Efficient reloading data / pushing data from server to client

Question

I'm looking for the 'way to go' (i.e. the most efficient, most used, general accepted way) when it comes to the reloading of data from a web server to a front end. In the end application, I will have several output fields where data has to be written to, for example like this:

The data streams will be different from each other in the end application. The lines will have to be reloaded with fresh, up to date data from the server.

I have been thinking of using Ajax requests to update like every second, but there has to be an other way to do this. Ajax requests will cause a lot data traffic. Also, when using the Facebook chat, you don't have to wait every second, chats are received almost instantly. Yet I don't see any Ajax polling requests being made when I use the developer tools of Mozilla Firefox. This made me think if there would be a different way to do this.

I've looked into Node.js, but it appears that isn't possible with my host.

I have heard people talking about Ajax Push, is that what I should use? If so, can you give me a basic usage example?

If not, what would then be the way to go when having multiple data streams that have to be reloaded within a second?

Requirements are speed and low data traffic. It therefore wouldn't be an option to continuously poll the server, I think, because that would create an enormous overhead.

^{I don't think it's of any importance, but I'm using PHP5.3 in the back end and JavaScript with jQuery 1.9.1 in the front end.}

Byrnie

Answer 1

This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:

• What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
• Using comet with PHP?
• Apache with Comet Support
• Server Scalability - HTML 5 websockets vs Comet
• How to implement event listening in PHP

In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).

HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.

Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.

Also see the realtime web technology guide.

Potpie

+1 Tracking down this information took me a while for a project I recently released. I had zero web development experience and was surprised to learn that pushing data from web server to browser was so uncommon. – Pithead

Answer 2

I think what you are looking for is generally called Comet. The was this technique is often made to work is as follows:

• The client (web browser) makes a request to the server for new data. This is not reloading the page, but rather is done in JavaScript
• The server responds to the request when it has some data for the client. Again, this doesn't impact the UI since it isn't the page itself that's getting reloaded: the loaindg of data is done "in background" so to speak, in JavaScript code.
• On the serve side, the request waits for new data, and returns the new data when available, or returns nothing if a timeout interval (defined on the server) is reached. This timeout is usually set to be lower than the browser HTTP timeout. The reason for this is so that the server can know whether a particular client got a particular piece of data. If the request is allowed to time out on the client side, the original request might be responded to by the server after the client has timed out, and the client will not get the data, even though the server thinks that it did.

The data is indeed usually transferred as JSON, but you can choose whatever encoding you'd like. See here for one example of how to do this. Goosh is another example of this technique, and so is Interactive Python Shell. The code for all is available.

---

On the PHP side you will want to create a page that will respond to these "background" JavaScript Comet requests. It could be the same page as the one that user loads, but let's say it is different, for ease of explanation. So the user loads index.php and the JavaScript Comet code calls getNewData.php to retrieve new data.

In your getNewData.php you will want to wait for your event and return the data then. You don't want to use polling for this, but there are PHP libraries that allow one to use various interprocess communication strategies to wait on events, see this question for instance. The high-level pseudocode for your getNewData.php would look as follows:

1. parse JSON request
2. Enter an efficient wait state (with timeout), waiting for your "new data is available" event
3. Did previous step time out?
   Yes: send response indicating no data
   No: send response with new data

Dagger

Whilst this answer is technically correct, it's recommending an out-dated and inefficient solution, better solved by more recent technologies. Comet is an umbrella term for HTTP hacks to achieve the sort of functionality requested. HTML5 introduced Server-Sent Events and WebSockets so that these hacks were no longer the first port of call to solve these types of problem. Additionally, a long-polling approach (as outlined in this answer), on a PHP server, will be highly inefficient. – Potpie