polling a HTTP server from J2ME client

Question

I have a J2ME app running on my mobile phone(client),

I would like to open an HTTP connection with the server and keep polling for updated information on the server.

Every poll performed will use up GPRS bytes and would turn out expensive in the long run, as GPRS billing is based on packets sent and received. Is there a byte efficient way of polling using the HTTP protocol?.

I have also heard of long polling, But I am not sure how it works and how efficient it would be.

Actually the preffered way would be for the Server to tell the phone app that new data is ready to be used that way polling won't be needed to be done, however I don't know of these techniques especially in J2ME.

Onega

Answer 1

If you want solve this problem using HTTP only, long polling would be the best way. It's fairly easy. First you need to setup an URL on server side for notification (e.g. http://example.com/notify), and define a notification protocol. The protocol can be as simply as some text lines and each line is an event. For example,

  MSG user1
  PHOTO user2 album1
  EMAIL user1
  HEARTBEAT 300

The polling thread on the phone works like this,

1. Make a HTTP connection to notification URL. In J2ME, you can use GCF HttpConnection.
2. The server will block if no events to push.
3. If the server responds, get each line and spawn a new thread to notify the application and loopback to #1.
4. If the connection closes for any reason, sleep for a while and go back to step 1.

You have to pay attention to following implementation details,

1. Tune HTTP timeouts on both client and server. The longer the timeout, the more efficient. Timed out connection will cause a reconnect.
2. Enable HTTP keepalive on both the phone and the server. TCP's 3-way handshake is expensive in GPRS term so try to avoid it.
3. Detect stale connections. In mobile environments, it's very easy to get stale HTTP connections (connection is gone but polling thread is still waiting). You can use heartbeats to recover. Say heartbeat rate is 5 minutes. Server should send a notification in every 5 minutes. If no data to push, just send HEARTBEAT. On the phone, the polling thread should try to close and reopen the polling connection if nothing received for 5 minutes.
4. Handling connectivity errors carefully. Long polling doesn't work well when there are connectivity issues. If not handled properly, it can be the deal-breaker. For example, you can waste lots of packets on Step 4 if the sleep is not long enough. If possible, check GPRS availability on the phone and put the polling thread on hold when GPRS is not available to save battery.
5. Server cost can be very high if not implemented properly. For example, if you use Java servlet, every running application will have at least one corresponding polling connection and its thread. Depending on the number of users, this can kill a Tomcat quickly :) You need to use resource efficient technologies, like Apache Mina.

I was told there are other more efficient ways to push notifications to the phone, like using SMS and some IP-level tricks. But you either have to do some low level non-portable programming or run into risks of patent violations. Long polling is probably the best you can get with a HTTP only solution.

Aardwolf

Fine answer!, However I did not understand regarding the following: 1. What does it mean that The server will block if no events to push. 2. What is HEARTBEAT? 3. Does the Tomcat have a limit for number of active users. – Onega

Answers: 1. When nothing to push, you want keep connection open and waiting for an event. Check out BlockingQueue. 2. Heartbeat is an administrative packet to tell the phone that server is alive. You should send this if idle too long to avoid phone going into stale connection recovery mode, which is more expensive. 3. For servlet like Tomcat, each HTTP request is a thread. The number of threads equals number of running applications. Even though the max is configurable, the server will not perform if the number is too high. I will try to stay in hundreds, not thousands. – Aardwolf

Answer 2

I don't know exactly what you mean by "polling", do you mean something like IMAP IDLE? A connection stays open and there is no overhead for building up the connection itself again and again. As stated, another possible solution is the HEAD Header of a HTTP Request (forgot it, thanks!).

Look into this tutorial for the basic of HTTP Connections in J2ME.

Pushing data to an application/device without Push Support (like a Blackberry) is not possible.

Lardaceous

Polling in this sense would be querying the server or rather asking it at fixed intervals. For instance the client would ask in a more general sense "is my data ready?" or "is anything changed since we last spoke?" The query would be done in a technical sense and a language that the server would understand. – Onega

HEAD doesn't send any HTTP content back (no body) but it's quite expensive if the goal is to save GPRS traffic. Lots have to happen under the rug (3-way TCP handshake, request and response with all the headers). If your message is small, it may make no difference from GET. – Aardwolf

Answer 3

The HEAD HTTP request is the method that HTTP provides if you want to check if a page has changed or not, it is used by browsers and proxy servers to check whether a page has been updated or not without consuming much bandwidth.

In HTTP terms, the HEAD request is the same as GET without the body, I assume this would be only a couple hundred bytes at most which looks acceptable if your polls are not very frequent.

Bryce

Answer 4

The best way to do this is to use socket connection. Many application like GMail use them.

Buber

Do you have any references? Maybe some AJAX code you can show us that does this? – Halftruth

I've tried to make a sample app: Midlet is listening to server (i've used example written in Ruby). You can minimize application by pressing "0". If some message is received, application popup and show it. depositfiles.com/files/68ix6gskv (sorry, I dont know how to attach files) – Buber