Restlet streaming data

Question

I have this task that I'm undertaking where I would be reading data from a device and make it available over a web service. The data is read 4 times a second. I want the web clients to be have an open HTTP connection and get the device readings as a stream using chunked transfer as long as the client keeps the connection open.

As a proof of concept, I want to start with a service that constantly generates a random number, 4 times a second, wraps it in json and stream that to clients. I'm trying to model it loosely based on twitter streaming api.

I'm using restlet 2.1.2 to create that webservice but I'm not sure which Representation I should be using to achieve this. I tried searching for this but didn't find anything useful. Could someone point me in the right direction as to what I should be using and maybe some examples, perhaps.

Thanks

Osteoid

Do you mean what SubType of Representation should you use? JsonRepresentation should be the one: restlet.org/learn/javadocs/snapshot/jse/ext/org/restlet/ext/… One of its constructors is handy: JsonRepresentation(Map<String,Object> map) – Moreno

Are you bound to Restlet? I don't think it supports what you need out of the box without some hacking. There's a web service library that handles async/steaming - Atmosphere. – Suitcase

@DiegoAlcántara JsonRepresentation cannot be used to stream dynamic content, I don't think. – Osteoid

@Suitcase Thanks for your response. unfortunately, I am bound to restlet. I did come across the hack you posted but did not have any luck gettin it to work with Thread.sleep() in the new Thread. – Osteoid

@ProfessorChaos what kind of client technology do you want/need to use to push and get data? F.ex. I don't think that you can consume twitter style streaming API with browsers' javascript+XHR. – Suitcase

@Suitcase The client right now is an android app. I plan to have a html5 style web app in the future. If I try twitter api's url with some hashtag tracking it streams in the browser. It shows the json representation of the tweets as they come in. Browser shows loading but no content comes through when there are no tweets. I was wondering if that's possible do with restlet. – Osteoid

@Suitcase This post explains the difficulty I'm having with what I've tried so far. – Osteoid

Answer 1

To achieve what you are trying to do, I'd use the WriterRepresentation (but see my answer to your other question), but I'm quite sure that you are going in the wrong architectural direction.

Indeed the following image from the documentation you linked

shows how even the Twitter streaming api is not intended to be connected by users, but by background processes that download messages in a store accessible by the HTTP. Users poll only the HTTP server, that reads the messages from the store and sends the back to the clients.

As a disconnected protocol, HTTP enable massive scalability that would not be possible otherwise. If each client establishes a persistent TCP connection backed by a dedicated server thread, you will rapidly exaust server resources! Moreover any HTTP proxy between the User Agent and the server could cause unexpected behaviours.

Thus, if you are bound to the HTTP protocol, the User Agent should poll. You can reduce the network load with headers like Last-Modified/If-Modified-Since or Etag/If-None-Match.

However, if you can adopt a different protocol, I strongly suggest to try a service bus over a connected TCP protocol.

Dead

I have a very unique requirement and hence I'm down this path. The webservice will only every have 1 to 2 clients connecting, and just one for the most part so scalability is not a concern. – Osteoid

Still, HTTP proxies could become a problem. BTW, what's exactly the problem you are facing? In particular, why polling is not an option? – Dead

It's like,I am trying to create a server around an already existing client. The server that used to exist was written using a different technology and I do not have the source base to see how it was implemented. Polling was not used because the connection establishment and closing was considered an overhead but now this while loop is not proving to be any better but it is what it is and i'm trying to see if I can make the restlet service work. – Osteoid