python - how can server initiate a connection to client

Question

So I created an app where the app connects to the server and tells it to do something. However the server takes a lot of time to do that thing (like an hour or so). So it doesn't make sense to connect the client to the server for that long. I want that the server must create a connection to the client to inform that he is ready to deliver the end result. Then after connection, he sends the data to the client.

I am thinking two ideas here:

1. I can create a server on the client side also, and client can listen for server to connect. However my clients are usually some desktop apps or mobile apps with limited resources and some of them also are behind a firewall/NAT. So even if they have a server running, I am not sure that my main server will be able to connect to them.

2. The client can keep checking periodically whether the results are prepared or not. This way the client does not have to maintain any server or anything. It will be the plain old client server architecture. But this costs me a lot of un-necessary traffic, as the result may be available within few minutes to few hours. A client constantly pining the server does not seems too good to me.

I first of all have no idea where to start or google this stuff. I know such a thing exists because I see all the time where servers such as "Skype" informs my desktop app that I have a new friend request, or gmail notification system where Google shows a message on my desktop when new mail is received.

Most google searches are yielding the same stuff where the client connects to the server. But the case is opposite here.

If this question is not upto standard, a few references would also be great, and I will delete this post. There's a great chance that I am just not thinking something and being stupid. Please help. Thanks.

Organic

Answer 1

So after I researched a lot of related articles, here is what I have come up with.

These links describe what is called a "server push" which basically pushes data to the client from server. There are a variety of methods discussed here. My point #2 was actually very near to the solution, and the term for point #2 is called long polling.

1. http://en.wikipedia.org/wiki/Push_technology
2. http://mrjoes.github.io/2013/06/21/python-realtime.html
3. http://flask.pocoo.org/snippets/80/

But methods discussed in both of these articles are about pushing the data back to the web browser. In some cases, the solution can be extended to server push in mobile clients, but I am still reading on them, so can't comment much.

However, for server to app push, which is kind of what my requirement was, two things really stood up:

1. Redis Pub/Sub: The functionality of pub/sub channels in redis is wonderful. All you have to do is subscribe the clients to the channel in server, and anytime the server publishes something, the clients automatically listen to that. Redis is widely used, is well maintained, is easy to use, and is designed for very large scale applications. I liked that.
2. Amazon SNS. Now people like me (I forgot to mention in the question, that I am using amazon as my servers) using amazon aws, this feature SNS (Simple Notification System) is designed entire for server to app push. Its cheap, easy to implement and comes with python libraries to use. So if you don't want to get in redis, just use this simple tool by amazon to push data to your clients. It also supports to push data directly to SQS (Simple Queue Service), if you ever need it.

I have currently decided to go with Amazon SNS and will also keep learning Redis side by side, just for future use.

Update:

Need server push? Forget everything else and use Redis + Socket-IO + Node-JS. How did I not know about Websockets. Its the next best thing.

In my app, I just send any messages to Redis that I want to be pushed to clients. Redis publishes those messages. Socket.IO is listening on the channels using psubscribe to listen on patterns rather than one channel. Category of user is defined by the channel name.

Once socket.io (written in node.js) hears about the message, it simply emit those messages to the client. Clients inturn are listening to socket.io on a specific port. Socket.io supports rooms like channel in redis.

For live examples just google real time chat with socket.io and node.js.

I will also write some examples if anybody requests me to. Its real easy.

Organic

Answer 2

In most cases it is as you say:

2. The client can keep checking periodically whether the results are prepared or not. This way the client does not have to maintain any server or anything. It will be the plain old client server architecture.

Because it is easy to implement.

Another thing you can try is UDP-hole-punshing. It is not much code but also needs a keep alive signal, I guess. Then the server can send to the client.

A third idea is from pwnat. It works over IP. The client sends out a predefined packet to an address that will not be reached. This allows anyone on the internet to answer if the excact packet is known. The sending and receiving the packet and the reply require IP-level access which often requires higher privileges.

Also you can use UPNP to configure the clients firewall to let through some packets.

Jurassic

pwnat and UPNP is not something I had in mind, though nice to know. UDP-hole-pushing looks promising, but then again, I found a similar article describing this process of keeping connection alive (see my answer), and its already implemented. Thanks anyways. Good to know a few more stuff. – Organic