Sending binary data over http

Question

I am looking for suggestions on the best way to send/receive data from a remote GPRS device, over port 80.

Creating a plain TCP socket on a random port works fine, but many carriers only allow port 80 HTTP traffic through their proxies, and then expect HTTP ascii data (for which they can modify headers as needed.

So, should my device create a POST request on a persistent http connection, and then receive a base64 encoded response from the web service? I am not sure how mobile proxies behave when binary data is involved. Is there a recommended way to do this?

I can adapt both device's firmware and the server side app.

[Edit]

I would like to know if there is a standard (more or less) way to do this. For various data logging and industrial systems, there is a need to send lots of binary data over a socket connections. For Ethernet connections, there are usually only problems involved with adapting some firewalls, but persistent binary connections have no trouble being established over arbitrary ports.

Mobile ISPs, however, tend to limit their "data plans" for port 80 only. They also take the liberty to mess with HTTP headers, and potentially the HTML data itself. This is where I need to identify potential pitfalls and ways to circumvent them.

• Will simply sending base64 encoded data work?
• How are the HTTP sessions handled? Arbitrary sockets can be kept alive for a long time, but HTTP verbs are usually short lived. Does this mean I will need to create a new connection for each packet of data? Or is there a way to send server responses in chunks, over a single connection?
• In what ways can an ISP proxy mess with the data, or the headers? For example, a proxy can sometimes keep a connection alive, even if the server closes it.

Group

Answer 1

Will simply sending base64 encoded data work?

There is no need to use base 64 encoding - this will simply increase the number of bytes you must transfer. Mobile operators normally limit mangling of responses to content types that they understand - i.e. images, stylesheets, etc.

How are the HTTP sessions handled?

HTTP sessions are normally handled either via a URL query parameter or via a cookie value. However, from what you have said it doesn't sound like sessions are necessary.

Arbitrary sockets can be kept alive for a long time, but HTTP verbs are usually short lived. Does this mean I will need to create a new connection for each packet of data?

HTTP requests can last for an arbitrarily long period of time, just as for raw TCP sockets. A GET request can last for hours if necessary. You need not create a new connection for each request — take a look at the Connection: Keep-Alive HTTP header.

Or is there a way to send server responses in chunks, over a single connection?

If you don't know the length of the response you can either omit a Content-Length header or, preferably, use the Transfer-Encoding: chunked HTTP header.

In what ways can an ISP proxy mess with the data, or the headers? For example, a proxy can sometimes keep a connection alive, even if the server closes it.

ISPs don't tend to reveal the changes they make to HTTP responses. If you are concerned about this a simple solution would be to encrypt the data and specify a Content-Encoding HTTP header. This would require you to control both the HTTP client and server.

Eurus

+1 Thanks. One tiny clarification: Mobile operators normally limit mangling of responses to content types that they understand - does that mean I should use a content type that they don't understand? – Group

The transformations made by mobile operators are typically for media-types where they know an 'equivalent' response can be returned. Examples would be compression of a JPEG image or inlining of CSS/Javascript into a HTML file. If you are sending data in a proprietary format you would use a media-type like application/vnd.company-name.file-type - this is unlikely to be transformed by intermediaries since they don't understand the format. – Eurus

@Eurus The Content-Length header specifies the length of any data in bytes. So, regardless of whether the data is standards compliant or proprietary, the operators can just forward those many bytes. – Irita

Answer 2

If possible, you could just send the data as HTTP requests and responses.

HTTP is perfectly capable of handling binary data: images are sent over HTTP all the time, and they're binary. People upload and download files of arbitrary data types all the time with no problem.

Just give it a mime type of "application/octet-stream" -- which is basically a generic mime type for binary data with no further specification of just what sort -- and any proxies along the way should leave it alone.

Liable

Answer 3

ASP.NET C# implementation of Uploading binary data like images as POST request to target URL:.

http://technowide.net/2012/09/01/upload-binary-data-http-post/

Pyromania