Compressing HTTP Post Data sent from browser

Question

I want to send a compressed POST data with Javascript to a server I control. Is there any way to let the HTTP layer deal with the compression.

I'm sending JSON. If I set the content type to GZIP/deflate will the browser automatically compress it and then Apache with the deflate mod automatically decompress it so my application doesn't have to think about the data being compressed at all?

I know it can work the other way around but any way to make it work this way?

Fernandofernas

Answer 1

Will the browser automatically gzip-encode your data for you? The short answer is no.

The long answer is that some user-agents can do things like this, but you definitely can't rely on it. The apache mod_deflate docs state:

some special applications actually do support request compression, for instance some WebDAV clients.

So, no, that's not going to work. You'll need to generate the appropriate HTTP request message yourself. The appropriate header in this case is Content-Encoding: gzip and NOT Content-Type: because the content itself is application/json, you're just looking to encode the entity body of your HTTP request message for transport.

Note that you need to also add the appropriate Content-Length: header specifying the size in bytes of the message entity body after compression -OR- send your HTTP message using Transfer-Encoding: chunked and forego the content-length specification.

On the receiving end, you can instruct mod_deflate to use an input filter to decompress the information:

<Location /dav-area>
SetInputFilter DEFLATE
</Location>

This is a bit heavy handed if you're only receiving compressed message bodies for a couple of resources. Instead, you should probably just use the client-side script to check for the Content-Encoding: gzip header and decompress the request body manually. How to do this in say, PHP, is another question entirely. If you need details for that you should post another question.

Lenticular

I would add that if you're sending small bursts of JSON text, the processing overhead needed to compress the data will likely outweigh the smaller transfer size benefits. If you're talking about streaming binary files in your requests it's another story. But for JSON? Probably not worth your time. – Lenticular

I figured as much that I couldn't do it but I said I'd ask anyhow. I've a version working where I compress one of the JSON fields and decompress in the application(Sinatra). I think this maybe be the best way to do it for my situation. The information in that field is a decent size so compressing is useful. Thank you – Fernandofernas

Note that "HAAHAHAAahahahahahhaha." is very compressible. – Ian

@MarkAdler Ha. I thought that was part of the answer. – Hannon

@MarkAdler unfortunately the PC police have removed that part of the answer. I guess it wasn't welcoming enough. – Urnfield

Answer 2

It's possible, but I would recommend strongly against accepting gzipped data incoming to your server. The main reason is to prevent your server from getting gzip bombed. It's generally not possible to know what the uncompressed data looks like before actually uncompressing it, so a user could send you a web request that looks like a harmless 1 KB or 1 MB of data but is really 100 GB of data, and then your web server (nginx or apache) hangs the next 10 minutes trying to decompress it all, eventually running out of memory/locking up.

Trinitrobenzene

But if you have "Note that you need to also add the appropriate Content-Length: header specifying the size in bytes of the message entity body after compression", then why your server cannot check this size first (primary filter), and then instruct decompressor to reject data expansion to more than to the expected size (secondary filter)? – Constantino

@Constantino So essentially check both compressed and uncompressed length? If so, then it’s simply a balance between security and functionality. With a higher allowed compression ratio (or longer time-out), the server is more vulnerable to DoS. With a lower allowed compression ratio (or shorter time-out), the compression is less useful. Not worth the hassle in most use-cases. Special case can be covered in application-level compression. – Pathoneurosis

@Constantino the Content-Length of compressed resource must be the compressed size. There's no info about the uncompressed size in the request. (in other words, Content-Length is "network length in bytes") – Shrewsbury

The server should have a max uncompressed length check. If during decompression, the decompressed stream goes over that length, immediately the server would drop connection and free all resources. – Valenzuela

There are a lot of good security reasons not to compress request bodies, not just GZIP bombs. Most firewalls will not decompress the request body when scanning for exploits and injection or DoS attacks, etc. Just don't do it. :) – Prestigious

Answer 3

Just achieved this using https://github.com/dankogai/js-deflate However the postdata for whatever reason will strip the + signs and replace them with spaces.

To send the data via javascript:

params.mapdata=  btoa(RawDeflate.deflate(JSON.stringify(mapdata)));

To receive the data via php:

$value = gzinflate(base64_decode(preg_replace('/\s/', '+',$value)));

Dosh

POST data must be URL-encoded, thus + should be sent as %2B. + is the old encoding for white space. – Rubin

this seems wrong. btoa expands the data 30% - so you're compressing text to binary, then expanding binary back to text. This will probably always make the transport bigger. – Bledsoe

Not if your compression is better than 30%, which seems reasonable depending on the payload. – Raptorial

POST data doesn't have to be encoded, you can get the raw payload with php://input. For Node.js, it gives you the raw payload buffer by default. – Chickadee