HTTP headers for chunked encoding POST - Error 411

Question

I'm sending data to a server with an Arduino which requires constructing an HTML POST line-by-line. I don't necessarily know the Content-Length a-priori, so I am using "chunked" encoding.

When I tried this example post from Wikipedia with the "Transfer-Encoding" option as specified in rfc2616

client.println("POST /myurl HTTP/1.1");
client.println("Host: 12.345.679.999"); // replaced with the test server's IP
client.println("User-Agent: Arduino/1.0");
client.println("Transfer-Encoding: chunked");
client.println();  
client.println("4");
client.println("test");
client.println("0");
client.println();

or, with escape characters explicit:

client.print("4\r\ntest\r\n0\r\n\r\n");

I receive the error from my server:

HTTP/1.1 411 Length Required
A request of the requested method POST requires a valid Content-length.
Server: Apache/2.2.22 (Ubuntu)

However, "chunked" encoding shouldn't require a Content-Length header field, see 4.4 - Message Length in rfc2616

Am I missing a field? Why doesn't this call work?

For the record, the non-Chunked-Encoding works:

if(client.connect(server, 80)){
    String PostData = "test";
    Serial.println("POST /myurl HTTP/1.1");
    client.println("Host: 12.345.679.999"); // replaced with the test server's IP
    Serial.println("User-Agent: Arduino/1.0");
    Serial.print("Content-Length: ");
    Serial.println(PostData.length());
    Serial.println();
    Serial.println(PostData);
}

UPDATE

From the apache2 error.log: "chunked Transfer-Encoding forbidden"

Meetinghouse

No, you are not doing anything wrong. It doesn't work because whomever wrote the server decided to require a Content-Length header for requests containing an entity body. Parsing request messages is more complicated if you allow chunked entity bodies because you have to parse the data as it arrives whereas if you know the length you can just read that number of octets. If the server has an option setting to disable the Content-Length requirement your request should work as expected. Otherwise, you're up the creek as the "1.1-compliant" server doesn't correctly handle chunked requests. – Couturier

Related: what server software is listening for the request? – Couturier

Thanks. Amazon EC2 server on Apache/2.2.22 (Ubuntu). So I guess I'll dig through the Apache httpd.conf file? – Meetinghouse

I haven't had problems sending chunked requests to apache servers in the past. Is your request line specifying POST /someurl HTTP/1.1 ? Because if you used 1.0 I would expect a 411 as chunked encoding was introduced with HTTP/1.1. Otherwise, you should just be able to do some googling to figure out what settings will convince apache to accept chunked requests. – Couturier

Yes, my request line is that. – Meetinghouse

Can you post a pastebin or something with the full raw request message? Also, do you have multiple virtual hosts running on this apache server? Because if you aren't including a Host: mydomain.com header it's possible that the request is being routed to another virtual host where content-length is required. Also, have you tried something basic like telnet to send a raw request that you know to be correct? – Couturier

I just updated the question with the rest of the header fields and the working non-chunked request. – Meetinghouse

Just dug out my apache error log: " chunked Transfer-Encoding forbidden" – Meetinghouse

Just get into this problem with Zend Server 6.1.0 which uses CGI/FastCGI for PHP and didn't found to the time any way to enable the chunked Transfer-Encoding. Apparently the problem isn't present when using mod_php as XAMPP apparently does. – Boatright

Answer 1

After finding

chunked Transfer-Encoding forbidden

in my Apache2 log I concluded that the error was not in the POST that I was making.

I found that modwsgi (the middle-layer between apache and django) does not enable chunked transfer-encoding by default. In the past, chunked wasn't supported at all

Refering to the change-log in the new version of modwsgi, I found that writing

WSGIChunkedRequest On

in my apache httpd.conf file allowed chunked requests (no more 411 error)

Meetinghouse

Ah, so WSGI was the culprit ... Glad you were able to work this out. – Couturier

With the above approach 411 is gone however I am getting 400 now. And if I disable the chunked transfer, it works as expected. – Vaas