Always send Content-Length in Apache?

Question

I'm loading a particularly large JSON string that is dynamically generated by PHP. To provide some feedback to the user, I want to show download progress.

I have the code figured out, and it works fine for static content such as images, JS files, etc. However, it doesn't seem to work for dynamic files.

This makes sense, since the dynamic files don't have predictable content length, but even if I add this in PHP:

ob_start(function($c) {
    header("Content-Length: ".strlen($c));
    return $c;
});

It still does not send the header (but if I add any other header it works fine).

Is there any way to force Apache to send the Content-Length header? Currently my only alternative is to save the output to a temporary file and redirect to it instead. This would work, but it's kind of messy so I'd rather avoid it if possible.

Evaluate

Possibly helpful: #1334971 – Monoclinous

... to be specific, the transfer encoding note in the Answer. – Monoclinous

Thanks for the pointer. It doesn't look like it's using chunked encoding (at least, Transfer-Encoding doesn't appear in the response headers) and I can't seem to get it to force HTTP/1.0... – Evaluate

Have you tried with a simple ob_start(); /* output json */ $content = ob_get_clean(); header('Content-Length: '.strlen($content)); print($content);? While it would be unexpected that your ob_start callback does not get executed, you can certainly expect anything unexpected from PHP. – Lucknow

What is the dynamic file extension? – Castigate

The request is similar to /ajax/getdata, the file is getdata.php. – Evaluate

Is there some public URL where one could try this out? (The ob_start-with-callback one, that is?) That it would send any header but Content-Length is really strange. – Mothball

so, maybe there is something in the Apache conf that overrides headers for php files? – Castigate

Could you post the output of curl -I <url>? – Arbitral

Are you using error_reporting(-1)? Are you sure you can still send headers at that point? Did you try what lanzz said? – Gravitt

Answer 1

I had similar problem but in my case Content-Length header wasn't sent by Apache because the response was gzip compressed. When I disable compression the Content-Length was calculated and sent properly.

Below is htaccess setting to disable gzip compression for swf file only

<FilesMatch "\.swf$">
  SetEnv no-gzip 1
</FilesMatch>

Remarkable

Answer 2

My best guess is that you have some changes in the modules/http/http_filters because Apache by default sends Content-Length.

Castigate

Interestingly, I just wrote a quick PHP script to retrieve the file and the Content-Length header is there. But in the browser xhr.getResponseHeader("Content-Length") returns null and it doesn't show up in the console either. – Evaluate

@Kolink What headers do you have in Google Chrome in Network tab? (when you inspect that call). And be sure to look at response not request headers. – Castigate

Sure enough the header is there. Okay, so why is IE10 not listing or acknowledging the Content-Length headers in some calls, but is in others??? – Evaluate

@Kolink It looks like another problem right now. What are you using to handle XHR call, jQuery? BTW I think that IE is still really unpredictable browser (like everything from Microsoft), PHP is dangerous. And Apache, well... It was nice years ago, maybe it's time to try something faster like based on event-loop Nginx? – Castigate

No jQuery, just plain XMLHttpRequest. It works fine for static files, which is the weird thing... – Evaluate

@Kolink IE is weird and you are brave using plain JS against IE :- ) Can you try the same thing using jQuery? (or any cross-browser library) – Castigate

I adamantly refuse to use any library. They cause more unpredictable behaviour than they solve. See also this image I made. – Evaluate

@Kolink I disagree, but it's for long discussion. Can you please use jQuery and if it works then we can look how jQuery handles that issue and you can extract vanilla JavaScript code from it. – Castigate

Not happening. Sorry. Even if it did work, how would it help? – Evaluate

@Kolink If jQuery works then we can have a look at its internals and see how it handles bug in IE. Just to make sure we are on the same page, CL in the headers is send but only IE has trouble to see it? – Castigate

Answer 3

The HTTP protocol needs a way to determine when a response has ended. According to

RFC2616 Section 4

4.4 Message Length

The transfer-length of a message is the length of the message-body as it appears in the message; that is, after any transfer-codings have been applied. When a message-body is included with a message, the transfer-length of that body is determined by one of the following (in order of precedence):

1.Any response message which "MUST NOT" include a message-body (such as the 1xx, 204, and 304 responses and any response to a HEAD request) is always terminated by the first empty line after the header fields, regardless of the entity-header fields present in the message.

2.If a Transfer-Encoding header field (section 14.41) is present and has any value other than "identity", then the transfer-length is defined by use of the "chunked" transfer-coding (section 3.6), unless the message is terminated by closing the connection.

3.If a Content-Length header field (section 14.13) is present, its decimal value in OCTETs represents both the entity-length and the transfer-length. The Content-Length header field MUST NOT be sent if these two lengths are different (i.e., if a Transfer-Encoding

 header field is present). If a message is received with both a
 Transfer-Encoding header field and a Content-Length header field,
 the latter MUST be ignored.

4.If the message uses the media type "multipart/byteranges", and the transfer-length is not otherwise specified, then this self- delimiting media type defines the transfer-length. This media type MUST NOT be used unless the sender knows that the recipient can parse it; the presence in a request of a Range header with multiple byte- range specifiers from a 1.1 client implies that the client can parse multipart/byteranges responses.

   A range header might be forwarded by a 1.0 proxy that does not
   understand multipart/byteranges; in this case the server MUST
   delimit the message using methods defined in items 1,3 or 5 of
   this section.

5.By the server closing the connection. (Closing the connection cannot be used to indicate the end of a request body, since that would leave no possibility for the server to send back a response.)

For compatibility with HTTP/1.0 applications, HTTP/1.1 requests containing a message-body MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant. If a request contains a message-body and a Content-Length is not given, the server SHOULD respond with 400 (bad request) if it cannot determine the length of the message, or with 411 (length required) if it wishes to insist on receiving a valid Content-Length.

All HTTP/1.1 applications that receive entities MUST accept the "chunked" transfer-coding (section 3.6), thus allowing this mechanism to be used for messages when the message length cannot be determined in advance.

Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non- identity transfer-coding, the Content-Length MUST be ignored.

When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. HTTP/1.1 user agents MUST notify the user when an invalid length is received and detected.

You can't just randomly add headers and expect them to be obeyed (other headers can override). You need to control all possibly overriding headers generated in the first place.

According to this question (How to make PHP generate Chunked response) a good way to force "Chunked" is to set Transfer-Encoding and to flush. Maybe those two aren't strictly needed. Would there be any extraneous flushes anywhere before you start buffering?

Bayne

Answer 4

From the ob_start documentation:

This function will turn output buffering on. While output buffering is active no 
output is sent from the script (other than headers), instead the output is stored 
in an internal buffer.

Note the "other than headers" bit -- the ob_start() callback is not called until the buffer is being flushed (or thrown away), at this point it's too late to use header(). I'm guessing you don't have error logging turned on, I see this in my error log:

PHP Warning:  Cannot modify header information - headers already sent 
  in /usr/local/apache2/apps/testing/test2.php on line 6

The callback lets you modify the buffer before sending, but as this contains only body data, you also cannot prepend a new header (it will appear in the content if you try, once for each chunk for chunked transfers).

Your code should call instead header("Content-Length: ...") once you know the size, before any other output. When a Content-Length: header is found, chunked transfer will not take place.

If you really need to force HTTP/1.0 (you don't), you can do it in httpd.conf with the special variables:

SetEnv downgrade-1.0 1
SetEnv force-response-1.0 1

You can put those in <Location> or <LocationMatch> for example, or use mod_rewrite's "env" flag for more control.

Incursion

Answer 5

I had a few JS files that were around 1MB that didn't get Content-Length header (unlike other, smaller JS files). The fact that Content-Length header is not mandatory with HTTP/2 and having a somewhat flaky Internet connection resulted in a very hard to debug error where the files were sometimes delivered only partially. The solution was to increase DeflateBufferSize https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#DeflateBufferSize

Enwind