How do I disable 'Transfer-Encoding: chunked' encoding in Varnish?

Question

Using Varnish 4, I have a set of backends that're responding with a valid Content-Length header and no Transfer-Encoding header.

On the first hit from a client, rather than responding to the client with those headers, Varnish is dropping the Content-Length header and adding Transfer-Encoding: chunked to the response. (Interestingly, the payload doesn't appear to have any chunks in it - it's one contiguous payload).

This causes serious problems for clients like Flash video players that are trying to do segment-size, bandwidth, etc analysis based on the Content-Length header. Their analysis fails, and they can't do things like multi-bitrate streaming, etc.

I've tried a number of semi-obvious things like:

• beresp.do_stream = true
• beresp.do_gzip = false
• unset req.http.Accept-Encoding

Sample backend response:

HTTP/1.1 200 OK
Cache-Control: public, max-age=600
Content-Type: video/mp4
Date: Tue, 13 May 2014 19:44:35 GMT
Server: Apache
Content-Length: 796618
Connection: keep-alive

Sample varnish response:

HTTP/1.1 200 OK
Server: Apache
Cache-Control: public, max-age=600
Content-Type: video/mp4
Date: Tue, 13 May 2014 23:10:06 GMT
X-Varnish: 2
Age: 0
Transfer-Encoding: chunked
Accept-Ranges: bytes

Subsequent loads of the object do including the Content-Length header, just not the first load into cache.

VCL: https://gist.github.com/onethumb/e64a405cc579909cace1

varnishlog output: https://gist.github.com/onethumb/e66a2bc4727a3a5340b6

Varnish Trac: https://www.varnish-cache.org/trac/ticket/1506

Confide

have the problem too, wait for update without bypassing the streamer. – Polston

Answer 1

For the time being, do_stream = false will do what you want.

Avoiding chunked encoding for the case where the backend sends unchunked is a possible future improvement to Varnish.

Example:

sub vcl_backend_response {
        if(beresp.http.Content-Type ~ "video") {
                set beresp.do_stream = false;
                set beresp.do_gzip = false;
                //set resp.http.Content-Length = beresp.http.Content-Length;
        }
        if(beresp.http.Edge-Control == "no-store") {
                set beresp.uncacheable = true;
                set beresp.ttl = 60s;
                set beresp.http.Smug-Cacheable = "No";
                return(deliver);
        }
}

Debose

Deleted my answer, yours is better – Creech

I tried set beresp.do_stream=false already, with the same result. :( – Confide

I should clarify, now that I've done more testing. beresp.do_stream=false improves the success rate, but errors still occur. I'm clocking at least a 10% rate of this problem even with do_stream set to false – Confide

@BrandonWamboldt Your answer is great as it delicately points out the relative code which could be helpful for newcomers (They might lack the required reputation to view the answer). Thanks and +1 Here anyway =) – Felicidadfelicie

@DonMacAskill Just curious, try enabling ESI per my answer and see if that works. I get 100% success rate. – Creech

I think beresp.do_stream only affects requests until the object is cached. Future responses are always chunked unless the origin server included a Content-Length header (ie. didn't chunk the response). – Pin

Answer 2

So the solution is not at all intuitive, but you must enable esi processing:

sub vcl_backend_response {
        set beresp.do_esi = true;

        if(beresp.http.Content-Type ~ "video") {
                set beresp.do_stream = true;
                set beresp.do_gzip = false;
                //set resp.http.Content-Length = beresp.http.Content-Length;
        }
        if(beresp.http.Edge-Control == "no-store") {
                set beresp.uncacheable = true;
                set beresp.ttl = 60s;
                set beresp.http.Smug-Cacheable = "No";
                return(deliver);
        }
}

So I discovered this by browsing the source code.

In particular, Varnish does this:

if (!req->disable_esi && req->obj->esidata != NULL) {
    /* In ESI mode, we can't know the aggregate length */
    req->res_mode &= ~RES_LEN;
    req->res_mode |= RES_ESI;
}

The above code sets the res_mode flag.

A little while later:

if (!(req->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
    /* We havn't chosen yet, do so */
    if (!req->wantbody) {
        /* Nothing */
    } else if (req->http->protover >= 11) {
        req->res_mode |= RES_CHUNKED;
    } else {
        req->res_mode |= RES_EOF;
        req->doclose = SC_TX_EOF;
    }
}

This sets the res_mode flag to RES_CHUNKED if the HTTP protocol is HTTP/1.1 or higher (which it is in your example) and the res_mode flag isn't set. Now even later:

if (req->res_mode & RES_CHUNKED)
    http_SetHeader(req->resp, "Transfer-Encoding: chunked");

Varnish sends the chuncked transfer encoding if the RES_CHUNKED flag is set.

The only way I see to effectively disable this is by enabling ESI mode. It gets disabled in a few other ways, but those aren't practical (e.g. for HTTP HEAD requests or pages with a 304 status code).

Creech

if (!(req->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) checks whether res_mode flag has either RES_LEN, RES_CHUNKED or RES_EOF set, but the code inside if (!req->disable_esi && req->obj->esidata != NULL) only unset RES_LEN and set RES_ESI. And according to here, the Content-Length will be stripped out eventually? – Felicidadfelicie

Answer 3

Upgraded from varnish 4.0 to 5.2 and now this works correctly also for the 1st request.

Thirtytwo