How can I set Transfer-Encoding to chunked, explicitly or implicitly, in an ASP.NET response?

Question

Can I simply set the Transfer-Encoding header?

Will calling Response.Flush() at some point cause this to occur implicitly?

---

EDIT
No, I Cannot call Response.Headers.Add("Transfer-Encoding","anything"); That throws.

any other suggestions?

---

Related:
Enable Chunked Transfer Encoding in ASP.NET

Trueblood

In Java Servlet, if you do not set the Content-Length header before writing/flushing the stream, it will be implicitly set to Transfer-Encoding: chunked and the data will be sent in chunks. See if similar behaviour exist in ASP.NET. – Televisor

In ASPNET, it caches the output implicitly, I think. There's a BufferOutput property on the HttpResponse. The documentation is on the property is completely insufficient though. msdn.microsoft.com/en-us/library/… It doesn't say anything about how the property affects Transfer-Encoding, or any of the other interesting questions. – Trueblood

Answer 1

TL;DR: Specifying the content-length is the best way to achieve a fast first byte; you'll allow chunking at TCP rather than HTTP level. If you don't know the content-length, setting context.Response.BufferOutput to false will send output as it's written the the output stream using chunked transfer-encoding.

---

Why do you want to set Transfer-Encoding: chunked? Chunked transfers are essentially a work-around to permit sending documents whose content-length is not known in advance. ASP.NET, however, by default buffers the entire output and hence does know the overall content length.

Of course, HTTP is layered over TCP, and behind the scene TCP is "chunking" anyhow by splitting even a monolithic HTTP response into packets - meaning that if you specify the content-length up front and disable output buffering, you'll get the best latency without requiring HTTP-level chunking. Thus, you don't need HTTP-level chunking to provide a fast first byte when you know the content-length.

Although I'm not an expert on HTTP, I have implemented a simple streaming media server with seeking support, dynamic compression, caching etc. and I do have a reasonable grasp of the relevance of a fast first byte - and chunking is generally an inferior option if you know the content-length - which is almost certainly why ASP.NET won't let you set it manually - it's just not necessary.

However, if you don't know the HTTP content length before transmission and buffering is too expensive, you turn off output buffering and presumably the server will use a chunked transfer encoding by necessity.

When does the server use chunked transfer encoding? I just tested, and indeed if context.Response.BufferOutput is set to false, and when the content length is not set, the response is chunked; such a response is 1-2% larger in my entirely non-scientific quick test of a 1.7MB content-encoding: gzip xml document. Since gzip relies on context to reduce redundancy, I'd expected the compression ratio to suffer more, but it seems that chunking doesn't necessarily greatly reduce compression ratios.

If you look at the framework code in reflector, it seems that the transfer encoding is indeed set automatically as needed - i.e. if buffering is off AND no content length is known AND the response is to an HTTP/1.1 request, chunked transfer encoding is used. However, if the server is IIS7 and this is a worker request (?integrated mode?), the code branches to a native method - probably with the same behavior, but I can't verify that.

Ite

@Eamon, Chunked transfers are not a workaround, they are a feature. I think you know this, but when the size of the content is unknown and potentially large at the time the first response bytes are written, then it is incorrect, potentially dangerous, and will result in realy poor performance if ASPNET attempts to cache the entire response before sending it. Regarding the use of BufferOutput - Can you cite the source that you "just checked"? Do you mean you tested it? I observed that behavior as well. What I'm looking for is a documented description. Does Response.Flush() do it? etc. – Trueblood

I observed it. If you look at the code in reflector, turning off bufferoutput is effectively equivalent to calling flush after each write; and each flush that is not final checks and sets chunked transfer encoding if headers haven't been written yet, haven't been suppressed, the client isn't disconnected, this isn't the "final" flush, the response's content length isn't set manually, and the http version is 1.1 - with the caveat that there's some code that checks for IIS7 and does something else for that, which looks more complex. – Ite

Anyhow, the point is that if you're worried about buffering overlarge responses - turn off buffering; don't explicitly worry about chunked transfer encoding (which is simply the mechanism by which the response can be sent when buffering is disabled and the server can't otherwise deduce content length). You don't need to manually enable chunked transfer encoding, and as far as I can tell - there's no reason to. – Ite

@Eamon - you seem to think that an app should never care whether buffering or chunked transfer occurs. But that's not true. Suppose the data to be transferred is large, and the size is known. Let's say the app knows the response is going to be exactly 1gb. It makes sense for the app to explicitly make sure that no buffering occurs, which implies chunked encoding. You seem to overlook that possibility. Or consider time-to-first-byte, which is smaller with chunked transfer than without. Just two examples. The point is, there are good reasons to make explicit use of this feature of HTTP. – Trueblood

No, that's not true - an app may care whether it's buffered or not - but chunked transfer encoding is just one way that an app can be "unbuffered". If you explicitly set a content-length and turn off buffering, you still won't have chunked transfer, but will have low latency - in fact, it'll be slightly faster than chunked transfer encoding since the total response size will be smaller. So, I'm not saying an app shouldn't care about being unbuffered, I'm saying an app shouldn't care about how being unbuffered is implemented - chunked transfer encoding is not needed for low latency! – Ite

Thank you for explanation, it helped me to solve a little issue on how Google Chrome handles Ajax request that are returned (and not handled) in chunked transfer encoding. – Moneyer

"Thus, you don't necessarily need HTTP-level chunking to provide a fast first byte." This is stupid. I need it for a certain scenario, either answer to the question or don't bother! – Almemar

@DotNetWise: edited answer to emphasize what you need to do to minimize initial latency (which presumably is what you're actually after, and doesn't require transfer-encoding chunked.) – Ite

@DotNetWise: also, the answer includes how to force transfer-encoding chunked if you really need that: context.Response.BufferOutput = false. – Ite

Refering to your first line in bold... in case you are relying on chunking at the TCP layer, your client will not be able to get a fast first byte because the client is also talking HTTP to the server. In that case TCP at client will wait for the full response and then package it to the client as http response. – Compline

@Compline you're wrong; if you set a content-length and use normal transfer enconding (i.e. non-chunked), the browser can and will interpret the partial tcp stream as it comes in. It will not necessarily wait for the complete response. The browser isn't required to do anything with those partial responses; but that's not an http/tcp issue; that depends on the content being rendered (e.g. all browsers will play partial videos/audio with only a little buffering, but many won't display partial images because that's ugly and slow). – Ite

@EamonNerbonne, thanks, great answer, I've started a bounty to award it to you once it's possible (24H). – Boman

Answer 2

It looks like you need to setup IIS for this. IIS 6 has a property AspEnableChunkedEncoding in the metabase and you can see the IIS 7 mappings for this on MSDN at http://msdn.microsoft.com/en-us/library/aa965021(VS.90).aspx. This will enable you to set TRANSFER-ENCODING: chunked in your header. I hope this helps.

Kristakristal

thanks. But AspEnableChunkedEncoding is by default true, so that's not the problem. Also, this doesn't answer the specific question about use of chunked encoding within ASPNET. – Trueblood

Answer 3

Although you set Buffer to false and leave empty the content length, you need to make sure that you have disabled "Dynamic Content Compressing" feature for IIS7 to make chunked response working. Also, client browser should have at least HTTP 1.1 .. Chunked mode won't be working for HTTP 1.0

Hathorn

Answer 4

Response.Buffer = False

This will set HTTP Header "Tranfer-Encoding:Chuncked" and send the response each callled response.write

Chromatic