How to make PHP generate Chunked response

Question

I googled for this problem but there is no answer for it.

I want my PHP script to generate HTTP response in chunked( http://en.wikipedia.org/wiki/Chunked_transfer_encoding). How to do it?

Update: I figured it out. I have to specify Transfer-encoding header and flush it.

header("Transfer-encoding: chunked");
flush(); 

The flush is necessary. Otherwise, Content-Length header will be generated.

And, I have to make chunks by myself. With a helper function, it is not hard.

function dump_chunk($chunk)
{
    echo sprintf("%x\r\n", strlen($chunk));
    echo $chunk;
    echo "\r\n";
}

Conjunction

You can flush after each chunk. – Plossl

Would you explain how the chunks are gotten from the file? Please? ^_º – Domination

Be careful: Setting header("Transfer-encoding: chunked"); explicitly breaks PHP's native chunked encoding and it won't calculate and send the chunk lengths anymore! – Heterozygote

Answer 1

A PHP response will always be chunked if you don't specify a content-length header, and a flush occurs. (which will happen automatically after x bytes, just don't know exactly how much).

This is a weird thing to care about. Is this some kind of academic/learning exercise or is there a real world problem you're trying to solve?

Dinghy

I want to force chunked transfer encoding because I'm testing a ruby script which is supposed to be able to read it. So if I understand you correctly, I need only stick some flush(); lines in to run periodically, and that will force php to respond in Chunked Transfer Encoding. No need to explicitly output any chunk size numbers? – Blackbird

Apache should handle that for you by itself. Basically, if apache won't know the length in advance, it has no option but to use 'chunked' – Dinghy

Apache buffers a certain amount of output before sending. If your entire output fits in the buffer then it won't be chunked, unless you force it. – Knapp

Answer 2

This has gotten a little vauge... if you don't mind big assed chunks, (0x1000 octets or so), then yes, PHP will make them.

<?php

while (true) {
    # output data
    flush()
    usleep(pow(2,18));
}
?>

PHP will generated the numbered sections, etc.

If you want to send tiny little chunks, as you might do with an AJAX client... well, I've combined the OPs question, with some research on PHP.NET, and it does look like he was on to a good thing.

$ echo -en "GET /chunked/ HTTP/1.1\r\nHost: ec\r\n\r\n" | nc localhost 80

HTTP/1.1 200 OK
Date: Wed, 23 May 2012 13:03:01 GMT
Server: Apache/2.2.9 (Debian) PHP/5.3.5-1 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8o
X-Powered-By: PHP/5.3.5-1
Transfer-encoding: chunked
Content-Type: text/html

14
Teachers have class.
50
We secure our friends not by accepting favors but by doing them.
            -- Thucydides
48
Vulcans never bluff.
            -- Spock, "The Doomsday Machine", stardate 4202.1
31
All kings is mostly rapscallions.
            -- Mark Twain
41
Reappraisal, n.:
    An abrupt change of mind after being found out.
49
He who knows, does not speak.  He who speaks, does not know.
            -- Lao Tsu

Whether or not it will eventually squeeze out it out's own (incorrect) chunk count, remains to be seen... but I saw no sign of it.

<?php
header("Transfer-encoding: chunked");
@apache_setenv('no-gzip', 1);
@ini_set('zlib.output_compression', 0);
@ini_set('implicit_flush', 1);
for ($i = 0; $i < ob_get_level(); $i++)  ob_end_flush();
ob_implicit_flush(1); flush();

function dump_chunk($chunk)
{
  printf("%x\r\n%s\r\n", strlen($chunk), $chunk);
  flush();
}

for (;;) {
  $output = array();
  exec("/usr/games/fortune", $output);
  dump_chunk(implode("\n", $output));
  usleep(pow(2,18));
}
?>

Bloodthirsty

If your intent is to get rid of every output buffer, this way will only remove half of them (rounded up). Use while(ob_get_level()){ob_end_flush();} instead. If you want to avoid repetitively calling the function: for($i=ob_get_level(); $i; --$i){ob_end_flush();} or $i=ob_get_level()+1;while(--$i){ob_end_flush();} (use post-decrement if the +1 seems odd). Thanks for the netcat example, HTTP is far simpler than I expected, which is funny since I've seen <method> <path> <HTTP version>, headers, and bodies, but they're usually shown separately when I view them. – Easel

Answer 3

Working sample as of 16.02.2013

<?php

function dump_chunk($chunk) {
    //echo sprintf("%x\r\n", strlen($chunk));
    echo sprintf("\r\n");
    echo $chunk;
    echo "\r\n";
}

ob_start();
?>
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Title</title>
    </head>
    <body>
        <?php
        ob_end_flush();
        flush();
        ob_flush();
        for ($i = 0; $i < 5; $i++) {
            sleep(1);
            dump_chunk('Sending data chunk ' . ($i + 1) . ' of 1000 <br />');
            flush();
            ob_flush();
        }
        sleep(1); // needed for last animation
        ?>
    </body>
</html>

Mcclelland

For some reason the page only displays when the entire page is rendered instead of increasing. Do you know what may have been the problem? – Trickster

What configuration are you using? (Apache version, modules, php version)? – Mcclelland

PHP 5.3.26, apache 2.2.24 + nginx hosting24.com/blog/… provided by hosting24's silver package hosting24.com/features.php – Trickster

Try code on this apachefriends.org/en/xampp.html. If it works on XAMPP, then your problem may be because of nginx (see #3222507 or nginx.org/en/docs/faq/chunked_encoding_from_backend.html for clues) – Mcclelland

Why are you using output buffering? You don't seem to need it unless you're changing headers after output would have begun, you're applying a function to the output, or saving the buffer to a variable via ob_get_clean(). – Easel

@ChinotoVokro You need the output buffering to capture the length in bytes of each chunk before it's sent. – Spark

@DamianYerrick Yes, but that's handled by system level write buffering, ob_*() functions have nothing to do with http chunks. You can use flush() to request (some setups won't work as expected) the system level write buffer to be flushed, which will create a new chunk. I also just noticed the ob_flush() and flush() are backwards; ob_flush() needs to come first otherwise the latest content won't be in the system buffer to flushed. Better yet, get rid of PHP's output buffers via while(ob_get_level()){ob_end_flush();} so you only have to use flush(). – Easel

Answer 4

The output buffer will not be sent to the browser until it is full. The default size is 4096 bytes. So you either need to change the buffer size or pad your chunks. Also, the browser may have it's own minimum buffer before the page is displayed. Note that according to the Wikipedia Article about chunked encoding, sending a 0\r\n\r\n chunk terminates the response.

If you want to change the output buffering size setting, I read that you cannot use ini_set('output_buffering', $value). Instead, change the output buffering setting by adding the following to your php.ini file.

php_value output_buffering 1024

Here's an example of padding your chunks

header("Transfer-Encoding: chunked");
header("Content-Encoding: none");

// Send chunk to browser
function send_chunk($chunk)
{
    // The chunk must fill the output buffer or php won't send it
    $chunk = str_pad($chunk, 4096);

    printf("%x\r\n%s\r\n", strlen($chunk), $chunk);
    flush();
}

// Send your content in chunks
for($i=0; $i<10; $i++)
{
    send_chunk("This is Chunk #$i.<br>\r\n");
    usleep(500000);
}

// note that if you send an empty chunk
// the browser won't display additional output
echo "0\r\n\r\n";
flush();

Here's a short version that demonstrates 0\r\n\r\n\ terminating the output:

$output = "hello world";
header("Transfer-Encoding: chunked");
header('Content-Encoding: none');
printf("%x\r\n%s\r\n", strlen($output), $output);
ob_flush();
print("0\r\n\r\n");
ob_flush();
flush();
sleep(10);
print('POST PROCESSING');

Koball

Answer 5

You should be able to use:

<?php header("Transfer-Encoding: chunked");

but you'll have to ensure yourself that the output follows the specifications.

Inclinometer

It seems not enough. I have to do flush() right after last header() statement to make sure "Content-Length" header is not generated. – Conjunction

Sometimes you must disable gz-compress on server and send header("Content-Encoding: identity"); – Beware

Answer 6

A simpler way (without creating your own data chunks) is to only use flush():

<?php 
$fp = fopen('data.bin', "rb");
flush();
fpassthru($fp);
?>

This worked for me as shown in the following HTTP request and response.

Request:

00000000  47 45 54 20 2f 67 65 74  2d 63 68 75 6e 6b 2e 70 GET /get -chunk.p
00000010  68 70 20 48 54 54 50 2f  31 2e 31 0d 0a 55 73 65 hp HTTP/ 1.1..Use
00000020  72 2d 41 67 65 6e 74 3a  20 63 75 72 6c 2f 37 2e r-Agent:  curl/7.
00000030  32 31 2e 37 20 28 69 36  38 36 2d 70 63 2d 6c 69 21.7 (i6 86-pc-li
00000040  6e 75 78 2d 67 6e 75 29  20 6c 69 62 63 75 72 6c nux-gnu)  libcurl
00000050  2f 37 2e 32 31 2e 37 20  4f 70 65 6e 53 53 4c 2f /7.21.7  OpenSSL/
00000060  31 2e 30 2e 30 64 20 7a  6c 69 62 2f 31 2e 32 2e 1.0.0d z lib/1.2.
00000070  35 20 6c 69 62 73 73 68  32 2f 31 2e 32 2e 37 0d 5 libssh 2/1.2.7.
00000080  0a 48 6f 73 74 3a 20 67  61 69 61 0d 0a 41 63 63 .Host: g aia..Acc
00000090  65 70 74 3a 20 2a 2f 2a  0d 0a 0d 0a             ept: */* ....

Response:

00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d HTTP/1.1  200 OK.
00000010  0a 44 61 74 65 3a 20 54  68 75 2c 20 32 32 20 53 .Date: T hu, 22 S
00000020  65 70 20 32 30 31 31 20  32 33 3a 35 33 3a 30 32 ep 2011  23:53:02
00000030  20 47 4d 54 0d 0a 53 65  72 76 65 72 3a 20 41 70  GMT..Se rver: Ap
00000040  61 63 68 65 2f 32 2e 32  2e 38 20 28 46 65 64 6f ache/2.2 .8 (Fedo
00000050  72 61 29 0d 0a 58 2d 50  6f 77 65 72 65 64 2d 42 ra)..X-P owered-B
00000060  79 3a 20 50 48 50 2f 35  2e 32 2e 36 0d 0a 54 72 y: PHP/5 .2.6..Tr
00000070  61 6e 73 66 65 72 2d 45  6e 63 6f 64 69 6e 67 3a ansfer-E ncoding:
00000080  20 63 68 75 6e 6b 65 64  0d 0a 43 6f 6e 74 65 6e  chunked ..Conten
00000090  74 2d 54 79 70 65 3a 20  74 65 78 74 2f 68 74 6d t-Type:  text/htm
000000A0  6c 3b 20 63 68 61 72 73  65 74 3d 55 54 46 2d 38 l; chars et=UTF-8
000000B0  0d 0a 0d 0a                                      ....
000000B4  61 0d 0a 54 45 53 54 20  44 41 54 41 0a 0d 0a 30 a..TEST  DATA...0
000000C4  0d 0a 0d 0a                                      ....

Rash

Answer 7

Andriy F.'s solution worked for me. Here's a simplified version of his answer:

function dump_chunk($chunk)
{
    echo $chunk;
    flush();
    ob_flush();
}

header('Content-Type: text/html; charset=UTF-8');

flush();

for ($i = 0; $i < 3; $i++) {
    dump_chunk('Sending data chunk ' . ($i + 1) . ' of 1000 <br />');
    sleep(1);
}

Although I don't understand why the ob_flush() call is needed. If someone knows, please comment.

Riobard

Answer 8

If the size is big enough, Apache do it for you. If you work with ob_gzhandler, I think it makes no sense. Better you let the buffer out as fast as possible. If PHP really does not send automatically the content-length header or the content-length is from the uncompressed content:

ob_start();
ob_start("ob_gzhandler");
...
ob_end_flush();  
header('Content-Length: '.ob_get_length()); 
ob_end_flush();

You could reduce buffer size with apache's mod_buffer (may be with PHP's ini_set).

PS: If the size is big enough and the content-length is unknown, the content will be send in chunks as a makeshift.

Eurystheus

Answer 9

I needed to do two additional things to get this to work. Call ob_start if an object had not yet been started. And also echo out a long empty string. The browser requires a certain amount of content to be sent in it's first chunk it appears.

header('Content-Encoding', 'chunked');
header('Transfer-Encoding', 'chunked');
header('Content-Type', 'text/html');
header('Connection', 'keep-alive');

if (ob_get_level() == 0) ob_start();
echo str_pad('',4096)."\n";
ob_flush();
flush();

Washout