Prevent output buffering with PHP and Apache

Question

I have a PHP script which sends a large number of records, and I want to flush each record as soon as it is available: the client is able to process each record as it arrives, it does not need to wait for the entire response. I realize it takes slightly longer for the entire transfer because it needs to be sent in multiple packets, but it still allows the client to start working sooner.

I've tried all the different flush() and ob_flush() functions but nothing seems to help get the data actually sent over the line before the page is finished. I've confirmed that it is not the web browser because I've tested it using telnet.

Guilt

Answer 1

The only solution that worked for me was to set the output_buffering directive in php.ini to "Off". I didn't want to do this for the entire server, just this one specific resource. Normally you could use ini_set from the PHP script, but for whatever reason php doesn't allow output_buffering to be set in this way (see the php manual).

Well it turns out that if you're using Apache, you can set some php ini directives (including output_buffering) from your server config, including a .htaccess file. So I used the following in a .htaccess file to disable the output_buffering just for that one file:

<Files "q.php">
    php_value output_buffering Off
</Files>

And then in my static server configuration, I just needed AllowOverride Options=php_value (or a larger hammer, like AllowOverride All) in order for that to be allowed in a .htaccess file.

Guilt

Not sure if this was always wrong or something has changed in PHP5.6, but I had to use php_flag output_buffering Off to get the Off to work. – Boo

I set "output_buffering" from php.ini from 4096 to Off, and rebooted entire server, verified with phpinfo(); but I still face the same problem, server is waiting for page load before sending data to the browser. Is there another setting in Apache2 or Ubuntu 16.04? I'm using PHP7. – Redemptioner

Answer 2

You don't mention what web server you are using, but I am going to go out on a limb here and guess Apache2. I hit almost the identical thing you describe. I was trying to get my cgi script to pass back information as it had it ready, instead of buffering the whole thing. Worked jiffy in curl, etc., but buffered in a browser (pretty much any browser), which was at least maddening. I went through the exact steps you describe. The resolution in my case was to modify sites-enabled/terrifico.com configuration file in Apache2 (the line in question starts with

SetEnvIfNoCase

(You can ignore the stuff above and below that line, I'm just showing it for reference of where I placed it.)

<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName test.terrifico.com
ServerAlias test.terrifico.com

SetEnvIfNoCase Request_URI \.cgi$ no-gzip dont-vary

DocumentRoot /var/www/test.terrifico.com

From staring at the network traffic going back and forth, it finally dawned on me that the browser was advertising that it accepted deflation for anything (that was text). That was the difference between the browser and curl, for example. The salient bit was

Accept-Encoding:gzip,deflate,sdch

There was a bit about chunking, but that didn't impact this particular problem. So, the browser was requesting mod_deflate to kick in, which defeated my carefully spewing out bytes as I got them in my cgi script. You could change it in the browser, but it seemed more sensible to change it on the server once for the works.

Perhaps this helps.

Solorzano

Answer 3

To turn off output buffering at run time in PHP without changing php.ini or having a .htaccess file, just use ob_end_flush() or ob_end_clean() at the beginning of the script. For example:

This should output without buffering:

<?php
@ob_end_clean();

for ($i = 0; $i < 5; $i++)
{
    echo "$i\n";
    flush();
    usleep(0.5e6);
}

This outputs with buffering (all at a time) if output_buffering is on, regardless of the flush() call:

<?php

for ($i = 0; $i < 5; $i++)
{
    echo "$i\n";
    flush();
    usleep(0.5e6);
}

Despite its name, ob_implicit_flush calls flush(), not ob_flush(), implicitly after every output. This can be handy in this instance after closing the output buffer at the beginning:

<?php
@ob_end_clean(); // disable output buffer
ob_implicit_flush(); // call flush() automatically after every output

for ($i = 0; $i < 5; $i++)
{
    echo "$i\n";
    usleep(0.5e6);
}

This fixes the PHP side. There may be something else going on with mod_deflate or similar (see the answer by Ted Collins), and I've observed that Firefox needs at least 1024 bytes before it starts to output anything at all.

Millepede

Best answer, ob_implicit_flush() is what was missing – Cesium

ob_end_clean(): Failed to delete buffer. No buffer to delete – Moynihan

@AntonDuzenko Apparently it now raises an E_NOTICE-level message when it's not active. This means that you don't need it in the first place, but if you want to keep your code compatible with other servers that do need it and want to keep it, just add an @ symbol at the beginning of ob_end_clean to silence the notice. I've edited the answer accordingly. – Millepede