HTTP File Download: Monitoring Download Progress

Question

I am in a situation, when I have to implement downloading of large files(up to 4GB) from a Web server: Apache 2.4.4 via HTTP protocol. I have tried several approaches, but the best solution looks to be the usage of X-SendFile module.

As I offer progress bar for file uploads, I would need to have the same feature for file downloads. So here are my questions:

• Is there any way, including workaround, to achieve file downloads progress monitoring?
• Is there any way, including workaround, to calculate file download transfer speed?
• Is there better way to provide efficient file downloads from a web server than usage of X-Sendfile module?

Is there better file download option in general, that would allow me to monitor file download progress? It can be a client (JavaScript) or server solution(PHP). Is there any particular web server that allows this?

Currently I use:

• Apache 2.4.4
• Ubuntu

Many times thanks.

Garris

You can use cURL in php. #13958803 – Waterway

Are you in a position to stream the file from your server - or are you downloading a single file? – Trusting

@web_bod, many files will be downloaded from the server. In other words, file download and upload will be the main functionality of the server. Since I am in control of the server, I can decide for any form of file download. I tried to download through PHP, but this brought limitations on file size and number of files downloaded. Therefore I tried X-SendFile which looks good. However, if streaming is the way to go, I will gladly forged X-SendFile – Garris

@positlabs, cURL looks interesting too. Thank you for this tip. Since I do not know much about cURL, I am going to study it. I hope, cURL does not put obstacles on filesize downloaded or number of simultaneous download. – Garris

I'll sketch something over the weekend for you - is PHP is your preferred language? – Trusting

@web_bod, that is very nice from you. Yes, PHP is my prefered server language. On the client side nothing special only: HTML 5, JavaScript. Of course, I would not wish the clients to install anything special like java or flash because of file downloads. The server should be able to handle multiple files and multiple users without any filesize limit, as I can have files that are several GB large. Thank you very much in advance. Please, forget me not. :-) – Garris

@web_bod: there's one more thing. the files on the servers are in their specific directories, away from the web serving directory. I study cURL now, but I am not sure, if it is possible for cURL to access files stored elsewhere than in the web directory, on the server. – Garris

@positlabs, I have tested cURL. It has even progress bar info. The problem with cURL is, that when there is a large file to download, on file download there is a long delay before the download actually happens. I do not know what causes this. But imagine, you click a link to get u a file, and now you have to wait several minutes when the download actually start. But it was good tip +1 – Garris

Answer 1

2 ideas (not verified):

First:

Instead of placing regular links to files (that you want to download) on your page place links like .../dowanload.php which may look sth like this:

<?php

    // download.php file
    session_start(); // if needed

    $filename = $_GET['filename']);

    header( 'Content-type: text/plain' ); // use any MIME you want here
    header( 'Content-Disposition: attachment; filename="' . htmlspecialchars($filename) . '"' );
    header( 'Pragma: no-cache' );

    // of course add some error handling

    $filename = 'c:/php/php.ini';

    $handle = fopen($filename, 'rb');

    // do not use file_get_contents as you've said files are up to 4GB - so read in chunks
    while($chunk = fread($handle, 1000)) // chunk size may depend on your filesize
    {
        echo $chunk;
        flush();
        // write progress info to the DB, or session variable in order to update progress bar
    }

    fclose($handle);
?>

This way you may keep eye on your download process. In the meantime you may write progress info to the DB/session var and update progress bar reading status from DB/session var using AJAX of course polling a script that reads progress info.

That is very simplified but I think it might work as you want.

Second:

Apache 2.4 has Lua language built in:

• mod_lua
• Creating hooks and scripts with mod_lua

I bet you can try to write LUA Apache handler that will monitor your download - send progress to the DB and update progress bar using PHP/AJAX taking progress info from the DB.

Similarly - there are modules for perl and even python (but not for win)

Diaspore

The weird URLs can be hidden using mod_rewrite. The outside world need not know about the download.php script. – Apograph

Instead of mod_lua you can use mod_perl, which is not in experimental state. – Apograph

using a DB to store progress updates? i see now why node.js becoming popular... i mean at least suggest APC... – Singletary

@dandavis: Sure, APC would be ok but newest PHP lacks it already (Opcache should allow similar thing). Node.js - this is what I need to read more about - just hear about it - nothing more. Thanks – Diaspore

@Artur, hi and thanks for your feedback. To be honest, it looks to me as too complicated and possibly overkill for what I need. What worked for me was combination of fopen(), fread(), print(), while monitoring download progress with AJAX requests. – Garris

@Artur, hi may I get to you regaring you answer, please. It looks, that it is not possobile to run download monitoring php script, while download. Please, see my post here: #21507060. Would you have any idea for solution? Thanks in advance. – Garris

Answer 2

I see main problem in that: In a php+apache solution output buffering may be placed in several places:

Browser <= 1 => Apache <= 2 => PHP handler <= 3 => PHP Interpreter process

You need to control first buffer. But directly from PHP it is impossible.

Possible solutions:

1) You can write own mini daemon which primary function will be only send files and run it on another than 80 port 8880 for example. And process downloading files and monitor output buffer from there. Your output buffer will be only one and you can control it:

Browser <= 1 => PHP Interpreter process

2) Also you can take mod_lua and control output buffers directly from apache.

3) Also you can take nginx and control nginx output buffers using built-in perl (it is stable)

4) Try to use PHP Built-in web server and control php output buffer directly. I can't say anything about how it is stable, sorry. But you can try. ;)

I think that nginx+php+built-in perl is more stable and powerful solution. But you can choose and maybe use other solution non in that list. I will follow this topic and waiting your final solution with interest.

Dander

Answer 3

Read and write to the database at short intervals is killing performance.

I would suggest to use sessions (incrementing the value of sent data in the loop) with which you can safely off by quite another php file, you can return data as JSON which can be used by the javascript function/plugin.

Dykstra

great idea. I had the same. Howver, I come to an unexpected problem: While the main PHP download script is running, the monitoring PHP script does not get executed by AJAX calls. I am stuck with this for long time and testing it. But AJAX monitoring scripts are apparently run only when there is no other PHP script currently running. See my other post – Garris

I have tried many approaches, and all of them had some sort of problem. Approach of using fopen() fread() print() works best to me and gives no limits on file size. I therefore mark this answer as useful answer, as the accepted answer, and award it with my bounty. – Garris