PHP cURL Realtime proxy (stream file)

Question

Currently I have a script like the following:

<?php
$filename = "http://someurl.com/file.ext";
header('Content-Type: application/octet-stream');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$filename);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 500);
$data=curl_exec($ch);
curl_close($ch);
echo $data;
?>

The problem is that the server just send the response after download the whole file. I want to make it work like a "stream", sending chunks of data as response while the file is downloaded.

Is that possible to achieve with PHP and cURL?

Conservatory

This might be what you're looking for. – Kirit

I never used composer before and I am in shared hosting, but I will search more about it. – Conservatory

Answer 1

It's possible. You can use the curl option CURLOPT_WRITEFUNCTION to specify a callback where you'll receive chunks of data so you can send them directly to the client as curl downloads the file.

<?php

$filename = "http://someurl.com/file.ext";
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="' . basename($filename) . '"');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$filename);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 500);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, function($curl, $data) {
    echo $data;
    return strlen($data);
});
curl_exec($ch);
curl_close($ch);

Macleod

Tested it with a 5 mb file. Instead of just gradually send the chuncks I believe that it send the data already transfered from begin many times or it is in some way corrupted. This requires a specific php version? – Conservatory

The writefunction should never receive the same data twice. The minimum PHP version is 5.3 to allow the anonymous function to be supplied to CURLOPT_WRITEFUNCTION. Otherwise no requirement. I tried it again with various files from 5 MB to 480 MB and it started downloading on my computer nearly immediately and the files checksums matched up and opened fine. – Macleod

I finally found the problem, was caused by cloudflare... The code works as expected. Thank you. – Conservatory

Working fine..! Thanks @Macleod – Commandeer

Is it possible to redirect the original content-length to the php proxy and pass it to header-function? – Hadwyn

@Hadwyn There is no guarantee a length is sent. If they send a content-length header, it's easy to grab that using CURLOPT_HEADERFUNCTION and send it before any data is sent. You may also be able to borrow some logic from this other answer of mine which shows how to get the content length of a remote resource. – Macleod

And when? After curl_exec is to late. Before there is no header data. – Hadwyn

actually there's no need for a CURLOPT_WRITEFUNCTION here - the default action for curl if CURLOPT_FILE is null and CURLOPT_WRITEFUNCTION is null, is to just write it to stdout anyway, which is exactly what your custom CURLOPT_WRITEFUNCTION here does :) – Stilla

@Hadwyn CURLOPT_HEADERFUNCTION makes it fairly easy to proxy all headers, but you must be careful to filter out the "Content-Encoding" header if you're using CURLOPT_ENCODING... something like (untested) curl_setopt($ch,CURLOPT_HEADERFUNCTION,function($ch,string $header){$ret=strlen($header);if(0===stripos($header,"Content-Encoding")){return $ret;}header(substr($header,0,-2));return $ret;}); (the substr -2 is because the headers from curl includes the \r\n separators, but php's header() function does not want the \r\n separators.) – Stilla

Code can be made a little more concise by using curl_setopt_array($ch, [CURLOPT_URL,...]) – Hypnotic

Answer 2

Curl will by default output the response directly, unless you specify CURLOPT_RETURNTRANSFER.

Your code will work just by removing CURLOPT_RETURNTRANSFER and the last echo:

<?php
$filename = "http://someurl.com/file.ext";
header('Content-Type: application/octet-stream');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$filename);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 500);
$data=curl_exec($ch);
curl_close($ch);
?>

Conductive

CURLOPT_RETURNTRANSFER is 1 of several options that stops that default behavior. others include: CURLOPT_NOBODY and CURLOPT_FILE, and CURLOPT_WRITEFUNCTION – Stilla