Guzzle 6 Large file uploads / Chunking

Question

I've read that if Guzzle cannot determine Content-Length, it will send Transfer-Encoding: Chunked headers and cURL on the back-end will handling the chunking. But I'm obviously hitting post_max_size limit. ("POST Content-Length of 524288375 bytes exceeds the limit of 8388608 bytes) when POSTing to a working uploadChunkerController. I know the upload handler (endpoint) works with smaller files. I feel I have something configured wrong with my Guzzle options. I have to set verify to false and I need to post an api_key with the request.

    $client = new Client();
    $fh     = fopen('../storage/random-500M.pdf', 'r');
    $url    = 'https://local:8443/app_dev.php/_uploader/bigupload/upload';

    $request = $client->request(
        'POST',
        $url,
        [
            'verify'    => false,
            'multipart' => [
                [
                    'name' => 'api_key',
                    'contents' => 'abc123'
                ],
                [
                    'name'     => 'file',
                    'contents' => $fh,
                    'filename' => 'bigupload.pdf'
                ]
            ]
        ]
    );

Editing php.ini settings is not an option nor the solution. I've found a lot of 'solutions' that appear to be for older versions of Guzzle. Am I thinking too hard about this? Is there a simpler solution?

Heliochrome

the server/script you are posting to is saying you can only upload a file that's 8MB, if you have access to that server/script which it appears you do you will need to change the limit. – Spay

That's the whole reason to use chunking. BlueImp jQuery File Uploader works from the front-end/UI side. I'm posting to the same controller action from another PHP script. So as stated before the action that consumes the POSTs handles chunked uploads and as long as I can get Guzzle/cURL to chunk the file and provide the proper headers, all should be golden. – Heliochrome

phpnews.io/feeditem/… ? – Spay

@Spay Yes, I've followed that blog post. It's for an older version. The EntityBody class doesn't exist in the current Guzzle 6 git repo. I used that as a model to get where I'm at now. Even attempted to use '$body = \GuzzleHttp\Psr7\stream_for($fh);' and pass that in as the resource. – Heliochrome

Answer 1

After digging through Guzzle and cURL source code, there's no 'automatic' way for them to send 'chunks'. Headers aren't set and there's no way for them to slice up the file being sent. I've come up with my own solution using Guzzle vs raw PHP cURL calls.

/**
 * @Route("/chunks", name="chunks")
 */
public function sendFileAction()
{
    $jar       = new \GuzzleHttp\Cookie\SessionCookieJar('SESSION_STORAGE', true);
    $handler   = new CurlHandler();
    $stack     = HandlerStack::create($handler);
    $client    = new Client(['cookies'=>true, 'handler' => $stack]);
    $filename  = 'files/huge-1gig-file.jpg';
    $filesize  = filesize($filename);
    $fh        = fopen($filename, 'r');
    $chunkSize = 1024 * 2000;
    $boundary  = '----iCEBrkUploaderBoundary' . uniqid();
    $url       = 'https://localhost/app_dev.php/_uploader/bigupload/upload';

    rewind($fh); // probably not necessary
    while (! feof($fh)) {
        $pos   = ftell($fh);
        $chunk = fread($fh, $chunkSize);
        $calc  = $pos + strlen($chunk)-1;

        // Not sure if this is needed.
        //if (ftell($fh) > $chunkSize) {
        //    $pos++;
        //}

        $request = $client->request(
            'POST',
            $url,
            [
                'cookies' => $jar,
                'debug'   => false,
                'verify'  => false,
                'headers' => [
                    'Transfer-Encoding'   => 'chunked',
                    'Accept-Encoding'     => 'gzip, deflate, br',
                    'Accept'              => 'application/json, text/javascript, */*; q=0.01',
                    'Connection'          => 'keep-alive',
                    'Content-disposition' => 'attachment; filename="' . basename($filename) . '"',
                    'Content-length'      => $calc - $pos,
                    'Content-Range'       => 'bytes ' . $pos . '-' . $calc . '/' . $filesize
                ],
                'multipart' => [
                    [
                        'name'     => 'api_key,
                        'contents' => 'aaabbbcc-deff-ffed-dddd-1234567890123'
                    ],
                    [
                        'name'     => 'file',
                        'contents' => $chunk,
                        'filename' => basename($filename),
                        'headers' => [
                            'Content-Type' => 'multipart/form-data; boundary=' . $boundary
                        ]
                    ]
                ]
            ]
        );
    }

    return new Response('ok', 200);
}

I hope this helps someone else out. Comments/Suggestions welcome.

Heliochrome

This function replicates the functionality of the front end github.com/blueimp/jQuery-File-Upload file uploader so you're able to programmatically (CLI) post to the same endpoint and your files will be handled with the same logic. – Heliochrome

Answer 2

Chunked transfer encoding doesn't help in this case.

It's used to provide content ASAP by sending (and generating) it by parts. It has nothing to do with size limits (like in your scenario).

The only way for you is to increase the limit on the server.

Bunsen

If Javascript can do it link, PHP can do it too. @Spay referenced an older version of Guzzle that claims to do chunking. I can only assume a newer version can also to this. I've attempted to use raw cURL (CURLOP_READFUNCTION) with no luck. – Heliochrome

Same question with an answer that seems to say Guzzle can post chunks. [Upload File in chunks to URL Endpoint using Guzzle PHP ](#45183154) – Heliochrome

Guzzle is able to do that (chunks I mean), but it won't help you. As I said before, the it doesn't work by design. Try to implement it with JS or any other language/library and see. – Bunsen

You are trying to upload a file that is bigger that the server's limits. This is not possible by design (that's why the feature is called limits). – Bunsen

Then can you explain how BlueImp jQuery File uploader works? Can you explain why chunking exists? (github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads) – Heliochrome