Using ZipStream in Symfony: streamed zip download will not decompress using Archive Utility on Mac OSX

Question

I have a symfony 2.8 application and on the user clicking "download" button, I use the keys of several large (images, video) files on s3 to stream this to the browser as a zip file using ZipStream (https://github.com/maennchen/ZipStream-PHP).

The streaming of files and download as a zip (stored, not compressed) is successful in the browser & the zip lands in Downloads. However, when attempting to open the zip in Mac OSX El Capitan using Archive Utility (native archive software on OSX), it errors out. The Error:

Unable to expand "test.zip" into "Downloads". (Error 2 - No such file or directory.)

I have seen older, identical issues on SO and attempted those fixes, specifically this post: https://mcmap.net/q/1057434/-dynamically-created-zip-files-by-zipstream-in-php-won-39-t-open-in-osx and followed up on the Issues & PRs in ZipStream that relates and the upstream fixes in Guzzle etc.

Problem is, the above fixes were back in 2011 and things move on in that time. So applying the same fixes I am not getting a working result.

Specific fixes I have tried is: 1. Setting "version to extract" to 0x000A as suggested. Also as '20' as recommended in another post. I've set the same for "version made by". 2. I tried to force the compression method to 'deflate' instead of 'stored' to see if I got a working result. A stored resulted is all I need and suitable for a zip used as container file for images & video.

I am able to extract the zip using a third party archive app called The Unarchiver. However, users won't know & can't be expected to install an alternative archive app to suit my web app. Thats not an effective solution.

Does anyone have knowledge or experience of solving this issue and can help me out with how to resolve it?

NB: A streamed zip to browser is the required solution. Downloading assets from s3 to the server to create a zip and then stream the resulting zip to the browser is not a solution given the amount of time and overhead of such an approach.

Added info if required:

Code & Setup: - Files are stored on s3. - Web app is symfony 2.8 on PHP7.0 runing on ec2 with Ubuntu. - Using aws-sdk-php, create an s3client with valid credentials and I register StreamWrapper (s3client->registerStreamWrapper()) on the s3client. This is to fetch files from s3 via fopen to stream to ZipStream library:

$this->s3Client = $s3Client;  
$this->s3Client->registerStreamWrapper();

// Initialize the ZipStream object and pass in the file name which
//  will be what is sent in the content-disposition header.
// This is the name of the file which will be sent to the client.
// Define suitable options for ZipStream Archive.
$opt = array(
        'comment' => 'test zip file.',
        'content_type' => 'application/octet-stream'
      );
$zip = new ZipStream\ZipStream($filename, $opt);

$keys = array(
      "zipsimpletestfolder/file1.txt"
  );

foreach ($keys as $key) {
    // Get the file name in S3 key so we can save it to the zip file 
    // using the same name.
    $fileName = basename($key);

    $bucket = 'mg-test';
    $s3path = "s3://" . $bucket . "/" . $key;

    if ($streamRead = fopen($s3path, 'r')) {
      $zip->addFileFromStream($fileName, $streamRead);        
    } else {
      die('Could not open stream for reading');
    }
}

$zip->finish();

Zip Output Results:

• Extraction on mac via Archive Utility fails with error 2

• Extraction on mac with The unarchiver works.

• Extraction on windows with 7-zip works.

• Extraction on windows with WinRar fails - says zip is corrupted.

Response Headers:

Edit I'm open to using another method of streaming files to browser as a zip that can be opened on Mac, Windows, Linux natively without using ZipStream if suggested. Just not to creating a zip file server side to subsequently stream thereafter.

Continuate

You may be doing something out of sequence with that library. Have you tested the simple cases before thr complex ones? One small text file? Two small text files? Two large text files? Open the resulting file and inspect it with a hex editor? – Ratiocinate

@Michael-sqlbot I tried a simple case with 1 image, 2 images, 1 video etc initially and got same issue. I'm actually trying it again now with basic txt files. Will revert back shortly. What am I looking for in inspect with hex editor? – Continuate

@Michael-sqlbot Tried with a single txt file on s3, got the download but on extract with Archive Utility I get the same error: Unable to extract "test.zip" into "Downloads". (Error 2 - No such file or directory.) – Continuate

Since you say you are not compressing the files, just using zip as a container you should be able to see all of the text with a hex editor. If it's not all there, this suggests that you have confirmed a problem, either with the library or with your use of it. If the text is there, you can then inspect the container structures and compare to known attributes if the zip file format. Missing from your code is $zip->finish(); which presumably writes the internal directory structure at the end of the zip file (stream). – Ratiocinate

@Michael-sqlbot $zip->finish(); is in my code just not in sample above. I've edited above. – Continuate

Answer 1

The issue with the zip downloaded is that it contained html from the response in the Symfony controller that was calling ZipStream->addFileFromStream(). Basically, when ZipStream was streaming data to create zip download in client browser, a controller action response was also sent and, best guess, the two were getting mixed up on client's browser. Opening the zip file in hex editor and seeing the html in there it was obviously the issue.

To get this working in Symfony 2.8 with ZipStream, I just used Symfony's StreamedResponse in the controller action and used ZipStream in the streamedResponse's function. To stream S3 files I just passed in an array of s3keys and the s3client into that function. So:

use Symfony\Component\HttpFoundation\StreamedResponse;
use Aws\S3\S3Client;    
use ZipStream;

//...

/**
 * @Route("/zipstream", name="zipstream")
 */
public function zipStreamAction()
{
    //test file on s3
    $s3keys = array(
      "ziptestfolder/file1.txt"
    );

    $s3Client = $this->get('app.amazon.s3'); //s3client service
    $s3Client->registerStreamWrapper(); //required

    $response = new StreamedResponse(function() use($s3keys, $s3Client) 
    {

        // Define suitable options for ZipStream Archive.
        $opt = array(
                'comment' => 'test zip file.',
                'content_type' => 'application/octet-stream'
              );
        //initialise zipstream with output zip filename and options.
        $zip = new ZipStream\ZipStream('test.zip', $opt);

        //loop keys useful for multiple files
        foreach ($s3keys as $key) {
            // Get the file name in S3 key so we can save it to the zip 
            //file using the same name.
            $fileName = basename($key);

            //concatenate s3path.
            $bucket = 'bucketname';
            $s3path = "s3://" . $bucket . "/" . $key;        

            //addFileFromStream
            if ($streamRead = fopen($s3path, 'r')) {
              $zip->addFileFromStream($fileName, $streamRead);        
            } else {
              die('Could not open stream for reading');
            }
        }

        $zip->finish();

    });

    return $response;
}

Solved. Maybe this helps someone else use ZipStream in Symfony.

Continuate