Extensions for VS Code not working when downloaded with wget or curl

Question

On a remote machine I am trying to use wget to download extensions for VS Code. However, the downloaded files are always corrupt and cannot be used. It turns out that using wget or curl will retrieve files which are a bit smaller than using a browser, so it appears reasonable that they cannot be unzipped. The URL I have used for downloading is https://marketplace.visualstudio.com/_apis/public/gallery/publishers/stkb/vsextensions/rewrap/1.9.1/vspackage

Downloads/tmp >ls -la
total 168
drwxrwxrwx 1 enno enno  4096 Feb  6 08:45 .
drwxrwxrwx 1 enno enno  4096 Feb  6 08:45 ..
-rwxrwxrwx 1 enno enno 85317 Feb  6 08:45 curl.vsix
-rwxrwxrwx 1 enno enno 85317 Feb  6 08:45 wget.vsix
BWP-181WD4:Downloads/tmp >unzip curl.vsix
Archive:  curl.vsix
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
note:  curl.vsix may be a plain executable, not an archive
unzip:  cannot find zipfile directory in one of curl.vsix or
        curl.vsix.zip, and cannot find curl.vsix.ZIP, period.

Downloads/tmp >unzip wget.vsix
Archive:  wget.vsix
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
note:  wget.vsix may be a plain executable, not an archive
unzip:  cannot find zipfile directory in one of wget.vsix or
        wget.vsix.zip, and cannot find wget.vsix.ZIP, period.

However, using a browser-downloaded file works just fine:

BWP-181WD4:Downloads/tmp >unzip ./browser.vsix
Archive:  ./browser.vsix
  inflating: extension.vsixmanifest
  inflating: [Content_Types].xml
  inflating: extension/CHANGELOG.md
  inflating: extension/images/logo.png
  inflating: extension/images/logo.svg
  inflating: extension/package.json
  inflating: extension/README.md
...

How come? The file sizes are clearly different:

Downloads/tmp >ls -la
total 256
drwxrwxrwx 1 enno enno  4096 Feb  6 08:52 .
drwxrwxrwx 1 enno enno  4096 Feb  6 08:50 ..
-rwxrwxrwx 1 enno enno 90035 Feb  6 08:49 browser.vsix
-rwxrwxrwx 1 enno enno 85317 Feb  6 08:45 curl.vsix
-rwxrwxrwx 1 enno enno 85317 Feb  6 08:45 wget.vsix

Is this related to how wget and curl retrieve files or how Microsoft serves these files?

Impartial

compare the files byte by byte with cmp and see which bytes are missing. Maybe tell curl to treat the file as binary. – Demiurge

cmp says curl.vsix browser.vsix differ: byte 1, line 1. So it appears curl does not simply download a part of a file, but something completely different. How can I tell curl to treat the file as binary? I could only find --data-binary which only affects uploads. – Impartial

if you look with od -x can you spot the possible cause of the difference in file size – Demiurge

I used od -x to convert both files and then used diff which shows that both files are completely different. – Impartial

Answer 1

The files downloaded with wget and curl are gzip files.

Change the filename to vspackage.gz and use gunzip to extract the vsix file

If you look at the response header in the browser you see they are gzip compressed.

Find a way for curl/wget to use the content-disposition and content-encoding of the response header.

Demiurge

Just to follow up- gunzipping works, but the resulting file is a zip- so you'll have to further unzip it – Phosphorescent

@Phosphorescent vsix is nothing but a file in zip format. You can just rename the zip into vsix and vscode can recognize it. – Trappist