Extract files with invalid characters in filename with Python

Question

I use python's zipfile module to extract a .zip archive (Let's take this file at http://img.dafont.com/dl/?f=akvaleir for example.)

f = zipfile.ZipFile('akvaleir.zip', 'r')
for fileinfo in f.infolist():
    print fileinfo.filename
    f.extract(fileinfo, '.')

Its output:

Akval�ir_Normal_v2007.ttf
Akval�ir, La police - The Font - Fr - En.pdf

Both files are unaccessable after extraction because there are invalid encoded characters in their filenames. The problem is zipfile module doesn't have an option to specify output filenames.

However, "unzip akvaleir.zip" escapes the filename well:

root@host:~# unzip akvaleir.zip 
Archive:  akvaleir.zip
  inflating: AkvalВir_Normal_v2007.ttf  
  inflating: AkvalВir, La police - The Font - Fr - En.pdf  

I tried capturing output of "unzip -l akvaleir.zip" in my python program and these two filenames are:

Akval\xd0\x92ir_Normal_v2007.ttf
Akval\xd0\x92ir, La police - The Font - Fr - En.pdf

How can I get the correct filename like what unzip command does without capturing output of "unzip -l akvaleir.zip"?

Palaearctic

Answer 1

Instead of the extract method, use the open method and save the resulting pseudofile to disk under whatever name you wish, for example with shutil.copyfileobj.

Tensile

@Alex, thanks, it works. Do you know how to escape invalid filename in python in the same way as unzip? – Palaearctic

+1 don't use extract or extractall unless you've throroughly checked all the filenames, as it can spew files out anywhere on your filesystem. – Stirrup

@jack, sure looks like they're encoding it in utf-8, but I don't know what encoding the zipfile itself is using -- try printing the repr of the filenames as it comes to Python from the zipfile and we'll see if we can guess that encoding too (basically you'll decode the filename into unicode, with whatever codec it's using, then encode it into utf-8 for the purpose of saving the file). – Tensile

or use read and simply write in a file opened with wb (binary mode), this seems to work better than shutil.copyfileobj for special binary file. – Lycaon

Answer 2

It took some time but I think I found the answer.

I assumed the word was supposed to be Akvaléir. I found a page description about that, in French. When I used your code snippet I had a string like

>>> fileinfo.filename
'Akval\x82ir, La police - The Font - Fr - En.pdf'
>>> 

That didn't work at UTF8, Latin-1, CP-1251 or CP-1252 encodings. I then found that CP863 was a possible Canadian encoding, so perhaps this was from French Canada.

>>> print unicode(fileinfo.filename, "cp863").encode("utf8")
Akvaléir, La police - The Font - Fr - En.pdf
>>> 

However, I then read the Zip file format specification which says

The ZIP format has historically supported only the original IBM PC character encoding set, commonly referred to as IBM Code Page 437.

...

If general purpose bit 11 is set, the filename and comment must support The Unicode Standard, Version 4.1.0 or greater using the character encoding form defined by the UTF-8 storage specification.

Testing that out gives me the same answer as the Canadian code page

>>> print unicode(fileinfo.filename, "cp437").encode("utf8")
Akvaléir, La police - The Font - Fr - En.pdf
>>>

I don't have a Unicode encoded zip file and I'm not going to create one to find out, so I'll just assume that all zip files have the cp437 encoding.

import shutil
import zipfile

f = zipfile.ZipFile('akvaleir.zip', 'r')
for fileinfo in f.infolist():
    filename = unicode(fileinfo.filename, "cp437")
    outputfile = open(filename, "wb")
    shutil.copyfileobj(f.open(fileinfo.filename), outputfile)

On my Mac that gives

 109936 Nov 27 01:46 Akvale??ir_Normal_v2007.ttf
  25244 Nov 27 01:46 Akvale??ir, La police - The Font - Fr - En.pdf

which tab-completes to

ls Akvale\314\201ir

and shows up with a nice 'é' in my file browser.

Sadiras

Yes, you have to know the source encoding in advance. The ZIP format contains absolutely no information from which you can work out what encoding filenames are using. Whilst Macs and most modern Linuxen sensibly use UTF-8 for their filesystems and inside ZIP, Windows machines use the system code page, which is locale-dependent and never UTF-8. It's a real headache. – Stirrup

@dalke, thanks for your info. How can I detect which encoding a string uses in python program? – Palaearctic

Read the spec and found that ZIP uses cp437 or utf-8. There is a flag (bit 11) which specifies which of those two was used. I haven't written code to check for that. – Sadiras

filename = unicode(fileinfo.filename, "cp437") should be filename = fileinfo.filename.encode('cp437').encode('???') here when using python3 – Valonia

Answer 3

I ran into a similar issue while running my application using Docker. Adding this lines to the Dockerfile, fixed everything for me:

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

So, I guess if you're not using Docker, give it a try and make sure locales are properly generated and set.

Predecease