Python - mechanism to identify compressed file type and uncompress

Question

A compressed file can be classified into below logical groups
a. The operating system which you are working on (*ix, Win) etc.
b. Different types of compression algorithm (i.e .zip,.Z,.bz2,.rar,.gzip). Atleast from a standard list of mostly used compressed files.
c. Then we have tar ball mechanism - where I suppose there are no compression. But it acts more like a concatenation.

Now, if we start addressing the above set of compressed files,
a. Option (a) would be taken care by python since it is platform independent language.
b. Option (b) and (c) seems to have a problem.

What do I need
How do I identify the file type (compression type) and then UN-compress them?

---

Like:

fileType = getFileType(fileName)  
switch(fileType):  
case .rar:  unrar....
case .zip:  unzip....

etc  

---

So the fundamental question is how do we identify the compression algorithm based on the file (assuming the extension is not provided or incorrect)? Is there any specific way to do it in python?

Corm

Answer 1

This page has a list of "magic" file signatures. Grab the ones you need and put them in a dict like below. Then we need a function that matches the dict keys with the start of the file. I've written a suggestion, though it can be optimized by preprocessing the magic_dict into e.g. one giant compiled regexp.

magic_dict = {
    "\x1f\x8b\x08": "gz",
    "\x42\x5a\x68": "bz2",
    "\x50\x4b\x03\x04": "zip"
    }

max_len = max(len(x) for x in magic_dict)

def file_type(filename):
    with open(filename) as f:
        file_start = f.read(max_len)
    for magic, filetype in magic_dict.items():
        if file_start.startswith(magic):
            return filetype
    return "no match"

This solution should be cross-plattform and is of course not dependent on file name extension, but it may give false positives for files with random content that just happen to start with some specific magic bytes.

Mcafee

This nicely identifies the file type. However, you should return in object created by opening the file and allowing access. Otherwise you will end up testing the file type again to see you it should be handled. This can be avoided by creating a common abstraction that can deal with all supported file types. The pattern is calls "factory". – Downing

You can also use this site to search for the signatures you want: filesignatures.net/index.php – Helmholtz

The zip file format allows arbitrary data to be appended to the beginning of the file, so checking for a magic number for zip files is not correct in all cases. – Panorama

@BradleyOdell AFAIK, that's RAR, not zip. – Fuentes

Excellent idea, however under Windows 10, I get the following error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 20: character maps to <undefined> – Copp

Answer 2

Based on lazyr's answer and my comment, here is what I mean:

class CompressedFile (object):
    magic = None
    file_type = None
    mime_type = None
    proper_extension = None

    def __init__(self, f):
        # f is an open file or file like object
        self.f = f
        self.accessor = self.open()

    @classmethod
    def is_magic(self, data):
        return data.startswith(self.magic)

    def open(self):
        return None

import zipfile

class ZIPFile (CompressedFile):
    magic = '\x50\x4b\x03\x04'
    file_type = 'zip'
    mime_type = 'compressed/zip'

    def open(self):
        return zipfile.ZipFile(self.f)

import bz2

class BZ2File (CompressedFile):
    magic = '\x42\x5a\x68'
    file_type = 'bz2'
    mime_type = 'compressed/bz2'

    def open(self):
        return bz2.BZ2File(self.f)

import gzip

class GZFile (CompressedFile):
    magic = '\x1f\x8b\x08'
    file_type = 'gz'
    mime_type = 'compressed/gz'

    def open(self):
        return gzip.GzipFile(self.f)


# factory function to create a suitable instance for accessing files
def get_compressed_file(filename):
    with file(filename, 'rb') as f:
        start_of_file = f.read(1024)
        f.seek(0)
        for cls in (ZIPFile, BZ2File, GZFile):
            if cls.is_magic(start_of_file):
                return cls(f)

        return None

filename='test.zip'
cf = get_compressed_file(filename)
if cf is not None:
    print filename, 'is a', cf.mime_type, 'file'
    print cf.accessor

Can now access the compressed data using cf.accessor. All the modules provide similar methods like 'read()', 'write()', etc. to do this.

Downing

in get_compressed_file function you are doing cls(f), f is a file handler, while your open functions are expecting filenames... I modified it to close f, and pass filename instead. is there a better way? – Phiz

my previous comment may be related to python version...in python2 bz2.BZ2File does only accept string – Phiz

Answer 3

The accepted solution looks great, but it doesn't work with python-3, here are the modifications that made it work -- using binary I/O instead of strings:

magic_dict = {
    b"\x1f\x8b\x08": "gz",
    b"\x42\x5a\x68": "bz2",
    b"\x50\x4b\x03\x04": "zip"
    }
''' SKIP '''
    with open(filename, "rb") as f:
''' The rest is the same '''

Untouchable

Answer 4

This is a complex question that depends on a number of factors: the most important being how portable your solution needs to be.

The basics behind finding the file type given a file is to find an identifying header in the file, usually something called a "magic sequence" or signature header, which identifies that a file is of a certain type. Its name or extension is usually not used if it can be avoided. For some files, Python has this built in. For example, to deal with .tar files, you can use the tarfile module, which has a convenient is_tarfile method. There is a similar module named zipfile. These modules will also let you extract files in pure Python.

For example:

f = file('myfile','r')
if zipfile.is_zipfile(f):
    zip = zipfile.ZipFile(f)
    zip.extractall('/dest/dir')
elif tarfile.is_tarfile(f):
    ...

If your solution is Linux or OSX only, there is also the file command which will do a lot of the work for you. You can also use the built-in tools to uncompress the files. If you are just doing a simple script, this method is simpler and will give you better performance.

Correspondence

Answer 5

"a" is completely false.

"b" can be easily interpreted badly, as ".zip" doesn't mean the file is actually a zip file. It could be a JPEG with zip extension (for confusing purposes, if you want).

You actually need to check if the data inside the file matches the data expected to have by it's extension. Also have a look at magic byte.

Unreflective

With option (a), I only meant code written in python to un-compress say Unix, must work for same file un-compression in WIN. Any specific reason that I am wrong? – Corm

A compression algorithm is OS-independant. You can compress a file in Unix, then uncompress it on WIndows, then send it to a Mac and compress it again, compare the compressed file from Unix and the one from Mac and they'll be bit-a-bit equal. – Unreflective

@Corm Generally (most likely) you can use same python code to uncompress a file across OSs using python. You meant to classify based on the python code needed to uncompress on different OSs(which is what platform independence brings) with the (incorrect) understanding that different OSs will need different python code (which is generally speaking not true). But you said so with a choice of words that means something else and alexandernst corrected you. – Henghold

Answer 6

If the exercise is to identify it just to label files, you have lots of answers. If you want to uncompress the archive, why don't you just try and catch the execptions/errors? For example:

>>> tarfile.is_tarfile('lala.txt')
False
>>> zipfile.is_zipfile('lala.txt')
False
>>> with bz2.BZ2File('startup.bat','r') as f:
...    f.read()
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
IOError: invalid data stream

Buatti

Answer 7

2019 update:
I was looking for a solution to detect if a .csv file was gzipped or not. The answer @Lauritz gave was throwing errors for me, i imagine it's just because the way files are read has changed in the past 7 years.

This library worked perfectly for me! https://pypi.org/project/filetype/

Litigation