How to read from a text file compressed with 7z?

Question

I would like to read (in Python 2.7), line by line, from a csv (text) file, which is 7z compressed. I don't want to decompress the entire (large) file, but to stream the lines.

I tried pylzma.decompressobj() unsuccessfully. I get a data error. Note that this code doesn't yet read line by line:

input_filename = r"testing.csv.7z"
with open(input_filename, 'rb') as infile:
    obj = pylzma.decompressobj()
    o = open('decompressed.raw', 'wb')
    obj = pylzma.decompressobj()
    while True:
        tmp = infile.read(1)
        if not tmp: break
        o.write(obj.decompress(tmp))
    o.close()

Output:

    o.write(obj.decompress(tmp))
ValueError: data error during decompression

Rosefish

Why don't you post your code and a sample file so we can reproduce your error and can see how we can help? – Garratt

.7z file are containers (archives) that can contain more than one file, so what's the name of file inside testing.7z you want to read? – Croissant

@martineau, testing.csv – Rosefish

Answer 1

This will allow you to iterate the lines. It's partially derived from some code I found in an answer to another question.

At this point in time (pylzma-0.5.0) the py7zlib module doesn't implement an API that would allow archive members to be read as a stream of bytes or characters — its ArchiveFile class only provides a read() function that decompresses and returns the uncompressed data in a member all at once. Given that, about the best that can be done is return bytes or lines iteratively via a Python generator using that as a buffer.

The following does the latter, but may not help if the problem is the archive member file itself is huge.

The code below should work in Python 3.x as well as 2.7.

import io
import os
import py7zlib


class SevenZFileError(py7zlib.ArchiveError):
    pass

class SevenZFile(object):
    @classmethod
    def is_7zfile(cls, filepath):
        """ Determine if filepath points to a valid 7z archive. """
        is7z = False
        fp = None
        try:
            fp = open(filepath, 'rb')
            archive = py7zlib.Archive7z(fp)
            _ = len(archive.getnames())
            is7z = True
        finally:
            if fp: fp.close()
        return is7z

    def __init__(self, filepath):
        fp = open(filepath, 'rb')
        self.filepath = filepath
        self.archive = py7zlib.Archive7z(fp)

    def __contains__(self, name):
        return name in self.archive.getnames()

    def readlines(self, name, newline=''):
        r""" Iterator of lines from named archive member.

        `newline` controls how line endings are handled.

        It can be None, '', '\n', '\r', and '\r\n' and works the same way as it does
        in StringIO. Note however that the default value is different and is to enable
        universal newlines mode, but line endings are returned untranslated.
        """
        archivefile = self.archive.getmember(name)
        if not archivefile:
            raise SevenZFileError('archive member %r not found in %r' %
                                  (name, self.filepath))

        # Decompress entire member and return its contents iteratively.
        data = archivefile.read().decode()
        for line in io.StringIO(data, newline=newline):
            yield line


if __name__ == '__main__':

    import csv

    if SevenZFile.is_7zfile('testing.csv.7z'):
        sevenZfile = SevenZFile('testing.csv.7z')

        if 'testing.csv' not in sevenZfile:
            print('testing.csv is not a member of testing.csv.7z')
        else:
            reader = csv.reader(sevenZfile.readlines('testing.csv'))
            for row in reader:
                print(', '.join(row))

Croissant

Answer 2

If you were using Python 3.3+, you might be able to do this using the lzma module which was added to the standard library in that version.

See: lzma Examples

Halfway

The question is tagged with python-2.7 so we can assume it is not Python 3 here. – Garratt

Also, you should mention python 3.3 (from the doc link) & not just 3. – Peridotite

@MartijnPieters didn't have that tag when I commented. – Halfway

Even if the OP was using Python 3.3+, the lzma module only provides functions for compressing and decompressing raw data with LZMA compression algorithm—which isn't the same thing as handling 7zip-format archive files which potentially contain multiple files/members, as the PyLZMA third-party module does. – Croissant

Answer 3

If you can use python 3, there is a useful library, py7zr, which supports partially 7zip decompression as below:

import py7zr
import re
filter_pattern = re.compile(r'<your/target/file_and_directories/regex/expression>')
with SevenZipFile('archive.7z', 'r') as archive:
    allfiles = archive.getnames()
    selective_files = [f if filter_pattern.match(f) for f in allfiles]
    archive.extract(targets=selective_files)

Trilogy

This does not accomplish what the OP asks, which is to stream a single output file. – Zygotene