S3 file to local using luigi raises UnicodeDecodeError

Question

I am copying a pdf file to local, using the following piece of code:

with self.input_target().open('r') as r:
    with self.output_target().open('w') as w:
       for line in r: 
           w.write(line) 

Which is based in this question (kind of)

But when I execute that code I get the following:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11:
      invalid continuation byte

I tried this other approach, without good results:

   with self.input_target().open('r') as r, self.output_target().open('w') as w:                                                                                                                        
       w.write(r.read())                         

What is the correct way of doing it?

Jumpoff

could you prepare a minimal example to reproduce it and the pdf? – Protasis

Answer 1

Another option would be to use the format kwarg luigi.format.Nop to Target to indicate that it is a binary file that is being read.

e.g.

def output(self):
    return LocalTarget(target, format=luigi.format.Nop)

Flannel

Answer 2

It seems that you're dealing with a binary file as if it was text - but it's not. You probably need to do something like this:

with self.input().open('r') as i, self.output().open('w') as o:
    o.write(i.read())

(not tested!)

Also, I think you may find this answer useful: Python writing binary files, bytes

Protasis

Thank you for the answer, but I didn 't work. Same error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11: invalid continuation byte :( I will update the question – Jumpoff

try open('rb') and open('wb') – Protasis

Do you know the encoding of the pdf file? If it's Latin-1, then you should read it as such: self.input_target().open('r', encoding="ISO-8859-1"). – Tertial

luigi doesn` t support the encoding argument – Jumpoff

Answer 3

I ended using luigi.s3.S3Clientand the method get which copies a binary file from/to the s3.

Snippet of code:

import luigi

class ATask(luigi.Task):
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()
    ...

    def run(self):
        client = luigi.s3.S3Client()
        client.get(s3_path, local_path)  ## This gets the file
        ...

I think that the underlying reason is that luigi uses boto for getting/putting files from/to the s3. (As you can see in the source code)

Jumpoff