Redshift COPY command Failed writing body (0 != 575) Cause: Failed to inflateinvalid or incomplete deflate data

Question

I have a file on S3 called data.csv.gz It is a gzipped CSV. I've successfully ungzipped it with the ungzip command, so I know it's gzipped correctly as far as I can tell.

Running the following command gives an error

COPY to_table ("id", "something", "something_else")
FROM 's3://my.domain.com/somewhere/data.csv.gz'
CREDENTIALS 'aws_access_key_id=********;aws_secret_access_key=********'
IGNOREHEADER 1 TRUNCATECOLUMNS CSV REGION 'us-east-1' GZIP;

The error is:

-----------------------------------------------
error:  Failed writing body (0 != 575) Cause: Failed to inflateinvalid or incomplete deflate data. zlib error code: -3
code:      9001
context:   S3 key being read : ...
...
-----------------------------------------------

What does this mean and what can be done to fix it?

The file is SSE-S3 encrypted, if that matters - which from what I can tell, it shouldn't.

Polysynthetic

Answer 1

This happens when you use gzip option during copy but the file cannot be read as gzip.

Kata

Answer 2

I was facing the same issue, i deleted existing file folder in s3 and re-ran unload script and then copy script.

Zooplasty

Answer 3

in my case, the gz file is intact. It is a legal gzip file. I doublechecked it using file -i and gzip -v -t cmds.

i even unzipped the file, re-zipped it and uploaded it to s3. but still having the same error.

then i found out that the last row in this gz file got "corrupted". it was cut in half somehow, i had to delete this row, re-zip it, and upload it to s3. everything is ok after that

however the mystery is we should have gotten the stl_load_errors if a row is cut in half like this ¯\_(ツ)_/¯ my only guess is that our original gzip file was still corrupted but undetectable using those 2 cmds and i was still able to unzip it.

Dhow