GZipStream complains magic number in header is not correct

Question

I'm attempting to use National Weather Service (U.S.) data, but something has changed recently and the GZip file no longer opens.

.NET 4.5 complains that...

Message=The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
Source=System
StackTrace:
   at System.IO.Compression.GZipDecoder.ReadHeader(InputBuffer input)
   at System.IO.Compression.Inflater.Decode()
   at System.IO.Compression.Inflater.Inflate(Byte[] bytes, Int32 offset, Int32 length)
   at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)

I don't understand what has changed, but this is becoming a real show-stopper. Can anyone with GZip format experience tell me what has changed to make this stop working?

A file that works:

http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201504/20150404/nws_precip_2015040420.tar.gz

A file that doesn't work:

http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz

Update with sample code

const string url = "http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz";
string appPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
string downloadPath = Path.Combine(appPath, Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "nws_precip_2015050505.tar.gz");
using (var wc = new WebClient())
{
    wc.DownloadFile(url, downloadPath);
}

string extractDirPath = Path.Combine(appPath, "Extracted");
if (!Directory.Exists(extractDirPath))
{
    Directory.CreateDirectory(extractDirPath);
}
string extractFilePath = Path.Combine(extractDirPath, "nws_precip_2015050505.tar");

using (var fsIn = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
using (var fsOut = new FileStream(extractFilePath, FileMode.Create, FileAccess.Write))
using (var gz = new GZipStream(fsIn, CompressionMode.Decompress, true))
{
    gz.CopyTo(fsOut);
}

Their

Yep, both files are fine. How are you getting the file? – Sparky

Apologies -- I had the incorrect URL posted for the failure URL. Sample code is attached. – Their

The links for the working and not working files aren't active anymore. – Lowis

@JavierRuiz I don't think the file contents are particularly relevant anymore. Check the accepted answer - I believe the server was returning malformed data at the time. – Their

Answer 1

It appears that this service SOMETIMES returns tar format files disguised as .tar.gz. This is very confusing, but if you check that the first two bytes are 0x1F and 0x8B, you can detect if the file is a GZip by checking its magic numbers manually.

using (FileStream fs = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
{
    byte[] buffer = new byte[2];
    fs.Read(buffer, 0, buffer.Length);
    if (buffer[0] == 0x1F
        && buffer[1] == 0x8B)
    {
        // It's probably a GZip file
    }
    else
    {
        // It's probably not a GZip file
    }
}

Their

Holy crap. These stupid qpe files were giving me some serious trouble. This message should be tagged with "Error GZIP header, first magic byte doesn't match" and "The magic number in GZip header is not correct. Make sure you are passing in a GZip stream." Thank you so much for solving this! – Roseliaroselin

Same problem with ZIP files downloaded from Google Cloud Platform buckets to my Windows machine: they have a .gz extension but 7ZIP info tell me they're actually zip files. – J

Answer 2

[Resolved] GZipStream complains magic number in header is not correct

//Exception magic number tar.gz file

Migcal error cause

1. File is not compress into tar.gz properly
2. File size is too big , above 1+GB

Solution over it

1. use .net framework 4.5.1 to over the this exception //OR//

2. manupulate the exsiting solution without change .net framework. Please follow the step for implementation.

   1. Remane abc.tar.gz as abc (remove extension).

   2. pass this file and directly name to compress function

*public static void Compress(DirectoryInfo directorySelected, string directoryPath)

    {
        foreach (FileInfo fileToCompress in directorySelected.GetFiles())
        {
            using (FileStream originalFileStream = fileToCompress.OpenRead())
            {
                if ((File.GetAttributes(fileToCompress.FullName) &
                   FileAttributes.Hidden) != FileAttributes.Hidden & fileToCompress.Extension != ".tar.gz")
                {
                    using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".tar.gz"))
                    {
                        using (System.IO.Compression.GZipStream compressionStream = new System.IO.Compression.GZipStream(compressedFileStream,
                           System.IO.Compression.CompressionMode.Compress))
                        {
                            originalFileStream.CopyTo(compressionStream);
                        }
                    }
                    FileInfo info = new FileInfo(directoryPath + "\\" + fileToCompress.Name + ".tar.gz");
                }
            }
        }
    }


3. implement this code in following exception handler try catch block
    try
    {
        TarGzFilePath=@"c:\temp\abc.tar.gz";
        FileStream streams = File.OpenRead(TarGzFilePath);
        string FileName=string.Empty;
        GZipInputStream tarGz = new GZipInputStream(streams);
        TarInputStream tar = new TarInputStream(tarGz);
        // exception will occured in below lines should apply try catch

        TarEntry ze;
            try
            {
                ze = tar.GetNextEntry();// exception occured here "magical number"
            }
            catch (Exception extra)
            {
                tar.Close();
                tarGz.Close();
                streams.Close();
                //please close all above , other wise it will come with exception "tihs process use by another process"
                //rename your file *for better accuracy you can copy file to other location 

                File.Move(@"c:\temp\abc.tar.gz", @"c:\temp\abc"); // rename file 
                DirectoryInfo directorySelected = new DirectoryInfo(Path.GetDirectoryName(@"c:\temp\abc"));
                Compress(directorySelected, directoryPath); // directorySelected=c:\temp\abc , directoryPath=c:\temp\abc.tar.gz // process in step 2 function 
                streams = File.OpenRead(TarGzFilePath);
                tarGz = new GZipInputStream(streams);
                tar = new TarInputStream(tarGz);
                ze = tar.GetNextEntry();                    
            }
            // do anything with extraction with your code
    }
    catch (exception ex)
    {
        tar.Close();
        tarGz.Close();
        streams.Close();
    }

Onepiece