Possible bug in GDAL?

Question

Given this file:

http://mtarchive.geol.iastate.edu/2019/02/18/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_20190218-150000.grib2.gz

… I get different output depending on the software I use.

wgrib2 2019021815.grib2 -csv wgrib2.csv

and

cdo outputtab,date,time,lat,lon,value 2019021815.grib2 > cdo.txt

both output:

-3 (undefined) :  8,869,250 records
0              : 14,848,865 records
Other values   :    781,885 records
___________________________________
Total          : 24,500,000 records

However:

gdal_translate.exe -of xyz 2019021815.grib2 gdal.csv

outputs -3 for all 24,500,000 records.

I get the same result if I parse the file using Python + GDAL.

Is this a bug in GDAL? I'd like to be able to parse these files using Python.

---

I'm using Windows for wgrib2, which can be downloaded from ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/Windows_7/

I'm also using Windows for GDAL, which can be downloaded from https://gdal.org/download.html

I'm using Linux for cdo, which can be downloaded from https://code.mpimet.mpg.de/projects/cdo/

Carnify

Did you ever get this to work? I get the same -3. – Holt

@David, thanks for the confirmation that this doesn't work in GDAL. I may make this a bounty question to get more interest. – Carnify

I work with MRMS quite a bit on my own personal project. It's worth knowing (just in case) that the Weather-Climate Toolkit from NOAA can export MRMS and other gridded/radial data to CSV, json, tiff, etc. If you are ever in a pinch, check it out. – Holt

@David, I've used the Toolkit but hadn't tried exporting to CSV. It's twice as fast as wgrib2 and provides a better output format. And it seems to automatically exclude the undefined values. This doesn't solve the GDAL issue, but it looks to be a better solution altogether. If you make a post and I don't get a better answer, I'll be happy to give you the bounty! – Carnify

thanks. I will do that. Hopefully we will see some other posts on this question. GDAL is a beast. – Holt

I'm assuming you're on Win. Could you share the tools you're using? Are there some that require a license? Did you download them or you build them? Can you share their versions? – Belonging

Good questions, @CristiFati. No license needed, and there are executables and instructions for all at the links I've now added to my post. Latest versions. – Carnify

Ok, it's what I was afraid of (regarding gdal - it comes as an archive from conda repos). And on Win, everything is under Cygwin. Will test, although my experience in this area is somewhere close to 0. – Belonging

Just to add another data point: I get correct results running gdal_translate on macOS. – Beals

Don't go to a lot of trouble. David's solution may be the best. – Carnify

Hmm, that's why I asked about the tool versions. The download URLs remain the same but their content is subject to change (update). I wasn't specific enough. Maybe the same URL will point to different versions for different platforms. @NikolasStevenson-Molnar kind of supports this. I don't know how to get gdal on OSX, but I imagine it's not from the conda repos. Bottom line: if it works on OSX, it should also work on Win (the same code version). – Belonging

@NikolasStevenson-Molnar, interesting! I'll see if there's an update to GDAL that fixes things, but traveling at the moment. – Carnify

Sorry for being pushy, but the gdal download (from conda repos) didn't include (as expected) executables (like gdal_translate). The whls are available at lfd.uci.edu/~gohlke/pythonlibs/#gdal (I don't have to build them myself :) ). – Belonging

@RickHitchcock I'm on version 2.4.2. – Beals

Thanks, @CristiFati. Been a while since I did the installation, and I've been traveling so I couldn't confirm. I actually got the Windows binaries from gisinternals.com/release.php. – Carnify

@NikolasStevenson-Molnar, that version fixes the bug! I'll probably still use David's solution, but your answer answers the question and could qualify for the bounty if you'd like to post. – Carnify

Thanks, @RickHitchcock, I'll turn it into an answer. – Beals

Just as a note, I got that error using gdal_translate (versions 2.4.1 and 3.0.1). which was included in the whls that I mentioned earlier. Anyway, good to hear the problem is fixed! – Belonging

Answer 1

As you suggest, this seems to be a bug in GDAL. I tested the same command using GDAL 2.4.2, and it works. The change log for 2.4.0 points to several fixes to the GRIB driver.

GRIB driver:

• replace DataSource? and derived classes with VSILFILE directly for > 4GB file support on Windows
• turn printf() warning as CPLDebug() messages
• read and write missing data values correctly for complex packing when original data is integer (#1063)
• g2clib: avoid int overflow. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8194
• g2clib: avoid potential out of bound access (CID 1393528)
• degrib: fix floating point division by zero. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9407.
• degrib: avoid potential floating point division by zero. Fixes ​https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=10291
• add UNIT[] node to SRS on reading, so that is valid

Beals

Thank you! The latest version does indeed fix the bug. – Carnify

Answer 2

My answer to your issue would be to switch to using the Weather-Climate Toolkit from NOAA.

Download Here for Windows/Mac/Linux

It can export gridded and radial NEXRAD/MRMS data to CSV. A sample batch command for anyone needing this assistance would be :

wct-export DataSetName.grib2.gz csv Output.csv wctbatchconfig.xml

If it throws errors, try a different/earlier/beta version of WCT from : https://www1.ncdc.noaa.gov/pub/data/wct/dist/ and be sure to check the sample commands on the website.

Holt

Even though you've provided me with a fantastic alternative, I had to give the bounty to Nikolas since he provided a direct answer to the question. But thanks again - I'm working on an API to serve weather data (including MRMS), which you may find useful. – Carnify