R: reading geotiff data straight from web url (httr::GET raw content)

Question

I would like to create a RasterLayer from GeoTIFF data provided by a server. I'll query the server for this data using a httr::GET call (the data is provided on-demand, so in the application there won't be a url ending in .tif but a query url).

After writing the result of this call to disk as a GeoTIFF file it's easy enough to create the RasterLayer from the resulting GeoTIFF file on disk:

library(httr)
library(raster)

url <- 'http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif'

geotiff_file <- tempfile(fileext='.tif')
httr::GET(url,httr::write_disk(path=geotiff_file))
my_raster <- raster(geotiff_file)
my_raster

However, I would like to skip the write to disk part and create the raster straight from the in-memory server response.

response <- httr::GET(url,httr::write_memory())
response

The content of the response is a raw string which I would need to interpret as geoTIFF data.

str(httr::content(response))

However, I can only find raster or rgdal functions to read from a file. Any suggestions on translating this raw string to a raster?

Thanks!

Menagerie

Answer 1

GDAL has some cool virtual file system driver, one of which is /vsicurl that

allows on-the-fly random reading of files available through HTTP/FTP web protocols, without prior download of the entire file. It requires GDAL to be built against libcurl.

Since the raster package builds on rgdal you can simply do this:

library(raster)

r <- raster('/vsicurl/http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif')

plot(r)

Jorrie

This does not work for me on windows. Perhaps because the windows rgdal was not built with libcurl? – Larsen

@RobertHijmans Haven't tested it on windows - works on Linux. Could be either an issue with libcurl or the gdal version. Do the other virtual file systems work for you? – Jorrie

Works on Linux, but on Windows the raster() line gives an error: Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", : Cannot create a RasterLayer object from this file. (file does not exist) – Menagerie

I tried with a path to a file on disk, but that gives the same error. Not sure how to check whether other virtual file systems work. – Menagerie

@Menagerie You could try /vsizip' which can read images from a .zip` archive on disk. If that works, it's probably an issue with libcurl. Another option would be trying python GDAL, to see if it's an issue with rgdal – Jorrie

I tried a couple of things on Windows. First off, gdal.Open("/vsicurl/download.osgeo.org/geotiff/samples/gdal_eg/cea.tif") does work in python. In R, gdalinfo() from package gdalUtils (rdocumentation.org/packages/gdalUtils/versions/2.0.1.14/topics/…) on the url works: gdalUtils::gdalinfo('/vsicurl/download.osgeo.org/geotiff/samples/gdal_eg/cea.tif'). Similarly /vsizip on a zip file works with gdalUtils::gdalinfo(), but not with raster(). Seems to point to how the paths are handled by the raster function, strange thing is that it's specific for windows. – Menagerie

@Menagerie nicely tested. Looks like the issue is with rgdal, either with the gdal version or with the libcurl functionality. – Jorrie

A workaround on windows is to build a vrt 1st (QGIS has this same issue) ``` library(gdalUtils) gdalbuildvrt('/vsicurl/download.osgeo.org/geotiff/samples/gdal_eg/…) r <- raster('cea.vrt') ``` – Polyzoarium

It looks like the package rgdak isn't available on CRAN anymore. It's no longer being maintained. Supposedly, the sf package has supplanted it. – Betrothal

This seems to work fine using the terra package instead of raster – Sheet