[gdal-dev] optimal vsicurl settings for merging range requests

[Scott Henderson]

There are many environment variables for optimizing access to remote geotiffs via GDAL’s /vsicurl/ interface. I’ve noticed that directly downloading a file via curl can be much faster (~4x) than pulling the entire file via gdal_translate or gdalmanage, and I think this is due to 1 GET request versus 10s of GET requests. Here is an example using Landsat8 on Google Cloud:

In brief it seems there is a limit for GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES that I’m missing. If the entire file is requested, I was expecting 2 GET requests (1 to read the metadata and another to retrieve the data).

Static:
https://github.com/scottyhq/pangeo-cog-test/blob/master/cog-timing.ipynb <https://github.com/scottyhq/pangeo-cog-test/blob/master/cog-timing.ipynb> 

Interactive:
https://mybinder.org/v2/gh/scottyhq/pangeo-cog-test/master?urlpath=lab/tree/cog-timing.ipynb <https://mybinder.org/v2/gh/scottyhq/pangeo-cog-test/master?urlpath=lab/tree/cog-timing.ipynb> 

Just the command:
CPL_VSIL_CURL_USE_HEAD=NO GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.TIF CPL_CURL_VERBOSE=YES GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES gdal_translate --debug ON /vsicurl/http://storage.googleapis.com/gcp-public-data-landsat/LC08/01/047/027/LC08_L1TP_047027_20130421_20170310_01_T1/LC08_L1TP_047027_20130421_20170310_01_T1_B4.TIF LC08_L1TP_047027_20130421_20170310_01_T1_B4.TIF

Thanks for any hints or clarification!
Scott



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181218/b3f0651c/attachment.html>