[gdal-dev] gdalwarp doesn't use the VSIL curl cache ?

[Even Rouault]

On mardi 15 janvier 2019 09:49:53 CET Maxime Riché wrote:
> I am developing a tile server for COGTiff.
> When I open a dataset, the COGTiff metadata are cached in the curl cache.
> The beginning of the file is only downloaded when the first tile is served
> (using the same tiff). This is as I was expecting.
> 
> But when I serve several time the same tile (only this one), I can see that
> the pixels are downloaded every time (the metadata are cached). The pixels
> are not cached by the VSIL curl cache. These pixels are retrieved with
> gdalwarp (using a new dataset everytime).
> Is that the expected behavior? Is there a way to cache the pixels (selected
> by gdalwarp) ?
> 
> (
> It is not a size problem:;
>         - name: VSI_CACHE_SIZE
>           value: "268435456"
>         - name: VSI_CACHE
>           value: "TRUE"
>         - name: CPL_VSIL_CURL_CACHE_SIZE
>           value: "268435456"
> )
> 
> Thanks for the clarifications !

That's a particular effect of using /vsicurl/ (or similar network filesystems) 
with the GeoTIFF driver that does a parallel multi-range request. In that 
mode, it bypasses the vsicurl cache since it uses the exact range requests, 
and doesn't round them on multiple of the 16 KB chunk size. That could 
potentially be improved

Two possible workarounds:
- do not close the dataset if you intend to read it several times. The GDAL 
block cache will then be used and will avoid going down to the /vsicurl I/O 
layer. That was the use case for which this multirange parallelized was 
originally developed
- or define the GDAL_HTTP_MULTIRANGE=SERIAL configuration option that will 
decompose the multirange request in a series of single range request and use 
the vsicurl caching.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com