[gdal-dev] Way to increase global LRU cache of the /vsis3/ driver

[Even Rouault]

On mardi 13 juin 2017 08:24:59 CEST teeschke wrote:
> Dear GDAL Devs,
> 
> the vsis3 driver truly is a masterpiece of a driver!!
> http://www.gdal.org/cpl__vsi_8h.html#a5b4754999acd06444bfda172ff2aaa16
> 
> I just wonder if there is any possibility to increase the global LRU cache
> which has a fixed size of 16MB.
> 
> In my use-case I have time-related Gribfiles, stored on S3. The timestamp is
> part of the filename. The front-end uses _tiles_ and has an _animation_
> functionality. That ends up in lots of data accesses from Mapserver to S3
> and I hope that increasing the global LRU cache to 2GB would be a
> performance kick.

In

https://github.com/OSGeo/gdal/blob/trunk/gdal/port/cpl_vsil_curl.cpp#L101

replace
static const int N_MAX_REGIONS = 1000;
with
static const int N_MAX_REGIONS = 122070;

(122070 * 16384 ~= 2 GB)

and recompile

However the algorithm that handles cached region eviction/lookup might exhibit bad 
performance with such a big numbers since it is rather native right now, so I'd suggest you to 
test carefully. It should probably use a hashset rather than an array.

Ultimately I'd like that this setting to be user tunable, so might be worth that you file an 
enhancement ticket at https://trac.osgeo.org/gdal/newticket

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20170613/bf0eb69a/attachment.html>