[gdal-dev] VSI_CACHE, open files and memory growth

[Peter Schmitt]

Hi,

I ran into something unexpected with VSI_CACHE while trying to understand
Memory consumption with MapServer
https://lists.osgeo.org/pipermail/mapserver-users/2017-October/080315.html

When MapServer layers have the setting CLOSE_CONNECTION=DEFER (this setting
keeps open file handles open indefinitely), images accessed with /vsis3/,
GDAL config variables GDAL_CACHEMAX=1 and VSI_CACHE=FALSE, my 8 MapServer
processes, Supervisord and Nginx consumes a fixed amount of memory which I
cannot grow (~ 470 mb, as reported by `docker stats`) after I have been
streaming WMS tiles for a while.

When I set VSI_CACHE=TRUE, my 8 MapServer processes grows more than I think
it should. The default VSI_CACHE_SIZE is 25 MB
https://trac.osgeo.org/gdal/wiki/ConfigOptions#VSI_CACHE yet I can get my 8
MapServer processes to consume > 1.4 GB and climbing. I'm not sure how high
it will go as the way I am reproducing this is a bit tedious. I would
expect the memory to level out around 470 + 25 * 8 = 670 MB

Regardless of VSI_CACHE and GDAL_CACHEMAX settings, I get a small memory
footprint when MapServer is configured to close file handles after every
request (CLOSE_CONNECTION=ALWAYS). I've never noticed this issue with
VSI_CACHE and MapServer before because I typically cnopfigure MapServer to
utilize a Tile Index, which defaults to behavior similar to
CLOSE_CONNECTION=ALWAYS. That is, file handles are closed after each
request.

I'm using MapServer-7.0.6 built with GDAL-2.2.2.  Is VSI_CACHE_SIZE the
cache size per process? per file handle?

Thanks,
Pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171027/8b1c6d06/attachment.html>