[mapserver-users] Understanding MapServer Memory Usage with WMS raster layers

[Peter Schmitt]

I think I understand what's going on now and want to share my findings for
future Google searchers wondering the same thing.

I see unbounded memory growth with many raster layers having a single image
because the default behavior of raster layers lacking a Tile Index is DEFER.
https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L838-L839

With DEFER, the Connection Pool will keep the rasters open until mapserv
closes.
https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65
My setup (https://github.com/pedros007/debian-mapserver Docker container
with Supervisord running 8 mapserv processes routed via nginx fcgi) ensures
mapserver never exits.  So with my Mapfile of many large single-file raster
layers results in many large rasters which are opened indefinitely by 8
separate mapserv processes.

My typical Mapfile setup has many raster layers each with a tile index of
many rasters. In this configuration, I don't see unbounded memory growth.
Looks like the default behavior with a tile index is to close each raster
in each MapServer request!
https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L845

In short, I can prevent unbounded memory growth by setting
CLOSE_CONNECTION=ALWAYS on single-image raster layers.  I suppose I figured
that out before writing my original question to the mailing list, but at
least I now know why this solves my problem.  Sorry for adding noise to the
list!



On Wed, Oct 25, 2017 at 11:42 AM, Peter Schmitt <pschmitt at gmail.com> wrote:

> Hi,
>
> I am using MapServer as a WMS endpoint serving rasters. My configuration
> has Supervisord running 8 mappserv processes which are listening on a
> socket and dispatched via Nginx FastCGI. This all runs in the following
> Docker image: https://github.com/pedros007/debian-mapserver
>
> I noticed my memory utilization monotonically increases over time only
> drops when there's a log in dmesg on the host Linux OS (... /not/ in the OS
> of the Docker container) like the following:
>
>       [508072.641250] Memory cgroup out of memory: Kill process 21147
> (mapserv) score 143 or sacrifice child
>       [508072.646072] Killed process 21147 (mapserv) total-vm:672384kB,
> anon-rss:449172kB, file-rss:0kB, shmem-rss:0kB
>
> I can dramatically lower my memory footprint by adding this directive to
> every raster LAYER as suggested by  https://github.com/mapserver/
> mapserver/issues/4389
>
>       PROCESSING "CLOSE_CONNECTION=ALWAYS"
>
> With all of that, I have a few questions:
>
> 1.  With the default CLOSE_CONNECTION=DEFER on a raster layer, are raster
> file handles ever closed?
> 2.  What does the config variable GDAL_CACHEMAX do in relation to
> CLOSE_CONNECTION = DEFER or ALWAYS?  I assume with processing directive
> CLOSE_CONNECTION=ALWAYS, GDAL_CACHEMAX does nothing for me.
> 3.  My Mapfile actually reads rasters at /vsicurl/ URLs.  I assume that
> VSI_CACHE=TRUE and VSI_CACHE_SIZE are a LRU cache which is in addition to
> the GDAL cache and any OS/host level caching of open file handles. Is the
> same data cached separately in up to 3 places (GDAL cache, Host cache and
> VSI cache)?
>
> Thanks!
> Pete
>
>


-- 
Pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20171027/5d119ccf/attachment.html>