<html>
<head><title></title></head>
<body><div class="iw_mail" dir="ltr">
<p style="margin:0;">Hi,</p>
<p style="margin:0;"><br></p>
<p style="margin:0;">Thanks for the good info Peter !!<br></p>
<p style="margin:0;"><br></p>
<p style="margin:0;">Looking at the text just below these rows:</p>
<p style="margin:0;"><br></p>
<p style="margin:0;"><a href="https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65">https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65</a></p>
<p style="margin:0;"><br></p>
<p style="margin:0;">I see that there is an option that is not documented:</p>
<p style="margin:0;"><br></p>
<p style="margin:0;">CLOSE CONNECTION=NORMAL</p>
<p style="margin:0;"><br></p>
<p style="margin:0;">Shouldn't that option be described in the manual page for LAYER, even if it is the default behavior ?</p>
<p style="margin:0;"><br></p>/Lars Schylberg<br><br><blockquote class="reply_block" dir="ltr" style="border-left: 2px solid #0088CC; margin: 5pt 0 5pt 10pt; padding: 0 0 0 5pt; font-size: 13px; font-family: roboto,tahoma,helvetica,sans-serif;">
<hr size="1">-----Originalmeddelande-----<br>Från: "Peter Schmitt" <<a href="mailto:pschmitt@gmail.com">pschmitt@gmail.com</a>><br>Till: <a href="mailto:mapserver-users@lists.osgeo.org">mapserver-users@lists.osgeo.org</a><br>Datum: 2017-10-27 20:58<br>Ämne: Re: [mapserver-users] Understanding MapServer Memory Usage with WMS raster layers<br><br><div dir="ltr">I think I understand what's going on now and want to share my findings for future Google searchers wondering the same thing.<div><br></div>
<div>I see unbounded memory growth with many raster layers having a single image because the default behavior of raster layers lacking a Tile Index is DEFER.</div>
<div>
<a href="https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L838-L839">https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L838-L839</a><br>
</div>
<div><br></div>
<div>
<div>With DEFER, the Connection Pool will keep the rasters open until mapserv closes.</div>
<div>
<a href="https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65">https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65</a><br>
</div>
<div>My setup (<a href="https://github.com/pedros007/debian-mapserver">https://github.com/pedros007/debian-mapserver</a> Docker container with Supervisord running 8 mapserv processes routed via nginx fcgi) ensures mapserver never exits. So with my Mapfile of many large single-file raster layers results in many large rasters which are opened indefinitely by 8 separate mapserv processes.</div>
</div>
<div><br></div>
<div>My typical Mapfile setup has many raster layers each with a tile index of many rasters. In this configuration, I don't see unbounded memory growth. Looks like the default behavior with a tile index is to close each raster in each MapServer request! <a href="https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L845">https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L845</a>
</div>
<div><br></div>
<div>In short, I can prevent unbounded memory growth by setting CLOSE_CONNECTION=ALWAYS on single-image raster layers. I suppose I figured that out before writing my original question to the mailing list, but at least I now know why this solves my problem. Sorry for adding noise to the list!</div>
<div><br></div>
<div><br></div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">On Wed, Oct 25, 2017 at 11:42 AM, Peter Schmitt <span dir="ltr"><<a href="mailto:pschmitt@gmail.com" target="_blank">pschmitt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div>
<div>I am using MapServer as a WMS endpoint serving rasters. My configuration has Supervisord running 8 mappserv processes which are listening on a socket and dispatched via Nginx FastCGI. This all runs in the following Docker image: <a href="https://github.com/pedros007/debian-mapserver" target="_blank">https://github.com/pedros007/<wbr></wbr>debian-mapserver</a>
</div>
<div><br></div>
<div>I noticed my memory utilization monotonically increases over time only drops when there's a log in dmesg on the host Linux OS (... /not/ in the OS of the Docker container) like the following:</div>
<div><br></div>
<div>
<div>[508072.641250] Memory cgroup out of memory: Kill process 21147 (mapserv) score 143 or sacrifice child</div>
<div>[508072.646072] Killed process 21147 (mapserv) total-vm:672384kB, anon-rss:449172kB, file-rss:0kB, shmem-rss:0kB</div>
<div><br></div>
<div>I can dramatically lower my memory footprint by adding this directive to every raster LAYER as suggested by <a href="https://github.com/mapserver/mapserver/issues/4389" target="_blank">https://github.com/mapserver/<wbr></wbr>mapserver/issues/4389</a>
</div>
<div><br></div>
<div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px">PROCESSING "CLOSE_CONNECTION=ALWAYS"</span></div>
<div><br></div>
<div>With all of that, I have a few questions:</div>
<div><br></div>
<div>1. With the default CLOSE_CONNECTION=DEFER on a raster layer, are raster file handles ever closed?</div>
<div>2. What does the config variable GDAL_CACHEMAX do in relation to CLOSE_CONNECTION = DEFER or ALWAYS? I assume with processing directive CLOSE_CONNECTION=ALWAYS, GDAL_CACHEMAX does nothing for me.</div>
<div>3. My Mapfile actually reads rasters at /vsicurl/ URLs. I assume that VSI_CACHE=TRUE and VSI_CACHE_SIZE are a LRU cache which is in addition to the GDAL cache and any OS/host level caching of open file handles. Is the same data cached separately in up to 3 places (GDAL cache, Host cache and VSI cache)?</div>
<div><br></div>Thanks!<br><div class="m_-2495392079459431185gmail_signature">Pete</div>
<div class="m_-2495392079459431185gmail_signature"><br></div>
</div>
</div></blockquote>
</div>
<br><br clear="all"><div><br></div>--<br><div class="gmail_signature" data-smartmail="gmail_signature">Pete</div>
</div>
<hr>_______________________________________________<br>mapserver-users mailing list<br><a href="mailto:mapserver-users@lists.osgeo.org">mapserver-users@lists.osgeo.org</a><br><a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users">https://lists.osgeo.org/mailman/listinfo/mapserver-users</a>
</blockquote>
<p style="margin:0;"><br></p>
<p style="margin:0;"><br></p>
<div class="signature">
<div class="separator">--</div>
<br> </div>
</div></body>
</html>