[mapserver-dev] Mapserver causes too many open files error

[Tamas Szekeres]

This change demonstrates what we need in #1

https://github.com/szekerest/mapserver/commit/87cacc6a37e802bc3af16656d45966c05deb4d62

For #2 this is the example:

https://github.com/szekerest/mapserver/commit/e16069972b43f837b47d0f736c2df4d5c0c7a475

Both of them should be switchable which is not yet added to this code
example.

Best regards,

Tamas



2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us>:

> What does #1 entail? The other change is Windows specific so seems ok – I
> don’t know what the right value should be of course.
>
>
>
> *From:* Tamas Szekeres [mailto:szekerest at gmail.com]
> *Sent:* Friday, December 01, 2017 1:53 AM
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us>
> *Cc:* mapserver-dev at lists.osgeo.org
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files error
>
>
>
> We did some investigation in this regard and the following 2 changes could
> solve this specific problem on Windows and ASP.NET.
>
>
>
> 1. Adding the ability to use a single cache instead of a per-thread cache.
>
>
>
> 2. Increase the number of open stdio file handle limit for Windows.
> Default is 512 the upper limit is 2048 accoding to this documentation:
> https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
>
>
>
>
>
> These settings should probably be controlled with compile options or
> environment variables.
>
>
>
> Any objections to apply such changes?
>
>
>
> Best regards,
>
>
>
> Tamas
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us>:
>
> The former is certainly simpler. How expensive is setting up and clearing
> out the cache? Probably should look at Thomas’ RFC on the topic.
>
>
>
> *From:* mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org] *On
> Behalf Of *Tamas Szekeres
> *Sent:* Sunday, November 26, 2017 2:01 PM
> *To:* mapserver-dev at lists.osgeo.org
> *Subject:* [mapserver-dev] Mapserver causes too many open files error
>
>
>
> Hi Devs,
>
>
>
> We experienced an issue with mapserver 7.x (comparing to 6.x) that it
> causes too many open file handles in a multi-threaded environment. Actually
> the process is using the ASP.NET backend where the web requests are
> served through the ASP.NET worker process by using a thread pool, and the
> process should be considered as a long running process, where msCleanup is
> not being called regularly.
>
>
>
> By looking into the code the issue should probably be dedicated to the
> global font cache approach implemented in mapserver 7. Since freetype keeps
> the font files open, we get the number of fonts used in the mapfile
> multiplied with the thread pool size which may exceed the number of stdio
> open file handle limit as per https://msdn.microsoft.
> com/en-us/library/kdfaxaay(v=vs.140).aspx
>
>
>
> With the former approach (ie. 6.x) the font cache was implemented in the
> renderers and the cache cleanup was happening on the renderer cleanup
> in msFreeOutputFormat/msFreeMap.
>
>
>
> Do you have any suggestions about a proper workaround to this?
>
>
>
> I might think that we should either bind the font cache to each map
> objects (like earlier) or use a single common font cache for the entire
> process. I would probably prefer the first one as soon as we want to clean
> up every resources what a rendering session allocates.
>
>
>
> Thanks,
>
>
>
> Tamas
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20171201/c98cb555/attachment.html>