[mapserver-dev] Mapserver causes too many open files error

[Tamas Szekeres]

Hi Steve,

Sorry for the confusion those commits were basically for testing the
changes.
I've now uploaded the commits I would prefer to see in mapserver/master:

https://github.com/szekerest/mapserver/commit/def90bfed47abaaba7ec12f49eb218494a39cbdc

https://github.com/szekerest/mapserver/commit/b74fd63f6838f1cf20cee7f1951e8cbac8da443e

With regards to the FastCGI configuration I think that should also work
without the USE_THREAD setting which is not affected with the changes. But
if we don't set these environment variables, we would get back the original
behavior.

Best regards,

Tamas




2017-12-05 21:40 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us>:

> Seem relatively minor. Couple of questions:
>
>
>
> ·         Since changes for item 1 would apply more generally, is there
> any downside for FastCGI or should there be a 3rd state where you have
> threads AND are using Windows (or is that what you mean by being
> switchable).
>
> ·         Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?
>
>
>
> Steve
>
>
>
> *From:* Tamas Szekeres [mailto:szekerest at gmail.com]
> *Sent:* Friday, December 01, 2017 9:51 AM
>
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us>
> *Cc:* mapserver-dev at lists.osgeo.org
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files error
>
>
>
> This change demonstrates what we need in #1
>
>
>
> https://github.com/szekerest/mapserver/commit/
> 87cacc6a37e802bc3af16656d45966c05deb4d62
>
>
>
> For #2 this is the example:
>
>
>
> https://github.com/szekerest/mapserver/commit/
> e16069972b43f837b47d0f736c2df4d5c0c7a475
>
>
>
> Both of them should be switchable which is not yet added to this code
> example.
>
>
>
> Best regards,
>
>
>
> Tamas
>
>
>
>
>
>
>
> 2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us>:
>
> What does #1 entail? The other change is Windows specific so seems ok – I
> don’t know what the right value should be of course.
>
>
>
> *From:* Tamas Szekeres [mailto:szekerest at gmail.com]
> *Sent:* Friday, December 01, 2017 1:53 AM
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us>
> *Cc:* mapserver-dev at lists.osgeo.org
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files error
>
>
>
> We did some investigation in this regard and the following 2 changes could
> solve this specific problem on Windows and ASP.NET.
>
>
>
> 1. Adding the ability to use a single cache instead of a per-thread cache.
>
>
>
> 2. Increase the number of open stdio file handle limit for Windows.
> Default is 512 the upper limit is 2048 accoding to this documentation:
> https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
>
>
>
>
>
> These settings should probably be controlled with compile options or
> environment variables.
>
>
>
> Any objections to apply such changes?
>
>
>
> Best regards,
>
>
>
> Tamas
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us>:
>
> The former is certainly simpler. How expensive is setting up and clearing
> out the cache? Probably should look at Thomas’ RFC on the topic.
>
>
>
> *From:* mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org] *On
> Behalf Of *Tamas Szekeres
> *Sent:* Sunday, November 26, 2017 2:01 PM
> *To:* mapserver-dev at lists.osgeo.org
> *Subject:* [mapserver-dev] Mapserver causes too many open files error
>
>
>
> Hi Devs,
>
>
>
> We experienced an issue with mapserver 7.x (comparing to 6.x) that it
> causes too many open file handles in a multi-threaded environment. Actually
> the process is using the ASP.NET backend where the web requests are
> served through the ASP.NET worker process by using a thread pool, and the
> process should be considered as a long running process, where msCleanup is
> not being called regularly.
>
>
>
> By looking into the code the issue should probably be dedicated to the
> global font cache approach implemented in mapserver 7. Since freetype keeps
> the font files open, we get the number of fonts used in the mapfile
> multiplied with the thread pool size which may exceed the number of stdio
> open file handle limit as per https://msdn.microsoft.
> com/en-us/library/kdfaxaay(v=vs.140).aspx
>
>
>
> With the former approach (ie. 6.x) the font cache was implemented in the
> renderers and the cache cleanup was happening on the renderer cleanup
> in msFreeOutputFormat/msFreeMap.
>
>
>
> Do you have any suggestions about a proper workaround to this?
>
>
>
> I might think that we should either bind the font cache to each map
> objects (like earlier) or use a single common font cache for the entire
> process. I would probably prefer the first one as soon as we want to clean
> up every resources what a rendering session allocates.
>
>
>
> Thanks,
>
>
>
> Tamas
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20171210/42627823/attachment.html>