[mapserver-dev] Mapserver causes too many open files error
Jeff McKenna
jmckenna at gatewaygeomatics.com
Mon Dec 11 15:31:37 PST 2017
+1 here. Thanks Tamas.
-jeff
On 2017-12-11 5:49 PM, Lime, Steve D (MNIT) wrote:
> I’m ok with the changes… +1 Anyone else?
>
> *From:*Tamas Szekeres [mailto:szekerest at gmail.com]
> *Sent:* Sunday, December 10, 2017 12:06 PM
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us>
> *Cc:* mapserver-dev at lists.osgeo.org
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files error
>
> Hi Steve,
>
> Sorry for the confusion those commits were basically for testing the
> changes.
>
> I've now uploaded the commits I would prefer to see in mapserver/master:
>
> https://github.com/szekerest/mapserver/commit/def90bfed47abaaba7ec12f49eb218494a39cbdc
>
> https://github.com/szekerest/mapserver/commit/b74fd63f6838f1cf20cee7f1951e8cbac8da443e
>
> With regards to the FastCGI configuration I think that should also work
> without the USE_THREAD setting which is not affected with the changes.
> But if we don't set these environment variables, we would get back the
> original behavior.
>
> Best regards,
>
> Tamas
>
> 2017-12-05 21:40 GMT+01:00 Lime, Steve D (MNIT) <steve.lime at state.mn.us
> <mailto:steve.lime at state.mn.us>>:
>
> Seem relatively minor. Couple of questions:
>
> ·Since changes for item 1 would apply more generally, is there any
> downside for FastCGI or should there be a 3^rd state where you have
> threads AND are using Windows (or is that what you mean by being
> switchable).
>
> ·Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?
>
> Steve
>
> *From:*Tamas Szekeres [mailto:szekerest at gmail.com
> <mailto:szekerest at gmail.com>]
> *Sent:* Friday, December 01, 2017 9:51 AM
>
>
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us
> <mailto:steve.lime at state.mn.us>>
> *Cc:* mapserver-dev at lists.osgeo.org
> <mailto:mapserver-dev at lists.osgeo.org>
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files
> error
>
> This change demonstrates what we need in #1
>
> https://github.com/szekerest/mapserver/commit/87cacc6a37e802bc3af16656d45966c05deb4d62
>
> For #2 this is the example:
>
> https://github.com/szekerest/mapserver/commit/e16069972b43f837b47d0f736c2df4d5c0c7a475
>
> Both of them should be switchable which is not yet added to this
> code example.
>
> Best regards,
>
> Tamas
>
> 2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT)
> <steve.lime at state.mn.us <mailto:steve.lime at state.mn.us>>:
>
> What does #1 entail? The other change is Windows specific so
> seems ok – I don’t know what the right value should be of course.
>
> *From:*Tamas Szekeres [mailto:szekerest at gmail.com
> <mailto:szekerest at gmail.com>]
> *Sent:* Friday, December 01, 2017 1:53 AM
> *To:* Lime, Steve D (MNIT) <steve.lime at state.mn.us
> <mailto:steve.lime at state.mn.us>>
> *Cc:* mapserver-dev at lists.osgeo.org
> <mailto:mapserver-dev at lists.osgeo.org>
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open
> files error
>
> We did some investigation in this regard and the following 2
> changes could solve this specific problem on Windows and ASP.NET
> <http://ASP.NET>.
>
> 1. Adding the ability to use a single cache instead of a
> per-thread cache.
>
> 2. Increase the number of open stdio file handle limit for
> Windows. Default is 512 the upper limit is 2048 accoding to this
> documentation:
> https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
>
> These settings should probably be controlled with compile
> options or environment variables.
>
> Any objections to apply such changes?
>
> Best regards,
>
> Tamas
>
> 2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT)
> <steve.lime at state.mn.us <mailto:steve.lime at state.mn.us>>:
>
> The former is certainly simpler. How expensive is setting up
> and clearing out the cache? Probably should look at Thomas’
> RFC on the topic.
>
> *From:*mapserver-dev
> [mailto:mapserver-dev-bounces at lists.osgeo.org
> <mailto:mapserver-dev-bounces at lists.osgeo.org>] *On Behalf
> Of *Tamas Szekeres
> *Sent:* Sunday, November 26, 2017 2:01 PM
> *To:* mapserver-dev at lists.osgeo.org
> <mailto:mapserver-dev at lists.osgeo.org>
> *Subject:* [mapserver-dev] Mapserver causes too many open
> files error
>
> Hi Devs,
>
> We experienced an issue with mapserver 7.x (comparing to
> 6.x) that it causes too many open file handles in a
> multi-threaded environment. Actually the process is using
> the ASP.NET <http://ASP.NET> backend where the web requests
> are served through the ASP.NET <http://ASP.NET> worker
> process by using a thread pool, and the process should be
> considered as a long running process, where msCleanup is not
> being called regularly.
>
> By looking into the code the issue should probably be
> dedicated to the global font cache approach implemented in
> mapserver 7. Since freetype keeps the font files open, we
> get the number of fonts used in the mapfile multiplied with
> the thread pool size which may exceed the number of stdio
> open file handle limit as per
> https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx
>
> With the former approach (ie. 6.x) the font cache was
> implemented in the renderers and the cache cleanup was
> happening on the renderer cleanup
> in msFreeOutputFormat/msFreeMap.
>
> Do you have any suggestions about a proper workaround to this?
>
> I might think that we should either bind the font cache to
> each map objects (like earlier) or use a single common font
> cache for the entire process. I would probably prefer the
> first one as soon as we want to clean up every resources
> what a rendering session allocates.
>
> Thanks,
>
> Tamas
>
>
More information about the mapserver-dev
mailing list