[mapserver-dev] Mapserver lexer/parser issues

[Lime, Steve D (MNIT)]

Ouch, this looks tough to debug. Are the processes all getting stuck in the same spot or could that be a red herring? Is there any way to re-create the behavior outside of ASP.Net/MapScript using just straight MapServer calls and maybe FastCGI? What sort of lexer/parser interactions are there on each request - there would be the map load but a point query shouldn't use the lexer/parser unless there are layer filters or expressions involved.

From: mapserver-dev <mapserver-dev-bounces at lists.osgeo.org> On Behalf Of Tamas Szekeres
Sent: Friday, August 13, 2021 5:05 AM
To: mapserver-dev at lists.osgeo.org
Subject: [mapserver-dev] Mapserver lexer/parser issues

Dear Developers,

We're currently having an issue with MapServer and I am hoping you can give some idea how to work around this.

We use asp.net<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fasp.net%2F&data=04%7C01%7Csteve.lime%40state.mn.us%7C146e3f692ac0429140db08d95e41e13e%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637644461111528028%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=kbfLO5GflI5L6q6CsAmRNhIkoz1Aw10n5aGhRrrPl7s%3D&reserved=0> and IIS, where the web requests are being served from a pool of threads within a single process. The application runs fine if we just use it manually, but when we run a load test which involves 20 concurrent users or so, the worker process CPU gets stuck at 25% after a while (there are 4 virtual cpu on this box, so it looks like it is maxing one of the vCPU). Every subsequent requests on that particular worker process times out.

The test consists of loading a map, doing a selection (ie queryBypoint) and generate a map, then move to another area in a grid like pattern and do another selection and map export.
Once finished the test starts again automatically, load the map, selection, etc... for each concurrent user. The test complete successfully a few iterations but after a while the CPU gets stuck.
Note that the same request executed successfully many times before.

After the process was getting stucked this way, we took a memory snapshot with some debugging information which shows the following results:

1.  One of the threads is operating in an endless loop in maplexer.c inside the following fragment:

do
{
YY_CHAR yy_c = yy_ec[YY_SC_TO_UI(*yy_cp)] ;
if ( yy_accept[yy_current_state] )
{
(yy_last_accepting_state) = yy_current_state;
(yy_last_accepting_cpos) = yy_cp;
}
while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state )
{
yy_current_state = (int) yy_def[yy_current_state];
if ( yy_current_state >= 2034 )
yy_c = yy_meta[yy_c];
}
yy_current_state = yy_nxt[yy_base[yy_current_state] + yy_c];
++yy_cp;
}
while ( yy_current_state != 2033 );

It looks like yy_current_state never gets the expected value to get out of the loop, but it is not too trivial for me how to interpret this situation in the lexer generated code. It may probably be a bug in the lexer itself or we didn't manage to protect the static variables against the multiple thread access well enough (?)

2. I can also see in the memory dmp that more than 75% of the threads are waiting for the parser lock at msLoadMapFromString which significantly degrades the performance by consuming the available threads from the pool. Do we have any ongoing effort to make the mapfile/expression lexer/parser reentrant?

The application is using a 7.4.x version of MapServer and it doesn't seem to be trivial for us to upgrade in short term, but if you can confirm this specific problem has already been addressed in a later version we'll consider to upgrade.

Thanks,

Tamas


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20210813/39e6dd62/attachment.html>