[mapserver-dev] RFC 64 - Call for a vote...

[Lime, Steve D (DNR)]

I got a pure, re-entrant parser working this evening (no globals). It's in the sandbox. There are no locks around logical, text or geomtransform expression handling. The flex-based lexer is still not thread safe but it's not called nearly as much as before. In the context of RFC 64 it's used:

  1 - to load a mapfile or symbol file (once, perhaps more if URL overloading is used)
  2 - to tokenize an expression (once per layer in msLayerWhichItems() if necessary)

This is a far cry from once per feature (or worse, once per pixel) in pre-6.0 versions.

I did run into an interesting problem that I'm not sure how to fix. With Bison/yyparse() you have to supply a yylex() function, either generated by flex/lex or written by hand (as I do in this case). So, you have to define a function prototype for yylex() at the top of the grammar. With a re-entrant parser you have to pass a parameter of type YYSTYPE to yylex() (to avoid using globals). Problem is that you can't define a function prototype for yylex() now since YYSTYPE isn't defined until after the prototype appears in mapparser.c. If you don't prototype yylex() then you get compiler warnings. I opted for the warnings in the sandbox. Any idea how to work around this? 

Probably need to move yylex()/yyerror() to their own file and include the header generated by bison. I'd think you'd still need the prototype in the grammar (.y) file.

Steve 
________________________________________
From: mapserver-dev-bounces at lists.osgeo.org [mapserver-dev-bounces at lists.osgeo.org] On Behalf Of Tamas Szekeres [szekerest at gmail.com]
Sent: Monday, November 29, 2010 12:59 PM
To: Lime, Steve D (DNR)
Cc: mapserver-dev at lists.osgeo.org
Subject: Re: [mapserver-dev] RFC 64 - Call for a vote...

Steve,

I just wanted to make sure whether a thread safe approach of passing in stuff to the parser is doable at this stage or not? By looking a the proposed mapparser.y it doesn't seem to be implemented with the current version. I recall Frank has suggested to take a look into the recent parser enhancement in gdal with a reasonable approach: http://trac.osgeo.org/gdal/browser/sandbox/warmerdam/gdal-rfc28/gdal/ogr/swq_parser.y

In case if we follow this version would this be such a big deal (and require to wait for a subsequent major release) if we would want to switch to a thread safe implementation in the future?

Do we have any reason to alter the original RFC number from 59 to 64?


Best regards,

Tamas




2010/11/29 Lime, Steve D (DNR) <Steve.Lime at state.mn.us<mailto:Steve.Lime at state.mn.us>>
It's been a week with little comment, not sure how that should be interpreted. I know it's a long RFC. I feel a little uncomfortable declaring it "passed" based on only two votes though.

Steve

-----Original Message-----
From: mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org> [mailto:mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org>] On Behalf Of Lime, Steve D (DNR)
Sent: Monday, November 22, 2010 11:38 AM
To: Stephen Woodbridge; mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>
Subject: RE: [mapserver-dev] RFC 64 - Call for a vote...

1) Thread safety. At the moment I use thread locks as we always have. So there's no regression there. I'd like to try and make things thread safe (at least for parser, as opposed to flex for mapfiles) for 6.0.

2) I would like to push the IN operator optimization off (unless someone is motivated now). I don't think it's trivial exercise since we need to support both string and numeric lists on the right-hand side.

Steve

-----Original Message-----
From: mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org> [mailto:mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org>] On Behalf Of Stephen Woodbridge
Sent: Monday, November 22, 2010 11:19 AM
To: mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>
Subject: Re: [mapserver-dev] RFC 64 - Call for a vote...

On 11/22/2010 11:39 AM, Lime, Steve D (DNR) wrote:
> Need to act on RFC 64 one way or another so I'll call for a vote.
>
> http://mapserver.org/development/rfc/ms-rfc-64.html

Steve,

What are you thoughts on the TODO list? Are these getting pushed out
post 6.0?

Are you planning to put thread locks around the parser global usage so
you don not break thread safety? As opposed to making the code thread
safe now? I do not think you can regress thread-safety and break that.

For the IN optimization, if the IN list is large, load the tokens into a
btree structure and search will be very fast. I believe this is how
postgres does it internally.

Unless there are some serious concerns from others on issues I'm not
seeing, this seems like a huge amount of value for relatively low risk,
other than changing a lot of code. I think this needs to go into 6.0.

+1 for 6.0

-Steve W
_______________________________________________
mapserver-dev mailing list
mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>
http://lists.osgeo.org/mailman/listinfo/mapserver-dev


_______________________________________________
mapserver-dev mailing list
mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>
http://lists.osgeo.org/mailman/listinfo/mapserver-dev


_______________________________________________
mapserver-dev mailing list
mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>
http://lists.osgeo.org/mailman/listinfo/mapserver-dev