[mapserver-users] fastcgi?

[Frank Warmerdam]

Bart van den Eijnden wrote:
> Hi list,
> 
> I see that some thread-safety issues have been addressed in the latest 
> version.
> 
> Are there any plans to implement mapserver as a fast-cgi, so that the 
> map configuration file does not have to be parsed all the time and 
> database connections can be "persistent"?

Bart,

I discussed this with Daniel a bit on the weekend, and have briefly reviewed
the FastCGI docs, and I think this would be a useful thing to implement for
some cases.  I don't think that map file parsing time is likely to be
significant most of the time, and a straightforward implementation of FastCGI
support is unlikely to avoid reparsing the mapfile anyways.  However, in the
short term I think it would help a great deal with database connections
which can be expensive to setup, and initializing the FMEObjects library for
use with MapServer.

So, I propose to try and implement simple FastCGI support for MapServer in
time for the 4.0 release.  This would basically just wrap the mainloop of
"mapserv".  I will look at enabling multi-threaded fast cgi use, but for
the usual reasons (big locks around some of the slower sections like GDAL
reading) running multiple threads is unlikely to help much in some cases.

What this initial effort will allow for is sharing connection setup, and
expensive library initializations over a whole bunch of cgi-bin requests
instead of redoing it for each.  I imagine there will be some sort of a
--fast-cgi configure option that would build fastcgi support into the
mapserv executable (or perhaps we will need to link a separate fastmapserv
executable).  The effort will also involve some checking to ensure there
aren't per render memory leaks.

In the future I would like to see:
  o A mechanism to preserve open handles for GDAL datasets.  Some raster
    formats are a bit expensive to open with GDAL.  Holding open the
    datasets would also provide dramatic improvments in some cases in that
    the existing GDAL raster data caching mechanism would then be useful.
    Note that GDAL already has a concept of "sharing" a GDAL dataset
    intended to allow multiple parts of an application to share a given
    GDAL dataset handle via a reference counting mechanism.  So I think for
    this to work well we would just have to implement a mechanism in the
    .map file (or something else controlling the FastCGI configuration
    perhaps) to declare GDAL files that should be opened when the server
    starts and left open till it terminates.

  o A mechanism to allowing preserving OGR dataset connections similarly.
    There is also a new "shared dataset" mechanism in OGR so a similiar
    approach to GDAL should be applicable.  Note that OGR does not generally
    provide any feature caching logic, so the main savings with OGR caching
    is to avoid the cost of reopening datasets all the time.  For some
    formats this is high.

  o A mechanism to preserve a parsed mapfile.  I don't know if there are
    issues with this though.  For instance, if URL parameters modify some
    values in the in-memory mapObj and layerObjs then they can't be reused
    directly.  Did Sean or someone implement a complete mechanism to
    copy mapObj's and all the associated stuff?

  o In the even longer term I would like to implement a more fine grained
    locking within GDAL so that the "big lock" around GDAL won't be such
    an issue.

I don't forsee alot of work implementing caching or preserving access
to shapefiles.  This is already very lightweight and fast as implemented
and so should need much work.  Nor would I add will a well organized
shapefile based map gain much from FastCGI support.  This is mainly
aimed at various addin libraries.

I don't know what the implications of FastCGI support for mapscript-cgi
applications are.  I am not really well organized to deal with that, but
perhaps once it is working well for mapserv that could be investigated.

I would add that FastCGI support will also benefit from cleanup in
MapServ related to multi-threaded support for server loadable modules
and so forth.

I am interested in hearing from a few folks who would be interested in
testing FastCGI support as I am working on it.

I imagine further FastCGI discussions should be moved to the -dev list
untill it is working well.

PS. I really wish I had made it to MUM.  It sounds like it was a blast.
I have already been talking to the guys about seminars I might present at
next years meeting on PROJ, GDAL and raster support in general.   Perhaps
even on FastCGI and multi-threading issues!

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent