[mapserver-dev] mappostgis.c

[Paul Ramsey]

No, epsg definitions are looked up by mapserver, or rather, by proj,
in the proj library. The PostGIS driver is "srs stupid".  PostGIS
layers suffer from exactly the same issues in this regard as any other
layer.

I think the expedient answer is an in-memory cache of PJ structs,
filled up on an "as needed basis".

I think of it like this:

You have a map file with 10 layers. Odds are, all/most of your data is
going to be in the same SRS. We cache the first object in memory, and
the rest use it. Instant 10x performance boost. Add in FCGI and the
boost goes to infinity (or something). It's something that is
self-contained in Mapserver and will work right away, on old versions
of Proj.

Paul

On Tue, Nov 4, 2008 at 6:11 PM, Stephen Woodbridge
<woodbri at swoodbridge.com> wrote:
> Daniel,
>
> Thank you for the excellent explanation that make prefect sense. So project
> is what needs to be fixed :)
>
> Well it might be fixed if PROJ had a very small abstraction API to
> virtualize the access to the EPSG data files and parsing the data. Then
> existing apps could use the default file access driver, but anyone so
> inclined could overload the methods with their own drivers to postgres,
> sqlite, memcache, whatever.
>
> So postGIS just implements its own version of PROJ library or what, given
> that it has its own spatial_ref_sys table or it just feeds the proj4text
> column into the existing PROJ functions.
>
> Does postGIS have the same performance issues with projections?
> If not, why not? can the postGIS pattern be mirrored in mapserver for
> example?
> Since Paul is looking into this, I'm sure he has thought of these things and
> their is probably some good reason why this will not work efficiently.
>
> -Steve W
>
>
>
> Daniel Morissette wrote:
>>
>> At the moment, the handling of the epsg definitions uses a mechanism
>> that's built into PROJ and that is shared by all apps that use PROJ on a
>> given system. That doesn't mean things can't or shouldn't change, but if we
>> come up with our own mechanism then it will be MapServer-specific and won't
>> be shared by other apps that use PROJ... unless we embed the new mechanism
>> directly in PROJ, but I doubt that adding a SQLite dependency in PROJ would
>> fly very well.
>>
>> All that being said, fixing that epsg performance issue has been on my
>> list for a while (for too long)... I just need to find the time to do it.
>>
>> Daniel
>>
>> Stephen Woodbridge wrote:
>>>
>>> Paul, Steve,
>>>
>>> I think I idly asked this in the past and it was ignored. Why would we
>>> not consider using something like SQLite to:
>>>  1) cache the parse epsg codes into, or
>>>  2) just compile the epsg codes into a SQLite DB that could be accessed
>>> instead of the textfile.
>>>
>>> Here are some random thoughts on the subject:
>>>
>>> o the data could be preparsed into fields or a C struct that is copied
>>> into a blob and retrieved as needed
>>> o SQLite licensing is compatible with mapserver
>>> o yes - people might not want to have to compile the epsg file into a DB
>>> o yes - it increases the technology footprint slightly
>>> o but it might be useful for many other mapserver caching needs
>>> o its thread safe, stable and very fast
>>> o it could be configured to support both models: DB or text file by a
>>> PROCESSING directive in the MAP PROJECT block to point to the DB. Although
>>> it is probably cleaner to just support one. A simple utility could be
>>> provided like "epsg2db /path/to/proj.db" to compile the text file into a db.
>>>
>>> I'm just thinking out loud here. Does this make sense to anyone else?
>>> 25% of the processing time is a huge win! that means a server would have
>>> 25% more capacity or you could serve the same number of requests with 75% of
>>> the hardware you currently need.
>>>
>>> -Steve W
>>>
>>> Paul Ramsey wrote:
>>>>
>>>> On Tue, Nov 4, 2008 at 10:54 AM, Steve Lime
>>>> <Steve.Lime at dnr.state.mn.us> wrote:
>>>>>
>>>>> A good threat, probably long overdue. Do you have any feel for the
>>>>> performance end of things? I recall the Geoserver tests showing
>>>>> GeoServer performing substantially better than MapServer w/regards to
>>>>> PostGIS.
>>>>
>>>> OK, this isn't testing against Geoserver, just against 5.2, and the
>>>> result is, 5.4trunk is maybe 5% slower. Which is not unsurprising, since now
>>>> the geometry is transiting a text encoding on the way. I
>>>> had my fingers crossed that the simpler database handling would speed
>>>>  things up, but no go.
>>>>
>>>> In terms of straight-up profiling information:
>>>>
>>>> 25% of cycles are still spent in pj_init, and this is *after* I moved
>>>>  the two relevant EPSG codes to the top of the file. Geoserver has
>>>> objects cached. 20% of cycles are spent in image compression (gif or
>>>> png). The Geoserver ImageIO might have optimized image compressors
>>>> (my work with IPP impresses on me how much gain is available in this
>>>> area).
>>>>
>>>> The penalty for parsing the proj4 definitions every time, even for
>>>> layers that aren't being drawn, is certainly high, and of course
>>>> grows the more layers you have.  However, these overheads also exist
>>>> in the shape file chain, so the difference between PostGIS and Shape
>>>> relative to Geoserver remains unexplained.
>>>>
>>>> P. _______________________________________________ mapserver-dev
>>>> mailing list mapserver-dev at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>
>>> _______________________________________________
>>> mapserver-dev mailing list
>>> mapserver-dev at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>>
>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>