[mapserver-dev] Re: Large Shape File Performance

[Steve Lime]

Paul: I've tried the patches in two cases and both error'd out.

A simple draw (shp2img) on a shapefile with ~350,000 features and no qix file segfaults in msGetBit(), backtrace is:

#0 0x080ea305 in msGetBit ()
#1 0x080ec88c in msShapeFileLayerNextShape ()
#2 0x080706de in msLayerNextShape ()
#3 0x0807bf35 in msDrawVectorLayer ()
#4 0x0807c2d1 in msDrawLayer ()
#5 0x0807da04 in msDrawMap ()
#6 0x08054546 in main ()

With a qix file it also segfaults in the same place but the back trace is different:

#0  0x080ea305 in msGetBit ()
#1  0x080f4391 in msFilterTreeSearch ()
#2  0x080eb7f9 in msShapefileWhichShapes ()
#3  0x080eb97f in msShapeFileLayerWhichShapes ()
#4  0x0807074b in msLayerWhichShapes ()
#5  0x0807be19 in msDrawVectorLayer ()
#6  0x0807c2d1 in msDrawLayer ()
#7  0x0807da04 in msDrawMap ()
#8  0x08054548 in main ()

A more complex draw using one of our application mapfiles that references a dozen or so layers (all
with indexes) errors out with this:

msDrawMap(): Image handling error. Failed to draw layer named 'bdry_adwma2py3'. <br>
msDBFReadAttribute(): DBASE file error. Invalid record number 1568. <br>

The patches were applied against the dev version. Both applied fine with no errors if I applied bitmap
first and then the shxperf last. This is a SuSE 9.2 system.

My first test case is self contained if you'd like access to it.

Steve

>>> On 4/21/2008 at 12:25 PM, in message
<30fe546d0804211025h16103005p3374dc7e0cbb2cd9 at mail.gmail.com>, "Paul Ramsey"
<pramsey at cleverelephant.ca> wrote:
> All, I have added another patch which addresses scanning the bitmap
> status structure more efficiently. Here's the results for the two
> patches.
> 
> Two performance patches at:
> http://trac.osgeo.org/mapserver/ticket/2282 
> 
> Summary:
> 
> Performance patches radically improved the selective case and
> left the un-selective case in reasonable shape.
> 
> Definitions of tests:
> 
> o Indexed: Render 20 features of 1.8M, using an index scan to only
>  pull the features needed from the SHP file.  Tests the selective
> data access case.
> o Unindexed: Render 20 features of 1.8M, using no index. Forces
> bounds from every feature to be read.  Tests the un-selective
> data access case
> 
> Mapserver 5.0
> 
>   Indexed: 164ms
>   Unindexed: 990ms
> 
> Mapserver SVN + SHX Patch
> 
>   Indexed: 48ms
>   Unindexed: 1015ms
> 
> Mapserver SVN + Bitmap Patch
> 
>   Indexed: 143ms
>   Unindexed: 975ms
> 
> Mapserver SVN + Bitmap Patch + SVN Patch
> 
>   Indexed: 28ms
>   Unindexed: 991ms
> 
> 
> On Sat, Apr 19, 2008 at 9:40 AM, Paul Ramsey <pramsey at cleverelephant.ca> 
> wrote:
>> All,
>>
>>  I've done a patch that seems to improve the large shape file
>>  performance while not overly affecting the standard performance. I am
>>  interested in folks trying it out on their own systems and seeing if
>>  the timings are better for them. Also in people who do file
>>  creation/writing with mapscript confirming I haven't broken that
>>  functionality.
>>
>>  Patch attached to ticket: http://trac.osgeo.org/mapserver/ticket/2282 
>>
>>  Thanks,
>>
>>  Paul
>>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org 
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev