VS: [mapserver-users] Mapserver search performance

[Varun saraf]

Hi,

I tried the shptree tool but did not see any performance improvement.
It fetched results about 4-5 seconds faster. Without QIX files, it was
5 minutes and with QIX files, it took about 4 minutes and 56 seconds.
All of my requests are based on the "MAPSHAPE" parameter in NQUERY
mode. I noticed that mapserver was taking about 1 second for
outputting the information in the template file for each feature.
Could it be because all of this information that I require is coming
from an external DBF file that I join to the layer/shape's DBF? Will
including all these fields/information in the shape's DBF file itself
help? Does mapserver pick up each feature and use a point in polygon
approach to check if it lies in the provided shape? If yes, how can i
make this process faster. Is there a way of making it multi-threaded
or do more checks per second?

I am providing sample code. Hope this helps. I have about 35 layers in
my map file. Should I merge all these into a single layer?

MAP FILE
========
    LAYER
        NAME "L1"
    	METADATA
    	    qstring_validation_pattern '.'
    	END
        STATUS DEFAULT
        TYPE point
        DATA BLKS_01
        TOLERANCE 0
        TOLERANCEUNITS METERS
        CLASS
	   STYLE
	      OUTLINECOLOR 255 0 0
	    END	
	END
	
	JOIN
	    NAME "external"
    	    TABLE "data/externalData.dbf"
            FROM "Field1"
    	    TO "Field1"
    	    TYPE ONE-TO-ONE
  	END
  	
        TEMPLATE "blockTemplate.xml"
    END

Template File
==========
[GEOID],[Field1],[LON],[LAT],[external_ID],[external_NAME],[external_STA],[external_NEEDSCORE],[external_MINSCORE];

Request
=======
http://localhost/cgi-bin/mapserv.exe?map=C:/ms4w/apps/GIS/centroides.map&mode=nquery&mapshape=-99.757833
32.474433 -99.758005 32.450679 -99.726591 32.447637 -99.727621
32.474723

Any help in this regard is greatly appreciated. Let me know if you
need any more information.

Thanks,
Varun

On Mon, Apr 11, 2011 at 3:06 PM, Varun saraf <vsaraf.gmu at gmail.com> wrote:
> Thanks a lot for the quick reply. I will give it a try.
>
> On Mon, Apr 11, 2011 at 1:50 PM, Rahkonen Jukka
> <Jukka.Rahkonen at mmmtike.fi> wrote:
>> Hi,
>>
>> Shptree will help and stopwatch will tell you how much. Without spatial
>> index Mapserver needs to go through the whole shapefile every time. Have
>> a try, it will not take very many seconds to run shptree. Make different
>> requests, take times with and without .qix files and you will some
>> numbers. Change the requests and request order (query 1 with
>> .qix/withour qix, query 2 without .qix/with .qix) so that you can see if
>> you are actually testing the speed of disk access and memory cache
>> access and not the effect of having spatial index.
>> By adding DEBUG 5 for your layer you don't need stopwatch but you'll get
>> timings into ms_errorfile.
>>
>>
>> -Jukka Rahkonen-
>>
>> Varun saraf wrote:
>>
>>>
>>> Hello Everyone,
>>>
>>> I have programmed a GIS application using Mapserver, Google maps and
>>> Tilecache. The functionality of this GIS application is to extract the
>>> data (from the dbf file) for all features (Points) within a randomly
>>> drawn user shape and doing some statistical operation on that data. I
>>> use an NQUERY mode with MAPSHAPE attribute to get all the data for the
>>> user drawn shape. Mapserver takes aout 5-10 seconds for a small shape
>>> (a couple of square miles) but as the shape gets bigger (hundreds of
>>> square miles), the time taken to fetch all data related to the
>>> features/points lying in the shape grows exponentially (Upto 2 hours
>>> for some shapes). Until now, we were restricting the maximum area a
>>> shape can have but we have to get rid of that. Is there a way to
>>> improve the performance in any way? Will SHPTREE work for this
>>> purpose? The features are currently points only but we may move to
>>> polygons in future. We use the .shp files for the shapes. Is it
>>> advantageous to move to a database instead? If yes, what database
>>> works best?
>>>
>>> What I did notice is that for any given request to mapserver, however
>>> large the shape, the CPU utilization never crosses 12%. Can we improve
>>> performance by increasing the RAM or maybe move to a solid state hard
>>> drive? There is also the possibility of moving this application to
>>> Cloud computing. Anything that will improve the performance actually.
>>> Can someone point me in the right direction as to what might be the
>>> current bottleneck?
>>>
>>> T current setup is on windows and uses MS4W on an Apache server.
>>>
>>> Thanks,
>>> Varun
>>> _______________________________________________
>>> mapserver-users mailing list
>>> mapserver-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>>
>