shapefile optimization for dynamic data

[Chris Tweedie]

Ben, for the situation you described i would of thought pushing the GPS 
points into a database would make much more sense than trying to 
add/remake the shapefile especially at that frequency.

Have you looked into using postgis (or any other db for that matter) as 
your datasource?

Chris

Quoting Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>:

> Using shptree will not help you that much in this scenario because of 
> the frequency of updating the file. You best bet would be use 
> multiple files and a tile index that you would have to add the new 
> files to as they are created. Then you can shptree on the non-active 
> file, but not on the active file. That will probably be the best 
> scenario. Also make sure you shptree the tileindex.
>
> If a shapefile does not have a qix spatial index, then mapserver 
> creates one on the fly and throws it away. If you are adding a point 
> a second the file is probably getting updated faster than you can 
> index it and then render it. Using the tileindex should really help 
> in this case also, because only the files the intersect you current 
> display window need to be opened and looked at.
>
> -Steve W.
>
> Ben Eisenbraun wrote:
>> Hi-
>>
>> I'm building a Mapserver application with dynamic data coming in 
>> every second, and I'm not sure how to organize the shapefiles for 
>> speediest access.
>>
>> I'm collecting data via a GPS and a sensor that reports a data point 
>> once per second.  I'm using Mapserver CGI to generate an overlay 
>> onto a map via a javascript frontend that auto-refreshes every few 
>> seconds.  The application has to run on a low-power embedded 
>> hardware device (roughly a p2-266), and I'm running into performance 
>> problems once I've collected a few thousand data points.  The 
>> Mapserver CGI process tends to consume all the CPU trying to render 
>> the overlays.
>>
>> Up to now, I've been using shpadd/dbfadd to add the data points to a 
>> single shapefile.  I've tried using shptree to index the shapefile, 
>> but under my
>> testing, it doesn't seem to speed up rendering time at all.  The 
>> largest a shapefile should ever get to be is about 25,000 data 
>> points; I'm not sure if that's large or small compared to other 
>> people's data.
>>
>> I've thought about breaking up that single shapefile into multiple 
>> shapefiles of a given size, but I'm not sure if that would be a win 
>> for this type of situation.
>>
>> Given that I'm constantly updating the source shapefile, what are my options
>> for optimizing it?
>>
>> Thanks for any tips or pointers.
>>
>> -ben
>>
>> --
>> simplicity is the most difficult thing to secure in this world; it 
>> is the last limit of experience and the last effort of genius.     
>> <george sand>
>>
>