[mapserver-users] Layer with over 6000 shapefiles

[Stephen Woodbridge]

On 1/29/2014 1:29 PM, Rahkonen Jukka (Tike) wrote:
> Oh, I thought you had something interesting :)
>
> If that little shapefile was slow you for sure had no qix index on that. One shapefile will be good for you but ogrtileindex is about as good alternative and if you have faulty data in some of the 6000 original shapefiles you can correct just those.  I believe that Spatialite db with spatial index and vacuumed would be as fast or faster.
>
> -Jukka Rahkonen-

As Jukka and others have commented it sounds like you might be missing 
your spatial index *.qix file when you test your all in one shapefile. 
256 MB shapefile is not that big unless you are trying to load the whole 
thing to display the contours for the whole world. If the later is the 
case then you probably need to rethink loading all 256 MB of data 
because it my guess is that it is not all useful at that resolution.

If you are filtering only some countours using and expression, then you 
would be better off separating them into another shapefile and adding a 
another layer.

-Steve

>   EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:
>
>> I was off on the size of the shape file.  It's 265MB, not GB.  Doh!  Anyway, the performance when processing that files was still surprisingly bad.  Quad core server and all 4 core's were maxed out for about 5 minutes after just loading that layer on a client.  I am running Server 2003, and only have 4GB ram on that server.  We are looking at replacing it.
> James
>
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Even Rouault
> Sent: Wednesday, January 29, 2014 11:09 AM
> To: mapserver-users at lists.osgeo.org; Brent Wood
> Subject: Re: [mapserver-users] Layer with over 6000 shapefiles
>
> Le mercredi 29 janvier 2014 18:40:49, Brent Wood a écrit :
>> This is less a mapserver issue than a data management issue - and
>> shapefiles are not a particularly good way to manage data.
>>
>> I suggest you look at a spatially enabled database, with attribute &
>> spatial indexes, such as Postgis, instead of shapefiles as a data source.
>> These are pretty much designed for such use cases.
>>
>>
>> That said, if you cannot use a spatial database in this case, you can
>> do things with mapserver to improve things - like have multiple layers
>> (though 6000 seems a bit excessive) in a single group - so addressable
>> as a single layer but each with its own shapefile spatial index file.
>
> A spatialite DB with spatial index could also perhaps do it. Not sure if it scales well up to 260 GB however.
>
>>
>> Brent Wood
>>
>>
>>
>> ________________________________
>>   From: "EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE"
>> <james.evans.4 at us.af.mil> To: "mapserver-users at lists.osgeo.org"
>> <mapserver-users at lists.osgeo.org> Sent: Thursday, January 30, 2014
>> 5:32 AM
>> Subject: [mapserver-users] Layer with over 6000 shapefiles
>>
>>
>> Hi,
>> Is there a way to add a layer that points to 6000 some odd shapefiles?
>> I've created a set of contour files for the whole world, and when I
>> had the info in a single shapefile it was about 260GBs.  Mapserver
>> would serve it up, but performance was horrible.  I now have the set
>> broken up into about 6000 separate shapefiles, but don't know how to
>> address that in a single layer.  I tried using gdaltindex, to create a
>> single index file that points to all the other shapefiles, but I get
>> an error saying the shapefiles are an unsupported type.  Is there some
>> other way to create an index that Mapserver can use to point to this
>> many shapefiles? Thanks, James
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>