[mapserver-users] Layer with over 6000 shapefiles

[EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE]

I was off on the size of the shape file.  It's 265MB, not GB.  Doh!  Anyway, the performance when processing that files was still surprisingly bad.  Quad core server and all 4 core's were maxed out for about 5 minutes after just loading that layer on a client.  I am running Server 2003, and only have 4GB ram on that server.  We are looking at replacing it.
James


-----Original Message-----
From: mapserver-users-bounces at lists.osgeo.org [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Even Rouault
Sent: Wednesday, January 29, 2014 11:09 AM
To: mapserver-users at lists.osgeo.org; Brent Wood
Subject: Re: [mapserver-users] Layer with over 6000 shapefiles

Le mercredi 29 janvier 2014 18:40:49, Brent Wood a écrit :
> This is less a mapserver issue than a data management issue - and 
> shapefiles are not a particularly good way to manage data.
> 
> I suggest you look at a spatially enabled database, with attribute & 
> spatial indexes, such as Postgis, instead of shapefiles as a data source.
> These are pretty much designed for such use cases.
> 
> 
> That said, if you cannot use a spatial database in this case, you can 
> do things with mapserver to improve things - like have multiple layers 
> (though 6000 seems a bit excessive) in a single group - so addressable 
> as a single layer but each with its own shapefile spatial index file.

A spatialite DB with spatial index could also perhaps do it. Not sure if it scales well up to 260 GB however.

> 
> Brent Wood
> 
> 
> 
> ________________________________
>  From: "EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE"
> <james.evans.4 at us.af.mil> To: "mapserver-users at lists.osgeo.org"
> <mapserver-users at lists.osgeo.org> Sent: Thursday, January 30, 2014 
> 5:32 AM
> Subject: [mapserver-users] Layer with over 6000 shapefiles
> 
> 
> Hi,
> Is there a way to add a layer that points to 6000 some odd shapefiles? 
> I've created a set of contour files for the whole world, and when I 
> had the info in a single shapefile it was about 260GBs.  Mapserver 
> would serve it up, but performance was horrible.  I now have the set 
> broken up into about 6000 separate shapefiles, but don't know how to 
> address that in a single layer.  I tried using gdaltindex, to create a 
> single index file that points to all the other shapefiles, but I get 
> an error saying the shapefiles are an unsupported type.  Is there some 
> other way to create an index that Mapserver can use to point to this 
> many shapefiles? Thanks, James
> 
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

--
Geospatial professional services
http://even.rouault.free.fr/services.html
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6912 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20140129/3e93127f/attachment-0001.bin>