[mapserver-users] Layer with over 6000 shapefiles

[EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE]

Geez.  Just added the .qix to the original .shp file.  What a difference.
My server doesn't even bump up on processor usage.  
Thanks!



-----Original Message-----
From: mapserver-users-bounces at lists.osgeo.org
[mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Rahkonen Jukka
(Tike)
Sent: Wednesday, January 29, 2014 11:29 AM
To: mapserver-users at lists.osgeo.org
Subject: Re: [mapserver-users] Layer with over 6000 shapefiles

Oh, I thought you had something interesting :)

If that little shapefile was slow you for sure had no qix index on that. One
shapefile will be good for you but ogrtileindex is about as good alternative
and if you have faulty data in some of the 6000 original shapefiles you can
correct just those.  I believe that Spatialite db with spatial index and
vacuumed would be as fast or faster.

-Jukka Rahkonen-


 EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:

> I was off on the size of the shape file.  It's 265MB, not GB.  Doh!
Anyway, the performance when processing that files was still surprisingly
bad.  Quad core server and all 4 core's were maxed out for about 5 minutes
after just loading that layer on a client.  I am running Server 2003, and
only have 4GB ram on that server.  We are looking at replacing it.
James


-----Original Message-----
From: mapserver-users-bounces at lists.osgeo.org
[mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Even Rouault
Sent: Wednesday, January 29, 2014 11:09 AM
To: mapserver-users at lists.osgeo.org; Brent Wood
Subject: Re: [mapserver-users] Layer with over 6000 shapefiles

Le mercredi 29 janvier 2014 18:40:49, Brent Wood a écrit :
> This is less a mapserver issue than a data management issue - and 
> shapefiles are not a particularly good way to manage data.
>
> I suggest you look at a spatially enabled database, with attribute & 
> spatial indexes, such as Postgis, instead of shapefiles as a data source.
> These are pretty much designed for such use cases.
>
>
> That said, if you cannot use a spatial database in this case, you can 
> do things with mapserver to improve things - like have multiple layers 
> (though 6000 seems a bit excessive) in a single group - so addressable 
> as a single layer but each with its own shapefile spatial index file.

A spatialite DB with spatial index could also perhaps do it. Not sure if it
scales well up to 260 GB however.

>
> Brent Wood
>
>
>
> ________________________________
>  From: "EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE"
> <james.evans.4 at us.af.mil> To: "mapserver-users at lists.osgeo.org"
> <mapserver-users at lists.osgeo.org> Sent: Thursday, January 30, 2014
> 5:32 AM
> Subject: [mapserver-users] Layer with over 6000 shapefiles
>
>
> Hi,
> Is there a way to add a layer that points to 6000 some odd shapefiles?
> I've created a set of contour files for the whole world, and when I 
> had the info in a single shapefile it was about 260GBs.  Mapserver 
> would serve it up, but performance was horrible.  I now have the set 
> broken up into about 6000 separate shapefiles, but don't know how to 
> address that in a single layer.  I tried using gdaltindex, to create a 
> single index file that points to all the other shapefiles, but I get 
> an error saying the shapefiles are an unsupported type.  Is there some 
> other way to create an index that Mapserver can use to point to this 
> many shapefiles? Thanks, James
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

--
Geospatial professional services
http://even.rouault.free.fr/services.html
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6912 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20140129/7451b9b8/attachment.bin>