[mapserver-users] Layer with over 6000 shapefiles

[Rahkonen Jukka (Tike)]

Oh, I thought you had something interesting :)

If that little shapefile was slow you for sure had no qix index on that. One shapefile will be good for you but ogrtileindex is about as good alternative and if you have faulty data in some of the 6000 original shapefiles you can correct just those.  I believe that Spatialite db with spatial index and vacuumed would be as fast or faster.

-Jukka Rahkonen-


 EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:

> I was off on the size of the shape file.  It's 265MB, not GB.  Doh!  Anyway, the performance when processing that files was still surprisingly bad.  Quad core server and all 4 core's were maxed out for about 5 minutes after just loading that layer on a client.  I am running Server 2003, and only have 4GB ram on that server.  We are looking at replacing it.
James


-----Original Message-----
From: mapserver-users-bounces at lists.osgeo.org [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Even Rouault
Sent: Wednesday, January 29, 2014 11:09 AM
To: mapserver-users at lists.osgeo.org; Brent Wood
Subject: Re: [mapserver-users] Layer with over 6000 shapefiles

Le mercredi 29 janvier 2014 18:40:49, Brent Wood a écrit :
> This is less a mapserver issue than a data management issue - and
> shapefiles are not a particularly good way to manage data.
>
> I suggest you look at a spatially enabled database, with attribute &
> spatial indexes, such as Postgis, instead of shapefiles as a data source.
> These are pretty much designed for such use cases.
>
>
> That said, if you cannot use a spatial database in this case, you can
> do things with mapserver to improve things - like have multiple layers
> (though 6000 seems a bit excessive) in a single group - so addressable
> as a single layer but each with its own shapefile spatial index file.

A spatialite DB with spatial index could also perhaps do it. Not sure if it scales well up to 260 GB however.

>
> Brent Wood
>
>
>
> ________________________________
>  From: "EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE"
> <james.evans.4 at us.af.mil> To: "mapserver-users at lists.osgeo.org"
> <mapserver-users at lists.osgeo.org> Sent: Thursday, January 30, 2014
> 5:32 AM
> Subject: [mapserver-users] Layer with over 6000 shapefiles
>
>
> Hi,
> Is there a way to add a layer that points to 6000 some odd shapefiles?
> I've created a set of contour files for the whole world, and when I
> had the info in a single shapefile it was about 260GBs.  Mapserver
> would serve it up, but performance was horrible.  I now have the set
> broken up into about 6000 separate shapefiles, but don't know how to
> address that in a single layer.  I tried using gdaltindex, to create a
> single index file that points to all the other shapefiles, but I get
> an error saying the shapefiles are an unsupported type.  Is there some
> other way to create an index that Mapserver can use to point to this
> many shapefiles? Thanks, James
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

--
Geospatial professional services
http://even.rouault.free.fr/services.html
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users