Best way to structure large datasets

[Stephen Woodbridge]

Walter Anderson wrote:
> I am most interested in performance.  I am interested
> in minimizing the end users wait times.

Then you should put the data in shapefiles with tile indexes. This is 
the fastest that we have today. I have All the US data in 3300 Counties, 
and nearly 33,000 shapefiles in about 10 data layers and use tileindexes 
for it.

I also have some polygon data in PostGIS that I use specifically for 
doing Thematic maps.

> I am interested in using Postgresql for the datasets
> to allow for the geoprocessing capabilities that it
> offers through SQL.  This geoprocessing will be
> separate from the MAPSERVER portions of the
> applications.

Load the data that you need from shapefiles into PostGIS. There is a 
simple tools that lets you copy shapefiles to and from PostGIS.

-Steve

> --- Ed McNierney <ed at topozone.com> wrote:
> 
> 
>>Walter -
>>
>>It depends on how you define "best".  Are you
>>looking for the best
>>performance, the smallest storage space, or the
>>easiest to update and
>>maintain?  The answer is different for all three.
>>
>>When in doubt, preprocess - unless your goal is ease
>>of maintenance!
>>
>>If you could fill us in a bit on what your goals
>>are, it would be easier
>>to give more specific advice.  Thanks!
>>
>>	- Ed
>>
>>Ed McNierney
>>President and Chief Mapmaker
>>TopoZone.com / Maps a la carte, Inc.
>>73 Princeton Street, Suite 305
>>North Chelmsford, MA  01863
>>Phone: +1 (978) 251-4242
>>Fax: +1 (978) 251-1396
>>ed at topozone.com 
>>
>>-----Original Message-----
>>From: UMN MapServer Users List
>>[mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
>>Behalf Of Walter Anderson
>>Sent: Monday, January 09, 2006 9:07 PM
>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>Subject: [UMN_MAPSERVER-USERS] Best way to structure
>>large datasets
>>
>>I have a general question on the best way to
>>structure large datasets.
>>I am compiling an internal web application that will
>>server GIS data to
>>a small workgroup (10-20) but the data set covers
>>the entire state of
>>Texas.  I have a couple of questions for the experts
>>out there.
>>
>>  1. Specifically the road centerline data is about
>>1GB in size (1:24k
>>scale centerlines).  The centerline files are
>>currently stored as one
>>file per county (254 counties).  
>>I want to use Postgresql to store all of my data but
>>am unsure of the
>>whether I can use the tiling index method with
>>postgresql or even if it
>>is needed.
>>  2. Also I would like to filter the roads system so
>>that at the
>>statewide level only the Interstates and US Highways
>>are displayed, then
>>as the user zooms in progressively more information
>>is displayed.
>>Should / can this be accomplished with a single
>>(tiled) vector file or
>>should I preprocess the data to generate subsets
>>suitable for each scale
>>level.
>>
>>Thanks for any input,
>>
>>
>>Walter Anderson
>>Walter Anderson 
>>
>>
>>		
>>__________________________________________
>>Yahoo! DSL - Something to write home about. 
>>Just $16.99/mo. or less. 
>>dsl.yahoo.com 
>>
> 
> 
> 
> 
> 		
> __________________________________________ 
> Yahoo! DSL – Something to write home about. 
> Just $16.99/mo. or less. 
> dsl.yahoo.com 
>