AW: [UMN_MAPSERVER-USERS] slighty OT: new hardware for a mapserver application

Tue Aug 9 05:55:59 PDT 2005

Frank,

Not that you asked me specifically, but ...

FWIW, our approach is to store vector data in different tables, although each table may have been created from many shape files (streets being created out some 3200 UWS counties, etc.), they tend to be all of the same sort (point, line or multipolygon). Some, e.g. states in the US, are small in number of rows, while others (US streets, US land parcel data, etc.) are fairly large with tens of millions of rows.

The postGIS queries using a bounding box are reasonably efficient, and areas that are commonly seen are cached in RAM which helps speed things up considerably. postGres tools help us decide what indexes are useful and do some tuning.

We do have a tracking system in place to be able to say which shapefile / other source was the origin of the data, which helps for maintenance and reproducing data sets. We aso have some meta-data tables that let us quickly determine which data is available is which areas.

HTH,

Greg Williamson
DBA
GlobeXplorer LLC

"When the only tool you have is a hammer, all your problems look like nails!"

-----Original Message-----
From:	UMN MapServer Users List on behalf of Frank Broniewski
Sent:	Tue 8/9/2005 5:40 AM
To:	MAPSERVER-USERS at LISTS.UMN.EDU
Cc:	
Subject:	[UMN_MAPSERVER-USERS] AW: [UMN_MAPSERVER-USERS] slighty OT: new hardware for a mapserver application
If I may interfere, I have also an question about putting vector data
into postgis.
If I may qoute you:

> ...though i would suggest to 
> merge the shapes and MySQl alpha data into one PostgreSQL/PostGIS 
> feature table. ...

Are you putting all your e.g. point shapes into one table and seperate
them by an attribute or do you create one table for one shapefile? Is
there an advantage of having all shapefiles of one geometry in one table
instead of having one table per shapefile?

I'm curious because here at the museum  we want to put our data also
into a postgres/postgis db and are unsure about the most clever way to
accomplish this.

Your Idea seemed somewhat revolutionary to me, so I wondered if you
could describe a little more in detail your way of storing shapefiles in
the database.

Greetings

Frank Broniewski
Musée National d'Histoire et d'Art
Section Préhistoire / Projet EPC
    Tél: +352 260 281-21
241, Rue de Luxembourg
L-8077 Bertrange

-----Ursprüngliche Nachricht-----
Von: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] Im
Auftrag von Arnulf Christl
Gesendet: Dienstag, 9. August 2005 13:09
An: MAPSERVER-USERS at LISTS.UMN.EDU
Betreff: Re: [UMN_MAPSERVER-USERS] slighty OT: new hardware for a
mapserver application

Dylan Beaudette wrote:
> Thanks for the tips Ed,
> 
> There should be about 100Gb of imagery in geotiff format, 5 to 10 Gb 
> of vector data stored in shapefiles, and about another 5 to 10 Gb of 
> tabular data stored in MySQL.

The vectorial data will not cause any problems though i would suggest to

merge the shapes and MySQl alpha data into one PostgreSQL/PostGIS 
feature table. We are truly happy with PG doing all the work and 
rendering sections out of 10 million and more polygons within
milliseconds. There are quite different opinions though regarding
performance of shape 
files versus databases.

> I am not expecting a huge number of hits per day, but right now it is 
> hard to say... perhaps 10 to 100 requests per day?

Is that individual requests or number of users? Larger installations are

known to render several hundred thousand maps a day so that your 
suggested numbers should not even get you start to think.

> We are considering a dual processor xeon system with SCSI disks, as 
> this closely resembles our test machine.

As suggested, keep it cheap, step up later. Dual CPU is cool, loads of 
RAM is cool when using PG, fast disks are good for large raster data 
(see Ed, he is the expert on rasters). If you need more database vector 
performance put it on a separate machine.

Best, Arnulf.

> Thanks again,
> 
> Dylan
> 
> On Monday 08 August 2005 10:11 am, Ed McNierney wrote:
> 
>>Dylan -
>>
>>I always recommend buying cheap stuff and making sure you ask it 
>>simple questions <g>!
>>
>>If you are running lots of imagery, a fast disk subsystem is a good 
>>idea.  But it's helpful to define "large quantity" more precisely, 
>>because one person's "large" is another's "minuscule".  RAM is cheap 
>>and always a good idea, as it can be used for disk caching and 
>>swapping is a big performance hit.
>>
>>If you can be more specific about (a) the size of your data sets, (b) 
>>the volume of requests and image sizes supported, and (c) the formats 
>>of your data (TIFF? PostGIS? Shapefiles?) perhaps we can be more 
>>specific about recommendations.
>>
>>	- Ed
>>
>>Ed McNierney
>>President and Chief Mapmaker
>>TopoZone.com / Maps a la carte, Inc.
>>73 Princeton Street, Suite 305
>>North Chelmsford, MA  01863
>>ed at topozone.com
>>(978) 251-4242
>>
>>-----Original Message-----
>>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] 
>>On Behalf Of Dylan Beaudette
>>Sent: Monday, August 08, 2005 12:34 PM
>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>Subject: [UMN_MAPSERVER-USERS] slighty OT: new hardware for a 
>>mapserver application
>>
>>Greetings,
>>
>>We are currently in the process of ramping up the scale of a 
>>mapserver-based application, and thus in search of some new hardware.
>>
>>So far we have priced out some dual processor systems, with the best 
>>deal from Dell. The machine will be running the UNIX version of 
>>mapserver. The large quantity of imagery and high density vector files

>>will require a powerful machine in order to keep latency low.
>>
>>Any suggestions or stories about previous purchases (related to a 
>>mapserver
>>application) would be much appreciated.
>>
>>Thanks in advance!
>>
>>--
>>Dylan Beaudette
>>Soils and Biogeochemistry Graduate Group University of California at 
>>Davis 530.754.7341

!DSPAM:42f8a47d263891639413654!