Hardware for serving lot's of Raster

[Ed McNierney]

Shoaib -

Much of your answer depends on the usage of your data, not the size of
it.  If you get one map request a day, and that user's in no hurry, then
you can do whatever you like!

Some of your answer depends on the details of that data usage as it
dictates the organization and structure of your imagery.  For example,
if you need to quickly served "zoomed out" views at lower resolution but
at high quality, you may need to prebuild "overview" images that will
require more disk space.  That's usually not much more, since the
overviews get a lot smaller very quickly.

If you're focusing on hardware, I think there are very few important
variables.  If your data does not change often, and you want the best
possible read performance, a RAID 5 array will do that - writes will be
slow, but that's OK.  Given your description of the data, I don't think
there's much reason to consider anything else.

Remember that in a RAID 5 array you potentially have many disks working
simultaneously for you, so a larger number of smaller disks will
generally be faster than an smaller number of larger disks.  And you'll
get more for your money with more disks per array - an 8-drive RAID 5
array retains 87.5% of its raw capacity, while a 4-drive RAID 5 array
only keeps 75% of its raw capacity.

You need to choose between IDE and SCSI disk hardware.  SCSI disks will
be faster, mainly due to higher rotational speeds - it's easy to find
15,000 RPM SCSI drives while IDE drives are usually 7,200 or 5,400 RPM
(there are 10K RPM IDE drives but not many).  That translates into
shorter seek times and quicker reads.  If you are mostly reading small,
random bits of data (common in a MapServer application) then the seek
time will dominate and SCSI will really outperform IDE.

The downside is that SCSI disks are MUCH more expensive and have MUCH
smaller capacity.  That means you'll also pay for more array enclosures,
controllers, etc.

As a sample data point, I see I can buy one Western Digital 320 GB 7,200
RPM IDE drive for $149.59.  By the way, that is a price per megabyte
that is just over 219,000 times cheaper than the very first hard disk I
purchased for myself, for my new IBM PC.  At the opposite extreme I can
get a Seagate 146 GB 15,000 RPM SCSI drive for $957.01.  It will be
considerably faster in real-world performance, but the price per unit
storage is 14 times higher!  If cost is really no object, buy 15K SCSI
drives in RAID 5 arrays and you'll have a great time.

If you'd like more information, please try to describe your application
and usage model in as much detail as possible, since that's going to
have a huge effect on the design.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Shoaib Burq
Sent: Sunday, November 20, 2005 6:10 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Hardware for serving lot's of Raster

Hi,

I am in the process of spec'ing out some hardware for serving ~1TB of
hi-res bathymetry data via mapserver. Having never done this before I am
looking for some guidance regarding what to look for in the hardware
design. I might be using ka-map and precache the tiles; Updates are not
very frequent.

Obvious concerns are:
1. Performance,
2. Security,
3. Backup,
4. Portability (if we ever need to change the server's location), 5.
Cost (we have a relatively healthy budget)

I had a read of the documentation from UCL's ICEDS project but besides
stating the use of RAID but is not very specific.

any suggestions?

chrs
Shoaib Burq