Hardware for serving lot's of Raster

[Ken Lord]

Hi Shoaib, Ed,

If I may interject into your great hard drive summary ...

The new SATA II hard drives should also be considered, they can also
be placed in RAID arrays.  They are roughly the same price as IDE but
perform much better, especially in RAID.  I'm sure the high end SCSI
drives are still better, but SATA II gives a lot of bang for the buck.

SATA is related to SCSI, my SATA II drive is listed by Windows XP as a
SCSI drive in my Device manager.

For a direct comparison,  my 7200rpm Samsung SATA II NCQ 8.9ms 80gb
drive transfers 2.5 times more data in a given time than my 7200rpm
Western Digital 8.9ms 80gb IDE drive. Both drives are worth around
$75.

The new NCQ feature (native command queing) allows the harddrive to
process requests in the most efficient order to reduce seek times
...in older hard drives request are handled in the order they are
received.  NCQ can give a 25% boost in performance by minimizing seek
times this way.

SATA II can handle a bandwidth of 3Gb/second, compared to 133Mb/s
bandwidth for IDE, and 150 - 300Mb/s for the older SATA drives ...
3Gb/s is far more bandwidth than what the hardware itself can fill.
Which is where RAID comes in.  IDE RAID is still limited to the
133Mb/s IDE bandwidth limit,  the more drives you add, the more you
will hit the bandwidth limit. SATA II RAID has a lot more bandwidth
room to work with.

There is a lot of confusion with the SATA II specification.  NCQ, and
hot-swapping for example are possible, but not required by the spec,
so you have to be sure of what you are buying. You have to be sure
that the rest of your system can run SATA II drives.

Unfortuneatly I can't compare any of this directly to SCSI, I don't
know the numbers on them, hardware is a hobby for me, I'm not an
expert ... but given the prices, I do hope that they are that much
better.

Cheers,
Ken Lord
Vancouver BC


On 11/20/05, Ed McNierney <ed at topozone.com> wrote:
> Shoaib -
>
> Much of your answer depends on the usage of your data, not the size of
> it.  If you get one map request a day, and that user's in no hurry, then
> you can do whatever you like!
>
> Some of your answer depends on the details of that data usage as it
> dictates the organization and structure of your imagery.  For example,
> if you need to quickly served "zoomed out" views at lower resolution but
> at high quality, you may need to prebuild "overview" images that will
> require more disk space.  That's usually not much more, since the
> overviews get a lot smaller very quickly.
>
> If you're focusing on hardware, I think there are very few important
> variables.  If your data does not change often, and you want the best
> possible read performance, a RAID 5 array will do that - writes will be
> slow, but that's OK.  Given your description of the data, I don't think
> there's much reason to consider anything else.
>
> Remember that in a RAID 5 array you potentially have many disks working
> simultaneously for you, so a larger number of smaller disks will
> generally be faster than an smaller number of larger disks.  And you'll
> get more for your money with more disks per array - an 8-drive RAID 5
> array retains 87.5% of its raw capacity, while a 4-drive RAID 5 array
> only keeps 75% of its raw capacity.
>
> You need to choose between IDE and SCSI disk hardware.  SCSI disks will
> be faster, mainly due to higher rotational speeds - it's easy to find
> 15,000 RPM SCSI drives while IDE drives are usually 7,200 or 5,400 RPM
> (there are 10K RPM IDE drives but not many).  That translates into
> shorter seek times and quicker reads.  If you are mostly reading small,
> random bits of data (common in a MapServer application) then the seek
> time will dominate and SCSI will really outperform IDE.
>
> The downside is that SCSI disks are MUCH more expensive and have MUCH
> smaller capacity.  That means you'll also pay for more array enclosures,
> controllers, etc.
>
> As a sample data point, I see I can buy one Western Digital 320 GB 7,200
> RPM IDE drive for $149.59.  By the way, that is a price per megabyte
> that is just over 219,000 times cheaper than the very first hard disk I
> purchased for myself, for my new IBM PC.  At the opposite extreme I can
> get a Seagate 146 GB 15,000 RPM SCSI drive for $957.01.  It will be
> considerably faster in real-world performance, but the price per unit
> storage is 14 times higher!  If cost is really no object, buy 15K SCSI
> drives in RAID 5 arrays and you'll have a great time.
>
> If you'd like more information, please try to describe your application
> and usage model in as much detail as possible, since that's going to
> have a huge effect on the design.
>
>        - Ed
>
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> Phone: +1 (978) 251-4242
> Fax: +1 (978) 251-1396
> ed at topozone.com
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
> Behalf Of Shoaib Burq
> Sent: Sunday, November 20, 2005 6:10 PM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: [UMN_MAPSERVER-USERS] Hardware for serving lot's of Raster
>
> Hi,
>
> I am in the process of spec'ing out some hardware for serving ~1TB of
> hi-res bathymetry data via mapserver. Having never done this before I am
> looking for some guidance regarding what to look for in the hardware
> design. I might be using ka-map and precache the tiles; Updates are not
> very frequent.
>
> Obvious concerns are:
> 1. Performance,
> 2. Security,
> 3. Backup,
> 4. Portability (if we ever need to change the server's location), 5.
> Cost (we have a relatively healthy budget)
>
> I had a read of the documentation from UCL's ICEDS project but besides
> stating the use of RAID but is not very specific.
>
> any suggestions?
>
> chrs
> Shoaib Burq
>