[mapserver-users] Mapserver Storage

[Bob Basques]

All, 

I'm working on a similar project currently.  Setting up 50tb of storage, we went the route of multiple CPUs, with large disks.   Redundant raid config, so half of physical disk available for storage.  We're in the 30+tb of real storage across a 4U setup right now.  Cost (with hardware/setup/initial config) is below those numbers below (so far), because we're building from scratch and learning along the way. 

I would tend to agree on not using the off site stuff, just considering the moving of the data and the idea of co-lo to some other remote location starts to fall apart.   The transfer costs, in bandwidth and/or time, really start to eat into things cost wise.  Some of this depends on the end uses as well.  We're building a data site for distribution of really large files and datasets. 

bobb 


>>> Paul Spencer <pspencer at dmsolutions.ca> wrote:


Hi,

I would personally recommend against AWS S3/EBS for anything of this scale as the I/O is pretty pathetic unless you invest in their very high end instances.  We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB EBS volumes and separate instances for each - the performance has been so poor that we have had to redesign our workflow to get copies of data onto EBS attached to each mapserver instance - for scaling that sucks and even then the I/O performance of EBS is not that great on the normal instances.

I'm not a hardware guy but I think the purpose of a dedicated SAN box is to provide high bandwidth access to large amounts storage so that the data can effectively be distributed to/from multiple machines over a network - ideal for scaling mapserver onto multiple servers but rendering from the same data.  I  read an article about a year ago from a company that provides petabyte storage for online storage, it details how they built their storage devices - they say $7867 for 67 terabytes

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Seems pretty geeky, but perhaps you are the hardware type or know someone who is :)


On 2011-01-28, at 3:41 AM, tigana.fluens at gmail.com wrote:

> Hello guys, we're a startup and new to mapserver. We're expecting large amounts of data to come by (at least on our scale) around 40-60TB of raster images for mapserver to render. My question is for the infrastructure, what is the best way to store this (cost-efficiently)?
>
> - Do we just get a dedicated server with a lot of HDDs? I'm looking at a 48TB setup in RAID 1+0 so i get 24TB right what happens now if we need more? Also, how can we scale from the mapserver side?  Is access to different storage servers possible?
> - I've considered SANs but then it's not practical right because only one machine will access the storage?
> - What about Amazon's S3? or EBS? Anything we can use on that?
>
> I wish to get awesome advice on this storage issue, basically what the considered best practice is for the mapserver people :P Thanks
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users


__________________________________________

   Paul Spencer
   Chief Technology Officer
   DM Solutions Group Inc
   http://research.dmsolutions.ca/

_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20110128/26df81e5/attachment.html