[mapserver-users] Mapserver Storage

[Paul Spencer]

Hi,

I would personally recommend against AWS S3/EBS for anything of this scale as the I/O is pretty pathetic unless you invest in their very high end instances.  We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB EBS volumes and separate instances for each - the performance has been so poor that we have had to redesign our workflow to get copies of data onto EBS attached to each mapserver instance - for scaling that sucks and even then the I/O performance of EBS is not that great on the normal instances.

I'm not a hardware guy but I think the purpose of a dedicated SAN box is to provide high bandwidth access to large amounts storage so that the data can effectively be distributed to/from multiple machines over a network - ideal for scaling mapserver onto multiple servers but rendering from the same data.  I  read an article about a year ago from a company that provides petabyte storage for online storage, it details how they built their storage devices - they say $7867 for 67 terabytes

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Seems pretty geeky, but perhaps you are the hardware type or know someone who is :)


On 2011-01-28, at 3:41 AM, tigana.fluens at gmail.com wrote:

> Hello guys, we're a startup and new to mapserver. We're expecting large amounts of data to come by (at least on our scale) around 40-60TB of raster images for mapserver to render. My question is for the infrastructure, what is the best way to store this (cost-efficiently)?
> 
> - Do we just get a dedicated server with a lot of HDDs? I'm looking at a 48TB setup in RAID 1+0 so i get 24TB right what happens now if we need more? Also, how can we scale from the mapserver side?  Is access to different storage servers possible?
> - I've considered SANs but then it's not practical right because only one machine will access the storage?
> - What about Amazon's S3? or EBS? Anything we can use on that?
> 
> I wish to get awesome advice on this storage issue, basically what the considered best practice is for the mapserver people :P Thanks
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users


__________________________________________

   Paul Spencer
   Chief Technology Officer
   DM Solutions Group Inc
   http://research.dmsolutions.ca/