[mapserver-users] Mapserver Storage
Paul Spencer
pspencer at dmsolutions.ca
Fri Jan 28 06:08:02 PST 2011
Hi,
I would personally recommend against AWS S3/EBS for anything of this scale as the I/O is pretty pathetic unless you invest in their very high end instances. We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB EBS volumes and separate instances for each - the performance has been so poor that we have had to redesign our workflow to get copies of data onto EBS attached to each mapserver instance - for scaling that sucks and even then the I/O performance of EBS is not that great on the normal instances.
I'm not a hardware guy but I think the purpose of a dedicated SAN box is to provide high bandwidth access to large amounts storage so that the data can effectively be distributed to/from multiple machines over a network - ideal for scaling mapserver onto multiple servers but rendering from the same data. I read an article about a year ago from a company that provides petabyte storage for online storage, it details how they built their storage devices - they say $7867 for 67 terabytes
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
Seems pretty geeky, but perhaps you are the hardware type or know someone who is :)
On 2011-01-28, at 3:41 AM, tigana.fluens at gmail.com wrote:
> Hello guys, we're a startup and new to mapserver. We're expecting large amounts of data to come by (at least on our scale) around 40-60TB of raster images for mapserver to render. My question is for the infrastructure, what is the best way to store this (cost-efficiently)?
>
> - Do we just get a dedicated server with a lot of HDDs? I'm looking at a 48TB setup in RAID 1+0 so i get 24TB right what happens now if we need more? Also, how can we scale from the mapserver side? Is access to different storage servers possible?
> - I've considered SANs but then it's not practical right because only one machine will access the storage?
> - What about Amazon's S3? or EBS? Anything we can use on that?
>
> I wish to get awesome advice on this storage issue, basically what the considered best practice is for the mapserver people :P Thanks
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
__________________________________________
Paul Spencer
Chief Technology Officer
DM Solutions Group Inc
http://research.dmsolutions.ca/
More information about the MapServer-users
mailing list