[mapserver-users] Mapserver Storage

Fri Jan 28 11:29:26 EST 2011

On 1/28/2011 11:14 AM, Mark Korver wrote:
> I think current S3 pricing for 50TB at 0.125/GB comes to about
> 6250/month.  That is for "Designed for 99.999999999% Durability".  I
> can't even count that many 9s.  I know you can buy HDs for about
> $50/TB = $2500 for that 50TB. Assuming you replace them all once a
> year its still only $5000.

Yeah, If MTBF for a disk is like 10,000 hours then you need to divide 
that by the number of disks. and you can expect to replace one disk a 
month is you have 14 disks. If it is 100,000 hours then you have one a 
month if you have 140 disks.

> But we all know that to even get to 99.99 there are lot more parts to
> the above equation.  Anybody have a ballpark on what it really costs
> to park that 50TB on your server room rack?

You need to have redundant systems, power supplies, drive controllers 
all with failover and and monitoring.

All that said, I have a lot of clients that like Dell's EqualLogic iSCSI 
SAN devices for this application. http://www.equallogic.com/

This is a good compromise for expandability, data security, performance, 
disk utilization, etc if you want to do this in-house. But if you are 
going to do this in-house I strongly recommend getting someone the knows 
this technology and and how to build systems to meet you needs.

-Steve W

> On Fri, Jan 28, 2011 at 9:29 AM, Bob Basques
> <Bob.Basques at ci.stpaul.mn.us>  wrote:
>>
>> All,
>>
>> I'm working on a similar project currently.  Setting up 50tb of storage, we went the route of multiple CPUs, with large disks.   Redundant raid config, so half of physical disk available for storage.  We're in the 30+tb of real storage across a 4U setup right now.  Cost (with hardware/setup/initial config) is below those numbers below (so far), because we're building from scratch and learning along the way.
>>
>> I would tend to agree on not using the off site stuff, just considering the moving of the data and the idea of co-lo to some other remote location starts to fall apart.   The transfer costs, in bandwidth and/or time, really start to eat into things cost wise.  Some of this depends on the end uses as well.  We're building a data site for distribution of really large files and datasets.
>>
>> bobb
>>
>>>>> Paul Spencer<pspencer at dmsolutions.ca>  wrote:
>>
>> Hi,
>>
>> I would personally recommend against AWS S3/EBS for anything of this scale as the I/O is pretty pathetic unless you invest in their very high end instances.  We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB EBS volumes and separate instances for each - the performance has been so poor that we have had to redesign our workflow to get copies of data onto EBS attached to each mapserver instance - for scaling that sucks and even then the I/O performance of EBS is not that great on the normal instances.
>>
>> I'm not a hardware guy but I think the purpose of a dedicated SAN box is to provide high bandwidth access to large amounts storage so that the data can effectively be distributed to/from multiple machines over a network - ideal for scaling mapserver onto multiple servers but rendering from the same data.  I  read an article about a year ago from a company that provides petabyte storage for online storage, it details how they built their storage devices - they say $7867 for 67 terabytes
>>
>> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
>>
>> Seems pretty geeky, but perhaps you are the hardware type or know someone who is :)
>>
>>
>> On 2011-01-28, at 3:41 AM, tigana.fluens at gmail.com wrote:
>>
>>> Hello guys, we're a startup and new to mapserver. We're expecting large amounts of data to come by (at least on our scale) around 40-60TB of raster images for mapserver to render. My question is for the infrastructure, what is the best way to store this (cost-efficiently)?
>>>
>>> - Do we just get a dedicated server with a lot of HDDs? I'm looking at a 48TB setup in RAID 1+0 so i get 24TB right what happens now if we need more? Also, how can we scale from the mapserver side?  Is access to different storage servers possible?
>>> - I've considered SANs but then it's not practical right because only one machine will access the storage?
>>> - What about Amazon's S3? or EBS? Anything we can use on that?
>>>
>>> I wish to get awesome advice on this storage issue, basically what the considered best practice is for the mapserver people :P Thanks
>>> _______________________________________________
>>> mapserver-users mailing list
>>> mapserver-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>> __________________________________________
>>
>>     Paul Spencer
>>     Chief Technology Officer
>>     DM Solutions Group Inc
>>     http://research.dmsolutions.ca/
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users