[mapserver-users] Mapserver Storage

Fri Jan 28 09:29:51 PST 2011

now we have some real numbers.

reputable provider
10TB/$1800

Amazon S3
10TB/$1250

Amazon EBS
10TB/$1000

its not that simple a comparison.  but looking at those numbers and
knowing that with AWS you can run everything else as a function of
need (EC2,LB,SQS,mail services... etc etc) AWS starts looking cheap.
It might be a bit more to park data ( not really ) but that is only a
small part of the final equation.

On Fri, Jan 28, 2011 at 10:30 AM,  <tigana.fluens at gmail.com> wrote:
> @Mark Yes, so big of a difference. HDDs dont even fail often : ) What can
> you say about our current plan:
>
> Assuming we need 10TB of storage,
> We get a dedicated server somewhere on a reputable provider like SoftLayer
> the max HDDs they can put in a single box is 24 so assuming we get 2TB HDDs
> that's 24TB in RAID 1+0. I just spoke to their sales team and with the
> config Xeon 5xxx something, 12GB DDR3 of memory among other things such as
> the raid card and whatnot, it would cost around $1800/month.
>
> Is this a viable solution? I would like to just invest in a local server but
> unfortunately our internet sucks here in our country so we'll just keep
> local backups and rely on the RAID in the said machine for the required
> "redundancy" if you can call it that.
>
> On 1/29/2011 12:14 AM, Mark Korver wrote:
>>
>> I think current S3 pricing for 50TB at 0.125/GB comes to about
>> 6250/month.  That is for "Designed for 99.999999999% Durability".  I
>> can't even count that many 9s.  I know you can buy HDs for about
>> $50/TB = $2500 for that 50TB. Assuming you replace them all once a
>> year its still only $5000.
>>
>> But we all know that to even get to 99.99 there are lot more parts to
>> the above equation.  Anybody have a ballpark on what it really costs
>> to park that 50TB on your server room rack?
>>
>> On Fri, Jan 28, 2011 at 9:29 AM, Bob Basques
>> <Bob.Basques at ci.stpaul.mn.us>  wrote:
>>>
>>> All,
>>>
>>> I'm working on a similar project currently.  Setting up 50tb of storage,
>>> we went the route of multiple CPUs, with large disks.   Redundant raid
>>> config, so half of physical disk available for storage.  We're in the 30+tb
>>> of real storage across a 4U setup right now.  Cost (with
>>> hardware/setup/initial config) is below those numbers below (so far),
>>> because we're building from scratch and learning along the way.
>>>
>>> I would tend to agree on not using the off site stuff, just considering
>>> the moving of the data and the idea of co-lo to some other remote location
>>> starts to fall apart.   The transfer costs, in bandwidth and/or time, really
>>> start to eat into things cost wise.  Some of this depends on the end uses as
>>> well.  We're building a data site for distribution of really large files and
>>> datasets.
>>>
>>> bobb
>>>
>>>>>> Paul Spencer<pspencer at dmsolutions.ca>  wrote:
>>>
>>> Hi,
>>>
>>> I would personally recommend against AWS S3/EBS for anything of this
>>> scale as the I/O is pretty pathetic unless you invest in their very high end
>>> instances.  We've set up a 4TB 'SAN' using glusterfs on AWS EC2 using 1TB
>>> EBS volumes and separate instances for each - the performance has been so
>>> poor that we have had to redesign our workflow to get copies of data onto
>>> EBS attached to each mapserver instance - for scaling that sucks and even
>>> then the I/O performance of EBS is not that great on the normal instances.
>>>
>>> I'm not a hardware guy but I think the purpose of a dedicated SAN box is
>>> to provide high bandwidth access to large amounts storage so that the data
>>> can effectively be distributed to/from multiple machines over a network -
>>> ideal for scaling mapserver onto multiple servers but rendering from the
>>> same data.  I  read an article about a year ago from a company that provides
>>> petabyte storage for online storage, it details how they built their storage
>>> devices - they say $7867 for 67 terabytes
>>>
>>>
>>> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
>>>
>>> Seems pretty geeky, but perhaps you are the hardware type or know someone
>>> who is :)
>>>
>>>
>>> On 2011-01-28, at 3:41 AM, tigana.fluens at gmail.com wrote:
>>>
>>>> Hello guys, we're a startup and new to mapserver. We're expecting large
>>>> amounts of data to come by (at least on our scale) around 40-60TB of raster
>>>> images for mapserver to render. My question is for the infrastructure, what
>>>> is the best way to store this (cost-efficiently)?
>>>>
>>>> - Do we just get a dedicated server with a lot of HDDs? I'm looking at a
>>>> 48TB setup in RAID 1+0 so i get 24TB right what happens now if we need more?
>>>> Also, how can we scale from the mapserver side?  Is access to different
>>>> storage servers possible?
>>>> - I've considered SANs but then it's not practical right because only
>>>> one machine will access the storage?
>>>> - What about Amazon's S3? or EBS? Anything we can use on that?
>>>>
>>>> I wish to get awesome advice on this storage issue, basically what the
>>>> considered best practice is for the mapserver people :P Thanks
>>>> _______________________________________________
>>>> mapserver-users mailing list
>>>> mapserver-users at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>>> __________________________________________
>>>
>>>    Paul Spencer
>>>    Chief Technology Officer
>>>    DM Solutions Group Inc
>>>    http://research.dmsolutions.ca/
>>>
>>> _______________________________________________
>>> mapserver-users mailing list
>>> mapserver-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>>> _______________________________________________
>>> mapserver-users mailing list
>>> mapserver-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>
>