[mapserver-users] Amazon EC2 vs. Collocation/Dedicated servers

[Paul Spencer]

Hi Adam,

we use mapserver on Amazon EC2 instances with some success.  The instances themselves seem decently spec'd for mapserver usage, although we have found that the medium high-cpu instances do seem to be more cost effective than normal instances.

The biggest detractor with the cloud is storage.  Basically you get a limited amount of ephemeral storage (it disappears when the machine is terminated) with each instance.  You can attach EBS volumes of up to 1 TB.  The I/O on local and EBS storage is not great but its okay.  The problems start when you want to use multiple machines sharing some form of storage.  For us, we have several TB of tiles cached and we want to be able to serve up tiles and dynamic map draws.  The tile cache is also not complete so it needs to generate tiles on demand.  Finally, we have a lot of data that is used in the maps, more than will fit on the ephemeral storage of a medium instance.  We used to use several 1TB EBS volumes mounted on a single instance, shared via NFS to all the map servers but we ran into periodic locking problems that brought the whole thing down.  We recently switched to use glusterfs which is an excellent open source product and allows us to essentially have a SAN where all our data can appear seamlessly to all our machines as a single volume even though it spans several EBS volumes.  The problem with this is that map draws accessing data via gluster are several times slower than using locally stored data.  Longer individual map draws were not acceptable so now we are working on co-locating the data with the map servers and using the shared storage for tiles and a master copy of the data.

One of the advantages of EC2 is that you can bring up extra computing power on demand (either manually or programmatically).  Unfortunately, having to copy the data locally to the machines negates this advantage as it takes several hours to copy and extract the data - not exactly real-time response to changing load on the system :(

Of course, its also hard to be responsive if you have colocated hardware.  Overall, despite the problems with I/O bandwidth and managing our large datasets I think EC2 has done well for us and we could not have provided the same from dedicated hardware within the same budget.

Cheers

Paul



On 2010-11-03, at 9:23 AM, Adam Eskreis wrote:

> Hey All,
> 
> I just wanted to get an idea of what the general consessus is about what the best platform for running mapserver in a live environment for many users (in the tens of thousands).  My company is trying to decide the best way to spend money on servers is for our mapserver application.  We have just 2 layers in our map, but one of them is heavily data intensive and is too dynamic to be cached (to put it simply, it is a dynamic layer that uses a different png symbols for each feature), the other is a cached baselayer.  The biggest concern is, what type of environment will allow us to quickly serve up not only cached tiles, but dynamic tiles as well, while still being able to handle tens of thousands of users.  The two choices we are deciding between are Amazon EC2, or purchased servers in a collocation facility.  I have put together a list of pros and cons of each, but I still wanted to get the opinion of the mapserver pros out there.  So, if anyone has any thoughts, please write back.
> 
> Thanks,
> -Adam
> 
> P.S. We are also looking for a developer who knows the mapserver code like the back of their hand who may be able to modify mapserver to be optimized for our project.  Any mapserver developer that is interested in a part-time contracted position, please reply to aeskreis at gmail.com with your resume as well as your hourly rate.  
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users


__________________________________________

   Paul Spencer
   Chief Technology Officer
   DM Solutions Group Inc
   http://research.dmsolutions.ca/