[SAC] OSGeo Ganeti Cluster

Thu Dec 28 11:47:26 PST 2017

I would have never suggested libvrt if I didn't know the compatibility.
Both are managers on top of KVM, with the disks of the vms being lvm
volumes. Moving a vm is a matter of copying the lvm volume and declaring
a config to use it as it's disk.

The feature we'd be losing is DRBD, which is a multi server hotcopy
failover system, which we don't really use because of performance issues
with load on some our machines. Ganeti also isn't really designed for
less than 3 servers, and kind of expects those servers to all be roughly
the same.

You are correct if we moved off ganeti we'd remove OSUOSL from anything
but the hardware management. This came up as SAC has never really had
good access to the hosts osgeo3 and osgeo4, and OSUOSL doesn't always
have time to troubleshoot some of our unusual issues.

Thanks,
Alex

On 12/28/2017 10:15 AM, Regina Obe wrote:
> Lance,
> 
>  
> 
> I'm afraid we are further behind on new server than I thought.   Seems more questions came out of the meeting than answers.
> 
> Any thoughts you have to add would be greatly appreciated.
> 
>  
> 
> One of the surprising outcomes for me was I thought sticking with Ganeti was done deal.  Seems it is not and libvrt is under consideration
> 
>  
> 
> You have any thoughts between Ganeti and Libvrt, what we would be losing if we switch to Libvrt.  Are the image formats even compatible?  I suspect they are not but haven't done the research.  I'm more concerned with OSUSL being able to support us if we decide to go with libvrt and rebuilding our currently in use VMs on libvrt.
> 
>  
> 
>  
> 
> Minutes from last meeting here: https://wiki.osgeo.org/wiki/SAC_Meeting_2017-12-21#Minutes
> 
> (transcript starts around 20:15 – 22:01ish  http://irclogs.geoapt.com/osgeo-sac/%23osgeo-sac.2017-12-21.log )
> 
>  
> 
> To summarize OSU specific outcomes
> 
>  
> 
> 1)      We still need to pick out specs on new server.  Alex is going to propose some options on the mailing list as I recall from here - https://www.siliconmechanics.com/ to fit in a $5000 ish budget.
> 
> 2)      We are debating with sticking with Ganeti or moving to something easier for us to manage like libvrt.  I'm concerned with having just one libvrt and it doesn't solve the problem we have of just having 1 Ganeti cluster we can trust so would just assume stick with Ganeti, but I'm less knowledgeable on the subject of the difference between the 2.  So I guess this means a hold-off for you on your plans unless you have any options we missed. :(
> 
> 3)      On the existing Ganeti clusters we have to inventory what is easy to move off and what we actually are still using cause on quick finding, I think a lot of things on those servers are not in use.  I think Martin was in middle of migrating stuff off because all those VMs are old Debian 5 or 6 and have to be rebuilt anyway, but I'm not confident we'll have enough bandwidth in next month or two to move everything off.
> 
>  
> 
> Thanks,
> 
> Regina
> 
>  
> 
> From: Sac [mailto:sac-bounces at lists.osgeo.org] On Behalf Of Lance Albertson
> Sent: Wednesday, December 27, 2017 7:25 PM
> To: tech at wildintellect.com
> Cc: System Administration Committee Discussion/OSGeo <sac at lists.osgeo.org>; systems at osuosl.org; sysadmin at osgeo.org
> Subject: Re: [SAC] OSGeo Ganeti Cluster
> 
>  
> 
> Any update from your last SAC meeting?
> 
>  
> 
> Thanks-
> 
>  
> 
> On Thu, Dec 14, 2017 at 4:41 PM, Lance Albertson <ramereth at osuosl.org <mailto:ramereth at osuosl.org> > wrote:
> 
>  
> 
>  
> 
> On Thu, Dec 14, 2017 at 2:46 PM, Alex M <tech_dev at wildintellect.com <mailto:tech_dev at wildintellect.com> > wrote:
> 
> I would say I don't trust osgeo4.
> 
> I believe it has a failed drive in it's raid, that we did not replace in
> anticipation of moving to new hardware. Also because it already burned
> through a couple of replacements, and the raid rebuild times were
> agonizing.
> 
>  
> 
> Right, I had forgotten that its in a failed drive state.
> 
>  
> 
> osgeo6, is already in, and is the replacement machine for osgeo4, we
> just haven't finished moving everything off. osgeo6 does not run ganeti
> or kvm at this time. We have debated if it should.
> 
> I'm not sure we are using drbd for an instances anymore. Would it be
> simpler to remove ganeti? Or is it possible to use other ganeti machines
> you have as the 2nd disks for the shuffle and upgrade?
> 
>  
> 
> You can't mixed Ganeti clusters unfortunately so we'd have to add a completely new node.
> 
>  
> 
> The new machine we are discussing is osgeo7 a replacement for osgeo3.
> 
>  
> 
> *nods*
> 
>  
> 
> Lance, what's the rack and PDU situation? If there is room we can order
> it sooner. Last I knew we needed to get osgeo4 off and out before we
> could add anything else. If there is room we can order sooner.
> 
>  
> 
> We have plenty of room now so feel free to get that started.
> 
>  
> 
> Alternate option, what would be the cost if we just want to buy in to
> existing Ganeti VM services OSUOSL is running? We aren't 100% sure the
> direction we are going with containers, virtualization, and cloud
> services. So an OSUOSL offer of "cloud" virtualization might be an option.
> 
>  
> 
> Our primary VM infrastructure is still based on Ganeti, however we've been exploring using OpenStack as an alternative for more elastic needs. We've been running an OpenStack cluster for the past several years on the ppc64le platform, but we haven't created a cluster for x86 yet. I was hoping we'd get something like that deployed sometime next year, but it depends on various factors.
> 
>  
> 
> What exactly are your needs in the medium and long term? We could put you on our primary Ganeti cluster but we have to be careful with any I/O intensive VMs so they don't impact other users.
> 
>  
> 
> Thanks-
> 
>  
> 