[SAC] OSGeo Ganeti Cluster

[Alex M]

Sandro,

Are you asking about the TracSVN machine or in general? It's a simple
configuration change and reboot in ganeti to change the CPU allocation,
assuming there are more CPUs to allocate. In this case I would say
hitting CPU steal indicates that services need to be moved to different
machines to lower the overall load. I was hoping now that the Drupal
site is not the main website the overall load on the machine would
reduce, however the Wiki VM appears to be a suspect with it's noticeable
swap which creates disk iowait which leads to CPU load. Though if you
look at the CPU load chart on TracSVN, it's got plenty of CPU available.
So I think increasing the ram on Wiki and modifying the kernel
swappiness might help. Let me check the configuration notes of osgeo3
and we can put in a request to increase the ram allocation of the Wiki VM.

Thanks,
Alex

On 01/19/2018 10:55 AM, Sandro Santilli wrote:
> How hard would it be to also add more CPU slices to an existing VM ?
> I'm thinking about TracSVN, which according to Munin has "steal"
> events ...
> 
> --strk;
> 
> On Thu, Jan 18, 2018 at 12:56:55PM -0800, Lance Albertson wrote:
>> Mind submitting a new ticket for that via support at osuosl.org? Also please
>> include what hostname you want to give it.
>>
>> Thanks!
>>
>> On Thu, Jan 18, 2018 at 12:53 PM, Alex M <tech_dev at wildintellect.com> wrote:
>>
>>> Lance,
>>>
>>> We're currently shopping for the new machine, and figuring out how we
>>> want to migrate off the older hardware.
>>>
>>> In the meantime could we request a VM on your ganeti cluster to explore
>>> if we want to try hosting more things on your systems instead of our own?
>>> We'd like a Debian 9, with 2 cpu, 4 GB Ram, 100 GB hard drive.
>>>
>>> We can supply ssh keys for access to root to configure the VM once up.
>>>
>>> Thanks,
>>> Alex
>>> OSGeo Sys Admin
>>>
>>>
>>> On 12/14/2017 04:41 PM, Lance Albertson wrote:
>>>> On Thu, Dec 14, 2017 at 2:46 PM, Alex M <tech_dev at wildintellect.com>
>>> wrote:
>>>>>
>>>>> I would say I don't trust osgeo4.
>>>>>
>>>>> I believe it has a failed drive in it's raid, that we did not replace in
>>>>> anticipation of moving to new hardware. Also because it already burned
>>>>> through a couple of replacements, and the raid rebuild times were
>>>>> agonizing.
>>>>>
>>>>
>>>> ​Right, I had forgotten that its in a failed drive state.​
>>>>
>>>>
>>>>> osgeo6, is already in, and is the replacement machine for osgeo4, we
>>>>> just haven't finished moving everything off. osgeo6 does not run ganeti
>>>>> or kvm at this time. We have debated if it should.
>>>>>
>>>>> I'm not sure we are using drbd for an instances anymore. Would it be
>>>>> simpler to remove ganeti? Or is it possible to use other ganeti machines
>>>>> you have as the 2nd disks for the shuffle and upgrade?
>>>>>
>>>>
>>>> ​You can't mixed Ganeti clusters ​unfortunately so we'd have to add a
>>>> completely new node.
>>>>
>>>>
>>>>> The new machine we are discussing is osgeo7 a replacement for osgeo3.
>>>>>
>>>>
>>>> ​*nods*​
>>>>
>>>>
>>>>> Lance, what's the rack and PDU situation? If there is room we can order
>>>>> it sooner. Last I knew we needed to get osgeo4 off and out before we
>>>>> could add anything else. If there is room we can order sooner.
>>>>>
>>>>
>>>> ​We have plenty of room now​ so feel free to get that started.
>>>>
>>>> Alternate option, what would be the cost if we just want to buy in to
>>>>> existing Ganeti VM services OSUOSL is running? We aren't 100% sure the
>>>>> direction we are going with containers, virtualization, and cloud
>>>>> services. So an OSUOSL offer of "cloud" virtualization might be an
>>> option.
>>>>>
>>>>
>>>> ​​Our primary VM infrastructure is still based on Ganeti, however we've
>>>> been exploring using OpenStack as an alternative for more elastic needs.
>>>> We've been running an OpenStack cluster for the past several years on the
>>>> ppc64le platform, but we haven't created a cluster for x86 yet. I was
>>>> hoping we'd get something like that deployed sometime next year, but it
>>>> depends on various factors.
>>>>
>>>> What exactly are your needs in the medium and long term? We could put you
>>>> on our primary Ganeti cluster but we have to be careful with any I/O
>>>> intensive VMs so they don't impact other users.
>>>>
>>>> Thanks-
>>>>
>>>
>>>
>>
>>
>> -- 
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab