[SAC] FOSS4G meet with OSUOSL update

[Martin Spott]

Hi, please excuse the late reply.  I was so enthusiastic over the
GRASS/PostGIS stuff I did last weekend so I completely forgot about the
admin side of the coin  ;-)

On Thu, Sep 11, 2014 at 11:10:01AM -0700, Alex Mandel wrote:

> 1. OSL can give Martin  (and others if we request) access to the host OS
> of osgeo3 and osgeo4 via their VPN. Which I think is the same way Martin
> setup osgeo5/backup.

Sounds good, so we can detect and inspect issues without depending on
OSL's support.
Setting up the backup machine was a little bit more complex. I was
given access to some non-public network and then, as far as memory
serves, connected to a console-to-IP adapter using some special browser
plugin.

> 3. Justin thinks there's enough space on other OSL hardware to shift
> osgeo4 vms for approximately 1 month so we can redo osgeo4. AKA switch
> to software raid 5, update the host OS, etc... We should schedule this
> in advance based on SAC availability to do required work.

I have to admit I have no idea about what exactly they did to get 'our'
Ganeti setup running.  I suspect they apply some sort of automated
install onto every machine in order to keep all affected systems
compatible to each other.  Right ?
Therefore I also suspect they have a preferred host OS for good reasons
and I don't think it's advisable to question their choice.  I also
didn't mean to question Ganeti in general, just having too limited
access was a bit unfortunate.

On Sat, Sep 13, 2014 at 10:37:55AM -0700, Alex Mandel wrote:

> 1. OSUOSL did point out that if we switch to software raid, it will be a
> little more difficult to swap disks. I assume Martin S. knows what needs
> to be done in order to tell the raid to drop a disk and be ready for the
> new one. Question, does software raid allow hot swapping disks?

Yes - but even though I've done this many times, I still always look at
the manual before removing a disk from the set  ;-)

> 2. I forgot to talk with OSUOSL about upgrading/changing the OS on
> osgeo4. Since we are redoing the disks I suspect this will need to
> happen. I will email them.

See above.

> Proposal Work Plan:
> 
> Step 1: Offload osgeo4
> 
> 1. Hotcopy mail from osgeo3 to osgeo4.

.... "from osgeo4 to osgeo3" in order to offload osgeoo4 !?

> 2. Get new VM or docker, or docker in a vm (Debian 7) on OSUOSL ganeti
> cluster. Invite projects to migrate their stuff from projects over to
> the new spot.

I think inviting projects to migrate their stuff is the most
unpredictable item in the entire plan  :-)

Practically speaking I'd prefer a plan which allows changing from HW
RAID to SW RAID (and maybe updating the host OS, if required)
independent from any other migration plans.  Thus, if OSL offers to run
our VM's on their cluster for a limited period, then I'd like to do
exactly this: Move our VM's off osgeo4, redo osgeo4 (using OSL's
favourite host OS and Ganeti) and move the VM's back to osgeo4.  If we
expect the projects to migrate their stuff in the same period, then we
might fail to meet the schedule.

Aside from that, I think the migration to Docker ("Docker" used as a
synonym for Linux containers) is a different taks.  One of the main
benefits of using containers is to make better use of the specific
topology of the hardware, much better than virtualization can do.
Therefore I'd refrain from using containers inside a VM, because the
net benefit isn't convincing and instead start using containers on new
hardware with no virtualization in place.

While we're at it: Are we talking about redoing osgeo4 only, not osgeo3
as well ?

> Step 4: Buy new hardware (sometime in the next year)

See above.

> Invite projects to a new sac-announce list which sends out notices about
> upcoming events.

I'd say we should request projects to subscribe to such list. As far as
I understand, communication to projects was mostly done over private
channels in the past and, as a consequence, they did expect the
continuation of private 'care' in case of upcoming changes.  This
didn't work out as planned, as we know  ;-)

> - 24 hour response team

Hah, as far as I understand, the primary reason why *I* was invited
into the admin team was the fact that I don't live in any of the US
time zones  :-)

Best regards,
	Martin.
-- 
 Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------