[SAC] Issues with osgeo3

[Lance Albertson]

Hi All,

Thanks again for your patience. Everything appears to be online and also
confirmed via Jeff McKenna on IRC. I wanted to fill you in on what happened
and how I'll prevent it from happening again.

We've been working on upgrading our Ganeti servers to be managed via Chef
on CentOS 7. A part of that process is upgrading Ganeti itself to a version
that works on CentOS 7 and then eventually scheduling a time to rebuild the
nodes with CentOS.

In order to upgrade Ganeti, I needed to upgrade some of the packages on
these systems (which currently run Gentoo). During that process I ran into
an issue with incompatible versions of ncurses and eventually incompatible
versions of glibc. In an attempt to resolve the issue, I tried to manually
update glibc which immediately caused all the processes to fail on the
machine. To fix the problem, I had to boot the machine up on a LiveCD and
manually copy the original versions of the libraries involved back onto the
system.

I had tried this same upgrade on another node without any issues related to
this but apparently I didn't do the steps in correct order this time.
Moving forward I'm going to make sure I document the process better and
make local backup copies of the current system packages installed. Also, I
won't try to manually upgrade glibc on the system as that was silly of me
to do :(

I'll likely be working on this later today again but will be more careful
on what I'm doing. It shouldn't have caused an issue like it did today and
I don't plan on it doing that further.

If you have any further questions please let me know.

Thanks-

On Thu, Sep 21, 2017 at 9:41 AM, Lance Albertson <lance at osuosl.org> wrote:

> I was doing to do some simple upgrades on osgeo3 this morning and
> unfortunately I caused it to go offline which effected all vms running on
> that node. We're working on fixing it right now.
>
> Apologize for the outage.
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>



-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/sac/attachments/20170921/4b52079d/attachment-0001.html>