[SAC] [support.osuosl.org #23649] [OSGeo] Failed disk in osgeo4.osuosl.bak

[Lance Albertson via RT]

On Thu, Apr 3, 2014 at 12:04 AM, tech at wildintellect.com via RT <
support at osuosl.org> wrote:

> Something seems amiss. The ProjectsVM stopped responding, high disk
> latency and iowait ( 10-11pm PST)
>
> http://webextra.osgeo.osuosl.org/munin/osgeo.org/projects.osgeo.org/index.html
>
> We couldn't ssh so I tried to restart via Ganeti Web interface. No luck.
> Can't even tell if it came back up.
>
> Other VMs on osgeo4 seem ok but also have sizeable iowait.
>

I've completed a forced restart of the VM so it appears to be back however
postgres/mysql both seemed to have failed on boot.

We're still rebuilding unfortunately:

Rebuild Progress on Device at Enclosure 32, Slot 3 Completed 82% in 200
Minutes.

I've never seen a rebuild take this long before but this hardware is
starting to show its age a little.

I think what happened to this VM in question is related to an
unpredictable bug we've seen on other VMs where they freeze up randomly. It
seems to be related to high i/o but we could never replicate it. Issuing
the reboot command I believe did work although it had to force a shutdown
it seems. I'm not sure why it didn't come back online however. Next time
you might try an "immediate" shutdown and then start it back up.

Anyways, hopefully the rebuild completes sometime later today and i/o gets
back to normal again.

Thanks-

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab