[SAC] [Hosting] Network outage 12/17/2016 08:26AM PST - 12/17/2016 11:14AM PST

[Lance Albertson]

Just got another update from the data center manager.

The power should be stable and we don't expect another outage to happen
today or the rest of the weekend. When the power went out, everything
worked as expected but for some reason the generator was out of phase and
the UPS went to battery and then ran out of power. They still aren't sure
why that happened but they are confident is it stable for now.

They are planning on fixing the wiring they changed today tomorrow morning
between 10am-11am PST. They are planning on doing it without an outage, but
there is a risk they could lose power again. The UPS will be on battery
while they make the change and have about six minutes to make the change.
They did it with three minutes today without an issue.

I am still working on the full incident report and will send it later today
or sometime tomorrow.

On Sat, Dec 17, 2016 at 11:47 AM, Lance Albertson <ramereth at osuosl.org>
wrote:

> I just got word that the power restoration is not stable and it may go
> down again today. I will send an update once the power has been stabilized.
>
> On Sat, Dec 17, 2016 at 11:27 AM, Lance Albertson <ramereth at osuosl.org>
> wrote:
>
>> All,
>>
>> We had a major network outage today caused by a UPS fault in the data
>> center which houses our upstream connection with NERO. All of our systems
>> remained online during the outage, however our upstream connection to the
>> Internet was down due to the NERO switches being powered off.
>>
>> Everything seems to be back online currently and I'm waiting to hear a
>> more detailed report back on the root cause.
>>
>> I will be working on sending a more detailed report later today once more
>> information is available.
>>
>> Thank you for your patience.
>>
>> --
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab
>>
>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>



-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/sac/attachments/20161217/edd0e1cd/attachment.html>
-------------- next part --------------
_______________________________________________
Hosting mailing list
Hosting at osuosl.org
http://lists.osuosl.org/mailman/listinfo/hosting