From lance at osuosl.org  Tue Jul  7 09:47:36 2026
From: lance at osuosl.org (Lance Albertson)
Date: Tue, 7 Jul 2026 09:47:36 -0700
Subject: [Hosting] Power event at SDC 7/7/2026
Message-ID: <CAORfjXTqES1Agb33r372x+zj2v=ZDtGKNdU5s7+Qqfgi33JRfw@mail.gmail.com>

Hi all,

We recently experienced a power event at the State Data Center where our
machines are hosted. I am currently waiting for an official update on the
cause.

One of our links to LinkOregon went down but has since reconnected, and
everything appears to be coming back online. I am currently checking the
single PSU hosts to ensure there are no lingering issues. I will also
investigate why part of the network went offline, as our infrastructure was
designed to handle these situations.

I will send an update as soon as I have more information.

Thanks-

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/sac/attachments/20260707/5220bd0e/attachment.htm>
-------------- next part --------------
_______________________________________________
Hosting mailing list
Hosting at lists.osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting

From lance at osuosl.org  Tue Jul  7 11:52:27 2026
From: lance at osuosl.org (Lance Albertson)
Date: Tue, 7 Jul 2026 11:52:27 -0700
Subject: [Hosting] Power event at SDC 7/7/2026
In-Reply-To: <CAORfjXTqES1Agb33r372x+zj2v=ZDtGKNdU5s7+Qqfgi33JRfw@mail.gmail.com>
References: <CAORfjXTqES1Agb33r372x+zj2v=ZDtGKNdU5s7+Qqfgi33JRfw@mail.gmail.com>
Message-ID: <CAORfjXQ=aPxqay=aiCcr7bgno7RK-SyTbmh0WnA3UNw-0scj-w@mail.gmail.com>

Hi all,

Following up on this morning's power event, I now have an explanation from
Oregon DAS, who manages the State Data Center.

The DAS Facilities team was performing routine building battery maintenance
that involved switching power from the A side to the B side. This work was
not run through their change management process ahead of time, so we did
not receive advance notice, which is why the alerts came as a surprise. DAS
has acknowledged this and will run this type of work through change
management with a heads-up notice going forward.

We have two uplinks to LinkOregon via an MLAG on their end. During one of
the switchovers, one of their two routers went down while the other stayed
up. As traffic failed over between the LinkOregon routers and ours, we saw
a brief network blip. The failover worked as intended and the downed router
has since come back online. We are following up with LinkOregon on why that
router dropped during a single-side switchover, since it should have stayed
powered.

Impact was minimal. Our infrastructure is designed to ride through A/B
switchovers, and I have verified the single-PSU hosts for any lingering
issues. Everything else looked fine and we are back to normal.

DAS had additional battery work planned through Friday, but they have *called
it off *for the rest of the week. The remaining work will be rescheduled
through proper change management, so we should get advance notice next time
and can pass that along to you.

I will follow up if anything further comes out of the LinkOregon
investigation.

Thanks-

On Tue, Jul 7, 2026 at 9:47?AM Lance Albertson <lance at osuosl.org> wrote:

> Hi all,
>
> We recently experienced a power event at the State Data Center where our
> machines are hosted. I am currently waiting for an official update on the
> cause.
>
> One of our links to LinkOregon went down but has since reconnected, and
> everything appears to be coming back online. I am currently checking the
> single PSU hosts to ensure there are no lingering issues. I will also
> investigate why part of the network went offline, as our infrastructure was
> designed to handle these situations.
>
> I will send an update as soon as I have more information.
>
> Thanks-
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/sac/attachments/20260707/47e94a07/attachment.htm>
-------------- next part --------------
_______________________________________________
Hosting mailing list
Hosting at lists.osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting