<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I got word that this outage was more campus wide which included impacting the OpenCompute hosts. I went through those hosts and ensured they are back online but let me know if I missed anything.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">OSU will be sending in a tech in a few days to see why the UPS didn't fail over properly in our primary datacenter which caused the power event. I'm also going to spot check a few hosts' power when I go in on Tuesday to ensure power is split properly between the power feeds. If you had any hosts that went down with dual power, please let me know ASAP so I can add it to the list of hosts to check.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks for your patience!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 1, 2021 at 8:15 AM Lance Albertson <<a href="mailto:lance@osuosl.org">lance@osuosl.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">It seems as though we had an unplanned power event that happened in our primary data center early this morning at 3:03AM PDT (1003 UTC) that affected one of the two power feeds. Virtually every system that has a dual power supply should have remained online. The one exception is some systems located in a row that are only being fed by that power feed which include:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- power8-aix</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- <a href="http://pieta.debian.org" target="_blank">pieta.debian.org</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc2-power8</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- All Buildbot/RTEMS systems</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc113</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc114</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc115</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc116</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc117</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">- gcc118</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I believe every system that we monitor should be back online but there might be others we aren't monitoring that are still down. If that's the case, please send an email to support and we'll take a look at it as soon as possible.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I'm still waiting to hear back about what happened and why it happened and will pass that information along once I learn more.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks for your patience.</div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Lance Albertson</font><div><div><font face="arial, helvetica, sans-serif">Director</font></div><div><span style="font-family:arial,helvetica,sans-serif">Oregon State University | </span><span style="font-family:arial,helvetica,sans-serif">Open Source Lab </span></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><font face="arial, helvetica, sans-serif">Lance Albertson</font><div><div><font face="arial, helvetica, sans-serif">Director</font></div><div><span style="font-family:arial,helvetica,sans-serif">Oregon State University | </span><span style="font-family:arial,helvetica,sans-serif">Open Source Lab </span></div></div></div></div>