<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>All,<br>
</p>
<p>We had an outage of the mysql1-vip cluster from 20:18 to 21:04
UTC today (Apr 18th) related to the final backup and preparation
for this upgrade. Xrabackup issued a "FLUSH TABLES WITH READ LOCK"
for the final stage of the backup process. When combined with a
long running query, this exhausted the MySQL connection pool and
resulted in a read/write outage.</p>
<p>We will take steps to mitigate this for the upgrade on the 20th.
Please let us know if you have any questions or concerns.</p>
<p>Best-<br>
--<br>
Ethan Hampton<br>
Student Systems Engineer<br>
Oregon State University | Open Source Lab</p>
<div class="moz-cite-prefix">On 4/13/22 1:53 PM, Ethan Hampton
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1ed3f250-bdb0-4a85-211b-ff4720976851@osuosl.org">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div id="magicdomid354" class="ace-line"><span class="">All,</span></div>
<div id="magicdomid355" class="ace-line"><br>
</div>
<div id="magicdomid484" class="ace-line"><span class="">We will be
upgrading one of our MySQL clusters (mysql1-vip
(mysql1/mysql2)) next Wednesday, April 20th, 2022 between
17:00-20:00 UTC (10am-1pm PDT).</span></div>
<div id="magicdomid357" class="ace-line"><br>
</div>
<div id="magicdomid899" class="ace-line"><span class="">We are
upgrading MySQL (Percona) from 5.5 to 5.6. This is the first
step towards managing the cluster under Chef and </span><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0">upgrading
to MySQL</span><span
class="author-a-z86zcea72fvz81zz66zrjdbz87zz65z"> (Percona)</span><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0"> 5.7 at
a future date</span><span class="">. Expect more maintenance
on this cluster in the near future.</span></div>
<div id="magicdomid359" class="ace-line"><br>
</div>
<div id="magicdomid578" class="ace-line"><span class="">This will
result in up to 20 minutes of downtime in the middle of the
window as we fail-over the cluster. We estimate that the
actual downtime will be far less than that bound.</span></div>
<div id="magicdomid363" class="ace-line"><br>
</div>
<div id="magicdomid371" class="ace-line"><span class="">We have
tested this upgrade in a separate environment and believe that
it will have minimal impact to projects on the cluster. We
will make a full backup of the cluster before the upgrade work
begins.</span></div>
<div id="magicdomid579" class="ace-line"><br>
</div>
<div id="magicdomid895" class="ace-line"><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0
b"><b>If you are using the vip1 cluster, please make sure you
are using the mysql1-vip.osuosl.org address instead of
mysql1/2.osuosl.org directly or additional outages will
occur.</b></span></div>
<div id="magicdomid365" class="ace-line"><br>
</div>
<div id="magicdomid372" class="ace-line"><span class="">If you
have any questions or concerns, please let us know. We'll be
in #osuosl on Libera Chat in case any one needs to contact us
directly.</span></div>
<p>Thanks!<br>
<br>
--<br>
Ethan Hampton<br>
Student Systems Engineer<br>
Oregon State University | Open Source Lab<br>
</p>
</blockquote>
</body>
</html>