<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>All,</p>
<p>Due to Xtrabackup requiring a read lock for part of the backup
process, there will be an additional outage at the beginning of
the maintenance window so we can complete a final backup before
the cluster upgrade. We will do our best to minimize the outage
time.<br>
</p>
<p>Please let us know if you have any questions or concerns.</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 4/18/22 2:41 PM, Ethan Hampton
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b8f6bba4-c4f7-b9de-266b-79010ebbd62b@osuosl.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>All,<br>
</p>
<p>We had an outage of the mysql1-vip cluster from 20:18 to 21:04
UTC today (Apr 18th) related to the final backup and preparation
for this upgrade. Xrabackup issued a "FLUSH TABLES WITH READ
LOCK" for the final stage of the backup process. When combined
with a long running query, this exhausted the MySQL connection
pool and resulted in a read/write outage.</p>
<p>We will take steps to mitigate this for the upgrade on the
20th. Please let us know if you have any questions or concerns.</p>
<p>Best-<br>
</p>
<div class="moz-cite-prefix">On 4/13/22 1:53 PM, Ethan Hampton
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1ed3f250-bdb0-4a85-211b-ff4720976851@osuosl.org">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div id="magicdomid354" class="ace-line"><span class="">All,</span></div>
<div id="magicdomid355" class="ace-line"><br>
</div>
<div id="magicdomid484" class="ace-line"><span class="">We will
be upgrading one of our MySQL clusters (mysql1-vip
(mysql1/mysql2)) next Wednesday, April 20th, 2022 between
17:00-20:00 UTC (10am-1pm PDT).</span></div>
<div id="magicdomid357" class="ace-line"><br>
</div>
<div id="magicdomid899" class="ace-line"><span class="">We are
upgrading MySQL (Percona) from 5.5 to 5.6. This is the first
step towards managing the cluster under Chef and </span><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0">upgrading
to MySQL</span><span
class="author-a-z86zcea72fvz81zz66zrjdbz87zz65z"> (Percona)</span><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0"> 5.7 at
a future date</span><span class="">. Expect more maintenance
on this cluster in the near future.</span></div>
<div id="magicdomid359" class="ace-line"><br>
</div>
<div id="magicdomid578" class="ace-line"><span class="">This
will result in up to 20 minutes of downtime in the middle of
the window as we fail-over the cluster. We estimate that the
actual downtime will be far less than that bound.</span></div>
<div id="magicdomid363" class="ace-line"><br>
</div>
<div id="magicdomid371" class="ace-line"><span class="">We have
tested this upgrade in a separate environment and believe
that it will have minimal impact to projects on the cluster.
We will make a full backup of the cluster before the upgrade
work begins.</span></div>
<div id="magicdomid579" class="ace-line"><br>
</div>
<div id="magicdomid895" class="ace-line"><span
class="author-a-0z83zz82zz70zqz65zz70zz86zz79zcz66zuz122zz82ze0
b"><b>If you are using the vip1 cluster, please make sure
you are using the mysql1-vip.osuosl.org address instead of
mysql1/2.osuosl.org directly or additional outages will
occur.</b></span></div>
<div id="magicdomid365" class="ace-line"><br>
</div>
<div id="magicdomid372" class="ace-line"><span class="">If you
have any questions or concerns, please let us know. We'll be
in #osuosl on Libera Chat in case any one needs to contact
us directly.</span></div>
<p>Thanks!<br>
</p>
</blockquote>
</blockquote>
<pre class="moz-signature" cols="72">--
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab</pre>
</body>
</html>