[SAC] [Hosting] mysql1-vip (mysql1/mysql2) 5.5 -> 5.6 Upgrade | April 20th, 2022 17-20 UTC

[Ethan Hampton]

All,

We had an outage of the mysql1-vip cluster from 20:18 to 21:04 UTC today 
(Apr 18th) related to the final backup and preparation for this upgrade. 
Xrabackup issued a "FLUSH TABLES WITH READ LOCK" for the final stage of 
the backup process. When combined with a long running query, this 
exhausted the MySQL connection pool and resulted in a read/write outage.

We will take steps to mitigate this for the upgrade on the 20th. Please 
let us know if you have any questions or concerns.

Best-
--
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab

On 4/13/22 1:53 PM, Ethan Hampton wrote:
> All,
>
> We will be upgrading one of our MySQL clusters (mysql1-vip 
> (mysql1/mysql2)) next Wednesday, April 20th, 2022 between 17:00-20:00 
> UTC (10am-1pm PDT).
>
> We are upgrading MySQL (Percona) from 5.5 to 5.6. This is the first 
> step towards managing the cluster under Chef and upgrading to 
> MySQL(Percona)5.7 at a future date. Expect more maintenance on this 
> cluster in the near future.
>
> This will result in up to 20 minutes of downtime in the middle of the 
> window as we fail-over the cluster. We estimate that the actual 
> downtime will be far less than that bound.
>
> We have tested this upgrade in a separate environment and believe that 
> it will have minimal impact to projects on the cluster. We will make a 
> full backup of the cluster before the upgrade work begins.
>
> *If you are using the vip1 cluster, please make sure you are using the 
> mysql1-vip.osuosl.org address instead of mysql1/2.osuosl.org directly 
> or additional outages will occur.*
>
> If you have any questions or concerns, please let us know. We'll be in 
> #osuosl on Libera Chat in case any one needs to contact us directly.
>
> Thanks!
>
> --
> Ethan Hampton
> Student Systems Engineer
> Oregon State University | Open Source Lab
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/sac/attachments/20220418/f6dd53a2/attachment.html>
-------------- next part --------------
_______________________________________________
Hosting mailing list
Hosting at osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting