Cluster Maintenance in the Wee Hours

We sometimes have to take the cluster down for scheduled maintenance and upgrades. We usually do this at 11:05pm or 12:05am on the Saturday/Sunday changeover. We did this this weekend.

We did a number of upgrades and improvements, including:

• Replaced two older SAN units with new ones. This will increase disk I/O performance for a number of customers.

• Updated firmware in all SAN units.

• Re-sync’d a database that had gotten out of sync with its replica on Friday evening.

• Increased temp and heap table space, which should reduce disk I/O, and increased innodb_buffer_cache as well, potentially easing disk I/O. The combination should result in better database performance.

• Upgraded early cluster servers to same kernel and slightly newer Xen to be the same as the newest servers in the cluster.

Getting ready at the data center 2 hours before the maintenance window.

Working on one of the servers.

Swapping SAN units.

Upgrading software and firmware.

Ready to restart the cluster.