This has been copied from the news section of our website, but that requires a login and we thought this was worth sharing with the broader community: An example of how automation can break things quite quickly.
Description of system in normal operating conditions
At Whatbox, we handle the administrative side of our servers so that you can focus on using your hosted applications. This includes hardware upgrades and repairs, adding new features, and keeping applications secure and up to date. Our servers run Gentoo Linux, and we use a configuration management tool called puppet to maintain our servers. When puppet is run on our servers, it generates a list of software packages that we use, and instructs the system to install and/or update these packages.
Our Gentoo systems install software packages by compiling them from source code, a process that is time consuming but typically occurs as a background operation. Once this operation has completed, puppet further instructs the system to remove any software packages that were not included in the previously generated list. These unlisted software packages typically are either temporarily installed for troubleshooting purposes, or were dependencies of other packages that are no longer required.
Incident summary
On October 1st, 2021, at 18:34 UTC a configuration update was sent to all of our servers. This update contained faulty instructions that caused puppet to generate a list of software packages containing only a single entry. As a result, all other software packages were scheduled to be removed.
At 18:57 UTC, servers began uninstalling all software packages that were not essential to booting the system.
This resulted in a variety of error messages across all applications. SSH utilities such as vim, mtr, and crontab were no longer available. Hosted applications such as ruTorrent returned an Access Denied error. The Whatbox Manage page showed services as "Restarting" or in other unusual states, and other information on the page was unavailable. If your system shell was set to zsh, you were unable to login via SSH as your shell no longer existed.
It is important to note that there was no risk of customer data loss. While system applications were uninstalled, customer data was untouched by puppet.
Incident / Root Cause Analysis
By 19:05 UTC, several Whatbox engineers began responding to the incident after seeing expected commands stop working.
$ mtr 1.1.1.1
bash: /usr/sbin/mtr: No such file or directory
$ sudo
bash: /bin/sudo: No such file or directory
At 19:08 UTC, the decision was made to terminate the currently running update, in order to halt the removal of additional software packages.
At 19:10 UTC, we determined a preliminary cause of the problem -- the list of software packages to maintain had been shortened to a single entry.
At 19:11 UTC, we identified the root cause of the problem. The puppet module that we use to manage software packages had been updated to a newer version. The newer version of this module introduced a software bug that caused the resulting list to be shortened.
Incident / Resolution Timeline
At 19:11 UTC, we reverted the puppet module to a prior known good version and attempted to push this change to the first server. The attempt to push this change failed, because puppet was no longer installed on the servers.
At 19:19 UTC, we began reinstalling puppet, git, and sudo in order to assist in repairing the servers. This had completed by 19:43.
At 19:59 UTC, we were continuing to evaluate next steps to repair the servers.
A complicating factor is the use of Python 2.7. This older version of Python was discontinued in 2020, and as a result, Gentoo's package manager no longer supports installing packages with Python 2.7 support. However, we still have certain software packages that require Python 2.7 support, such as libtorrent-rasterbar. When these packages were uninstalled, it became a challenge to reinstall them while maintaining Python 2.7 support as the package manager no longer provided this functionality.
At 20:03 UTC, we considered an alternative idea to more quickly recover, but initial testing showed signs of greater risk.
At 20:30 UTC, we began reinstalling more software packages that did not depend on Python 2.7.
At 20:55 UTC, we considered an early upgrade to Deluge 2.0, which is already planned for December 10th, 2021, as this would allow us to upgrade libtorrent-rasterbar and resolve the Python 2.7 troubles. We decided against this as we could not ensure adequate testing.
At 21:10 UTC, we attempted to push an update to a single server to resolve some software dependency issues.
At 21:19 UTC, an update was pushed to servers to temporarily resolve connection issues to Deluge and its WebUI.
At 21:45 UTC, we continued looking for ways to allow the package manager to compile the necessary packages with Python 2.7 support.
At 21:56 UTC, we continued resolving errors and reinstalling more software packages that did not depend on Python 2.7.
At 22:16 UTC, we saw positive results in a method to allow us to override the package manager.
At 22:32 UTC, initial testing was performed to compile libtorrent-rasterbar with the necessary Python 2.7 support.
At 22:49 UTC, the system's FTP service was reinstalled across all servers.
At 23:04 UTC, the system's nginx and ruTorrent services were repaired across all servers.
At 23:40 UTC, we continued to experience trouble with the package manager installing packages with Python 2.7 support.
At 00:18 UTC, we attempted rolling out binary packages for libtorrent and deluge with Python 2.7 support.
At 00:24 UTC, we confirmed the binary packages were functioning as expected.
At 00:33 UTC, we began pushing a complete system update to initial servers.
At 00:37 UTC, the complete system update was pushed to all servers.
At 00:40 UTC, another update was pushed to all servers to resolve the remaining Python 2.7 troubles.
At 00:52 UTC, special attention was given to one server that did not correctly apply the updates.
At 01:55 UTC, another update was pushed to resolve ongoing issues with the FTP service.
At 02:33 UTC, we continued monitoring servers as software installation continued to progress.
At 03:37 UTC, another update was pushed to increase the speed in which software packages are installed.
At 03:39 UTC, half of the servers have completed all software package reinstallations. All services on these servers are functioning normally.
At 04:04 UTC, a missing package is reinstalled to resolve communications issues between the Whatbox site and servers.
At 05:11 UTC, all but one server have completed their software package reinstallations.
At 05:35 UTC, we reviewed our processes for sending credits to all customers, while waiting for the final server to finish updating.
At 06:43 UTC, all servers have completed their software package reinstallations.
At 06:56 UTC, we begin issuing a 2 day service credit to all customers, for the 12 hours of downtime that occurred.
At 09:49 UTC, we completed issuing service credits to all customers.
Setbacks Encountered
We sincerely apologize for the inconvenience that was caused by this service outage.
The process of compiling software from source code is more secure than relying on binary packages provided by third parties, as it allows us to enable certain security features that third parties might not enable themselves. To help ensure reliability, we run unit tests on the software programs and libraries we install, which makes sure the software is functioning in the way its developers expect it to. When these unit tests fail, the package manager blocks the software program from being installed. Together, these actions help us to run a more secure, more reliable service, but they extend the time it takes to install new applications. When routine updates occur, this extended time is not a problem as services will continue running. However, in this case, services had been stopped as they were no longer installed, and it took many hours for everything to reinstall as there were 400+ packages to compile.
Upon further review, we found that unit tests were failing on the puppet module that was found to be the root cause of this event. However, the unit tests for this module are not performed by the package manager and therefore it could not block this update. Instead, these unit tests were performed using an external service, and a recent change of ownership in the external service led to us no longer receiving notifications of failed unit tests. With no failure notifications received, the assumption was mistakenly made that all tests had passed. Manual QA checks were also performed, but nothing unexpected had been observed. Had we observed the failed unit tests, we could have prevented this event from occurring.
The requirement for Python 2.7 support caused additional hours to be added to the overall outage duration. Our operating system's package manager gives us lots of flexibility, but it has been removing support for compiling packages with Python 2.7, a version of Python that has been discontinued and no longer receives security updates. Python 2.7 is still required to support one of our hosted applications, so we have been maintaining this support in the package manager ourselves. However, completely uninstalling and reinstalling these packages introduced additional challenges that resulted in the extended outage time. In about two months we will be upgrading Deluge to version 2.0, which will remove the dependency that we currently have with Python 2.7, and would have shortened the duration of this outage.
Corrective Actions
A two day service credit has been issued for the 12 hours of downtime that occurred.
We have identified methods to increase the speed in which software packages are compiled and tested prior to installation.
Additional protective measures will be added between testing of new changes and the deployment across our servers.
In the case of the puppet module where unit test failures were silently lost, we will make changes to ensure a confirmed success is required.
We currently deploy new changes to a small number of servers initially to test for any unintended actions. More extensive testing will be implemented to verify proper functionality before deploying to a larger number of servers.
We will investigate whether additional tests can help detect unintended software removals.
Python 2.7 support will be removed from our package manager in two months time, which will simplify software package management.
FAQ
Q: Was there any customer data loss?
A: No, there was no loss of any customer data. The system packager manager gracefully uninstalled hundreds of software packages, but system configuration files along with customer data were retained.
Q: Was there a security breach?
A: No. The individual that issued this update was authorized to do so, they just didn't realize the update contained a critical software bug.
Q: Why did I receive an email about exceeding my monthly traffic limit?
A: In rare cases, traffic was incorrectly counted during the duration of the outage. We have reset the monthly traffic counters for these affected users.
