“On September 5, Twitter experienced the loss of its Sacramento data region (SMF) due to extreme weather. The unprecedented event resulted in a total shutdown of physical equipment at the SMF,” said Carrie Fernandez, the company’s vice president of engineering, in an internal message to Twitter engineers on Friday.
Large technology companies typically have multiple data centers, in part to ensure that their service can stay online if one center fails; this is known as redundancy.
As a result of the outage in Sacramento, Twitter is in a “non-redundant state,” according to Fernandez’s Friday memo. She explained that Twitter’s data centers in Atlanta and Portland are still operational, but warned, “If we lose one of those remaining data centers, we may not be able to serve the traffic of all Twitter users.” .”
The memo continues to halt non-critical Twitter product updates until the company can fully restore its Sacramento data center services. “All production changes, including deployments and releases to mobile platforms, are blocked except for those changes required to address service continuity or other urgent operational needs,” Fernandez wrote.
The restrictions highlight the apparent vulnerability of some of Twitter’s most fundamental systems, a problem that Peiter “Mudge” Zatko, Twitter’s former security chief turned whistleblower, raised in a disclosure sent legislators and government agencies in July.
In his whistleblower disclosure, first reported by CNN and The Washington Post, Zatko warned that Twitter had “inadequate data center redundancy” that raised the risk of a brief service outage or even the possibility of Twitter going out. line forever.
“Even a temporary but overlapping outage of a small number of data centers is likely to result in service [Twitter] going offline for weeks, months or forever,” according to Zatko’s whistleblower disclosure. (Twitter has criticized Zatko and has widely defended itself against the allegations, saying the disclosure paints a “false narrative” of the company.)
News of the data center outage comes a day before Zatko is scheduled to testify before the Senate Judiciary Committee.
In a statement about the Sacramento outage, a Twitter spokesperson told CNN, “There have been no outages affecting people’s ability to access and use Twitter at this time. Our teams remain equipped with the tools and resources that were needed to send updates and will continue to work to provide a smooth experience on Twitter.”
Data centers need “water, power, reliable humidity controls and cooling to live,” said retired Brigadier General Greg Touhill, who served as the US government’s chief information security officer in 2016 and 2017.
“You want redundancy, not duplication, of your data locations to increase your cyber resilience so you can take a hit from a natural disaster [or other event] that can destroy a single piece of equipment or data center,” Touhill, who now heads the CERT division at Carnegie Mellon University’s Software Engineering Institute, told CNN.
