Plenty of popular websites suffered a couple of hours of unanticipated downtime when a truck hit a transformer and took down Rackspace’s Dallas datacenter.
Fans of Agent Pendergast from the novel Relic and its sequels should remember him counseling the museum to not rely on its overly sophisticated security system during a major society event, taking place while a hypothalamus-eating serial killer was on the loose.
The security system turned out to have a single point of failure, with disastrous results. Rackspace suffered a similar indignity, one that could not be put aside like a well-worn paperback.
Sites like GigaOm and 37Signals, the latter a software-as-a-service company, both experienced downtime as a truck took out a transformer near Rackspace’s Dallas facility. Rackspace explained what happened in a statement on their site:
In the second incident at approximately 6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended.
When we transferred power to our secondary utility power system, the data center’s chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim.
This repeated cycling of the chillers resulted in increasing temperatures within the data center. As a precautionary measure we decided to take some customers’ servers offline.
“Our Internet infrastructure despite all the talk is fragile as a fine porcelain cup on the roof of a car zipping across a pot-holed goat track,” Om Malik said of the outage.
Jesse Robbins at O’Reilly Radar looked at the incident from the SLA perspective, giving credit to Rackspace for having a clear agreement with its customers. That SLA promises a 5 percent credit per 30 minutes of downtime, which should salve the disruption for the affected customers.
