Organizations are sometimes reluctant to share particulars about information heart fires resulting from NDAs and PR considerations. As such, it’s usually troublesome sufficient to hint situations of knowledge heart fires in any respect, except studies to native hearth departments or newsrooms have been made, or prospects expertise important downtime and demand explanations.
This tendency to withhold detailed studies about information heart fires would possibly assist corporations shield their reputations, however it will possibly additionally make it tougher for information heart operators to acknowledge vulnerabilities, be taught from incidents, and implement measures to make sure the security of their staff and prospects.
In response to an Uptime Institute weblog publish on information heart hearth frequency – which was written after a catastrophic hearth destroyed OVHcloud’s information heart in France in 2021 – there had been 11 studies of knowledge heart fires since its data started in 1994 – a median of 0.5 fires a yr.
Extra just lately, the info heart requirements group mentioned it recognized 14 “high-profile information heart outages” brought on by hearth or hearth suppression techniques between 2020 and early 2023.
Whereas fires account for a comparatively small share of incidents that impression information facilities, their potential ramifications shouldn’t be neglected. Along with the hazards they pose to workers, information heart fires may end up in prolonged downtime. This could doubtlessly price corporations tens of millions of {dollars} and result in critical inconveniences for patrons, leading to an erosion of belief.
Whereas particulars could be scarce, Data Center Information reviewed its archives to revisit main information heart fires and energy outages over the previous decade. We additionally spoke with trade consultants to offer insights that information heart staff can use to evaluate vulnerabilities and develop crucial security plans to guard towards future incidents.
A Google information heart in Iowa (Picture: Alamy)
1. Google Data Center Hearth, Iowa
Simply earlier than midday on August eighth, 2022, a hearth broke out in a big Google information heart in Council Bluffs, Iowa. The fireplace, which was first reported as an “electrical incident,” was brought on by an arc flash that sparked an explosion in a substation close to the primary information heart constructing.
Whereas not technically hearth, an arc flash is {an electrical} explosion that generates warmth upwards of 30,000 levels Fahrenheit, doubtlessly igniting supplies and inflicting fires. The explosion occurred whereas three staff have been accessing {an electrical} cupboard within the information heart’s fundamental room.
The fireplace injured three staff, who have been taken to a close-by hospital for remedy. It occurred on the identical day as outages of the corporate’s maps and search service, though Google mentioned the 2 incidents have been unrelated.
The Council Bluffs information heart is one among Google’s first services and one of many largest information heart campuses on the earth.
2. Evocative Data Center Hearth, New Jersey
Firefighters responded to a fireplace on the Evocative information heart facility in Secaucus, New Jersey, on October 12, 2023. The fireplace was contained to the uninterruptible energy provide (UPS) space and was rapidly extinguished. Regardless, it took its toll on the 105,000 sq.ft information heart, which required a full energy shutdown. Luckily, nobody was injured.
Evocative, previously INAP, supplies web connectivity to many corporations within the New York metro space.
A fireplace swept by means of OVHcloud’s SBG2 information heart in Strasbourg, France, in 2021 (Picture: Alamy)
3. OVHcloud Data Center Hearth, France
A fireplace on March 10, 2021, destroyed one among OVHcloud’s Strasbourg information facilities and a part of a second one.
No OVH, firefighting, or native authorities companies employees members have been injured, the French cloud computing firm mentioned.
The fireplace destroyed OVH’s SBG2 information heart utterly and 4 rooms in SBG1, in accordance with an incident report on the corporate’s web site. UPS was down within the SBG3 facility, and the remaining SBG4 information heart had “no bodily impression.”
3. AT&T Data Center Hearth, Texas
AT&T customers within the Dallas space misplaced web and cable companies after an “undetermined hearth” broke out in an AT&T information heart in Richardson, Texas heart in Richardson, Texas, on October 15, 2018.
In response to studies, the hearth started at a energy change and triggered prospects as much as 12 hours of downtime. The fireplace didn’t trigger any accidents.
4. Fisher Plaza Data Center, Washington
Round midday on June 2, 2009, {an electrical} hearth sparked inside Seattle’s Fisher Plaza information heart, which housed the servers of fashionable websites like Adhost.com, Microsoft’s Bing Journey, Verizon, and cost portal Authorize.web.
All information heart staff have been evacuated, and nobody was injured. Nevertheless, the incident resulted in $6.8 billion in damages and downtime.
An investigation carried out by Energy Science Engineering, a Washington-based engineering firm, discovered that the hearth was seemingly brought on by insufficient insulation in {an electrical} duct connecting the constructing to town energy grid.
This wasn’t the primary time Fisher Plaza had skilled energy outages and electrical fires. Only a yr earlier, a fireplace broke out in a garage-level electrical room, leading to the true property firm Redfin going offline for 5 hours.
5. SK Inc. C&C Hearth, South Korea
On October 15, 2021, a fireplace on the SK C&C information heart in Pangyo, South Korea, affected two main tech corporations, Kakao Company and Naver Company.
Whereas Naver rapidly restored its servers, Kakao confronted extended outages, disrupting its messaging platforms, cost apps, and rideshare companies for hours.
Regardless of having a catastrophe restoration protocol, Kakao’s plan didn’t account for the facility outage in the course of the hearth, delaying its restoration efforts. In response, Kakao established a ‘recurrence prevention committee’ to forestall comparable incidents sooner or later.
A CO2 hearth extinguishing system for an information heart (Picture: Alamy)
Constructing Resilience: Stopping Data Center Fires
The above examples are reminders that information heart fires can escape unexpectedly and have quite a lot of causes, together with arc flashes, defective infrastructure, {hardware} failures, and human error.
Whereas there are additionally unexpected threats like pure disasters, there are a lot of situations by which energy failures and electrical fires may need been prevented. Guaranteeing infrastructure security is essential to mitigating the chance of knowledge heart fires.
Sadly, bettering the security of crucial infrastructure usually comes with a big price ticket. In response to Chris Brown, Uptime Institute’s chief technical operator, infrastructure security is usually compromised resulting from price range constraints.
Learn extra of the newest information heart safety and threat administration information
“Restricted funds out there and the necessity to put money into crucial electrical and mechanical techniques have resulted in funds needing to be pulled from different areas,” Brown mentioned. “Moreover, in some areas, repurposing present buildings is important resulting from an absence of area and the present constructing buildings might not have been initially designed to the degrees most would take into account crucial for an information heart.”
He added: “Extra issues and investments are wanted within the precise construction but additionally the compartmentalization of complementary techniques to make sure that hearth is just not allowed to propagate and likewise is just not allowed to close down your entire information heart.”
Combating Fires By means of Compliance and Regulation
Over current years, there was a stronger push to create stronger compliance and regulation requirements for information heart infrastructure throughout the US and globally.
In December 2023, Colorado handed the Federal Data Center Enhancement Act, which outlined minimal requirements of infrastructure resiliency within the occasion of cyber and bodily assaults, in addition to pure disasters.
In Might 2024, Maryland handed the Vital Infrastructure Streamlining Act. The UK handed an analogous invoice, in January 2024, which launched new laws round reporting information heart incidents and strengthened security and infrastructure necessities.
Regulation and security measures to extend infrastructure resiliency are key to mitigating information heart disasters, but it surely’s additionally vital to do not forget that some incidents are unavoidable. This makes it crucial to have emergency protocols that facilitate a protected and environment friendly restoration.
How rapidly information facilities and consumer service could be restored usually will depend on the protocols and catastrophe restoration plans of the operator. Information heart catastrophe restoration plans embody catastrophe restoration groups, threat evaluation, redundant infrastructure, and backup energy turbines, which shield information and scale back downtime.
Key parts of an information heart catastrophe restoration plan. The data on this picture is reproduced with type permission from SANS Institute.
Staying on the Ball
Catastrophe restoration plans will not be simply paperwork and protocols. In addition they contain sturdy partnerships between information heart operators and their prospects. Krista Shepard, a spokesperson for Cologix, an information heart developer, mentioned catastrophe restoration plans will not be ‘set-it-and-forget-it’ measures, however dwelling paperwork that should be adaptable to match a continuously evolving panorama.
“The flexibility to swiftly restore operations within the occasion of a catastrophic occasion requires information backups in safe offsite places,” Shepard instructed Data Center Information. “It additionally requires rigorous testing, drilling, and proactive collaboration to make sure the catastrophe restoration plan is applied as seamlessly as attainable to spare invaluable time and sources within the occasion of a catastrophic loss.
She added: “It’s vital to periodically replace and refine catastrophe restoration plans as your enterprise and expertise evolve, and to adapt to altering environmental and climate situations.”
