AT&T provides initial insights about FirstNet performance, recovery efforts after Nashville explosion
Some FirstNet subscribers experienced service outages after the massive Dec. 25 explosion at AT&T’s Nashville network hub, but backup battery power at the facility and the use of deployable cell sites meant that many FirstNet users were able to communicate “within hours,” according to AT&T.
Although the Christmas-morning explosion caused significant damage to AT&T’s Nashville facility—“including a crater on the east-facing side”—the network hub remained operational for several hours that day, an AT&T spokesperson said today in a prepared statement provided to IWCE’s Urgent Communications
“Our network equipment was not directly knocked out by the blast, and mobility services—including FirstNet—continued operating on temporary battery power throughout the morning,” according to the AT&T statement. “First responders on FirstNet at the scene had prioritized service immediately following the blast.
“Due to the loss of commercial power during the explosion and the subsequent impact to our fixed back-up generators from water and fire damage, once the temporary battery power ran out, service disruptions occurred in the region, including parts of Kentucky and Alabama. FirstNet customers were temporarily impacted, but within hours, dedicated FirstNet portable cell sites were on air.”
One challenge to restoring service was the fact that the area around AT&T’s Nashville building—a regional network facility that handles voice, data and video traffic for wireless and fiber networks—was deemed to be an active crime scene. This access issue was particularly problematic in trying to work on the first floor of the building, while the basement level was flooded and required the water to be pumped, according to the AT&T spokesperson
“Building access was delayed up to 12 hours before we could begin recovery work as fires were extinguished, the crime scene was secured and building structural assessments were completed,” the AT&T spokesperson stated.
“Thanks to the heroic efforts of our team and the cooperation of federal, state and local law enforcement, we restored most service within 48 hours of the event. The unmatched capability and experience of our National Disaster Recovery (NDR) and FirstNet teams was the critical difference in the speed and efficiency of the recovery effort.”
This around-the-clock effort including pumping water out of the Nashville facility and drilling cabling access holes through concrete walls, so outside generators could power equipment inside the facility—a building located mere feet from the recreational vehicle owned by Anthony Quinn Warner that officials believe held the explosives. This process was completed within 36 hours of the blast, according to AT&T.
“Unlike many other nearby buildings, [AT&T’s Nashville network hub] remained structurally stable—illustrating the resiliency and design elements of this heavily hardened facility,” the AT&T spokesperson said.
Immediately after the blast, AT&T personnel focused their efforts to provide communications to the impacted area, according to the carrier.
“We mobilized resources, including portable cell sites, generators, our fleet of specialized equipment trailers and hundreds of personnel and partners from around the country,” according to the AT&T spokesperson. “Because our dedicated FirstNet fleet is housed across more than 40 sites nationwide, we’re physically staged for fast response in any emergency.
“We prioritized first responders in the immediate area, with dedicated FirstNet portable cell sites deployed to the scene within hours. Our FirstNet teams worked around the clock to support public safety’s emergency-communications needs. Our mobility network was also prioritized for consumers, enterprise business customers and public-safety agencies to help in managing their operations, fleets and connecting their employees.”
Overall, AT&T’s FirstNet Response Operations Group (ROG) handled almost 50 requests for additional network support from public-safety agencies, according to AT&T. A full fleet of 23 National Disaster Recovery (NDR) network recovery trailers were located at the Nashville site in preparation to provide recovery fallback, if the structure or the equipment inside the facility could not be recovered.
“By Sunday [Dec. 27], a majority of broadband and video services and more than 96% of affected mobility sites were restored, and more than 17 portable cell sites were on air across the region,” according to the AT&T spokesperson. “We continued around the clock work on remaining power issues and resolving individual customer impacts.
“We had 26 portable cell sites (SatCOLTs) deployed or staged to support this response, and—at peak—had 21 on-air simultaneously. FirstNet customers had prioritized service on all the portable sites.”
With the worst of the service-outage challenges from the Nashville explosion have been addressed, AT&T personnel will begin the process of assessing the company’s response to the incident, according to the AT&T spokesperson.
“Thanks to the heroic efforts of our team and the cooperation of federal, state and local law enforcement, we restored most service within 48 hours of the event,” the AT&T spokesperson stated.
“While we are proud of the resiliency of our building and equipment in the face of a direct hit and the quick and effective response, we always learn from disasters and improve our processes and protocols. We will do the same here.”
I guess no one is going to discuss the elephant in the room as to why there was no redundancy in the core network as required of FirstNet. Our county is directly attached to Nashville. We are Police Dept and our 130+ FirstNet Aircards were down for over 30 hours. Our Fire Dept was down for the same amount. We ordered a Colt and it never came. Our 911 was down for over 50 hours. AT&T dropped the ball spectacularly during this outage. Yes their men and woman worked tirelessly to get things restored but it should never have happened in the first place. We build redundancy into every 911 system and every critical infrastructure, why didn’t AT&T do the same? That’s what we need to focus on. Not this glossed over success story.
Mike
Sounds like Urgent Communications should contact you about writing an article about the real impact of this event. I hope they do but I’m not going to hold my breath.
I did reach out to Mike in an e-mail on Saturday; I hope we will be able to talk about what he experienced. Meanwhile, if anyone else has information about the impact the Nashville explosion had on communications services to public safety (including 911) or critical infrastructure, please call me at 262-442-9507 or e-mail me at [email protected]. Thanks, Donny
I sure hope you’r department has an LMR system that is not IP based, that can talk from radio to radio with no infrastructure. With what we have seen lately, that may end up being all of the communications that a department may have.
I am now convinced that First Net was a pipe dream from the beginning. Never will be suitable for police or mutual aid. Too fragile. Maybe it could be used for some mutual aid work. Thats it. For the amount of money being spent anyone who is a LE pro can see what a joke it was. I worked for over 20 years running a system and I would catch a pike of shit if people lost radio for a half hour. And stop calling this a major bombing. It wasnt. A vehicle borne device very local impact. New York was a major incident and LE radios continued on.