AT&T provides initial insights about FirstNet performance, recovery efforts after Nashville explosion

Some FirstNet subscribers experienced service outages after the massive Dec. 25 explosion at AT&T’s Nashville network hub, but backup battery power at the facility and the use of deployable cell sites meant that many FirstNet users were able to communicate “within hours,” according to AT&T.

Although the Christmas-morning explosion caused significant damage to AT&T’s Nashville facility—“including a crater on the east-facing side”—the network hub remained operational for several hours that day, an AT&T spokesperson said today in a prepared statement provided to IWCE’s Urgent Communications

“Our network equipment was not directly knocked out by the blast, and mobility services—including FirstNet—continued operating on temporary battery power throughout the morning,” according to the AT&T statement. “First responders on FirstNet at the scene had prioritized service immediately following the blast.

“Due to the loss of commercial power during the explosion and the subsequent impact to our fixed back-up generators from water and fire damage, once the temporary battery power ran out, service disruptions occurred in the region, including parts of Kentucky and Alabama. FirstNet customers were temporarily impacted, but within hours, dedicated FirstNet portable cell sites were on air.”

One challenge to restoring service was the fact that the area around AT&T’s Nashville building—a regional network facility that handles voice, data and video traffic for wireless and fiber networks—was deemed to be an active crime scene. This access issue was particularly problematic in trying to work on the first floor of the building, while the basement level was flooded and required the water to be pumped, according to the AT&T spokesperson

“Building access was delayed up to 12 hours before we could begin recovery work as fires were extinguished, the crime scene was secured and building structural assessments were completed,” the AT&T spokesperson stated.

“Thanks to the heroic efforts of our team and the cooperation of federal, state and local law enforcement, we restored most service within 48 hours of the event. The unmatched capability and experience of our National Disaster Recovery (NDR) and FirstNet teams was the critical difference in the speed and efficiency of the recovery effort.”

This around-the-clock effort including pumping water out of the Nashville facility and drilling cabling access holes through concrete walls, so outside generators could power equipment inside the facility—a building located mere feet from the recreational vehicle owned by Anthony Quinn Warner that officials believe held the explosives. This process was completed within 36 hours of the blast, according to AT&T.

“Unlike many other nearby buildings, [AT&T’s Nashville network hub] remained structurally stable—illustrating the resiliency and design elements of this heavily hardened facility,” the AT&T spokesperson said.

Immediately after the blast, AT&T personnel focused their efforts to provide communications to the impacted area, according to the carrier.

“We mobilized resources, including portable cell sites, generators, our fleet of specialized equipment trailers and hundreds of personnel and partners from around the country,” according to the AT&T spokesperson. “Because our dedicated FirstNet fleet is housed across more than 40 sites nationwide, we’re physically staged for fast response in any emergency.

“We prioritized first responders in the immediate area, with dedicated FirstNet portable cell sites deployed to the scene within hours. Our FirstNet teams worked around the clock to support public safety’s emergency-communications needs. Our mobility network was also prioritized for consumers, enterprise business customers and public-safety agencies to help in managing their operations, fleets and connecting their employees.”

Overall, AT&T’s FirstNet Response Operations Group (ROG) handled almost 50 requests for additional network support from public-safety agencies, according to AT&T. A full fleet of 23 National Disaster Recovery (NDR) network recovery trailers were located at the Nashville site in preparation to provide recovery fallback, if the structure or the equipment inside the facility could not be recovered.

“By Sunday [Dec. 27], a majority of broadband and video services and more than 96% of affected mobility sites were restored, and more than 17 portable cell sites were on air across the region,” according to the AT&T spokesperson. “We continued around the clock work on remaining power issues and resolving individual customer impacts.

“We had 26 portable cell sites (SatCOLTs) deployed or staged to support this response, and—at peak—had 21 on-air simultaneously. FirstNet customers had prioritized service on all the portable sites.”

With the worst of the service-outage challenges from the Nashville explosion have been addressed, AT&T personnel will begin the process of assessing the company’s response to the incident, according to the AT&T spokesperson.

“Thanks to the heroic efforts of our team and the cooperation of federal, state and local law enforcement, we restored most service within 48 hours of the event,” the AT&T spokesperson stated.

“While we are proud of the resiliency of our building and equipment in the face of a direct hit and the quick and effective response, we always learn from disasters and improve our processes and protocols. We will do the same here.”