Tennessee board reports Nashville bombing impact on 911, future plans

In the wake of the Nashville bombing that create extended outages for 66 public-safety answering points (PSAPs) AT&T has committed to improving its network resiliency, communications with 911 centers, and its backup-power capabilities, according to a report to the Tennessee Emergency Communications Board (TECB).

TECB Vice Chairman Steve Martini presented key findings of the report to the board during its May 5 meeting, although “sensitive” network-security information was not shared publicly. Based on five meetings with AT&T officials between April 6 and May 3, Martini said that  that main cause of the 911 outages stemmed from the fact that the AT&T personnel were not allowed to provide supplemental power sources to the key network in a timely manner after the bomb detonation resulted in the failure of the commercial power grid in the area.

“The primary issue with the outages to 911 and other services was the inability to access the building to replace commercial power before batteries failed,” Martini said during the meeting, a video of which can be viewed online.

“Going forward, AT&T is actively taking steps to additionally fortify the power supply, and the company is also talking with PSAPs to improve communications and transport options.”

In his presentation, Martini reiterated a finding that AT&T officials have stated in the aftermath of the Dec. 25 explosion in front of a key AT&T network hub in downtown Nashville: despite the blast from Anthony Quinn Warner’s RV at 6:30 a.m., communications services were maintained for several hours as backup batteries automatically powered equipment.

However, the batteries eventually ran out of power, because the backup generators—expected to keep the batteries powered in the event of the commercial electric grid being unavailable for an extended period of time—were damaged by the blast and did not work as designed.

AT&T personnel might have been able to bring power sources to the scene before the batteries were drained, but they were not allowed to access the site for almost 12 hours, according to the report. Officials at the location did not permit AT&T or others to access the site for two reason: (1) there was uncertainty that the building was structurally sound enough to enter safely after the blast, and (2) it was an area that was being assessed as a crime scene as evidence was gathered.

With this in mind, AT&T told TECB officials that it plans to address the issues by “evaluating engineering electrical-power-shutoff infrastructure into central offices nationwide” and “conducting engineering studies to enhance backup-generator flexibility and capability,” Martini said.

AT&T also is working to find solutions to ensure that its personnel can access communications gear in a timely manner, according to Martini.

“AT&T is talking with government officials about establishing national protocols to expedite access in the event that a future crime creates a communications outage,” he said. “While this a national undertaking—talking with the FCC and FBI, for example—local AT&T leaders have already briefed members of metro Nashville government and the state government on the need to revise crime-scene protocols to accommodate emergency-communications restoration. If necessary, these considerations can be addressed in legislation during the upcoming session.”

Outages related to the Nashville bombing did create hardships for 911 centers in the area, but some developed creative solutions that leveraged cloud-based service RapidSOS to maximize the number of 911 calls that received responses, even though many could not receive emergency calls through traditional paths.

Making the problems of the widespread outage—66 PSAPs were impacted negatively for days, according to the report—even worse was the fact that some 911 centers did not receive notifications or updates about the issue for six days, according to the TECB. AT&T said it sent communications to PSAPs in Tennessee and Kentucky about the situation via the Everbridge alerting system, but representatives of some PSAPs said they did not receive them.

TECB’s report indicates that the Everbridge system worked as designed, but in some cases the transmission link to the PSAP had been damaged, so the communications from AT&T were not received. Martini said diversifying the communications paths to 911 should be considered.

“AT&T will explore redundant PSAP network connections to include terrestrial—like diverse AVPN circuits—and non-terrestrial options like LTE wireless,” Martini said. “TECB can facilitate a meeting with AT&T for those PSAPs to explore these available options. In addition, AT&T has committed to providing any support or information needed to seek supplemental funding from state and local resources.”

Martini said that TECB could help the situation by prompting PSAPs to update the contact information for the Everbridge alert system more frequently than the annual updates that AT&T traditionally has sought “to make sure that communications are targeted, received and understood by PSAP decision makers.”

Detailed information that was deemed too sensitive to be shared publicly may be available to PSAP directors through individual meetings with AT&T officials that can be scheduled through the TECB.

TECB Chairman Phillip Noel emphasized that the release of the report about the bombing’s impact on 911 does not mark the end of the board’s work about the matter.

“This is not a final discussion,” Noel said during the meeting. “This discussion will continue for weeks or months—however long it takes.”

TECB is not the only entity pursuing greater clarity about what occurred in the aftermath of the Nashville bombing and what can be done to better prepare emergency communications for such situations. The FirstNet Authority also has been getting input from public-safety entities about the situation, as AT&T is the nationwide contractor building FirstNet, the nationwide public-safety broadband system (NPSBN).

“The Nashville bombing that impacted FirstNet users on Christmas Day is an example of how we have been able to work together to continuously improve the network and address resiliency challenges,” FirstNet Authority Chairman Tip Osterthaler said during the FirstNet Authority board’s meeting early this month.

“The FirstNet Authority has conducted an after-action review of the event, and I recently led a team back to Tennessee and Kentucky to follow up on our earlier visit with public-safety stakeholders who were most impacted by the blast and the subsequent service outage. These discussions were very helpful and productive to the development and completion of our after-action review of the event, and we are grateful for the cooperation of the participating agencies.”