FirstNet Authority board, task force to assess AT&T outage, suggest potential prevention measures
AT&T’s hours-long network outage that impacted FirstNet service last week will be discussed by the FirstNet Authority board during an open meeting next week, as well as the focus of a new task force created specifically to assess the incident and recommend strategies to help avoid a repeat in the future.
FirstNet Authority Executive Director and CEO Joe Wassel shared the information in a blog posted today on the FirstNet Authority website, noting that the FirstNet nationwide public-safety broadband network (NPSBN) was “built to be a reliable tool” for public-safety subscribers.
“From experience, I know that networks do fail, and it matters how we communicate during an event, surge to fix, and learn from those failures to help prevent future events and improve response and recovery,” Wassel states in the blog. “We can and will do better for our public-safety stakeholders.”
With this in mind, Wassel wrote that the FirstNet Authority is “committed to identifying the circumstances that led to the outage and working with AT&T to implement strategies and corrective actions to help prevent FirstNet from experiencing an outage like this in the future.
“Our contract requires AT&T to provide an after-action assessment of the root-cause analysis of the outage, and we look forward to receiving this as soon as possible. I am also meeting with AT&T leaders to discuss this outage in depth.”
FirstNet Authority board members next week are scheduled to convene next Wednesday for their quarterly open meeting, and last week’s AT&T outage “will be the top order of business at this meeting,” according to Wassel.
“We will discuss steps to strengthen the network and ensure FirstNet continues to be the most resilient and reliable broadband network for public safety,” Wassel states in his blog.
In addition, Wassel states that he has created a FirstNet After-Action Task Force, which will have “public-safety, technical and emergency-management experts from our team” assessing the circumstances associated with last week’s outage and recommending practices designed to prevent such incidents from happening again.
“The task force will help us strengthen our preparedness and emergency communications processes, so that in the event of any future outages, the FirstNet Authority, together with AT&T, can understand the impact faster and surge to communicate with the public-safety community,” Wassel states in his blog.
“And as we conduct our assessment of the magnitude and impact of this outage, the FirstNet Authority looks forward to sharing more information with our public safety stakeholders about how we plan to address, prepare for, and respond in the future. We are committed to improving our approach to understanding, communicating, and mitigating impacts on the network.”
Wassel did not provide any new details regarding the outage’s cause, reiterating AT&T’s explanation that the disruption for the cause of the outage, stating that the carrier giant has attributed it to “the application and execution of an incorrect process used while expanding its network; AT&T stated it was not the result of a cyberattack.”
AT&T said that it prioritized restoring service to FirstNet subscribers when last week’s outage occurred—a point Wassel confirmed in his blog.
“Based on initial reviews, the network outage occurred in the early hours of the morning on Thursday,” according to Wassel’s blog. “The FirstNet network was restored by around 5:00 a.m. CST—about 3 hours since service was initially affected for some FirstNet subscribers across the country.
“As the network operator for FirstNet, AT&T took immediate action to prioritize the restoration of public safety services. The FirstNet Authority recognizes that AT&T put public safety first during the outage. Resilience is crucial in the face of adversity, and AT&T stepped up and prioritized the restoration of FirstNet.”
Despite the problems associated with last week’s AT&T outage, Wassel expressed optimism about the long-term outlook for the FirstNet NPSBN.
“Challenging times have a way of bringing truths to light and making us stronger,” Wassel states in his blog. “Our truth is this: the FirstNet Authority will always put the public-safety community first in everything we do. We remain confident about the dependability of FirstNet and are committed to the continuous improvement of public safety’s network.”
Recent trends in telecom transport products, have headed toward “No Touch Provisioning”. This allows network engineers to provision a whole network from one location; saving labor time. The downside of that is, if something goes wrong, it risks losing most or all of the network at once.
Firstnet should probably be broken up into areas; and provisioning done in one area at a time. This way if something going wrong, it does not risk such a large outage. One then tests the changes in a minimal impact area, to identify bugs, before pushing it out any further. “No Touch Provisioning” also makes the network a much more tempting target for sabotage, since one act can bring down the whole network. Not sure this is what happened, but it’s my bet!!
AT&Ts nonsensical explanations don’t inspire confidence! They should have had an explanation to Firstnet withing a few hours after the incident–Not waiting for Firstnet to go after this information!!