AT&T suffers extended nationwide outage, prioritizes FirstNet restoration
AT&T today suffered a nationwide outage of its wireless services that included interruption to the FirstNet nationwide public-safety broadband network (NPSBN), which the carrier giant has been building and maintaining for almost seven years.
During the early hours this morning, AT&T customers began reporting what the carrier giant referenced as a “nationwide outage” that also impacted Cricket Wireless customers. As of 11:15 a.m. Eastern time, AT&T reported that “our network teams took immediate action and, so far, three-quarters of our network has been restored.”
Three hours later, AT&T stated that its entire commercial network had been restored.
“We have restored wireless service to all our affected customers,” according to a statement posted on AT&T’s web site. “We sincerely apologize to them. Keeping our customers connected remains our top priority, and we are taking steps to ensure our customers do not experience this again in the future.”
FirstNet service was restored nationwide even sooner, according to an AT&T statement provided in response to an inquiry from IWCE’s Urgent Communications.
“Some FirstNet subscribers may have experienced a service disruption this morning,” according to the AT&T statement. “FirstNet is operating normally. We took immediate action to prioritize public-safety restoration.”
A FirstNet Authority spokesperson also noted the prioritized restoration effort for FirstNet users, noting that service was fully restored as of 11:30 a.m. Eastern time—the time of the statement—for these subscribers.
“The First Responder Network Authority is aware of wireless-service outages experienced this morning,” according to a FirstNet Authority statement provided to IWCE’s Urgent Communications. “Our nationwide network contractor, AT&T, took immediate action to prioritize restoration for public-safety users of FirstNet and has confirmed service is currently running normally across the FirstNet network. The FirstNet Authority will work with AT&T to conduct an assessment of the outage.”
Neither AT&T nor the FirstNet Authority provided any information indicating the cause of the outage during the day on Thursday, but AT&T issued the following statement on Thursday night:
“Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” according to the AT&T statement. “We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
Media reports on Thursday indicate that the FCC has said it is investigating the outage, while the FBI and the U.S. Department of Homeland Security (DHS)—specifically, the Cybersecurity and Infrastructure Security Agency (CISA)—also reportedly were trying to assess the matter.
“The bottom line is we don’t have all the answers,” White House national-security communications adviser John Kirby said. “We’re being told that AT&T has no reason to think that this was a cybersecurity incident. But again, I want to be careful. We won’t know until an investigation has been completed.”
Early media reports indicated that other U.S. carriers also suffered outages, but to the extent experienced by AT&T. In fact, Verizon issued a statement to IWCE’s Urgent Communications that problems experienced by its customers occurred only when trying to communicate with users of another carrier.
“Verizon’s network is operating normally,” according to the Verizon statement. “Some customers experienced issues this morning when calling or texting with customers served by another carrier. We are continuing to monitor the situation.”
This marks the second time in a little more than three years that there has been a widespread FirstNet outage. On Christmas Day in 2020, multiple states in the southeast U.S. experienced an extended outage when a bomber exploded a vehicle near an AT&T network hub in Nashville, Tenn. On that day, the explosion did not immediately halt communications, which remained operational for several hours before backup generators ran out of fuel.
Today’s AT&T outage occurred just a week after the FirstNet Authority announced plans to invest more than $8 billion into the FirstNet system during the next decade. In December, the FirstNet Authority validated that AT&T completed the initial five-year nationwide buildout of FirstNet on time and on budget, as AT&T had announced last spring.
AT&T has received a total of $6.5 billion from the FirstNet Authority—the maximum possible amount under the contract–with the validation of the initial five-year deployment of LTE equipment operating on 20 MHz of 700 MHz Band 14 spectrum licensed to the FirstNet Authority, an AT&T spokesperson informed IWCE’s Urgent Communications.
That $6.5 billion represented the bulk of $7 billion in funding—money generated from the proceeds of FCC spectrum auctions—allocated by Congress to the FirstNet Authority in 2012. That law also directed the FCC to award the FirstNet Authority with a nationwide license to the 700 MHz Band 14 spectrum, which AT&T pays the FirstNet Authority annually to use for both public-safety and commercial purposes.
Those payments from AT&T are scheduled to total $18 billion during the 25-year life of the contract with the FirstNet Authority. Of this amount, about $3 billion is expected to pay for the FirstNet Authority’s operations, and the other $15 billion is required to be invested in a manner that enhances the FirstNet system.
As a trivial matter, today is the 12th anniversary of former President Barack Obama signing into law massive legislation that included the creation of the FirstNet Authority in 2012.
Can’t speculate at this point but 0400hrs sound like a maintenance window. Remember this outage? https://urgentcomm.com/2022/09/20/reflections-on-recent-lte-outages/
With that in mind, I thought FirstNet assured us that it is running a separate core from AT&T commercial? https://firstnet.gov/newsroom/blog/firstnet-authority-roadmap-technology-domains-core
Quoted – “Neither AT&T nor the FirstNet Authority provided any information indicating the cause of the outage during the day on Thursday, but AT&T issued the following statement on Thursday night:
“Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” according to the AT&T statement. “We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
Media reports on Thursday indicate that the FCC has said it is investigating the outage, while the FBI and the U.S. Department of Homeland Security (DHS)—specifically, the Cybersecurity and Infrastructure Security Agency (CISA)—also reportedly were trying to assess the matter.”
– Stop Quote
Why in the world are the FBI and Homeland Security involved for an ATT Procedural Error?
Can agencies please start telling the truth?
Were we hacked?
Are people going to realize that a hybrid emergency comms system is THE ONLY ANSWER??
Don’t throw out LMR and HAMS for Cell-phone and network backhaul solutions…