https://urgentcomm.com/wp-content/themes/ucm_child/assets/images/logo/footer-new-logo.png
  • Home
  • News
  • Multimedia
    • Back
    • Multimedia
    • Video
    • Podcasts
    • Galleries
    • IWCE’s Video Showcase
    • IWCE 2022 Winter Showcase
    • IWCE 2023 Pre-event Guide
  • Commentary
    • Back
    • Commentary
    • Urgent Matters
    • View From The Top
    • All Things IWCE
    • Legal Matters
  • Resources
    • Back
    • Resources
    • Webinars
    • White Papers
    • Reprints & Reuse
  • IWCE
    • Back
    • IWCE
    • Conference
    • Special Events
    • Exhibitor Listings
    • Premier Partners
    • Floor Plan
    • Exhibiting Information
    • Register for IWCE
  • About Us
    • Back
    • About Us
    • Contact Us
    • Advertise
    • Terms of Service
    • Privacy Statement
    • Cookie Policy
  • Related Sites
    • Back
    • American City & County
    • IWCE
    • Light Reading
    • IOT World Today
    • Mission Critical Technologies
    • TU-Auto
  • In the field
    • Back
    • In the field
    • Broadband Push-to-X
    • Internet of Things
    • Project 25
    • Public-Safety Broadband/FirstNet
    • Virtual/Augmented Reality
    • Land Mobile Radio
    • Long Term Evolution (LTE)
    • Applications
    • Drones/Robots
    • IoT/Smart X
    • Software
    • Subscriber Devices
    • Video
  • Call Center/Command
    • Back
    • Call Center/Command
    • Artificial Intelligence
    • NG911
    • Alerting Systems
    • Analytics
    • Dispatch/Call-taking
    • Incident Command/Situational Awareness
    • Tracking, Monitoring & Control
  • Network Tech
    • Back
    • Network Tech
    • Interoperability
    • LMR 100
    • LMR 200
    • Backhaul
    • Deployables
    • Power
    • Tower & Site
    • Wireless Networks
    • Coverage/Interference
    • Security
    • System Design
    • System Installation
    • System Operation
    • Test & Measurement
  • Operations
    • Back
    • Operations
    • Critical Infrastructure
    • Enterprise
    • Federal Government/Military
    • Public Safety
    • State & Local Government
    • Training
  • Regulations
    • Back
    • Regulations
    • Narrowbanding
    • T-Band
    • Rebanding
    • TV White Spaces
    • None
    • Funding
    • Policy
    • Regional Coordination
    • Standards
  • Organizations
    • Back
    • Organizations
    • AASHTO
    • APCO
    • DHS
    • DMR Association
    • ETA
    • EWA
    • FCC
    • IWCE
    • NASEMSO
    • NATE
    • NXDN Forum
    • NENA
    • NIST/PSCR
    • NPSTC
    • NTIA/FirstNet
    • P25 TIG
    • TETRA + CCA
    • UTC
Urgent Communications
  • NEWSLETTER
  • Home
  • News
  • Multimedia
    • Back
    • Video
    • Podcasts
    • Omdia Crit Comms Circle Podcast
    • Galleries
    • IWCE’s Video Showcase
    • IWCE 2023 Pre-event Guide
    • IWCE 2022 Winter Showcase
  • Commentary
    • Back
    • All Things IWCE
    • Urgent Matters
    • View From The Top
    • Legal Matters
  • Resources
    • Back
    • Webinars
    • White Papers
    • Reprints & Reuse
    • UC eZines
    • Sponsored content
  • IWCE
    • Back
    • Conference
    • Why Attend
    • Exhibitor Listing
    • Floor Plan
    • Exhibiting Information
    • Join the Event Mailing List
  • About Us
    • Back
    • About Us
    • Contact Us
    • Advertise
    • Cookie Policy
    • Terms of Service
    • Privacy Statement
  • Related Sites
    • Back
    • American City & County
    • IWCE
    • Light Reading
    • IOT World Today
    • TU-Auto
  • newsletter
  • In the field
    • Back
    • Internet of Things
    • Broadband Push-to-X
    • Project 25
    • Public-Safety Broadband/FirstNet
    • Virtual/Augmented Reality
    • Land Mobile Radio
    • Long Term Evolution (LTE)
    • Applications
    • Drones/Robots
    • IoT/Smart X
    • Software
    • Subscriber Devices
    • Video
  • Call Center/Command
    • Back
    • Artificial Intelligence
    • NG911
    • Alerting Systems
    • Analytics
    • Dispatch/Call-taking
    • Incident Command/Situational Awareness
    • Tracking, Monitoring & Control
  • Network Tech
    • Back
    • Cybersecurity
    • Interoperability
    • LMR 100
    • LMR 200
    • Backhaul
    • Deployables
    • Power
    • Tower & Site
    • Wireless Networks
    • Coverage/Interference
    • Security
    • System Design
    • System Installation
    • System Operation
    • Test & Measurement
  • Operations
    • Back
    • Critical Infrastructure
    • Enterprise
    • Federal Government/Military
    • Public Safety
    • State & Local Government
    • Training
  • Regulations
    • Back
    • Narrowbanding
    • T-Band
    • Rebanding
    • TV White Spaces
    • None
    • Funding
    • Policy
    • Regional Coordination
    • Standards
  • Organizations
    • Back
    • AASHTO
    • APCO
    • DHS
    • DMR Association
    • ETA
    • EWA
    • FCC
    • IWCE
    • NASEMSO
    • NATE
    • NXDN Forum
    • NENA
    • NIST/PSCR
    • NPSTC
    • NTIA/FirstNet
    • P25 TIG
    • TETRA + CCA
    • UTC
acc.com

Public Safety Commentary


Commentary

Reflections on recent LTE outages

Reflections on recent LTE outages

  • Written by Ken Rehbehn / CritComm Insights
  • 20th September 2022

Rogers Communications Canada experienced a national service outage on July 8, 2022, that halted LTE communications for millions of Canadians, including public-safety agencies counting on mobile broadband for operations. In light of this jarring failure, should public-safety leaders rethink plans to collapse all wireless communications onto a single wireless technology platform operated by a service provider?

The root cause of the Rogers outage was relatively mundane. It was not a hack. It was not a disaster. The failure came from a routine, well-planned, early-morning configuration change to core internet router reachability. The faulty configuration change flooded Rogers’ network-routing equipment with invalid route information, halting all incoming and outgoing traffic until the national routing fabric could be restarted—a laborious process that took many hours. Restoral efforts continued into the night.

While Rogers’ national outage was significant in scope and impact, it was not a unique event. Modern cellular networks operate with a core IP network serving as the network’s central nervous system. Equipment failure, configuration errors, and security attacks can affect the core network subsystems that govern LTE network operation. Subsystems implementing the Border Gateway Protocol (BGP), Directory Name Service (DNS), IP Multimedia Subsystems (IMS), and Home Subscriber Server (HSS) each can trigger national outages.

Recent history shows how prevalent these failures are. Just days before Roger’s outage, the KDDI LTE network in Japan experienced a national two-day shutdown caused by congestion in its IP Multimedia Subsystem hosting Voice over LTE. In Europe, an attempted June 2021 upgrade to Orange’s network in France disrupted voice services across the country. Christmas 2020 brought about a mass failure of AT&T cellular operations, including FirstNet, across the Southeast U.S. following the Christmas-day explosion outside a significant MPLS routing hub in Nashville. And in June 2020, T-Mobile’s Voice over LTE network experienced an outage across the southeast US following an IP-routing misconfiguration. The list can go on.

LTE’s foundational role in public-safety operations is well established in many countries today. Mobile broadband communications based on LTE serve as the essential enabler of vital tools, ranging from e-mail to incident dispatch. The future will expand this role. With all mobile network infrastructure vendors offering support for 3GPP Mission Critical LTE specifications, some nations are planning to retire legacy Land Mobile Radio (LMR) systems that are based on analog radio or digital trunked radio transmission technology.

The argument for a transition from narrowband to broadband is logical. As the term implies, narrowband radio technology lacks channel capacity for extensive data flows beyond basic messaging. In contrast, LTE was architected to provide an IP-based super-set of functionality that can accommodate the needs of outdated technology silos. For cost-conscious governments, the argument is enticing. Rather than funding two network technologies – LMR for push-to-talk tactical communications and LTE for data-centric communications – they can fund a single converged mission-critical LTE network that handles all requirements.

Unfortunately, the recent massive network outages raise several issues that local and national public safety authorities must consider before taking the final step of powering down the LMR networks. Chief amongst those is the ability of a dedicated public safety core network to operate following widespread IP or transport (e.g., MPLS, optical) failure that impacts the radio access network. Resilience tools such as MOCN architectures, roaming, multi-SIM user devices, and satellite communications can be essential elements of an authority’s strategy. Likewise, the continued operation of a parallel LMR network provides significant redundancy. Specific points to consider include:

  • Mission-critical LTE does not mean bullet-proof LTE. Modern LTE networks support policy-based priority handling of voice and data traffic. Likewise, these networks have special-purpose signaling that helps the network meet transmission time requirements for push-to-talk communications. But these mechanisms protect users and traffic from contention with non-public safety users. Mission-critical LTE and mission-critical push-to-talk cannot function when the core network fails.
  • Commercial core network scale may be a factor. The mass failures that catch the public’s eye are in large-scale networks serving millions of users. That scale may be a contributing factor in the extended time that it takes to correct a fault. Restarting a complex national network requires a careful sequence of operations that takes time. Dedicated public-safety core networks are, by nature, smaller in scale. Faults in the public-safety core network may be easier to find and fix than faults in a large-scale mobile operator network.
  • Dedicated public-safety core networks are useless if there is no functional LTE radio access network. While a dedicated public safety core network may be less prone to failure thanks to its reduced scale, the radio access traffic still needs to find its way to the dedicated core. National public-safety communications authorities must ensure that their selected mobile-network operator can forward radio access network traffic to the dedicated public-safety core, even if the operator’s main commercial core network is not responding. Because some of the radio access network functionality is based on dips into the operator’s main core network for an initial subscriber and policy lookup, that assurance may be an impossible dream.
  • LMR networks can fail, but they degrade in a layered approach. A point that sometimes gets glossed over when austerity-minded politicians think about converging all emergency communications on a single technology is that LMR technology is designed to fail. In contrast, LTE is based on the assumption that the core network is always available. Worse, if an LTE handset loses connectivity with the network, the handset cannot use LTE to communicate directly with other devices. While the 3GPP Proximity Services (ProSe) feature was intended to fill this gap, the ProSe capabilities fail to meet the requirements of firefighting and law enforcement teams operating deep in buildings. LMR failure is different. In the case of digital trunked systems, the individual radio tower sites will continue to function even if the LMR core network is lost. And end users can communicate directly if the radio tower site is not accessible. The layered degradation provides a time-proven approach to resilience that LTE cannot match.
  • LTE push-to-talk can work in tandem with LMR. As companies like ESChat and Catalyst Communications have proven, LTE-based support for push-to-talk communications works well in tandem with legacy Land Mobile Radio technology. 3GPP mission-critical push-to-talk (MCPTT) interworking gateways for TETRA and Project25 are now bringing this mix-and-match capability for 3GPP-based approaches. This versatility means that public safety officials can use MCPTT as a failback for failed LMR and LMR as a failback for failed LTE. Expense?
  • Cost savings from LMR retirement may be overstated. Much of the austerity argument driving the push to retire legacy LMR technology comes from an assumption that subscriber handset costs will go down thanks to LTE economy-of-scale benefits. That handset argument is failing as the need for an effective ProSe alternative forces a move to dual-technology devices supporting both LTE and a DMR/TETRA/Project 25/Analog direct mode radio. Of course, the austerity argument includes eliminating LMR network infrastructure and maintenance costs. But the cost of operating modern digital trunked networks may be more than offset by the value of having a simple, reliable alternative communications path.
  • A hidden cost of LMR retirement. A significant public-safety in-building communications footprint is based today on LMR, not LTE. In the U.S., fire-protection regulations mandate support for public-safety communications in new large buildings. Practices vary, but most developed nations worldwide also ensure that critical facilities get built with sufficient radio support for public-safety communications. Coverage is guaranteed for portions of the building that commercial cell operators do not cover. Builders must typically fund this coverage and install in-building systems that support LMR. A shift to LTE requires a substantial new investment for existing properties . Who pays for that upgrade?

Perhaps the cadence of mass LTE network outages will slow down or stop. Emerging AI-assisted network management tools, automation, and operational experience may lead to an elimination of mass outages. But the complexity of inter-connected IP networks dependent on fragile protocols such as BGP, DNS, and IMS suggests that mass outage risk remains a lingering factor for years to come.

Placing all communications requirements in a single technology basket is tempting. But public-safety communications planners must provide a robust PACE communications methodology for resilience that incorporates each PACE element: primary, alternate, contingency, and emergency communications functionality.

Tags: homepage-featured-4 Applications Backhaul Broadband Push-to-X Commentary Commentary Newsletters Coverage/Interference Critical Infrastructure Federal Government/Military Funding In-Building Incident Command/Situational Awareness Interoperability Land Mobile Radio Long Term Evolution (LTE) Mobile Data Commentary Mobile Voice Commentary Networks & Systems Commentary NTIA/FirstNet Policy Project 25 Project25 Commentary Public Safety Public Safety Commentary Public-Safety Broadband/FirstNet Regional Coordination Standards State & Local Government Subscriber Devices System Design System Installation System Operation Tower & Site Tracking, Monitoring & Control Wireless Networks Commentary

Most Recent


  • AT&T claims LTE coverage edge, FirstNet build more than 99% done
    AT&T claims a 250,000-square-mile coverage advantage and that the planned five-year deployment of the FirstNet public-safety broadband network operating on the 700 MHz Band 14 spectrum licensed to the FirstNet Authority is more than 99% complete as a contractual deadline approaches this week. AT&T—the contractor responsible for building and maintaining the FirstNet public-safety broadband system—made […]
  • Verizon
    Verizon Frontline supports U.S. Forest Service efforts against wildfires
    Verizon Frontline increased its support of entities responding to wildland fires during 2022, particularly the U.S. Forest Service (USFS), which accounted for more than half of this activity by the carrier’s Crisis Response Team, according to the carrier. Cory Davis, Verizon’s assistant vice president for public safety, said that Verizon Frontline provided communications support to […]
  • Autonomous-vehicle consequences could include more traffic
    Most discussions about driverless, autonomous cars, have led to claims that they will help reduce congestion. Not so, says the RAC Foundation. It cites a UK government report which believes the opposite is true. Claiming that driverless cars could increase congestion by 85% by 2060. Presently, it’s thought that drivers lose more than 80 hours […]
  • Tesla Model 3 hacked in less than 2 minutes at Pwn2Own contest
    Researchers from France-based pen-testing firm Synacktiv demonstrated two separate exploits against the Tesla Model 3 this week at the Pwn2Own hacking contest in Vancouver. The attacks gave them deep access into subsystems controlling the vehicle’s safety and other components. One of the exploits involved executing what is known as a time-of-check-to-time-of-use (TOCTTOU) attack on Tesla’s Gateway energy […]

3 comments

  1. Avatar JP 21st September 2022 @ 4:35 pm
    Reply

    Christmas morning explosion in downtown Nashville is another prime example.
    Communications aftermath persisted regionally for days afterward.

  2. Avatar GBH 21st September 2022 @ 5:14 pm
    Reply

    This all points out to why this national cry to move 9-1-1 to an all-IP infrastructure sounds pretty suspect to me. I am sure that the vendors love it as that is what they are selling, but the pubic safety agency that buys it, and ends up with injured or deceased parties when it collapses like a house or cards, is gonna be the one who gets to play musical lawyers. Older Technolgies like T1s are decidedly lower tech, but also decidedly more resilient and more decoupled, so that an error at one place doesnt collapse the whole blasted network!

  3. Avatar John Facella, P.E. 5th January 2023 @ 11:08 pm
    Reply

    This article is well researched, and discusses many of the same issues that I warned about in “Rethinking FirstNet and LMR”, in Mission Critical, June 2016. In the 6+ years since that article, we have now seen, as the author records, multiple failures of LTE or other nationwide systems. LTE is a complex network, comprised of many parts, which only a very few skilled people are capable of understanding and repairing. Putting all of public safety communications ultimately into just one nationwide network is to invite bad actors to take it down. Compare that with tens of thousands of LMR systems and a large network of service shops and vendors to support them. The issue of numbers of cores can work against public safety both ways. If there are too many cores, the interaction among cores, and the software updating of them, becomes complex. If there are only a few cores (I have been told FirstNet has 2) it makes for a more attractive target for bad actors because they can focus their energies on fewer targets. As this article points out, these recent outages were not created by bad actors. But as anyone who has been watching the cyber security world for the past 10 years knows, no agency or government has been immune to being hacked, even NSA. Why do we believe FirstNet or other such centralized national networks are immune to hacking?

Leave a comment Cancel reply

To leave a comment login with your Urgent Comms account:

Log in with your Urgent Comms account

Or alternatively provide your name, email address below:

Your email address will not be published. Required fields are marked *

Related Content

  • Necessity indeed is the mother of invention
  • LTE and liability: Why the fire service must move forward with digital incident command
  • Partnership and collaboration must be the foundation for emergency communications
  • FirstNet success means no hypothetical 'shots' need to be fired, Swenson says

Commentary


Updated: How ‘sidelink’ peer-to-peer communications can enhance public-safety operations

  • 1
27th February 2023

NG911 needed to secure our communities and nation

24th February 2023

How 5G is making cities safer, smarter, and more efficient

26th January 2023
view all

Events


UC Ezines


IWCE 2019 Wrap Up

13th May 2019
view all

Twitter


UrgentComm

AT&T claims LTE coverage edge, FirstNet build more than 99% done dlvr.it/SlXZfr

27th March 2023
UrgentComm

Verizon Frontline supports U.S. Forest Service efforts against wildfires dlvr.it/SlX1g3

27th March 2023
UrgentComm

Autonomous-vehicle consequences could include more traffic dlvr.it/SlWr67

27th March 2023
UrgentComm

Tesla Model 3 hacked in less than 2 minutes at Pwn2Own contest dlvr.it/SlVJg9

26th March 2023
UrgentComm

SES: JP Hemingway on satellites’ role in the digital divide, D2D and disasters dlvr.it/SlTL4h

25th March 2023
UrgentComm

House members introduce $15 billion NG911 funding bill dlvr.it/SlS0Lr

25th March 2023
UrgentComm

ADRF: Sun Kim discusses company’s new hybrid in-building wireless solution dlvr.it/SlRtSQ

25th March 2023
UrgentComm

U.S. cell towers and small cells: By the numbers dlvr.it/SlRn6N

25th March 2023

Newsletter

Sign up for UrgentComm’s newsletters to receive regular news and information updates about Communications and Technology.

Expert Commentary

Learn from experts about the latest technology in automation, machine-learning, big data and cybersecurity.

Business Media

Find the latest videos and media from the market leaders.

Media Kit and Advertising

Want to reach our digital and print audiences? Learn more here.

DISCOVER MORE FROM INFORMA TECH

  • American City & County
  • IWCE
  • Light Reading
  • IOT World Today
  • Mission Critical Technologies
  • TU-Auto

WORKING WITH US

  • About Us
  • Contact Us
  • Events
  • Careers

FOLLOW Urgent Comms ON SOCIAL

  • Privacy
  • CCPA: “Do Not Sell My Data”
  • Cookie Policy
  • Terms
Copyright © 2023 Informa PLC. Informa PLC is registered in England and Wales with company number 8860726 whose registered and Head office is 5 Howick Place, London, SW1P 1WG.