Survivable Communications: Much more than hardened radios

Without question, Hurricane Katrina was a terrible natural disaster, resulting in a chilling number of deaths and injuries and a staggering amount of people displaced from their Gulf Coast homes. But it was more than that. For the communications community, the storm provided a rude awakening that has everyone from vendors to politicians reassessing the makeup of mission-critical communications.

Although the inability of commercial communication networks to function in the aftermath of Katrina was predictable, a variety of circumstances that caused even the most hardened land mobile radio (LMR) systems to fail during the crisis were not expected. And the impact of the resultant confusion was telling.

Without a solid communications foundation, recovery efforts ranged from inefficient to almost nonexistent immediately after Katrina passed, particularly in the New Orleans area. Numerous calls to the 911 system went unanswered, creating an unhealthy sense of uncertainty and abandonment for victims.

Compounding the anxiety were the rampant rumors emanating from the region, as word-of-mouth became the most prevalent form of communication. Desperate for any nuggets of information on the disaster, media outlets and policymakers inadvertently fell into the same trap, treating unsubstantiated speculation as gospel.

Many of the horrific stories that shaped public perception of a region in uncivilized pandemonium — reports of widespread looting, rioting, gunfire directed at rescuers, homicides and rapes — later proved to be wildly exaggerated or simply untrue. Nevertheless, the damage was done, as those aiding the recovery effort were forced to use valuable resources in an attempt to protect themselves instead of mobilizing. Similarly, first responders spent time and effort reacting to the myriad stories, often to learn that there was little or no truth to them.

“It is at times like these that we are reminded of the importance of being able to communicate,” said FCC Chairman Kevin Martin during the commission’s meeting after Katrina. “While no communications network could be expected to remain fully operational in the face of a direct hit from a category four or five hurricane, that fact was little consolation to the people on the ground.”

Flood of problems

Certainly the wind damage from Katrina was considerable, as missing rooftops and compelling photographs of downed wireless towers indicated. However, commercial wireless carriers owned nearly all of those towers. For the most part, LMR towers — typically designed to survive hurricane-force winds — remained intact physically, as they did last year when four hurricanes swept through Florida over a six-week period.

A few Motorola-built towers were damaged but were quickly replaced or repaired, said Kelly Kirwan, a Motorola vice president heading the company’s emergency-response team. None of the towers in the M/A-COM systems in the region were lost, even those in areas such as Biloxi, Miss., which absorbed some of the heaviest damage Katrina inflicted, said David Cerqua, M/A-COM’s district sales director for the area.

Flooding posed a much bigger problem, particularly in the New Orleans area, where waters rose much higher than ever expected when the levee protecting the city broke in multiple places. Floodwaters submerged many facilities housing the electronic gear for commercial wireline and wireless systems, rendering them inoperable. Furthermore, the power grid in the area largely failed, meaning operations effectively were limited to the life of backup power supplies such as generators and batteries.

“The wind was almost inconsequential,” said Stephen J. Gordon, commander of police communications for the city of New Orleans. “It was the water that was the killer.”

For the Louisiana State Police, only one of the 46 towers in its Motorola 800 MHz trunked system was significantly damaged by wind, but 20 other sites lost connectivity to the system hub at some point because the T-1 lines used to backhaul traffic stopped working, said Rex McDonald, director of information and communications services for the state police.

“We lost connectivity — not coverage, but connectivity — with our central site,” said McDonald, who noted that a patchwork of temporary microwave links and extensive use of Raytheon JPS’s ACU-1000 interconnection solution kept the communication system operating, though at reduced capacity.

In a few instances, some electronics gear was submerged, despite being located as much as 15 feet above the normal flood plain, McDonald said.

Cerqua said floodwaters did not directly harm any of the sites developed by M/A-COM — which designed and deployed the 800 MHz network used by the city of New Orleans — as all electronics and power backup gear were located high enough to keep them out of harm’s way. However, the floods still were a problem, as they made it difficult to access sites to refuel generators that ran dry and to perform other repair work, he said.

In addition, a generator located on top of the 42-story Energy Centre (see photo on page 45) was disabled when debris blown by hurricane winds punctured the radiator on the primary backup generator, Cerqua said.

“It wasn’t that the radio system didn’t work, it just ran out of gas.”

What next?

Regardless of the cause of the problems, lawmakers on Capitol Hill seem more focused than ever to enact measures to ensure first responders have reliable communications as they try to protect citizens during extreme crises.

“The time to find a solution is now — in fact, it was yesterday or the day before yesterday, and we still haven’t done it,” said Sen. Barbara Boxer (D-Calif.) during a Senate Commerce Committee hearing on the subject. “We didn’t learn our lesson after the ’93 World Trade Center bombing; we didn’t learn it after Sept. 11; the wildfires raging in California two years ago didn’t teach us … and Hurricane Katrina showed us. Enough is enough.”

With this in mind, Senate Commerce Committee Chairman Ted Stevens (R-Alaska) and Co-chair Daniel Inouye (D-Hawaii) have agreed to co-sponsor a bill designed to tackle many of the long-standing issues surrounding public-safety communications.

Key components of the legislation include clearing UHF television broadcasters from the 700 MHz band and granting public safety its long-promised 24 MHz of airwaves by April 2009. Commercial wireless operators would vie for the remaining spectrum in the band in an auction, with a large percentage of the proceeds earmarked to pay for upgrades to the 911 emergency-calling system and emergency communications, with Stevens publicly citing the need for interoperability (see news story on page 8).

A lack of spectrum and interoperability proved to be a problem for the Louisiana State Police, McDonald said. With the New Orleans system down, the statewide system was the only working communication network for a couple of days after the hurricane. However, the state system did not have access to enough airwaves to accommodate the hundreds of new users arriving on the scene to give aid, especially because widespread use of ACU-1000s to provide interoperability is not spectrally efficient, McDonald said.

But others said the traditional issues of public-safety spectrum and interoperability were not the primary communications problems related to Katrina. Indeed, Louisiana is one of the few places where television broadcasters do not occupy the 24 MHz swath of 700 MHz spectrum earmarked for public safety, and Motorola did provide some equipment to use those airwaves on a limited basis.

Gordon said the city of New Orleans’ system had plenty of capacity and provided interoperability paths once M/A-COM was able to resolve the power-generator problem on top of the Energy Centre and redirect a microwave dish that was turned askew by the hurricane winds when a supporting bracket broke.

Unfortunately, it took a couple of days for M/A-COM technicians to gain access to these sites because of flooding and glitches in the security-clearance process (see MRT, October, page 6).

With this in mind, legislation has been introduced in the Senate Homeland Security and Governmental Affairs Committee to address mission-critical network survivability.

“The attack of 9/11 highlighted the problem of interoperability, dramatically showing how vulnerable our first responders are in an emergency when police and firefighters are unable to communicate with each other,” said bill co-sponsor Joseph Lieberman (D-Conn.) in a statement. “Hurricane Katrina spotlighted an even more severe problem, operability — the need for systems that themselves can survive a disaster, either natural or man-made.”

Resiliency and redundancy

It’s a difficult task but not an insurmountable one, said sources interviewed for this article. Keys to success include redundant forms of connectivity, networks designed to re-route traffic around possible points of failure, hardened and better-protected equipment, and alternative sources of power — not only for the radio equipment but also for the support mechanisms necessary to ensure that it works properly.

In addition, remote monitoring and the ability to quickly access sites to repair and maintain equipment are vital in large-scale disasters.

McDonald said he believes the Louisiana state police system must develop connectivity outside of the T-1s supplied by local telecom carriers. Microwave links were a particularly popular alternative in the aftermath of Katrina, with Harris Microwave officials stating their company sold more than 100 links in the region after the hurricane ravaged the area.

Satellite connectivity also was an alternative explored thoroughly after the hurricane. Certainly its ubiquitous coverage and independence from terrestrial network problems were attractive to emergency workers, and satellite links from communications centers proved to be very valuable.

But satellite phones exhibited limited value for frontline rescue workers because they require a clear view to the southern sky and for the user to keep the antenna in an upright position — something that can be difficult to do while performing other recovery tasks, SAFECOM Director David Boyd testified during a Senate hearing.

McDonald agreed.

“Every satellite handset I passed out was returned to me,” McDonald said. “I’m sure they’re fine in the country, but they don’t work well in and around buildings.”

Proponents of IP technologies noted that it proved its resiliency after the hurricane, as the mayor of New Orleans’ first communications with the outside world were accomplished with an off-the-shelf voice-over-IP (VoIP) phone because the city’s radio system failed to work. Others noted that IP’s ability to re-route traffic around points of failure should be leveraged, particularly to prevent a repeat of the numerous 911 calls to disabled or evacuated public-safety answering points (PSAPs) that were lost.

“We received reports that there were 911 operators who couldn’t receive emergency calls but were surfing the Internet and sending instant messages,” said Morgan Wright, Cisco Systems’ global industry solution manager for justice and public safety. “What does that tell you?”

Although IP technology has some advantages, re-routing 911 calls can be accomplished using current telephony-based systems, said 911 consultant James Cavanagh of the Consultant Registry. However, the hurricane highlighted the need for better planning, he said, possibly including the establishment of regional 911 call centers where displaced PSAPs could continue to function as if operating from their usual facilities (see story on page 22).

“They could do it today without touching a packet, but it requires a level of planning that’s maybe one or two steps beyond anything they’ve done before,” Cavanagh said. “I think Katrina is a catalyst for doing something like that.”

Resiliency and power

Of course, re-routing can only work if network equipment is intact and working. The fact that most public-safety LMR towers withstood the mighty winds of Katrina is testament to the manufacturers meeting disaster-focused specifications, but future radio contracts must ensure that other critical parts of the network also are resilient.

And more than just radio equipment must be secured in areas that are safe from the elements. Cerqua said network designers did not consider the possibility that the glass walls of the Energy Centre could break and leave the city of New Orleans’ backup generator exposed to flying debris that caused it to fail. In the future, he said, M/A-COM would consider strategies such as erecting a brick wall to protect such critical infrastructure.

In addition, designers of mission-critical networks must consider factors outside their traditional realm to ensure survivability during such extreme circumstances. For instance, when repairing the generator at the Energy Centre, itioning necessary to prevent the generator from overheating was not supported by the backup power source, Cerqua said. After Katrina, portable air-conditioning units were used to cool the generator, but such circumstances should be considered during the network-planning stages, he said.

“Those are things you wouldn’t have thought to do pre-Katrina,” Cerqua said.

But none of these network-design strategies are relevant unless power sources are available to run the equipment. And when the ubiquitous commercial power grid fails, the dynamics of all systems change drastically.

Although radio base stations typically are equipped with backup power, that’s often not the case for the handset user in the field. As a result, there were several instances where recovery workers found their handsets drained of power with no convenient way to recharge them, sources said.

It’s a message also being sent by Mike Lee, president and CEO of Elevated Security, which designs fixed and portable communications systems that use alternative power sources such as wind power, solar power and hydrogen fuel cells (see photo on page 41).

This combination of sources powers Elevated Security’s portable video surveillance equipment. Elevated Security also has a Canopy broadband solution — powered independent of the power grid and traditional backup generator sources — that enables remote monitoring and control of the system.

Lee said a similar power setup could be deployed in the field to recharge radios or to serve as a bridge when backup generators run out of fuel. Eventually, the development of more power-efficient communications systems could let them operate completely independent of the power grid, he said.

“We’ve got to design systems that don’t rely on that kind of power,” Lee said. “If you can’t use it when the grid goes down, it’s useless.”

Is it practical?

While all of these alternatives and redundancies appear possible, they certainly would add cost to mission-critical networks — in some cases, very significant amounts. That, plus the fact such extreme specifications are needed only in very rare instances — if at all, in some locations — could create a dilemma for budget-conscious elected officials.

“You can make a system with all sorts of redundancies … you have four kinds of generators at every site,” New Orleans’ Gordon said. “It just depends on whether the jurisdiction wants to spend the money to do that.”

Particularly frustrating for those tackling this daunting task is that even an unlimited budget and the best planning in the world may not be enough to ensure that a network works in situations that can’t be imagined.

“You can plan for 99% of the contingencies, but it’s the 1% you didn’t plan on that always gets you,” Motorola’s Kirwan said.

All sources interviewed agreed that there is no single solution, with many noting that specific environmental and geographic concerns can make a significant difference. For instance, although a buried fiber cable might be considered a reliable backup in a flood situation, it would be useless if snapped by an earthquake.

McDonald said he believes it’s important that all alternatives be reviewed, with their respective strengths and weaknesses considered, when designers plan mission-critical networks.

“You really can’t rely on any one technology in a disaster of this magnitude,” McDonald said. “You really have to have a complement of different technologies to really have a comprehensive approach [to creating a survivable network].”