School of hard knocks

A plethora of disasters in the U.S. over the past few years has underscored the need to address weaknesses when it comes to keeping wireless communications running — and has taught local governments, public-safety communications operators and commercial carriers — which are playing a more vital role in emergency communications — some hard lessons in risk assessment.

As such, the wireless communications industry as a whole is beginning to embrace a new attitude: Plan for the what-ifs, and spend the money even if that means equipment sits unused 365 days of the year.

“Since Hurricane Katrina, we’ve seen the industry determining what the risk really is,” said Ben Holycross, radio systems manager with Polk County, Fla., who worked to keep communications systems operating in the aftermath of three major hurricanes that slammed the state over a six-week period in 2004. He also led a team to ravaged Hancock County, Miss., after Hurricane Katrina, which blasted the Gulf Coast in August 2005, to help restore communications.

“People are aware that public-safety communications systems indeed are mission critical, and they are looking at how to either build or beef up existing systems in order to have systems that will survive anticipated risks,” Holycross said. “That’s certainly refreshing, and it is a change from the past when even those people who did understand the risks were certainly less than encouraged to pursue upgrades because it was — and still is — a hard sell.”

Art Vanags, director of operations with Motorola, echoed that sentiment. Motorola now hosts a subcommittee within the Mutual Trunked Users Group called Disaster Preparedness and Recovery. It consists of representatives from state and local agencies as well as industries such as oil and gas. Once a month, the group brainstorms potential challenges communications systems might face and how to prepare for them. The goal is to publish a checklist of response-and-recovery issues and procedures, ranging from vital equipment to agreements needed to move quickly when disaster strikes. Motorola itself offers audits of cell sites and makes recommendations to operators about how they can make their sites more hardened and redundant.

“I came from the oil industry, where mitigation and disaster plans are second nature,” Vanags said. “Public-safety communications is now definitely aware of the need, and they are trying to understand what their like-in-kind entities are doing, and looking for us to give them some ideas about what they are overlooking.”

The general trend for both public-safety wireless operators and commercial carriers is spending more money to harden sites, said Rich Berliner, president and chief executive officer of Berliner Communications, which outsources network buildouts across the country for both types of operators.

“Hardening [tower] sites has been going on for some time, but the surge came after Sept. 11 with the stories of the phone systems working but not the fiber and T-1 backhaul,” Berliner said. “Katrina kicked it up.”

Hardening tower sites primarily involves three actions: installing backup generators; incorporating backhaul that isn’t easily disrupted, such as microwave links; and stockpiling portable sites.

Sprint Nextel, a key supplier of push-to-talk service to public safety, is determined to avoid being caught off guard as it was during the string of hurricanes in 2004 and Hurricane Katrina in 2005. The carrier is making an aggressive push to harden storm-prone tower sites in coastal communities. In mid-2006, it announced plans to spend $100 million to install permanent generators for critical sites and network facilities and to purchase portable generators, and satellite cell on wheels (SatCOWs) and satellite cell on light trucks (SatCOLTS) mobile communications centers, which can be deployed as part of the carrier’s disaster response and recovery program.

By the end of 2006, Sprint Nextel had permanent generators at approximately 800 sites serving both the nationwide Sprint PCS and Nextel networks, primarily in Florida. Sprint Nextel also announced plans to install 1500 additional permanent generators at sites in Alabama, the Carolinas, Georgia, Louisiana, Mississippi, Texas and other locations in Florida by the end of 2008. Prior to 2006, the operator only used a handful of generators to provide redundant power at its sites.

Those involved in site construction say generator sales have skyrocketed during the last two years. “All of the critical infrastructure people looked at 2004 and 2005 and analyzed their systems failures. One of the primary system failures was loss of power,” Holycross said.

In preparation for the 2006 hurricane season, which was substantially less severe than predicted, Cingular invested $60 million to harden its network with additional generators and implement its emergency response program, which includes two mobile command centers. Alltel announced it purchased more than 100 new generators specifically for the hurricane season.

The story is much the same for public safety, even away from hurricane-prone areas. York County, Va., for instance, has a backup generator at every site. Most of them are propane generators buried underground that can run for two weeks without the need for additional fuel.

An emphasis on dual-path backhaul also is becoming prevalent among wireless communications system operators. T-1 lines leased from local phone companies usually are the first to get knocked out of service during a disaster, so operators are looking to compensate by making sure they have more than one backhaul solution. Microwave backhaul is becoming the technology of choice for both public-safety and commercial operators.

“Microwave for commercial operators was not originally done in the context of hardening, but saving money,” Berliner said. “Now if you look at it in terms of saving money and providing secure communications, it looks to be an even better alternative.”

While fiber lines are typically severed, microwave antennas are usually just knocked around and can easily be re-positioned. Polk County connects its microwave links in a bi-directional circle. If any segment of the microwave path is severed, the signal can then travel in the opposite direction, Holycross said. York County’s new Astro P-25 system includes a Sonet ring, several extra DS-1 Sonet loops and a microwave system, which has a 99.999% availability guarantee, said Terry Hall, emergency communications manager with York County.

One prime example of the value of a multi-path backhaul strategy was the performance of SouthernLinc, a commercial iDEN operator in the Southeast, after Hurricane Katrina hit Alabama, Louisiana and Mississippi on Aug. 29, 2005. Clay Brogdon, manager of radio frequency engineering and construction, said SouthernLinc had 98% of its cell sites operating on Sept. 1, while other commercial and public-safety operators were down for weeks.

SouthernLinc is in the unique position of being owned by Southern Co., the parent company of five electric utilities in the Southeast, all of which also are SouthernLinc customers. As a result, SouthernLinc developed its system with utility-grade standards in mind. It maintains its own backhaul transport, in addition to using microwave links and leased T-1 lines, and has hardened every site within 100 miles of the coast with large reinforced concrete shelters and backup generators that include 50-gallon propane tanks.

The state of Alabama subsequently has teamed up with SouthernLinc, inviting it to be part of the state’s coordinated response to emergencies.

“They understand our affiliation with the electric utilities [is] an advantage in restoring communications after a storm,” Brogdon said.

Public safety now is catching on to something commercial mobile operators have been using for years to temporarily increase network capacity or fill in the gaps when the network goes down: portable sites, or what the commercial sector calls cell sites on wheels, or COWs. Motorola’s Vanags said his company has been stockpiling portable sites for years, shipping them out after the Sept. 11 terrorist attacks and Hurricane Katrina. Now a significant number of public-safety communications operators are interested in acquiring them, he said.

“We’ve gotten more requests in the last year for us to bring our transportable sites to trade shows,” Vanags said. “They all realize they have to do something more proactive. Katrina opened a lot of eyes.”

But the ye olde problem remains: justifying the costs. Tight budgets are the norm in the public-safety communications realm, which means it’s often difficult for decision-makers to justify having a million-dollar piece of equipment, such as a portable cell site, sitting in storage waiting for the next disaster that everyone hopes will never happen. Costs also dictate how extensively a permanent site can be hardened.

York County has responded by taking advantage of public/private partnerships when it comes to tower construction as a way to reduce the cost of its P25 system. For years, tower co-location has been the norm among commercial operators as municipalities nationwide have mandated the practice. But public-safety long has been left out of the loop. Today, however, there is perceived value in enticing a commercial operator to build a site complete with generators in exchange for free rent.

“Everyone knows that towers are real estate, and real estate is money,” Hall said. “The economics make great sense when we are looking at trying to justify a radio system. A tower is about $800,000, so we offset the cost by going to a cellular provider and asking them to build the tower to our specs in lieu of rent for 20 years. That can pay for a lot of maintenance costs.”

The situation also is a win for commercial operators, as they not only avoid paying rent but also can economically justify the hardening of their own sites. And because 60% of calls to 911 are wireless, the line between public-safety wireless and commercial wireless is blurring.