Article
Why outsourcing network management makes sense
This article originally appeared in print with the headline, "Don't be afraid to ask for help."
The goal of all public-safety agencies long has been to unify and streamline their separate, siloed communications systems. Doing so creates reliable, dedicated network links, both within individual agencies as well as across agency boundaries. Enhancing first-responder communications, particularly interoperability, is a critical goal.
This was what Santa Clara County, the city of San Jose and 14 other communities within the California county hoped to achieve when they formed the Silicon Valley Regional Interoperability Authority (SVRIA). From its beginning in 2009, the SVRIA has built a redundant network of microwave radio systems supporting the operations of 18 public-safety agencies.
“Part of what we do is try to enhance interoperability for both voice and data, and we also play a significant role in collaboration to enhance the communications tools of public-safety personnel,” said SVRIA Executive Director Mike Milas, who represents the public-safety interests of government bodies in the greater Santa Clara County area, aka Silicon Valley.
The SVRIA’s emergency communications (ECOMM) microwave radio infrastructure is a protected OC-3 ring of microwave terminals designed in both a loop and spur configuration.
“One thing we’ve done is to improve our infrastructure with the ECOMM network,” Milas said. “We have 41 sites now, and our area of operation—the county of Santa Clara—covers over 1,400 square miles where almost 2 million people live.”
It is one thing to build a public-safety microwave network, but it’s another thing to sustain it. Local government agencies may not always have the staff or higher-level technical expertise necessary to optimize, administer and maintain such a network, and that is the position that the SVRIA was in when this joint-powers authority was considering how to best provide these services.
With a widely dispersed network of microwave sites, Milas didn’t have the staff needed to maintain and operate the ECOMM network.
“We have a small staff and I don’t have my own technical team,” Milas said. “In order to support a system like ECOMM, we have to contract out services.”
As a result, the SVRIA partnered with Aviat Networks, which not only built the system, but also is providing a variety of support services, including network management via its network operations center.
Distributed network management. In adopting a centralized network operations center (NOC), the SVRIA sidestepped several challenges that face most public-safety agencies in managing and maintaining an emergency communications network. One of the key challenges is staffing the network. Many regional communications authorities are working with very tight budgets and don’t have microwave radio personnel on staff.
An agency often must rely on member agencies to provide maintenance and support for specific radio sites. However, this multi-party approach means that monitoring and management of the entire network may not be adequately cohesive and integrated.
Another negative aspect of the multi-party approach to network management concerns the inefficient housing of radio parts. With individual agencies charged with maintaining individual radio sites, each agency must stock and maintain an inventory of spare parts. This results in duplication of spares across the entire network, and it means that money is tied up for long periods of time in duplicate equipment that is not being used.
A third issue is that individual agencies may have personnel who are more versed in legacy analog radio designs, rather than with the digital and IP transport technology on which most modern radio systems are based today.
With no centralized authority monitoring the entire network, it is difficult to achieve broad interoperability with end-user equipment. The ability to have data information exchanges is lacking, and technical personnel often are unfamiliar with the methods of ensuring communications between proprietary systems and devices.
Every network needs maintenance, yet with multiple agencies involved with maintaining various microwave sites, it is difficult to establish a workable maintenance schedule. Maintenance is necessary, because environmental factors such as wind, snow and ice can cause antennas to shift out of alignment or corrode mounting brackets on outdoor equipment. Meanwhile backup battery power systems for indoor equipment must be maintained to ensure reliable disaster protection.
Another challenge with distributed network management is that it is difficult to coordinate the addition of new members to the network. When nobody is responsible for the overall network, it’s hard to integrate a new member into the overall scheme.
Centralized network management. In contrast, having a single entity responsible for the entire network brings efficiency and overall control. Specifically, the NOC service delivers centralized network control and management while ensuring that the right resource is onsite in a specific timeframe, with the right replacement component, to restore the network in the shortest period of time.
Having a centralized NOC eliwminates many of the challenges of distributed management. Staffing is not an issue, because the NOC provider delivers 24/7/365 staffing with its own personnel. Spare-parts inventory and maintenance of the system can be directed from a single site, so it is easy to optimize both. And, with an overall manager of the network, the entire network’s health is continually monitored in real time.
Providing a centralized NOC for the SVRIA network makes it possible to employ the following best practices:
There are specific service level agreements (SLAs) that dictate problem-resolution procedures and timing, as well as turnaround time on replacement parts (which are available the next business day).
The NOC is the sole “keeper of the network,” and a NOC representative participates in agency meetings to report on moves, adds or changes to the network over time.
There are specific disaster-recovery procedures in place, including a second, redundant NOC located in a different geographical location.
NOC personnel follow key security procedures to ensure that network data and access are protected from unauthorized personnel. The primary and redundant NOC are secured with access limited to network operations personnel. At customer locations, the customer interface demarcation points are established via dedicated communication lines protected by firewall gateways. Proprietary customer network information—for example, key contacts and site locations—are stored and managed via a secured portal exclusively for use by the network operations personnel.
The NOC monitors the network continually. As NOC personnel notice an outage, they initiate problem-resolution practices, including notifying the user community, opening case tickets, trying to resolve the problem remotely, and notifying the maintenance personnel, so that dispatch can be alerted if the problem can’t be solved remotely. The NOC provides people to guide the field technician, if required. Once a field technician has completed corrective action, any failed equipment is sent to the manufacturer’s factory for repair.
All actions taken to resolve a problem are updated by the ticketing process, so each step can be reported to the SVRIA. This continues until the problem is resolved, a new spare part is in stock and the failed-unit repair is underway. One major advantage with the NOC coordination is that there is positive control on the equipment being replaced (spares management).
NOC maintenance personnel have access to all sites without an escort. Because some sites are located at hospitals and other sensitive locations, it is essential that network-maintenance personnel have easy access to each microwave site. All employees receive thorough background checks before employment.
Maintenance personnel conduct informal training with SVRIA member agency technicians to make them more familiar with the system.
For the SVRIA, using a partner who offers a centralized NOC and committed support levels has provided real peace of mind about the microwave network’s status and operations. New members can be added to the system easily, and every member is ensured that the microwave sites in its locations are properly maintained and managed. For a public intercommunications agency without appropriate local technical resources, such a partnership is a winning scenario. n
Patrick Davis is vice president of customer care and services management for Aviat Networks.
The need for network
The need for network monitoring and management of the ECOMM system was intended as an interim solution to allow the governance body “SVRIA” time to facilitate a discussion among participating agencies on how best to support this and future shared systems. Among concepts to consider were dependence upon vendors, the evolution of agency resources into some sort of consolidated effort or a combination of options.
NOC services provided have not met the original intent to date. Evidence of this is easily proven by the chronic loss of system visibility created by equipment integration problems that have existed shortly after Acceptance Testing was completed. This situation occurs seemingly random and can affect the management and monitoring of part or the entire network. So far when the system is blind, the event in its self has not created an interruption of public safety operations. It is the attempt to remedy the events as it occurs by the maintenance vendor that has caused interruptions to police and fire communications systems.
When the ECOMM system was evolving there were many challenges answered. Each unique challenge was aggressively pursued and put to rest. Agency communications technicians engaged at an engineering level relentlessly sought to resolve each problem because they knew that at some point they could be solely responsible for the systems health during the inevitable 2:00 AM emergency call out. This became convoluted by the leadership style of the governance body which favored continuing vendor dependence.
Most communities never hear of or become affected by the challenges of public safety communications. I believe this is due to the many professional technicians who are dedicated to the smooth operation of these systems because it is their police officers, Deputies, Firefighters, Animal Control Officers and others who serve their community, that depend on these systems for not only their own lives but those they are serving.
There is a stark reality to technology; it will only perform as well as you understand it. When interests become dependent, you must come to terms with perception.