The backup plan (Blan B?)
Even the best-laid plans can run amuck. No matter how many backup systems you have in place, you will never achieve 100% reliability. You might achieve 99.9999% or better, but never 100%. I am speaking of communication systems in general and dispatching systems in particular.
Why is it that the 0.0001% of downtime is what sticks in the mind of those who sign our checks? Never mind the 99.9999% of the time that the system was “up” and operating normally. Another thing-why is it that the “down” time always seems to occur at a critical time? Is it a perceived problem or a real problem?
If the boss perceives it as a problem, it is a real problem to us! And believe me-when the dispatch center goes down, the boss will perceive it as a problem! As our world of electronics has become more sophisticated (read, “computer-dependent”), our capabilities have greatly expanded. The capabilities of such sophisticated equipment are limited primarily by the imaginations of those designing or buying such equipment. All of the proverbial “bells and whistles” that can be attached to the ultra-sophisticated, computerized equipment are nice to have but may not be so essential when the chips are down. This sophistication comes with a price beyond the purchase and initial installation costs.
Maintenance requirements are more stringent and require greater technical expertise than they did for older technology. The problem becomes more complex when we “put all our eggs into one basket.”
A perfect example is the dispatching center of my own employer, the South Carolina Forestry Commission. Just a few years ago we had a large network of lookout towers, as did most of the forestry services in this country. Our radio system was simple-literally, simplex! We had no repeaters (unless you can call one tower attendant relaying to another a “repeater”-a human repeater, if you will). Each county had several lookout towers, with one tower designated as the dispatch tower for each county. Thus, we had single-county dispatching.
The time came when the old lookout towers were phased out and replaced with air patrol-the passing of an era. There is something about a lookout tower that can’t be replaced by airplanes-although our pilots do a good job of detecting and directing the suppression of wildfires.
Then came more reorganization. Districts were combined to form regions, and several forestry “areas” were carved out of the regions. A region typically has five forestry areas, typically consisting of about four counties each. Every area was set up (crudely) to dispatch for the counties within it. Thus, we had four-county dispatching in typical areas.
Ultimately, dispatching was moved to a central point within the region. The dispatching center now serves the entire region (19 counties). The region is so large that the weather might be wet in one part of the region while wildfires might rage in another part. The radio system consists of several VHF high-band repeaters, some on mountaintops, along with several simplex stations controlled via telco lines back into the dispatch center. Now, with single-point regional dispatching, the dispatch center has become an all-important hub in our communication system. It has to remain operable even in the face of powerful storms that seem to plague the southeast.
For that reason, lightning and surge protection, along with backup power, were my foremost concerns. With single-point dispatching, downtime is an unaffordable luxury. However, we could hand off dispatching to another region through the T1 lines in an emergency (assuming the T1 circuits were operable). A large backup generator was installed along with a UPS system. Only critical components were served by the UPS. The generator could service most of the building. Because the generator is fueled by natural gas, it could run indefinitely without concern for fuel.
Figure 1 below shows a simplified block diagram of the interconnection of the generator, UPS and dispatching equipment. The UPS supplies only the critical components of the dispatch system. The UPS is a “standby” unit as opposed to an “online” unit. Even with the UPS in the standby mode, the power feeding the UPS outlets passes through the UPS and is conditioned by it. In the event that the UPS doesn’t “like” what it sees from the commercial power source (or generator), the UPS will switch to “inverter” operation, using the internal batteries to produce power. As long as the power feeding the UPS is within tolerance (frequency and voltage), the UPS simply passes the power through (while using it to float-charge the UPS batteries).
If the commercial power fails, the UPS will automatically go to the “inverter” mode and generate power for the UPS outlets. At the same time, the generator will start. Once the generator comes up to full power and stabilizes, the transfer switch will switch the electrical system to the generator. If the generator power is within tolerance, the UPS will revert to the “normal” mode, and the internal batteries will resume charging from the generator power source. Thus, the UPS is only online for a short period of time (assuming that the generator is operating correctly).
Recently, we experienced a power failure late in the afternoon. No thunderstorms were in the area, but the power was out for an hour or so. The UPS took over until the generator came on line. However, once the generator came on line, the UPS would switch back and forth between “normal” and “inverter” operation. This indicated that the UPS did not like the power coming from the generator. A quick check of the generator revealed that the generator frequency was fluctuating. The UPS frequency window was set to accept a frequency variance of +/-3Hz, the maximum window allowed.
We called a service company to service the generator. During the testing and “adjusting” of the generator, a voltage rise or surge set off something akin to a fireworks display in the dispatch center-complete with smoke, fire and screams as the dispatchers ran out the back door! Several of the lower-priced plastic-cased surge protectors literally smoked and tripped circuit breakers on the breaker panel. These are basically MOV shunt devices that ultimately fail in the shorted mode-often in an impressive display that will scare off even the most daring dispatcher. After we finally got everything stabilized and back “up,” we were able to coax the dispatchers back into the building.
At that point, I promised that I would get those plastic-cased cheap surge protectors replaced with better units that would not threaten to catch fire. I will continue this saga next month.
Until next time-stay tuned!
