Florida city says water system hacked, warns critical-infrastructure community

Critical-infrastructure systems are potential targets for cyberattacks, local Florida officials warned after a hacker last week infiltrated the computer system of a Tampa-area city’s water-treatment plant and made a potentially dangerous change to the lye level in the system that was thwarted by an alert plant operator.

Eric Seidel, mayor of Oldsmar, Fla.—a city with a population of less than 15,000—said the public was never in danger of drinking poisoned water, but he acknowledged the hack to the Oldsmar water-treatment system and emphasized the broader implication of the breach on critical-infrastructure entities.

“The important thing is to put everybody on notice,” Seidel said during a press conference Monday about the incident. “I think that’s really the purpose of today, to make sure that everyone realizes that these kind of bad actors are out there. It’s happening, so really take a hard look at what you have in place.”

Pinellas County Sheriff Bob Gualtieri echoed this sentiment.

“Because of this security breach, we are asking that all governmental entities within the Tampa Bay area with critical-infrastructure components actively review their computer-security protocols and make any updates that are consistent with the most up-to-date practices,” Gualtieri said during the press conference.

Last Friday morning, an operator at the Oldsmar water-treatment plant first noted that someone accessed the plant’s computer system, but the operator was not alarmed, because no changes were made and supervisors often checked the system via a remote-access function, Gualtieri said. But the hacker returned at about 1:30 p.m. later that day and was much more active while in the system for 3-5 minutes, changing the level of sodium hydroxide—commonly known as lye, a primary ingredient in liquid drain cleaners—to dangerous levels.

“The hacker changed the sodium hydroxide [level] from about 100 parts per million to 11,100 parts per million,” Gualtieri said. “This is obviously a significant and potentially dangerous increase.

“After the intruder increased the parts per million from 100 to 11,100, the intruder exited the system, and the plant operator immediately reduced the level back to the appropriate amount of 100. Because the operator noticed the increase and lowered it right away, at no time was there a significant effect on the water being treated. Importantly, the public was never in danger.”

Other local, state and federal agencies were notified and are helping investigate the incident, Gualtieri said. As of Monday, no suspects had been identified. Gualtieri said that officials have not determined a motive or whether the hack originated from inside the U.S.

Gualtieri said that it would have taken 24 to 36 hours before water with the high level of lye could have been distributed to the public. Seidel said that the water-treatment plant system includes safeguards within other levels of the system that would have prevented the dangerous water from being consumed by the public.

“The reality of it is that the redundancies we have in place, they work … That lye would have never made it through the process to someone’s tap,” Seidel said. “The systems are set up to catch it. But everyone should be on notice.”

Oldsmar City Manager Al Braithwaite reiterated the role that multiple alarms in the water-treatment system play in ensuring the public’s safety in this case, but he also acknowledged the concern around cyberattacks—a sentiment that has been echoed by many governments and enterprises throughout the United States.

“I think we’ve anticipated that this was coming,” Braithwaite said during the press conference, although he did not cite any other example of critical-infrastructure systems being hacked.

Braithwaite and Gualtieri both noted that the water-treatment plant has disabled the remote-access capability to the facility’s computer system.

Sivan Tehila, a cybersecurity strategist for the Israel-based cybersecurity firm Perimeter 81, said that the lack of publicly acknowledged instances of critical-infrastructure hacks in the United States does not mean that such incidents do not occur.

“I’m not very surprised that it [the Oldsmar water-treatment hack] happened, and there are many similar cases that we probably are not aware of,” Tehila said during an interview with IWCE’s Urgent Communications. “I assume it happens more often than we actually think.”

The fact that the Oldsmar hack resulted in the intrusion of infrastructure as important as the city’s water supply likely will enhance the growing awareness of cybersecurity for critical infrastructure, Tehila said.

“When it comes closer to our day-to-day life—like water, a subway or things like that—it make us much more worried,” she said. “It’s one thing if someone is getting access to your webcam and can see what you’re doing, but it’s another thing when it comes to human life.”

Tehila applauded the quick actions of the Oldsmar water-treatment operator.

“This operator who immediately recognized that there was something wrong with the [lye] level should get a nice award for that,” Tehila said. “Most analysts who are sitting in behind the screen are so overwhelmed that it is hard to decide what is a false-positive alert and what is an actual alert.

“It’s not only about the technology; it’s about the people—if they are skilled enough, if they are trained, and if they are not too tired … Many times, [enterprises] are investing in technology, but they’re not investing in the analysts or the people who sit behind the screen.”

Tehila, who has helped design security solutions for railway and air-pollution systems during her career, acknowledged the challenges facing critical-infrastructure network administrators, particularly as they try to make remote-access opportunities to employees during the COVID-19 pandemic to promote security and convenience. The age of equipment used in many critical-infrastructure systems can create difficulties when trying to integrate them with modern-day IT networks and access protocols, she said.

“One of the reasons why it is so hard to monitor this environment—because, basically, in order to monitor, you need to connect directly to the controllers of a specific system,” Tehila said. “All of these controllers usually are not advanced, so you really need to find a way to connect them and to monitor them without causing any damage.”

“There is no choice but to monitor these systems, but because of the lack of visibility, it’s really hard to identify real-time potential hacks.”

When combined with high-profile hacking issues associated with SolarWinds network-management software, ransomware at hospitals and other critical-infrastructure systems, the incident at the Oldsmar water-treatment plant could spark renewed awareness—and potentially funding—in cybersecurity efforts, according to Tehila.

“I feel like that, because of SolarWinds, we do see government take cybersecurity more seriously,” she said. “I believe that is going to help other organizations and cities get more budget for cybersecurity.

“It’s never too late. But, on the other hand, it’s never enough. It’s a space race, and that’s how it works.”

Perimeter 81 advocates that its clients use Defense in Depth and the company’s zero-trust model to implement remote-access functionality as part of its “firewall as a service” strategy, Tehila said.

“We are a creating a modern remote-access solution to replace VPNs and other legacy solutions,” Tehila said. “Basically, users can connect from everywhere to any environment. We are cloud-agnostic. Besides that, we also have an option to manage policies for the application level and the network level.”

For enterprises without a significant budget for cybersecurity, Tehila said they still can do a lot to enhance cybersecurity by following basic security guidelines that are publicly available. At some point, regulations should be considered, particularly as Internet of Things (IoT) systems are deployed that add connected devices—some with little or no security—that many fear can be exploited to access sensitive networks.

“Many people don’t like regulation, but I think regulation is a great thing, when it comes to security, because it means that companies, industry, government and organizations have to align with this regulation,” Tehila said. “It just makes us, in general, more secure.

“I hope that maybe the SolarWinds incident, all of these things that happened during COVID—like the hospitals [hit by ransomware—and this case with the water will push the regulators to actually have regulations for IoT.”

Despite the considerable cybersecurity challenges associated with modern networking, returning to an architecture with siloed systems likely is not realistic, Tehila said.

“I don’t think there is an option to go back, honestly,” Tehila said. “I think that these companies will have to deal with a new situation, and we’re just going to need to get used to the fact that we’re living in a different world. We’re going to have better technologies to protect ourselves. There will always be someone who will be able to hack that, but we are going to improve ourselves.

“I don’t see any companies today, in the new reality, that are able to not use the cloud for the specific needs of their organization. Besides that, there is no way to monitor in this environment without connecting to the Internet somehow.”