Actions

Richmond only had 3 operators at water plant at time of outage, likely didn't test backup systems

Procurement records show the city knew as far back as 2012 that electrical equipment was "deteriorating," "aged," and had "decreasing reliability"
Posted 12:13 AM, Jan 14, 2025
and last updated 3:24 AM, Jan 14, 2025

RICHMOND, Va. — The backup systems at Richmond's water treatment plant were not tested ahead of an expected winter storm, according to initial information gathered by state regulators.

Those backup systems failed to keep the facility operating when the storm knocked out power at the plant on Jan. 6, causing a citywide water crisis with impacts felt across the region.

“This was a completely preventable event. It shouldn’t have happened," Dwayne Roadcap, the director of the Virginia Department of Health's Drinking Water Office, told CBS 6 Monday.

That sentiment was echoed by Robert Rains, a former wastewater plant operator with decades of experience at Richmond and Henrico facilities.

“It was this series of calamities and tragedies that happened, just pancaked on each other. Unfortunately, it shouldn't have happened to begin with. It shouldn't have happened," Rains said.

Watch: Tyler Layne questions Richmond leaders about storm readiness before water crisis

Tyler Layne questions Richmond leaders about storm readiness before water crisis

Rains spent 12 years working in Richmond's wastewater plant, where he was a chief operator, and then another 26 years working at Henrico's water reclamation facility. He retired 12 years ago.

He said part of his responsibilities included testing the facility's power sources and redundancy systems on a regular basis.

“We had diesel generators. It would automatically start up. We just had to check it once a week, run it for an hour and check the fuel level in it," Rains said.

According to briefings from city and state officials, when a Dominion Energy outage caused by tree branches on an overhead power line knocked out the plant's primary power supply, an automatic switch to transfer power to a secondary Dominion Energy source did not work.

An offsite electrician was called to the plant to assess the situation. He did not turn on a backup generator because he wanted to manually transfer power to the secondary Dominion Energy source.

The manual transfer of power was successful, but the computer system called SCADA, which Mayor Danny Avula described as the "brains" of the facility's operations, did not reconnect to the server.

A backup battery was being used to keep SCADA operating, but it did not last as long as expected.

Roadcap said it's his understanding that two more backup batteries, with different functions, were out of service last week. The city has not yet answered questions about whether those two batteries were working before the Jan. 6 storm.

Additionally, Roadcap said operators were not aware of a manual procedure, after all backup systems failed, to close valves in order to prevent water from flooding into the facility and damaging equipment.

“We need to be checking our valves to make sure that they are positioned where they're supposed to be. They shouldn't be open, and we need to manually close them and vice versa," Rains said.

Rains believed the crisis also brings up questions surrounding the reliability of critical infrastructure.

CBS 6 combed through procurement documents spanning many years and found records showing the city knew as far back as 2012 that the plant's electrical equipment was "deteriorating," "aged," and had "decreasing reliability."

An external facility and operations assessment was done at the time and determined the facility's main electrical utility equipment needed to be replaced and repaired, including the switchgear that should be able to automatically transfer power to the secondary Dominion source.

That's the component that Avula said triggered the "cascade" of failures.

“I think the failure of our switch is on our side, and so I think that switch may be part of the infrastructure that we'll obviously need to replace," Avula said during a briefing last week.

Procurement records revealed the city began soliciting contractors to perform the replacements and switchgear "modifications" in 2016, as the Richmond Times-Dispatch first reported.

Interim city spokesperson Julian Walker said a contract was eventually issued six years later in late 2022. Walker did not say when the actual work began, but he said it is happening right now.

Records show four contractors bid on the project in 2022, and the estimated costs ranged from $3.9 million to more than $7 million.

Watch: Councilmembers support water crisis probe, say they've funded all DPU requests

Councilmembers support water crisis probe, say they've funded all DPU requests

When asked what took so long for the project to get underway, Walker said previous rounds of solicitations for bids did not result in enough bids or reasonable bids.

CBS 6 also requested an interview with Former Richmond Mayor Levar Stoney, who is now running for lieutenant governor. Stoney's campaign manager Zach Marcus said he was not available.

In a written statement, Marcus said Stoney proposed budgets that included the funds for needed maintenance updates at the 100-year-old plant.

"That said, the mayor did not and would not be involved in any procurement decision per the Virginia Public Procurement Act. Ultimately, solutions for aging water infrastructure throughout our Commonwealth will require real, sustained federal and state investment in addition to local and regional partnerships," Marcus said.

State regulators will be reviewing the city's capital improvement plans and priorities as part of their investigation.

They'll also evaluate the Department of Public Utilities' policies, emergency plans, asset maintenance, and whether DPU properly prepared ahead of an emergency weather event.

“The early indications are that Richmond plant staff did not test or evaluate their backup systems before the event," Roadcap said.

Additionally, he said DPU has the capability to operate the plant with both of its Dominion Power sources at the same time. However, in the winter months, DPU turns one of those sources off.

"You want to have as many power systems online and available as you can, so it was somewhat surprising that the plant has one system turned off. So, we're looking into why they felt that they could operate the plant effectively in the event of an emergency like we had with only one of them operational," Roadcap said.

The level of staffing on the day of the outage also appeared to be an issue.

“We do know there were three operators on staff when the power went out," Roadcap said.

“Three operators? Is that enough?” reporter Tyler Layne asked.

“Clearly it was not," Roadcap said.

Roadcap said the state investigation will likely result in a series of recommendations with timelines to address them.

That investigation will be separate from a third-party probe Mayor Avula has committed to commissioning as well.

CBS 6 is committed to sharing community voices on this important topic. Email your thoughts to the CBS 6 Newsroom.

📲: CONNECT WITH US

Blue Sky| Facebook| Instagram| X| Threads| TikTok| YouTube

CBS6-News-at-4pm-and-Jennifer-Hudson-480x360.jpg

Entertainment

Watch 'The Jennifer Hudson Show' weekdays at 3 p.m. on CBS 6!

📱 Download CBS 6 News App
The app features breaking news alerts, live video, weather radar, traffic incidents, closings and delays and more.