Member Login Menu
Civil Engineering Magazine THE MAGAZINE OF THE AMERICAN SOCIETY OF CIVIL ENGINEERS

Risk, Resilience, Response

By JAY LANDERS

In accordance with the requirements of the America's Water Infrastructure Act of 2018, U.S. drinking water providers must complete risk and resilience assessments and emergency response plans. Large water utilities had to complete their assessments earlier this year, offering valuable lessons to their smaller counterparts as they seek to prepare their own.

Signed into law on October 23, 2018, the America's Water Infrastructure Act (AWIA) imposed new security requirements for U.S. drinking water systems serving more than 3,300 people. Section 2013 of the AWIA amended the Safe Drinking Water Act to require that such providers conduct a risk and resilience assessment (RRA) to examine risks to their systems "from malevolent acts and natural hazards," according to the law. As part of its RRA, a water provider also must evaluate the resilience of its entire enterprise, from its source water to its distribution system as well as its monitoring, operation and maintenance, and chemical storage and handling practices. Such assessments also are to include a water provider's financial infrastructure and its protection against cyber threats (see the related article: What Must a Risk and Resilience Assessment Include?).

These assessments, in turn, are to serve as the basis for the other main requirement of Section 2013 of the AWIA, the development by water utilities of an emergency response plan (ERP). According to the act, the plans must include the following four components:

  • "strategies and resources to improve the resilience of the system, including the physical security and cybersecurity of the system;
  • plans and procedures that can be implemented, and identification of equipment that can be utilized, in the event of a malevolent act or natural hazard that threatens the ability of the community water system to deliver safe drinking water;
  • actions, procedures, and equipment which can obviate or significantly lessen the impact of a malevolent act or natural hazard on the public health and the safety and supply of drinking water provided to communities and individuals, including the development of alternative source water options, relocation of water intakes, and construction of flood protection barriers; and
  • strategies that can be used to aid in the detection of malevolent acts or natural hazards that threaten the security or resilience of the system."

The law included staggered deadlines for utilities to complete their RRAs and certify to the U.S. Environmental Protection Agency (EPA) that they had done so. Larger systems, defined as those serving 100,000 people or more, had until March 31 to complete this task. These same utilities were given another six months to complete their ERPs. Medium-size utilities have until the end of the year to finalize their RRAs, while smaller utilities have until June 30, 2021. After certification of their RRAs, utilities have six months to develop or update their ERPs. (See the table.) The AWIA also requires that utilities review their RRAs every five years and certify to the EPA that they have done so and have updated their assessments, if necessary.

Section 2013 of the AWIA amounts to an "update" of the Public Health Security and Bioterrorism Preparedness and Response Act of 2002, commonly known as the Bioterrorism Act, says Kevin Morley, Ph.D., the manager of federal relations for the American Water Works Association (AWWA), of Denver. "It's everything that was in the Bioterrorism Act, plus," Morley says. The key difference between the AWIA and its predecessor is that the Bioterrorism Act "was strictly focused on terrorism" and did not address natural disasters or other possible threats to water utilities, he notes. For its part, the AWIA "maintains a need to focus on malevolent acts, but it also brings in natural hazards," Morley says," because that's what is more likely to cause a utility a problem on any given Sunday."

To illustrate his point, Morley gives the example of a large water utility in the southeastern United States that upgraded its physical security after it had conducted its vulnerability assessment as required by the Bioterrorism Act. "They spent a lot of money on fences, cameras, and access control," he says. "Then they had an F5 tornado. Fences and cameras don't do anything to get you ready for an F5 tornado and what that requires from a preparedness perspective." By contrast, "much of what you do to prepare for a hurricane is transferrable to these other threats," Morley says.

Essentially, the AWIA mandates that water utilities use an "all-hazards approach" when assessing the various threats to their systems, says Will Williams, the associate vice president for asset management in the Atlanta office of Black & Veatch, which has its headquarters in Overland Park, Kansas. Besides the aspects of physical security that were the focus of the Bioterrorism Act, the AWIA also considers factors related to cybersecurity as well as natural hazards, Williams says.

The importance of ensuring adequate security against cyber threats is "huge," says Simon Watson, CMRP, the operations and maintenance practice leader for Brown and Caldwell, which has its headquarters in Walnut Creek, California. Increasingly, many water utilities rely on internet-based systems for their customer service processes as well as their operational procedures, particularly as a result of remotely accessing supervisory control and data acquisition (SCADA) systems used to operate treatment systems, pump stations, and the like. "Because we're starting to go to more virtual remote access for those, it opens you up for more cyber hacking," Watson says. "You didn't have that in the older way of doing business." In the past, when a SCADA system displayed an alarm, an operator would have to physically check on the equipment in question. That is not always the case today, Watson says. If an operator receives an alarm, he or she might use an iPad or other device to access the SCADA system and ascertain the problem. "That's another cyber link that's in flux," Watson notes.

The cybersecurity hazard is the "one universal threat to everybody," Morley says. "It is one hundred percent probable that you will experience some type of cyber incident. It may not be catastrophic, but it certainly can be disruptive."

In fact, the cyber threat is what prompted Congress to include in the AWIA the requirement that RRAs assess the "financial infrastructure" of water utilities, meaning their business or enterprise systems, Morley says. "This is about ransomware," he says, referring to a type of malware increasingly used by hackers to prevent users from accessing their systems until they have paid a "ransom" by means of an anonymous online payment service. For this reason, RRAs must assess the integrity of any systems used to manage cyber risks associated with a utility's information technology system as well as its operational technology system (i.e., SCADA system).

Ultimately, cybersecurity is "about risk management, not risk elimination," Morley says. "There are many things that can be done to do some basic blocking and tackling to make it harder to get into that network," he says.

Clearly, water utilities conducting their RRAs must consider a "much broader range of risks" than used to be evaluated as part of similar assessments, says Forrest Gist, P.E., the global technology lead for security in the intelligent solutions practice at Jacobs, which has its headquarters in Dallas. For example, RRAs should also include an analysis of dependency hazards and proximity threats, Gist says. A dependency hazard involves the loss of something that a utility depends upon, while a proximity threat concerns an incident involving a nearby organization or entity that impairs the functions of the water utility. As an example of the latter, Gist posits a scenario in which a chemical plant neighboring a water utility experiences an explosion, fire, or spill. "How does that impact your facility?" he asks.

Common dependency hazards include electrical power, diesel fuel for generators, and chemicals and other products needed as part of the treatment process. "A critical one that's being looked at right now, especially with COVID-19, is loss of staff," Gist says. "How do utilities work around having to have some of their employees shelter in place or work from home?"

In some cases, a dependency hazard may not be readily obvious, Watson says. "Some of the chemicals needed for water treatment may have multiple suppliers, but it's all from only one manufacturer," Watson notes. "It still only comes from one place. That's a challenge. What is the backup plan if you can't get XYZ chemical?"

Because proximity threats pose risks to source water, Congress specifically mandated that source water be evaluated as part of RRAs. In doing so, Morley says, lawmakers were responding to two episodes in 2014 of large-scale contamination of the source water supplying major American cities. The first, in January of that year, involved the spill of a coal-washing chemical into West Virginia's Elk River upstream of the main drinking water intake used to supply the city of Charleston. As a result, about 300,000 people in the Charleston metropolitan area lost access to potable water for several days. In August 2014, a toxic algal bloom in western Lake Erie forced the city of Toledo, Ohio, to warn its residents not to drink or touch the water from their taps for three days.

In requiring assessments of threats to source water, Congress aimed for utilities to gain better understanding of how to ensure the resilience of these critical resources. "The idea there was [that] there's stuff outside your fence line that can make you have a bad day," Morley says. "You don't have any control over it, but you should evaluate that threat and recognize what the possibility is and understand what you can and can't do." To help utilities better understand potential threats to their source waters, Section 2018 of the AWIA directs states to allow water utilities access to reports prepared as part of Emergency Planning and Community Right To Know Act detailing substances stored in aboveground tanks.

"All of those four threats—malevolent acts, natural hazards, dependency, and proximity—are reviewed in a well-rounded risk assessment," Gist says. However, additional threats need to be considered, depending on the particular situation of an individual utility. For example, Gist cites the case of a client he worked with that assumed its largest threat concerned cybersecurity. However, during the process of developing its RRA, Jacobs and the utility discovered that the imminent retirement of senior staff presented a much more tangible problem. "For them, their biggest threat was loss of staff," Gist notes. "There were one or two people who had a lot of really important information in their head. It wasn't really written down, or if it was, nobody knew where it was written down." As a result, Jacobs recommended that the utility take steps to document the processes used by key staff and implement programs aimed at disseminating the information held by senior staff to others within the organization.

Besides risks, an RRA must address the resilience of a water system's physical and electronic components. The issue of resilience, Gist says, must be examined to answer the question, "How can a utility bounce back after an impact?

Of course, the answers to this question will depend entirely on the individual risk under evaluation. At the same time, responses to certain risks may need to evolve over time as circumstances change. For example, Watson notes that the growing incidence of wildfires in California in recent years has prompted some electric utilities, especially those in the northern portion of the state, to engage in public safety power shutoffs. During such an incident, an electric utility turns off power in areas experiencing strong winds and other conditions that increase fire risk. Because the power now tends to be shut off for longer periods of time compared with past efforts to prevent fires, this has implications for water utilities conducting RRAs, Watson says. "They're having to come up with longer-term mitigations," he explains. "We've had a lot of utilities that are starting to look at putting in more permanent backup power, whereas in the past maybe they had a portable generator they could take around to different areas."

Although Section 2013 of the AWIA specifies the topics to be addressed in an RRA, the law is silent on the process to be used when conducting such assessments. "Utilities are free to use the methodology that they wish, provided they cover the risk topics required under the law," Gist says.

The EPA has developed tools and guidance for use in developing RRAs, including the Vulnerability Self-Assessment Tool (VSAT) designed for assessing risk and resilience at drinking water and wastewater systems. The VSAT is intended for medium- or larger-size utilities. To help smaller utilities comply with the RRA requirements of Section 2013 of the AWIA, the EPA prepared the guidance document Small System Risk and Resilience Assessment Checklist.

Additional material is available to help utilities assess potential threats and hazards and the likelihood of occurrence. For example, the EPA released in 2019 Baseline Information on Malevolent Acts for Community Water Systems, which is intended for use in selecting malevolent threats to be included in an RRA. The U.S. Department of Homeland Security and other law enforcement agencies also have prepared guidance for this purpose.

The "gold standard" method for developing RRAs, Gist says, is spelled out in the 2010 document Risk Analysis and Management for Critical Asset Protection (RAMCAP) Standard for Risk and Resilience Management of Water and Wastewater Systems, which was developed by the AWWA and the American National Standards Institute, of Washington, D.C. Commonly known as the J-100-10 standard, the document describes a seven-step process that "covers all the bases that are needed within the risk and resilience assessment," Gist says. Briefly, the seven steps are:

  1. characterizing assets
  2. characterizing threats and hazards
  3. analyzing the consequences of threats and hazards
  4. analyzing vulnerabilities that would allow a threat or hazard to occur
  5. analyzing the likelihood of the various threats and hazards
  6. assessing a utility's current level of risk and resilience
  7. considering options for reducing risks and increasing resilience

By conducting the first six steps of the seven-step process spelled out in the J-100-10 standard, a utility determines the consequences, vulnerabilities, and likelihoods for every threat and asset and multiplies those factors to determine the corresponding risk. "All those previous steps come together in step six, which is a risk analysis developing the actual numbers for each risk," Gist says. "The numbers are listed in risk-dollars per year. It makes it very easy to understand and compare risks."

Armed with this information, a utility identifies ways to reduce those risks during the seventh step. For example, a utility would look to develop "mitigation recommendations, improvement recommendations, and staffing or policy recommendations" to reduce those risks, Gist says. The utility then assesses the recommendations in terms of their benefits and costs and develops a schedule for implementation.

Having a monetized estimate of the annual baseline cost associated with various risks can go a long way toward helping a utility "justify some expenditures for reducing risk," Watson says. He uses the example of a utility that faces a threat having a risk value of $500,000 annually. If utility staff develop a mitigation plan that costs $100,000, the board is able to "see the cost-benefit analysis" and may be more likely to approve the expense, he says. "Board members like to make decisions based on risk and dollars."

In light of these and other methodologies and accompanying guidance documents, the question looms as to which approach is best for a given utility. Seeking to answer this question, the Water Research Foundation—which has offices in Denver and Alexandria, Virginia—undertook Project 5014, Practical Framework for Water Infrastructure Resilience. The ongoing research effort seeks to help utilities "navigate through the maze to find the best way of assessing resilience for their specific circumstances and size," says Black & Veatch's Williams, who is leading the project.

"What we're developing is a framework to help utilities decide on the level of resilience that they intend to put in place," Williams says, "the best balance of performance, cost, and risk that they want to achieve." To this end, the project is intended to help utilities of all sizes and types, not just drinking water providers, understand which guidance or standards would best assist them in these efforts. The goal is to provide utilities with a "decision-making framework" that accounts for such variables as utility type and size and can be used to determine how best to proceed with efforts to develop resilience within their systems, he says.

This framework "will allow water utilities to make the right decisions about how they most efficiently get to the right level of resilience and how that might improve over time," Williams says. In this way, utilities can comply with the requirements of the AWIA but also integrate resilience more deeply into their systems as they move forward.

As part of the research project for the Water Research Foundation, Black & Veatch is working with 20 partners that include small, medium, and large utilities that encompass those providing services related to drinking water, wastewater treatment, or stormwater management. The team is developing a series of case studies and guidance documents that are intended to "raise the profile of the focus on resilience," Williams says. The finished product is scheduled to be completed by the end of this year, he notes.

As an interim measure, the Water Research Foundation and Black & Veatch released in late May AWIA Execution: Lessons Learned , a report that highlights practical guidance distilled from the first round of RRAs completed earlier this year by large utilities. The report includes recommended resources, tips for completing RRAs and ERPs, and suggestions for how to incorporate resilience more fully into utility programs in the future. "We're encouraging utilities to embrace resilience as part of business as usual," Williams says.

In an era when many utilities face significant challenges in addressing aging infrastructure, the idea of having to spend money to increase resilience may prove a tough sell. However, smart organizations will find ways to balance both needs, Williams says. "Utilities need to make sure they have a risk-based planning approach, so [that] they can make sure their next dollar is spent on the most benefit," he says. "The utilities that get the most out of this will be those that combine programs," by linking their RRAs with other objectives such as their asset management programs, workforce development efforts, or aging infrastructure replacement programs.

The AWWA's Morley agrees. After identifying the risks they face, utilities are better positioned "to do some potentially more directed capital improvement planning," he says. Once aware of a given risk, a utility can seek ways to address it as part of a separate capital program. "I've got this need that was a potential risk when I did my assessment," Morley explains. "So, I can knock that [risk] down a little bit by doing this other capital program. I might be able to kill two birds with one stone."

Additional Insight: WHAT MUST A RISK AND RESILIENCE ASSESSMENT INCLUDE?

California Water Service, which has its headquarters in San Jose, has 23 service areas within the state, including Bakersfield, Stockton, and many smaller cities. Of those service areas, eight serve more than 100,000 people, which necessitated the development of a separate RRA for each of those service areas by the March 31 deadline. "We kind of had to scramble to get everything done for those first eight," says Darin Duncan, P.E., the director of field operations for Cal Water.

Cal Water kicked off its initial work on the RRAs in fall 2019 by evaluating a host of existing reports regarding its facilities. "We looked at our water supply facility master plans, our urban water management plan, and some of our rate case filings and justifications," Duncan says. "We had security reports and emergency response plans. Pretty much all the reports we've done in the past, we put them out there and said, 'Let's take a look at these, and let's try to figure out risk from this.' It was kind of a shotgun approach."

After assessing risks in this broad fashion, Cal Water and its consultant, Brown and Caldwell, interviewed the local operators and management staff of individual systems to vet the information and obtain their input regarding risks. Overall, the RRA process resulted in a better, more complete understanding of smaller-scale, localized risks, rather than "big-picture risks," Duncan says. "We're in California. We're in earthquake country. We have systems that cross the San Andreas Fault. We've known about those risks. We know about the flooding risks."

Instead, the development of the RRAs helped provide greater insight regarding the "individual risks" of each of Cal Water's systems that was the subject of an RRA, Duncan says. For example, an area served by a single pipeline faces the risk of a complete loss of service should something happen to it. Although local operators tended to be aware of such risks, this information had not always been passed along to the corporate level, Duncan says. As a result of the RRA process, Cal Water is better positioned to address these risks. "Now everybody is aware of these major and minor risks to these systems," he notes.

With the information learned from developing the RRAs, Cal Water plans to take steps to reduce the risks. "We're going to work with our public utilities commission to show what we found and go forward and really work to eliminate this risk," Duncan says.

As it developed its RRA in advance of the March 31 deadline, the Salt Lake City Department of Public Utilities benefited from the existence of a previous vulnerability assessment it had conducted in 2003, says Natalie Moore, P.E., an engineer for the department. "We used that to assess the gaps between that study and the AWIA requirements," Moore says. "We looked for the gap and then, obviously, tried to fill in some of those things that we needed to update." To this end, the department used master plans that had been developed for individual water treatment plants, as well as a 2019 drought contingency plan. "There was a lot of information about some natural waterways and some of our mitigation efforts for drought that were helpful," she says. "We had a range of studies that we tried to glean some information from to help us with the RRA."

Working with Jacobs, the Salt Lake City Department of Public Utilities developed its RRA in a "pretty straightforward" manner, Moore says. "Jacobs laid out the pathway well." However, a significant amount of staff time was required to conduct the RRA, Moore notes. "It was a huge time dedication. We had twenty or thirty staff members in the room for half-day or all-day workshops and meetings." Staff were present to provide input on various aspects of the department's systems. Site visits to certain critical facilities also were held. "Organizing all the staff and finding time for everybody to participate was [the] most difficult but most important part," Moore says.

Although difficult, having the proper staff present for key discussions related to the RRA process is critical to ensuring success, Moore maintains. "It was most helpful to have the right staff in the room," she says. "We had everybody from customer service to billing to operations and management." In some cases, staff members were able to relay important information that was not widely known outside their immediate realms. "It's really helpful to get everybody at the table talking," Moore says.

Cal Water's Duncan concurs. "Make it a multidepartment, multidiscipline effort," he says. "Make sure you have your systems engineers who have different concerns from water quality people who have different concerns from operators. You get all those groups together, and you can really do well."

To help utilities with the preparation of their ERPs, the EPA published in July 2019 Community Water System Emergency Response Plan: Template and Instructions. The template is organized in a manner that helps utilities ensure that they have included all the information necessary to comply with the AWIA's requirements pertaining to ERPs. Such information includes details regarding resilience strategies, emergency plans and procedures, mitigation actions, and detection strategies.

Davidson Water Inc., of Welcome, North Carolina, is a private nonprofit membership cooperative that provides drinking water for about 150,000 people in Davidson County and portions of Randolph and Forsyth Counties. In summer 2019, Davidson Water hired Merrick and Company, of Greenwood Village, Colorado, to develop its RRA and ERP. For these tasks, Merrick brought onboard two subconsultants: Enterprise Management Associates Inc., of Boulder, Colorado, for cybersecurity and iParametrics LLC, of Alpharetta, Georgia, for facility security.

Although a security committee within Davidson Water has worked over the years to update its emergency response procedures, "we've needed to do a deeper dive," says Robert Walters, the vice president of construction and engineering for Davidson Water. "That's why I'm excited to see what the consultant will bring to us" for the ERP due on September 30, Walters says. "I want it to be more user-friendly, if you will."

To this end, as part of the overall ERP, Walters intends for Davidson Water to prepare small "flip books" that describe how operators and other staff should respond in the face of certain relatively common emergencies such as tornadoes or ice storms. Unlike a large manual, the small flip books could be kept by staff in their vehicles or other convenient locations and retrieved as necessary. The flip books would offer quick guidance on the steps to take, preparations to be made, and people to contact in the event of a given incident, Walters says. "That's what I mean by a user-friendly kind of thing."

In its own small way, the example from Walters highlights the ultimate goal of the AWIA—helping ensure that water utilities run as smoothly as possible, even in the event of hazards both likely and unlikely.

Jay Landers is a contributing editor to Civil Engineering.

Civil Engineering, July/August 2020, © American Society Of Civil Engineers. All Rights Reserved

related

Read Civil Engineering magazine on your smart device: download our apps.

app store play store