Cybersecurity of the Grid

LAS VEGAS — For Cybersecurity and Infrastructure Security Agency Director Jen Easterly the doomed CrowdStrike software update that took global IT systems and networks offline last month holds a "big lesson" for critical infrastructure.

“The CrowdStrike incident was such a terrible incident,” Easterly said Aug. 7 during a media briefing at the Black Hat cybersecurity conference, but “it was a useful exercise, like a dress rehearsal for what China may want to do to us.”

The outage was not the result of a malicious act, but rather a basic field input error that caused an out-of-bounds memory read. Yet, to Easterly, the widespread chaos it caused offers a clear example of what could occur if China-affiliated attackers make good on its efforts to cause systemic disruption to U.S. critical infrastructure.

When Easterly learned of the outage, around 2 a.m. on July 19: “What was going through my mind was ‘oh, this is exactly what China wants to do.’”

The outage highlighted the need for resilient systems to keep operations running in the wake of an incident or disruptive attack. But, for many of CrowdStrike's customers, normal operations ground to a halt.

Easterly gave the example of Volt Typhoon, a China state-sponsored threat group which has intruded and embedded in multiple U.S. critical infrastructure sectors to potentially launch disruptive or destructive attacks in the event of a conflict in the Taiwan Strait.

Federal authorities early this year warned that intrusions by the state-sponsored threat group and other China-linked groups are part of an extensive effort to maneuver in preparation for future attacks. The nation’s drinking and wastewater sector has confronted heightened threat activity from state-linked and criminal hackers targeting vulnerable water utilities.

The would-be attackers are effectively lying in wait.

Easterly and other U.S. officials worry these potential attacks could target pipelines, water systems, transportation and communications to incite panic and societal chaos.

CrowdStrike outage exemplifies unmet need for resiliency

CISA is a voluntary partnership agency, but as America’s cyber defense agency and the national coordinator for critical infrastructure security and resilience, it played a big role early on in trying to assess the impact of the outage CrowdStrike caused and worked with CrowdStrike to release mitigation guidance.

“It just reinforced what we’ve been saying about the importance of technology vendors designing, developing, testing and deploying software that is secure by design. And we saw that cybersecurity vendors were not immune from issues around software quality and design,” Easterly said.

“And it really reinforced just how ubiquitous software is and how much we depend on it working properly,” Easterly said.

While the CrowdStrike incident is widely regarded as one of the largest IT outages in history, it could have been worse, especially if malicious attackers were involved.

Federal cyber authorities say the China state-linked intrusions of U.S. critical infrastructure, at least the ones they're aware of, are likely just the tip of the iceberg.

“There is, we believe, much we are not seeing which is why we have been very clear that we shouldn’t focus on preventing,” Easterly said.

“I mean obviously, we want to prevent, but it really is about building resilience into our networks and our systems so that we can withstand significant disruption, at least drive down the recovery time to be able to provide services.”

Disclosure: Black Hat and Cybersecurity Dive are both owned by Informa. Black Hat has no influence over Cybersecurity Dive’s coverage.

The National Renewable Energy Laboratory last week published two reports highlighting cybersecurity tools that aim to improve utility asset management, risk quantification and visibility into the industrial control systems, or ICS, that are increasingly attached to the U.S. electric grid.

Utilities can request to pilot NREL’s Cyber100 Compass application, which models cyber risk associated with energy system upgrades, the national lab said July 25. And a July 24 report on the runZero cyber asset attack surface management, or CAASM, system that aims to reveal potentially “hidden” risks in a utility system concluded its scanning methods “can improve visibility without affecting the performance of ICS assets.”

The power grid is increasingly distributed and renewable but “right now, there’s a lot of uncertainty about how risky the transition is,” Maurice Martin, senior cybersecurity researcher at NREL, said in a statement. “It’s hard for utilities to know what kind of risk level they’re exposing themselves to, and that uncertainty can have a cooling effect.”

NREL’s Cyber100 Compass takes a new approach to managing cyber risk for utilities, combining data from subject matter experts with user-inputted information to perform a probabilistic risk assessment and produce a monetary estimate of potential losses due to a cyber attack, including one resulting in power outages.

The application asks utilities to provide information about their current energy system and plans for new resources, and then provides an assessment of how certain upgrades could change cybersecurity posture and risk.

“While the Cyber100 Compass application is still a proof of concept and not yet ready for industry use, it is an important first step toward developing guidance for system planners about potential cybersecurity risks as they integrate more renewable energy into their generation mix,” NREL said.

The lab has created a form for utilities to to request access to the application in exchange for providing feedback on its interface, usability and results. Utilities will need to input data on their risk tolerance and the value they place on avoiding certain types of cyber attack-induced physical events, including loss of power and harm to equipment attacks.

“The whole concept behind Cyber100 Compass is about mining the knowledge of these subject matter experts and capturing it in a form that is reusable across many different utilities of various sizes, loads, and generation mixes,” Martin said. “Cyber100 Compass provides a monetary expression of risk to help utility decision makers feel confident in their planned upgrades.”

Separately, NREL’s Clean Energy Cybersecurity Accelerator, or CECA, published a report assessing the efficacy of runZero’s CAASM  system. The report concluded runZero’s platform identified all internet-protocol-addressable assets in the test environment and collected detailed information about each device and all open ports.

“Evaluations also showed no adverse effects on deployed ICS assets or ongoing supervisory control and data acquisition communications and processes,” NREL said. “Evaluations show that runZero’s active scanning methods can improve visibility without affecting the performance of ICS assets.”

While passive scanning only looks at network traffic, active scanning involves sending data to assets and analyzing the response. 

“Running active scanning solutions built for IT environments can be unsafe and inappropriate in an [operational technology] environment due to bespoke assets, legacy firmware, or proprietary protocols,” NREL noted in its report. But runZero’s platform did not cause any problems.

“We are seeing more sophisticated attacks against critical infrastructure, particularly energy infrastructure,” Rob King, runZero’s director of security research, said in a statement. “Working with CECA allowed us to prove that active scanning of OT/ICS infrastructure can be done safely and effectively and is important to securing these vital systems.”

“It was interesting to see active scanning used safely,” said CECA technical team lead Nick Blair. “Active identification methods have been taboo in OT systems — for good reason — for a long time. While we can't claim our findings apply universally, hopefully they can break the ice and allow these methods to be considered as an option.”

Sponsored content

By Dragos

By: Jason Christopher, Dragos Principal Cyber Risk Advisor

The electric utility industry is a valuable target for adversaries seeking to exploit industrial control systems (ICS) and operations technology (OT) for a variety of purposes. A power disruption event from a cyberattack can occur from multiple components of an electric system including disruptions of the operational systems used for situational awareness and energy trading, targeting enterprise environments to achieve an enabling attack through interconnected and interdependent IT systems, or through a direct compromise of cyber digital assets used within OT environments. Attacks on electric systems – like attacks on other critical infrastructure sectors – can further an adversary’s criminal, political, economic, or geopolitical goals. As adversaries and their sponsors invest more effort and money into obtaining effects-focused capabilities, the risk of a disruptive or destructive attack on the electric sector significantly increases.

The number of publicly known attacks impacting ICS environments around the world continues to increase, and correspondingly the potential risk due to a disruptive cyber event impacting the North American electric sector is currently assessed as high. This report highlights multiple threats and adversaries focusing on critical infrastructure and their capabilities. Dragos anticipates the threat landscape associated with the sector will remain high as the detected intrusions continue to rise.

Of the activity groups that Dragos is actively tracking, nearly two-thirds of the groups performing ICS specific targeting and disruption activities are focused on the North American electric sector. Additionally, existing threats to ICS are expanding and establishing new interest in electric utility operations in North America. For example, the Dragos tracked activity group XENOTIME – the most dangerous and capable activity group – initially focused its targeting efforts on oil and gas operations before expanding to include North American electric utilities. Dragos also identified the MAGNALLIUM activity group expanding targeting to include electric utilities in the US. This activity group expansion and shift to the electric sector coincided with increasing political and military tensions in Gulf Coast Countries (GCC).

Dragos research of the CRASHOVERRIDE attack indicates ELECTRUM targeted recovery operations. Such activity, if successful, could prolong outages following a cyberattack and cause physical damage to equipment or harm to operators. These findings suggest the group had greater ambitions than what it achieved during its 2016 attack, and represent worrying possibilities for safety and protection-focused attacks in the future.

Historically, adversaries have demonstrated the capabilities to significantly disrupt electric operations in largescale cyber events through specialized malware and deep knowledge of targets’ operations environments. Although North America has not experienced similar attacks, ICS-targeting adversaries exhibit the interest and ability to target such networks with activities that could facilitate such attacks.

The electric sector, as a whole, has been working for over a decade to address cyber threats through board level decisions, 1 preparedness exercises like GridEx, the NERC CIP standards, and direct investment in ICS specific security technologies. However, adversaries will continue to evolve and the industry must be ready to adapt

The Federal Bureau of Investigation on July 1 warned private industry that expansion of U.S. renewable energy capacity increases the risk of being targeted by hackers who may want to disrupt power generation, steal intellectual property, or ransom critical information.

Attacks on residential solar systems “have been rare,” the FBI said, but hackers looking to make a bigger impact could target microgrids, or inverters at larger solar farms. “However, researchers are working to counter this potential risk through a passive sensor device that can detect unusual activity in the electrical current,” the federal law enforcement agency said.

The problem with rolling out new defenses to existing infrastructure, however, is that it leaves hackers with gaps of time when renewables may not be properly guarded. A “secure by design” approach, where monitoring and defenses are built in from the beginning, can address the issue “but the reality of it is, most companies don't do that yet,” said ​Avishai Avivi, chief information security officer of SafeBreach, a California-based cybersecurity firm.

Most of the FBI’s recommendations to protect renewable resources from hackers are general best practices, but Avivi said they are necessary because many people’s security practices simply aren’t very good.

“It's amazing how many people don't practice basic cyber hygiene,” he said, comparing malware to the covid pandemic. “It’s like washing your hands. ... It's a very simple, primitive solution, but it's very effective in staving off the infection. Not reusing passwords, segregating between functional areas, all kinds of very basic concepts that can help you, at the very least, minimize the potential impact of a malicious incident.”   

Unpatched systems can lead to threat actors gaining access to critical systems, said Tom Marsland, vice president of technology and technical service for security training group Cloud Range. But the FBI’s warning contained little specific advice, he said.

“Nothing here is special. People just need to do the basics, and companies need to invest in the basics," he said.

The FBI’s warning recommended:

• Renewable industry stakeholders should routinely monitor network activity for unusual or suspicious traffic;
• Company networks should be updated to patch security vulnerabilities, along with the use of firewalls and antivirus software;
• Offline backups of data should be maintained and all backup data should be encrypted;
• The security posture of third-party vendors should be examined;
• All passwords should comply with the National Institute of Standards and Technology’s standards for developing and managing password policies;
• Networks should be segmented to prevent the spread of ransomware.

“The FBI encourages current and former employees of companies within the renewable industry to report cyber intrusions targeting either themselves or their organization, as well suspected elicitation attempts by foreign nationals outside of the organization,” the FBI said.

The FBI’s recommendations are a good start, “but really constitute a minimum baseline of security controls necessary to mitigate the specific threats they outline: manipulating power inverters and targeting microgrids,” Mike Hamilton, former chief information security officer for the city of Seattle and now chief information security officer and founder of cybersecurity firm, Critical Insight, said in an email.

Developing and deploying specific power monitoring technology to detect attempts at compromise “may take time in existing deployments, however might be included in projects going forward without much delay,” Hamilton said. But he added that the recent decision by the U.S. Supreme Court to strike down the Chevron doctrine and limit federal agency authority, means “uptake of these FBI recommendations is likely to be spotty without an enforcement mechanism.”

It may be “technically correct” that expansion of the U.S. renewable energy industry could increase the risk of targeting by malicious cyber actors, said Malachi Walker, security advisor, DomainTools. But the majority of risks outlined by the FBI “seem to be applicable to any industry that grows in size and scope or leverages IoT connected devices.” 

“The development and deployment timelines of a standardized passive sensor are uncertain,” Walker said, but when combined with more general defense approaches would likely serve to protect renewable resources. However, those solutions “should not be excluded to only renewable energy projects.” 

"The problem facing renewable energy isn't unlike the problem facing the rest of the electric sector,” Gregory Pollmann, principal industrial hunter at Dragos, said in an email.

The developing nature of inverter-based resources means they rely heavily on vendors and third-party organizations for operation and installation, he said. “Those connections can add attack surface to industrial networks and be very difficult to monitor without robust visibility. With all of that said, the FBI's recommendations ... is what the OT cybersecurity industry has been advocating for years.”

The Department of Energy released a new framework of best practices for securing clean energy cyber supply chains, including key technologies used to manage and operate electricity, oil and natural gas systems.

The principles outline 10 best cybersecurity practices for suppliers, as well as 10 for consumers, with a focus on risk management, transparency, operational resilience and proactive incident response. 

The Biden administration called out the heightened need for such guidance as the threat of cyberattacks against the energy sector continues to grow from both foreign and domestic actors.

The Energy Department's Office of Cybersecurity, Energy Security, and Emergency Response developed the guidelines with input from energy automation and industrial control system manufacturers, as well as the Idaho National Laboratory, which specializes in cybersecurity research.

The department lists 10 best practice areas for both suppliers and end users. They include priorities such as maintaining vulnerability management processes for suppliers that follow industry best practices, as well as providing product support, including security patches and mitigations throughout the lifecycle of an end user transaction.

For end users, the department encourages the inclusion of contractual language for “those terms, conditions, and testing requirements that will influence your security outcomes,” and working with suppliers to fully understand and integrate appropriate cybersecurity controls and platforms.

The U.S. is not alone in its boosted efforts related to manufacturing cybersecurity — the issue was discussed among leaders at the G7 Summit in Apulia, Italy, earlier this month. Officials there committed to "continue discussions" on how to improve cybersecurity resilience in key sectors, including how to improve supply chain security. 

"As new digital clean energy technologies are integrated, we must ensure they are cyber secure to prevent destruction or disruption in services," National Security Advisor Jake Sullivan said in a June 18 White House statement. "The G7 will work to establish a collective cybersecurity framework for operational technologies for both manufacturers and operators."

The cyber threat to U.S. critical manufacturing is growing. The sector experienced the second highest number of cyberattacks among U.S. industries last year at 218, falling behind only the healthcare sector, according to FBI data. On a global scale, nearly half of critical manufacturers are at risk of a cyberattack, with many organizations lacking visibility into their broader business ecosystems to successfully fend off an attack. 

To combat the heightened risk, the Biden administration has taken an increased interest in fortifying U.S. manufacturing and supply chain security. In November, the administration created the White House Council on Supply Chain Resilience, which was formalized earlier this month by executive order. 

At the agency level, the DOE has been working with energy distributors in recent months to improve cybersecurity. The department created similar "baselines" in February aimed at improving the security of distribution systems and distributed energy resources.

The department also rolled out $30 million in funding in January to fund research, development and demonstration projects focused on improving the cybersecurity of clean energy resources.

Ransomware is “the foremost widespread cybersecurity threat impacting industrial organizations worldwide,” Abdulrahman Alamri, senior adversary hunter at Dragos, wrote in an April blog post analyzing global cybersecurity incidents in the first quarter. But the electric sector was relatively unscathed, notching just a single incident, compared with more than 100 attacks on the manufacturing sector.

Utility Dive reached out to Alamri to ask about the disparity: Is the electric sector better protected than others, and is there a risk of complacency among utilities?

To some extent, the difference in attack numbers has to do with the size of the industries, Alamri said in an email. Since manufacturing is the largest industrial sector “by number of entities,” it reports the highest numbers of incidents, Alamri said. Ransomware groups are “generally opportunistic and financially motivated, aiming their attacks at entities where they perceive the greatest opportunity to achieve their goals,” he said.

Since the start of 2023, Dragos has observed a rise in ransomware attacks “leading to operational disruptions in numerous industrial organizations,” he said. The organization, which monitors the activities of ransomware groups, including their postings on dark web leak sites, is aware of “many instances” where ransomware operators achieved some level of disruption to operational technology when an attack on an information technology environment “prompts an organization to shut down elements of OT environments as a precautionary measure,” he said.

That type of disruption occurred in 2021, when Colonial Pipeline was shut down following a ransomware attack. The ransomware never migrated into the pipeline’s OT environment, but the company shut down operations as a proactive safety measure, leading to disruptions in gasoline and jet fuel deliveries along the East Coast.

The manufacturing, transportation and industrial control systems equipment and engineering sectors accounted for about 90% of first-quarter incidents around the world, according to Dragos. The oil and gas sector experienced eight incidents, or about 4% of attacks. The mining, communications, electric and renewable energy sectors each had two or less attacks, according to the cybersecurity firm.

NERC CIP rules help protect electric sector

The electric power sector has security rules and best practices in place that have helped to create a culture of security, Alamri said.

The Critical Infrastructure Protection standards managed by the North American Electric Reliability Corp. “do not directly address ransomware as a separate risk,” Alamri said. However, “many of the policies, procedures and technologies that organizations have had to implement related to different NERC CIP standards do assist electric sector organizations in preventing their exposure to ransomware.”

In particular, utility personnel are trained on NERC CIP-005, which focuses on electronic security perimeters, and CIP-007, which covers systems security management, Alamri said, noting that those rules “emphasize both network and system security in ways that are commonly described as best practices for preventing malware, specifically ransomware.”

NERC and the Federal Energy Regulatory Commission are evaluating new standards, including CIP-015 on internal network security monitoring and the inclusion of virtualization in many of the revisions to other CIP standards, Alamri added.

“Outside of the existing work processes currently being undertaken by the CIP drafting teams, it is unlikely that additional rules and regulations are necessary at this time other than normal revisions,” he said.

Matt Calligan, director of growth markets at ArmorText, said the utility sector doesn’t necessarily need more rules, but “what’s lacking is clarity on the application of those rules and how the overlapping requirements of the various regulations often put utilities in a Catch-22.”

Calligan pointed to questions regarding whether utilities can use cloud-based systems. NERC’s rules require grid asset owners to have certain control of the devices operating their software, and cloud computing makes that difficult.

“So you have utilities going the forgiveness-versus-permission route so they can utilize the latest cloud-based technologies, and other utilities who have taken a hard ‘no CIP data in the cloud’ stance until things are clarified,” Calligan said. The situation is “forcing many departments into older technologies that might be approaching end of life.” 

For organizations that are not subject to NERC CIP, “additional rules may be warranted,” Alamri said. The National Association of Regulatory Utility Commissioners is working on cybersecurity guidance, and in February it published a set of cybersecurity baselines with the U.S. Department of Energy aimed at improving the security of distribution systems and distributed energy resources.

NARUC has “taken many of the experiences and requirements from NERC CIP and incorporated them into their appropriate guidance documents around cybersecurity,” he said.

Ransomware can “spill over” into OT environments

Ransomware is “certainly one of the most significant and most prolific ongoing threats to utilities/energy organizations,” Alamri said. While ransomware is not typically designed to target an organization's operational technology, ransomware adversaries “are increasingly adopting [industrial control system]-specific process kill lists, demonstrating the ability to stop industrial processes in the OT environment,” he said.

Dragos is aware of a ransomware incident that impacted a U.S. energy company where the attacker “successfully traversed the enterprise into the OT environment,” Alamri said. However, it is possible that “the adversary did not explicitly target the OT environment and that successful navigation was incidental.”

About three-quarters of cyberattacks impacting the OT environment originate on the IT side, said Stellar Cyber founder and Chief Technical Officer Aimei Wei.

“They spill over from IT to OT, so our view here is that people really need one solution that can affect both. ... You gain visibility into both environments, and are able to find and detect that lateral movement from IT to OT,” she said. The attacks are “really difficult to prevent,” so it is important to “catch the early signs.” 

Utilities often “air-gap” OT environments from IT, but this leads to its own set of challenges, Calligan said.

“I know many teams in major utilities who have two different colored screens on their desks, one connected to OT and one connected to IT — this really isn’t scalable, and isolation can breed ossified thinking,” he said. “The blending of OT and IT is inevitable as more and more on the OT side becomes digitally enabled and controlled. This requires dialogue and cooperation between both sides of the house around tools and strategies to create resiliency.”

Utilities must guard against complacency

With relatively few ransomware attacks impacting electric utilities’ operations, relative to other sectors, is there a danger of complacency?

Calligan suggested that treating manufacturing as one large industry for OT cyber activity while breaking other industries into subcategories “could lead to misguided conclusions” and make utilities look more secure than they are.

Complacency and a false sense of security can arise when incidents are infrequent or appear less severe compared with other sectors, Alamri said. “However, given recent high profile cyber attacks to critical infrastructure, we would not call utilities complacent,” he said. “Last year, the Dragos team saw a 104% growth in the number of tabletop exercises we conducted with customers in the electric industry. Ransomware remains the most common scenario selected.”

It would be a mistake for utilities to get complacent on security, particularly heading into the election, said Steve Garrison, senior vice president of marketing for Stellar Cyber. It is good that utilities have avoided most ransomware attempts recently, but “we don't think that short-term trend is all that meaningful.”

Ransomware attackers have also been hitting the healthcare sector recently, which could account for a decline in utility attacks, according to Wei. “Attackers might be focusing their efforts on a particular sector at certain point of time,” she said. 

“There's going to be a lot of interesting things that happened during the election this year,” and cyberattacks are among the top threats utilities are watching for, Garrison said.

FBI Director Christopher Wray said state-linked threat groups are ramping up threat activity against the U.S. and pose a continued risk to key critical infrastructure sectors, in a speech April 9 before the American Bar Association's Standing Committee on Law and National Security. 

Threat actors linked with the People’s Republic of China are continuing to build out offensive capabilities, setting up access to various sectors such as the water, energy and telecommunications industries, according to Wray. 

“We’re seeing hostile nation-states become more aggressive in their efforts to steal our secrets and our innovation, target our critical infrastructure, export their aggression to our shores, and front and center is China,” Wray said.

Wray and other top national security and cybersecurity officials testified before the House Select Committee on the Chinese Communist Party in January about Volt Typhoon, a China-linked threat actor that has been working to embed itself within key critical infrastructure sectors. 

The threat actor has been working to potentially launch a diversionary attack against the U.S. in case of military action breaking out in the Asia-Pacific region, Wray testified. 

Just last month, the Five Eyes intelligence partners issued an advisory urging critical infrastructure leaders to take the threat from China seriously. 

Russia-affiliated threat actors are also targeting critical infrastructure in the U.S. and elsewhere around the world, focused on underwater cables and industrial control systems, Wray said. 

Wray noted that Section 702 of the Foreign Intelligence Surveillance Act  — which is scheduled to expire in a few weeks without Congressional action — has been critical in combating cyber threats. 

FISA Section 702 queries allow the U.S to conduct surveillance of foreign intelligence adversaries that are using electronic communications. These targets often include terrorists, spies and hackers in many cases.

Wray said 702 allowed the FBI to alert 300 victims in the U.S. and abroad using the ability to conduct U.S. person queries. He also cited the use of 702 to help the FBI discover compromised network infrastructure at a transportation hub in the U.S.

However, fierce opposition remains over the surveillance authority and how it can be used to sweep up Americans. 

U.S. and Canadian electric grids face a growing threat from hackers and physical attacks, and greater communication, coordination and advance planning are required to counter them, officials at the North American Electric Reliability Corp. said April 4.

“The current geopolitical situation has significant ramifications for the North American grid,” said Manny Cancel, senior vice president of NERC and CEO of the Electricity Information Sharing and Analysis Center.

NERC on April 2 issued its “GridEx VII Lessons Learned Report,” laying out recommendations based on a biennial grid attack exercise hosted in November. More than 250 organizations and 15,000 participants took part in the exercise, officials said. 

“In addition to the ongoing war in Ukraine, the Israel-Hamas conflict, which began last October, is also a cause for concern,” Cancel said during a call with media to discuss the report and threat environment. “This turmoil has contributed to a dramatic increase in malicious cyber activity, including new versions of malware and ransomware that constantly pressure operational and information technology networks.”

China, Russia, Iran and North Korea have demonstrated advanced cyber capabilities, Cancel said. And there are risks associated with software and supply chain vulnerabilities. Earlier this year, the Cybersecurity and Infrastructure Security Agency confirmed that state-sponsored threat actor Volt Typhoon had compromised the IT environments of multiple critical infrastructure providers in the U.S. 

“All these vulnerabilities and threats just further emphasize the need for the industry to remain vigilant and ensure good internal network monitoring is in place, especially on critical OT systems,” Cancel said.

There were also 2,800 physical grid security events reported to the E-ISAC in 2023, Cancel said. About 3% of them led to grid impacts, including outages or the need for operational contingencies. The most serious impacts were the result of firearms attacks, thefts, tampering and vandalism.

The E-ISAC has previously seen a correlation between malicious activity targeting the power sector and elections, “so we anticipate that there could be an opportunity for an increase. As we know, activists continue to use this as a vehicle to get their ideology and other political thoughts across,” Cancel said.

Against a growing threat backdrop, the GridEx simulated attack allows utilities the opportunity to gauge their responses, communications protocols and cross-sector coordination. NERC’s assessment of the November exercise revealed the opportunity for greater cooperation and communication, in particular between utilities and non-federal government partners.

There was an “overall reduction in the number of government entities that participated” in the simulated attack, according to the after-action report. “Municipal government entities, such as city, town, and county governments, as well as state energy offices would benefit from greater involvement in emergency response planning, training, and exercises.”

Organizations are also still identifying best practices for communications in a hybrid work environment, the report said. The simulated attack “helped participating organizations identify challenges with hybrid response and interoperable communications with internal and external response partners.”

The electric power sector must also evaluate options to manage grid reliability impacts when power market systems may be unavailable for a lengthy time, the report found.

“Market operators and participants should review their market rules to ensure a common understanding of how generation dispatch and financial settlements would be administered through an extended period of market system or data unavailability,” the report said.

Cancel said the E-ISAC is developing an action plan to track progress made addressing each recommendation, and to update industry and government partners on that progress. Some GridEx participants have already begun implementing the report’s recommendations, he said. 

“Continued coordination across the industry helps ensure our vigilance and allows us to respond quickly, should the need arise,” he said.

Cybersecurity requirements and questions for vendors should be included in utilities’ procurement processes, state regulators and the U.S. Department of Energy recommended Feb. 22 in a set of “cybersecurity baselines” aimed at improving the security of distribution systems and distributed energy resources.

The National Association of Regulatory Utility Commissioners and DOE’s Office of Cybersecurity, Energy Security, and Emergency Response developed the baselines together and are planning to publish implementation guidelines later this year.

The baselines are intended to be a resource for state public utility commissions, utilities and DER operators and aggregators and “provide a common starting point for cyber risk reduction activities,” said NARUC Executive Director Greg White.

The electric grid is increasingly distributed and cybersecurity “is an integral underpinning of power system resilience,” DOE and NARUC said.

“Addressing cybersecurity is essential as electric distribution systems continue to evolve, spurred by new technologies and operational models, as well as the ever-increasing threat of cyberattacks,” White said. The baselines are “tailored for electric distribution systems and the DER that connect to them.”

NARUC and DOE will host a pair of webinars to discuss the baselines, on March 14 and March 19. The recommendations include:

• As new devices or services are procured, utilities should make a “good-faith effort to negotiate procurement documents and contracts” stipulating information sharing requirements around security incidents;
• Utilities should develop policies requiring a minimum password length of 15 or more characters “for in-scope IT and OT assets that are not otherwise protected” behind multifactor authentication or other authentication mechanism;
• Departing employees should lose access to physical and online critical resources within 24 hours of a separation;
• Split up IT and OT networks and utilize “an appropriate network security device to enforce a deny-by-default policy on communications between networks,” only permitting explicitly-allowed connections; and,
• Multi-factor authentication should be required for remote access to assets “using the strongest available method for that asset.”

NARUC and DOE are now launching the second phase of the cybersecurity baselines initiative, to develop implementation strategies and adoption guidelines which they said will include “recommendations for prioritizing assets to which the cybersecurity baselines might apply, as well as prioritizing the order in which the baselines might be implemented based on cyber risk assessments.”

More than 250 organizations participated this week in a simulated attack on the North American electric grid, known as GridEx, in order to gauge utility responses, communications protocols and cross-sector coordination, should an actual disruption occur. While risks are rising, some lessons from the COVID-19 pandemic can help make the grid more secure, experts said. 

The exercise is held every two years by the North American Electric Reliability Corp.’s Electricity Information Sharing and Analysis Center, or E-ISAC. Officials said this year’s scenario drew from real-world events and mimicked supply chain challenges, loss of access to utility buildings due to physical threats, and the inclusion of a nation-state attacker.

NERC has also worked to expand the role of the vendor community in GridEx, including cybersecurity firms and equipment manufacturers. Including vendors in the exercise allows utilities “to review and examine all of the relationships, processes and tools they have available to respond to a real-life event,” said Katherine Ledesma, head of public policy and government affairs for cybersecurity firm Dragos.

Grid officials are not aware of “specific, credible threats” that could impact the power system, but the landscape is “dynamic, and continues to present challenges that are increasingly difficult to detect and defend against,” said Manny Cancel, NERC senior vice president and head of the E-ISAC.

“The electricity sector is under constant attack from nation-states and organized criminals,” Cancel said in a call with reporters, pointing in particular to China, Russia, Iran and North Korea.

“Geopolitical turmoil has contributed to a dramatic uptick in malicious cyber activity, including new iterations of malware and ransomware that are constantly pressuring information technology and operational technology platforms,” he said.

Utility security posture “has never been more important,” said Pedro Pizarro, president and CEO of Edison International, the parent company of Southern California Edison. “We see critical infrastructure targeted in conflicts overseas. That heightens our risks in North America.” 

Cross-sector and federal partnerships are essential to Edison’s cybersecurity approach, said Pizarro, who is also co-chair of the Electricity Subsector Coordinating Council. “One of the essential components of our strategy is planning and preparing to respond to major incidents and helping all stakeholders to understand their roles and responsibilities,” he said.

The electric power sector has also seen a significant increase in serious physical security incidents. “While most of these incidents do not result in grid impacts, incidents continue to increase and we expect this trend to continue,” Cancel said.

To help utilities prepare for these threats, the GridEx simulation brings in a wide range of participants. The event consists of a two-day simulated attack to test utilities’ responses, followed by an executive tabletop exercise which focuses on policy-level issues. GridEx participants include the electricity, natural gas, telecommunications and finance sectors. And NERC launched a vendor affiliate program last year to bring equipment manufacturers and security firms into the conversation.

Dragos participated in GridEx this week, and Ledesma said she is “encouraged to see owners and operators, as well as government partners, include vendors and cybersecurity service providers” in the exercise. Vendor expertise, intelligence and capabilities are “vital during an incident, and so it’s beneficial to include them in an exercise under blue sky conditions as well,” she said.

The GridEx attack scenarios are “authentic and reflect the contemporary threat landscape,” Ledesma said. “GridEx also provides the opportunity for organizations to coordinate regionally and across organizations on issues affecting interconnected generation, transmission and distribution systems.”

Utilities’ ability to respond to a cyberattack has been bolstered by the COVID-19 pandemic and the transition to work from home, said Duane Highley, CEO of Tri-State Generation and Transmission Association and ESCC co-chair. “The ability to work remotely has really been enhanced,” he said.

“Part of our simulation was simulating the loss for a building because of a threat at the building location. And in prior years, that might have been a lot more of a scramble than it is now since we just went through two years of working remotely,” Highley said. “Establishing remote work for a workforce was a much simpler task today than it would have been pre-pandemic.”

This was the seventh iteration of GridEx. The E-ISAC plans to publish an assessment of the exercise by the end of the first quarter of 2024.

Cloud technologies could provide significant cost, security and reliability benefits to the U.S. electric grid but critical infrastructure rules do not allow them to be used for certain larger assets, multiple speakers said Nov. 9 at the Federal Energy Regulatory Commission’s annual reliability conference.

The Critical Infrastructure Protection rules, or CIP, are managed by the North American Electric Reliability Corp. and currently require grid asset owners to have certain control or knowledge of the devices operating their software. Cloud computing makes that difficult or impossible, experts agreed, in particular for what are known as high- or medium-impact grid assets.

Current NERC standards “do not provide clear guidance” on how regulated entities can implement new technologies that may not have been envisioned by the current CIP rules, Joseph Mosher, portfolio manager at EDF Renewables, told the commission. “Attempts to incorporate newer technology into the NERC CIP standards can be painful and time consuming,” he said.

Experts expressed concerns over the outdated CIP rules, at a time when grid officials say they face growing threats.

“One can definitely make the argument that the grid is less secure today than it would be” if cloud computing solutions were allowed, “and that gap is growing every day,” security consultant Tom Alrich said. “This is the biggest problem with NERC CIP today.”

A related problem — that important information about those systems can't today be stored in the cloud — will be fixed beginning next year when two revised CIP standards come into effect, he said.

A sector under attack

The cyber threat to the electric power sector is growing, and grid officials say they must utilize new tools to counter it.

“The electricity sector is under constant attack by nation states and organized criminals. We see billions of attempts a day to survey our networks, identify vulnerabilities or gaps in protection, steal credentials or data, or exact a ransom,” Manny Cancel, senior vice president and CEO of the Electricity Information Sharing and Analysis Center, told regulators Nov. 9. The E-ISAC is operated by NERC.

China is the biggest cyber threat to the U.S. grid, Cancel said, followed by Russia.

“China will continue to target critical infrastructure here in the U.S., to grow their knowledge and identify access points for future use,” he said. Hackers are constantly looking to exploit vulnerabilities in enterprise software platforms “that are used pervasively across the electricity industry,” he added.

The database of cyber vulnerabilities maintained by the National Institutes of Standards and Technology is on pace to report over 27,000 vulnerabilities in 2023, Cancel said, representing a 25% increase over 2022, and "many of them are critical.”

Those vulnerabilities can combine with the rapid expansion of grid-connected assets to create a threat-rich environment, experts agreed.

The rate at which smaller generators, such as solar, wind and batteries, are connecting to the grid is multiplying not only the number of generators but also the vendors necessary to support the infrastructure, Mosher added. “In many cases” these new vendors may not fully support on-premise solutions “and instead require a full or partial cloud implementation, potentially making the generator owner noncompliant with NERC CIP,” he said.

‘These tools cannot be used’

Some of ISO New England’s most robust security tools are not allowed to protect certain electric system assets because they use cloud technology, said Rudolf Pawul, vice president of information and cyber security services for the grid operator.

“Traditional log monitoring and malware detection is insufficient in the face of modern threats,” Pawul said. The ISO “has augmented its traditional security tools with learning-enabled endpoint detection, backed by world class threat hunters, and similar tools for application performance monitoring that aids with anomaly detection. Unfortunately, these tools cannot be used for several ISO New England systems, including some of the most important ones.”

ISO New England has submitted a standards authorization request to NERC “with the hope that the standards can be revised to allow the use of cloud-based services,” Pawul said. “Despite the broad support and a positive reception from both NERC and [the Northeast Power Coordinating Council] the CIP revision process will take years.”

Pawul also said the process of integrating new technologies — not just cloud computing — should be reviewed.

“Whether it is artificial intelligence, quantum computing or the next disruptive technology, the industry will again be at a disadvantage unless it adapts. ... the energy industry must assess the ramifications of the CIP standards on disruptive technology sooner [and] reduce the time spent to make revisions, or create a process for compliance exceptions within the CIP reliability standards,” Pawul said.

The need for cloud computing solutions on the power grid extends beyond security, experts said.

Decarbonization will require cloud computing

Cloud technologies “are key in augmenting security teams’ abilities, with increased visualization, automation and resilience,” Maggy Powell, security assurance principal for the power and utility sector at Amazon Web Services, told the commission. But beyond that, the operational demands of decarbonization, decentralization and digitalization “translate to a more than 100 times increase in data volumes to reliably operate the grid, making computing capacity essential,” she said.

Cloud computing can offer utilities opportunities to gather instructive information from the new data “and create value for applications such as predictive maintenance, outage management, power flow analysis and other operational applications,” Powell said.

The last complete update of NERC CIP standards was finished in 2017, and that endeavor took more than eight years, Alrich said. Because of the critical nature of cloud computing, he said a solution will likely be developed much sooner. 

“This is one of those cases where the vendors’ interests and the users’ interests and the regulators’ interests are all the same,” Alrich said. “There's got to be a way this can be done sooner than eight or nine years ... Pretty soon there's not going to be any software to run on premises anymore.”