The North American Electric Reliability Corp. on May 15 issued its highest alert level ever, urging generators and transmission owners to take measures to prepare for winter.

In its first Level 3 Essential Actions alert, NERC cited three “extreme winter weather events between 2018 and 2022. Severely cold winter weather knocked power plants offline in 2018 in the South Central U.S., in 2021 in Winter Storm Uri and last year in Winter Storm Elliott, leading to power shortages and outages, NERC said.

“When cold weather events such as Winter Storm Uri occur, system operators may need to shed firm customer load to prevent uncontrolled load shedding and cascading outages which may not only result in major disruption but also have very real human consequences,” the grid watchdog organization said.

The alert asks power plant owners to calculate the “extreme cold weather temperature,” or ECWT, for their generating units and include that number — the lowest 0.2 percentile of the hourly temperatures measured in December, January and February since 2000 — in their winter preparedness plans.

Generators should identify which units need additional freeze protection measures to operate at the ECWT and put them in place before the winter season if possible. Power plant owners also should identify critical components of their facilities and the steps they took to protect them from winter weather, according to the alert.

The alert calls on transmission operators to update their operating plan to include provisions such as minimizing the overlap of circuits that are designated for manual load shed and circuits that serve designated critical loads.

The operating plans should also include provisions for manual load shedding that can occur quickly enough to ease an emergency, according to the alert.

Balancing authorities, which maintain electric supply and demand in their areas, should update their operating plans to manage generating resources in their area to address various issues, including fuel supply and inventory concerns, and fuel switching capabilities, according to the alert.

Organizations won’t be fined for failing to implement the alert’s recommended actions, according to NERC.

The electric system is becoming more reliant on variable energy resources, such as wind and solar, and on natural gas, according to NERC.

“Extreme winter weather events have stressed the supply of traditional fuels and the dependability of new resources,” NERC said. “Preparation of resources for operation during extreme winter weather and situational awareness in both planning and operations by applicable registered entities is necessary for optimal reliability.”

The PJM Interconnection aims to offer incentives for “black start” generators that have assured fuel supplies, while allowing wind, solar and energy storage to qualify for that category of power plant, according to a proposal filed May 12 with the Federal Energy Regulatory Commission.

Black start units are used to re-energize the grid during widespread blackouts, but PJM doesn’t require them to have secure fuel supplies, according to the grid operator.

During widespread outages on the bulk electric system, grid operators restore power by running black start units, building “cranking paths” to other key power plants, forming “islands” of restored service, which are synchronized, and connecting to neighbors, PJM said.

Nearly half of PJM’s black start units lack fuel assurance and the distribution of fuel-assured units is spread unevenly, potentially extending outages on the bulk electric system, the grid operator said.

Power plant fuel supply could be disrupted by natural gas pipeline outages, equipment failures at compressor stations or cyber attacks on pipeline systems, PJM said.

“Fuel supply availability could be a very real concern during the type of severe and widespread disruption to the [bulk electric system] that triggers a need for black start generation,” PJM said.

PJM proposed establishing Fuel Assured Black Start Units as a new category of black start resources. They would need to meet minimum commitment periods, performance requirements and testing requirements.

The black start units must be able to run for 16 hours without grid-supplied power.

Eligible gas-fired power plants without on-site storage would have to connect to two interstate pipelines or have a direct connection to a gas-gathering system.

The proposal helps address a need for additional diversity in the types of resources that can qualify as a black start unit by establishing qualification rules for wind, solar, run-of-river hydro and battery storage resources, PJM said.

As an incentive, black start units that recover their costs through base formula rates would receive 10% more revenue.

PJM intends to require one fuel-assured black start unit in each of its 21 transmission zones, supplemented by additional “high impact” units that are needed to quickly restore power, according to the filing.

PJM asked FERC that its proposal take effect by July 11 so it can issue a request for proposals for black start resources this summer. The proposal was widely supported by PJM’s stakeholders, the grid operator said.

Ukraine’s power grid reached a major milestone this week, resuming electricity exports to Europe following months of missile and drone attacks by Russia. 

From Oct. 2022 to Feb. 2023 Russia launched almost 200 missiles and 46 drones at the nation’s power grid, according to Volodymyr Kudrytskyi, CEO of Ukraine's national power company and transmission operator, Ukrenergo.

Kudrytskyi on April 11 remotely addressed the MIT Energy Forum. During those four months, he said Ukraine lost approximately 10 GW of generation and 43% of its high voltage transmission network was destroyed. The operator was forced to impose rolling blackouts, keeping up to 12 million people without power at any given time.

“There was no large thermal or hydro power plant in Ukraine that was not either damaged or destroyed,” Kudrytskyi said.

Despite the destruction, the country on April 11 resumed energy exports to Moldova and is expected to begin exporting to Poland on April 12. How did the embattled nation’s electric grid survive the Russian attacks, and are there lessons for grid operators in the United States?

“When we finish this war, we will be happy to share our know-how with different countries,” Kudrytskyi said.

Preparing for the invasion

Russia invaded Ukraine on Feb. 24, 2022, but preparations had been in the works for years to separate the two nation’s electric grids, Kudrytskyi said.

Efforts to synchronize Ukraine's grid with Europe's began in 2017, and there was about a year of more work needed for the transition when Russia invaded. Those efforts were accelerated and Ukraine connected to the EU grid in March 2022, allowing the country to maintain service to some customers even as Russian attacks disabled other parts of its system.

Also important, ahead of the invasion the grid operator built up a supply of replacement equipment and “secretly prepared a reserve control room two weeks before the invasion,” Kudrytskyi said.

Anticipating the invasion, the grid operator “worked out, together with distribution system operators, emergency schemes for power to be supplied to every region of Ukraine. We imagined different scenarios of disconnections of different power lines and shelling of different substations, to model how we would deliver power to a specific region in a situation when some transmission elements would be off,” he said.

Sandbags were used to shore up critical substations, he said.

“We had to cover the most critical elements of our substations with engineering defenses,” he said, including autotransformers, relay protection equipment and other assets.

Speeding recovery

Despite physical defenses, Russian strikes damaged or destroyed scores of substations and power plants, forcing the grid operator to improvise.

“We invented a technique that allowed us to secure and preserve the system balance even in the course of these attacks,” Kudrytskyi said. Experts say this could include islanding procedures that prevent a grid collapse.

Last November, Russian attacks successfully destabilized the Ukranian grid, disconnecting it from the EU. In the wake of that attack, the grid operator developed a procedure “that allows us to maintain the balance of the power grid and prevents it from falling apart,” he said.

The technique “proved to be very efficient,” Kudrytskyi said, though he declined to elaborate. “We survived probably eight or nine attacks after November, and there were no instances when our system was unstable,” he said.

“I suspect they have developed islanding procedures where the grid can be (or has been) separated into a number of independent islands,” Kevin Perry, formerly the director of critical infrastructure protection at Southwest Power Pool, said in an email.

Perry said the key to islanding a portion of the grid is to have sufficient generation to meet the load in each segment, with load shedding that maintains the balance. “Each island, separated from the rest of the grid, cannot influence what happens outside of the island, hence any disruption will be contained and limited to just the affected island,” he said.

Ukraine’s grid workers also had to adapt, said Kudrytskyi, with a goal of restoring power three to four times as fast as in the past.

If a transformer might have taken two to three months to install before the invasion, “we managed to do it sometimes in one week, sometimes in three, four days, depending on the scale of destruction,” Kudrytskyi said. Ukrenergo had hundreds of people working  in frontline regions with a dedicated stock of materials, and working closely with regional power administrations and distribution system operators, he said.

Sometime in February the Russians “seemed to exhaust their heavy missile fleet” that had been targeting Ukraine’s energy infrastructure, said Kudrytskyi, “And the good news is that this infrastructure, this power system, survived and is still operating.”

Looking ahead

Kudrytskyi is optimistic about the future of Ukraine’s power system. The grid modernization work, even for a nation under siege, looks similar to other parts of the world.

”We have plans to phase out coal-fired generation in the next five to 10 years, and to replace it with clean, sustainable, but also efficient sources of power,” he said. “And I'm sure that after our victory, we will not be alone in this process.”

Ukraine will also lean on its nuclear fleet to reduce carbon emissions, he said.

From a security perspective, “we are now finalizing the design of protected substations,” Kudrytskyi said, and could be the first in the world to implement new defenses.

Perry said he is “intrigued” by Ukrenergo claims to have devised a way to protect substations from physical attacks, including missiles. “I am envisioning them enclosing the equipment in some sort of blast-hardened structure, or burying the equipment underground,” he said. “Either way, they will have to overcome the heat dissipation challenges.”

It remains unclear whether any new techniques would be applicable in North America, he said. The U.S. has faced a rise in physical attacks on its energy infrastructure, and federal regulators are mulling tighter security rules.

There were almost 1,700 physical grid security incidents reported in 2022, according to the North American Electric Reliability Corp., up 10.5% from 2021, though only a fraction cause outages or other grid impacts. Most of the incidents are categorized as gunfire, ballistic damage, intrusion or tampering and vandalism.

Kudrytskyi stressed the importance of stockpiling equipment, particularly high voltage transformers. 

In North America, there is an industry-led initiative to stockpile long-lead-time transmission equipment led by Grid Assurance. In December, officials said the time to procure equipment has grown to 20 to 39 months from 16 to 20 months the year before. Meanwhile costs have risen 20% to 50%.

”We did not manage to find a lot” of high voltage transformers, Kudrytskyi said. “It’s obvious no one is ready. ... Transmission system operators all around the world should think about creating a strategic stock of this equipment, autotransforms, relay protection terminals [and] some other critical elements of the substations which will possibly be targets.”

The physical security rule in place now for critical substations “appropriately focuses limited industry resources” and should not be expanded to a broader set of assets on the bulk power system, or BPS, the North American Electric Reliability Corp. concluded in a report published April 14. 

However, NERC also found a need to “evaluate additional reliability, resiliency and security measures” following a rise in substation attacks last year. The reliability organization and the Federal Energy Regulatory Commission are planning a technical conference “to further study appropriate levels of physical protections.”

NERC’s physical security reliability standard, known as CIP-014, will likely eventually be modified, according to Mike Hamilton, chief information security officer of Critical Insight, a cybersecurity firm. But that process could be lengthy and “the gist of this announcement is that operators must conduct physical risk assessments of substations. Now,” he said.

The scope of the North American grid is extensive, with tens of thousands of substations in remote areas. NERC’s physical security rules focus on assets that are critical to the reliable operation of the BPS.

The criteria for a substation to fall under the CIP-014 rule is “broad enough” to capture critical facilities, the report said. “NERC did not find evidence that an expansion of the applicability criteria would identify additional substations that would qualify as ‘critical’ substations.”

While NERC stopped short of recommending a broad expansion of its physical security rule, it did acknowledge the growing threat and need for additional study and action.

“Given the increase in physical security attacks on BPS substations, there is a need to evaluate additional reliability, resiliency, and security measures designed to mitigate the risks associated with those physical security attacks,” NERC said.

Nearly 1,700 physical security incidents were reported to the Electricity Information Sharing and Analysis Center in 2022, up 10.5% from 2021. The majority of reported incidents were categorized as gunfire, ballistic damage, intrusion or tampering and vandalism.

Supplementary data “could show that additional substation configurations would warrant assessment” under the rule, NERC said.

NERC said it would work with FERC staff to hold a technical conference to “identify the type of substation configurations that should be studied to determine whether any additional substations should be included in the applicability criteria.” 

FERC called for NERC to consider expanding CIP-014 applicability in December, following a North Carolina firearms attack that knocked power out to about 45,000 Duke Energy customers. Later that month, multiple substations in Washington were damaged, leading to more than 14,000 outages on the Tacoma Power and Puget Sound Energy systems. 

But NERC’s report concluded establishing a “uniform, bright line set of minimum physical security protections” for all BPS substations and associated primary controls centers “is unlikely to be an effective approach to mitigating physical security risks and their potential impacts” because it “fails to provide for a methodical approach necessary to address site-specific threats or objectives.”

“NERC finds that this more holistic approach will provide greater long-term flexibility and minimize the impacts of physical attacks on BPS reliability,” the report concluded. NERC is recommending “further evaluation of the appropriate combination of reliability, resiliency and security measures that would be effective in helping to mitigate the impact of physical security attacks.”

Critical Insight’s Hamilton said he found NERC’s recommendation against expanding CIP-014 “curious,” but that new security rules were likely coming — as they have for other critical sectors. The health, aviation, pipeline and water sectors have all received new requirements within the last six weeks, he said.

CIP-014 “will likely be modified to reflect the increased scrutiny of substation physical security and document potential compensating controls,” he said. But waiting for a rulemaking and comment period “is not sufficiently rapid to counter a known and prevalent threat.”

“NERC is playing ‘catch up’ to address recent physical attacks and not waiting for the bureaucracy,” Hamilton said.

The National Renewable Energy Laboratory will test the ability of real-time meter data to coordinate rapid energy restoration in remote communities in a $3 million partnership with Copper Labs, a Boulder, Colorado-based manufacturer of real-time meter data collection technology.

The partnership will first identify and define community needs. It will eventually see the creation of a plan to automate power restoration during outages and the development of a virtual emergency operations center. The location for the project has not yet been announced but will be in either Colorado or New Mexico, according to Copper Labs.

Real-time meter data will become increasingly important to utilities and customers as the percentage of solar and other renewable resources on the grid grows, Copper Labs CEO Dan Forman said, because it will help align demand with the availability of intermittent resources.

A new cooperative project funded by the Department of Energy is set to find out whether modern interventions could help remote communities get the lights back on more quickly after power outages.

NREL, Copper Labs and other partners will install and use sensors and advanced distribution management systems in a trial project to test whether the equipment can improve community resilience, according to Fei Ding, the project’s principal investigator from NREL. According to Copper Labs’, the company’s data-reading hardware will be installed at some 20 sites at an as-yet unannounced community in the Holy Cross Energy, Connexus or United Power service territories. The hardware will be used to provide greater behind-the-meter visibility, Ding said.

Copper Labs makes data collectors that can read gas, water and electric meters every 30 seconds. This gives electric utilities more immediate insight into who is experiencing a power outage, and what resources might be available to help restore power, Forman said.

“In outlying areas, the faster you can get them online, the better it is for health, safety and comfort,” he said.

As larger percentages of solar and other renewable resources dominate the grid, Forman said, access to real-time data will become more important for avoiding outages and maintaining reliability.

Real-time meter data could also help communities take greater advantage of demand-side management programs, he said. Case studies by the company have found that making real-time data available to customers results in greater customer engagement and has twice the impact on peak load as demand management programs that lacked a real-time data web portal.

Although the NREL project will focus on installing hardware at customers’ homes, Copper Labs also has a version of its device that can be attached to a fixed point such as a utility pole to read the meters from an entire neighborhood. The device can use cellular data, Ethernet or fiber connections. This eliminates the need for wi-fi, which is often a barrier to smart meter devices and programs in remote and rural communities, Forman said.

The Federal Energy Regulatory Commission on Feb. 16 approved two extreme cold weather reliability standards for U.S. generators in response to Winter Storm Uri in 2021, when about 4.5 million Texans lost power because power plants were unable to run.

FERC approved the standards proposed by the North American Electric Reliability Corp. even though they included “undefined terms, broad limitations, exceptions and exemptions and prolonged compliance periods.” FERC ordered NERC to address those issues within a year.

FERC also directed NERC to monitor and assess power plant owners’ implementation of the new requirements. NERC is preparing to propose a second set of cold weather standards late this year to address remaining recommendations from the FERC-NERC inquiry into Winter Storm Uri.

The just-approved reliability standards come less than two months after widespread power plant outages during Winter Storm Elliott, including outages by Duke Energy and the Tennessee Valley Authority in the Southeast.

The Midcontinent Independent System Operator reported it had 50,000 MW in unplanned power plant outages during the late-December cold snap. About 45,950 MW, or 23% of PJM’s generating fleet, was unexpectedly offline on Dec. 24 while ISO New England had about 2,275 MW in unplanned outages that day.

One standard addresses emergency operations. FERC said it will improve how transmission operators account for the overlap of manual load shed and automatic load shed in their emergency operating plans while addressing the need to minimize the use of manual load shed that could make emergencies worse and threaten system reliability.

The other standard covers extreme cold weather preparedness and operations. The standard requires generator owners to put in place freeze protection measures, develop enhanced cold weather preparedness plans, conduct annual training and implement corrective action plans to address freezing issues.

Generators must also provide certain cold weather operating parameters to reliability coordinators, transmission operators and balancing authorities for their analysis and planning, FERC said.

“We believe that these measures begin to address many of the issues identified as contributing to generating unit failures during extreme cold weather conditions,” FERC said.

NERC’s proposed weatherization requirements were “not up to the task,” FERC Commissioner Allison Clements said during the agency’s monthly meeting, noting the harm caused during Winter Storm Uri, with at least 210 deaths in Texas and up to $130 billion in losses there.

The proposal, for example, requires power plant owners to weatherize generators so they can operate for one hour in extreme cold temperatures beginning in April 2027, she noted.

“Yes – one hour – in 2027, over four years from now,” Clements said. “Needless to say, that doesn’t bring comfort that we can get through a multi-day winter event like Winter Storm Uri, and waiting for four additional winters before weatherization requirements kick in does not reflect the urgency we feel.”

FERC said the one-hour requirement was too short and ordered NERC to extend it, but was not specific.

FERC rejected arguments by the Electric Power Supply Association, the PJM Power Providers Group and Invenergy that the agency and NERC lack the authority to set some of the requirements in the standards.

Also, FERC said concerns about how generators would recover their costs in meeting the standards was outside the scope of the proceeding.

Power systems across the U.S. faced challenging grid reliability conditions over the past year but managed to avoid the worst-case scenario of prolonged outages thanks to a combination of policy measures implemented over the last couple of years and luck, experts say. 

All over the country, a changing climate and extreme weather events – whether due to high temperatures, low temperatures or storms and hurricanes – are posing a threat to grid reliability, according to Eric Gimon, senior fellow with Energy Innovation. Weather patterns are changing from what the U.S. historically experienced and are going to continue to get more extreme, he said, and “what seems unusual today will seem more normal tomorrow. This is a one-way street right now.”

In Texas, the Electric Reliability Council of Texas grid experienced extreme temperatures and peak loads but has so far avoided widespread blackouts in 2022; California underwent a heat wave in September and, despite very tight power supply conditions, also managed to prevent large-scale outages. The Pacific Northwest, meanwhile, experienced high temperatures this year, although not as severe as during the heat wave it underwent in 2021, and it also avoided prolonged outages.

Power system stakeholders are grappling with these issues and doing what they can in a relatively short period to prepare the grid for changing weather and demand patterns, according to Arne Olson, senior partner with Energy and Environmental Economics. 

In general, the grid fared better in 2022 than the previous year or two, which experts attribute in part to luck — that is, relatively less challenging weather conditions — as well as a host of measures implemented by grid operators and planners, like mobilizing demand response and bringing online new battery storage in California, which, “if they hadn’t done those things, we would not have got through that situation without a blackout,” said Olson.

Looking to the future, however, experts say that different regions of the country will need to invest in transmission and other grid infrastructure, build more power resources and demand response capabilities, and work in a more organized fashion with each other to make sure they’re able to keep the lights on. 

Utility Dive took a closer look at three parts of the country — California, Texas and the Pacific Northwest — that are facing grid reliability challenges to see how they fared in 2022 and provide an outlook for 2023 and beyond.

‘2022 showed the value of the changes that have been made’ in California

In September 2022, the Western U.S. was hit with a 10-day heat wave that broke temperature records across the region, including a 116-degree high in downtown Sacramento on Sept. 6. However, the California Independent System Operator did not have to initiate rolling outages, which it attributed to increased battery storage on the grid as well as the expanded use of demand response.

Roughly 3.5 GW of storage has been brought online in California since mid-2020, and calls for conservation on Sept. 6 — the day that the grid experienced a record-breaking peak load of about 52 GW — prompted energy use to drop by roughly 1.5 GW. 

The grid operator has also pointed to market enhancements, like tweaking resource sufficiency evaluations, that helped avert the need for rolling outages. 

“I think if we compare it with 2020,” when CAISO did have to initiate outages amid another record-breaking heat wave, “2022 showed the value of the changes that have been made and the investments that have been made since…” said Severin Borenstein, professor at the Haas School of Business at the University of California, Berkeley, and member of CAISO’s governing board. 

“The battery and other capacity installation showed their value, as did the coordination with other areas, and sorting out some of the issues about imports and exports,” Borenstein added. “So I think the good news is it showed that we’ve made a lot of progress — and the bad news is we still came very, very close to rotating outages.”

The outcome of the 2022 heat wave in the Western U.S. indicated that while a lot of reliability-related improvements were tiny, marginal gains over what was in place in the past year, “the aggregation of all those little improvements made a big difference,” agreed Ben Kujala, senior research lead at Dunsky Energy + Climate Advisors.

Following the August 2020 blackouts, California regulators rolled out a series of demand-side and supply-side measures to bolster the electric grid. The state has made progress on the supply side, Borenstein said, with the influx of batteries being the best example. At the same time, California still has a lot of work to do in bringing more resources, especially a lot more storage, online. And in the medium run, the events of September reinforced the need to keep current plants online — whether the 2.2 GW Diablo Canyon nuclear plant or thermal facilities, he added, noting that “we really have to be cognizant of how tight this grid is when we are talking about trying to phase out thermal plants quickly.”

On the demand side, there has been a sea change in California’s attitude toward demand response over the last several years, said Cisco deVries, CEO of OhmConnect.

“I don’t think there is any question that demand response became a very important component of how the state responds to extreme weather, and I don’t think there’s any question at all that over Labor Day weekend this year, demand response saved the day,” he added. 

On Sept. 6, Pacific Gas & Electric fulfilled a record peak demand of 22,672 MW, utility spokesperson Paul Doherty said in an email. At the same time, roughly 1.2 GW of new battery storage capacity has come online in PG&E’s portfolio since the August 2020 outages, and the utility also procured additional emergency reliability energy supply, he added.   

The utility also modified and expanded demand response efforts this summer, Doherty said. More than 1.5 million customers are enrolled in a voluntary program that rewards them to temporarily reduce electricity use during high demand periods, and other customers participated in its SmartAC program, which cycles air conditioners on and off every 15 minutes for up to six hours. In addition, thousands of PG&E customers with a Tesla Powerwall battery participated in a virtual power plant that discharged back to the grid during times of high demand.

"We were better prepared to meet peak loads this summer, in part due to significant new supply that has been brought online. However, it was Californians and PG&E customers who heeded the call for conservation during the historic September heat wave and averted the need for rotating outages," Doherty said.

In the Pacific Northwest, ‘we need to be prepared for more heat waves’

In 2021, the Pacific Northwest underwent a heat wave that led to blackouts caused by strains on the region’s electric infrastructure and served as a “wake-up call” for the utility sector, experts said.

This summer, the region faced high temperatures again, with around 11 million people under excessive heat warnings and another 12 million under heat advisories at some point, the Washington Post reported July 29. 

While this year’s temperatures were nowhere as severe as the 2021 heat wave, the fact that another such event occurred so soon definitely cements the notion that expected temperatures are changing, and “we need to be prepared for more heat waves — more frequent heat waves and more extreme heat waves — even in the northerly parts of the country that certainly didn’t have them,” Olson said. 

For utilities, this presents at least two issues.

“When they do their power planning… they need to anticipate the changed weather,” Olson said. Historically, this meant planning the system around weather variability over many years. “But the past weather is no longer a very good guide for the future, so they need to really start factoring in changes to their weather over time.” 

Utilities will also need to ensure that their guidelines for transmission and distribution infrastructure — for example, for line sag for transmission lines — are reflective of temperatures they are experiencing. The hotter it gets, the more power lines can sag and “heat definitely can do a number on distribution system transformers,” Olson said.

But addressing these issues presents challenges of their own. When it comes to building out new capacity, the Pacific Northwest faces the same hurdles the rest of the country does, like siting and local opposition, said Kujala. Moreover, project developers across the country are facing issues with supply chain constraints, shortages of key materials and other delays. 

Portland General Electric used the reliability challenges of 2021 as learning tools to create new grid hardening projects, implement technology to help with reliability and fine-tune programs that addressed shifting customer behavior, Larry Bekkedahl, the utility's senior vice president of advanced energy delivery, said in an email. This included identifying high-risk areas where they could bury power lines, adding batteries to their system, and installing cameras to more quickly identify and respond to wildfires in its service territory. 

PGE is also offering customers options like peak-time rebates, as well as two-way grid enablement that supports dispersed energy resources, Bekkedahl said.

“Going forward, we see this being a key area to focus on — how can we use analytics and smart forecasting to work with customers to create a virtual power plant and manage load on the hottest and coolest days. We have found that many industrial customers are eager to take part in these programs given the mutual benefits, but need to know ahead of time to adjust production and the smarter we can make our grid and modeling, the easier time customers will have in taking part in decarbonization efforts without even realizing it," he added. 

While demand response has shown very positive results in bolstering reliability in California, it tends to be more difficult in the Pacific Northwest, due to the hydropower system, Kujala said.

“What conditions are going to necessitate demand response from one year to the next will widely change based on the amount of hydro generation that you have from one year to the next,” he said. While there are really good examples of regional utilities using demand response, “it’s just a harder path for demand response in the Pacific Northwest.”

In Texas, the focus has been on the thermal fleet

Texas did not experience a reliability crisis like the events that unfolded during Winter Storm Uri in 2021, but the ERCOT grid experienced very challenging conditions this year — all of the grid’s top 10 peak demand events occurred in 2022 and it touched a record 80 GW of demand on July 20.

Following Winter Storm Uri, Texas regulators began to contemplate a series of market reforms, including new requirements around the winterization of generators and some expansion of distributed energy resources, according to Caitlin Marquis, managing director at Advanced Energy Economy. 

Regulators in Texas are also considering a new market mechanism that would require utilities to purchase performance credits earned by generators based on their availability during high-risk periods for the system. 

“Texas is still a summer peaking system, so some of the policies aimed at winter, we’ll see how those continue to hold up as we go into the next winter,” she said. So far, the grid has maintained reliability through peak demand days, but there remain opportunities to get more out of technologies like distributed energy resources and battery storage, Marquis added. 

“The focus I think so far has been more on... the thermal fleet and having those resources stick around and show up,” she said.

More broadly, one aspect that Texas has going for it is that there are a lot of resources, including solar, wind and batteries, being built in the region, which will increase expected power margins going forward, according to Energy Innovation’s Gimon. 

“The real key dynamic during the energy transition is how long does the old stuff stick around while the new stuff comes on. Texas gave a very strong boost in the incentive for old stuff to stick around, by adjusting their so-called operational reserve demand curve,” a program that increases power prices once the hour-ahead reserves start to look low, he explained. 

What happens next?

Looking to 2023 and beyond, experts largely agree that power systems will continue to face more extreme weather events, and the electricity sector will need to make investments in infrastructure, demand response and additional power supply resources to address such events.

On the demand side, regulators need to take a closer look at pricing programs that are better communicated and have stronger incentives for people to move their demand to the middle of the day, Borenstein said. This could take a while because medium-term and long-term adjustments are likely to involve capital investments — “we’re talking about automated systems that do things like water heating in the off-peak, in order to reduce demand on-peak, heat pump installations and so forth,” he explained. 

In the shorter term, states like California need to implement more effective critical peak pricing programs, as opposed to the simpler time-of-use pricing that currently exists, Borenstein added, since peak and off-peak differentials are fairly small and don’t necessarily reflect the 10 days a year when the grid is in a crunch. 

Another question that regulators will need to work on in the West is how California and its neighboring states work together in a more organized fashion to ensure they are able to keep the lights on, added Amisha Rai, AEE managing director. Several Western states are ratcheting up their usage of renewables, and there is a heavy demand for affordable, clean energy, which means policymakers need to look at utilizing transmission assets in the most efficient way possible, she said.

This includes “really thinking about how an RTO in the West can better enable all of that delivery, facilitate renewables across the system and ensure that we are not leaving any renewables behind if there is excess supply…” she added. A July report from AEE found that the creation of a West-wide regional transmission organization — something that policymakers have been discussing for more than two decades — could lead to $79.2 billion in additional annual gross regional product across 11 states.

In Texas, meanwhile, regulators in November approved a pilot project to allow for aggregations of distributed energy resources to participate in the ERCOT market, which aims at harnessing some 80 MW of flexible resources on the grid. The pilot is estimated to last for three years, and several utilities are expected to participate in it. 

Looking over the next few years, a key priority for regulators in the region will need to be getting lessons from the first phase of the pilot and expanding those to better allow DERs to participate in the markets, AEE’s Marquis said. 

Another priority will need to be expanding opportunities for energy efficiency, according to Marquis. 

“Texas is one of the least energy-efficient states. It also has a really quickly growing residential population, so there’s a ton of potential for energy efficiency improvements… that will be critical to making sure that the state is able to meet customer needs in both winter and summer,” she added.

And then there’s the issue of transmission planning, which has emerged as a key priority at the federal level as well. In July 2021, the Federal Energy Regulatory Commission opened a proceeding to take a closer look at overhauling federal transmission policy, which stakeholders say will be critical to meeting longer-term clean energy goals. Congress is also looking at ways to facilitate new transmission and other infrastructure.

Addressing transmission infrastructure needs to make sure that new resources like wind, solar and battery storage are able to come online in Texas, and generation is able to reach load where needed, will be critical to reliability, Marquis said.

The California Independent System Operator got through a record-setting heat wave in September without resorting to rolling outages — a “remarkable outcome” — in part because of 3.5 GW of storage that was added since mid-2020, the grid operator said in a report released Nov. 2.

Also, consumers cut their electric use by about 1,510 MW in response to calls for conservation and other efforts on Sept. 6 when the CAISO grid hit a record peak load of 52,061 MW, according to the report.

“As we continue to integrate new resources onto the grid and make other necessary adjustments, our experience and lessons learned during the September 2022 heat wave will help us navigate the next climate-driven challenge,” CAISO said.

Utilities and grid operators across the United States are increasingly challenged by extreme weather, such as heat waves.

Starting Aug. 1, the West endured a 10-day heat wave, setting records across the region. Downtown Sacramento hit a record 116 degrees Fahrenheit on Sept. 6, according to the report.

CAISO’s market systems and processes largely responded as designed, according to the grid operator. “Because of greater regional coordination, reciprocity, and cooperation within the market, significant amounts of imported energy were available during the most stressed grid conditions,” CAISO said.

CAISO was also able to import power from the Desert Southwest and Pacific Northwest where the heat wave wasn’t as intense or long compared to California, according to the report. Net imports totaled 6,300 MW on Sept. 6. There were also 1,000 MW of transfers via the Western Energy Imbalance Market, according to the report.

“Despite the stressed system conditions during the heat wave, the market honored all wheeling-through market transactions that enable electricity to flow through the ISO but do not serve ISO load,” the grid operator said.

Recent market enhancements helped CAISO avoid rolling outages, according to the report. They include clarification of scheduling priorities, enhancements to resource sufficiency evaluations and electricity market pricing designed to incentivize generation when demand is high, the grid operator said.

During the heat wave, 60 storage resources with more than 4,000 MW of storage capacity and 15,000 MWh capability participated in the ISO markets, according to the report.

However, because of a software issue that has since been fixed, storage resources were not charging sufficiently early in the day during the heat wave, CAISO said.

“We continue to learn how best to integrate storage into the market, and the high prices experienced during the heat wave presented new scenarios for the ISO to learn about the complexities and challenges of managing battery state of charge,” the grid operator said.

In addition, the owners of fossil-fueled power plants have worked to improve the performance of their fleets in the last two years, CAISO said.

“These resources were critical during the most stressed part of the heat wave; however, some of these resources still experienced high levels of unavailability during other times of the event,” the grid operator said.

During the heat wave, the ISO’s daily average power prices rose to $600/MWh, with maximum prices reaching $2,000/MWh in the real-time market, the grid operator said.

CAISO held a conference call on Nov. 17 to review the report and answer stakeholder questions.

A U.S. Department of Energy national laboratory has shown how blockchain technology can be used to secure the grid in what it says is a first-of-its-kind demonstration of a new power system communications framework.

“This framework gives us a totally new capability to rapidly respond to anomalies,” Raymond Borges Hink, who led the research team at the Oak Ridge National Laboratory, said in a statement. Borges Hink is a cyber security research scientist at the lab.

A blockchain is a distributed ledger more commonly associated with crypto currencies, but he said his team has shown it can be used to “more quickly identify an unauthorized system change, find its source and provide more trustworthy failure analysis. The goal is to limit the damage caused by a cyberattack or equipment failure.”

The blockchain-based communications framework was demonstrated at the Grid Research Integration and Deployment Center at Oak Ridge this year. According to an August study of the project it utilized measurement, communication and protection devices that electric utilities commonly employ.

In order to ensure grid components were properly functioning, the communications framework would verify that configuration data on devices had not been illegitimately modified from the last known correct settings by comparing it “with the last known correct baselines.”

“Our system helps determine in near real-time whether a fault was triggered by a cyberattack or induced by natural events,” Borges Hink said. “This is the first implementation of blockchain enabling this kind of data validation between a substation, a control center and metering infrastructure.”

While most of the energy discussion around blockchain technologies focuses on their energy use — a group of federal lawmakers is investigating the electricity consumption of bitcoin — some use cases focus on how distributed ledgers can benefit clean energy development and grid management. Decarbonization-focused non-profit RMI has been studying how distributed ledgers can be used in carbon markets.

A Ukraine war-provoked Russian cyberattack on the U.S. power system has not happened, but experts agree the threat is real because of a key shortcoming in cybersecurity preparations.

The 2021 Colonial Pipeline shutdown that disrupted Eastern U.S. gasoline deliveries hinted at the danger of cyberattacks on the energy sector. A May 12, 2021, Biden executive order, requiring major power system cybersecurity actions, implicitly acknowledged that Russia’s 2015 attack on Ukraine’s power system can happen here. But current and planned responses to the Biden order may not be enough to protect electricity delivery, cyber specialists said.

Russia may have so far withheld cyber warfare against the U.S. and its allies because of “a balance of power issue,” OPSWAT operations technology and industrial cybersecurity expert Oren Dvoskin said. “If a cyberattack is stopped, whoever stopped it knows the adversary, which is why nation-states are careful about if and when to deploy cyber weapons,” he said.

But the cyber threat to the energy sector goes beyond attacks to communications networks like the recent headlined ransomware attacks, analysts said. Using the growing internet access of power system operations that allow companies to monitor and control engineering processes online, attackers could disrupt critical infrastructure to create environmental devastation, losses of life, and severe economic impacts, they said. 

Power system “security and safety” depends on “the reliability and accuracy of sensor data that informs operations,” Applied Control Solutions Managing Partner and Cybersecurity Analyst Joe Weiss told Utility Dive. And “Russia, China, and Iran are aware of the lack of cybersecurity in process sensors and have access to them” in critical electric system operations, he said.  

The recent discovery of cryptocurrency’s vulnerability is a reminder that cybersecurity requires constant attention. But threats can be minimized by first recognizing protections to internet technology networks are inadequate to protect operational technology hardware, and then putting the best people, processes and technologies in place to protect electricity delivery, cyber analysts said.

Recognizing threats

The Biden executive order recognized “persistent and increasingly sophisticated malicious cyber campaigns” and the need “to identify, deter, protect against, detect, and respond” to them. And protections must include systems “that run the vital machinery that ensures our safety (operational technology (OT)),” along with “systems that process data (information technology (IT)),” it said.

Threats to energy sector networks were apparent before the executive order. But vulnerability expanded with the “convergence of IT and OT systems” and the use of “common software and security systems” in monitoring and control, a November 2021 Congressional Research Service report said.

Shields Up, the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency, or CISA, was launched in February by the Biden administration as a response to the Ukraine war’s increased cyber threat.

The May 2021 executive order required federal agencies, including CISA, to develop “Zero Trust Architecture.” A zero trust system design assumes that “anomalous or malicious activity” is “inevitable or has likely already occurred.” It eliminates “implicit trust in any one element” and allows online access only with “real-time” and “multiple sources” verifications, the order said.

But that leaves a shortcoming in cybersecurity, analysts said.

The shortcoming

The federal approaches assume IT attacks are the concern, control systems engineer and cyberanalyst Weiss said. They overlook OT-focused cyberattacks, which “are not always easily identifiable or recognized at all,” and “can be mistaken for accidents or malfunctions,” he warned.

Former Federal Energy Regulatory Commission Chair Jon Wellinghoff, now CEO of power system consultant GridPolicy, agreed. “The executive order was a great step forward in coordinating federal agencies’ cybersecurity efforts under Homeland Security,” he told Utility Dive. “But there has not been enough progress on securing operations technology.”

The still unexplained August 2019 federal seizure of a Chinese transformer on its way to installation on the Western Area Power Administration system demonstrates the complexity of the OT threat, both Wellinghoff and Weiss said. It was diverted to Sandia National Laboratory for study when it was discovered to have unexplained electronics built into it, they said.

Transformer manufacturer JiangSu HuaPeng denied allegations of an effort to create a hidden backdoor to the U.S. power system, but there is no public report of Sandia findings confirming that, a September 2020 investigation by Motherboard reported.  

“There are probably thousands of China-originated control devices in our electric and natural gas infrastructure that we can't be sure of,” Wellinghoff said. “The core of the problem is that the bulk of cybersecurity focuses on IT networks and not industrial control operations technologies.”

Even sensors in smart operations systems may have no capability for passwords, authentication or encryption, but nevertheless remain “the 100% trusted input to OT networks,” Weiss added.

Sensors without security capabilities make malicious and unintentional operational disruptions difficult to distinguish and could allow power system cyberattacks to go unnoticed, Weiss said. Inadequate sensor protections contribute to continued uncertainties about the specific cause of the 2005 Stuxnet attack on an Iranian nuclear facility and a 2008 Florida nuclear plant shutdown resulting from a substation disruption that left no proof of its supposedly accidental cause, he added.

Even where internet communications are secure, if OT sensors “are compromised or defective, it will not be possible to have a safe, secure, reliable, or optimized process,” Weiss wrote in March 2021.

These vulnerabilities are “a real plague in OT environments,” OPSWAT’s Dvoskin agreed. The malware that attacked the Colonial pipeline in 2021 compromised so many of Colonial’s IT components that it forced a preemptive shutdown of operations, he said.

The current alternative is “unidirectional security gateways” that “segment the OT network” to limit malware access between the IT network and OT devices, he said. But “attack patterns are always one step ahead,” and even the zero trust concept is compromised when OT networks “not architectured for cybersecurity leave the entire network vulnerable,” he said.

To date, most attacks have been through ransomware and companies have paid the attackers, he said. Unfortunately, that has created “an incentive for attackers to pursue bigger returns by targeting operations, like the Colonial incident,” Dvoskin said.

As a result, cybersecurity has become a high electric system priority, many stakeholders agreed.

Utilities and system operators respond

Major U.S. utilities and power system operators are using the CISA and Biden order guidance to prepare for attacks, they told Utility Dive. They are also participating in public agency and industry protective exercises, including the North American Electric Reliability Corporation GridEx and the Department of Energy Clear Path simulations.

As early as 2020, cybersecurity attacks and vulnerabilities to “ransomware and supply chain compromises” were “a significant concern,” according to NERC’s 2021 State of Reliability report. And in December 2020, there was a complicated attack “that leveraged SolarWinds’ Orion software and Microsoft’s Azure cloud environment.” 

The attack on software provider SolarWinds allowed Russian attackers to access power sector networks, NERC reported. Though it caused no loss of system load, it “highlighted the extraordinary capability and persistence” of “the Russian Foreign Intelligence Service,” CISA added. A post-incident investigation revealed “25% of electric utilities had downloaded malicious software,” NERC said.

While energy industry leaders have limited “expertise” about cybersecurity threats, the “shock of recent incidents” has driven “major changes,” a February-March 2022 international survey by global consultants DNV found.

Electric utilities are working on cybersecurity from a “defense-in-depth” perspective, with simultaneous planning for prevention of and remediation from attacks, both Edison Electric Institute Senior VP of Security and Preparedness Scott Aronson and American Public Power Association Senior Director, Cybersecurity, Bridgette Bourge emailed Utility Dive.

As part of the defense-in-depth approach, an Essence 2.0 Cybersecurity Technology is now being developed that integrates the monitoring of IT and OT systems and “continuously assesses the electric grid for anything out of the ordinary,” according to the National Rural Electric Cooperative Association.

Individual utilities reported unhesitating seriousness about cybersecurity.

“We can never be too safe or prepared,” Duke Energy spokesperson Caroline McMillan Portillo said. “As the scale and sophistication of potential cyber events grow, so do our defensive capabilities for all of our IT and OT assets, and for both the bulk system and the distribution system.” Arizona Public Service spokesperson Yessica del Rincon and Entergy spokesperson Neal Kirby concurred.

American Electric Power, which provides regulated, unregulated, and transmission service across a 12-state, 200,000 square mile territory, “has about 200 cyber people focused on security,” AEP VP of Cyber and Physical Security Steve Swick said.

“We are reluctant to speak in specifics,” but “we hire top cyber security experts,” and “hold regular drills,” Consolidated Edison spokesperson Allan Drury emailed. “Threats are increasing in sophistication, magnitude and frequency,” and “would-be attackers are inventive and relentless,” he added.

The California and Midcontinent independent system operators declined to comment in detail. The New York Independent System Operator “relies on the NERC Critical Infrastructure Protection standards and other cybersecurity frameworks, guidelines, and best practices,” its Power Trends 2022 reported.

And better practices are emerging, cybersecurity analysts reported. 

Emerging cybersecurity practices

The Biden executive order, CISA’s Shields Up protections, and the NERC guidelines are the most frequently mentioned cybersecurity standards, stakeholders agreed.  

But most of the Biden order and CISA guidance only explicitly address IT intrusions and ransomware, Weiss and others stressed. And while securing IT systems from attacks can prevent financial consequences, losing control of OT systems could lead to significant economic or environmental impacts or cost lives, a recent DNV “insights” paper warned.

Historically, OT networks were not online, which allowed use of less secure equipment and practices, DNV Cyber Security Managing Director Trond Solberg told Utility Dive. And the IT and OT departments were often organizationally siloed, “exchanging risk reports that were frequently forgotten or building firewalls that people weren’t trained to use.”

Many energy industry executives acknowledge “OT security is now getting attention,” he said. But because many have not realized the extent to which OT requires new practices, “too few executives are willing to invest significant time and money, and OT cybersecurity problems remain.” 

Better practices in OT security will depend on how “people, processes, and technologies” move from “prevailing practices” to meet “current conditions” and take advantage of foreseeable “future directions,” a DNV December 2021 white paper on new OT security practices summarized.

Prevailing practices have too few people engaged in OT security, inadequate plans to define responsibilities, and do not focus on developing dedicated OT security teams, DNV said. OT-specific, risk-based standards are being adopted, but policy enforcement is needed, it added.

These prevailing practices do not effectively address current nation-state attacks and the lack of industry-wide governance, the paper said. Investments are needed to close gaps in incident response, asset identification, and, especially, in management of currently emerging attacker accesses through IT-OT interfaces, it said.

In the next two to four years, the pool of personnel dedicated to OT security should be doubled, DNV said. Because attacks through supply chains are accelerating, a converged IT and OT team and a wider view of where threats can come from will also be needed in auditing vendor security.

Converged IT and OT systems can drive adoption of more zero trust measures and advanced and enabling technologies, DNV concluded. The result could be the optimization of today’s isolated and fragmented security practices.

A “radical shift” can be driven by safeguarding OT if energy company leaders recognize that protecting sensors and communications systems will transform a “non-security” cost into “a security win,” DNV said. With that shift, zero trust and access management programs can become bigger priorities, which will lead to better security practices in both the IT and OT domains, it added.

Skilled OT security specialists, managers, engineers operators, and others must be developed, the paper stressed. That kind of team will have the organizational and technical skills to develop critical “cyber readiness” by planning for and responding to “security compromises and data breaches” on both OT and IT systems, and also ensuring supply chain and third-party security, it added.

Cyberanalyst Weiss was more succinct than DNV. Close the “culture gap” between the Chief Information Security Officer who “owns no hardware” and the vice president for power delivery who “owns everything but isn't part of cybersecurity,” he said. “Engineers care about reliability and safety and network people care about cybersecurity and they currently are not talking to each other.”