Introduction

Markets cannot solve the problem of electricity market design. Given current technology, chiefly in requiring nearly instantaneous balancing with strong interdependencies across the transmission grid, efficient electricity markets have and need central coordination to support competition among generators, suppliers, and load customers. The questions are about how but not whether to organize that coordination. The green agenda to reduce greenhouse gas emissions through widescale deployment of renewable energy resources raises challenges for electricity market design. The new generation technologies have high capital costs but low variable costs, in contrast to older thermal technologies with lower capital costs but higher variable costs. Combined with the intermittency of renewable supply, there could be requirements to modify, extend, or replace existing electricity market designs.

A high-level review of efficient electricity market design as it has developed in the USA, in theory and practice, provides a foundation for addressing key requirements for the green agenda. The focus is on underlying principles applied to the requirements and contributions of efficient electricity markets with lower, low, or zero carbon emissions. Importantly, the analysis here allows for fundamental and significant changes without a priori rejecting ideas that are admittedly politically difficult. For example, high and volatile energy prices may be part of the solution; subsidies and mandatory forward capacity mechanisms may be part of the problem; and higher reliability standards may be expensive illusions.

After reviewing the basics of efficient electricity market design, attentions turn to scarcity pricing, the growing importance of demand participation, improvements in multi-period dispatch and pricing, expanding the wholesale competitive model to the distribution level, and unit commitment pricing issues. An underlying issue that interacts with deployment of renewable resources appears in the discussion of carbon pricing and the main components of the social cost of carbon. The real-time market design interacts with forward markets to help support innovation and investment.

Efficient Electricity Market Design

Liberalized wholesale energy markets have been a major focus of policy. “The core idea motivating these reforms was to lower costs by using competitive forces to drive efficiency improvements relative to the well-known inefficiencies of regulated monopolies” [, p. 2]. The details differ across oil, natural gas, and electricity markets. In the case of electricity, a critical element is in the treatment of the transmission system. “The practice of ignoring the critical functions played by the transmission system in many discussions of deregulation almost certainly leads to incorrect conclusions about the optimal structure of an electric power system” [, p. 63].1, 2

A key step in developing an efficient wholesale electricity market design is to begin the process at the end. In a system of monopoly and central direction, real-time incentives may be of secondary importance, and electricity system design can and did focus on cost recovery rather than incentives. However, in a competitive market operating under principles of open access and non-discrimination, good design begins with the real-time market and works backward. A common failure mode starts with the forward market, without specifying the rules and prices that would apply in real-time. In a market where participants have discretion, the most important prices are those in real-time. “Despite the fact that quantities traded in the balancing markets are generally small, the prevailing balancing prices, or real-time prices, may have a strong impact on prices in the wholesale electricity markets. … No generator would want to sell on the wholesale market at a price lower than the expected real-time price, and no consumer would want to buy on the wholesale market at a price higher than the expected real-time price. As a consequence, any distortions in the real-time prices may filter through to the wholesale electricity prices” [, p. 53].3

Electricity systems require system operators to coordinate real-time activities. The various actions include changes in the dispatch to rebalance the system and manage utilization of the transmission grid. Changing the pattern of net generation across locations is the principal tool for ensuring reliable power flows. The principles of efficient markets imply that market participant incentives will be compatible with efficient use of the grid. In the electricity system, this efficient use amounts to the long-standing practice of economic dispatch subject to transmission and security constraints.

Hence, the requirement was not to find a way to allow the market to discover an efficient dispatch. Rather, the challenge was to provide consistent real-time or spot prices that would support the efficient or economic dispatch coordinated by the system operator. The prices would reflect the locational differences in the marginal value of net generation. Under reasonable simplifying assumptions found in common practice, the solution amounted to determine what became known as locational marginal prices (LMP) [4]. For a given dispatch interval, the LMPs capture the marginal value of incremental generation balancing the marginal cost of load reductions.

“Locational marginal pricing (LMP) is the electricity spot pricing model that serves as the benchmark for market design – the textbook ideal that should be the target for policy makers. A trading arrangement based on LMP takes all relevant generation and transmission costs appropriately into account and hence supports optimal investments” [, p. 116].5

This model evolved from a cost-based engineering foundation to a bid-based system that approximated the idealized supply–demand framework of basic economics with the coordinated solution that could account for generator supply curves and load demand corves [6]. The foundation of economic dispatch and locational prices provided the ingredients for a system of financial transmission rights that could replace the unworkable prior contract-path approach to transmission rights and hedging basis risk [7].

The development and application of this efficient real-time wholesale electricity market framework was difficult and expensive [8]. Eventually this model was adopted by all the organized markets in the USA. This is the only model for dispatch and pricing that supports open access and non-discrimination. The model continues to evolve, maintaining the close connections to first principles [6].

How might this model have to change due to the new requirements of the green agenda?

The real-time supply curve or dispatch merit order for the traditional electricity market is upward sloping, reflecting the increasing variable costs of generation from existing renewables, to nuclear, to efficient gas to inefficient coal generators. Marginal system prices move accordingly as load changes. As a first approach to answering the question about the new requirements of the green agenda, consider a thought experiment. Suppose that this supply is replaced by 100% zero-marginal cost green energy, up to the limits of its capacity. This is an extreme case, and it ignores the important effect of storage and intertemporal interactions, both of which will be more important with increasing penetration of intermittent generation. It is not likely to happen, but the hypothetical example does illustrate the necessary changes in real-time market design.

In particular, a change to the green supply curve would not change the basic design principles [9].

Market outcomes would be different. When demand is low relative to capacity, the marginal cost would be zero, so the price would be zero. And when the demand curve shifts to the right, price would have to rise to reduce the quantity demanded to match the available capacity. The associated market-clearing LMP price would be (very) high. But the efficient market design says nothing about the level of real-time prices. The prices would change and would be driven entirely by scarcity pricing, but the market design would be unchanged.

This thought experiment provides context for considering the implications of the green agenda for real-time electricity market design. The example highlights the importance of scarcity pricing, demand participation, and multi-period dispatch.

Scarcity Pricing

The description of the real-time LMP model often simplifies to marginal-cost pricing, which then collapsed to the treatment of the marginal cost of generators. In part this derives from assuming that demand was fixed. But this descriptive convenience was never exactly correct, nor necessary. For example, when load reached the capacity of a given swath of generation, there would always be an additional price component that would reflect the scarcity of lower cost generation. This would include high load periods when all the available generation capacity was in use. Then scarcity prices would be necessary to balance supply and demand.

Less obvious was the treatment of associated operating reserves. The usual analysis focused on energy and ignored the interactions with the required capacity to be set aside for operating reserves. The efficient market implementations continued this practice by failing to account for the interaction of energy and the marginal values of reserves. The assumption was that this would not be very important, because the interaction would be handled through demand participation. However, demand participation did not develop, and thus, efficient design required a more immediate attention to the roles of operating reserves and scarcity pricing.

Improvements in scarcity pricing have been slow to come, with the leading affirmative example being the operating reserve demand curve (ORDC) adopted in the Texas ERCOT system, starting in 2014 [10, 11]. The essential idea is to recognize the imminent expected marginal value of operating reserves, as derived through an operating reserve demand curve, and incorporate that implied scarcity price as part of the market-clearing price of energy. Full implementation would include co-optimization in the economic dispatch. This reform was overdue, and it will be of even greater importance as part of the adaptation of markets to accommodate intermittent renewable energy.

 The Texas experience through 2020 reinforced the need for scarcity pricing and the analysis of the benefits. Prices were high during scarcity conditions, helped alleviate stress on the system, and were supporting new generation investment. “These results demonstrate that the suppliers in ERCOT respond to price signals and associated incentives” [, pp. 82–83]. The exceptional emergency during February 2021 remains a subject of important further study and investigation as part of the regulatory review [12, 13]. However, the weather conditions were a one-in-fifty year event [14], so extreme and well outside the traditional one-in-ten year reliability standard that it is not clear that any electricity system design would have fared well [15].

Demand Participation

The basic model of economic efficiency treats supply and demand symmetrically. When prices rise, supply increases and demand decreases. The market-clearing price balances supply and demand. While this works in theory, the experience is that demand participation in the real-time dispatch has been weak to non-existent. This is distinct from the problematic “demand response” programs, primarily used to substitute for operating reserves, where operators pay loads to reduce their demand. The well-known problems of pricing, baselines, and measurement for demand response are a separate issue [16]. Demand participation differs principally in charging loads for what they consume, which can be measured, rather than paying them for what they did not consume, which can only be estimated to obtain deemed demand reduction.

Demand participation in the real-time dispatch has not been consistent with the expectations that underly the efficient market theory. There are several possible explanations that have to do with market failures, particularly under extreme conditions [17]. However, incomplete scarcity pricing created a chicken-and-egg problem in that there would be little incentive for demand participation. The ERCOT experience is that scarcity pricing and the related peak charges for transmission cost recovery produce material reductions in demand. With greater demand participation, the ORDC contribution to scarcity pricing would recede in importance and demand should set the market-clearing price. In the market with high penetration of renewables, every source of flexibility will be of increasing importance, so reforms to facilitate demand participation would be of increasing importance in the operation of the real-time market.

Multi-period Dispatch and Pricing

Another aspect of system flexibility, that will be of increasing value with the increased penetration of renewables, is the ability to change the dispatch over time in response to rapidly changing conditions. In part this has been met by reducing the dispatch and price settlement intervals to 5 min. The original development of the efficient market model followed a static perspective, where each period is independent [4]. In the presence of ubiquitous ramping constraints, however, dispatch intervals are not independent. This creates a challenge for the basic electricity market design and this challenge will become more important in the future.

The need for ramping capability requires prepositioning generators to make sure that adequate capacity is available in subsequent periods, and this dispatch configuration may not be consistent with marginal cost conditions within the dispatch interval. This has produced market extensions to create new ramping products to deal with the expected multi-period trajectory of energy demand requirements [18, 19].

This need for new products follows from an assumption that the efficient market design cannot produce supporting prices that reflect the ramping requirements. However, in the same way that the basic static real-time framework provides LMP derived from marginal conditions to support the optimal dispatch, a multi-period optimization with look-ahead can produce LMPs that incorporate the ramping constraints [20].

The pricing model would be embedded in a rolling dispatch where the system is optimized with a suitable look-ahead across multiple intervals, updating both the dispatch and the prices. With no change in expected conditions, and for the special case of strictly increasing supply curves, the rolling price conditions would support the optimal dispatch [21]. In the more general case, with only non-decreasing supply curves, the rolling model would require some added constraints to maintain the required dispatch and price interdependence across the rolling horizon [22]. When conditions change in the rolling price framework, there will be a requirement to select a dynamic settlements model, ranging from settlement for the current dispatch interval only to repeated settlements for the full look-ahead period [23]. The multi-period extension of the standard efficient dispatch and pricing can incorporate scarcity pricing through the ORDC required to address the uncertainty about future deviations from the forecast. The basic efficient market design model can incorporate the expected ramping requirements without creating new products.

Distribution Level Pricing

The focus of efficient real-time electricity market design has been on the wholesale market with generator and load connections to the high voltage grid. However, with the increasing impacts of intermittent renewables and the growing investment in distributed energy resources, policy and regulatory attention has turned to the consideration of how to incorporate demand participation and energy supplies that are connected to the distribution system [24].

There is nothing in principle that prevents the extension of the analysis to include the distribution system, and the basic model could treat real power flows and calculate variable energy costs in the same way. However, there are added complications to address the reactive power conditions that can be more material on the low voltage distribution systems [25]. These problems need to be addressed in the context of extending active market participation and trading to the distribution level [26]. However, the scale of the problem is daunting. For example, in 2021 the PJM bus model included approximately 12,000 locations. Extending this to every connection on the distribution system would increase the number of LMPs to be calculated and applied by orders of magnitude. Although an active area of research, it remains to define a workable implementation that encompasses the added constraints on the distribution system [27]. This is not a new condition caused by expansion of renewable energy resources. But it becomes more important with expanding demand participation and distributed energy resources.

Efficient calculation of the marginal costs for energy would increase the importance of improving pricing for delivered energy. This would separate fixed from variable costs to support demand participation for loads connected at the distribution. The challenge of reforming existing retail rate designs is an old and well-known problem [28, 29]. The expansion of intermittent renewables does not change the fundamentals, but it does increase the importance of price reform.

Unit Commitment, Dispatch, and Pricing

The basic theory of efficient electricity markets and pricing included simplifying assumptions, such as convexity for cost functions, that assured the existence of market-clearing prices that would support the cost-minimizing dispatch solution [4]. The actual electricity system includes important complications, such as unit commitment, startup costs, minimum run times and so on, that violate these simplifying assumptions. System operators recognize these constraints and incorporate them in their optimization models. The definition of efficient dispatch expands to include these unit commitment and related constraints. The present discussion emphasizes deterministic models capable for implementation using tools consistent with current practice rather than complete stochastic optimization frameworks [30, 31]. Although computationally challenging, the efficient dispatch interpretation remains.

For pricing, however, there is a complication because the discrete conditions in the unit commitment model violate the convexity conditions and produce circumstances where there may be no energy prices that would clear the market. In other words, the energy prices cannot support the dispatch solution. The resolution of the dilemma has been to move to two-part pricing with energy prices and then various possible “uplift” side-payments that ensure that the market participants optimal individual choice is to follow the efficient unit commitment and dispatch [32]. The particular methods employ different computational formulations, including relaxation of the constraints to form various pricing models that conform to a closely related convex problem [33].

Active approaches include constructing the convex hulls of the components and then optimizing the resulting problem. This provides exact solutions for a useful class of problems and reports good approximations in other cases [34]. An expanded unit commitment characterization can provide convex hull prices with most of the existing unit commitment constraints including ramping constraints [35]. Reformulating the original problem by adding constraints and variables, to construct an equivalent master problem that characterizes the convex hull provides exact pricing solutions and reports good computational performance [36,37,38]. However, the reformulated problem may be hard to connect to the native formulation used by the system operator. An alternative approach is to maintain the native unit commitment but apply either the well-known Dantzig-Wolfe decomposition [39] or heuristics to accelerate Benders decomposition applied to the dual problem of the original unit commitment [40].

This line of developing theory and practice is important, but it is part of the natural evolution of efficient market design. Greater penetration of renewables does not affect the basic theoretical issues, but it does reinforce the importance of improved pricing models.

Carbon Pricing

Pricing carbon and other emissions fits naturally with efficient electricity market design. The price defines a variable cost for generation and is included along with other variable costs such as for fuel. Economic dispatch seeks the lowest total for the aggregate system variable costs and translates the impacts into the LMP energy prices.

Prices and Subsidies

The natural fit of carbon pricing is familiar because it is already part of the market, such as in the Regional Greenhouse Gas Initiative (RGGI).Footnote 1 Although the RGGI prices have been low, they are material and provide an ample demonstration of the natural accommodation of carbon pricing. The best policy would be for a single price of carbon emissions across the interconnected grid. When the carbon price is different for connected locations, as with RGGI or the Western Energy Imbalance Market, there are issues of how to treat exports and imports [41, 42]. However, the main experience is that carbon pricing is consistent with the basic electricity market design assumptions and theory. Carbon pricing is critical for actually achieving the objectives of addressing the climate challenge [43].

Subsidies for clean energy are not the same as carbon pricing. The attractions of markets include incentives for efficient operation and investment, changing the incidence of risk bearing from loads to generators to better match the investment decisions, and simplifying otherwise complicated to impossible analyses of how the subsidies connect to the goal of emissions reductions. A focus on the pathways for subsidies compromises the underlying objective. Tracking carbon emissions without pricing is a fraught endeavor. In principle, we do not even know the sign of the effect at the margin [44, 45]. A clean energy standard (CES) would be better than technology specific subsidies, but the imperfect link between renewable production and emission reduction means that a CES is not equivalent to carbon pricing [, p. 100].46

Measuring emissions is easy; unpacking and tracking substitution via subsidized activities is hard. For example, “[t]he movement to have companies measure and disclose their emissions is just an enormous waste of time. If you had a proper price on carbon, we wouldn’t have to do that any more than we need companies to do an inventory of their wheat use or silicon use. It’s another example of how we’re going down a rabbit hole of measures. Even the central banks are getting involved.”Footnote 2

Subsidies produce unintended consequences and undermine the incentives provided by markets. To illustrate, the production tax credit is well known to create a perverse incentive for the wind generator which turns the real zero variable cost into a perceived negative variable cost equal to the amount of the subsidy. The results can be negative energy prices. This, for example, creates an incentive for storage operators to charge and discharge simultaneously, making money while dissipating energy by operating the battery as a radiator and thereby increasing load [47].

“Subsidies are contagious. Competition in the markets could be replaced by competition to receive subsidies” [, p. 2]. The problem is pervasive and grows with increasing reliance on subsidies. “Subsidies pose a more general problem in this context. They attempt to discourage carbon-intensive activities by making other activities more attractive. One difficulty with subsidies is identifying the eligible low-carbon activities. Why subsidize hybrid cars (which we do) and not biking (which we do not)? Is the answer to subsidize all low carbon activities? Of course, that is impossible because there are just too many low-carbon activities, and it would prove astronomically expensive. Another problem is that subsidies are so uneven in their impact. A recent study by the National Academy of Sciences looked at the impact of several subsidies on GHG emissions. It found a vast difference in their effectiveness in terms of CO482 removed per dollar of subsidy. None of the subsidies were efficient; some were horribly inefficient; and others such as the ethanol subsidy were perverse and actually increased GHG emissions. The net effect of all the subsidies taken together was effectively zero! … So in the end, it is much more effective to penalize carbon emissions than to subsidize everything else” [, p. 266].49

Social Cost of Carbon

Determining the appropriate price of carbon is challenging. Part of the difficulty is conceptual. Popular debate on climate policy includes an embrace of policy targets such as a “net zero” level of carbon and related emissions by a given date. Studies then develop cost-effective strategies to achieve the objective. A by-product of these studies is an estimate of the marginal cost of carbon emission reduction [, p. 204]. The marginal cost estimates can be very large, so large as to call into question the stated policy objective.50

By contrast, the social cost of carbon arises from a cost–benefit framework. “In the context of climate change, the application of cost–benefit analysis to inform mitigation policies can help to achieve the best outcomes and avoid the worst: spending trillions of dollars but failing to get the job done (fn). The costs of a climate policy are the abatement costs of reducing emissions of carbon dioxide (CO2) (or other greenhouse gases). The standard measure of the benefits of a climate policy is the social cost of carbon (SCC), which measures the avoided economic damages associated with a metric ton of CO2 emissions” [, p. 850]. The SCC provides a yardstick for deciding how much is enough, rather than starting with a target set without balancing costs and benefits [, pp. 46–47].51, 52

Integrated assessment models (IAM) provide a tool for implementing this cost–benefit analysis. The first and most notable model is the Dynamic Integrated Climate-Economy (DICE) model developed by Nordhaus [53]. There is a continuing controversy over the details of this model and the implications for the estimate of the SCC [54]. The US government provided interim guidance and will provide an updated analysis of the estimate and use of the SCC [55].

Consider three elements of the debate, described in terms of the respective components of DICE: the discount rate, the treatment of uncertainty, and the impact of tipping points. This list is not exhaustive, but it captures the most important analytical issues.

Discount Rate

The interim guidance reports a SCC ranging from $14 to $76 per metric ton (2020 dollars), depending on the choice of the discount rate applied to marginal damages, with a central value of $51 using a 3% discount rate. The underlying debate includes a normative argument that the appropriate formulation of the cost–benefit analysis should apply a prescriptive utility function that gives relatively high weight to future (much richer) generations and therefore lower discount rates [56]. By contrast, Nordhaus argues for the descriptive approach that chooses the parameters of the utility function to be reasonable but to produce discount rates that initially approximate actual implied tradeoffs between investment and consumption and gradually decline over time [57, 58].

There have been attempts to embrace both approaches by adopting an alternative utility function separating the tradeoffs between contemporaneous states from the comparison between different periods [59]. But this merely relocates rather than resolves the argument [60]. The debate also connects to the largely unresolved issue in finance known under the heading of the equity premium puzzle [61,62,63].

Absent climate change, the prescriptive approach would imply that consumption of the current population should be reduced significantly to increase further the consumption of richer future populations. The descriptive approach accepts the tradeoffs between consumption and investment as they are found historically. On balance, the descriptive approach provides a firmer foundation for carbon pricing policy.

The DICE model computes an optimal trajectory. The Climate Leadership Council proposed a similar “Gradually Rising Carbon Fee … Carbon Dividends for All Americans … Significant Regulatory Simplification … Border Carbon Adjustment” [64]. The Princeton Net-Zero America (NZA) report sets a 2050 emissions target and estimates the implied marginal cost of emission reduction for a range of trajectories [, p. 204]. As shown in the figure, the near term estimates are similar, but by 2050 the implied SCCs diverge from substantially from the DICE optimum [50, 65].

figure a

The figure illustrates the dramatic difference between the SCC based on a cost–benefit framework and the implications of policy targets set without reference to the cost–benefit tradeoffs.

Uncertainty

The future is uncertain. Future marginal damages vary substantially across different scenarios. The treatment of uncertainty is especially challenging for long horizons because the sequential resolution of uncertainty creates the opportunity to act now, learn, and then act again later with an adjusted policy [66,67,68]. A material problem with these approaches incorporating uncertainty is the computational burden which leads to embedded model simplifications or reduced model transparency.

The principal alternative is to conduct sensitivity analyses using probabilistic Monte Carlo analysis. “When uncertainties are accounted for, the expected values of most of the major geophysical variables, such as temperature, are largely unchanged. However, the social cost of carbon is higher (by about 10 percent) under uncertainty than in the best-guess case because of the asymmetry in the impacts of uncertainty on the damages from climate change. … the relative uncertainty is much higher for economic variables than for geophysical variables” [, p. 335].69

Tipping Points

The prospect of tipping points captures the imagination. For example, loss of the Greenland ice sheet would produce a dramatic increase in sea level and would be irreversible on human timescales. However, an analysis by Nordhaus that incorporates both the timing very far in the future and the sequential resolution of uncertainty implies that the effect of this tipping point is to increase the current estimate of the SCC by less than 5% [70]. This estimate is confirmed by a separate study that examines a more comprehensive list of eight possible tipping points and concludes that the effect is to increase the current estimate of the SCC by 25% [71]. This estimate is remarkably similar to the judgmental damage adjustment embedded in DICE [, p. 11].57

Forward Markets

The impact of greater penetration of intermittent generation underscores the importance of continuing to improve implementation of the basic efficient market design without requiring a fundamental change in the underlying theory. The real-time design supports static or productive efficiency. Given the capital-intensive nature of the electricity sector, which would increase with many of the renewable energy sources, forward markets are important in supporting dynamic efficiency of both investment and innovation.

Well-designed forward markets for energy, ancillary services, and financial transmission rights already anticipate settlement against real-time conditions, so the broad outline of the forward market structure remains unchanged. The outcomes will be different depending on different anticipated real-time conditions, but the basic elements of forward market pricing and contracts would continue.

Importantly, with the exception of some long-lead time unit commitment decisions, forward markets and their instruments are financial contracts that generally do not require, or cannot provide, specific performance for power flows. For example, forward energy contracts are essentially contracts for differences settled against the real-time quantities and prices. Hence, forward markets provide a natural venue for so-called virtual transactions which are in effect strictly financial. These virtual arrangements expand the market participants to include financial institutions and provide a valuable arbitrage function to drive forward market prices towards the expected real-time values [72]. This both disciplines the market and yields a valuable corrective that avoids any need, for example, to try and equate forward estimates of transmission flows with real-time flows. In other words, the arbitrage function addresses the prices but not the quantities [73].

Characterizing uncertainty is an example of an important forward activity that is affected by the expansion of intermittent generation but does not change the structure of efficient electricity market design. Almost by definition, greater penetration of intermittent renewable energy creates greater volatility in the availability of power. Furthermore, whereas the traditional system could be treated under various assumptions of independence of loads and generation types, the correlation of across types is now material. When the wind stops blowing in the region, all the wind generators reduce their output. Improving analysis and forecasting abilities is an important line of research and improved practice, but for purpose of the present discussion it has little impact on forward market design. As long as forward auctions and contracts are organized using deterministic optimization models and schedules based on expected values, the forward market design should be largely unaffected. Moving to a formal stochastic design with state contingent contracts would be another matter, but this is beyond the scope of the discussion here.

Supporting Investment

The idealized efficient market provides a guide for supporting investment. With efficient real-time prices, market participants can participate in forward markets and make investments that support long-run efficiency. The so-called energy-only market would be sufficient. However, this ideal case is incomplete for a number of reasons, and a better description would be as an “energy-only” market which captures the essence of the idea while recognizing that a more hybrid structure is required [74]. Three prominent long-term investment issues include the challenges of innovation, transmission infrastructure, and resource adequacy.

Innovation

If we know what to do, we should do it. The problem is that we do not know exactly what to do. The history of the electricity sector is filled with good and bad surprises, shifting policies based on the assumed certainty of the moment. Given the fundamental challenges of climate and related issues, it is clear that new technologies and new investment approaches will be required [43].

A significant advantage of efficient electricity market design is allowing for open access and non-discrimination for ease of entry and exit, with rewards for innovation obtained primarily from the market. By contrast, when costs are socialized and the investment decisions are far removed from the market consequences, the system reverts to a machine for creating stranded assets. It is such mistakes in the past that precipitated the interest in efficient electricity markets [2].

Transmission Infrastructure

Expanding transmission investment is widely cited as a critical need for addressing the increased penetration of intermittent renewables. The best places for siting renewables are often far from loads [75]. Although markets investments are possible in some limited cases, markets alone cannot solve the challenges of transmission expansion.

A hybrid system is required, with a planning perspective for applying cost–benefit principles and sifting through alternative transmission investments. This is a large and important topic that goes beyond the discussion of electricity markets. A principal connection to the rest of the electricity market will be through the application of the basic principles of cost allocation summarized by the mantra that the beneficiaries pay. Transmission investment can be a complement or a substitute for generation and load investments, so there is a strong interaction with the market. The beneficiary pays approach can, in principle, mitigate conflicts between planning and market incentives [76, 77].

Resource Adequacy

The resource adequacy controversies concern the level of investment, especially in generation, required to meet various reliability and policy objectives. The reliability characteristics of variable energy resources differ from the traditional thermal generation in important ways, but this is a matter of degree rather than a structural difference. The implications would be to modify the level but not the type of reserve requirements. Hence, the dispatch and prices in real-time might change, but would be produced by and consistent with the existing market design.

These reliability issues go well beyond market design, but there are at least two important strands that connect to efficient markets having to do with inadequate incentives and incomplete markets.

The incentive problem arises from the failure to implement the necessary scarcity pricing and demand participation in real-time markets. Hence, prices have not been volatile enough and the result has been the familiar “missing money” problem where the revenues from the energy market are not sufficient to support efficient investment. The incomplete markets problem includes the possible substantial gap between the reliability levels embedded in efficient investments versus the traditional reliability standard which leads to much higher reserve margins [17].

The reforms outlined above for real-time scarcity and carbon pricing would address the first of these incentive problems. They would not be sufficient to address the reliability externality, but they would expand the policy choices if needed to support conservative reliability standards. For example, the Texas regulators have made deliberate policy choices to bias the parameters of the ORDC to provide a conservative (higher) estimate of the loss of load probability and thereby raise the real-time scarcity prices. By contrast, forward capacity mechanisms have been struggling to recreate the expected revenue and real-time performance incentives that follow naturally from better scarcity pricing.

The problems with forward capacity mechanisms and stimulating investment arise in part because ensuring specific performance of physical capacity contracts is beyond the capability of our knowledge. If we knew how to guarantee deliverability of specific generation determined years ahead in capacity auctions, we would not need organized markets to manage the complex conditions that arise in the real-time market. Recognizing that capacity mechanisms are in effect financial hedging contracts, as in Australia, would allow market reforms and the gradual atrophy of the existing capacity markets [78].

The anticipated effects of climate change include higher probabilities of extreme events [, p. 23]. Reliability standards can protect against many adverse events, but not everything. For truly extreme events, emergency response will continue to be essential and all that can be provided. Promising complete protection against unusual extreme events would be an expensive illusion.79

Summary

Electricity markets are necessarily hybrid systems that must incorporate a variety of policy objectives and conditions. The reforms to introduce greater competition were motivated in large part by the experience with vertically integrated monopoly and the technological change that allowed for competition in many if not all parts of the electricity system. The theory of efficient markets and the practice of its implementation have fundamentally changed the structure of operations, pricing, and investment. As part of the continuing evolution, expanded adoption of intermittent generation sources creates new concerns that the central elements of efficient electricity market design may need to be revisited. Although there are many technical and economic challenges, the broad conclusion here is that the basic framework of efficient real-time electricity markets becomes even more important in this greener future. The further reforms suggested, especially for better scarcity pricing, demand participation, and carbon pricing, would be fully compatible with and would capitalize on efficient markets design.