Long-term equilibrium in electricity markets with renewables and energy storage only

Abstract

In this paper, we study the optimal generation mix in power systems where only two technologies are available: variable renewable energy (VRE) and electric energy storage (EES). By using a net load duration curve approach, we formulate a least-cost optimization model in which EES is only limited by its power capacity. We solve this problem analytically and find least-cost and market equilibrium conditions that lead to the optimal capacities of VRE and EES. We show that, mathematically, an electricity price structure that depends on the period of the year (i.e. EES charging or discharging, VRE curtailment, load shedding) and on investments costs leads to cost recovery for VRE and EES. We show that when EES is the marginal technology (either charging or discharging) the price must be non-zero. More specifically, the equilibrium prices during EES charge or discharge are functions of the EES and VRE fixed costs. We confirm our analytical findings using a numerical model. We argue that, although the system we study is hypothetical and simplified, our findings provide insights and research directions for how to recover fixed costs in a future electricity market based on VRE and EES only.

1 Introduction

Most driving forces that apply to the power sector point towards a very large increase of variable renewable energy (VRE) sources such as wind and solar. First, unsubsidized costs of wind and solar are reaching unprecedented lows, with a 72%-respectively 90%-decrease over the last 12 years [15]. Second, energy policy in general pushes towards VRE: renewables contribute to the reduction of global emissions, one of the main objectives of the Paris Agreement: “Parties aim to reach global peaking of greenhouse gas emissions as soon as possible” [34]. Finally, exacerbated concerns of security of supply related to global gas and oil trade also push towards VRE that is also deemed to solve this concern. As such, VRE would contribute to meeting the essential goals of delivering a secure, sustainable and affordable electricity system [6, 29]. According to IRENA [8], 82% of all electricity generation capacity expansion in the world in 2021 was renewables, mostly wind and solar. In the United States, the deployment of clean electricity technologies is an essential part of the 50–52% target for reduction in U.S. Greenhouse Gas Pollution by 2030 [33]. Finally, IEA considers that “A massive expansion of clean electricity is essential to giving the world a chance of achieving its net zero goals” [7].

With thousands of GW of VRE installations expected towards 2050,¹ new integration challenges are expected, because of the very nature of VRE technologies. VRE has limited dispatchability, at least upwards,² and its output is entirely weather-driven and therefore variable and uncertain.. For this reason, VRE increases the need for flexible resources such as storage, interconnections, demand-side-management and peaker generation. Among flexibility solutions that enable highly renewable generation mixes, energy storage is receiving increasing interest from industry and research communities. Energy storage refers to a large variety of technologies, basically from Li-ion batteries to large hydro reservoirs. Li-ion batteries are a powerful solution for shorter storage durations (up to a few hours), but its cost advantage rapidly declines with increasing duration [18, 25]. Technologies with a discharge duration of more than 10 h are often referred to as long duration energy storage (LDES) [25]. LDES include many potential technologies, including heat energy storage, electrochemical storage, hydrogen storage, pumped storage hydro, geomechanical storage etc. [20]. In current systems, the only technology that provides very long duration storage to the grid is hydro storage, such as Hydro Québec in Canada, which is able to store 176 TWh across multiple years [17]. In general, most of the current power systems have very limited availability of very long duration energy storage (i.e. exceeding 100 h), although they have the greatest impact on lowering electricity costs in zero-carbon systems, according to Sepulveda et al. [23].

When excess VRE generation can not be stored locally or transferred to other regions using interconnections, the surplus leads to zero or negative prices in wholesale electricity markets; an effect that increases nonlinearly with the amount of VRE [12]. In markets based on marginal pricing, it is an open question whether the needed flexible resources will be able to earn the revenue they need to be deployed adequately [35]. These challenges add up to existing concerns regarding the design of electricity markets and their ability to provide sufficient incentives for resource adequacy [11]. Grubb [6] argues that there is currently “no obvious international example of a market structure fully appropriate for renewables at scale”. While there is already significant market-design literature for the case of medium levels of VRE integration [6, 13], there is less literature focusing on the case with a very large VRE share. Various regulatory and market design avenues are proposed, that range from a fully vertically integrated structure to a fully competitive system [9, 26, 29]. In between those two forms, there is the proposition in which two different market structures would handle the short-term and long-term management of power generation systems, which is summarized as “competition in the market” and “competition for the market” in Joskow [9].

In this paper, we consider the case where all electricity is provided by VRE and the only source of flexibility is electrical energy storage (EES). Under this assumption, there are no supply or demand resources with an explicit non-zero marginal costs in the system. We are interested in analyzing whether existing electricity market designs may still continue to function in such an extreme scenario. Our analysis addresses challenges specific to 100% VRE markets, addressing the issues related to investment signals for EES in zero-marginal cost systems, which are identified as complex problems in existing literature (e.g. [11]).

This work expands on our previous research on market equilibrium in low-carbon systems [14], to look analytically for long-term electricity market equilibrium and its implications for cost recovery, in a case where the capacities of VRE and EES are optimized without other investment alternatives available. This is in contrast with previous literature that often treats VRE capacity exogenously, as pointed by Joskow [19]. Our aim is to analyze the market properties of this “extreme” and stylized example, although we of course recognize that our underlying assumptions are simplistic and that all features of current markets do not apply directly to this case. The main innovation we introduce is to derive long-term equilibrium solutions for future systems with VRE and EES only, while also investigating the implications for price formation in corresponding electricity markets. The overall approach we take is based on traditional work on cost recovery and system optimality for thermal generators that have marginal costs based on their fuels [5, 24]. Through a simple formulation using net load duration curves we are able to derive analytical equilibrium solutions. However, a drawback is that a number of operational considerations are ignored, such as inertia constraints [22].

The rest of the paper is organized as follows: Sect. 2 is a general presentation of the analytical problem, which is solved in Sect. 3 for the case where load shedding is not allowed. Section 4 focuses on the situation with load shedding and corresponding scarcity prices when this occurs. Finally, Sect. 5 concludes the paper and discusses future research directions.

2 Methodology

We first describe a centralized planner’s problem which is to minimize the total expected cost of electricity generation considering a set of reliability constraints and physical limitations [5]. We study an “energy only” situation, meaning that only energy actually delivered is compensated: there is no pricing of either firm capacity or operating reserves. We include a “price cap” (assumed to be given by market regulations), although it could also be interpreted as a “demand price” given by a general bid for reduction of demand. It is a parameter that is difficult to estimate and to justify [2, 10]. Outside load shedding, no other demand side-management solution is present. This is a limitation because, unlike demand-side management programs, the load shedding we model applies to all consumers and does not make a distinction between power users [24].

In the classical model with conventional generators only, the solution to the centralized generation expansion planning problem leads to time-dependent markets prices equal to the variable cost of the marginal generator, and optimal capacities as illustrated in Fig. 1. Such a pricing structure is proven to lead to optimal incentives for investment in generators (fixed costs are recovered, average economic profit is zero) [5, 24]. ${t}_{s}$ is the optimal duration of blackouts, which is given by the fixed costs of the peaker divided by VOLL.

Fig. 1

Adapted from [14], load duration curve and optimal capacities of conventional plants. ${t}_{s}$ is the optimal duration of load shedding

Adding VRE and EES in addition to conventional generators leads to a more complicate formulation as described in [14]. Their analysis shows that when VRE and/or EES, that have no direct marginal costs, are added, the general price structure based on marginal costs is still conserved [14], as long as conventional generators remain present. In other words, conventional generators continue to be price setting resources, and these resources also influence the optimal price during EES charge and discharge through the principle of opportunity cost. However, the higher the VRE capacity, the more often are market prices zero or negative [9, 30, 31], putting the profitability at risk, mostly for peak generators. A similar approach is used in Newbery [22], who also introduces constraints related to inertia needs in the power system, leading to increased curtailment of VRE.

In this paper, our aim is to evaluate the results given by a similar model but this time with no conventional thermal generators, i.e. with no marginal costs. This 100% VRE system is interesting partly because of the rapid expansion of renewable sources, but also because it allows us to set aside the question of VRE plants possibly not being viable without support when in competition with conventional generators [12, 29, 30].

We are interested in the market equilibrium in a system where VRE and EES are serving a time-varying inflexible load, described by the following system of equations.

$$C ={F}_{e}{x}_{e}+{F}_{v}{x}_{v}+{v}_{s}{\int }_{0}^{T}{q}_{s}(t)dt$$

(1)

$$s.t. \, {q}_{d}\left(t\right)-{q}_{v}\left(t\right)-{q}_{e-}\left(t\right)+{q}_{e+}\left(t\right)-{q}_{s}\left(t\right)=0$$

(2)

$$-{q}_{k}\left(t\right)\le 0, k\in \left\{s,v,e+,e-\right\}$$

(3)

$${q}_{e+}\left(t\right)-{x}_{e}\le 0 , {q}_{e-}\left(t\right)-{x}_{e}\le 0$$

(4)

$${q}_{v}\left(t\right)-{{AF}_{v}(t)x}_{v}\le 0$$

(5)

$$\frac{d{E}_{e}(t)}{dt}={\eta }_{e}\cdot {q}_{e+}\left(t\right)-{q}_{e-}\left(t\right)$$

(6)

$${\eta }_{e}{\int }_{0}^{T}{q}_{e+}\left(t\right)dt-{\int }_{0}^{T}{q}_{e-}\left(t\right)dt=0$$

(7)

where indices s, v, e refer to load shedding, VRE and EES respectively.

The objective function $C$ in Eq. (1) is the sum of three terms. The first two terms are the annualized fixed costs for EES capacity (${x}_{e}$) and VRE capacity (${x}_{v}$), where we neglect variable costs. The third term in Eq. (1) represents the total costs associated with load shedding ${q}_{s}(t)$, where ${v}_{s}$ represents the shedding costs, set administratively to VOLL or some other level.

Note that we assume no separate investment costs for energy that can be stored in the EES, that is only the MW capacity of storage is considered. In other words, the duration of the storage device is not optimized, but assumed sufficient to meet the system need. Though simplified in many cases, this approach is relevant to existing systems dominated by hydropower. For instance, the Norwegian hydropower reservoirs are designed for inter-annual and seasonal storage—traditionally from summer to winter. North-European wind power counteracts the seasonal storage needs as winds tend to be higher in winter. Thus, wind power reduces the need for seasonal storage capacity, and thereby reduces the need for long-term energy storage from existing reservoirs. On the other hand, power capacity, and especially pumping capacity, becomes a more limiting factor. However, we recognize that the omission of the energy storage constraint is a limitation and that further development of the theoretical framework to include MWh constraints of storage will be part of future research.

Equation (2) states that, at each instant t, the demand ${q}_{d}\left(t\right)$ must be equal to renewable generation ${q}_{v}(t)$ plus the net charging of EES, computed as ${q}_{e+}\left(t\right)-{q}_{e-}(t)$.

Charging and discharging ${q}_{e+}$ and ${q}_{e-}$ are bounded by the capacity of EES ${x}_{e}$, as seen from Eq. (4). The upper limit of renewable generation ${q}_{v}(t)$ is described by Eq. (5). In this respect, we make two assumptions. First, we assume VRE generation can be fully reduced at no cost (“curtailment”) when needed [21]. Then, ${q}_{v}\left(t\right)\le {AF}_{v}(t)\cdot {x}_{v}$ where ${AF}_{v}(t)$ is the availability factor at time instant $t$. This means that ${AF}_{v}(t)$ applies regardless of the installed capacity, which is a simplification which in general holds better for large installed VRE capacities, unless the availability of sites with good resource starts to reduce. This linear scaling is used in other research works [1, 3, 4, 32].

The conservation of the energy stored from one step to the next is described by Eq. (6), where ${\eta }_{e}$ is the round-trip efficiency; finally, Eq. (7) imposes that all energy stored during a year must be released within the same year, i.e. the storage content must be the same at the beginning and the end of the year.

In addition to the idealized EES representation discussed above, this set of equations comes with additional simplifications; in particular, the impacts of the transmission network are ignored and operating reserve constraints are neglected.

3 The case with no load shedding

To start exploring the cost recovery implications of this problem, we make an additional preliminary simplification, which is to remove the possibility of load shedding by removing the term ${q}_{s}(t)$ from the set of equations. Removing load shedding makes the model simpler to analyze and serves as a benchmark for the analysis that follows in the next section, which considers load shedding.

The situation with no load shedding focuses on the system equilibrium between VRE and EES and describes a situation where high prices (load shedding or price caps in general) are removed by a regulatory/policy decision. It is actually called a “regulatory solution” according to [24].

Schematically, the dispatch follows variations in net demand as illustrated in Fig. 2: extra wind energy is used to charge the storage (between ${t}_{e}$ and $T$), because it is better to use VRE directly instead of suffering losses through EES.

Fig. 2

Load duration curve for demand (grey line) and net demand (black) line for a system with EES and VRE plants only, without load shedding. The optimal capacity of EES is indicated in the figure as ${x}_{e}$

The detailed analytical solution of the problem is presented in Appendix 2, and leads to the following optimality conditions:

$$\frac{{F}_{v}}{{t}_{e}{AF}_{v}^{\left[0,{t}_{e}\right]}+{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}}=\Lambda =\frac{{F}_{e}}{{\eta }_{e}(T-{t}_{v})}$$

(8)

$${\eta }_{e}{\int }_{{t}_{e}}^{T}{q}_{e+}\left(t\right)dt-{\int }_{0}^{{t}_{e}}{q}_{e-}\left(t\right)dt=0$$

(9)

A system with VRE and EES capacities $({x}_{v}$,${x}_{e})$ leading to durations $({t}_{v},{t}_{e})$ that fulfill conditions (8) and (9) is optimal, meaning that it minimizes total costs $C$ and meets constraints (2)–(7). From the point of view of markets, this is a neutral starting point: we have simply formulated and solved a capacity expansion problem expressed as a cost-minimization set of equations.

If we now apply the paradigm of existing spot markets (marginal cost pricing) to such a system, we must set price to 0 all the time in the absence of shedding or variable costs. In turn, it leads to the market revenue of both VRE and EES to be 0, and their profit to be negative: neither VRE nor EES does recover its investment costs through the market. In addition, the Weighted Average Price of Electricity (WAPE) is 0 all the time, with WAPE defined as follows:

$$WAPE=\frac{{\int }_{0}^{T}{p}_{e}\left(t\right){q}_{d}\left(t\right)dt}{{E}_{d}}$$

(10)

This is in contrast with the average cost of electricity (ACE) that must reflect investment costs and is therefore nonzero and should be transferred one way or another to end-users. ACE is defined as follows:

$$ACE=\frac{{\sum }_{i\in \left\{v,e\right\}}{F}_{i}\cdot {x}_{i}}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}$$

(11)

Setting the market price to 0 all the time therefore separates the investment and the operational problems, as the market clearing prices resulting from the operational problem clearly do not recover the investment costs.

Now, note that $\Lambda$, defined as in Eq. (8), is mathematically the Lagrange multiplier associated with the supply–demand equilibrium constraint. This means that $\Lambda$ represents the marginal cost of increasing the load when the net load is positive, i.e. a price for electricity that reflects the investment cost in the system, as shown in Eq. (8). Consequently, the corresponding marginal cost of electricity when the EES is the marginal load will be $\Lambda$ times the storage efficiency. Hence, based on this approach, we can set the following time-dependent pricing structure ${p}_{e}\left(t\right)$:

Appendix 3 shows that using ${p}_{e}\left(t\right)$ as defined above in Table 1 leads to zero profit of VRE and EES, meaning that both VRE and EES exactly recover their total costs. Appendix 2 also details the calculations that lead to the weighted average price of electricity to be equal to the average cost of electricity, that is:

Table 1 Proposed prices corresponding to dispatch, with no load shedding

$$WAPE=ACE$$

(12)

The price structure proposed in Table 1 is therefore a mathematically acceptable solution to the system planner’s problem with no load shedding. Said otherwise, if VRE and EES are paid such as in Table 1 for every hour of the year, they will eventually recover their costs. Note that we do not address the unicity of this solution which is left for further work. In Appendix 4, we verify that this analytical solution coincides with the multiplier found by a numerical model, with 99.8% accuracy.

4 Addition of load shedding

Here, we add load shedding as a price cap which was not considered in the previous section. Defining price caps is a challenging trade-off between too-low and too-high. In general, prices caps below VOLL lead to the missing money problem in wholesale markets, and lead to required out-of-market actions due to the price suppression. At the same time, high prices at the VOLL level may lead to the exercise of market power [10, 29]. Finally, caps may be changed by policy makers and regulators over time [29].

In the situation we describe (VRE and EES only), we must first note that load shedding can happen in different ways. In the typical description of a system based on fossil generators (see Sect. 2), load shedding happens when load exceeds the total available generation capacity, ignoring energy supply issues such as limited gas supply, forced or planned outages of power plants, or depleted hydro reservoirs; such situations are usually ignored in power market models based on the load duration curve. In the 100% VRE case, load shedding can also happen because the energy content of the storage is insufficient, i.e. not only because the total EES power capacity ${x}_{e}$ at the time of peak net-load is insufficient. In this paper, the energy capacity of EES is not treated explicitly in the analysis. Rather, we are interested in the traditional capacity-limited load shedding, as described in Fig. 3 (left) below. In Appendix 5, we provide detailed results regarding how the type of shedding is influenced with relative VRE and EES costs, and that it applies at least in the case with VRE investments costs low with respect to EES costs, in which there is no energy-limited load shedding.

Fig. 3

Description of two possible load shedding situations. Left: case with shedding due to insufficient generation capacity from energy storage. Right: case with shedding due to insufficient energy storage capacity

A detailed derivation of the optimal dispatch is given in Appendix 1 and is summarized in Table 2.

Table 2 Optimal dispatch

We solve the problem analytically which leads to the following optimality conditions (see Appendix 1 for the detailed derivation):

$$\frac{{F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}}{\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}}=\Lambda =\frac{{F}_{e}-{v}_{s}{t}_{s}}{{\eta }_{e}(T-{t}_{v})-{t}_{s}}$$

(13)

$${\eta }_{e}{\int }_{{t}_{e}}^{T}{q}_{e+}\left(t\right)dt-{\int }_{0}^{{t}_{e}}{q}_{e-}\left(t\right)dt=0$$

(14)

Similarly to the previous section, we obtain the value $\Lambda$, the Lagrangian multiplier associated with the supply–demand equilibrium condition, now given by Eq. (13). We check that it is equal to the solution found by a numerical solver.

The proposed price structure is as in the Table 3 below. We observe that this set of prices, leads to WAPE = ACE and to a cost recovery for both VRE and EES, as detailed in Appendix 2. It is composed of four segments: the shedding period $[0;{t}_{s}]$ in which price is ${v}_{s}$ because load shedding happens. The discharge and charge periods $\left[{t}_{s},{t}_{e}\right]$ and $\left[{t}_{e},{t}_{v}\right]$, in which prices are $\Lambda$ and ${\eta }_{e}\Lambda$ respectively. Finally, the curtailment period in which excess generation leads to a price of 0.

Table 3 Proposed prices corresponding to dispatch, with no shedding

The proposed structure is different to pricing electricity in a uniform way using either 0 $/MWh or the ACE. On the contrary, it is time dependent, with higher values occurring for higher loads and 0 when there is VRE curtailment. As already discussed in the previous section, the main finding here is that, unlike the case with conventional generators, we can no longer separate the planning and operational problems. This is because the Lagrangian used to determine the short-term price, $\Lambda$, is dependent on fixed capital costs.

Still, knowing whether ${p}_{e}\left(t\right)$ is an actual price poses two types of challenges. First, the price structure is dependent on the load and VRE time series chosen as an input in the optimization phase, since $\Lambda$ is dependent on ${t}_{v},{t}_{e}$ (which in turn depend on optimal capacities ${x}_{v},{x}_{e}$). This first issue, though, also appears when optimizing generation mixes based on conventional generators. However, adding another time series subject to variation from one year to another (VRE generation through ${AF}_{v}(t)$) increases the uncertainty. Generally, that issue could be partially solved using either a set of different weather years or superimposed capacity constraints when calculating the prices dependent on $\Lambda$. Most likely, future power markets will require new security of supply “modules”, for example capacity mechanism or decarbonization module, such as introduced in [29].

More specific to this 100% VRE case is the fact that generators (either VRE or EES) should get an incentive to offer their energy at prices following Table 1 instead of offering the lowest possible prices, a situation which could eventually lead to a collapse of prices to 0 throughout the year. While it is not the objective of this paper to enter into detailed market design propositions, we argue that the supervision of prices offered by generators already exists. For example, NYISO Market Power Mitigation Measures help to avoid “unnecessary interference with competitive price signals” [27, 28]. Among the categories that may warrant mitigation are “Physical Withholding” (“not offering to sell or schedule the output of or services provided by an Electric Facility capable of serving”) and Economic withholding (“submitting Bids for an Electric Facility that are unjustifiably high”). While the proposed price structure does not pose a problem of economic withholding but rather the opposite, the very existence of power mitigation measures is an interesting element.

5 Conclusion

In many countries, the population, and in turn governments and policy makers offer an increasing support to VRE generation. This movement will likely increase in the future, since VRE addresses three main criteria for electricity planning: affordability, security of supply, and low carbon emissions.

Currently, most VRE generation is procured through calls for tenders for power purchase agreements (PPAs). In many jurisdictions, this type of procurement coexists as a matter of fact with other generators that are mostly compensated through spot markets, although other instruments such as for example capacity markets and zero emissions credits may be used to recover costs. While this approach has been successful to some extent in delivering large amounts of new generation, new procurement schemes may appear in the future in the transition towards low-carbon solutions.

In this paper, we analyze a stylized case with 100% renewable energy supported by EES, and in which all the incentives for the build-up of the power system comes from short-term prices. We show that a solution to minimizing investment and load shedding costs is given by VRE and EES capacities that fulfill two equilibrium conditions, namely energy storage conservation and one given by the dual value of the supply demand equilibrium. In turn, these conditions lead to short-term prices that incorporate fixed costs, which ensure that VRE and EES recover their costs. These equilibrium conditions differ from the cases with conventional generators in the mix, since in those cases the operational problem and corresponding short-term prices can be separated from the fixed costs of the generation technologies.

While we argue that our research is an interesting exploration of a competitive equilibrium between resources that have no variable costs, there are various complementary research avenues that should be explored in future work. First, the competition between a set of possible VRE generators and EES technologies instead of one of each type must be analyzed. Second, adding explicit investment costs for the storage energy capacity of the EES is an area that must be explored. Also, the fixed costs of a storage system will be influenced by the way it will be operated, which adds complexity to our analysis. Finally, adding price responsive demand [29] will be key since inelastic demand is often recognized as a barrier to the wider deployment of VRE generation and price responsive demand can influence both price formation and the need for EES in the system.

Appendices

Appendix 1: Analytical derivation of optimality conditions, in the case with load shedding

In this section, we provide the detailed derivation of equilibrium conditions for the case with load shedding, as discussed in Sect. 4. The case with no shedding is simply a sub ensemble of what is presented below.

The Lagrangian of the operational problem for an arbitrary time instant $t$ is:

$$\begin{aligned} {L}_{op}\left(t\right)&={v}_{s}{q}_{s}\left(t\right)+{\lambda }_{d}\left(t\right)\left({q}_{d}\left(t\right)-{q}_{s}\left(t\right)-{q}_{v}\left(t\right)-{q}_{e-}\left(t\right)+{q}_{e+}\left(t\right)\right)\\&\quad +\sum _{l}{\mu }_{l}^{max}\left({q}_{l}\left(t\right)-{x}_{l}\right)+{\mu }_{v}^{max}\left({q}_{v}\left(t\right)-{AF}_{v}\left(t\right){x}_{v}\right)\\&\quad +\sum _{k}{\mu }_{k}^{min}\left({-q}_{k}\left(t\right)\right)+\theta \left(t\right)\left[\frac{d{E}_{e}(t)}{dt}-{\eta }_{e}\cdot {q}_{e+}\left(t\right)+{q}_{e-}\left(t\right)\right] \\&\quad with\; l\in \left\{e+,e-\right\}\; and, \;k\in \left\{s,v,e+,e-\right\} \end{aligned}$$

(15)

The KKT-conditions for this problem consist of (8)–(13), in addition to the following equations:

$$-{\lambda }_{d}\left(t\right)+{\mu }_{e-}^{max}-{\mu }_{e-}^{min}+\theta \left(t\right)=0$$

(16)

$${\lambda }_{d}\left(t\right)+{\mu }_{e+}^{max}-{\mu }_{e+}^{min}-{\eta }_{e}\theta \left(t\right)=$$

(17)

$$-{\lambda }_{d}(t)+{\mu }_{v}^{max}-{\mu }_{v}^{min}=0$$

(18)

$${v}_{s}-{\lambda }_{d}(t)-{\mu }_{s}^{min}=0$$

(19)

$${\mu }_{l}^{max}\cdot \left({q}_{l}\left(t\right)-{x}_{l}\right)=0 \forall l\in \left\{e,e-\right\}$$

(20)

$${\mu }_{v}^{max}\cdot \left({q}_{v}\left(t\right)-{AF}_{v}(t){x}_{v}\right)=0$$

(21)

$${\mu }_{k}^{min}\cdot \left(-{q}_{k}\left(t\right)\right)=0 \forall k\in \left\{s,e,e-,v\right\}$$

(22)

$$\theta \left(t\right)\frac{d{E}_{e}(t)}{dt}-{\eta }_{e}\cdot {q}_{e+}\left(t\right)-{q}_{e-}\left(t\right) )=0$$

(23)

From the KKT-conditions, we get the dispatch according to the merit-order. The resulting optimal dispatch levels of EES and VRE in each period are provided in the table below, where the time parameter $t$ is sorted after the net demand.

Shedding	Discharge	Charge	Curtailment
$t\le {t}_{s}:$ ${x}_{e}\le {q}_{nd}\left(t\right)$	${t}_{s}<t\le {t}_{e}:$ $0\le {q}_{nd}\left(t\right)<{x}_{e}$	${t}_{e}<t\le {t}_{v}:$ $-{x}_{e}\le {q}_{nd}\left(t\right)<0$	$t>{t}_{v}:$ ${q}_{nd}\left(t\right)<-{x}_{e}$
${\mu }_{s}^{min}=0$	${\mu }_{s}^{min}$	${\mu }_{s}^{min}$	${\mu }_{s}^{min}$
${\mu }_{e-}^{min}=0$	${\mu }_{e-}^{min}=0$	${\mu }_{e-}^{min}$	${\mu }_{e-}^{min}$
${\mu }_{e+}^{min}$	${\mu }_{e+}^{min}$	${\mu }_{e+}^{min}=0$	${\mu }_{e+}^{min}=0$
${\mu }_{v}^{min}=0$	${\mu }_{v}^{min}=0$	${\mu }_{v}^{min}=0$	${\mu }_{v}^{min}=0$
${\mu }_{e-}^{max}$	${\mu }_{e-}^{max}=0$	${\mu }_{e-}^{max}=0$	${\mu }_{e-}^{max}=0$
${\mu }_{e+}^{max}=0$	${\mu }_{e+}^{max}=0$	${\mu }_{e+}^{max}=0$	${\mu }_{e+}^{max}$
${\mu }_{v}^{max}$	${\mu }_{v}^{max}$	${\mu }_{v}^{max}$	${\mu }_{v}^{max}=0$
${p}_{e}\left(t\right)={v}_{s}$	${p}_{e}\left(t\right)=\theta \left(t\right)$	${p}_{e}\left(t\right)={\eta }_{e}\theta \left(t\right)$	${p}_{e}(t)=0$
${q}_{v}\left(t\right)={AF}_{v}(t){x}_{v}$	${q}_{v}\left(t\right)={AF}_{v}(t){x}_{v}$	${q}_{v}\left(t\right)={AF}_{v}(t){x}_{v}$	${q}_{v}\left(t\right)={q}_{d}\left(t\right)+{x}_{e}$
${q}_{e-}\left(t\right)={x}_{e}$	${q}_{e-}\left(t\right)={q}_{nd}\left(t\right)$	${q}_{e-}\left(t\right)=0$	${q}_{e-}\left(t\right)=0$
${q}_{e+}\left(t\right)=0$	${q}_{e+}\left(t\right)=0$	${q}_{e+}\left(t\right)=-{q}_{nd}\left(t\right)$	${q}_{e+}\left(t\right)={x}_{e}$
${q}_{s}\left(t\right)={q}_{nd}\left(t\right)-{x}_{e}$	${q}_{s}\left(t\right)=0$	${q}_{s}\left(t\right)=0$	${q}_{s}\left(t\right)=0$

Of particular interest is the following subset of equations which is the optimal dispatch (Table

Table 4 Optimal dispatch

4).

What it shows is that the short-term price ${\lambda }_{d}\left(t\right)$ during shedding is ${v}_{s}$, while it is 0 during VRE curtailment, in line with previous work [14]. More interesting at this point of our analysis is that the short-term price is proportional to $\theta \left(t\right)$ during $[{t}_{s}, {t}_{e}]$ and $[{t}_{e}, {t}_{v}]$ (i.e. during marginal charging or discharging of EES). During these periods ${\lambda }_{d}\left(t\right)$ is necessarily nonzero: since the storage constraint is always the limiting one, $\theta \left(t\right)$ is necessarily non-zero during those periods. ${\lambda }_{d}\left(t\right)$ is therefore undetermined but non-zero (Table

Table 5 Value of ${\lambda }_{d}\left(t\right)$ for each time period

5).

The optimal dispatch described above contains the operational constraints solved for optimal operation over the whole duration [0,T]. We can use these operational conditions to express the total cost minimization problem as:

$$C ={F}_{v}{x}_{v}+{F}_{e}{x}_{e}+{v}_{s}{\int }_{0}^{{t}_{s}}\left[{q}_{d}\left(t\right)-{AF}_{v}\left(t\right){x}_{v}-{x}_{e}\right]dt$$

(24)

$${\text{s}}.{\text{t}}. {\eta }_{e}{\int }_{{T}}^{T}{q}_{e+}\left(t\right)dt-{\int }_{0}^{{T}}{q}_{e-}\left(t\right)dt=0$$

(25)

The optimization problem can now be written in the Lagrangian form,

$$L\left({x}_{e},{x}_{v}\right)=-C\left({x}_{e},{x}_{v}\right)-\Lambda g\left({x}_{e},{x}_{v}\right)$$

(26)

where $\Lambda$ is the Lagrange multiplier associated with the storage conservation constraint (7). Using the Leibniz integral rule, the following equations hold for the derivatives used to derive the optimality conditions:

$$\frac{\partial g\left({x}_{e},{x}_{v}\right)}{\partial {x}_{e}}={t}_{s}-{\eta }_{e}(T-{t}_{v})$$

(27)

$$\frac{\partial g\left({x}_{e},{x}_{v}\right)}{\partial {x}_{v}}=- ({t}_{e}-{t}_{s}){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}-{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}$$

(28)

$$\frac{\partial C\left({x}_{e},{x}_{v}\right)}{\partial {x}_{e}}={F}_{e}-{v}_{s}{t}_{s}$$

(29)

$$\frac{\partial C\left({x}_{e},{x}_{v}\right)}{\partial {x}_{v}}={F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}$$

(30)

where we have defined${\int }_{{t}_{1}}^{{t}_{2}}{AF}_{v}\left(t\right)dt=\left({t}_{2}-{t}_{1}\right){\cdot AF}_{v}^{\left[{t}_{1},{t}_{2}\right]}$. The first-order optimality conditions for this problem are $\frac{\partial L\left({x}_{e}*,{x}_{v*,}\lambda *\right)}{\partial {x}_{i}}=0 (i=e,v)$ and$\frac{\partial L\left({x}_{e}*,{x}_{v*,}\lambda *\right)}{\partial \lambda *}=0$, which leads to:

$${F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}=\Lambda \left[\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}\right]$$

(31)

$${F}_{e}-{v}_{s}{t}_{s}=\Lambda \left[{\eta }_{e}(T-{t}_{v})-{t}_{s}\right]$$

(32)

$${x}_{e} \left({F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}\right)+{x}_{v}\left(\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{\eta }_{e}\left({t}_{v}-{t}_{e}\right){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}\right)={E}_{d}^{{0,t}_{s}}+{\eta }_{e} {E}_{d}^{{{t}_{e},t}_{v}}$$

(33)

With ${E}_{d}^{[{t}_{a}{,t}_{b}]}={\int }_{{t}_{a}}^{{t}_{b}}{q}_{e}\left(t\right)dt$

In particular, Eqs. (31) and (32) lead to

$$\Lambda =\frac{{F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}}{\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}}=\frac{{F}_{e}-{v}_{s}{t}_{s}}{{\eta }_{e}(T-{t}_{v})-{t}_{s}}$$

(34)

As seen from Table 1, our modeled generation mix has in the end two unknowns ${x}_{e}$ and ${x}_{v}$ which in turn determine ${t}_{e}$, ${t}_{s}$ and ${t}_{v}$ or vice-versa. The two equilibrium conditions above should therefore lead to optimal capacities ${x}_{e}^{*}$ and ${x}_{v}^{*}$.

Appendix 2: Cost recovery of VRE and VRE in the case with load shedding

The profit function of EES is

$${\pi }_{e}={\int }_{0}^{T}{\lambda }_{d}\left(t\right)\left({q}_{e-}\left(t\right)-{q}_{e+}\left(t\right)\right)dt-{F}_{e}\cdot$$

(35)

where the instantaneous charging and discharging power is given by the storage operation strategy and is generally a function of the storage capacity (power and energy) and the market price. By using the segments from the optimal dispatch, the profit function can be expressed as

$${\pi }_{e}={v}_{s}{\int }_{0}^{{t}_{s}}{x}_{e}dt+{p}_{e}{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e-}(t)dt-{p}_{e}{\eta }_{e}{\int }_{{t}_{e}}^{{t}_{v}}{q}_{e+}\left(t\right)dt-{F}_{e}\cdot {x}_{e}$$

(36)

where ${p}_{e}$ is the short-term electricity price during period $[{t}_{p},{t}_{e}]$, when the storage is the marginal contribution to the system. From the KKT-conditions, we also know that the price during period $[{t}_{e},{t}_{v}]$ should be equal to ${p}_{e}{\eta }_{e}$ (see Table 1).

By using the optimal dispatch (Table 1), we can show that ${\pi }_{e}$ is equal to:

$${v}_{s}{x}_{e}{t}_{s}+{p}_{e}{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e-}(t)dt-{p}_{e}{\eta }_{e}\left[\frac{{x}_{e}{t}_{s}}{{\eta }_{e}}+{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e-}\left(t\right)dt-{x}_{e}(T-{t}_{v})\right]-{F}_{e}{x}_{e}$$

(37)

By applying the storage conservation ${\eta }_{e}{\int }_{{t}_{e}}^{tv}{q}_{e+}\left(t\right)dt={x}_{e}{t}_{s}+{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e-}\left(t\right)dt-{x}_{e}(T-{t}_{v})$, the profit function becomes:

$${\pi }_{e}={p}_{e}({x}_{e}\left(T-{t}_{v})-{x}_{e}{t}_{s}\right)-{F}_{e}\cdot {x}_{e}+{{v}_{s}x}_{e}{t}_{s}$$

(38)

By setting ${\pi }_{e}=0$ we obtain

$${p}_{e}=\frac{{F}_{e}-{v}_{s}{t}_{s}}{{\eta }_{e}(T-{t}_{v})-{t}_{s}}=\Lambda$$

(39)

Therefore, the short-term price during marginal discharge that allows cost recovery is equal to $\Lambda$, the Lagrange multiplier associated with the storage conservation constraint introduced in Eq. (31).

The general profit function for VRE is:

$${\pi }_{v}={\int }_{0}^{T}{\lambda }_{d}\left(t\right){q}_{v}\left(t\right)dt-{F}_{v}\cdot {x}_{v}$$

(40)

Following the same arguments as above, we get:

$${\pi }_{v}={v}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}{x}_{v}{t}_{s}+{p}_{e}{AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}{x}_{v}\left({t}_{e}-{t}_{s}\right)+{p}_{e}{\eta }_{e}{AF}_{v}^{\left[{t}_{e},{t}_{v}\right]}{x}_{v}\left({t}_{v}-{t}_{e}\right)-{F}_{v}{x}_{v}$$

(41)

Cost recovery, ${\pi }_{v}=0$, gives the same result as above:

$${p}_{e}=\frac{{F}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}}{\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{\eta }_{e}({t}_{v}-{t}_{e}){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}}=\Lambda$$

(42)

Therefore, the short-term price during marginal charging and discharging is found to be $\Lambda$, the Lagrange multiplier for the energy storage conservation constraint.

Appendix 3: Average cost of electricity and weighted average price of electricity in the case with load shedding

If the analytical solutions derived previously imply that the market solution gives the optimal solution, then the weighted average price of electricity (WAPE) for the consumers in the market will be equal to the annual cost of electricity (ACE):

$$ACE=\frac{{v}_{s}{\int }_{0}^{{t}_{s}}{q}_{s}\left(t\right)dt+{\sum }_{i\in \left\{v,e\right\}}{F}_{i}\cdot {x}_{i}}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}$$

(43)

The general equation for WAPE used here is

$$WAPE=\frac{{\int }_{0}^{T}{\lambda }_{d}\left(t\right){q}_{d}\left(t\right)dt}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}$$

(44)

where the price is ${\lambda }_{d}\left(t\right)$. In our case, WAPE becomes:

$$WAPE=\frac{{v}_{s}{\int }_{0}^{{t}_{s}}{q}_{d}\left(t\right)dt+\Lambda {\int }_{{t}_{s}}^{{t}_{e}}{q}_{d}\left(t\right)dt+\Lambda {\eta }_{e}{\int }_{{t}_{e}}^{{t}_{v}}{q}_{d}\left(t\right)dt}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}$$

$$=\frac{{v}_{s}{\int }_{0}^{{t}_{s}}{q}_{d}\left(t\right)dt+\Lambda [{x}_{v}\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{x}_{v}{\eta }_{e}\left({t}_{v}-{t}_{e}\right){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}+{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e}\left(t\right)-{\eta }_{e}{\int }_{{t}_{e}}^{{t}_{v}}{q}_{e-}\left(t\right)]}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}$$

(45)

Using the storage conservation constraint:

$${\eta }_{e}{\int }_{{t}_{e}}^{tv}{q}_{e+}\left(t\right)dt={x}_{e}{t}_{s}+{\int }_{{t}_{s}}^{{t}_{e}}{q}_{e-}\left(t\right)dt-{\eta }_{e}{x}_{e}(T-{t}_{v})$$

(46)

One gets

$$\begin{aligned} WAPE &= \frac{{v}_{s}{\int }_{0}^{{t}_{s}}{q}_{d}\left(t\right)dt+\Lambda [{x}_{v}\left({t}_{e}-{t}_{s}\right){AF}_{v}^{\left[{t}_{s},{t}_{e}\right]}+{x}_{v}{\eta }_{e}\left({t}_{v}-{t}_{e}\right){AF}_{v}^{\left[{t}_{v},{t}_{e}\right]}-{x}_{e}{t}_{s}+{{\eta }_{e}x}_{e}(T-{t}_{v})]}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}\\& =\frac{{v}_{s}{\int }_{0}^{{t}_{s}}{q}_{s}\left(t\right)dt+{v}_{s}{x}_{e}{t}_{s}+{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}+{F}_{e}{x}_{e}-{v}_{s}{x}_{e}{t}_{s}+{F}_{v}{x}_{v}-{v}_{s}{t}_{s}{AF}_{v}^{\left[0,{t}_{s}\right]}}{{\int }_{0}^{T}{q}_{d}\left(t\right)dt}\end{aligned}$$

(47)

which leads to $WAPE=ACE$.

Appendix 4: Analysis of equilibrium conditions using a numerical model

To illustrate how the two equilibrium conditions lead to the solution of our least-cost optimization problem, we use an example with the following input costs and technology data (Table 6):

Table 6 Cost data and power system data for the illustration, and the numerical example in next section

Time series are from the ENTSO-E 2040 GCA dataset for load and JRC EMHIRES data set (European Commission 2019) for wind generation, following the approach in [11].

Those two conditions are hardly solvable explicitly, so we shall solve it using an exploration of the (${x}_{e}$, ${x}_{v}$) plane. This is the purpose of the graph below which uses VRE capacity (${x}_{v}$, x-axis) and EES capacity (${x}_{e}$, y-axis). Darker color is used when condition of Eq. (8) or Eq. (9) is fulfilled. We see that there is one single intersection between the two conditions, leading to the solution (Fig. 4).

Fig. 4

Exploration of possible xv (VRE capacity, in MW) and xe (EES capacity, in MW); the close to linear curve is when condition (20) holds; the other one is when condition (19) holds. The region in which each conditions is valid is very narrow so we had to use a logarithmic gray scale to enhance constrast

The optimal solution, for which both conditions are valid, is in this case given by (Table 7):

Table 7 Equilibrium solution in our example

To assess the solution found analytically, we have modelled the system as a time-sequential Linear Programming (LP) problem based on the basic formulation (1)-(7) for comparison. The LP optimization model is implemented in Matlab and is identical as the model in Julia v.0.64 used in [11].

The input costs and technology data are summarized in Table 6. From the numerical results, we can extract the values of ${t}_{s}$, ${t}_{e}$ and ${t}_{v}$. We first verify that optimal dispatch obtained using the LP model is compatible with the dispatch described in the previous section. VRE is dispatched first, as expected, and EES is used as modeled in our optimal dispatch (Fig. 5).

Fig. 5

Duration curves extracted from the LP model

We can then compare the optimal VRE and EES capacities found using the numerical model with those found using the equilibrium conditions. The relative error is less than a percent (Table 8).

Table 8 Cost data and power system data for the numerical example

As shown above, $\Lambda$ is the short-term price that applies during the discharging period. It coincides with prices computed from the LP model as the dual value of the supply–demand equilibrium constraint (Fig. 6).

Fig. 6

Results from the LP model displaying both the net demand curve and short-term prices as computed from the dual value of the supply–demand equilibrium constraint

Regarding cost of electricity, we checked for various conditions that WAPE (Weighted Average Price of Electricity) for the costumers in the market is equal to the Annual Cost of Electricity (ACE), as show analytically. For example, this was tested against varying investment costs for EES (Fig. 7):

Fig. 7

ACE and WAPE computed from the results of the LP model are equal

Appendix 5: Numerical analysis of two load shedding options

In general, load shedding is parametrized using the VOLL, and happens when there is no longer enough generation capacity available to meet demand. However, there are two causes for not being able to meet demand, namely energy limitation and capacity limitation. To analyze this situation in more details, we use the numerical model described in the previous section (Appendix 4): we explore a large set of combinations of investments costs for VRE (Fv) and EES (Fe). The result is the heat map of Fig. 8, which exhibits two main zones; the one we are mostly interested in is the top left zone, where shedding happens because EES capacity is too small, as depicted in the top-left insert. While there exists another situation (top-right panel), our research focuses on the case of shedding because of limited capacity, which happens when VRE is rather inexpensive and EES is rather expensive.

Fig. 8

Heat map of the amount of shed load as a function of VRE investment costs (Fv, x-axis, $/MW-yr) and EES investment costs (Fe, y-axis, $/MW-yr). Shedding is equal to 0 for black areas. Top-left: shape of the solution for expensive EES. Top-right: shape of the solution for expensive RES