1 Introduction

The need for large-scale energy storage to balance intermittent and stochastic renewables in future energy systems has become apparent. The IEA Energy Outlook predicts that utility-scale battery storage will increase from less than 20 GW in 2020 to over 3000 GW by 2050 [1]. It remains an open question how to best integrate storage in energy markets. The urgency of this question is illustrated by a recent order from the Federal Energy Regulatory Commission that requires system operators to facilitate market participation of electric storage, and to provide fair compensation for the provided services, given the physical and operational characteristics of these assets [2].

In [3], three possible market designs for storage systems in electricity markets are identified. They differ in their representation of storage in the objective function and constraints of the market clearing. In the first design, there is no separate bidding format for storage systems, so that storage operators must submit price-quantity bids in the same way as loads and generators. This approach is common in current energy markets. The second market design additionally includes storage state-of-energy constraints to the market clearing, ensuring feasibility of the storage dispatch. In the corresponding bidding format, storage owners must therefore submit relevant operational parameters. We use the term merchant storage for a storage submitting price bids in the market, which is the case in these two market designs. One major disadvantage of merchant storage is that it can have negative effects on social welfare [4, 5], depending on its market power.Footnote 1

To mitigate these effects, the concept of non-merchant or passive storage in energy markets has been investigated in recent years [7,8,9,10]. It is the third market design option in [3]. A non-merchant storage does not submit price bids, but its operation is co-optimized with generation and loads to achieve the highest social welfare. An appealing property of this set-up is that it is comparable to network-aware markets, which are prevalent in practice, e.g., in the U.S. markets. Similar to the way that power lines are included in the market to perform spatial arbitrage, a non-merchant storage performs temporal arbitrage. Thus, the storage is scheduled to achieve more efficient use of energy systems. Non-merchant storage can be a public asset owned by the system or network operator, but it could also be privately owned.

Time-linking constraints are inevitably part of optimal dispatch problems for storage. With merchant storage, these time-linking constraints can be considered in the individual optimal bidding problem of the storage operator. The challenges associated with time-linking constraints in this case have been studied extensively for both price-taker [11, 12] and price-maker storage [13,14,15,16]. These challenges are transferred to the market operator in the case of non-merchant storage. Energy markets are cleared sequentially for subsequent finite time horizons, but due to intertemporal constraints, the different market horizons do influence one another. This raises the question of how to account for the subsequent market clearing in the current clearing, and vice versa. For an overview of market design approaches for markets with intertemporal constraints we refer to [17].

As the market-clearing problem is solved with a finite horizon, the relation with the following market horizons is embedded in the choice of a final state of energy for the storage system. Existing works on market clearing with non-merchant storage often neglect or simplify relations between subsequent market horizons, leading to ‘myopic’ decision-making regarding the state of energy of the storage at the end of horizon. This is pointed out as a research gap by the authors of [18]. A non-myopic, optimal final state of energy is challenging to determine, because it depends on both current and future market horizons. The non-merchant storage literature largely bypasses this problem using the following common simplifying assumptions. Most works do not impose any constraints on the final state of energy in the storage [7,8,9,10]. Another common simplification is to enforce that the storage state of energy is equal at the start and end of a horizon [19,20,21]. A possible state-of-energy profile resulting from either of these myopic approaches is plotted in blue in Fig. 1, whereas an example optimal profile given perfect information about future market intervals is depicted in black. Such simplifying assumptions can lead to suboptimal use of the storage, loss of social welfare, and market inefficiencies, which we will show in this paper. These effects are sometimes overlooked in existing works that consider market properties within a single horizon [10]. As an alternative, a ‘future-aware’ method, in red in Fig. 1, could approach the perfect profile if informed end-of-horizon decisions were made.

Fig. 1
figure 1

Illustrative example: the effect of (simplifying) assumptions on the state of energy of the storage over several market clearing horizons

Full size image

Several previous works do consider setting a future-aware end-of-horizon level for the non-merchant storage, but this has only been done using rolling-horizon methods [17, 22, 23]. However, problems can even occur when the final state of energy is set perfectly, both in rolling-horizon or other kinds of look-ahead markets. They result from the fact that subsequent optimization horizons are not aware of losses and gains incurred by market participants in past horizons [22]. As a result, it can happen that the storage improves social welfare, while not recovering its cost. Solutions to this problem have been proposed in a rolling-horizon setting, and with a focus on ramping constraints [22, 24].

The end-of-horizon issue for non-merchant storage is systematically overlooked, as illustrated by many studies that make simplifying assumptions on the final state of energy. To draw attention to this, we aim to convey two key messages in a rigorous manner. Our first message is that it is important to set a future-aware end-of-horizon storage level, instead of making simplifying assumptions. Second, we show that a new problem arises in the absence of the simplifying assumptions, namely, that market prices in subsequent market horizons may fail to reflect the value of storage. This problem is essential to address, as the market can fail to provide dispatch-following incentives and cost recovery for the storage, as we will show in this work. Previous works have touched upon these issues, but only in a rolling-horizon setting, and with a focus on ramping constraints. Instead of using a rolling-horizon approach, we consider a more general formulation where the final storage level is constrained to take a certain value, to be determined for each market clearing.

Our first contribution is to provide intuitive understanding of our two key messages using two illustrative examples. Our second and main contribution is the analysis of market properties under simplifying assumptions on the storage system final state of energy, as well as in the absence of these assumptions. To support the first message, we prove that such common simplifying assumptions ensure cost recovery for the storage, but lead to market inefficiencies. Related to the second message, we provide sufficient conditions in Theorem 1 under which the market prices in subsequent market horizons will fail to reflect the value of storage, and thus fail to provide dispatch-following incentives and possibly cost recovery for the storage. A final contribution of this work is a proposed market-clearing procedure with non-merchant storage which is efficient and ensures cost recovery, given perfect foresight about future market-clearing parameters.

The remainder of this paper is organized as follows. In Sect. 2, we introduce the set-up of this work, including the market-clearing model and pricing scheme. Section 3 analyzes the impact of common assumptions on market properties. In Sect. 4, we show why and under which conditions dispatch-following incentives and cost recovery for the storage are not ensured, even when the final state of energy is set to its optimal value. In Sect. 5, we propose a method for ensuring that the desirable market properties hold, in the perfect foresight setting. Finally, Sect. 6 concludes the paper, followed by several appendices.

2 Market model and definitions

We formulate the market-clearing problem with a non-merchant storage and detail our assumptions. Hereafter, market horizon refers to a set of time periods covered by a given market clearing. For example, a day-ahead market would have a market horizon of 24 h. The next market horizon would include the following 24 h. We disregard what happens in other markets trading for the same day, e.g., intraday and balancing. We first introduce the storage model in Sect. 2.1 before proceeding to the market formulation in Sect. 2.2. Next, we formulate the dual problem and discuss the pricing mechanism in Sect. 2.3. Finally, we define market properties in Sect. 2.4. Nomenclature and the rules followed for notations are available in Appendix A.

2.1 Storage model

We assume there is a single storage system. This is equivalent to including multiple storage systems, as we do not consider grid constraints. The storage system is described by

$$\begin{aligned}&0 \le e_t \le {\overline{E}}, \quad \forall t \in {\mathcal {T}}{} & {} :({\underline{\nu }}_t, {\overline{\nu }}_t) \end{aligned}$$
(1a)
$$\begin{aligned}&e_t = e_{t-1} + b_t , \quad \forall t \in {\mathcal {T}}\setminus \{1\}{} & {} :(\rho _t)\end{aligned}$$
(1b)
$$\begin{aligned}&e_1 = E^{\textrm{init}} + b_1{} & {} :(\rho _1) \end{aligned}$$
(1c)
$$\begin{aligned}&e_\text {H} = E^{\textrm{end}}{} & {} :(\xi )\,. \end{aligned}$$
(1d)

Here and in the following, dual variables are indicated within parentheses on the right of the corresponding constraint. For time step t in the market-clearing horizon \({\mathcal {T}}\), the state of energy of the storage system is a decision variable denoted by \(e_t\). It is bounded between 0 and the storage capacity \({\overline{E}}\) in (1a). The variable \(b_t\) represents the energy charged (\(b_t>0\)) or discharged (\(b_t<0\)). The storage energy balance is described by (1b) and (1c), where \(E^{\textrm{init}}\) is the initial state of energy. We omit charging and discharging losses, so that a single variable for charging and discharging suffices. The inclusion of charging and discharging losses would change the dual problem slightly, but as will be argued later on, it would not alter our main message. We disregard charging and discharging limits, since they make the formulations and derivations heavier by introducing additional dual variables, while they do not affect conclusions. Our storage-system model is stylized as we focus on the time-linking aspect in (1b). The representation of the storage system would be more detailed in a practical market-clearing model. For a discussion on the storage model assumptions, we refer to [25] and [26]. The final state of energy for \(t=H = |{\mathcal {T}}|\) can be set to a predefined value \(E^{\textrm{end}}\) with (1d). Constraints (1c) and (1d) generalize what is found in the literature, where the initial level is often assumed to be equal to zero and the final level is unconstrained, or where \(E^{\textrm{end}} = E^{\textrm{init}}\).

The best value of \(E^{\textrm{end}}\) could be determined in many different ways, e.g., rolling horizon, online learning, reinforcement learning, perfect foresight, etc. It is a topic in itself and we will not address it here. In Sect. 4, we consider the situation where \(E^{\textrm{end}}\) is chosen optimally, i.e., with perfect foresight of future market horizons. Our formulation is general in the way that it includes various approaches proposed in the literature as special cases. Alternatives to imposing the final level with (1d) could be to have a rolling-horizon set-up or to include an estimated value of stored energy in the objective function of the market clearing [15]. Similar end-of-horizon problems also arise in those settings.

2.2 Market-clearing formulation

In the market-clearing problem with non-merchant storage, the storage does not submit price-quantity bids. However, storage operational constraints (1a)–(1c), and in some cases (1d), are included in the clearing. The storage is used as an asset to move cheap energy between time periods, similar to the way that power lines can be included in the market clearing to move energy between nodes or zones. As we focus on the time-linking effects of storage, network constraints are excluded for simplicity. The addition of network constraints would, however, not alter our main results. Storage can also modify the dispatch between nodes of the system and have an impact on line congestions, which is studied in [9].

We consider two versions of the market clearing with non-merchant storage, namely, a free and a constrained market. In the constrained market, (1d) is included in the optimization problem, which gives the market operator the option to enforce an optimal end-of-horizon storage level by considering future market horizons. In the free market on the other hand, this constraint is omitted. The constrained market-clearing problem for the time periods \(t \in {\mathcal {T}}\), denoted by \({\textbf {C}}({\mathcal {T}})\), is

$$\begin{aligned} \max _{\textbf{x}} \quad&\sum _{t \in {\mathcal {T}}} \left( \sum _{l \in {\mathcal {L}}} U_{lt} d_{lt} - \sum _{g \in {\mathcal {G}}} C_{gt} p_{gt} \right) \end{aligned}$$
(2a)
$$\begin{aligned} \text {s.t.} \quad&\sum _{l \in {\mathcal {L}}} d_{lt} + b_t - \sum _{g \in {\mathcal {G}}} p_{gt} = 0 , \quad \forall t \in {\mathcal {T}} \quad :(\lambda _t)\end{aligned}$$
(2b)
$$\begin{aligned}&0 \le p_{gt} \le {\overline{P}}_{gt}, \quad \forall g \in {\mathcal {G}}, t \in {\mathcal {T}} \quad :({\underline{\mu }}_{gt}, {\overline{\mu }}_{gt})\end{aligned}$$
(2c)
$$\begin{aligned}&0 \le d_{lt} \le {\overline{D}}_{lt}, \quad \forall l \in {\mathcal {L}}, t \in {\mathcal {T}} \quad :({\underline{\chi }}_{lt}, {\overline{\chi }}_{lt})\end{aligned}$$
(2d)
$$\begin{aligned}&(\textrm{1a})-(\textrm{1d}) \, . \end{aligned}$$
(2e)

Here, \(\textbf{x}\) is the vector grouping all primal variables, which are indicated with Roman lowercase letters. The generators are indexed by \(g \in {\mathcal {G}}\), and loads by \(l \in {\mathcal {L}}\). The production of g at t is a decision variable \(p_{gt}\), and similarly \(d_{lt}\) gives the demand of l at t. The individual bid and offer prices are \(U_{lt}\) and \(C_{gt}\). These correspond to the demand utility and the generation cost under the assumption that participants bid truthfully. The objective function (2a) is to maximize the social welfare, calculated as the difference between total demand utility and total generation cost over the given time horizon. Constraint (2b) enforces balance between production and demand at each time t, including the charged or discharged energy. Production and demand limits are enforced in (2c) and (2d), with maximum levels \({\overline{P}}_{gt}\) and \({\overline{D}}_{lt}\), respectively.

The free market-clearing problem for the time periods \(t \in {\mathcal {T}}\), denoted by \({\textbf {F}}({\mathcal {T}})\), is obtained from the constrained problem by removing (1d).

2.3 Dual problem and pricing

The dual problem \({\textbf {CD}}({\mathcal {T}})\) of (2) is

$$\begin{aligned} \min _{\varvec{\zeta }} \quad&\sum _{t \in {\mathcal {T}}} \left( \sum _{g \in {\mathcal {G}}} {\overline{P}}_{gt}{\overline{\mu }}_{gt} + \sum _{l \in {\mathcal {L}}} {\overline{D}}_{lt} {\overline{\chi }}_{lt} + {\overline{E}} {\overline{\nu }}_t \right) + E^{\textrm{init}} \rho _1 - E^{\textrm{end}} \xi \end{aligned}$$
(3a)
$$\begin{aligned} \text {s.t.} \quad&C_{gt} - {\underline{\mu }}_{gt} + {\overline{\mu }}_{gt} - \lambda _t = 0, \quad \forall g \in {\mathcal {G}}, t \in {\mathcal {T}}\end{aligned}$$
(3b)
$$\begin{aligned}&- U_{lt} - {\underline{\chi }}_{lt} + {\overline{\chi }}_{lt} + \lambda _t = 0, \quad \forall l \in {\mathcal {L}}, t \in {\mathcal {T}}\end{aligned}$$
(3c)
$$\begin{aligned}&- \rho _t + \lambda _t = 0, \quad \forall t \in {\mathcal {T}}\end{aligned}$$
(3d)
$$\begin{aligned}&- {\underline{\nu }}_t + {\overline{\nu }}_t + \rho _t - \rho _{t+1} = 0, \quad \forall t \in {\mathcal {T}}\setminus \{H\}\end{aligned}$$
(3e)
$$\begin{aligned}&- {\underline{\nu }}_H + {\overline{\nu }}_H + \rho _H - \xi = 0 \end{aligned}$$
(3f)
$$\begin{aligned}&{\underline{\mu }}_{gt}, {\overline{\mu }}_{gt}, {\underline{\chi }}_{lt}, {\overline{\chi }}_{lt}, {\underline{\nu }}_t, {\overline{\nu }}_t \ge 0 \,. \end{aligned}$$
(3g)

The vector \(\varvec{\zeta }\) groups all dual variables. We include the dual variables associated with the non-negativity constraints of the primal variables as they are used in our subsequent proofs. We choose the signs of the free variables \(\rho _1\) and \(\xi\) in a way that will facilitate our derivations. The dual problem \({\textbf {FD}}({\mathcal {T}})\) of the free market clearing is obtained from (3) by removing \(\xi\) and the terms related to it.

A thorough analysis of the dual problem is available in [10]. One useful interpretation of \(\rho _t\) is that it represents the marginal value of having an additional unit of energy stored at the end of hour t. Constraint (3d) implies \(\rho _t = \lambda _t\) for all t. Therefore, we use \(\rho\) and \(\lambda\) interchangeably in the remainder of this article. The inclusion of charging and discharging losses would change relation (3d) by a factor.

We consider a pricing system where all market participants, including the storage system, buy and sell energy at the hourly market price \(\lambda _t\). Alternatively, the participants could be paid rents evaluated from other dual variables, as is done in [7, 8, 10]. The two payment systems are equivalent, as proven in [7, 10] for the case that \(E^{\textrm{init}} = E^{\textrm{end}} = 0\).

Constraints (3e) and (3f) establish a relation between the value of \(\rho _t\) in subsequent time periods. These constraints show that the market prices \(\lambda _t\) in two subsequent hours can only differ if the storage state of energy is at a bound. This can be seen as intertemporal congestion, similar to the way line congestion in network-aware markets may lead to differences in nodal or zonal prices.

We define the concept of a time zone similar to a spatial zone in network-aware markets. A time zone is the longest possible set of consecutive time steps with the same market price. We illustrate the concept of a time zone in Fig. 2.

Fig. 2
figure 2

Bottom: Example of market-clearing price dividing the market horizon into four time zones. Notation for sets of time indices used in later sections is introduced. Top: Storage state-of-energy (\(e_t\)) profile related to the below market price signal

Full size image

Definition 1

(Time zone) A set of time steps \({\mathcal {Z}}\) with \(\min _{t\in {\mathcal {Z}}} t = z_0\) and \(\max _{t\in {\mathcal {Z}}} t = Z\) is a time zone if and only if

  1. 1.

    \({\mathcal {Z}}\) only includes consecutive time steps

  2. 2.

    for all \(t\in {\mathcal {Z}}\) it holds that \(\lambda _t = c\) for some constant c

  3. 3.

    \(\rho _{z_0-1} \ne c\) and \(\rho _{Z+1} \ne c\).

Denote the time zone of time step t by \({\mathcal {Z}}_t\).

The bottom plot in Fig. 2 shows four time zones. Within each time zone, the market price \(\lambda _t = \rho _t\) is constant. The upper plot in the figure shows an example state-of-energy profile for the storage that can accompany the price profile below. As \(\rho\) decreases from \(t=1\) to \(t=2\), the storage must be at a lower bound at \(t=1\), due to constraint (3e). The same constraint implies that when the price increases, e.g., from \(t=3\) to \(t=4\), the storage must be at an upper bound at \(t=3\).

2.4 Market properties

Here, we define the desirable properties of the market clearing that will be evaluated in the next sections.

Market prices are dispatch supporting when no market player desires to unilaterally deviate from the outcomes of the market clearing. Alternatively, one could say that the market provides dispatch-following incentives. The market is efficient if social welfare is maximized and the market provides dispatch-following incentives. There is cost recovery if the profit of every participant is guaranteed to be non-negative. Related works might use other terms to describe similar market properties, e.g., individual rationality. Another critical market property is revenue adequacy, which holds when the market operator has no financial deficit. We do not mention it in the rest of the paper because it is ensured regardless of the assumptions on the final storage level.

In the remainder, we assume that there is perfect competition, in the sense that demands and generators participating in the market bid their true utility and cost. Indeed, we focus on the effects of the storage system on the market properties and disregard the possibility of strategic bidding for the other participants, which would also exist without the storage system.

3 Market properties under common assumptions on the final storage level

In this section, we analyze the influence that common simplifying assumptions on the final state of energy have on market properties, in particular, cost recovery for the storage system and market efficiency. We evaluate two assumptions. The first is to not have a constraint on the final level. The market clearing will then schedule the storage in order to maximize the social welfare in the current horizon only, while neglecting future social welfare. As a consequence of not having such a constraint, storage state of energy is guaranteed to be at its lower bound by the end of each market horizon, unless negative prices occur.Footnote 2 The second assumption is to assume that storage state of energy is equal at the start and end of a horizon. Often this level is set to zero, which is not necessarily a good choice, as the storage operator could prefer to start the day with some energy available in case the prices are high. We explore how these assumptions ensure cost recovery over each individual market horizon, but may lead to market inefficiency.

3.1 Illustrative Example I: Impact of common assumptions

In this purely illustrative example, we clear the market for two sequential days of two hours each, identified by the sets \({\mathcal {T}}^{\textrm{d1}}\) and \({\mathcal {T}}^{\textrm{d2}}\). Two hours is the shortest possible market horizon for which the impact of common assumptions can be illustrated. Similar results can be obtained for longer market horizons. The storage capacity is \({\overline{E}} = 2.5\, \textrm{Wh}\). We consider a single load and two generators, a cheaper one and a more expensive one. The time-dependent inputs are summarized in Table 1. The corresponding code is available online [27].

Table 1 Time-dependent inputs for Illustrative Example I of Sect. 3
Full size table

We look at the profit of the storage system with different assumptions on the initial state of energy \(E^{\textrm{init}}\) and final state of energy \(E^{\textrm{end}}\) on the first day. These results are shown in Table 2. In the first case, the initial level is set to 0 and the final level is left free, or equivalently set to zero (if no negative prices occur), which is the most common assumption in the literature. In the second case, the initial and the final levels on each day are equal, which is also a common assumption. The value is set to \(1.25 \, \textrm{Wh}\) (half of the capacity). In the third case, the preference of the storage, given perfect information about the second day, is taken into account. The storage starts empty, and the final level on the first day is set to \(2.5\, \textrm{Wh}\). The final level on the second day is left free, under the assumption that there is no subsequent market clearing, which is sufficient to show the potential issues. Indeed, in this last case, the storage earns a negative profit on the first day, but this is compensated for by a higher profit on the second day. Note that the storage never obtains a negative daily profit in the other two cases.

Table 2 Profit (€) of the storage system for various initial and final state of energy (\(\textrm{Wh}\)) for Illustrative Example I
Full size table

The total social welfare obtained is equal to 46€ in the first and second cases and 55.5€ in the last case. So not only the profit of the storage is higher in the last case, but the social welfare is too. This difference in social welfare is due to the fact that in the last case, the storage system is used in the first hour of the second day instead of the most expensive generator. In order to charge the storage, the less expensive generator produces more on the first day.

In the following, we generalize these results and show that if the final state of energy is forced to be equal to the initial state of energy or not fixed at all, the daily profit of the storage is never negative. However, these assumptions on the final storage level will most often give a solution that is not optimal for the storage, nor the social welfare.

3.2 Cost recovery for the storage system

In both market-clearing formulations \({\textbf {F}}({\mathcal {T}})\) and \({\textbf {C}}({\mathcal {T}})\), cost recovery holds for the generators and loads. For the storage system however, cost recovery is not automatically ensured within a single market interval. For example, if the final level is set to be higher than the initial level and the prices are positive and decreasing during the market interval, the storage system pays for charging at the last hour and does not recover that cost in this market interval. However, we show in this section that under certain conditions, cost recovery for the storage within each market interval is ensured.

3.2.1 Cost recovery in \({\textbf {C}}({\mathcal {T}})\) with equal initial and final state of energy

In order to evaluate cost recovery for \({\textbf {C}}({\mathcal {T}})\), we express the profit of the storage as a function of the initial and final state of energy. We use the profit-maximization problem of the storage and strong duality to find this relationship. Related to the market-clearing model (2), the profit maximization for the storage is

$$\begin{aligned} \max _\textbf{b,e} \quad&\sum _{t \in {\mathcal {T}}} - \lambda _t b_t \end{aligned}$$
(4a)
$$\begin{aligned} \text {s.t.} \quad&0 \le e_t \le {\overline{E}}, \quad \forall t \in {\mathcal {T}}&\quad :({\underline{\nu }}_t, {\overline{\nu }}_t) \end{aligned}$$
(4b)
$$\begin{aligned}&e_t = e_{t-1} + b_t , \quad \forall t \in {\mathcal {T}}, t \ne 1&\quad :(\rho _t) \end{aligned}$$
(4c)
$$\begin{aligned}&e_1 = E^{\textrm{init}} + b_1&\quad :(\rho _1) \end{aligned}$$
(4d)
$$\begin{aligned}&e_H = E^{\textrm{end}}&\quad :(\xi ) \,. \end{aligned}$$
(4e)

The objective of the storage is to maximize its profit (4a), considering that it is paid the market price \(\lambda _t\) when discharging (\(b_t \le 0\)), and it pays the market price \(\lambda _t\) when charging (\(b_t \ge 0\)). The market price is taken as an input. The dual problem of (4) is

$$\begin{aligned} \min _{\varvec{\rho , \nu }} \quad&\sum _{t \in {\mathcal {T}}} {\overline{E}} {\overline{\nu }}_t + E^{\textrm{init}} \rho _1 - E^{\textrm{end}} \xi \end{aligned}$$
(5a)
$$\begin{aligned} \text {s.t.} \quad&\lambda _t - \rho _t = 0, \quad \forall t \in {\mathcal {T}}\end{aligned}$$
(5b)
$$\begin{aligned}&-{\underline{\nu }}_t + {\overline{\nu }}_t + \rho _t - \rho _{t+1} = 0, \quad \forall t \in {\mathcal {T}}, t \ne H\end{aligned}$$
(5c)
$$\begin{aligned}&-{\underline{\nu }}_H + {\overline{\nu }}_H + \rho _H - \xi = 0 \end{aligned}$$
(5d)
$$\begin{aligned}&{\underline{\nu }}_t, {\overline{\nu }}_t \ge 0 \,. \end{aligned}$$
(5e)

At optimality, strong duality holds, and the values obtained for the primal and dual objective functions are equal (optimal variable values are denoted by an asterisk). The profit at optimality is thus

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} - \lambda _t b_t^* = \sum _{t \in {\mathcal {T}}} {\overline{E}} {\overline{\nu }}^*_t + E^{\textrm{init}} \rho _1^* - E^{\textrm{end}} \xi ^* . \end{aligned}$$
(6)

Adding (5c) for all t to (5d), and multiplying by \(E^{\textrm{end}}\), we get

$$\begin{aligned} E^{\textrm{end}} \xi ^* = - \sum _{t \in {\mathcal {T}}} E^{\textrm{end}} {\underline{\nu }}^*_t + \sum _{t \in {\mathcal {T}}} E^{\textrm{end}} {\overline{\nu }}^*_t + \rho _1^* E^{\textrm{end}} . \end{aligned}$$
(7)

Substituting (7) in the right side of (6), we obtain

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} - \lambda _t b_t^* = \sum _{t \in {\mathcal {T}}} ({\overline{E}}-E^{\textrm{end}}) {\overline{\nu }}^*_t + \sum _{t \in {\mathcal {T}}} E^{\textrm{end}} {\underline{\nu }}^*_t + \rho _1^* (E^{\textrm{init}} - E^{\textrm{end}}) . \end{aligned}$$
(8)

This equality shows that if the final level is set equal to the initial level, the profit of the storage system is non-negative, due to the non-negativity of \({\overline{\nu }}_t\) and \({\underline{\nu }}_t\), and the fact that \({\overline{E}}\ge E^{\textrm{end}}\). However, \(E^{\textrm{init}} = E^{\textrm{end}}\) is a special case. In general, we do not have guarantees that the profit of the storage system is non-negative, and in fact it is straightforward to find counterexamples, as was provided in Sect. 3.1.

3.2.2 Cost recovery in \({\textbf {F}}({\mathcal {T}})\)

We show here that not having a constraint on the final state of energy is also a special case for which the profit of the storage system is non-negative. For \({\textbf {F}}({\mathcal {T}})\), constraint (5d) is modified to

$$\begin{aligned} -{\underline{\nu }}_H + {\overline{\nu }}_H + \rho _H = 0 . \end{aligned}$$
(9)

Strong duality from (6) simplifies to

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} - \lambda _t b_t^* = \sum _{t \in {\mathcal {T}}} {\overline{E}} {\overline{\nu }}^*_t + E^{\textrm{init}} \rho _1^* . \end{aligned}$$
(10)

Summing all constraints in (5c) and (9) for the optimal solution, we get

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} {\overline{\nu }}^*_t = \sum _{t \in {\mathcal {T}}} {\underline{\nu }}^*_t - \rho _1^* . \end{aligned}$$
(11)

Since \({\underline{\nu }}^*_t\) is non-negative,

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} {\overline{\nu }}^*_t \ge - \rho _1^* . \end{aligned}$$
(12)

Multiplying both sides by \(E^{\textrm{init}}\), and using that \({\overline{E}} \ge E^{\textrm{init}}\), we derive

$$\begin{aligned} {\overline{E}} \sum _{t \in {\mathcal {T}}} {\overline{\nu }}^*_t \ge E^{\textrm{init}} \sum _{t \in {\mathcal {T}}} {\overline{\nu }}^*_t \ge - E^{\textrm{init}} \rho _1^* \,. \end{aligned}$$
(13)

We conclude

$$\begin{aligned} \sum _{t \in {\mathcal {T}}} - \lambda _t b_t^* = {\overline{E}} \sum _{t \in {\mathcal {T}}} {\overline{\nu }}^*_t + E^{\textrm{init}} \rho _1^* \ge 0 . \end{aligned}$$
(14)

In other words, the profit of the storage system is certainly non-negative when the final state of energy is unconstrained.

The absence of cost recovery over a single market interval is not necessarily an issue: the storage may still recover its cost in the subsequent market intervals. In that sense, it would be more relevant to evaluate cost recovery for the storage over an infinitely long time horizon (or equivalent). However, if market intervals are considered completely separate, cost recovery cannot be ensured. Setting the initial and final levels to the same value is a way to overcome this, but it has an impact on other market properties, as we discuss next.

3.3 Market efficiency

We now show that enforcing the initial level to be equal to the final level can pose problems in terms of market efficiency. We argue that under this simplification, the storage system could have an incentive to unilaterally deviate from the market-clearing outcomes. Indeed, the storage system profit-maximization problem would actually take into account a longer horizon than just one market interval, to best utilize the potential of temporal arbitrage. Ideally, the planning horizon for the owner of the storage system would also correspond to this longer horizon. The storage profit maximization should be evaluated over a longer horizon \({\mathcal {T}}^{\textrm{long}}\), where \(H^{\textrm{long}}>H\), with \(H=|{\mathcal {T}}|\) and \(H^{\textrm{long}}=|{\mathcal {T}}^{\textrm{long}}|\):

$$\begin{aligned} \max _\textbf{b, e} \quad \sum _{t \in {\mathcal {T}}^{\textrm{long}}} - \lambda _t b_t \end{aligned}$$
(15a)
$$\text {s.t.} \quad 0 \le e_t \le {\overline{E}}, \quad \forall t \in {\mathcal{T}}^{\textrm{long}}$$
(15b)
$$\begin{aligned}e_t = e_{t-1} + b_t , \quad \forall t \in {\mathcal{T}}^{\textrm{long}}, t \ne 1 \end{aligned}$$
(15c)
$$\begin{aligned}e_1 = E^{\textrm{init}} + b_1. \end{aligned}$$
(15d)

Thus, if the final level at the end of the market interval is not set to a value found in the storage profit-maximization problem, i.e. if \(E^{\textrm{end}}\ne e_H^*\), where \(e_H^*\) is an optimal solution for (15), the market output will not be optimal for the storage system and the storage will have an incentive to deviate.

The social welfare is also impacted, as shown in Illustrative Example I from Sect. 3.1. Clearing a set of market intervals together will always give the best value of the social welfare over the entire set of time periods included, since the social welfare is maximized. When we instead clear these market intervals sequentially without setting the final value of the storage properly for every single clearing, the social welfare may be reduced.

In conclusion, it is recommended to allow system operators to set the final storage level to a sensible level, based on information about future market intervals. However, doing so brings new challenges. In the following, we analyze this type of market clearing and discuss the duality issues that arise.

4 Market properties of the multi-interval market clearing

While the previous section focused on market properties within a single market interval, this section establishes several results for the multi-interval case. The main aim is to determine the effect of splitting the market clearing in multiple intervals on the optimal primal and dual solutions. Ideally, the splitting should not affect the solutions. We show that even though the primal solution may be unaffected, this is not necessarily the case for the dual solution.

First, we specify the formulations used in the remainder of this work in Sect. 4.1, including the definition of the full-horizon and split-horizon problems. In Sect. 4.2 we use an illustrative example to show how solutions to these optimization problems may differ, and why this deserves attention. The following sections formalize and generalize the results of this illustrative example. In Sect. 4.3, we show that the primal solutions to these two problems are equivalent. However, as will be shown in Sect. 4.4, market efficiency is not ensured in the split-horizon problem, while it does hold for the full-horizon problem. We provide conditions under which market inefficiencies will occur in the split-horizon problem.

4.1 Definition of the full-horizon and split-horizon problems

We consider a horizon \({\mathcal {T}}\) that consists of two market intervals, assumed to be days. The time periods of first and second days are collected in the respective sets \({\mathcal {T}}^{\textrm{d1}}\) and \({\mathcal {T}}^{\textrm{d2}}\), where \(H = |{\mathcal {T}}^{\textrm{d1}}|\) and \(H+1\) is the first hour of \({\mathcal {T}}^{\textrm{d2}}\). The end of the entire horizon is \(T = |{\mathcal {T}}|\). We define the full-horizon problem as the market clearing for the entire horizon \({\mathcal {T}}\). The full-horizon optimization model is given by \({\textbf {F}}({\mathcal {T}})\), as already defined in Sect. 2.2.

We use the term split-horizon problem for the case where we clear the market sequentially for \({\mathcal {T}}^{\textrm{d1}}\) and then for \({\mathcal {T}}^{\textrm{d2}}\). The split-horizon optimization model is given by \({\textbf {C}}({\mathcal {T}}^{\textrm{d1}})\) for the first and \({\textbf {F}}({\mathcal {T}}^{\textrm{d2}})\) for the second day. In the split-horizon market with perfect foresight, the final storage level on \({\mathcal {T}}^{\textrm{d1}}\) is set optimally. That is, \(E^{\textrm{end}}\) is equal to the final optimal storage level \(e_H^*\) when solving the full-horizon problem. For fair comparison, the storage level at the end of the second day is unconstrained in both the full-horizon and the split-horizon problem. The two optimization problems for the split-horizon problem are independent, besides the fact that one must choose the parameters right, which means that \(E^{\textrm{end}}\) on the first day must equal the initial storage level \(E^{\textrm{init}}\) on the second day. As the optimization problems are independent, they can be solved in an equivalent combined optimization without changing their solution. The term split-horizon problem refers to this combined optimization. The objective functions of the full-horizon and split-horizon problems are equal.

We assume that we can disregard what happens after time \(t = T\), i.e. we assume that there is a finite horizon for which the market with non-merchant storage needs to be cleared. This simplification is justified by the need for a perfect baseline to which we can compare the split-horizon solution. Over a finite horizon, the results from Sect. 3 show that the market clearing with non-merchant storage achieves the optimal solution, and cost recovery is ensured for the storage. Therefore, we can consider this full-horizon market clearing as a perfect benchmark.

Optimal values in the full- and split- horizon problem are denoted with \(*\) and \('\), respectively. Our proofs use the Karush-Kuhn-Tucker (KKT) conditions of the full- and split-horizon problems, in particular those corresponding to dual constraints from (3). For completeness, we provide the full sets of KKT conditions in Appendix B.

4.2 Illustrative Example II: Dual multiplicity

Suppose the market is cleared for two sequential hours. The market participants include a single load, two generators, and a non-merchant storage with capacity \({\overline{E}} = 2.5\, \textrm{Wh}\) and initial level \(E^\text {init} = 0\). The time-dependent parameters, demand and production limits, utility, and cost, are summarized in Table 3, and will also be visible in market-clearing diagrams. The corresponding code is available online [27].

Table 3 Time-dependent inputs for Illustrative Example II of Sect. 4
Full size table

In order to evaluate the effect of splitting the market clearing on primal and dual solutions, we solve both the split-horizon problem with perfect foresight and the full-horizon problem. We thus show that the lack of dispatch support is not due to imperfect information, but an inherent effect of the market splitting. The optimal values for relevant variables are summarized in Table 4. The primal variables of the full- and split-horizon problems have equal optimal values and therefore appear in the table only once.

Table 4 Selected optimal primal and dual values in Illustrative Example II
Full size table
Fig. 3
figure 3

Market-clearing diagram for full- and split-horizon problem in Illustrative Example II. The common legend is provided in Subfigure (c)

Full size image

In the full-horizon problem, the two hours are cleared together. As the storage is not at a bound after the first hour, the two hours comprise a single time zone, so that the market-clearing diagram for the two hours can be depicted as in Fig. 3a. The unique optimal solution is to charge the storage to \(e_1^* = 1\textrm{Wh}\), and discharge it fully at \(t=2\). As the storage is charging 1 unit and discharging 1 unit within the full horizon, its net power injection is zero, and therefore the storage is not depicted in Fig. 3a. The corresponding optimal values for the market price are \(\varvec{\lambda }^* = [5, 5]\) €/Wh. The storage improves the optimal value of the objective function by \(1 \textrm{Wh} \cdot (9 - 5)\) €/Wh = 4€ compared to the case without storage, as one unit of generation by the more expensive generator with cost of 9€ at \(t=2\) is replaced by the cheaper generator with cost of 5€at \(t=1\).

The split-horizon problem with perfect foresight clears the two hours separately, while enforcing in the first market interval that \(E^{\textrm{end}} = e^*_1 = 1\textrm{Wh}\). The primal solution is equal to that of the full-horizon market — this result we formalize in Sect. 4.3. In the first hour, \(\lambda _1' = 5\) €/Wh, as illustrated in the market-clearing diagram in Fig. 3b. In the split-horizon market for the second hour, none of the loads and generators are marginal. This is depicted in the market-clearing diagram in Fig. 3c. Therefore, there is no longer a unique dual solution for the split-horizon problem, as all values \(\lambda _2'\) in the range [2, 9] €/Wh are optimal. This interval includes the full-horizon optimum for \(\lambda _2\), so the full-horizon dual solution is still optimal for the split-horizon dual problem. This is a general result, as we will prove in Sect. 4.3.

The dual multiplicity that can arise in the split-horizon problem deserves attention for several reasons. If \(\lambda _2' \in [2, 5)\) €/Wh is selected, the market does not provide dispatch-following incentives to the storage, which can increase its profit by decreasing \(e_1\). In this example, the storage even earns a negative profit, illustrating that cost recovery is not ensured. At the same time, this dispatch of the storage does achieve the optimal social welfare. Furthermore, as shown in this example, there exists a price that supports the dispatch.

One could argue that just selecting the ‘correct’, dispatch-supporting dual value for \(\lambda _2' = 5\) €/Wh would solve this problem. However, in order to do so the market operator first of all needs to be aware of the possible existence of multiple dual solutions. Solvers usually return only one solution, which is not necessarily the property-preserving one. Second, if dual multiplicity occurs in the first market interval rather than the second, then the market operator cannot yet know which dual solution will preserve market properties. Finally, selecting a dispatch-supporting dual solution becomes more complicated in an imperfect-foresight setting, where the chosen level \(E^{\textrm{end}}\) may be suboptimal in hindsight.

4.3 Shared solutions of the full- and split-horizon problems

In this section, we investigate whether the solutions of the full-horizon are also solutions to the split-horizon problem. Lemma 1 shows that primal solutions to the split-horizon problem with perfect foresight and those to the full-horizon problem are identical.

Lemma 1

Let \(e_H^*\) be part of an optimal solution \(\textbf{x}^*\) to the full-horizon problem. Then \(\textbf{x}^*\) is an optimal primal solution to the full-horizon problem if and only if \(\textbf{x}^*\) is an optimal primal solution to the split-horizon problem with \(E^\textrm{end} = e_H^*\).

Proof

In Appendix C. \(\square\)

Next, Lemma 2 shows that any optimal primal and dual solution to the original problem can be converted to an equivalent solution for the split-horizon problem. This correspondence is sometimes overlooked in the literature. For example, [22] discusses the existence of dual solutions to the split-horizon problem that are not dispatch supporting, but does not mention that the original dual solution remains a solution to the split-horizon problem too.

Lemma 2

Any optimal primal and dual solution pair \(\{ \textbf{x}^*, \varvec{\zeta }^*\}\) to the full-horizon problem is also an optimal solution to the split-horizon problem, with additional split-horizon variable \(\xi '\) taking the value \(\xi ' = \rho _{H+1}^*\).

Proof

In Appendix C.2. \(\square\)

4.4 Weak solutions to the split-horizon problem

While every optimal dual solution to the full-horizon problem is also an optimal dual solution to the split-horizon problem, the reverse generally does not hold. In this section, we discuss the existence of a weak optimal dual solution admitted by the split-horizon problem, but not by the full-horizon problem. First, we define this type of dual solution, and show why it is a problem. Next, we provide sufficient conditions for the existence of weak dual solutions in the split-horizon problem in our main Theorem 1.

Definition 2

(Weak dual) A dual solution to the split-horizon problem is weak if the resulting price is not dispatch supporting for the non-merchant storage. Such a weak dual exists if and only if one of the following situations occurs.

  1. 1.

    \(e_H \in ({\underline{S}}, {\overline{E}})\) and \(\rho _H' \ne \rho _{H+1}'\)

  2. 2.

    \(e_H = {\underline{S}}\) and \(\rho _H' < \rho _{H+1}'\)

  3. 3.

    \(e_H = {\overline{E}}\) and \(\rho _H' > \rho _{H+1}'\)  .

For a weak dual solution, the storage operator has incentive to unilaterally deviate from the schedule determined in the market. This incentive exists if the solution to the storage’s profit maximization problem (4) given market prices \(\lambda _t\) is different from the schedule determined in the market. We distinguish two cases:

  1. 1.

    If \(\rho _H' = \xi ' > \rho _{H+1}'\), the storage can improve its profit by decreasing \(e_H'\). This is possible in case \(e_H \ne {\underline{S}}\), i.e. both in situations (1) and (3) of Definition 2.

  2. 2.

    If \(\rho _H' = \xi ' < \rho _{H+1}'\), the storage can improve its profit by increasing \(e_H'\). This is possible in case \(e_H \ne {\overline{E}}\), i.e. both in situations (1) and (2) of Definition 2.

In extreme cases, a weak dual solution can cause the loss of cost recovery for the storage operator. For example, this happens if the storage bought all of the energy \(e_H\) at the price \(\rho _H'\) and is scheduled to sell all of it for a lower price \(\rho _{H+1}'\).

Theorem 1

Assume all cost and utility bids are unique.Footnote 3 If the optimal solution to the full-horizon problem is such that \(H+1\) and T are in different time zones, and

$$\begin{aligned} \rho _H^* = \rho _{H+1}^*, \end{aligned}$$
(16)

then the split-horizon problem with perfect foresight admits a weak dual solution.

Proof

We construct a weak dual solution to the split-horizon problem, based on the given dual solution to the full-horizon problem. More precisely, we derive values of \(\delta >0\) for which

$$\begin{aligned} \rho _{H+1}' = \rho _{H+1}^* + \delta \end{aligned}$$
(17)

is part of an optimal dual solution \(\varvec{\zeta }'\) that satisfies the KKT conditions of the non-myopic split-horizon problem with perfect foresight. We construct \(\delta\) in such a way that \(\rho ^*_H\) can remain unchanged, i.e. \(\rho _H' = \rho ^*_H\). In this proof, we refer to KKT conditions coming from the dual and primal problems, which have been presented in previous sections.

By (16), H and \(H+1\) belong to the same time zone \({\mathcal {X}}\). Denote the time periods in \({\mathcal {X}}\) that are part of day 1 by the set \({\mathcal {X}}^{\textrm{d1}}\), and those that are part of day 2 by \({\mathcal {X}}^{\textrm{d2}}\). Note that \({\mathcal {X}}^{\textrm{d1}} \subseteq {\mathcal {T}}^{\textrm{d1}}\) and \({\mathcal {X}}^{\textrm{d2}} \subseteq {\mathcal {T}}^{\textrm{d2}}\). These sets are illustrated in Fig. 2.

Due to uniqueness of cost and utility bids, there can at most be one marginal load or generator in \({\mathcal {X}}\) for a single time period \(t\in {\mathcal {X}}\), which is part of either \({\mathcal {X}}^{\textrm{d1}}\) or \({\mathcal {X}}^{\textrm{d2}}\). Therefore, at least one of \({\mathcal {X}}^{\textrm{d1}}\) and \({\mathcal {X}}^{\textrm{d2}}\) does not have a marginal load or generator. We assume that this is the case for \({\mathcal {X}}^{\textrm{d2}}\). The proof is similar for the other case.

By uniqueness of cost and utility bids, \(\rho _H^*\) can at most be equal to a single cost or utility bid within \({\mathcal {X}}\). If there is no marginal load or generator in \({\mathcal {X}}\), it can happen that \(\rho _H^*\) is not equal to any cost or utility bid.

From the KKT conditions of the full-horizon problem corresponding to dual constraints (3b) and (3c), for scheduled loads and generators in \({\mathcal {X}}^{\textrm{d2}}\) it holds that

$$\begin{aligned} {\overline{\chi }}^*_{lt}&= U_{lt} - \rho _{t}^* \ge 0{} & {} \text {for } \forall t \in {\mathcal {X}}^{\textrm{d2}},\, l \in {\mathcal {L}}_t^{\mathrm {+}} \end{aligned}$$
(18)
$$\begin{aligned} {\overline{\mu }}^*_{gt}&= \rho _{t}^* - C_{gt} \ge 0{} & {} \text {for } \forall t \in {\mathcal {X}}^{\textrm{d2}},\, g \in {\mathcal {G}}_t^{\mathrm {+}} \,, \end{aligned}$$
(19)

where \({\mathcal {L}}_t^{\mathrm {+}} = \{ l \in {\mathcal {L}}|\, d^*_{lt} > 0\}\) and \({\mathcal {G}}_t^{\mathrm {+}} = \{ g \in {\mathcal {G}}|\, p^*_{gt} > 0\}\). The inequalities are actually strict for all but a single scheduled load or generator, except possibly the load or generator that has a cost bid equal to \(\rho _H^*\). Similarly, for unscheduled loads and generators in \({\mathcal {X}}^{\textrm{d2}}\),

$$\begin{aligned} {\underline{\chi }}^*_{lt}&= \rho _{t}^* - U_{lt} > 0{} & {} \text {for } \forall t \in {\mathcal {X}}^{\textrm{d2}},\, l \in {\mathcal {L}}_t^{\textrm{0}} \end{aligned}$$
(20)
$$\begin{aligned} {\underline{\mu }}^*_{gt}&= C_{gt} - \rho _{t}^* > 0{} & {} \text {for } \forall t \in {\mathcal {X}}^{\textrm{d2}},\, g \in {\mathcal {G}}_t^{\textrm{0}} \,, \end{aligned}$$
(21)

where \({\mathcal {L}}_t^{\textrm{0}} = \{ l \in {\mathcal {L}}|\, d^*_{lt} = 0\}\) and \({\mathcal {G}}_t^{\textrm{0}} = \{ g \in {\mathcal {G}}|\, p^*_{gt} = 0\}\).

We define the maximum positive change in \(\rho _t\) for \(t\in {\mathcal {X}}^{\textrm{d2}}\) as

$$\begin{aligned} {\overline{\Delta }}^+ = \min _{t\in {\mathcal {X}}^{\textrm{d2}}} \big \{{\overline{\chi }}^*_{lt} \; | \, l \in {\mathcal {L}}_t^{\mathrm {+}} \big \} \cup \big \{{\underline{\mu }}^*_{gt} \; | \,g \in {\mathcal {G}}_t^{\textrm{0}} \big \} \,. \end{aligned}$$
(22)

This is defined so that an increase of \(\rho _{H+1}\) by \({\overline{\Delta }}^+\) can be counterbalanced by changing \({\overline{\mu }}\), \({\underline{\mu }}\), \({\overline{\chi }}\), \({\underline{\chi }}\), so that KKT conditions corresponding to (3b) and (3c) still hold. Similarly, the maximum negative change in \(\rho _t\) for \(t\in {\mathcal {X}}^{\textrm{d2}}\) is

$$\begin{aligned} {\overline{\Delta }}^- = \min _{t\in {\mathcal {X}}^{\textrm{d2}}} \big \{{\underline{\chi }}^*_{lt} \; | \, l \in {\mathcal {L}}_t^{\textrm{0}} \big \} \cup \big \{{\overline{\mu }}^*_{gt} \; | \, g \in {\mathcal {G}}_t^{\mathrm {+}} \big \} \,. \end{aligned}$$
(23)

Non-negativity of \({\overline{\Delta }}^+\) and \({\overline{\Delta }}^-\) follows from (18)–(21) and (19)–(20), respectively. In fact, at least one of \({\overline{\Delta }}^+\) and \({\overline{\Delta }}^-\) is strictly positive

$$\begin{aligned} {\overline{\Delta }}^+> 0 \, \vee \, {\overline{\Delta }}^- > 0 \,, \end{aligned}$$
(24)

since the inequalities in (18) and (19) are strict for all but a single load or generator.

Other than KKT conditions corresponding to (3b) and (3c), the dual variable \(\rho _t\) for \(t\in {\mathcal {X}}^{\textrm{d2}}\) is bound by an end of time-zone relation due to the KKT condition corresponding to dual constraint (3e):

$$\begin{aligned} \rho _X^* = \rho _{X+1}^* + {\underline{\nu }}_X^* - {\overline{\nu }}_X^* \,, \end{aligned}$$
(25)

where X denotes the final time period in \({\mathcal {X}}\). As \(X < T\), it holds that \({\underline{\nu }}_X^*> 0 \vee {\overline{\nu }}_X^* >0\) by the definition of a time zone. From (25) we can derive the following additional constraint on \(\delta\):

$$\begin{aligned} \delta&\in [-\infty , {\overline{\nu }}_X^*]{} & {} \text {if } {\overline{\nu }}_X^* > 0 \end{aligned}$$
(26)
$$\begin{aligned} \delta&\in [-{\underline{\nu }}_X^*, \infty ]{} & {} \text {if } {\underline{\nu }}_X^* > 0 \,. \end{aligned}$$
(27)

Combining (24), (26), and (27),

$$\begin{aligned} \delta \in&\left[ -{\overline{\Delta }}^-, \min \big \{{\overline{\nu }}_X^*,\, {\overline{\Delta }}^+\big \}\right]{} & {} \text {if } {\overline{\nu }}_X^* > 0 \end{aligned}$$
(28)
$$\begin{aligned} \delta \in&\left[ -\min \big \{{\underline{\nu }}_X^*, \, {\overline{\Delta }}^- \big \}, {\overline{\Delta }}^+\right]{} & {} \text {if } {\underline{\nu }}_X^* > 0 \,. \end{aligned}$$
(29)

Condition (24) ensures that either the upper or lower bound on \(\delta\) is nonzero, in both these cases.

Finally, we can construct the weak dual solution accordingly:

$$\begin{aligned}&\rho _{t}' = \rho _t^* + \delta&\forall t \in {\mathcal {X}}^{\textrm{d2}} \end{aligned}$$
(30)
$$\begin{aligned}&{\overline{\chi }}_{lt}' = {\overline{\chi }}_{lt}^* - \delta&\forall t \in {\mathcal {X}}^{\textrm{d2}}, l \in {\mathcal {L}}_t^{\mathrm {+}} \end{aligned}$$
(31)
$$\begin{aligned}&{\underline{\chi }}_{lt}' = {\underline{\chi }}_{lt}^* + \delta&\forall t \in {\mathcal {X}}^{\textrm{d2}}, l \in {\mathcal {L}}_t^{\textrm{0}}\end{aligned}$$
(32)
$$\begin{aligned}&{\overline{\mu }}_{gt}' = {\overline{\mu }}_{gt}^* + \delta&\forall t \in {\mathcal {X}}^{\textrm{d2}}, g \in {\mathcal {G}}_t^{\mathrm {+}} \end{aligned}$$
(33)
$$\begin{aligned}&{\underline{\mu }}_{gt}' = {\underline{\mu }}_{gt}^* - \delta&\forall t \in {\mathcal {X}}^{\textrm{d2}}, g \in {\mathcal {G}}_t^{\textrm{0}} \end{aligned}$$
(34)
$$\begin{aligned}&{\underline{\nu }}_X' - {\overline{\nu }}_X' = {\underline{\nu }}_X^* - {\overline{\nu }}_X^* + \delta \,. \end{aligned}$$
(35)

\(\square\)

Intuitively, the conditions in Theorem 1 describe the setting in which the full-horizon optimal solution has a time zone which extends over the two different days, but does not cover the entire second day. When this time zone is split over two market-clearing intervals, at least one of the parts will lack a marginal load or generator, and the optimal market-clearing price for the affected time periods can take multiple values. These conditions are sufficient but not necessary, i.e. there are other cases in which a weak dual solution to the split-horizon problem exists.

Regarding the assumptions in Theorem 1, the uniqueness of cost and utility bids excludes certain miscellaneous exceptional cases, such as the case that all market participants are either on an upper or lower bound, but there is still no freedom in the related dual variables. We further assume that the time zone \({\mathcal {X}}\) that spans the two days does not include \(t=T\), the final time period of day two. This assumption is used to exclude the following very specific and unlikely case. It could happen that \(T\in {\mathcal {X}}\), while at the same time \(e_T^* \in ({\underline{S}}, {\overline{E}})\). By KKT condition given by dual constraint (3f), this implies that \(\rho _t^* = 0\) for all \(t \in {\mathcal {X}}\). If there is a marginal generator bidding 0 in the part of \({\mathcal {X}}\) that is on day one, this fixes \(\rho '_H = 0\) for the split-horizon problem too. Furthermore, due to dual constraint (3f), it must hold that \(\rho _{H+1}'=0\).

5 Restoring market efficiency

In this section we propose a method to ensure equivalence between dual solutions to the split-horizon problem and the full-horizon problem. We explain how our method solves the problem in case of perfect foresight, and discuss its use in an imperfect foresight setting.

5.1 Proposed method

In the perfect foresight setting, the optimal dual variables to the full-horizon problem, including \(\rho ^*_{H+1}\), are assumed known or predicted perfectly when clearing the first-day market. We denote solutions to the restored split-horizon problem using \(''\). When clearing the market for the second day, \(\rho _H''\), \({\underline{\nu }}_H''\), and \({\overline{\nu }}_H''\) are known.

Our method aims to modify the split-horizon problem in order to restore the dual constraint that is missing in the split-horizon problem:

$$\begin{aligned} \rho _{H+1} = \rho _H - {\underline{\nu }}_H + {\overline{\nu }}_H \,, \end{aligned}$$
(36)

while it is present in the full-horizon problem.

First, we modify problem \({\textbf {C}}({\mathcal {T}}^{\textrm{d1}})\). Instead of fixing the final storage level \(e_H\) using a constraint, we use the value \(\rho _{H+1}^*\) to steer the optimal dispatch of the storage. This is done by adding \(e_H \, \rho _{H+1}^*\) to the objective, where \(\rho _{H+1}^*\) is the perfect prediction of this dual variable. As a result, the first-day dual problem includes the dual constraint

$$- {\underline{\nu }}_H + {\overline{\nu }}_H + \rho_H - \rho_{H+1}^* = 0$$
(37)

instead of (3f), where it should be noted that \(\rho _{H+1}^*\) is a parameter in the day 1 problem.

Next, we modify problem \({\textbf {F}}({\mathcal {T}}^{\textrm{d2}})\), using \(\rho _{H}''\), \({\underline{\nu }}_{H}''\), and \({\overline{\nu }}_{H}''\) determined by the modified day 1 problem as parameters. The initial storage level \(e_{H}\) is now included in the second day problem as a variable instead of a parameter. We add \(-e_{H} \, (\rho _{H}'' - {\underline{\nu }}_{H}'' + {\overline{\nu }}_{H}'')\) to the objective function. The dual problem for day 2 will then include the following constraint related to \(e_{H}\):

$$\begin{aligned} - \rho _{H+1} + \rho _{H}'' - {\underline{\nu }}_{H}'' + {\overline{\nu }}_{H}'' = 0 \,, \end{aligned}$$
(38)

which equals the missing dual constraint, with dual variables related to day 1 as parameters.

However, these modifications introduce the possibility to have multiple solutions in the primal problem. We can overcome this by reintroducing the constraint to fix the final storage level in the first day problem. In that case, the price for the last hour of the first day is the sum \(\lambda _H + \xi\). The dual variable \(\xi\) needs to be included to the objective function for the second day too, such that the added term is \(-e_{H} \, (\rho _{H}'' + \xi '' - {\underline{\nu }}_{H}'' + {\overline{\nu }}_{H}'')\). We also include a constraint to fix the variable \(e_{H}\) to be equal to \(e_{H}^*\) in the problem of the second day, with an associated dual variable \(\zeta\). In that case, the price for the first hour of the second day is the sum \(\lambda _{H+1} + \zeta\). Note that a similar approach can be followed to restore missing dual constraints when splitting a problem with any type of time-linking constraint. In the context of market clearing, this could be, for example, ramp rates of generators.

The added term \(e_H \, \rho _{H+1}^*\) in the objective of the first-day market resembles the ‘future value of stored water’ commonly applied in optimal scheduling problems for hydropower. In that application, \(\rho _{H+1}^*\) would represent the opportunity cost of each unit left in the storage at the end of the considered period. The future-value-of-storage function \(f(e_H)=e_H \, \rho _{H+1}^*\) in our proposed solution only represents that value correctly at the optimal value of \(e_H^*\). In an imperfect foresight setting, previous works estimated this function for all feasible values of \(e_H\), for instance using dynamic programming and Benders decomposition [28]. On the other hand, our proposed added term to the objective function of the second-day market is not seen in hydropower scheduling applications. This term represents the value of stored energy from past market-clearing intervals, which is essential for ensuring that the prices support the dispatch.

5.2 Illustrative example

We apply the proposed method to Illustrative Example II from Sect. 4.2. In the perfect foresight case, we assume that the optimal value \(\rho _2^* = 5\) €/Wh is known at the time of market clearing for day 1. Using this value, we obtain \(\rho ''_H +\xi '' = 5\) €/Wh. The second day problem no longer admits multiple dual solutions, because the new dual constraint (38) enforces that \(\rho _{H+1}'' +\zeta '' = \rho ''_H +\xi '' = 5\) €/Wh. In summary, prices for both H and \(H+1\) are equal to their original values in the full-horizon problem. As a result, market efficiency (including dispatch-following incentives) and cost recovery for the storage are again ensured.

5.3 Imperfect foresight

In an imperfect foresight setting, the market operator would make an error in estimating \(\rho _{H+1}\) and \(e_H\). The resulting market prices might be misleading. As we force the use of the storage to reach the specified storage end level, some loads could be asked to pay a price that is higher than their utility and some generators could get paid with a price that is lower than their costs. Market efficiency is thus not ensured, as well as cost recovery for loads and generators.

6 Discussion and conclusion

The inclusion of non-merchant storage in energy market clearing has received attention, among others for its potential to increase social welfare. In this work, we have argued against several simplifying assumptions that are commonly made in the literature regarding the final state of energy of the non-merchant storage, in particular, to set the final state of energy equal to the initial state of energy, or to disregard the final level altogether. We have shown that under these assumptions, market efficiency is not ensured, as the market may fail to provide dispatch-following incentives for the storage system. In addition, the storage can only perform time arbitrage within a single market interval, but not between market horizons, resulting in a loss of overall social welfare. However, allowing the final state of energy to take any value can also have negative consequences on the market properties if not handled carefully. First, one must determine the value of the final state of energy that is optimal for the storage system. Second, the market prices may not reflect the relation between different market horizons.

Regarding the latter, we have shown that the split-horizon market may fail to provide dispatch-following incentives for the storage, even when the final state of energy is set perfectly. However, we have shown that any solution to the full-horizon problem is also a solution to the split-horizon problem, in the perfect-foresight setting. This changes the nature of the problem, compared to what is discussed in the literature. It shows that there always exists a property-preserving dual solution to the split-horizon problem, namely, the dual solution to the full-horizon problem. However, there may in addition exist optimal dual solutions to the split-horizon problem, which are infeasible to the full-horizon dual problem and lead to market inefficiencies. Therefore, the challenge in the perfect foresight setting becomes that of selecting the correct dual solution. We have provided sufficient conditions for the existence of weak dual solutions. Finally, we have proposed a method that restores market properties in the split-horizon problem, in the perfect-foresight case.

However, it becomes more complicated in the imperfect-foresight case, where the value of \(E^{\textrm{end}}\) may turn out to be suboptimal in hindsight. The proposed solution cannot be applied in case \(e_H\) is suboptimal in hindsight. Future work should develop solutions for the imperfect-foresight case. Here, it should be considered that the suboptimal final state of energy leads to a social welfare loss compared to the perfect-foresight case. It is a nontrivial question how this loss should be fairly divided among the market participants, especially since the storage may improve social welfare in expectation.

Furthermore, we note that illustrative examples and proofs in this work are valid for a market clearing with linear cost and utility functions. We have restricted ourselves to this setting for simplicity, and because this type of market clearing is common in practice. However, the market efficiency problem can also arise in the nonlinear convex case, under similar conditions. Future works should analyze this case in further detail.

The negative effects of simplifying assumptions on the final level of the storage are most critical for very large storage systems, as the seasonal arbitrage that these assets could provide would not be harnessed. Future work should assess for which storage characteristics these simplifying assumptions have the greatest negative impact. Finally, we have neglected the problem of determining the value for the final state of energy. Future work should focus on how to determine this value, both under perfect and imperfect information.