A tutorial review of complementarity models for decision-making in energy markets

Abstract

Complementarity models can represent the simultaneous optimization problems of one or several interacting decision-makers, and thus they have become an increasingly important and powerful tool for formulating and solving bottom-up energy market models. This paper provides an overview of the full range of complementarity-based formulations and how these can be applied to assist the different market participants and organizations with their decision-making processes. To this end, the first part of the paper introduces the mathematical formulation of some basic complementarity models, which are illustrated by highly simplified but illustrative energy market applications. Considering these models, the second part of the paper is devoted to describing in broad terms four areas of their potential application: electricity markets, emission markets, natural gas markets and economies comprising several interacting markets.

Introduction

Since the 1970s, modeling of price and quantity equilibria in some regional and national energy markets, e.g., electricity, emissions or natural gas markets, has become increasingly important (Hogan 2002). Private companies apply these models to forecast prices, to estimate profits of alternative operating strategies (including making offers in auction-based markets), and to evaluate investments in energy supply, energy transport facilities, and energy-consuming equipment. Policy-makers use these models to assess how alternative environmental and energy policies as well as infrastructure investments could affect public objectives such as consumer costs, overall market efficiency, energy security, and environmental impacts. Market regulators, monitors, and operators meanwhile use models to evaluate the competitiveness of markets by comparing market outcomes to what models say could happen under perfect competition, and also to project how market outcomes could be affected by changes in market rules.

Many energy market models are engineering-economic or “bottom-up” simulations. Such models include more or less realistic representations of the various types of market participants and their decision variables while accounting for the physical energy supply and transport constraints that are crucially important to the determination of market prices. Others are “top-down” models that feature aggregate representations of supply and demand. Top-down models are often statistically estimated from market data, rather than constructed in bottom-up fashion from market fundamentals such as input costs and conversion efficiencies.

The purpose of this paper is to provide an introduction to complementarity formulations, an increasingly popular framework for formulating and solving bottom-up energy market models. Very simply, the most basic complementarity model is one that solves for a vector of variables x (of dimension n) to meet the conditions of the form f(x) ≥ 0,  x ≥ 0, and f(x)^T x = 0, where f(x) is a vector-valued function also of length n. These conditions can be more compactly written using the “perp” symbol ⊥ as 0 ≤ f(x)⊥ x ≥ 0. We will also introduce more complicated variations, including mixed complementarity problems (MCPs, whose conditions include equalities as well as complementarity conditions), mathematical programs with equilibrium constraints (MPECs, which are optimization problems containing complementarity conditions in the constraint set), and equilibrium problems with equilibrium constraints (EPECs, in which a Nash equilibrium is sought among several players, each of whom solves an optimization problem in the form of an MPEC).

Because the first-order (Karush–Kuhn–Tucker, or KKT) optimality conditions for continuous optimization problems are a special case of complementarity conditions, complementarity is a natural way to cast equilibrium problems among market players whose profit maximization problems are stated as mathematical programs. In particular, different behavioral assumptions of the players lead to different model structures that we discuss in the paper. Further, because modern complementarity software makes it practical to solve large complementarity problems (having tens or even hundreds of thousands of variables) (Dirkse and Ferris 1995), these models can capture rich details concerning supply options, demand variability, and transport constraints. Finally, there is a rich body of theory that allows analysis of such models for properties such as solution existence and uniqueness (Cottle et al. 1992; Luo et al. 1996). These advantages of complementarity models have motivated applications to a wide variety of engineering and economic problems (Ferris and Pang 1997). In particular, the complementarity framework is increasingly used to represent energy markets, taking its place alongside older methods that have been used since the 1960s and 1970s, including pure optimization approaches (e.g., MARKAL, Fishbone et al. 1981), methods based on Gauss–Seidel iteration among optimization and other models (e.g., the PIES algorithm, Hogan 1975), and system dynamics (e.g., FOSSIL, Naill 1992).

Previous surveys of complementarity models address particular sectors of application (such as electricity or gas, e.g., Daxhelet and Smeers 2001; Hobbs and Helman 2004; Gabriel and Smeers 2006; Smeers 2008), or the use of particular types of complementarity models (e.g., variational inequalities, Daxhelet and Smeers 2001). Here, we provide an overview of the full range of complementarity-based formulations that are used in energy modeling. The paper’s purpose is tutorial, so rather than an exhaustive survey of applications, we present mathematical statements of the basic complementarity formulations and then a few abstract energy market formulations in order to illustrate the basic types of models. Our goal is to provide the reader some understanding of the mathematical fundamentals and their application so that their applicability to market intelligence, policy analysis, and market design and monitoring can be appreciated. The reader who wishes to apply the models of this paper can use available optimization modeling languages (e.g., GAMS 2013; AMPL 2013; AIMMS 2013) to build most of those models, taking advantage of solvers for particular classes of complementarity models (e.g., PATH and NLPEC) that can be called from the modeling software. Readers who then wish to develop their own models to address particular energy market operations, planning, or policy problems can consult the rapidly growing literature in their area of application; reference (Gabriel et al. 2013a), upon which this paper is based, provides much more detail than is presented here on formulations and implementation, and can be used as a gateway to much of that literature.

This paper is organized as follows. In the next section, mathematical fundamentals of complementarity problems, including MCPs, MPECs, and EPECs are presented. We discuss their relationship to optimization models and to Nash equilibria for market games. “Market modeling” reviews in broad terms four areas of application: electricity markets, emissions markets, natural gas markets and economies comprising several interacting markets. A brief conclusion is offered in “Conclusions and research challenges”.

Mathematical framework

In this section, we introduce the complementarity models that will be treated in this paper. The interested reader is referred to Gabriel et al. (2013a) for further details.

Optimality and complementarity

Suppose that there are I players, indexed by $i=1, 2, {\ldots}, I,$ each trying to optimize its decisions, e.g., to maximize profit or minimize cost. If player i controls variables in the vector x _i , then the following is a general formulation ¹ of the problem of player i:

$$ \begin{aligned} &\hbox{Minimize}_{x_{i}} \quad f_{i}(x)\\ &\hbox{s.t.}\\ \end{aligned} $$

(1a)

$$ g_{i}(x_{i}) \leq 0 {:}\,\mu_{i} $$

(1b)

$$ -x_{i} \leq 0 {:}\,\beta_{i}, $$

(1c)

where ${x_{i}\in\mathbb{R}^{n_{i}}}$ is the vector of decision variables, ${f_{i}{:}\,\mathbb{R}^{n}\rightarrow \mathbb{R}}$ is the objective function and ${g_{i}{:}\,\mathbb{R}^{n_{i}}\rightarrow \mathbb{R}^{m_{i}}}$ defines the inequality constraint (1b) whose associated dual variable is ${\mu_{i}\in\mathbb{R}^{m_{i}}. }$ The nonnegativity of x _i is imposed by (1c) and its associated dual variable is ${\beta_{i}\in\mathbb{R}^{n_{i}}. }$ We assume that the objective function (1a) depends on ${x\in\mathbb{R}^{n}}$ which includes decision vector ${x_{i}\in\mathbb{R}^{n_{i}}, }$ and also the decision vectors of the other players, i.e., $x=\{ x_{1},{\ldots},x_{i},{\ldots},x_{I}\},$ where ∑ ^I _i n _i  = n.

Assuming that functions f _i and g _i are continuously differentiable, we can write the Karush–Kuhn–Tucker (KKT) conditions (Bertsekas 1999) associated with problem (1):

$$ \nabla_{x_{i}}f_{i}(x)+\mu_{i}^{\rm T}\nabla_{x_{i}}g_{i}(x_{i})-\beta_{i}=0 $$

(2a)

$$ 0\leq -g_{i}(x_{i}) \perp \mu_{i} \geq 0 $$

(2b)

$$ 0\leq x_{i} \perp \beta_{i} \geq 0, $$

(2c)

Equation (2a) results from differentiating the Lagrangian function of (1) with respect to x _i . Constraints (2b) and (2c) are the complementarity conditions derived from constraints (1b) and (1c), respectively. Note again that 0 ≤ z ⊥ w ≥ 0 indicates that z ^T w = 0, z ≥ 0 and w ≥ 0. Observe that problem (2) can be classified as a mixed complementarity problem (MCP) since it includes both equality and complementarity conditions, i.e., constraints (2a) and (2b)–(2c), respectively.

Moreover, we can combine equations (2a) and (2c) to eliminate the dual variable β _i so that problem (2) is equivalent to the following more compact complementarity system:

$$ 0\leq x_{i} \perp \nabla_{x_{i}}f_{i}(x)+\mu_{i}^{\rm T}\nabla_{x_{i}}g_{i}(x_{i}) \geq 0 $$

(3a)

$$ 0\leq -g_{i}(x_{i}) \perp \mu_{i} \geq 0. $$

(3b)

It is important to note that if f _i and g _i are convex and problem (1) satisfies standard constraint qualification conditions (Bertsekas 1999), then the KKT conditions (3) are both necessary and sufficient optimality conditions, so that solving the complementarity system (3) is equivalent to solving the optimization problem (1).

The system of conditions (3) is said to be “square” because there are as many equations (or inequalities) as there are variables: for each variable in the vector x _i , either the variable itself equals zero, or the corresponding expression on the right of (3a) equals zero (or both); and for each variable in μ _i , either it is zero, or the corresponding function in −g _i (x _i ) is zero (or both).

Illustrative example: optimality and complementarity

In this example, we consider a general firm i that wants to trade its production ${x_{i}\in \mathbb{R}^{1}}$ (which can be identified as gas, coal, electricity, oil, etc.) in its corresponding energy market. We assume that this firm aims to maximize its profit (minimize cost minus revenues) from trading in the market subject to its technical constraints, so that it solves the following optimization problem:

$$ \begin{aligned} &\hbox{Minimize}_{x_{i}} \quad c_{i}x_{i}-p(x)x_{i}\\ &\hbox{s.t.} \\ \end{aligned} $$

(4a)

$$ x_{i}-X_{i}^{\rm max} \leq 0 {:}\,\mu^{\rm max}_{i} $$

(4b)

$$ -x_{i} \leq 0, $$

(4c)

where c _i is the per unit cost associated to x _i and p(x) represents the market price (i.e., the inverse demand function). Note that although the decision variable of problem (4) is x _i , we assume that the market price p(x) depends also on other firms’ production levels, i.e., $x=\{ x_{1},{\ldots},x_{i},{\ldots},x_{I}\}.$ Constraint (4b), whose associated dual variable is μ ^max _i , sets the maximum capacity level (X ^max _i ) for production x _i . Constraint (4c) forces the nonnegativity of x _i .

Similarly to (3), we can formulate the complementarity system equivalent to problem (4) as follows:

$$ 0\leq x_{i} \perp c_{i}-\frac{\partial p}{\partial x_{i}}x_{i}-p(x)+\mu_{i}^{\rm max} \geq 0 $$

(5a)

$$ 0\leq X_{i}^{\rm max}-x_{i} \perp \mu_{i}^{\rm max} \geq 0. $$

(5b)

Assuming the convexity of objective function (4a) (relative to x _i ) and noting that the constraints (4b) and (4c) are convex (they are actually linear), we can state that solving the complementarity system (5) is equivalent to solving problem (4).

Nash equilibrium

Consider a game-theoretic framework in which several players $i=1,{\ldots},I$ seek to simultaneously maximize their objective functions by solving optimization problems, one per player, similar to (1). Note that these optimization problems are interrelated because each player’s strategy (optimal decision variables x _i ) influences the rest of players’ objective functions f _k (x), k ≠ i, since $x=\{ x_{1},{\ldots},x_{i},{\ldots},x_{I}\}.$ Under this setting, the Nash equilibrium (Nash 1951) is defined as a set of strategies x _i for $i=1,{\ldots},I$ that guarantee that no player can improve its objective function by unilaterally changing its strategy.

Therefore, the Nash equilibrium can be identified as the joint solution of problems (1) for $i=1,{\ldots},I$ (Fig. 1), which is equivalent to solving the following complementarity system (assuming convexity of (1) and a constraint qualification):

$$ \begin{aligned} &\hbox{find:}\, \{x_{1}, {\ldots},x_{i},{\ldots},x_{I}\}\\ &\quad \hbox{that solve } (3) \hbox{ for } i=1, {{\ldots}}, I. \end{aligned} $$

(6)

Fig. 1

NE Problem: joint solution of several optimization problems/complementarity systems

Illustrative example: Nash–Cournot equilibrium

As an extension of the example presented in “Illustrative example: optimality and complementarity”, we assume that there are three firms participating in the energy market under study, each of them solving an optimization problem similar to (4) for i = 1, 2, 3. Additionally, we consider that the price p(x) depends linearly on the total energy production, i.e., p = α − β∑ ³ _i=1 x _i , where α and β are positive demand parameters. Note that in this example the strategic variable of each firm is x _i , i.e., the optimal energy quantity to produce.

Similarly to (6), the resulting Nash equilibrium in this market can be characterized as a solution x = {x ₁, x ₂, x ₃} of the following complementarity system:

$$ \left. \begin{array}{l} 0\leq x_{i} \perp c_{i}+\beta x_{i}-\alpha+\beta\sum_{j}x_{j}+\mu_{i}^{\rm max} \geq 0 \\ 0\leq X_{i}^{\rm max}-x_{i} \perp \mu_{i}^{\rm max} \geq 0 \end{array}\right\} \quad i=1,2,3. $$

(7)

In particular, the solution of system (7) can be identified as a Nash–Cournot equilibrium (Cournot 1927), which is a special type of Nash equilibrium. The Cournot model is characterized by the ability of each firm to anticipate its impact on the market through the knowledge of the inverse demand curve p(x). Note that there are other alternatives to represent the imperfect competition among firms, for instance, the Supply Function Equilibrium (SFE) (Klemperer and Meyer 1989) or the Conjectural Variation (CV) model. In particular, the CV model can be viewed as a generalization of the Cournot model in which a firm not only anticipates the impact of its production quantity on the market price, but also the impact that this quantity has on rival firms’ production quantities (Bowley 1924).

Consider that the demand function in (7) is characterized by α = 10 and β = 1, and that the different firms have the following per unit costs and maximum capacity levels: c ₁ = 3, c ₂ = 4, c ₃ = 5, X ^max _i  = 3, X ^max₂  = 4 and X ^max₃  = 2. The market outcomes obtained from the solution of the equilibrium problem (7) are presented in Table 1, second row ². Note that firm 1, which has the lowest per-unit costs is the one that achieves the highest production level and thus the highest profit.

Table 1 Illustrative examples results

Illustrative example: perfect competition and social welfare maximization

A price-taker firm does not anticipate any effect of its decisions on the market price. If all firms in Illustrative “Example: Nash–Cournot equilibrium” are price-takers, then p(x) in (4a) and (5a) becomes just p, the market price that is a variable in the overall equilibrium model, but it is treated as a parameter by each firm, and in particular, $\frac{\partial p}{ \partial x_i}$ in (5a) becomes zero. Assembling all the firms’ KKT conditions into one MCP, and using the linear form of the inverse demand function in place of p produces a model of perfect competition:

$$ \left. \begin{array}{l} 0\leq x_{i} \perp c_{i}-\alpha+\beta\sum_{j}x_{j}+\mu_{i}^{\rm max} \geq 0 \\ 0\leq X_{i}^{\rm max}-x_{i} \perp \mu_{i}^{\rm max} \geq 0\\ \end{array}\right\} \quad i=1,2,3. $$

(8)

The MCP (8) is almost the same as (7), except for the missing term β x _i . Results are presented in Table 1, third row. If this equilibrium is compared with the Cournot equilibrium, the competitive behavior of the firms renders lower market price and profits, higher total production quantity, and a higher social welfare.

It is a remarkable fact that a single optimization model, known as the Social Welfare Maximization model, is equivalent to the MCP (8) because the KKT conditions of the Social Welfare Maximization are the same as (8). The objective function of the Social Welfare Maximization model is the integral of the inverse demand function, minus the costs of production, summed over all firms, and the constraints are all of the constraints for the firms:

$$ \begin{aligned} &\hbox{Maximize}_{x_{1},x_{2},x_{3}} \quad \alpha \sum_{j=1}^{3}x_{j}-\frac{1}{2}\beta\left(\sum_{j=1}^{3}x_{j}\right)^{2}-\sum_{j=1}^{3}c_{j}x_{j}\\ &\hbox{s.t.} \\ \end{aligned} $$

(9a)

$$ x_{i}-X_{i}^{\rm max} \leq 0 {:}\,\mu^{\rm max}_{i} \quad i=1,2,3 $$

(9b)

$$ -x_{i} \leq 0,\quad i=1,2,3. $$

(9c)

The same result holds more generally for any price-taker model of equilibrium of a single commodity, e.g., for models with nonlinear inverse demand functions. Consequently, many single commodity equilibrium models are formulated as Social Welfare Maximization models, without explicitly formulating KKT conditions and the corresponding MCP.

Generalized Nash equilibrium

A generalization of the standard Nash equilibrium problem (6) can be considered if the feasible region of each player’s strategy (1b) is extended to be dependent on the other players’ strategies, i.e., g _i (x) ≤ 0 for $i=1,{\ldots},I$ where ${g_{i}{:}\,\mathbb{R}^{n}\rightarrow \mathbb{R}^{m_{i}}. }$ Taking this into account, the generalized Nash equilibrium can be formulated as follows:

$$ \begin{aligned} &\hbox{find}{:}\,\{x_{1}, {\ldots},x_{i},{\ldots},x_{I}\}\\ &\hbox{that solve}:\\ \end{aligned} $$

(10a)

$$ 0\leq x_{i} \perp \nabla_{x_{i}}f_{i}(x)+\mu_{i}^{\rm T}\nabla_{x_{i}}g_{i}(x) \geq 0 \,\, i=1,{{\ldots}},I. $$

(10b)

$$ 0\leq -g_{i}(x) \perp \mu_{i} \geq 0.\quad i=1,{\ldots},I. $$

(10c)

However, these problems are in general more difficult to solve than the standard Nash equilibrium case because they usually present nonsquare systems, with more variables than equations (Harker 1991), and consequently, infinitely many solutions.

Illustrative example: generalized Nash equilibrium

Consider that we incorporate the following constraint to each firm’s problem (4), for i = 1, 2, 3:

$$ \sum^{3}_{j=1}x_{j} \leq X^{\rm total}{:}\,\delta_{i}, $$

(11)

where X ^total represents the total available production to be traded in the market, which can be associated with a scarce natural resource or transport capability, and δ _i is the dual variable associated with this constraint. Note that condition (11) links the feasible regions of the different firms since it simultaneously includes decision variables x ₁, x ₂ and x ₃. Therefore, the joint solution of problems (4), in which condition (11) is incorporated as constraint, for i = 1, 2, 3, constitutes a generalized Nash equilibrium problem.

Thus, the generalized Nash equilibrium problem can be formulated similarly to the complementarity system (7), but adding the complementarity conditions:

$$ \left. \begin{array}{l} 0\leq x_{i} \perp c_{i}+\beta x_{i}-\alpha+\beta\sum_{j}x_{j}+\mu_{i}^{\rm max} +\delta_{i}\geq 0 \\ 0\leq X_{i}^{\rm max}-x_{i} \perp \mu_{i}^{\rm max} \geq 0\\ 0\leq X^{\rm total}-\sum_{i}x_{i} \perp \delta_{i} \geq 0 \end{array}\right\} \quad i=1,2,3. $$

(12)

Note that (12) includes the complementarity conditions associated with constraint (11), with only one constraint (11), but as many variables δ _i as there are players; this makes the generalized Nash model nonsquare.

It should be added that in some situations, there might be a reason to believe that all players have the same value for the dual variable, δ _i  = δ, which would result in a square model (Facchinei and Kanzow 2007; Harker 1991). This has the following economic interpretation: the scarce resource X ^total is rationed among the players by a market clearing price mechanism, in which the total demand for the resource is less than or equal to the supply, and all players are price takers with respect to that resource’s price.

Results corresponding to system (12), with a common dual variable δ, assuming X ^total = 4, are presented in Table 1, fourth row; the profits are calculated as (p − c _i )x _i . (If firms must pay for using x _i of the resource X ^total, then profits would be lower by δ*x _i , where delta = 0.67 in this solution.) The effect of limiting the total production is that the market price increases, as well as individual profits, as compared with the previous equilibrium models. Note also that there is a significant reduction of the Social Welfare.

Mathematical programs with equilibrium constraints

A mathematical program with equilibrium constraints (MPEC) is an optimization problem whose constraints include equilibrium conditions (Luo et al. 1996) (e.g., KKT conditions of a lower-level optimization problem or a Nash–Cournot game). MPECs are closely related to Stackelberg games (von Stackelberg 1934) in which a leader anticipates the reaction of one or several followers.

A general formulation of an MPEC is presented below:

$$ \begin{aligned} &\hbox{Minimize}_{y,x} \quad f^{L}(y,x)\\ &\hbox{s.t.} \\ \end{aligned} $$

(13a)

$$ y\in Y $$

(13b)

$$ \hbox{EP}(x|y), $$

(13c)

where ${f^{L}{:}\,\mathbb{R}^{n+p}\rightarrow \mathbb{R}}$ is leader’s objective function while vector ${y\in\mathbb{R}^{p}}$ (whose feasible region is Y) is the vector of decision variables of the leader. The vector ${x\in\mathbb{R}^{n}}$ is the set of decision variables for the followers. EP(x|y) represents the equilibrium problem of the followers, which is parameterized in the leader’s decision variables y. EP(x|y) can be formulated as a complementarity system similar to (6), as depicted in Fig. 2, or it can be formulated in other ways, e.g., as a variational inequality problem (Gabriel et al. 2013a). In particular, if the EP is formulated as a complementarity system, then problem (13) can be also referred to as a mathematical problem with complementarity constraints (MPCC).

Fig. 2

MPEC whose equilibrium constraints are included as a complementarity system

Illustrative example: MPEC

In this example, we consider again the three firms of “Illustrative example: Nash–Cournot equilibrium”. In this case, we assume that firm 1 behaves as a leader anticipating the reaction of firms 2 and 3, which are the followers. Thus, the MPEC solved by firm 1 can be formulated as follows:

$$ \begin{aligned} &\hbox{Minimize}_{x_{1},x_{2},x_{3}} \quad c_{1}x_{1}-\left(\alpha-\beta\sum_{i=1}^{3}x_{i}\right)x_{1} \\ &\hbox{s.t.} \end{aligned} $$

(14a)

$$ x_{1}-X_{1}^{\rm max}\leq 0 $$

(14b)

$$ -x_{1}\leq 0 $$

(14c)

$$ \hbox{EP}(x_{2},x_{3}|x_{1})$$

(14d)

where EP (x ₂, x ₃|x ₁) is represented by (7) for i = 2, 3. The complementarity system (7), for i = 2, 3, represents the equilibrium between firms 2 and 3 given the actions of firm 1. Note that, the MPEC’s decision variables are x ₁, x ₂ and x ₃ so that firm 1 is able not only to select its optimal production x ₁, but also influence the decision variables of the follower firms.

The fifth row in Table 1 presents the market outcomes resulting from model (14). Compared to the NE-Cournot model, the extra market power exercised by firm 1 (anticipation of reactions of the follower firms) allows it to increase its profit while reducing its rivals’ profits.

Equilibrium problems with equilibrium constraints

An equilibrium problems with equilibrium constraints (EPEC) can be interpreted as a multiple-leader Stackelberg game. Consider J leaders (indexed by j) each of one solves an MPEC problem similar to (13), i.e., each leader selects y _j solving

$$ \begin{aligned} &\hbox{Minimize}_{y_{j},x} \quad f_{j}^{L}(y,x)\\ &\hbox{s.t.} \\ \end{aligned} $$

(15a)

$$ y_{j}\in Y_{j} $$

(15b)

$$ \hbox{EP}(x|y), $$

(15c)

where $y=\{y_{1},{\ldots},y_{j},{\ldots},y_{J}\}.$ Note that the individual MPECs (15), one per leader j, are interrelated because both the objective function (15a) and the equilibrium conditions (15c) depend also on the decision variables of other leaders. In this regard, an EPEC can be considered as a type of generalized Nash equilibrium problem because each leader’s feasible region of y _j and x variables depends on other leaders’ decision variables through the follower’s equilibrium conditions ${EP}(x|y_1,{\ldots},y_{J}).$

Thus, an EPEC can be formulated as the joint solution of all the above MPECs (Fig. 3), one per leader, i.e.,

$$ \hbox{find: } \{y_{1}, {\ldots},y_{j},{\ldots},y_{J},x_{1}, {\ldots},x_{i},{\ldots},x_{I}\}\\ $$

(16)

Fig. 3

EPEC: joint solution of several interrelated MPECs

Note that if the equilibrium problem (15c) is formulated as a complementarity system like (3), then EPEC (16) includes several complementarity constraints.

Illustrative example: EPEC

In this example, we consider that both firms j = 1 and 2 are leaders that anticipate the reaction of firm 3 (the follower) by solving the following MPEC:

$$ \begin{aligned} &\hbox{Minimize}_{x_{j}} \quad c_{j}x_{j}-(\alpha-\beta\sum_{i=1}^{3}x_{i})x_{j}\\ &\hbox{s.t.} \\ \end{aligned} $$

(17a)

$$ x_{j}-X_{j}^{\rm max}\leq 0 $$

(17b)

$$ -x_{j}\leq 0 $$

(17c)

$$ \hbox{EP}(x_{3}|x_{1},x_{2})$$

(17d)

where EP (x ₃|x ₁, x ₂) is represented by (7) for i = 3. Note that MPEC (17) is similar to (14) but differs in that only firm 3’s complementarity conditions are included as equilibrium constraints.

Thus, the EPEC is defined as the joint solution of the two MPECs, i.e.,

$$ \begin{aligned}&\hbox{find: } \{x_{1},x_{2},x_{3}\} \\& \hbox{that solve } (17) \hbox{ for } j=1,2. \end{aligned}$$

(18)

The EPEC results for (17) are provided in Table 1, sixth row. Note that firm 2 is also a leader and thus it is able to increase its profit as compared with the previous MPEC results. Firm 3, which is the common follower of the two leaders, is excluded from the market and forced to stop production.

Note that, depending on the assumptions that each leader makes about its followers, other variants of EPEC (18) can be considered. For instance, firm 1 can presume to anticipate the reaction of both firms 2 and 3 (although firm 2 is another leader) while firm 2 only anticipates the reaction of firm 3. These variants are characterized by the number of followers i that each leader j considers in the equilibrium problem (17).

Solution methods

In the previous sections we have shown how, under certain conditions, an optimization problem, a Nash equilibrium problem or a Generalized Nash equilibrium problem, can be recast as a complementarity system similar to (3). In particular, if $\nabla_{x_{i}} f_{i}(x)+\mu_{i}^{\rm T}\nabla_{x_{i}}g_{i}(x)$ and g _i (x _i ) are linear functions then problem (3) can be referred to as a Linear Complementarity Problem (LCP). In contrast, if these expressions include nonlinear terms, then problem (3) can be denoted as a Nonlinear Complementarity Problem (NCP).

The main solution algorithms to solve LCP are based on pivoting methods (Cottle et al. 1992), such as the widely used Lemke’s method (Lemke 1965), or iterative methods, such as the matrix splitting one. On the one hand, pivoting methods can reach the solution in a finite number of pivots but may present computational complications, mainly for large problems, due to the accumulation of roundoff errors. On the other hand, iterative methods do not accumulate roundoff errors, but they may require a large number of iterations to properly approximate the true solution of the problem.

The most widely used algorithm to solve NCP is PATH (Dirkse and Ferris 1995) which is based on Newton’s method. Roughly speaking, at each Newton iteration of the algorithm, the nonlinear functions in the NCP are replaced by their first-order Taylor approximations about the previous iterate, which renders a LCP. Then the LCP can be solved using pivoting methods described in the above paragraph, sometimes with “damping” procedures inspired by those used in Newton’s method applied to systems of nonlinear equations. The iterations are repeated until the convergence of the solution is achieved. There are many other features of the PATH algorithm, but they are beyond the scope of this discussion. Convergence results may be found in Dirkse and Ferris (1995), and in practice, the algorithm has been very successful in solving many different NCP models.

Regarding the MPEC solution methods, it is observed that standard NLP solvers fail to tackle them directly because of the violation of some standard constraint qualification (Luo et al. 1996). Thus, the solution methodologies are based on the regularization of the complementarity constraints (Scholtes and Stöhr 2001), on their penalization in the objective function (Leyffer and Munson 2010), or on the used of Sequential Quadratic Programming (SQP) (Chen et al. 2006). Other approaches include the reformulation of the complementarity conditions (Fortuny-Amat and McCarl 1981; Siddiqui and Gabriel 2012), the use of decomposition techniques such as Bender’s (Conejo et al. 2006) or the application of interior point methods (Luo et al. 1996).

EPECs are difficult to handle (DeMiguel and Xu 2009) due to the non-convex feasible regions of each leader’s MPEC. The main solution techniques for EPECs are diagonalization (Hu and Ralph 2007), which is based on solving the MPECs sequentially until convergence, and the nonlinear programming reformulations of the resulting complementarity system (Leyffer and Munson 2010), i.e., the system formed by all the KKT conditions associated to each MPEC. However, the existence of a solution is generally not guaranteed and thus the development of appropriate solution techniques for EPECs is a line of active research.

Market modeling

Electricity markets

Introduction

Electricity markets allow trading of electricity in the long-run (one week to one year or more) through the futures market and bilateral contracts, and in the short-run (one day to several minutes prior to power delivery) through the pool, and bilateral contracts as well.

The pool and short-term bilateral trading are generally linked to the actual delivery of electric energy and since these two trading floors are short-term in nature, we generally refer to them as the spot market.

For the sake of simplicity, the analysis below refers to the spot (short-term) market, and particularly to its pool version. We begin with a description of an optimization model for a pool to run its auction; the model is based on Social Welfare Maximization. It is not a complementarity model, but it plays an important role in several subsequent models that do use complementarity concepts.

Market-clearing auction (optimization problem)

The first problem to be addressed in a pool is how to clear its auctions and to produce trading quantities and prices. We initially assume that demand plays a passive role, i.e., it does not actively bid in the pool, and the market operator instead forecasts the quantity of demand for the periods under consideration. More general formulations are readily derived.

An auction to match supply and demand is conveniently formulated as an optimization problem, similar to the Social Welfare Maximization model described in “Illustrative example: perfect competition and social welfare maximization”. The input data are the production offers by the producers (maximum quantities and minimum acceptable prices which the market operator treats as marginal costs) as well as the demand levels. The objective function is “social cost” which is computed as the sum, over all producers’ offers, of the minimum acceptable price times the variable representing the quantity to be produced. The output is the production of all the units of each producer and the hourly prices. The actual formulation is:

$$ \begin{aligned} &\hbox{Minimize} \quad \hbox{Social cost}\\ &\hbox{s.t.} \\ &\quad \hbox{Offer limits}\\ &\quad \hbox{Energy balance equations}\\ &\quad \hbox{Transmission flow constraints}\\ \end{aligned} $$

At an optimal solution, prices are the dual variables of the energy balance equations, because these dual variables are interpreted as the system-wide marginal costs. If the demands play an active role, instead of minimizing the social cost, the social welfare should be maximized, including the benefits of consumption.

The problem above is of the type analyzed in “Optimality and complementarity”. An example of a market clearing auction is reported in Motto et al. (2002).

Non-strategic producer self-scheduling (optimization problem)

As a prelude to models in which producers try to behave strategically by attempting to exert some influence over the equilibrium outcome of the pool, we consider first a price-taker producer. The key problem that a non-strategic producer faces is to schedule itself and to derive appropriate pool offers. Such a problem is an optimization problem as described below. For example, relying on records of market prices under similar conditions in the recent past, the producer can forecast 24 hourly prices for the next day, and then decide on the most profitable levels to operate its generation units. Each generation unit has many technical characteristics that limit it in various ways, e.g., in how fast its output rate can be changed from one hour to the next, or the maximum output rate that may correspond to the maximum offer quantity.

Given 24 next-day price forecasts, and the technical and cost characteristics of the units of the producer (input data), the non-strategic producer solves the optimization problem below:

$$ \begin{aligned} &\hbox{Maximize} \quad \hbox{Expected profit}\\ &\hbox{s.t.} \\ &\quad \hbox{Operational constraints of the production facilities}\\ \end{aligned} $$

The decision variables for this problem (problem outcome) are the hourly production outputs of each of the generating units of the producer.

Once the desired outputs of the production units are known, appropriate offering rules should be derived. One simple rule is offering the desired production levels at zero price and the remaining production capacities at infinity. This works as long as the producer is a price-taker, i.e., as long as its actions do not influence the market-clearing price.

This optimization problem is of the type analyzed in “Optimality and complementarity”. An example of a self-scheduling model is reported in Arroyo and Conejo (2000).

Equilibrium (Nash equilibrium)

Market regulators, monitoring entities and operators are concerned about how far a pool or bilateral market might stray from the ideal of perfect competition, with price-taker producers. They can use models to help assess the degree to which some firms are able to influence the market price, e.g., by withholding some of their capacity to try to drive the price up, in the Cournot or CV fashion.

Consider now a number of strategic producers competing in the pool or in a bilateral market to supply the demand. Each one of the producers solves a profit maximization problem, considering that: (i) its actual outputs influence the clearing prices (Cournot model), or that (ii) its actual outputs influence both the clearing prices and the outputs of other producers (conjectural variation model). Moreover, the price and the demand in each time period are related through an explicit equation (e.g., a linear demand like the one considered for the examples in “Mathematical framework”).

One appropriate way to tackle this problem is to write the optimality conditions (as in “Optimality and complementarity”) of the problem of each producer. Then, the optimality conditions of all producers are solved together. Note that this set of conditions may allow identification of a unique Nash equilibrium among producers, as this model avoids interactions through constraints which would result in a generalized Nash equilibrium problem and possibly many equilibria.

If the transmission network is represented in the model, then since all generation firms share the use of the network, there are constraints that link the generation firms’ models, and the model becomes a generalized Nash equilibrium, as in “Generalized Nash equilibrium”. However, by carefully specifying how the network is operated (i.e., conditions for access to the network by the generation firms), the MCP model can be made square. This is economically equivalent to saying that transmission capacity is rationed among firms by a price mechanisms, and that producers are price takers with respect to transmission prices.

The outcomes of this problem are equilibria characterized by the production levels of the units of all producers and the hourly clearing prices.

The problem described above is of the type analyzed in “Nash equilibrium”. Examples of competitive equilibrium are reported in Hobbs and Helman (2004) and Ruiz et al. (2008).

Strategic offering (MPEC)

Another way in which a large firm may be able to manipulate the outcome of the auction process in a pool is to choose its offer parameters strategically, e.g., to give a declared minimum price that is not equal to the actual marginal cost of operation, but is instead chosen with a view to producing an auction outcome that gives the largest possible profit to the firm. Strategic offering models, which represent such behavior, are used by market regulators, monitoring entities and operators, to assess the degree to which a large firm may be able to influence the pool outcomes in this way. These models can also be used by the firms themselves, to formulate offer strategies.

In such a model, a strategic producer makes offers into the pool anticipating the pool outcomes with the purpose of altering clearing prices to its benefit. This requires a hierarchical problem formulation (Stackelberg model) of the type described in “Mathematical programs with equilibrium constraints”. Such a problem has the following general structure:

$$ \begin{aligned} &\hbox{Maximize} \quad \hbox{Profit} \\ &\hbox{s.t.} \\ &\quad \hbox{Meeting}\, \hbox{the}\, \hbox{operational}\, \hbox{constraints}\, \hbox{of the units of the producer} \\ &\quad \hbox{Market equilibrium conditions} \\ \end{aligned} $$

where the market equilibrium conditions are the result of a market-clearing auction (optimization problem), as described above in “Market-clearing auction (optimization problem)”, whose inputs are the demand levels and the offers of all producers and whose outputs are accepted production levels and clearing prices. It has the general form below:

$$ \begin{aligned} &\hbox{Maximize} \quad \hbox{Social welfare} \\ &\hbox{s.t.} \\ &\quad \hbox{Meeting the operational constraints of the units of all producers} \\ &\quad \hbox{Energy balance conditions} \\ \end{aligned} $$

The decision variables of the upper-level problem (strategic offering) are the offers (quantities and prices) of the strategic producer and the decision variables of the lower-level problem (market-clearing) are the market-clearing outcomes: trading quantities and prices.

An appropriate manner to solve this problem is to derive the optimality conditions of the market-clearing auctions and to incorporate such conditions into the producer profit maximization problem, resulting in a MPEC, as explained in “Mathematical programs with equilibrium constraints”.

The problem above is of the type analyzed in “Mathematical programs with equilibrium constraints”. An example of a strategic offering model is reported in Hobbs et al. (2000) and Ruiz and Conejo (2009).

Strategic equilibrium (generalized Nash equilibrium, EPEC)

The interaction in the pool of a number of strategic producers, each one of them solving an MPEC of the type described in “Strategic offering (MPEC)” above, with common market equilibrium conditions as linking constraints, results in a generalized Nash equilibrium problem, as described in “Generalized Nash equilibrium”.

The producers interact through their respective objective functions that depend on clearing prices, and through the market-clearing conditions that are common to all producers. This model is also referred to as multiple-leaders (the producers) common-follower (the clearing procedure) game (Leyffer and Munson 2010).

An appropriate way to find equilibrium outcomes is jointly solving the optimality conditions of all MPECs, which constitutes an EPEC.

The problem above is of the type analyzed in “Generalized Nash equilibrium” and “Equilibrium problems with equilibrium constraints” (EPEC). Some examples of equilibria involving strategic producers are reported in Hobbs et al. (2000) and Ruiz et al. (2012).

Emissions trading

Introduction

Environmental legislation impacts energy industries in many ways. Water and air pollution emissions can be capped or taxed, siting of new facilities may be prohibited in sensitive areas, retrofits of pollution control equipment might be mandated, and investment in greener supply facilities or more efficient energy using equipment may be required.

Here, we consider an example of how one particular environmental policy can be modeled: pollution cap-and-trade policies. These policies work by allocating or selling permits to pollute (or “allowances”) to eligible pollution sources, which can then trade permits among themselves so that every polluter holds a number of allowances at least equal to their emissions. If the cap is below the emissions that would otherwise occur, the allowances have a positive market price that encourages emissions reductions . The attraction of cap-and-trade systems is that under certain assumptions, such systems motivate least-cost control of the emissions, while emissions allowances can be allocated to the affected industry, thus lessening the financial pain of compliance. Models similar to that summarized below can provide insight on how policies for limiting emissions and allocating allowances might influence future energy prices, power plant operations and investment, imports and exports, and other market outcomes of interest to policy-makers.

A simple model

We outline an approach to modeling cap-and-trade systems that builds upon the Nash equilibrium framework introduced in “Nash equilibrium” and applied to the electricity market in “Equilibrium (Nash equilibrium)”. The market actors are the same as in that model, with one addition: an auctioneer for the emissions allowances market.

Just as in “Equilibrium (Nash equilibrium)”, each producer optimizes its profit. But here, the model not only represents the producer’s decisions concerning the amount of energy to supply in each market it participates in, what fuel inputs to buy, and perhaps what transport services to purchase to bring its production to its markets, but also decisions concerning how many pollution allowances to buy or sell from the market and how to control its emissions. These decisions (and their associated decision variables) are of four types:

• 1. A producer may decide to emit fewer tons of pollution than would be permitted under the allowances it has been freely granted by the regulator. In that case it can sell its excess allowances, which is reflected in a revenue term in its profit objective. In a competitive allowances market, the producer takes the allowance price as fixed; but many pollution markets are concentrated so that the producer might recognize that it can affect the price of allowances by how many it buys. (Alternatively, in a dynamic market model, the producer can bank excess allowances for use in future time periods.)

• 2. The producer might decide to emit more than the allowances it has been given, so it must buy extra allowances from the allowances market at the market price. Again, the producer might be modeled as a price-taker or price-maker.

• 3. The producer might utilize “soft” strategies to reduce its emissions. In the case of electricity producers, the owner might simply shift output from dirty plants to cleaner ones, which is called “emissions dispatch”. Or the producer might reduce its sales (perhaps by subsidizing investments in energy efficiency by its customers), substitute bulk purchases of electricity from cleaner suppliers, or switch to cleaner fuels within its plants.

• 4. Or “hard” strategies might be pursued. These could involve retrofits of hardware (such as stack gas controls at existing plants for SO _x and NO _x ). Or, more drastically, the producer could retire older dirtier plants and replace them with clean newer (perhaps even renewable) facilities.

If these decisions are represented by continuous decision variables and, further, the resulting optimization problem for a producer is convex, then the natural next step is to take the KKT conditions for each producer’s model and concatenate them as part of a market equilibrium model, as in “Nash equilibrium” and “Equilibrium (Nash equilibrium)”. If they are discrete decisions, as is the case for many investments, the presence of binary variables is a challenge since KKT conditions no longer are necessary nor sufficient for optimality, and questions of equilibrium existence arise. The approach usually taken is to approximate capacity decisions with continuous variables instead, and then use the KKT conditions in a complementarity model. See Gabriel et al. (2012a, 2013b) for an investigation of models that need to satisfy both integrability and complementarity.

Compared to the models of “Electricity markets”, we now need sets of market-clearing conditions not only for the energy output markets, but also for the emissions input markets of the form:

$$ 0 \leq \hbox{Pollution Cap} - \hbox{Total Supplier Emissions} \perp \hbox{Price Allowances} \geq 0 $$

That is, if emissions are below the cap, the allowances price is zero; only if the cap is binding can the price be (potentially) positive. Alternatively, this can be viewed as the first order condition for an allowance auctioneer who maximizes revenue from selling allowances. If suppliers are price-takers with respect to allowances, this condition is just concatenated with market-clearing conditions for energy and the KKT conditions for producers to produce an MCP model of the energy and allowances markets, as in “Equilibrium (Nash equilibrium)”.

Attention must be paid to the appropriate time scale for the various markets, for instance, allowance markets might clear yearly, but energy markets might clear hourly or even every five minutes. As an example of this type of model, Chen et al. (2008) calculates equilibrium prices for CO₂ emissions allowances in the European Emissions Trading System, while simulating the electricity market for Northwest Europe considering peak and off-peak demand periods separately for each season. Another time scale issue is raised by the ability to overcomply with emissions limits in one year and then “bank” the excess allowances for use in a subsequent year, with a requirement that the amount in the bank always be nonnegative. A multiyear version of the equilibrium problem is then necessary to determine the equilibrium amounts of allowances to enter and exit the bank, and the resulting allowance price trajectory over time (and, in turn, the energy prices that are impacted by those allowance prices). If the bank is strictly positive for two or more consecutive years, the allowance price will increase, following the classic Hotelling price profile for nonrenewable resources (which has price rising at the rate of interest). This is the outcome of profit maximizing firms who are trading off control costs today versus the present worth of control costs in later years.

Oligopolistic markets for emissions allowances can also be modeled. For instance, if one supplier has market power with respect to allowances, the above complementarity condition is included in its constraint set, as well as the KKT conditions of other suppliers who buy or sell allowances. This situation is therefore modeled as an MPEC similar to (13) (as in Chen and Hobbs 2005). Alternatively, a variation on the theme of conjectural variations can be used, in which one or more energy suppliers anticipate that if they increase their demand for allowances, the price will go up according to an assumed market reaction (which might, however, be grossly inconsistent with how the market and other players actually respond) (Hobbs and Pang 2004).

This type of model can simulate trading systems that instead make consumers (or energy marketers and traders who sell to consumers) responsible for tracking emissions associated with the various suppliers that they purchase energy from. As a result, it would be consumers and/or marketers who would need to buy and sell emissions allowances rather than energy producers. Questions such as whether such systems would lower costs to consumers or increase incentives for energy conservation relative to producer trading systems can be addressed with complementarity models. Like all the other models of this paper, these models might be simple ones that can be analyzed for general results (Hobbs et al. 2010) or they may be richly detailed numerical models that represent energy sources, transport, and emissions trading (Chen et al. 2011; Yang et al. 2012). Both types of models have their uses. Simple models are helpful in fast-moving policy debates, showing in a transparent and replicable fashion the logic that leads to conclusions about possible outcomes or general relationships. In contrast, complex models are needed to quantify more precisely the impacts of policy alternatives on particular facilities or market players.

Natural gas markets

Introduction

Natural gas is a key element of global energy markets. Furthermore, it is important in a number of products and is used in a variety of sectors such as: residential, industrial, electric power, and transportation. For the transportation sector, while natural gas has at present only a relatively minor role in the US, in other countries it plays a much more significant part. Given that both compressed natural gas (CNG) and liquefied natural gas (LNG) have been shown to be substantially less polluting than petroleum-based fuels, ³ the connection with transportation and natural gas may be increasing in the future.

Natural gas is perhaps the cleanest-burning of the fossil fuels emitting for example 53.06 kg of carbon dioxide per million Btu, dramatically less than lignite (96.43), anthracite coal (103.62), distillate fuel oil (73.15), residential fuel oil (78.80). ⁴ The supplies of natural gas worldwide are not equally distributed geographically. This has led to business opportunities given the rise of liquefied natural gas which allows gas to be transported more globally than pipelines and has enabled gas to be traded more on a worldwide scale in both contract and spot markets. Conversely, this uneven distribution of the resource has also led to geopolitical tensions and in some cases to what some would say is the manifestation of market power. Take for example, the natural gas price disputes in 2006 and 2009 between Russia as supplier and Ukraine as downstream customer as well as other European countries further downstream. ⁵ Another geopolitical aspect is the Gas Exporting Countries Forum (GECF). This loose confederation of gas supplier countries collectively controls over 70 % of the global gas reserves (Dietsch 2012) and several recent studies have analyzed the possible influence of GECF (e.g., Gabriel et al. 2012b) in global markets.

Despite this heterogenous distribution of resources, there appears to be plenty of natural gas around. According to MIT Energy (2010), there is perhaps 150 times the current annual global consumption levels in recoverable resources [i.e., 16,200 trillion cubic feet (Tcf) of natural gas]. Moreover, about 9,000 Tcf is perhaps economically recoverable at less than or equal to $4 per million Btu. Part of this resource is what is characterized as “unconventional” with shale gas being one example. Due to the two engineering techniques of horizontal drilling and hydraulic fracturing (“fracking”), shale gas is a sizeable resource, especially in the US but also in other parts of the world (US Energy Information Administration 2011). According to the US Energy Information Administration (EIA) in its Annual Energy Outlook 2011, the US alone has some 862 Tcf (24,411 Bcm) of technically recoverable shale gas or about 34 % of its total natural gas resource base of 2543 Tcf. ⁶ EIA further projects that shale gas production will play a large part of the US supply picture constituting 46 % of the total US natural gas production by 2035. These numbers are subject to change given the current methods for estimating resources and EIA revised down some 40 % its 2011 estimate of 827 Tcf of US shale gas to 482 total due to more drilling data (Urbina 2012b). Along with the abundance of shale gas (e.g., in the U.S.), are some fracking-related environmental concerns regarding the potential contamination of the water table (Urbina 2012a) and a possible link with fracking-induced earthquakes (Fountain 2012). However, the US Environmental Protection Agency (EPA) is considering how to best proceed with hydraulic fracturing given health and environmental issues (Natural Gas Extraction–Hydraulic 2012) and these concerns are also present in other countries (Urbina 2011).

Another interesting aspect of natural gas is its interaction with intermittent, renewable forms of energy such as wind and solar. The intermittency of these supplies of power require some sort of thermal back-up and natural gas being the most environmentally friendly of the fossil fuels is a preferred choice. Thus, since there is not yet an effective way to store power on a large-scale, the rise of these intermittent sources of renewable energy is tied to some extent with natural gas (McMahon 2011).

For all these reasons, models of the natural gas market are important to energy planners. In the next few subsections we provide some possible gas market equilibrium formulations based on MCPs and MPECs and refer the reader to Chap 10 in Gabriel et al. (2013a) for more details.

Models for market participants

Overview The natural gas supply chain is composed of the following major market participants: producers, gas traders/exporters, pipeline operators, storage operators, liquefiers, regasifiers, LNG shippers and consumers. The major function of the producers is to explore for and produce gas from onshore or offshore wells. The gas traders/exporters then take this gas and bring it to the market. They can be independent or linked to a production company. They can be exporters to different countries or just taking care of domestic or local sales. Pipeline operators manage the flow and pressure of gas in pipelines, ⁷ compressors and other equipment related to the transport of gas. Storage operators manage underground reservoirs to both inject gas in the off season and extract it in the higher demand seasons. They need to take into account the engineering aspects as part of the injection/extraction cycles. ⁸ Natural gas can be converted to liquid form (LNG) and for that liquefiers and regasifiers are needed. In between these two market players are LNG shippers who transport the gas between nodes (cities, countries, regions). Lastly, there are the consumers which as previously mentioned come from different sectors. These sectors do not necessarily have the same demand curves for gas.

Company or market perspective There are many approaches to formulating a model for the natural gas market. Some of them are based on a company-specific perspective such as Guldmann (1983) which considered supply, storage, and service reliability issues for a natural gas distribution utility or de Wolf and Smeers (1996) which analyzed the optimal dimensioning of pipelines from both an operations and investment perspective or Midthun (2007) who developed models for optimal dispatch or optimal production.

From the market perspective, typically either there is one overall optimization problem maximizing an objective such as social welfare or the like [e.g., the International Gas Model used at EIA Model Documentation Report: International natural gas 2012, or the Gas Systems Analysis Model (GSAM) Gabriel et al. 2000, 2003]. Alternatively, a cost minimization for the entire market may be the perspective as in the TIGER (Transport Infrastructure for Gas with Enhanced Resolution) model (Lochner and Bothe 2007; Lochner 2011; Dieckhöner 2012). Such models usually have the assumption that the market is efficient.

Conversely, another market perspective involves taking the individual players’ optimization problems and then concatenating their KKT conditions possibly along with market-clearing conditions. The collection of all these conditions then constitutes an instance of an MCP. The individual players can either be price-takers or price-makers, the latter having the ability to adjust prices directly by altering gas volumes. Examples of the former are given in Gabriel et al. (2013a) (e.g., Illustrative Example 1.2.7, Chapter 10 models). Examples of the latter usually having some Nash–Cournot aspects and include: the World Gas Model (Gabriel et al. 2012b) and its precursor form (Gabriel et al. 2005a, b); GASTALE (Gas Market System for Trade Analysis in a Liberalizing Europe) (Boots et al. 2004; Egging and Gabriel 2006), and more recently (Lise and Hobbs 2008) with mostly a European market focus; GASMOD (Holz et al. 2008), the recent Former Soviet Union-centric model by Chyong and Hobbs (Chyong and Hobbs 2011), and GaMMES (Gas Market Modeling with Energy Substitution) (Abada et al. 2012; Abada 2012), a generalized Nash equilibrium model, to name just a few.

Natural gas market equilibrium formulations

The basic optimization problem for each of the players is the following:

$$ \begin{aligned} &\hbox{Maximize} \quad \hbox{Profit} \\ &\hbox{s.t.} \\ &\quad \hbox{Conservation of material} \\ &\quad \hbox{Upper and lower limits on equipment processes} \\ &\quad \hbox{Engineering considerations} \\ \end{aligned} $$

Lets consider each of these parts separately. First, the objective function is profit being defined as the difference between revenues and costs. Ignoring all subscripts except for player i, profit is of the form:

$$ p\left( \sum_{i}q_{i}\right) q_{i}-c_{i}\left( q_{i}\right) $$

(19)

where q _i is the quantity to be determined by player i (e.g., production for a producer), $c_{i}\left( \cdot \right)$ is the (convex) cost function, and $p\left( \cdot \right)$ is the inverse demand function. It is assumed that the summation over i in the inverse demand function is only over similar market players (e.g., just producers) and requires the decisions of the other similar players. This equation assumes that each of these players has market power and can adjust the level of q _i to their advantage since they “see” this inverse demand function in their objective function. If instead, they are price-takers, it is simply $p\left( \sum_{i}q_{i}\right) =p$ where the market price p is determined by market-clearing constraints of the form:

$$ \hbox{Total supply}-\hbox{total demand} =0 {:}\,p \quad \hbox{where } p \hbox{ is the associated multiplier} $$

(20)

Overall, the objective of profit should be a concave function of the decision variable q _i along with other conditions on the constraints to ensure that the KKT conditions are valid.

The various markets (production, storage, etc.) will determine different market-clearing prices either using (19) or (20). However, if there is more than one level of market-power, this will be either an EPEC as explained in “Equilibrium Problems with Equilibrium Constraints” or some sort of nested EPEC. In either case, the resulting equilibrium problem will be challenging to solve. In most cases (e.g, the World Gas Model), just one of the group of players is endowed with market power.

The cost function $c_{i}\left( \cdot \right)$ depends on the player type but usually is assumed to be convex and increasing. For example, for production, one choice is the Golembek cost function (Gabriel et al. 2012b) which rises exponentially as one approaches the production limit. In the case of the pipeline operator, such a cost might relate to the operational aspects of running the pipeline network. Clearly, one could also include investment costs and discounting if the optimization models are over different time periods. The conservation of material just stipulates that the total input equals the total output. The upper and lower limits on equipment are pretty straightforward with the lower value usually being zero. The upper limit is either fixed or some original endowment plus what is added if investments are considered. In the case of the pipeline operator, such an upper bound could relate to the maximum flow in the pipeline expressed in cubic feet or meters/day.

The engineering considerations can be some approximation of what really happens or the actual formulas. As mentioned above, both the Weymouth equation and the backpressure equation are two examples of such equations. Typically such equations might be ignored, linearized, or approximated in some fashion in order to make sure that the KKT conditions are still valid; for example see Midthun (2007), O’Neill et al. (1979) and Chap 10 and Appendix D of Gabriel et al. (2013a).

Then, the KKT conditions of each of the players are combined along with any market-clearing conditions to get a square, complementarity system. If some of the players have a first-mover type advantage, then an MPEC similar to (13) can be used.

Multicommodity markets

Several of the models summarized above have focused on the market for a single commodity with a single price, such as power at a particular location in a particular hour. However, most energy market models instead consider several markets simultaneously, recognizing that linkages among them imply that equilibrium prices in one market cannot be calculated without considering how they affect, and are affected by, prices elsewhere. As described below, energy markets may be distinguished by type of energy, timing, or location. Policies affecting supply or demand in one market have rippling (or, occasionally, tsunami) effects on other markets, and policy-makers are (or should be) particularly interested in these interactions. Several types of supply and demand interdependencies are described below as well as how complementarity market models can readily capture them.

The different markets that could be considered simultaneously in a multimarket model might be for physically distinct commodities, such as coal, natural gas, emissions allowances, and electricity. Technical relationships on the supply side may mean that two commodities can substitute for each other (e.g., gas for coal in power plants). Or, two commodities can be complements (e.g., if coal is burned in a power station, the generator must also buy emissions allowances). In either case, what a producer would be willing to pay for one input depends on the price of the other. Another supply relationship occurs when one commodity is transformed into another through the production process, so that the expense of inputs (say, natural gas sold to electric utilities) affects the cost of producing outputs (in that case, electricity). Changes in input prices then affect the costs and supply of the output. These phenomena can be represented in complementarity models based on producer optimization problems with production functions and constraints that appropriately represent these complementary, substitution, or input–output relationships in production functions.

Alternatively, the multiple markets may be for essentially the same commodity (e.g., natural gas), but because of transport costs or storage limits, commodities might be differentiated over space (e.g., different prices for gas in different countries) or time (e.g., higher prices for peak season gas supplies than for off-peak supplies). In a spatial market model, constrained transport capacity limits how much energy can be exported by one market or imported by another. Or transport costs can raise prices in importing regions. These spatial linkages can be quite complicated, as in load flow models for electric power [accounting in the most complex cases for reactive power, voltage, and frequency (Bautista et al. 2006)] or pipelines for gas transport [which could account for, e.g., line packing and other dynamics (Midthun et al. 2009)]. In such circumstances, prices at different locations diverge. In complementarity models of competitive markets, these price differences will equal the marginal cost of energy transport, if there are no binding capacity limits in the transport system, as competitive producers or traders will arbitrage away non-cost-based price differences. But when transport constraints bind, differences in prices will include a shadow price component accounting for the marginal value of transmission capacity, and competitive markets will ensure that the price paid to those providing transport services equals price differences among markets.

Such trade and arbitrage can be viewed as taking place through time as well, if an energy form can be put into storage in one time period and then withdrawn and consumed later when price is higher. An example is natural gas storage described in “Natural gas market equilibrium formulations”. In this case, there are multiple markets for the same commodity, but at different times.

Markets are also linked on the demand side. A final consumer’s willingness-to-pay for one commodity can depend on prices of other commodities. Commodities can be complements (natural gas and gas-fired furnaces, or electricity and light-bulbs) or substitutes (different types of light bulbs, alternative fuels for heating, and alternative fuels for vehicle transport, including gasoline, diesel, and, increasingly, natural gas and electricity). Commodities at different times can also have these demand relationships. For example, a key tool of the “Smart Grid” is to use real-time electricity prices (or other signals) to encourage consumers to be more flexible in the timing and amount of their energy use. As a result, the price of peak energy will affect off-peak demands because a residential consumer may decide to delay the timing of doing the wash or dishes, or may have an energy storage device in the home. Use at different times can be substitutes (when energy-using activities can be rescheduled to take advantage of low prices at certain times). They can also be complements (in the short run, if, e.g., a particular manufacturing process takes two adjacent hours to complete its operation, while in the long run, consistently low prices at some times during the day could incent purchases of energy-using equipment that increases energy consumption in all hours).

In a complementarity model, these demand relationships can be captured in a general demand function q = f(p), with q and p representing equal-lengthed vectors of quantities and prices, respectively. Such functions are often estimated econometrically. However, this behavior can also be represented by the KKT conditions of a model that maximizes consumer benefits of consuming energy at different times minus her or his expenditures, accounting for technical relationships in, for instance, storage devices. This is closely related to the household production framework, which represents the consumer as a kind of “producer”. In that framework, the final consumer optimizes the benefits of energy services that are consumed (light, heat, transport, ...), net of the capital, electricity, labor, and other inputs to produce those services. The complementarity market equilibrium model then combines the consumer’s KKT conditions with the conditions for suppliers plus market-clearing conditions. If one builds an energy model in which each energy supplier optimizes the profit subject to the actions of other suppliers, while explicitly accounting for the optimizing behavior of consumers with household production functions, the resulting model has an EPEC structure similar to (16). As an example of the use of a household production framework in a complementarity-based energy model, Hobbs and Nelson (1992) presents an MPEC where an electric utility (the upper level, or Stackelberg leader) anticipates how consumers (the lower level, or Stackelberg followers) will react to two different types of incentives: electricity prices and subsidies for investments in efficient energy-using equipment.

Models that capture the interactions of final demand for different energy commodities are increasingly useful for many policy and planning activities. Recognizing these interactions allows policies to exploit substitution possibilities in order to lower the cost and environmental impact of energy use. Smart Grids, for instance, can decrease the amount of power plant and transmission capacity that is needed by encouraging consumers to shift their uses to off-peak periods. Policies to promote electric or natural-gas vehicles can reduce oil imports. Pollution can be lessened by electric vehicles as well, if the fuels used to generate electricity are cleaner than gasoline and diesel fuels.

But on the other hand, disregarding such interactions can frustrate the objectives of policy. As an example, taxing residential natural gas use to encourage conservation may simply motivate consumers to heat homes with portable or baseboard electric resistance heaters. Then, if that electricity is produced by gas-fired power plants, the resulting total gas use may be higher, not lower because most of the energy value of the fuel is lost when generating and transmitting electricity. For these reasons, representing demand linkages was considered to be a critical feature in the original PIES model (Hogan 1975) and its descendent, the National Energy Modeling System (Gabriel et al. 2001).

Conclusions and research challenges

Complementarity models are often used to represent equilibrium since they offer the ability to consider simultaneous optimization problems of multiple interacting players. Different assumptions about competitive behaviors or types of interactions lead to different model structures. As such, complementarity modeling is very versatile and flexible and can encompass many aspects of game theory, optimization, economics, and engineering.

In particular, in this paper we have described equilibrium models with applications in energy. Mixed complementarity models as well as two-level optimization problems (MPEC and EPEC) were considered and shown to be important in aspects of energy planning.

Energy sector outcomes are increasingly driven by market fundamentals constrained or incented by government policy and infrastructure, as opposed to the government-owned firms or politicians making supply investment decisions by fiat. For that reason, and because of the wide availability of market data, energy economic equilibrium models have become widely used by private and public sector parties for decision support. In particular, complementarity is an increasingly popular way of formulating energy equilibrium problems because of its flexibility (being able to model both policy constraints and profit maximization behavior in the face of many complicated technical constraints) and the availability of efficient solvers.

Private sector companies use energy market equilibrium models to project the profitability and risks of alternative short-run strategies, such as market offers and fuel contracting decisions, and also long run investments in supply capacity. In the public sector, simple equilibrium models can inform policy debates by quickly and transparently showing the logical implications of alternative assumptions and policies. Often, such models can point out, through general proofs or simple numerical examples, where commonly accepted wisdom can be incorrect or where problems might occur that should be addressed by revising policy proposals. Meanwhile, more complex models can simulate the effects of policy changes on individual market players, geographic regions, or interests, which are often of intense interest to stakeholders. Such models can also be used to quantify the benefits and costs of alternative infrastructure designs, such as electricity networks, gas and oil pipelines, or import or export facilities. To avoid wasted resources, it is important to understand how markets might respond to infrastructure or policy changes under a wide range of scenarios, and complementarity models can help provide that understanding.

There are two main research challenges associated with complementarity problems.

The first one is the development of efficient and robust solution techniques that allow solving large scale problems. This is especially the case for MPECs and EPECs, for which most of the actual solution techniques rely on heuristic methods that cannot guarantee the convergence to a global optimum.

The second challenge is related to the market modeling. There are several modeling aspects that need to be further explored, for instance, the improvement of multicommodity models (which may also account for financial markets), the appropriate representation of uncertainties, the realistic formulation of physical and economical constraints, or the appropriate modeling of rivals’ behavior.