Representation of uncertainty in market models for operational planning and forecasting in renewable power systems: a review

Abstract

As the power system is becoming more weather-dependent and integrated to meet decarbonization targets, the level and severity of uncertainty increase and inevitably introduce higher risk of demand rationing or economic loss. This paper reviews the representation of uncertainty in power market models for operational planning and forecasting. A synthesis of previous reviews is used to find the prevalence of stochastic tools in power and energy system applications, and it concludes that most approaches are deterministic. A selection of power market tools handling uncertainty is reviewed in terms of the uncertain parameters they capture, and the methods used to describe them. These all use probabilistic methods and typically cover weather-related uncertainty, including demand. Random outages are also covered by several short-term power market models, while uncertainty in fuel and CO$_2$ emission prices were generally not found to be included, nor other types of uncertainty. A gap in power market models representing multiple dimensions of uncertainty, solvable on a realistic, large-scale system in a reasonable time, is identified. The paper concludes with a discussion on topics to address when representing uncertainty, where the main challenges are that uncertainty can be difficult to describe and quantify, and including uncertainty adds additional complexity and computational burden to the problem.

1 Introduction

Current power system development makes modeling electricity markets more challenging due to increasing penetration of intermittent and uncertain power production caused by variable renewable energy sources (VRE), electrification of energy systems and sector-coupling, climate change, and liberalization of markets. In addition, other developments, such as geopolitics, can significantly affect markets, as demonstrated lately by the Russian war in Ukraine and the energy shortage in Europe. The impact on the power markets of these factors, in combination with increasing demand in the wake of the COVID-19 pandemic and more extreme weather events, has been record-high European electricity prices [1]. This situation was hardly accounted for in any of the analyses a year ahead [2].

The timely question is, which types of uncertainty and unforeseen events are current power market models suitable to capture? Extreme and coincidental unfortunate incidents might be impossible to handle with modeling tools. There are various tools available for analyzing power systems and markets, many of which are based on a physical description of the power system and underlying costs. These tools are fundamental and can provide transparent explanations to complex relationships, enabling the analysis of the market clearing process by matching supply and demand and forecasting power prices for both present and future systems [3]. However, several assumptions are made when forecasting electricity markets due to the numerous uncertain parameters affecting the power system and power markets.

Uncertainty arises when describing future outcomes using modeling parameters. There are various power market uncertainties emerging from different sources, including demand forecasts, wind and solar power production, hydro inflow, fuel prices, CO$_2$ emission prices, market behavior, and generation and transmission capacity availability. Historically, price forecasting and production planning in hydrothermal power systems need to account for uncertainty, thus it is one of the fields developing and applying stochastic modeling [4]. As the power system is experiencing rapid changes to meet decarbonization targets, it is expected that uncertainties will impact the operation and planning of the power system to a larger extent than previously, and thus there is a need to account for uncertainty in modeling tools. The level of uncertainty increases due to the extensive deployment of VRE production, changing policy and regulations, demand side management programs, technology development, and a tighter connection to other sectors and markets. Moreover, climate change affects weather-driven power production and temperature-related demand. Important questions include if and how the uncertainty can be described, if and how it affects results and thus whether or not it should be accounted for in modeling.

The above-mentioned uncertain factors are different and have distinct characteristics. Some are technological/physical, some are economical, and others are regulatory/geopolitical. They differ by what timescales they become important and are revealed, by what decisions they affect, and by how they might be mathematically described and represented in a model. The description of the uncertain parameter may also depend on the time horizon and the perspective of the decision-maker [5]. Uncertainty can thus be classified according to several dimensions [6, 7]. Velasquez et al. classify uncertainty in one conceptual and three practical dimensions, by its nature, timescale, structure, and source, and it serves as a background for how to consider different uncertainties in the transmission expansion planning modeling process [6], though it is also suitable in other contexts. The presence of uncertainty affects decisions on all time scales, including operational decisions, planning and expansion decisions, and more long-term strategic decisions, and makes decision-making in power markets more challenging, and the forecasting more complex. An overview of how the Nordic power markets are structured, the decision-making process of power producers, and what uncertainties affect the different decisions are given by [8].

Uncertainty inevitably introduces risk [6]. Higher levels of uncertainty in power systems and power markets therefore lead to increased risk for decision-makers. However, this risk will be of a different character depending on the decision-maker’s perspective. The reliability and resilience of the power system are of the highest priority for system operators [9]. From a system perspective, the risk is therefore related to the secure operation of the system, and the system operator will try to reduce the risk of blackouts and rationing. From the perspective of a market actor, risk due to uncertainty can be in the form of a financial risk, which can entail both positive and negative outcomes, or a risk of failing to supply contracted electricity. However, growing uncertainty from a system perspective can also provide an opportunity for market actors, to compete to develop tools accounting for uncertainty in forecasting and planning.

Möst et al. point out that the need for decision-support tools in the energy business has significantly increased [10]. The literature also suggests that despite adding model complexity, decision-makers in power systems should consider uncertainty as both the severity of uncertainty increases and new uncertain factors are introduced through market liberalization and higher shares of VRE [11, 12]. Representing uncertainty will give a more realistic representation of the power markets and can provide valuable insight and decision-support. Handling uncertainty, both in the long-term and short-term, is critical for many types of decisions and analysis in the power system: e.g., operational decisions of a VRE producer, investment decisions, evaluating the impact of different energy policies, and the security, reliability, and resilience of a power system.

This paper focuses on uncertainty and assesses the representation of uncertainty in power market models used by decision-makers in operational planning and forecasting today. The review primarily focuses on models developed in Europe or applied specifically to hydrothermal power systems. The aim is to identify important uncertainties and how these are modeled today, plus to suggest what should be taken into account in future research and development.

This paper presents a three-fold contribution. Firstly, it summarizes previous review articles on power and energy system models, aiming to determine the prevalence of models that account for uncertainty and identify challenges in simulating future power systems, specifically focusing on uncertainty. Secondly, it evaluates 13 power market tools currently used by decision-makers in operational planning and forecasting. The assessment focuses on the consideration of uncertain parameters and the methods utilized to address them. Finally, the paper offers a discussion regarding the essential factors to consider when conducting forecasts in the presence of uncertainty. This includes determining the types of uncertainty to incorporate, describing the uncertainty and its correlations, and formulating and solving large-scale problems efficiently.

The paper is structured as follows: Sect. 2 provides a short overview of methods used to describe and manage uncertainty in power systems, including both deterministic and stochastic approaches. Section 3 presents a survey of previous reviews of energy and power system tools. The review of power market models handling uncertainty is given in Sect. 4, specifically focusing on stochastic optimization techniques implemented in power markets with high penetration of renewable energy, particularly hydropower. Section 5 discusses critical considerations and trade-offs when formulating power market models under uncertainty. The article concludes in Sect. 6.

2 Methods to describe and handle uncertainty

2.1 Uncertainty modeling

Uncertainty modeling generally revolves around different methods to measure the impact of uncertain input parameters on the system output parameters [13]. A classification of uncertainty modeling for decision-making in energy systems is proposed by Soroudi and Amraee and includes probabilistic, possibilistic, hybrid possibilistic–probabilistic approaches, information gap decision theory, robust optimization, and interval analysis [14]. These classifications have been applied in several review papers. Figure 1 provides a brief overview of these methods, including how the uncertain parameters are described and the advantages or disadvantages of each method. The reader is referred to the provided references for a detailed review of these methods, as it is outside the scope of this paper.

A comprehensive review of the six uncertainty modeling techniques for power system studies is given by [13]. Jordehi et al. focus on the first three approaches in their review and find that probabilistic methods based on probability distribution functions (PDFs) are the most used methods to deal with uncertainties in electric power systems [11]. Such methods, including Monte Carlo simulation [15] and scenario-based analysis, are simple to implement, but the challenge is their computational expensiveness. These methods also require the uncertain input to be described by PDFs and do not support uncertain parameters that cannot be described probabilistically.

Zakaria et al. give a thorough overview of uncertainty modeling, sampling methods for scenario generation, and stochastic optimization methods in renewable energy applications focusing primarily on Monte Carlo simulations, importance sampling, and (approximate) stochastic dynamic programming [12]. Singh et al. review power system uncertainties and approaches to handle uncertainty, focusing mainly on probabilistic power flow [7].

Fig. 1

Overview of uncertainty modeling [11, 13, 14]

2.2 Deterministic models

The classification of uncertainty modeling provided in Fig. 1 is based on how the uncertainty is described, e.g., by PDFs, membership functions, or intervals. There are also different methods for dealing with and evaluating uncertainty in power system and power market models, and it is common to classify these methods into deterministic methods and stochastic methods [16]. A deterministic tool optimizes a single deterministic scenario but can be run many times with varying assumptions on input data to assess the effects on the results.

Scenario analysis is often used to gain insight into the development of the power system under various assumptions. Different values or forecasts for uncertain parameters are manually chosen to obtain several scenarios, and the model is solved for each scenario [17, 18]. In sensitivity analysis, uncertain parameters are varied one at a time to test the impact on model outcomes. Sensitivity analysis is also used to test decisions made by the model under varying assumptions to evaluate which input parameters impact the output variables most and to find parameter ranges over which the decisions remain optimal [18]. However, sensitivity and scenario analysis do not give any insight into the probabilities of different events. Probabilistic methods take into account these probabilities and require that probability distributions represent the random parameters. As such, probabilistic methods include both deterministic and stochastic approaches.

Monte Carlo simulation is similar to scenario analysis, but it is a probabilistic method that can provide information about the distribution and probabilities of the results. A deterministic model is run several times and the scenarios are sampled in a structured way based on the PDFs of the uncertain parameters [14]. This method is popular, as it allows researchers to analyze a wide range of scenarios, including extreme probability scenarios, and provides statistics on the optimal solutions. However, a large number of scenarios typically have to be considered to obtain stable results [19], and different sampling methods exist to reduce the computational burden (e.g., importance sampling) [20]. Scenario analysis, sensitivity analysis, and Monte Carlo simulation are standard methods for evaluating uncertainty in power system applications, as stochastic models quickly become intractable, especially if considering multiple uncertain parameters.

2.3 Stochastic models

In stochastic models, uncertainty is included in the formulation of the model and is taken into account in the decision-making. Scenario-based analysis includes stochastic programming [21], and similar to Monte Carlo simulation, such methods also describe uncertainty using PDFs and generate scenarios. However, as the uncertainty is represented inside the model, the number of scenarios that can be handled within the computational limits of the model is limited. For Monte Carlo simulation, the solution time increases proportionally with the number of scenarios or samples, while for stochastic programming, the solution time increases exponentially. This typically leads to an unmanageable increase in computation time. In stochastic programming, appropriate scenario generation and reduction techniques are therefore essential to capture the possible outcomes by as few scenarios as possible [12]. This area of research has been paid increased attention in recent years [22, 23]. When stochastic parameters described by PDFs evolve over time, they follow a stochastic process. Scenarios can be generated from simulation of the uncertain parameters described by stochastic processes (e.g., mean-reversion processes, auto-regressive processes or Markov processes) [3]. Equiprobable scenarios can be determined based directly on historical data. Scenarios for short-term models can also be based on weather forecasts.

Stochastic models can be formulated as single-stage or multi-stage problems [3]. In single-stage formulations, decisions are made at the beginning of the planning period, prior to the realization of uncertainty, and no recourse actions can be taken. In multi-stage formulations, decisions are taken at several points in time as the uncertainty is gradually revealed, and decisions can be adjusted when new information arrives. Most problems in power markets resemble multi-stage problems, but due to the rapid increase in scenarios when there are many decision stages and branches in the scenario tree or lattice, these problems are often formulated as two-stage problems.

Roald et al. focus on two-stage stochastic optimization and give an overview of methods for optimization under uncertainty in power systems, including stochastic programming, chance-constrained optimization, robust optimization, and distributionally robust optimization [5]. The authors also discuss some methods for representing uncertainty depending on data availability. The different formulations of optimization problems under uncertainty depends on how the uncertainty is described (through scenarios based on PDFs, a set of possible PDFs, or an uncertainty set), but also on the risk-preference of the decision-maker. How the decision-maker relates to risk, i.e., risk-neutral or risk-averse, can be incorporated in decision-support tools by different risk measures. A popular risk measure is Conditional Value-at-Risk (CVaR), which is both convex and coherent [24]. The problem can also be formulated in a way that enhances certain risk-preferences. Chance-constraint optimization limits the risk of constraint violation to a chosen probability [25], while robust optimization and distributionally robust optimization will find the optimal solution to a problem that minimizes the cost of the worst case scenario or distribution [26].

Stochastic methods used in power system operations with high penetration of renewable energy, with a particular focus on unit commitment problems, are reviewed by [27]. The report emphasizes the stochastic programming formulations, including uncertainty modeling, scenario generation and reduction, solution algorithms, and chance-constrained, robust, interval, and fuzzy set-based formulations. An overview of the deterministic and stochastic methods to handle uncertainty in power system applications is illustrated in Fig. 2.

Fig. 2

Deterministic and stochastic methods for addressing uncertainty in power system applications [5, 27]

3 Previous reviews of energy and system models

Several review papers and reports on modeling tools for energy and power systems are found in the literature. An overview of this work and its coverage of uncertainty is presented in this section. The literature is reviewed to determine the prevalence of stochastic tools used in energy and power systems (Sect. 3.1), and to pinpoint trends and obstacles related to modeling of the future power system in general (Sect. 3.3), with a particular focus on the representation of uncertainty (Sect. 3.2). The review papers were also used to identify power market models addressing uncertainty, which were then considered for inclusion in the model review of Sect. 4.

The predominant approach of review papers is to review existing literature. However, some studies are based on surveys where model developers or users have answered a questionnaire [28,29,30]. The majority of the papers study named models or frameworks, while some assess models proposed by academia, or both, and others focus more on general findings and not particular tools. The following list summarizes some typical aspects covered by model review papers. The most common perspective is to provide a classification of the models, identify trends and highlight challenges, or address the suitability of the frameworks for specific applications:

• Provide a classification, taxonomy, or mapping of models [16, 28, 29, 31,32,33,34,35,36,37,38,39,40].

• Help analysts and decision-makers choose the appropriate model for their applications [9, 28, 30, 33, 41,42,43,44].

• Focus on a subset of models or specific applications, for example:

  • open-source models [45]

  • agent-based models [46,47,48]

  • stochastic models [3]

  • operational decisions [40]

  • planning [49], expansion planning [9], generation expansion planning [18, 50, 51], or storage expansion planning [17]

  • smart grids [36]

  • hydrogen energy systems [39]

  • storage modeling [52]

  • models used in a specific country/region [29, 33, 34]

• Address the suitability of current models for different energy and power system applications, for example:

  • VRE integration [30, 32, 42, 43, 50]

  • developing county applications [53, 54]

  • application at island level [38]

  • policy analysis [9, 34, 44]

  • modeling the impact of electric vehicles [55]

  • hybrid renewable energy systems [56]

  • integrated community energy systems [57]

• Focus on trends and challenges in tools and approaches [17, 28, 35] or current status [49, 58].

• Present a new model and review other models for comparison [52, 59, 60].

Some review papers give a thorough overview of previous review papers on energy and power system models [28, 44, 51]. The first two references provide a classification of the review papers according to their purpose. Savvidis et al. use four categories focusing on model description, classification scheme, field of use, and identification of suitable models [44]. Chang et al. build on the categorization from [44] and use seven categories including descriptive overview, classification, practical application, inter-comparison and suitability, transparency, accessibility and usability, policy relevance, and model linking [28]. Both papers find that almost all review papers contain elements from several of these categories. Siala et al. emphasize the difference between model reviews and model comparisons. Model reviews discuss qualitative properties and usually cover many models or frameworks, while model comparisons seek to quantify the impact of different modeling features on the outcomes [51].

This review paper will not classify review papers but instead address whether these papers cover aspects of uncertainty. Table 1 summarizes a diverse selection of existing review papers and their primary focus, and indicates if the topic of uncertainty is addressed. Several papers address uncertainty, however only [3] focuses solely on stochastic tools. In the listed work, models are classified according to several dimensions. They can be mapped according to their mathematical structure, application or purpose, technical detail, availability, technologies they represent, grid modeling, spatiotemporal characteristics, treatment of uncertainty, etc. Even though addressing uncertainty is seen as increasingly important in decision-making, several papers classify models without addressing whether they treat uncertainty or not. The following subsection summarizes findings of review papers that have categorized specific power, or both power and energy system models, to determine how frequently stochastic approaches are used.

Table 1 Papers reviewing energy and/or power system modeling tools

3.1 Most tools are deterministic

Möst and Keles focus on stochastic methods to support decision-making in liberalized electricity markets [3]. The paper covers the stochastic processes of electricity prices and other relevant commodities, scenario generation and reduction, and review 20 stochastic optimization models found in recent literature. The models are reviewed in terms of their application (investment decisions, short-/mid-term power production planning, and long-term system optimization), uncertain parameters, and the stochastic process used to describe the uncertain parameters. However, the main focus is modeling uncertainty in electricity prices and other commodities, and most of the reviewed tools take electricity prices as an input and are thus not fundamental power market models, which is the focus of this review.

Foley et al. also target liberalized markets, and review trends in electricity system modeling moving towards increased market complexity from, e.g., VRE integration [42]. They provide an overview of techniques and models for power systems (AURORAxmp, EMCAS, GTMax, PLEXOS, UPLAN, WASP IV, and WILMAR). Uncertainty is mentioned for some of the tools related to what parameters are considered uncertain, and the overall methodology. However, they conclude that in general the stochastic nature of VRE is not realistically represented, and that uncertainty due to risk and error has not been fully quantified in the modeling [42].

The readiness to analyze increasing levels of VRE is also covered by several other review papers, as seen in the introduction of this chapter. Connolly et al. present 37 tools suitable for analyzing renewable energy integration [30]. The review comprises both energy system and power system models, and several characteristics, like type of tool, sectors covered, geographical area, and length of model time-step, are given together with a more detailed description of each tool. A structured overview of whether the tools are stochastic or deterministic is not provided. However, uncertainty is mentioned for about 20% of the tools.

In the same context, Després et al. focus on both long-term energy and power system models, and present a topology that includes the general context and positioning of the model, the spatiotemporal characteristics, and details on technical and economic features [32]. Five modeling tools for the power sector (PRIMES, SWITCH, ReEDS, E2M2, and ELMOD) are reviewed and compared according to the proposed topology. The topology includes what approach is used to represent production from renewable energy sources, i.e., historical, statistically determined, or stochastic, but the aspect of uncertainty is not covered beyond this. Only E2M2 uses a stochastic representation of VRE, and this model is further discussed in Sect. 4.

Ringkjøb et al. review 75 state-of-the-art modeling tools for energy and electricity systems capable of addressing challenges faced in today’s energy system, ranging from small-scale power systems to long-term global energy systems [43]. They categorize the models into power system analysis tools, operational decision-support tools, investment decision-support tools, and scenario tools, and the review aims to help decision-makers find the appropriate tool for their problems. They build on the topology presented by [32], and conclude that most tools are deterministic and only a few of the models take into account the uncertainty of VRE generation.

Planning is an important task in energy and power systems. Both operational planning and expansion planning of transmission, generation, and storage capacity are covered by works listed in Table 1. Oikonomou et al. focus on models for operational planning and propose a taxonomy based on four core processes [40]. Twenty-three operational models are reviewed based on the taxonomy. Generally, uncertainty is incorporated in the optimisation procedures by deterministic or stochastic approaches, as depicted in Fig. 2, but only eight of the reviewed tools are found to be stochastic. According to Pourbeik et al., transmission and operational planning have historically been primarily deterministic, but increased attention has been drawn to probabilistic methods as uncertainties in the power system have increased [49]. The report reviews 18 planning tools primarily using Monte Carlo simulation methods and identifies the limited capability to address market-related uncertainties as a gap in commercial software.

Siala et al. review five power market models (DIMENSION, EUREGEN, E2M2, Urbs, and HECTOR) used for capacity expansion planning [51]. A comparison is made along the four axes of model type, planning horizon, temporal resolution, and spatial resolution. The planning horizon can be either intertemporal or myopic, which has consequences for the representation of uncertainty. Intertemporal models assume perfect foresight as the whole time horizon is known. Myopic formulations solve consecutive sub-problems, and decisions made in a certain period are arrived at without knowledge of future investment periods. Myopic models avoid the unrealistic assumption of perfect foresight. However, a myopic approach does not account for the information available regarding the future, such as expected outcomes. Beyond this, uncertainty is not covered by the paper. Another study, looking into long-term uncertainties in generation expansion models, found that uncertainty is often overlooked or simplified, that very few papers cover uncertainty in more than one input dimension, and that primarily short-term uncertainties are considered [61].

Haas et al. review trends and challenges in storage expansion planning models by analyzing and classifying 87 papers from 1970 to 2016 [17]. Uncertainty treatment and solution method are some of the classification criteria, and they review whether the proposed models are deterministic, use scenario analysis, Monte Carlo simulation, or stochastic optimization. The paper gives an overview of how the treatment of uncertainty in storage expansion planning models has evolved from 1970 to 2016, and what methods are used for different types of uncertainties. They find that uncertainties in capital costs, CO$_2$ emission prices, energy costs, VRE integration, and maximum curtailment levels are mainly studied by scenario analysis, but that the use of stochastic modeling in research has increased in the last couple of decades. However, most storage expansion planning models still have a deterministic formulation, and scenario analysis is the preferred method to account for uncertainty.

Gacitua et al. also review the literature on expansion planning models, but from the perspective of energy policy analysis [9]. The authors claim that handling uncertainty is critical when evaluating different energy policies, as uncertainty in, e.g., VRE production, load and load growth, technology costs, fuel costs, and water availability increases the risk associated with investments, and that stochastic modeling should be used to increase insight in the results obtained. They find an increased focus on stochastic programming, and robust optimization, to account for uncertainty on different timescales. However, a review of 21 existing decision-support tools finds that only six of them handle a stochastic representation of VRE. Likewise, Koppelaar et al. found a limited use of techniques dealing with uncertainty among 11 German electricity models used for scenario studies [34]. The survey was performed to investigate the ability and necessary improvements of available power system models to provide system-wide insights for policy purposes. Uncertainty is one of the six characteristics covered in the paper, in addition to system scope and modeling paradigms (optimization, equilibrium, and simulation), decision structure, technological change, and socio-political-technical interactions.

Savvidis et al. also center their research around energy policy questions and present a comparison method for energy system models based on four main categories: model-theoretic specifications, detail of modeling, market representation, and general information [44]. The model-theoretic specifications are based on a classification scheme proposed by [33]. This scheme comprises 14 categories accounting for the purpose, structure, approach, mathematical, and technological detail. Uncertainty is only addressed in the criterion describing the underlying methodology, which can be stochastic or Monte Carlo. Analysis and categorization of 22 energy system models used in the UK reveals that only one of the models uses Monte Carlo, and none of the models are stochastic [33]. Savvidis et al. include three additional criteria compared to [33], one of which is the representation of uncertainty and risk. However, this criterion is merely a yes/no question and does not allow for any information on how the uncertainty is modeled. The author provides a linkage between policy issues and model features to aid in identifying suitable energy system models for specific policy research questions. However, due to the small set of policy questions and the limited number of models, the modeling of uncertainty is not identified as an essential feature nor a gap in addressing the policy issues analyzed.

A broader, but systematic, mapping of 82 power system models available in, or used by, European organizations has been developed by [29] and concludes that most applied power system tools are deterministic. Both software- and model-related features, problems addressed, technologies represented, sectors covered, and applicability have been mapped to provide an overview of the models and their applications. The survey finds that 37 tools solve stochastic or probabilistic problems, but only 23 use stochastic programming (the rest use probabilistic Monte Carlo approaches). According to the survey, uncertainty in load and renewable energy are given most consideration, followed by fuel prices.

Another review providing a thorough classification is offered by Groissböck et al., where 81 functions are proposed to assess the detailed technical capability of modeling tools, including model foresight, risk level, and uncertainty [45]. However, the paper only covers whether the functions are considered or not, with no details on how they are considered. Despite the high number of functions, the authors state that additional aspects could be included, for example, within uncertainty, by considering scenario tree generation and scenario tree reduction methods. Thirty-one open-source optimization models for energy systems are reviewed according to the proposed functions. Uncertainty in profiles is considered in eight of the tools, but risk is not considered.

From the reviewed literature including uncertainty in the classification or description of the models, it can be concluded that most approaches used in decision-making today are deterministic. Several review papers have noted a growing emphasis on stochastic techniques and the importance of addressing uncertainty in future power and energy system forecasting. However, few of the papers listed in Table 1 focus on the details of what and how uncertainty is handled in each reviewed tool. This review paper explores fundamental power market models for operational use that take into account uncertain input data, and it will provide details on the types of uncertainty considered and how they are described. First, the next subsections will summarize our findings on why uncertainty is often overlooked or simplified, and the competing considerations that complicate the modeling of future power systems.

3.2 Challenges and trends in accounting for uncertainty

The main reason for using deterministic approaches over stochastic is the increased computational burden from including stochastic parameters [18, 40]. In addition, there are challenges related to data availability and data management [40, 49], and describing different uncertain parameters and their associated probabilities is not straight-forward. When modeling several uncertainties, questions arise regarding their correlations and how the occurrences of these uncertain parameters combine [3, 18]. The decision-maker must also decide if the tool should convey an attitude toward risk, and defining appropriate and acceptable risk-levels can be challenging [49]. Lastly, when including uncertainty in models, challenges related to balancing uncertainty and transparency [16], and how to interpret results [49], must also be addressed.

Many identify uncertainty as a key challenge or gap in power system modeling [16, 18, 45] and contend that further efforts in this field are needed, primarily when stochastic models are used in supporting decision-making on a daily basis [3, 49]. Several directions for future research related to uncertainty modeling are proposed in the literature. Fernandez et al. conclude that stochastic or probabilistic approaches should be used to deal with the stochastic nature of renewable energy sources, but that more attention should be devoted to other uncertain parameters such as hydro inflows, thermal power plant availability, investment costs, or policy impacts [29]. Möst and Keles find that CO$_2$ emission prices are rarely considered uncertain and suggest this topic for future research [3]. Treatment of uncertainty both in the long-term and short-term is identified by [18] as one of three gaps offering interesting opportunities for future research on generation expansion models, in addition to enhanced representation of operational flexibility and smart grid technologies. Developing more efficient scenario reduction and decomposition techniques and improving solution methods are areas of future research to address the challenges in increased computation time [3, 17].

Some papers also highlight the extreme outcomes of uncertain events. Unexpected events that can have a high impact on the power system, but are characterized by very low probabilities, are called high-impact low-probability (HILP) events [49]. Pourbeik et al. argue that HILP events should be considered in both planning and operational studies, and that there are several improvements related to methods, data, and software to analyze HILP events adequately [49]. Sufficient knowledge of such events is needed, including methods to appropriately represent such events in modeling tools. A review focusing on risk assessment in state-of-the-art generation expansion models found that investment decisions are significantly influenced by a series of economic, political, regulatory, environmental, technical, social, and climate uncertainties and that they should be considered, at least partially, in the models to account for the increased level of risk [62]. In addition, extreme events such as fluctuations in fuel prices, problems with natural gas or nuclear energy supply, and sudden outages from, e.g., terror attacks on energy infrastructure, should also be taken into account, especially if the focus of the models is to evaluate investments or support security-of-supply.

3.3 General modeling challenges and trends

Several other challenges in power system modeling are identified in the review process. The literature suggests that power system models must develop along many axes to adapt to the ongoing changes in the power system. To summarize, models should have a finer temporal and spatial detail to be able to represent the flexibility of storage [17] and the variability and decentralized production of VRE [16, 28, 32]. Short-term dynamics must be included in long-term models to verify investments and properly evaluate storage assets [43, 52]. Energy storage will play a critical role in future power systems as a flexibility provider, and a detailed representation of different storage assets and the time-linking constraints introduced by storage needs to be considered to adequately capture the dynamics of future power markets [17, 42, 52].

A tighter integration between the electricity sector and other sectors such as heat, transport, and gas, mainly due to electrification and sector-coupling, must also be taken into account in power system models as it adds flexibility to the system but also introduces additional uncertainty and risk [28, 40, 43, 63].

In energy and power system models, technical and economic aspects are typically considered. However, how human behavior (e.g., consumer behavior and technology deployment) and social aspects (e.g., local opposition to wind farms) affect power markets and the development of the power system is challenging to incorporate. Social and environmental aspects have been a factor in the development of hydropower projects for a long time, and it is expected that restrictions will become tighter in the future. More attention should be paid to human, social, environmental, and political factors as they are a primary driver for uncertainty, especially on longer time scales [16, 43].

The behavior of different market participants also adds uncertainty in electricity markets, and this behavior is not taken into account in optimization models that assume a perfect market. With the transition to deregulated electricity markets, there has been a notable increase in the utilization of agent-based models (ABM) for simulating strategic behavior and gaining valuable insights [64], and future work should focus on reducing the complexity of such models [48] and developing suitable learning algorithms [46]. As decision-makers can have several objectives, e.g., emission reductions in addition to the most cost-optimal solution, multi-objective optimization has gained increased interest to fulfill multiple goals [17, 18]. There is also an increased focus on openness, accessibility, and transparency within energy and power system modeling [16, 28].

Many of the proposed improvements will, similar to uncertainty treatment, increase model complexity. The increasing complexity of power and energy system models is evaluated according to temporal, spatial and mathematical complexity, and system scope by [37]. The approach to account for uncertainty is a part of the mathematical complexity of a model, but it is not discussed further in the paper. The study finds that complexity is allocated to prioritize those features and properties that are particularly important for the purpose of the tool. Pfenninger et al. suggest that complexity can be reduced to allow for more extensive uncertainty and sensitivity analysis [16]. They find it challenging to extend existing large-scale energy system models to include stochastic parameters, as only a few scenarios can be considered within the computational limits. Increased complexity leads to increased computational burden and reduced transparency. Computational tractability is identified as a critical challenge. Scenario reduction and clustering-techniques, decomposition techniques like Benders decomposition [65], and progressive hedging [66] are methods used to make optimization problems tractable [9]. Due to the increased complexity and related computational burden of models adapting to address these challenges, research should also focus on solution methods and techniques to solve these problems [9, 17]. Linking models with different scope and strengths can be used to improve modeling without adding complexity [28, 32].

4 Power market models with uncertainty representation

Based on the literature study presented in the previous section (Sect. 3), it can be concluded that a proper representation of uncertainty is seen as crucial when developing power market models. Still, only a few existing frameworks take uncertainty into account. Furthermore, many of the presented papers and reports identify if uncertainty is regarded, but details on what types of uncertainties are represented and how they are described and revealed to the model are not covered. A contribution to bridging this gap is provided in this section with a more detailed review of fundamental power market models handling uncertainty.

Around 200 models for energy and power system applications were identified in the initial process of this review, where most tools were found to be either purely deterministic or covering several energy sectors, and thus not included in this study on fundamental power market models handling uncertainty. To limit the selection further, we distinguish between models supporting operational decisions and investment decisions. However, this distinction is not absolute as some tools can include elements of both. Nonetheless, frameworks primarily addressing expansion planning problems are not included.

Historically, representation of uncertainty in models for operational planning has been important in hydrothermal power systems [4]. With the increasing penetration of intermittent energy sources, uncertainty during the operational phase in power systems is likely to increase. Therefore, tools simulating the operation of the power system or power market under uncertainty over short or long time horizons are reviewed. Furthermore, another challenge in the application of models to realistically sized problems is striking a balance between incorporating modeling details and ensuring the problem remains tractable for effective solution. This trade-off is most prominent for larger-scaled systems. Therefore, this review’s scope is also limited to operational models applied or applicable to a realistic, large-scale system typically covering one or several countries. These tools are used for analyzing system and market dynamics, in addition to operational decisions and planning. Hence, they are used for decision-making or decision-support. This implies that the models must be solvable within a reasonable time with realistic input data (e.g., ENTSO-E system or Nordic system).

The power market models presented in Table 2 are chosen based on the above criteria. Still, the list is not exhaustive, as some models were not included due to the limited availability of proper technical/scientific documentation. Furthermore, applied tools can have a lot of functionality to fulfill different use cases, but the availability of detailed information about all functionalities can be limited. Therefore, we aim to describe how the model is presented in the provided references instead of trying to outline all possible functionality.

This section starts by describing the main characteristics of the tools in Sect. 4.1, followed by a review of the uncertain parameters considered and how they are described in Sect. 4.2.

4.1 Model overview

Table 2 shows the reviewed models, including the time horizon, the country or region in which the tool is primarily used, the main problem addressed, and how the model is formulated and solved. Operational tools covering uncertainty are found for all time horizons, from models solving the hydrothermal coordination problem (HTC) or analyzing system adequacy (SA) for long time horizons (years), to short-term (days) models solving unit commitment and economic dispatch (UC &D) problems or modeling multiple markets (MM). Several countries and regions are represented, but the majority are developed and applied in Europe. The problem formulation describes the structure of the problem that is solved, including how uncertainty is represented on a general level. Most reviewed tools are based on stochastic programming, and either solve a multi-stage/two-stage stochastic linear problem (MSSLP/TSSLP) or a stochastic mixed integer linear problem (SMILP). Others use agent-based or Monte Carlo simulation. In agent-based models (ABM), several optimization problems are solved to represent the behavior of market participants. These small optimization problems can be stochastic, and data from the past and forecasts for the future can be used to support the decision-making process of each agent. Notice that all frameworks are based on probabilistic approaches, e.g., stochastic programming or Monte Carlo simulation, and none use the other stochastic methods shown in Fig. 2.

The recent attention on accounting for weather-related unpredictability in renewable power production is not novel to systems that heavily rely on hydropower technology. For decades, sophisticated tools solving the long-term hydrothermal coordination problem have been implemented in hydropower-dominated power systems. These models can be used as benchmarks for introducing uncertainty into power market models. The representation of uncertainty is crucial in the long-term operational planning of hydropower assets due to uncertainty about future inflow. Hydropower inflow varies on both short and long time scales and can have strong seasonal patterns. The long-term storage capacity of reservoirs gives rise to the question of whether to produce today or store the water for later. The concept of employing stochastic methods for optimal scheduling of reservoir hydropower can be traced back to 1946, where Massé [67] argues why deterministic models are too optimistic and the necessity of considering the stochastic nature of future conditions [4]. Today, state-of-the-art hydropower planning involves finding the value of the stored water using stochastic multi-stage long-term power market models, traditionally solved by variants of stochastic dynamic programming (SDP) [68] or stochastic dual dynamic programming (SDDP) [69], as we see for EMPS, VALORAGUA, NEWAVE, PSR-SDDP, and DOASA. A proposed methodology, applied by FanSi, solves a two-stage stochastic problem using a rolling horizon approach and Benders decomposition [70]. These models are applied in, e.g., the power systems of the Nordic countries, Portugal, Brazil, and New Zealand. In addition to operational planning, such long-term power market models are applied by hydropower producers, regulators, and transmission system operators for analyzing power systems and investments, and making predictions about the future.

The ANTARES simulator was developed to address system adequacy and transmission efficiency in the long-term [71]. To analyze system adequacy, a large number of random variables and possible combinations must be modeled, and sequential Monte Carlo simulation is suitable for analyzing all these possible outcomes. E2M2 is used for both generation expansion (GE) and unit commitment (UC) and has been applied to estimate the operational and investment costs associated with integrating more wind power in the German power system [72]. The problem horizon is long-term, but to limit the computational burden, each year is solved sequentially in a myopic planning horizon where each year is described by 12 representative days. The EMCAS model is used to model electricity market dynamics on several time-scales [73]. Agent-based modeling is used to represent the strategic behavior of different market participants with limited knowledge interacting on different layers over different planning periods where decisions are made.

Due to increased levels of uncertainty in power markets, representation of uncertainty in short-term market models is becoming more important when solving unit commitment and economic dispatch problems [74]. WILMAR and stELMOD solve the stochastic mixed-integer linear optimization problem for unit commitment with a rolling horizon approach, and cover several markets [75, 76]. Multiple markets are also modeled in the short-term tool SiSTEM, where optimization and agent-based simulation are combined to sequentially model the day-ahead market, intraday market, balancing activation, and imbalance settlement [77]. METIS is a modular tool covering the electricity, gas, and heat sector, and it is used by the European Commission to support policy making for electricity and gas [78]. A fundamental power market module simulates the successive clearing of markets for reserves, day-ahead, intraday, and balancing.

Table 2 Reviewed models handling uncertainty

4.2 Uncertain parameters

Table 3 offers an overview of the parameters considered uncertain in the reviewed tools. Uncertainty in demand, hydro inflow, VRE production (wind and solar), fuel prices, and forced outages were the uncertain parameters found to be represented by the reviewed models, i.e., other uncertain factors are represented by one single outcome. Most models represent uncertainty in demand and wind power production, and all long-term power market models focusing on solving the hydrothermal coordination problem include uncertainty in inflow. Random outages of generating units (or transmission lines) are covered by most of the models not addressing hydrothermal coordination. Few power market models include stochastic fuel prices, and no models were found to address uncertainty in CO$_2$ prices or other uncertain parameters.

Table 3 Parameters considered uncertain by the reviewed models

Uncertainty in demand, VRE, and hydro inflow is typically represented by most of the models included in this review. These are uncertain parameters that can be described and represented by PDFs, and often these are estimated based on historical data. Using historical time series directly as scenarios in stochastic models has the benefit that correlations in both time and space are captured. This property is hard to identify using sampling methods [70]. Historical series are used for inflow scenarios in VALORAGUA, and the uncertainty is revealed in weekly time periods (weekly perfect foresight) [87]. Historical data is also used in the EMPS model, where the probability distributions for the stochastic variables in different weeks are calculated based on statistics for actual outcomes for these variables in the past [83], and the uncertainty is revealed in weekly decision stages. In DOASA, weekly decision stages incorporate the use of inflow series sampled from the historical weekly inflow series. The inflows are assumed to be stage-wise independent in the model. However, inflow is typically stage-wise dependent, and the model can take this dependency into account by using an approach called Dependent Inflow Adjustment to adjust the inflows, while still assuming they are independent [80].

In the NEWAVE model, uncertainty is revealed in monthly stages, and the energy inflows are generated from a statistical monthly energy inflow model based on a periodic auto-regressive (PAR(p)) model [84, 88]. This synthetic inflow generation model has recently been extended to generate monthly multivariate synthetic sequences of both inflows and wind speeds, while also considering their correlations [89]. A periodic auto-regressive model is also used to generate inflow data for PSR-SDDP [85]. Scenarios for VRE can be provided directly to the model or generated in a Time Series Lab where a statistical model uses historical data, either from real measurements or from synthetic historical records created based on reanalysis data, to generate future scenarios for the model. The model supports both weekly and monthly decision stages. Uncertainty in demand can be represented by a normal distribution. The PSR-SDDP model has the functionality to handle fuel price scenarios, but no information is found on how these scenarios combine with the other uncertain parameters. In addition, the scenarios are user-defined, and the question arises of how to generate good scenarios for fuel prices.

For ANTARES, uncertainty in monthly hydro energies is estimated by a random process where the distribution of the monthly energies follows a Log-Normal fitting of the historical data [71]. The auto-correlation of the successive monthly hydro energies follows an exponential fitting. Randomized time series for wind and load data is generated based on the historical data of wind and temperature. The uncertainty is revealed in weekly stages, and correlations in time and space are maintained.

For short-term models, uncertainty is typically revealed between the clearing of sequential markets. A Scenario Tree Tool is used in the WILMAR model to generate a multi-stage scenario tree based on historical generation profiles or wind speed data, data on electricity demand, and data on outages and the mean time to repair [75]. Forecast errors for wind and demand are simulated using an auto-regressive moving average (ARMA) time series model. Scenario reduction is performed by combining the individual forecast scenarios with the smallest Euclidean distance. A scenario tree is also used to represent the stochastic generation from renewable energy sources in stELMOD, where the same Scenario Tree Tool is used to simulate forecast errors. In E2M2, uncertainty in wind power production is represented by a recombining tree. The method used to generate the tree is also based on the Scenario Tree Tool. METIS includes a stochastic module, which simulates power plant outages and forecast errors for demand and VRE generation from day-ahead to 1-h ahead. The spatial and temporal correlation between temperature (affecting demand), wind, and irradiance are preserved, as it builds on historical time series [78].

Derating the capacities of power plants and power lines is an easy method to take into account random outages in long-term models. However, this method only accounts for the loss of production and transmission capacity on average, and the actual uncertainty from random outages is not taken into account in the decisions made by the model. As a result, this method does not catch the variations that occur due to such random outages. This method is used by, e.g., VALORAGUA. It is challenging to make scenarios for both planned and especially unplanned outages, and to know the correlations between such events or the combined probability of failures. The forced outages of power plants in the WILMAR tool are simulated with semi-Markov chains [75]. For ANTARES, both planned and unplanned outages of thermal power plants are also modeled as a semi-Markovian process [71].

From a system perspective, the traditional measure of dealing with uncertainty from load forecast errors and unexpected outages of transmission and generation capacity has been to allocate reserve capacity in the markets to ensure system security. Reserve capacity is activated in the balancing markets to maintain the balance between supply and demand in real-time. Reserve requirements can be included in deterministic models, and are often based on simple heuristics (e.g., N-1 criteria) and static reserve rules, but the trend is moving toward dynamic reserve requirements based on forecast uncertainty [27]. In such approaches, uncertainty is not explicitly represented when solving a model and procuring reserves, as it would be with a stochastic market clearing or a stochastic model. The E2M2 model indirectly handles the uncertainty in availability of power plants by estimating reserve requirements. These requirements are determined by a probabilistic method where a cumulative outage distribution is calculated for similar generation units that are assumed to have the same probability for an unscheduled outage [81].

5 Discussion

The level of uncertainty in decision-making processes in power markets is increasing. To handle the risks associated with uncertainty, knowledge of how the outcome of a decision is affected by different realizations of uncertainty, and the probability of different unfortunate events, is highly valuable for decision-makers to obtain robust and flexible solutions. Such insights can and should be provided by using modeling tools. However, there are several questions that must be answered and challenges to address when tackling uncertainty, including:

• 1. What types of uncertainty have the most impact on the problem being solved?

• 2. How can this uncertainty, and its possible correlations, be described?

• 3. How should the problem be formulated and the uncertainty be revealed to the model?

• 4. How can this large-scale problem be solved in a reasonable time at a desired level of accuracy?

In the following, we elaborate on these topics. The questions and challenges should not be addressed in isolation as they are highly dependent on each other. The process of formulating a power market model to make decisions under uncertainty can be summarized in four main steps, as illustrated in Fig. 3.

Fig. 3

Questions and challenges exist within all steps in the process of developing a power market model handling uncertainty

5.1 Representation of uncertainty

Most often several types of uncertainty are found to have an impact on decision-making, and a number of assumptions on input data is made when solving a mathematical model of the power market. Some assumptions will typically have a higher impact on the results than others. What type of uncertainty has the most impact on an outcome is highly dependent on the problem that is being solved, and the characteristics of the power system, market design, and policy and regulatory frameworks. For problems with a long time horizon, some of these factors may themselves be uncertain. For example, for VRE investment decisions, uncertainty about load growth, support schemes, and emission reduction targets can be important, in addition to technology cost. How the impact of uncertain parameters is measured, i.e., what risk measure and problem formulation is used, will also guide the decision regarding what uncertainty to include or analyze [5].

The time horizon analyzed is also highly relevant to the question of what uncertainties to include. Velasquez et al. categorize uncertainty based on in what timescales they are revealed [6]. Short-term uncertainties like hourly load, VRE production, and outages are important for short-term operation hours to months ahead. Looking years ahead, demand growth, fuel costs, and hydro inflow are examples of medium-term uncertainties affecting medium-term planning, like generation expansion planning or hydro reservoir operation. For long-term strategic planning over decades, consumption patterns, disruptive technologies, and the effects of climate change are difficult to predict. Depending on the decision-support needed and analysis made, e.g., investment decisions, different policy analysis, power system adequacy, hydropower scheduling, or bidding in sequential markets, different parameters should therefore be considered uncertain. Due to the increased model complexity from adding uncertainty and limited computational resources, care should be taken in representing the randomness of the uncertain parameters that have the strongest effect on the results. Uncertainty that is hard to describe are in danger of being left out. One study warns against modeling what is easily quantifiable rather than the essential driving variables of the system [16].

In hydropower dominated areas, uncertainty in inflow will have a large impact on the outcome of the power market. Similarly, for power markets with a high penetration of wind power production, uncertainty in wind speed will have a high impact. Weather-patterns can vary a lot across all time-scales and, as illustrated in Sect. 3, weather-related uncertainty is the type of uncertainty most frequently covered by power market models handling uncertainty. To understand the possible future outcomes of power prices, a correct representation of wind speed, solar radiation, inflow, and temperature is critical as the power systems become more weather-dependent. In addition, uncertainty about the availability of transmission capacity and generation units is often included, especially in short-term models.

Some important price drivers associated with uncertainty are fuel and CO$_2$ emission prices. This was especially prominent during 2021 when energy shortage and climbing fuel and CO$_2$ emission prices resulted in record-high electricity prices in Europe. Uncertainty in fuel and CO$_2$ prices is generally not represented in the reviewed power market models, and the impact of different outcomes has traditionally been assessed by performing scenario or sensitivity analysis. However, the overall impact when several uncertain assumptions materialize in a strained power situation is difficult to model if the randomness of these factors is not included. In addition, the combination of several uncertain factors, including random outages of generation and transmission capacity, will help to more accurately capture the variation in electricity prices. When examining storage and other flexible assets, it is crucial to consider this variation. Moreover, uncertainties that are not considered in today’s tools may be of increased importance in future operational power market models, and new uncertainties introduced in power systems (e.g., from demand response) should be investigated and modeled to obtain realistic results [7].

5.2 Description of uncertainty

There are many different types of uncertainty, and how they can be described will depend on the available knowledge and data, and on its structure. The structure of uncertainty can be classified as “known”, “unknow”, and “unknowable”, and pertains to how the uncertainty can be described mathematically [6]. This is connected to the methods for uncertainty modeling briefly presented in Sect. 2.1 and summarized in Fig. 1. “Known” uncertainties are the most structured and can be described mathematically by PDFs. “Unknown” uncertainties do not follow a probabilistic pattern and are harder to describe mathematically (e.g., investment costs, outages). A common approach is to use intervals or set bounds on the outcome. “Unknowable” uncertainties are the least structured type and represent unforeseen events called black swans. They can be terror, war, natural disasters, and are best described by scenarios. However, assigning probabilities to such scenarios is difficult.

The relevant time frame will also affect how some parameters are described, as the short-term and long-term description might follow different distributions [5]. In addition, some parameters are highly correlated in the short-term to the current situation, like weather conditions and fuel prices, but the current situation may have limited impact on the parameter further into the future. Weather-related uncertainty can be predicted quite well a few days ahead based on forecasts, but looking years ahead, historic data will provide a better description of the possible outcomes of these parameters. However, a description of the past will never be a perfect description of the future, and future outcomes may be distinct from the historical ones. In addition, climate change affects weather-patterns, and the historical data may not fully represent future weather conditions [90, 91]. One example is the weather observed in Norway during 2020 and the preceding years, which is not found in the historical data [92]. This makes it questionable whether history is suited for accurately predicting the future. Efforts should therefore be made to adjust the data for climate-related changes and develop appropriate techniques to account for this, such as evaluating synthetic future scenarios. However, as seen in Sect. 4.2, historical time series for weather-related uncertainties are frequently used as scenarios in a stochastic model due to their ability to effectively capture the geographic, cross-, and auto-correlation of various time series for renewable energy and temperature. This is particularly difficult to replicate in synthetic scenarios or probabilistic models [70, 93].

Regardless of where they come from, many scenarios are needed to accurately capture the possible realizations of uncertainty. The number of scenarios must, however, be limited in a stochastic model to make it tractable, and appropriate scenario reduction techniques must be applied. Random parameters that follow stochastic processes for which there exist large amounts of historical data can be fitted to probability distributions and simulated to generate a large amount of scenarios. However, the true distributions of the random parameters are not fully known, and distributionally robust optimization is a method that can handle several possible distribution functions. For other types of uncertainty where PDFs are hard to obtain, the possible outcomes can be described as intervals or scenarios. Such values can be obtained from experts in the field or from other modeling exercises.

Power market outcomes are to an increasing extent affected by weather conditions. Due to the large amount of historical weather data, the stochastic nature of VRE, inflow, and temperature can be estimated and included in modeling tools. Random outages can also be described relatively easily if their probabilities of occurrence, failure rate, and repair rate are known [7]. Although historical data exist for electricity prices, fuel prices, and partly CO$_2$ prices, these time series cannot directly be used to estimate how future prices might develop as they are affected by several underlying factors. Alternatives can be to use forward prices, scenarios based on expert assessments, or other models to obtain forecasts, e.g., fundamental models of gas markets. Other uncertainties can be even harder to predict and describe, and this review found no models accounting for “unknowable” uncertainties.

Describing the individual uncertainties is one challenge, but in a power market where uncertainties are growing both in number and severity, a key challenge is how to capture and describe the combined uncertainty. According to [49], the probability that several random outages happen simultaneously is larger than could be expected by assuming all events were independent. When modeling multiple uncertainties, there are often correlations that should be taken into account. However, many uncertainty modeling methods are not suited to handling correlations. Weather-related uncertainty can be seen as one dimension of uncertainty as it is correlated in time and space. This uncertainty is often factored in, as seen in Sect. 4, but its correlation to other uncertain factors are hard to estimate and most research assumes that there are no correlations [11]. Developing appropriate techniques for modeling the correlation among different uncertain power system parameters in a realistic way is identified as a future research need by [11, 94]. In addition, including several dimensions of uncertainty will rapidly increase the number of scenarios in a model when these dimensions are combined. One possible solution is to adapt techniques that determine the worst or most extreme combinations, if the goal is to find solutions that are robust.

5.3 Problem formulation and solving approaches

In addition to what sources of uncertainty to include, how to formulate the problem under uncertainty is also an important choice [95]. How the uncertainty to consider is described, and the risk preference of the model user, will impact how the model is formulated [5]. Two-stage or multi-stage stochastic problem formulations are often used as they resemble the interaction between decision-making and the arrival of new and updated information. In a short-term multi-market model, decisions on how to bid into the markets are taken in a sequence moving closer to real-time operations. Uncertainty is reduced between the clearing of different markets as more information becomes available and more certain. The long-term hydrothermal coordination problem is also often formulated as a multi-stage stochastic problem, where the decision stages are weekly or monthly.

Many electricity market models assume a perfect market where all participants have access to the same information, and the uncertainty of how different market participants will bid into the market is neglected. Under this assumption, optimization models are often used. In markets where this assumption is weak, representing the behavior of agents can provide valuable insight. After market competition was introduced by the deregulation of electricity markets, agent-based modeling gained increased attention due to its suitability for representing the behavior of different market participants in the electricity markets [64].

The problem formulations can also be deterministic. Solving deterministic tools several times will provide insight into what different futures might look like. However, such methods provide limited support for how to plan as best as possible for different outcomes. In a deterministic framework, decisions will never be made “just in case”, and the solutions might be sub-optimal if they are based on wrong assumptions. The robustness and flexibility of a solution measures its effectiveness to withstand uncertainties [18]. The solutions provided by deterministic optimization models can be too optimal, e.g., the solutions are vulnerable and hard to modify if assumptions on uncertain inputs turn out wrong. A method called modeling to generate alternatives is a structured way to obtain and explore several solutions to an optimization problem that are close to optimal [16].

Lastly, as discussed in Sect. 3.3, when formulating a problem under uncertainty, the size of the problem increases significantly. To keep the problem manageable, it can be split into several problems with different time horizons, levels of detail, and time resolutions. Some of the models reviewed in Sect. 4 (e.g., WILMAR, NEWAVE, and EMPS) are part of a model hierarchy or scheduling toolchain where models with different scope are linked together, and information can be passed between the models to refine the results. This allows for different types of uncertainty to be considered, as the problem formulation and time horizon can be customized for each model. However, as concluded in Sect. 3, the main reason for not using stochastic tools is the increased computational burden. Therefore, one of the greatest challenges when it comes to including uncertainty in power market models is within solving the problems.

Solving huge problems that include a stochastic representation of several uncertain parameters and represent a realistic, large-scale system represents a significant task. As elaborated on in Sect. 3.3, the complexity of tools for modeling power systems and markets is increasing, and advancements in decomposition algorithms and solution methods are necessary to cope with the increased problem sizes. Different decomposition techniques exist to divide the problem into several sub-problems. Examples are Benders decomposition and SDDP based on cutting plane methods or dual decomposition and progressive hedging based on Lagrangian relaxation [5]. When solving large problems, approximations can be made to obtain a solution at the cost of solution quality [21, 96]. Heuristic methods trade precision for speed and can also be applied to solve such problems.

In addition to the solution methods described, another challenge is to utilize hardware development in an efficient way to be able to solve ever-growing problems.

5.4 Recommendations

From the reviewed work, it is evident that uncertainty is often covered in model reviews addressing expansion planning, policy analysis, system security and adequacy, and long-term storage, or in papers focusing on modeling tools for VRE integration and liberalized markets. From the reviewed power market models, we find that uncertainty is represented in long-term tools for hydrothermal coordination, short-term multi-market models including unit commitment and dispatch, and models for analyzing system adequacy and market dynamics. Uncertainty is thus especially important to account for in weather-dependent power systems, in managing long-term energy storage, for planning investments and supporting decisions with limited possibility for recourse action, and for long-term system adequacy analysis.

The main goal of representing uncertainty in decision-support tools is to make better decisions, e.g., more cost-beneficial, secure, or risk-reducing, and to provide insight into how power markets will evolve under different assumptions. Incorporating uncertainty will not, however, automatically lead to better decision-making and analysis. Assumptions made regarding uncertainty can be based on insufficient knowledge or data, and including uncertainty may come at the expense of other details, as discussed in Sect. 3. The trade-off between model detail, details in describing uncertain parameters, and the number of uncertain parameters to include should be carefully considered to obtain tractable problems. Proper assessment of the value of including different details (e.g., uncertainty versus granularity) for each unique problem is therefore recommended.

The diversity in markets and types of problems addressed in electricity market forecasting requires that tools are tailored to the specific area and application in mind to capture the dynamics of the market and uncertainties in the best possible way, instead of forcing all problems into the same framework. There is a danger that methods and tools will be chosen due to their familiarity, and not necessarily because they are best suited for the particular problem at hand [16]. To bridge the gap between research and real-world application, a close dialogue is recommended between tool developers and decision-makers to guide future research and to identify which features decision tools have and need [28].

By reviewing existing power market models, we identified a gap in models representing multiple dimensions of uncertainty and that can be solved for a large system in a reasonable time. Scott et al. investigate the importance of representing a wide range of long-term uncertainties in electricity market modeling for generation expansion planning and descriptive market modeling [61]. They compare the solutions of a deterministic model, the average solutions of performing scenario analysis, and Monte Carlo analysis with the stochastic solution and find that the stochastic solution outperforms the deterministic approaches. In addition, they conclude that the value of the stochastic solution increases when several uncertain factors are considered, and that combining uncertainty sources outperforms adding additional scenarios to any individual source of uncertainty. In other words, representing a variety of uncertainty trumps more accuracy in the representation of individual uncertainties in their study. Whether the same results hold for other types of both long- and short-term applications should be further addressed. Future research should also investigate the added value of accounting for uncertainty.

Powell reviews the different views of stochastic optimization and concludes that relatively little attention is given to how uncertainty is modeled in the literature on stochastic optimization [97]. It is proposed that more research efforts should focus on the intersection between uncertainty modeling and stochastic optimization. Stochastic optimization literature is also found to be mostly focused on expectation-based objectives, as was the case for the power market models reviewed in this work, where only the NEWAVE model was found to include a risk measure (CVaR) [98]. There is growing literature on the use of risk measures, but [97] recommends that research should dig deeper in terms of addressing issues related to specific applications. Additionally, related to an increased focus on system resilience and HILP events, exploring problem formulations that incorporate elements of robust optimization and distributionally robust optimization could be an interesting direction for future research.

In the end, not all parameters can be considered uncertain. Predictions will never capture all possible outcomes, and the results are only meant to support the decision-making process [42]. The unexpected events not represented can always happen, and hedging against the unknown risk that cannot be predicted is valuable [16]. “Unknowable” events, often referred to as black swans, are difficult to capture in stochastic models that only deal with things that are known and can be described. Extreme events that are not thought of will not be considered with the traditional methods but can have a large impact on the market. With the increasing number of uncertain parameters affecting the power markets, other more common events that are disadvantageous can occur simultaneously and result in quite extreme situations. The energy crisis in Europe has been called a “perfect storm”, as it is driven by several coincident factors including increased CO$_2$ prices, low wind power production, dry conditions, and rising demand in addition to climbing fuel prices [99]. This development demonstrates that the combination of several uncertain factors, including HILP events, needs to be analyzed and taken into account to fully capture extreme outcomes. However, such situations are not adequately considered by the power market models reviewed in this paper, and the solutions provided by these models risk not being robust against such incidents.

6 Conclusion

From the literature survey presented above, we uncover that several review papers and reports focus on the classification and suitability of modeling tools for energy and power system applications. Stochastic modeling has gained increased attention in the literature on power system modeling in recent years, especially for studying VRE integration. However, an extensive review of the literature shows that uncertainty is often overlooked or simplified in modeling tools due to the added complexity and computational burden introduced by stochastic variables, and the impact of different scenarios are often analyzed using deterministic tools. In addition, representing uncertainty in models requires knowledge and data about the distribution and outcome space of uncertain parameters, including their correlations, and generating appropriate scenarios is challenging.

Previous model review papers addressing uncertainty primarily focus on whether uncertainty is treated by the tool, but they do not cover details on what types of uncertainties are represented and how they are described and revealed to the model. This paper reviews a selection of power market models handling uncertainty and finds that weather-related uncertainty, including demand, is typically covered. Random outages are also covered by several short-term power market models, while uncertainty in fuel and CO$_2$ emission prices were generally not found to be included, nor other types of uncertainty. Most reviewed tools are based on stochastic programming, and we identified a lack of models covering multiple dimensions of uncertainty and that are also suitable for realistic, large-scale applications. The effects of analyzing a larger outcome space by including more extreme events and multiple uncertain factors in the models should be further investigated, and model developers should keep in mind that balancing transparency, complexity, and computation time is crucial when adopting models for this purpose.