1 Introduction

One of the biggest challenges Europe has to face is how to reach COP21 targets [1] for reducing greenhouse gas emissions or even Carbon Neutrality. At the same time, the European energy system must maintain its high-quality energy supply that always meets energy demands. This task calls for a massive transition of the pan-European energy system that will not only affect the electrical energy system but also impacts all energy-consuming sectors and their relevant energy modes. It includes the sectors of electricity, heating, cooling, and energy needed to transport passengers and freight. These are connected via sector coupling effects and cannot be optimized independently from one another.

Increased exploitation of renewable energy sources (RES) is indispensable. On the one hand, a higher share of RES will support the transition. On the other hand, it will bring new challenges to the system, among them the increased need for flexibility and additional requirements for ensuring the security of supply. Furthermore, the correlated energy transport systems, i.e., electricity distribution, transmission networks, and the gas transport network, must be adjusted accordingly. Additionally, consumers will have to switch to enhanced use of low-emission energy sources and implement highly efficient technologies to reduce their consumption. This requirement holds not only for electricity but also for all other energy vectors of relevance. Ongoing competition for the best applied low-cost and low-carbon-emitting “green, efficient” technologies will support this change. This will lead to a fundamental transition of the existing pan-European energy systems, requiring sophisticated planning and analysis of all options and possible pathways.

To address these challenges, an integrated pan-European system promoting strong cooperation and alignment between the individual countries is needed. In a pan-European context, holistic investment planning must be done for an optimal energy mix, while the macro-economic costs of the transition shall be minimized. Such a holistic view includes sector coupling. For example, the coupling of electricity transport—on cross-border, transmission, and distribution levels—to gas transport becomes more vigorous.

During the transition, two disruptive changes have happened that have introduced further challenges. First change is the emergence of new technologies to decarbonize gas networks by repurposing them to transport other molecules such as hydrogen or ammonia together with natural gas or separately. Second one is the political crisis in Europe changing the gas supply sources of Europe drastically. These changes, which have directly affected the operation of the gas networks, have made the detailed mathematical models with higher precision indispensable to account for the operational changes in the energy system as the experience from past has become insufficient and even no longer valid in some cases.

In recent years, several studies have addressed macro-economic planning of the energy transition, sector coupling, and power system modeling in an integrated way with various spatio-temporal resolution and span [2,3,4,5,6]. The studies integrating strategic and operational perspectives mostly focus on macro-economic planning of the energy system and its operational details in the electricity system [4, 5]. The gas network is perceived as one of the essential enablers in the energy transition as it transports gas to gas power plants (GPPs) to produce electricity, and hydrogen or synthetic methane that is produced by power-to-gas (P2G) technology using surplus electricity [7]. Besides, electric-driven gas transport network components have allowed the exploitation of the temporal flexibility of the gas transport network in recent years [8]. Therefore, the question of whether it is feasible to transport the demanded gas within the gas transport system has increased significantly in the energy transition planning for mid-to-long-term decisions. The gas system is integrated into the energy system in recent studies [9, 10] to analyze P2G with a linear gas network model that does not take into account the physical nature of gas in the networks. The detailed operational analysis of integrated electricity-gas networks to analyze the P2G has been studied mostly in an academic setting either with limited or no application to practical cases [7]. In a recent study, [11], the integrated electricity and gas grid is modeled using a single non-linear mathematical model that also accounts for the physical flow of gas in the network. In this study, the European Energy system is analyzed for 2040 and a detailed analysis of the German energy system is provided. On the other hand, the macroeconomic modeling with sector coupling for investment decisions is not included in the analysis. The included gas network model is also not detailed enough to model the operational modes of the active components such as compressor stations. Besides, the detail of the technical analysis of electricity and gas grids is reduced by the integrated modeling of the grids using a single model, since the practical process is sequential in real life as the electricity grid dispatch takes place after market evaluation and gas network adjustments are done after electricity grid dispatch. In [12], the results of a long-term pan-European investment model are tested with integrated electricity and gas network models. However, market dynamics and extensive interactions from sector coupling are not included in the analysis. Instead, they test investment results for the UK using the physical integrated grid models. Last but not least, although there are studies on scenario analysis at a pan-European scale [13], the scope of the integrated energy system analysis is mostly limited to a single country [14], i.e., Germany [2,3,4,5,6, 9, 10].

To understand the state-of-the-art regarding the disruptive changes that we mentioned above, some knowledge of the operating principle of gas networks is required. The gas flows in pipelines as a result of the pressure difference between the two ends of a pipeline, and this pressure difference depends on the properties of the gas flowing in the pipe as well as existence of an active component that regulates the pressure in the vicinity of the pipeline. These active components such as compressor stations and control valves have operational modes and different configurations that affect the connectedness of the network and direction of gas in pipelines [15,16,17]. From the studies in the literature that address the integration of gas networks in the European energy system, we observe that only a small number of the existing gas network models include the pressure drop such as [11] and none of them include detailed models to analyze the operational modes of the active components. Instead, they make assumptions, which may be no longer valid after the disruptive changes that address cases that may have never happened before. Hence, expert opinion on the validity of the assumptions that substitute the detailed operational gas network models based on past experience is not sufficient.

To the best of our knowledge, no published study exists addressing the integration of strategic and operational perspectives to plan the future European energy system to achieve the \({\text {CO}}_2\) emission reduction targets in the existence of disruptive changes with the techno-economic detail provided in our analysis.

In this paper, we focus on the planning and analysis methods to be employed by a holistic approach when dealing with Europe as a pan-European coupled system of national energy systems with individual characteristics, but all strongly connected. We address integrating the strategic and operational planning perspectives by interconnecting models with different spatio-temporal resolution and technical detail levels. Therefore, we implement various optimization and simulation models in an integrated way as in the plan4res [18, 19] modeling framework [20, 21] constituting a novel workflow. First, we use perfect foresight and optimization algorithms for linear problems investigating a cost-effective pathway to establish this new energy world. Thereby we find the optimal energy mix and a simplified (fleet) dispatch schedule for each year along the pathway. Then, we use mixed integer linear problems and a detailed bottom-up modeling approach to determine optimal energy system operation and electricity grid congestion management for specific focus years. Finally, we challenge the future energy mix results, especially P2G topics, using a physical flow-based gas grid model that also accounts for the operational modes and configurations of the active components in the gas network. Hence, we perform a detailed analysis of the resulting gas supply and demand using mixed-integer nonlinear programming models.

We report the results of these models within the proposed workflow for the pan-European scale through a case study to demonstrate the capability of the proposed workflow. The focus of the case study is on the incorporation of different energy sectors to reduce the overall \({\text {CO}}_2\) emissions of the energy system and increase flexibility of the energy system. The case study aims to optimize the investment trajectory necessary to establish a future European energy concept for achieving the COP21 goals while ensuring that the resulting future energy mix is operationally feasible regarding the electricity and gas grids and the limitations of their interconnection. Hence, the results of this particular case study enable us to demonstrate the capabilities of the proposed workflow. Besides, the problems encountered in implementing the models to the case study address the limitations of the workflow, which needs further elaboration.

With the implementation in the case study, we successfully demonstrate the interaction and data transfer between the models for a pan-European scale scenario. Consequently, we successfully project the multi-modal multi-regional energy mix at a pan-European scale with the investment trajectory along the pathway from 2020 to 2050, providing a detailed view of the overall energy generation and demand, including electricity, gas and fuel, thermal heat, and mobility. Nevertheless, we encountered challenges primarily due to limitations in the availability of open data sufficient for the complex models at the operational level, and the uncertainty of regulations that are to be applied to the use of emerging technologies. For the former, we provide tools and discuss evaluating the third-layer analysis results by distinguishing between the data errors and scenario infeasibility. For the latter, we abstract from the present European energy system by additional constraints or operational modes on the models to demonstrate the analysis of interactions.

Overall, the proposed methodology offers decision-makers a powerful tool to analyze unconventional scenarios, employing an integrated tactical and operational analysis approach. In situations where validating results using expert knowledge or historical data is problematic, this methodology becomes especially valuable for assessing the reliability of high-level models. Besides, by incorporating this methodology into their decision-making processes, stakeholders gain the capability to comprehensively evaluate long-term decisions, including investment policies concerning emerging technologies. It is crucial to consider a long-term planning horizon, i.e., 20–30 years, in such studies, allowing ample time for emerging technologies to mature and seamlessly integrate into existing systems. During this critical period, decision-makers require tools to assess the requirements and consequences of alternative solutions. Uncertainty and data scarcity arising from the high resolution of lower-level models and the long-term planning horizon pose challenges as we report through the case study. Thus, in this study, we also address these obstacles head-on and provide effective remedies to ensure well-informed decision-making while using the proposed methodology.

We present the proposed workflow in Sect. 2 and the employed models in the next sections. Then, in Sect. 6, we demonstrate the capabilities of the proposed workflow through a case study whose results highlight the identified explicit levers and challenges for the future energy system. Finally, we discuss the implications in Sect. 7.

2 Proposed work flow for the analysis of multi-modal pan-European energy concepts

Within plan4res, an end-to-end planning and operation tool for the future energy system of Europe has been developed [20,21,22]. It consists of a set of optimization models for integrated modeling of the pan-European energy system. The scope of these models is defined based on the following three layers [20, 21]:

  • A strategic investment layer that will compute investment scenarios for the energy system,

  • A scenario evaluation layer that evaluates investment decisions for the electricity system by including operational planning and market consolidation

  • An analysis layer that provides a further examination of the results provided by the layers described before on the electricity and gas grid. In this layer, technical details for the energy models are added, i.e., a physical flow-based gas network model is used.

The workflow is illustrated in Fig. 1. With the workflow, a multi-year future pathway, i.e., 2020–2050, is analyzed as a first step. Then, the detailed analysis of the second and third layers is performed for individual years such that by selecting a focus year, i.e., 2040.

Fig. 1
figure 1

The proposed work flow

Full size image

The workflow comprises the following four consecutive models and their interactions:

  • The Multi-modal Investment and Operation Model (MIM) constitutes the strategic investment layer of the modeling framework for the case study. It provides several scenarios optimized from an integrated view of the pan-European energy system and an aggregated perspective along specific pathways. The feasibility of one of the pathway steps is tested with the below-listed models.

    • Aim: To investigate the optimized multi-modal energy mix along the pathway to a future year considering the sector coupling to achieve the \({\text {CO}}_2\) emission reduction targets and the energy security of supply.

    • General Method: linear optimization with perfect foresight assumption.

    • Output: energy mix, investments, and retirements, and the simplified dispatch of generation and demand of all considered technologies with a coarse spatial resolution along a pathway involving multiple years.

  • The Disaggregated Multi-Energy Model/European Unit Commitment model (DMEM/EUC), which maps to the scenario valuation layer of the modeling framework, enables a market simulation under the given scenario framework. The focus is on decentralized dispatch of energy generation and supply considering different sectors by detailed bottom-up modeling. Given the electricity generation and demand schedules, the feasibility of electric transport in the projected grid is tested. DMEM/EUC focuses on a single year from the pathway because of the complexity induced by the detailed operational model involving high spatial and temporal resolution.

    • Aim: to analyze the feasibility of MIM results with a higher temporal and spatial resolution considering the constraints from the European energy market.

    • Input: MIM’s future energy mix and investment decision results.

    • General Method: a mixed integer linear model is solved with Lagrangian relaxation.

    • Output: the optimal dispatch of all electricity generation and storage units, and load profiles on an hourly basis.

  • The Disaggregated Multi-Energy Model/Transmission Grid Operation model (DMEM/TGO) is carried out to include a power flow analysis and a congestion management simulation. It is part of the analysis layer.

    • Aim: perform an ex-post analysis to get a detailed view on the overload situation in the European transmission grid resulting from the operation schedules of the DMEM/EUC

    • Input: the operation schedules from DMEM/EUC

    • General Method: simulation of power flows and optimization of mixed-integer linear congestion management problem

    • Output: the new schedules of the generation facilities and electricity loads ensuring secure grid operation and balancing the occurred network losses

  • A Gas Network Optimization model (GNO) analyzes the effect of gas consumption and injection of synthetic gases (e.g., from P2G) on the gas grid within the scenario with electricity generation and demand schedules from the DMEM/EUC. GNO and DMEM/TGO together constitute the analysis layer of the modeling framework.

    • Aim: to analyze the feasibility of the MIM and DMEM results from the gas infrastructure point of view

    • Input: the high-spatial-resolution hourly gas demand and supply schedules from DMEM

    • General Method: Linear optimization and mixed integer nonlinear optimization models

    • Output: Cross-border gas exchange at the European level, the feasibility of gas transport in a restricted region on the peak gas days, and feedback on results in case of infeasibility, i.e., the feasibility of location of a newly invested P2G facility

The workflow and the models’ interactions are designed to provide adequate feedback for the previous step(s) to refine/evaluate their results when an infeasibility is detected. This design facilitates investigating cost-optimal strategic energy transition decisions that consider the impact of sector coupling and practical applicability on the operational level.

3 Analysis of the pathway to carbon neutrality via multimodal investment and operation modeling

In the first layer of the proposed workflow, we analyze the alternative pathways to carbon neutrality at the pan-European scale by the MIM [4, 5]. MIM optimizes the future energy mix and the national investment plans along a pathway, considering hourly operational schedules, costs and revenue streams, and security of supply.

MIM models the linear energy conversion processes with a simplified approach by transforming the content of one energy commodity type into another type. The energy-converting processes are grouped into subprocesses that can be associated with technical parameters like efficiency, availability, etc., and also financial parameters like capital expenditures (CAPEX) and operational expenditures (OPEX) in a meaningful way.

The skillful combination of several of these linear subprocesses allows to model more complicated technologies or processes with multiple input and/or output energies. These interconnections are modeled as an interconnected energy network called a technology graph and used as an input to the MIM. This framework allows finding cost-optimal technology combinations, which can be interpreted as a time-resolved competition of technologies.

Using MIM, we create a meshed implementation of the integrated pan-European energy system in the form of a linear problem that models a coupled representation of the energy-relevant sectors, including electricity, heating and cooling, mobility and gas/fuel, and the competing technologies along with their interactions and feedback loops. It also considers technological trends (costs, efficiencies, etc., in addition to existing public policies), demand levels, and installed capacities. We solve this linear problem with perfect foresight to optimize the total expenditure (TOTEX). The decisions for investments and operation for all time steps, all planning steps, and all regions are determined in one single optimization run to achieve optimal intertemporal allocation. Several planning steps are considered (e.g., 10 years intervals) in parallel to optimize the system development until the target year of the planning horizon (e.g., 2050). In addition to the information about energy respectively technology mixes and investment decisions along the pathway, for every base year, we get insights into the operational schedules of generation units and consumers, the hourly price profiles for all energy vectors modeled, as well as the corresponding hourly carbon footprints. Likewise, we get the regional distribution of these data. The modeling approach is illustrated in Fig. 2. For further details of the model, the reader is referred to [3, 5] and [22] for a functional description of the MIM model with the relations to the DMEM model.

Fig. 2
figure 2

Overview of the multi-modal investment and operation model (MIM)

Full size image

In the case study, over 100 technology classes are implemented in the MIM model building a meshed representation of the pan-European energy system including 33 countries [23]. Setting up a solvable linear problem representing the pan-European Energy system requires consideration of multiple input data, assumptions, and constraints, as summarized below.

3.1 Simplifying the complexity of conversion processes

First of all, we use a simplified approach to describe individual technologies such that we clustered all units within a region into technology clusters representing fleets of similar technologies. We use per-region aggregated capacity and generation to average-out individual differences in operational schedules. Besides, we model the conversion processes in a simplified manner, i.e., input commodities are converted to output commodities considering technical and financial parameters.

  • Technology graph: We use energy cascades representing the possible interactions of the different conversion processes and technologies, which are already installed or have the potential to be installed, each connected by their input and output commodities. This constitutes a mesh constructed from simple one-step (sub-)conversion processes describing the conversion of an input commodity into another output commodity resp. energy vector. In this mesh, each technology is represented. From the possible manifold interactions between technologies and energy vectors, we retrieve a complex mesh called “technology graph” representing the whole complexity of the simulated energy system. Cascades of the energy flow highlighting some sub-clusters out of this technology graph are presented in [23].

  • Technical parameters: The technological parameters that we use in the conversion processes are, e.g., fleet availability, the maximum contribution to secure supply, and efficiency as well as fuel efficiency for conversion processes with multiple outputs, e.g., co-generating power plants; boundary values for installations, availability of installed facilities, maximum or given annual energy consumption and generation [23].

  • Financial parameters: We take into account the technology-dependent technical lifetime, specific CAPEX and operations and maintenance costs, WAAC (7%), depreciation (1%), financial lifetime (10 years), minimum \({\text {CO}}_2\) penalties as a projected trend over the pathway [23].

3.2 Reduction of the size of the linear problem

The dimensions of the resulting linear problem combined with usually limited computational power do not allow pathway optimization by calculating a whole year with 8760 h per interval. Thus, instead, the optimization is performed based on five selected representative weeks per interval year. 2015 is selected for the case study for historical demand and generation profiles, with the selected weeks considering total and regional peak demands, seasonal changes, weather patterns, and also “average” demands for every country individually.

3.3 Constraints

To set up a meshed energy system and derive a linear problem from different sectors’ interactions, we consider some sector-specific boundary conditions, either from technical constraints or national policies. These constraints are implemented in the MIM using very large data sets. This data constitutes regional distributions and time profiles, and time regional profiles which can be found in [24, 25].

  • Sector-based constraints These constraints include sector-based limitations or technical constraints as well as the impact of national policies for operating or retiring the existing facilities or installing new facilities. For the implementation details, the reader is referred to [23].

  • Constraints from national politics for nuclear power plants and national phase-out policies for the coal power plants are taken into account. For the implementation details, the reader is referred to [23].

3.4 Assumptions

Our assumptions for modeling the pan-European energy system are summarized below. For the implementation details, the reader is referred to [23].

  • Electric transport grid assumptions The focus of the MIM model is on the capacity expansion of the existing generation and consumer fleet. The expansion of the interconnecting electricity grid is only the secondary focus of analysis. Nevertheless, some “freedom” for grid extension optimization must also be implemented. Therefore, a representation of the interconnections as bidirectional NTCs is considered. For the case study, we base the representation on the European Association for the Cooperation of Transmission System Operators (ENTSO-E) data; and we divide the electricity grid into a fixed and an optional “additions” part (see [23] for details).

  • Security of supply considerations Considering the volatility and limited contribution to the security of supply requirements of some renewable energy sources, we simulate an additional “darkest hour” with the annual dispatch. This “darkest hour” approach should ensure that there is always enough generation capacity installed so that the security of supply can be guaranteed. In this “darkest hour” some technologies cannot fully contribute—like (distributed) wind power—or not at all—like PV. Besides, some demands do not need to be met to the full extent, such as eCar charging, or others may be assumed to be even higher such as local peak demand. The resulting demand must be covered by the remaining technologies that can contribute to the security of supply in this “darkest hour”. For the details of the assumed parameters, the readers are referred to [23].

  • Pseudo-transport grid for cross-border exchange of hydrogen “Green” hydrogen, which is produced from RES by electrolysis, may be a suitable new fuel type and a suitable solution to store (surplus) energy even from a long-term or seasonal perspective. Thus, it is essential to include \({\text {H}}_2\) in a transport system between the region of generation to the region of usage. However, such a transport grid infrastructure, for example, planned as the European hydrogen backbone [26], has not been realized. Therefore, in this study, we implement an optional pseudo grid to simulate hydrogen cross-border exchange, analogously to the NTC approach for the electricity grid as presented in [23].

4 Detailed market and grid analysis using the disaggregated multi-energy model (DMEM)

In the second and third layers of the workflow, we want to investigate, how a multi-energy scenario can be implemented in detail considering market and electricity grid operation with a high resolution using the DMEM. DMEM addresses the operational optimization of the results of MIM with higher detail, i.e., higher spatio-temporal resolution, using single years. MIM results include the installed capacities for conventional and renewable power plants on a country level and the heat supply structure for CWE countries. DMEM model provides operation schedules for power plants, storage units, and distributed generation units (employing energy cells), considering technical constraints like ramping and minimum up/downtimes. The resulting schedules depend not only on the electric demand but also ensure that the heat demand is met. Furthermore, power flow on the electricity transmission grid and the amount of dispatch are analyzed.

The input data for DMEM includes the MIM results and the installed generation capacities for the year under investigation. Since the MIM data has a coarser spatial resolution, we need to disaggregate the energy mix scenarios from MIM. Therefore, we disaggregate the MIM results of installed capacities for conventional and renewable power plants on the country level to wind power plants and PV units at suitable locations and generate location-specific hourly generation profiles using existing tools and commercial databases [27,28,29]. Then, we use a bottom-up modeling approach (see Sect. 4.1) to compute the high-resolution distributed demand structures of CWE countries. We also include the power plant models, exchange capacities for the electricity market, and Europe’s detailed electricity grid model in our analysis.

As a first step, we evaluate the investment decisions from the MIM by modeling the operation of the existing assets in the energy system using the EUC model.

Then, we use the results of the EUC for ex-post analysis of the transmission grid in terms of electric power flow and redispatch calculations of DMEM/TGO. Thereby, we subsequently disaggregate the optimization results of the EUC on a nodal resolution to generate electric load/generation time series. Thus, the impact of the expansion and operation on transmission grid utilization and the amount of redispatch is analyzed.

4.1 Bottom-up modeling of high-resolution distributed demand structures

To model distributed demand structures, a bottom-up modeling approach employing statistical and socio-economic data is applied.

Based on the socio-economic data, buildings and businesses are allocated to the spatial levels. These registers comprise residential buildings (either single-family homes, multi-family homes, or apartment blocks) and businesses classified as commerce, trade, and services, or industry. The businesses are further classified by their sizes (dependent on the number of employees) and industry/commerce sectors. The individual building and business characteristics, as well as weather data, are further used to determine the individual heating and electricity demands and profiles.

For every single entry (e.g. a specific household or a specific business location), the registers include a predefined electric and thermal demand (warm water, space heating, process heating) with an hourly temporal resolution. To meet these heat demands, heat generation and storage technologies are assigned to every single building/business based on the predefined energy-mix scenario generated by MIM. The heat generation units might also have an interaction with the electricity sector (e.g. heat pumps as additional electricity loads or combined heat and power (CHP) units as electricity generators), thus providing flexibility to the energy system through sector coupling.

The process of building these registers results in “energy cells”, which define a regionally connected part of each country in CWE. Each energy cell consists of aggregated heat demands and the associated distributed generators, storage units, and power plants, providing their aggregated flexibilities as a sub-model to the European unit commitment model. The flexibilities result from temporal shift potentials, e.g. in the heat supply through heat storages or in the charging of electric vehicles.

The information flow of the bottom-up method is illustrated in Fig. 3. We refer to [3] for a more detailed discussion.

Fig. 3
figure 3

Process of bottom-up modeling used in DMEM

Full size image

4.2 Market simulation

We use EUC to evaluate the decisions proposed by MIM at an operational level with a high spatial and temporal resolution.

The EUC simulates the European electricity market while incorporating further sectors such as heating and mobility. It optimizes the dispatch of all generation and storage units concerning the cross-market electricity exchange.

We apply a decomposition approach based on a Lagrangian relaxation to reduce the complexity of the resulting large optimization problem [30]. In this approach, the load coverage constraint is relaxed. For each market area, a Lagrange coordinator is used, which can be interpreted as the electricity price. The coordinators are adjusted in the same process until a convergent market state is reached. This enables the consideration of the cross-market electricity exchange through market coupling. Due to the simultaneous dispatch and market coupling, electricity prices can be derived [31]).

Fig. 4
figure 4

Integration of decentral flexibility into the market simulation

Full size image

For the market-driven simulation of distributed technologies, the aggregated flexible technologies (electric vehicles, CHPs, P2H, etc...) that are allocated to the buildings and industries are modeled as local virtual power plants (LVPPs) as illustrated in Fig. 4 [32]. The LVPPs are submodels of the market simulation, analogously to power plants, renewables and (hydro-) storages. We have observed that integrating LVPPs might lead to convergence problems due to the concurrency of the behavior of the corresponding technology submodels. Thus, we extended the market simulation as described in [33] by implementing methods to improve the model’s convergence.

4.3 Grid analysis

Finally, in the third layer of the proposed workflow, the impact of the market schedules, determined by the upstream models, on the European transmission network is evaluated via the TGO model. A detailed description of the model can be found in [4, 22, 34]. This model consists of two steps: At first, the TGO model determines power flows in the grid and can identify bottlenecks via \(N-1\) analysis. In the second step, a congestion management model is carried out to identify the cost-minimal interventions for alleviating the bottlenecks. The power flows are calculated using an AC formulation of the power flow equations. The simulation includes an integrated control of high-voltage direct current systems and phase shifting transformers as well as generator control for covering power imbalances/losses. Based on the power flows congestions are identified. Within the congestion management simulation, there are different options to clear those congestions like redispatching of power plants or pump storage units as well as the curtailment of RES generation. Other options are employing the flexibilities resulting from the coupling of the electricity sector with other energy sectors, e.g. using P2G units to shift the energy transport from the electricity grid to the gas grid. The required congestion management measures are determined using a mixed integer linear unit commitment problem under linearized network constraints ensuring \(N-1\) security. The objective of the simulation is the minimization of the overall operation costs considering all the technologies mentioned above.

5 Detailed gas grid analysis with an integration to electricity grid

Finally, we analyze the results of the MIM and DMEM models in terms of gas network capacities to answer the questions whether

  • the imported gas is enough for meeting the European countries’ gas demand;

  • European countries have enough capacity to transport gas via the existing network;

  • \({\text {H}}_2\) can be injected into the pipelines in the projected amount;

  • the location of the projected P2G facilities/\({\text {H}}_2\) dispensers are feasible.

We also study methods to further analyze infeasibility results for identifying bottlenecks in the gas transport network and give proper feedback to DMEM and MIM where applicable.

The gas flows into pipelines according to the thermodynamic rules. It is considerably slower than the flow of electricity. Besides, there are complex devices inducing complexity to the gas networks, such as compressor stations that regulate the gas flow. Thus, to model an interconnected operation of electricity and gas grid systems consistently at the operational level, we should not only account for the physical differences between the gas and electricity by using a physical flow-based gas network optimization model but also spatio-temporally integrate the gas network model to the electricity grid model to answer the above-stated questions consistently. Such integration enables us to improve/fix the electricity grid results using the feedback from the gas grid. However, as reflected by the very limited scales of the state-of-the-art academic studies [7], it is very ambitious to couple electricity and gas grids on the pan-European scale. Pan-European gas transport network data is scarcely available in public data sources, and the available data is insufficient for use with a physical-flow-based gas network optimization model ([35]). Therefore, we tailor the integrated analysis methodology by integrating the electricity and gas network models at the data exchange level and focusing on a restricted region in central Europe instead of the full European scale during the detailed operational analysis of the gas network. Note that the amount of gas entering and exiting from a region highly depends on how supplied gas is transported through Europe. Thus, this tailored method accounts for the pan-European gas supply/demand and gas transport capacity constraints induced by the pan-European gas transport network.

To implement the tailored methodology, we first select a restricted region where mandatory data for a physical-flow-based stationary gas network optimization model (see [35]) can be constructed consistently. Then, we use the available pan-European supply and demand data together with the market simulation results from DMEM (see Sect. 6.2) and high-level pan-European gas network topology to generate valid gas in-/out-flow scenarios, i.e., amount of gas entering to/flowing out of the network, for the restricted region. We evaluate the feasibility of the gas in-/out-flow scenarios with a stationary GNO model ([36] by solving a validation of nominations (NoVa) problem [15, 37], which is modeled as an MINLP. Finally, we analyze the infeasible gas in-/out-flow scenarios to isolate the reason for infeasibility and provide feedback to the MIM and DMEM models once we can diagnose the root cause as the input data from these models.

5.1 Network topology data construction/improvement

The results of the gas grid analysis are highly affected by the quality of the gas grid data. However, gas transport network topology data is not readily available from public resources in the sufficient detail required by the GNO, even for Germany, which we use as the restricted region for the detailed analysis. Although there have been readily available data for network topology for Germany, constructing consistent high-pressure gas transport network data for a physical gas network optimization model is an intricate and time-demanding task. We spent a considerable effort modeling the German gas network’s graph representation in sufficient detail to run our analysis with the GNO from the readily available network topology data of the LKD-EU project [38,39,40].

In addition to the improvements presented in [35], we improved the publicly available Germany high-pressure gas transport network topology published in [38, 39] by the following:

  • Association of the European Network of Transmission System Operators for Gas (ENTSOG) high-level topology with Germany network topology: All high-level topology nodes (cross border points and storage facilities) in the ENTSOG database are associated with at least one physical entry/exit node for Germany. A semi-manual method is used for this association, as also described in ([35])

  • Entry and exit points redefined using Transmission System Operator (TSO) data

  • Added height data to nodes: We used a geographical information system software to match the shape files provided with the LKD-EU data set to a digitized European map with height data.

  • Associating known node pressures for pipeline capacity computation

  • Perform consistency check for pipeline properties: We use simple heuristics to check whether the data is consistent and correct inconsistencies.

  • Pipeline decoupling, market area points, control valves

  • Adding major pipelines built after the available data set was published including potential and planned expansions that are taken into account for making the supply and demand forecasts, e.g., in ENTSOs Ten Year Network Development Plan (TYNDP) [41].

Besides, we augmented the data in [38, 39] with a data set of active components such as compressor stations, valves, and control valves. We have developed and implemented a methodology to estimate compressor and driver data based on partially available public data and network topology. Using this methodology, we modeled 58 compressor stations around Germany as presented in Fig. 5 and incorporated these models in our gas network optimization.

Fig. 5
figure 5

German high-pressure gas transport network data

Full size image

We kept improving the data with the analysis results of the GNO model by adopting an iterative process (see Sect. 5.4).

5.2 Scenario generation

The second input of the GNO is a gas in-/out-flow scenario for the gas network. In this analysis, we have high-pressure gas transport network data of a restricted region in the European continent to be analyzed with the GNO.

A gas in-/out-flow scenario consists of amounts of gas flowing into this network via its entry nodes and out from it via its exit nodes. These amounts are dependent on the gas demand of the region as well as how this demand is met, i.e., whether the gas is indigenously supplied from production and P2G facilities, taken from storage facilities in the region, or supplied from another one over the cross-border connection points via gas pipelines or Liquefied Natural Gas (LNG) terminals. Hence, a generated gas in-/out-flow scenario can only be practically relevant for the restricted region if the amounts of gas at the entry and exit nodes of the gas network in this scenario can be obtained at the pan-European scale, too.

To generate such a scenario, we consider constraints from the pan-European high-pressure gas network capacities, the amount of gas entering Europe from import via pipelines and LNG terminals, and the amount of gas demanded by other countries, e.g., for household and industrial usage, storing in underground storage facilities or caverns, or exporting to other countries.

In this study, we analyze the future state of the pan-European gas network. Thus, our analysis uses the gas demand and supply forecast, such as the data provided in the TYNDP [41] in our analysis. However, the forecast data is given for each country yearly. Hence, we propose a 3-step methodology [35] to spatio-temporally disaggregate the available pan-European gas demand/supply forecast data and merge it with the results of DMEM. In this methodology, we incorporate the constraints from the transport topology and capacities of the high-pressure European transport network, which are published ENTSOG [42, 43], to generate balanced gas in-/out-flow scenarios for Germany.

As a first step, we use the historical physical flow data as meta-distribution to temporally disaggregate the yearly cumulative forecasts from ENTSOG to daily time series per country. In this step, we also merge the hourly gas supply/demand schedules of CWE countries from DMEM with forecast data to finalize the daily time series for European countries. We use the method which is presented in [24] and [23] in detail.

In the second step, we further disaggregate the supply and demand forecast data given per country geospatially. We define the high-level pan-European network topology as a directed graph based on ENTSOG definitions provided in the modeling, and the data warehouse context [44, 45]. The topology and capacities are illustrated in Fig. 6.

Fig. 6
figure 6

Total capacity of pipelines between European countries in GWh/d [43]

Full size image

ENTSOG partitions the whole pan-European network into clusters that include significant portions of the network such as TSO gas networks, storage facilities, international connections like OPAL and North Stream, suppliers, and bottlenecks and provides an entry-exit topology of the pan-European gas transport network using these partitions. This high-level network topology shows cumulative capacities between these partitions instead of providing the network topology in terms of physical pipelines and components such as compressor stations and valves. The former gives information about the distribution of gas in Europe in terms of cross-border gas exchange, gas exchange between market areas, and gas withdrawn from/injected into storage facilities. However, the latter is required for an operational analysis of a gas grid. Hence in the second step of our scenario generation methodology, we use a capacitated network flow model based on ENTSOG’s high-level network topology [42, 43] and investigate feasible ways of geographical disaggregation of pan-European gas supply to meet country based gas demand.

We should note that the second step also models the gas exchange with the underground storage facilities and caverns with relevant rates for injection and withdrawal. The gas injection and gas withdrawal rates that depend on the volume of existing gas in the storage are modeled as capacity constraints in the model (for details, please refer to [23]). The result of this model is the amount of gas exchanged within the significant partitions of the gas network. However, these amounts are accumulated, and they should also be dispatched to physical nodes of the network in the region of interest, i.e., the German network.

We use another capacitated network flow model to dispatch the cumulative amounts of gas in-/out-flow to the physical nodes of the gas network in the selected region, i.e., Germany, in the third step. We also ensure that the electricity grid model and the GNO are spatially aligned at the data exchange level. We associate the physical entry and exit nodes of the gas network with the significant partitions of the ENTSOG high-level network topology. We model the German gas network topology as a directed graph and compute the maximum allowable capacities of the pipelines given the maximum pressure allowable in the pipeline, gas properties, pipeline diameter, and pipeline length. The \({\text {H}}_2\) input from the DMEM results in case P2G facilities exist, the gas in-feed from these facilities is associated with appropriate physical entry nodes of the gas grid such that it cannot exceed a predefined amount, i.e., 10% of the total natural gas inflow of that particular entry node. Thus, we assume that \({\text {H}}_2\) is mixed with natural gas in pipes up to an allowable level such that the effect of the mixture on the physical properties of the gas can be neglected. We aim to generate a feasible solution to the capacitated network flow model. Therefore, we use a slack formulation resulting in a balanced gas in-/out-flow scenario for GNO with step 2 results. The objective function value provides us feedback on the feasibility of the DMEM results.

The implementation details for both of the network models are provided in [23].

5.3 Infeasibility analysis

The NoVa problem is solved to evaluate the feasibility of the gas demand and gas supply [15]. The NoVa problem is a feasibility-seeking problem: when feasible, it results in a solution consisting of the set of operational modes for the active components of the network to enable routing the amount of gas entering the network to the exit nodes, and also the state of the network in terms of the amount of gas in the pipelines and the gas pressure at nodes. We investigate a feasible solution to the generated gas in-/out-flow scenarios by solving the mixed integer nonlinear programming (MINLP) model presented in [36]. Whenever the models detect an infeasibility, we further analyze the nomination by employing the GNO model slack formulations or detecting the irreducible infeasible subsystem of the gas network [46]. We use the C++ software framework Lamatto++ to conduct our computations and infeasibility analysis [47]. Please refer to [23] for the further implementation details.

5.4 Iterative framework

We use the steps in Sects. 5.1–5.3 in an iterative way as illustrated in Fig. 7 during our analysis for the case study. This iterative framework enables us to comment on the root cause of the infeasible solutions detected by NoVa such that whether they are because of an error in the data topology data set that we use or because of the gas in-flow/out-flow scenarios that are induced by the upper stream models, i.e., MIM and DMEM. For example, a zero objective function value of the capacitated network flow model that is used in the third step of scenario generation in Sect. 5.2 indicates that a gas in-/out-flow scenario can be generated given the DMEM gas schedules, while a positive objective function indicates, that either

  • some amount of gas from the high-level pan-European gas entry-exit network model results cannot be routed in the network, or,

  • all of the gas from the high-level pan-European gas entry-exit network model can be routed in the network, but there should be an exchange between the locations of gas leaving/entering the network.

Fig. 7
figure 7

Gas network analysis-iterative framework

Full size image

The former gives clues about the infeasibility of meeting the demanded amount of gas given in the gas schedules of DMEM. While the latter indicates that the DMEM gas demand/supply schedules are not realistic, and gas should be redispatched to other locations(s), e.g., electricity production can be scheduled using another GPP, or the \({\text {H}}_2\) injection should be made from another entry. The deviation is evaluated for each scenario individually, though infeasible results that occur systematically may also be a warning for an error in the physical network topology data set. On the other hand, the non-zero objective function results of the slack formulation and the minimum irreducible infeasible subsystem solution for infeasible scenarios help diagnose the problems in the data if they point out a certain part of the network with various scenarios consistently.

When we detect an infeasibility related to a certain region of the network topology systematically, we investigate the probable error sources related to this region such as data modeling assumptions, subnetworks induced by active components in the region, and expansion plans or changes in the future state of the grid. Using these iterations, we have diagnosed some data errors in the physical network data set and corrected them, for example by:

  • Adding missing pipelines such that Stolberg-Porz Line, Drohne-Nowal Line, NEL-NETRA link

  • Decoupling pipelines by splitting nodes, or adding control valves especially if the joining node is a market area point,

  • Updating the assumptions at the low-calorific gas region in Germany

6 Multi-scale multi-modal energy system transition analysis with the proposed workflow

6.1 Case study setting

We implement the proposed methodology for a case study in the project plan4res. The aim of the case study is to investigate which pathway we can/should take best from 2020 to 2050 to meet the European \({\text {CO}}_2\) emission targets. Thereby, we focus on the following questions:

  • What will the optimal future energy mix that enables achieving the COP21 targets look like?

  • How can we reach these goals with a cost-effective investment pathway?

  • What impact has sector coupling had on the future generation fleet, e.g., the potential role of emerging technologies like power-to-heat (P2H), eMobility, and P2G?

  • Will the optimal future energy mix be practically applicable considering the pan-European electricity and gas transport infrastructure, and limitations induced by their interconnected operation?

To answer these questions, we design the case study to capture the investment trajectory for a set of countries, the impact of international energy exchange, and the impact of sector coupling on the energy mix including electrification of mobility, heating, cooling; flexibility provided by P2H, heat storage, e-mobility, synthetic fuels; coupling of electricity and gas sector by P2G.

The base scenario (COP21 Targets) of the case study forces a 90% reduction in \({\text {CO}}_2\) emission in 2050. Besides, to analyze the impact of variations to this base scenario we made adequate changes in the modeling assumptions or constraints in the analysis steps. The scenarios are listed in Table 1 and for the details of other scenarios, the reader is referred to the case study report [23].

To analyze the strategic decisions made by MIM at an operational level, we focus on a single year in the rest of the case study to cope with the complexity induced by the high spatial and temporal resolution and technical details inherited by the models at this level. In the case study, we have selected 2040 as the focus year as in 2040 transition between the gas grid and a separate \({\text {H}}_2\) grid will still be ongoing. A separate hydrogen grid is expected to be operational after then.

Based on the outcome of the MIM model, the feasibility of the COP21 scenario is examined in more detail by DMEM/EUC for 2040. During the analysis, a distinction is made between different operating modes on the market simulation side. Among those, heat-driven market simulation assuming decentralized supply-based consumption is used as a base scenario for the operational analysis by the third layer of the workflow. The other cases show the capabilities of the workflow to evaluate the future energy mix resulting from MIM through what-if scenarios. The details are presented in “DMEM scenarios” of appendix.

The results of the market simulation are used for an electricity grid simulation with DMEM/TGO. This determines the grid load and, based on this, the necessary amount of congestion management.

Based on the market schedules and the findings of the electricity grid simulation, a detailed gas network analysis is performed. To assess a wide range while keeping the computational effort within reasonable limits, we focus on the peak days which require the minimum and the maximum amount of gas transport through the network. In our analysis, we also include a case where an artificial P2G facility at the onshore wind site with maximum redispatch volume is added in eastern Germany. The list of the scenarios is presented in Table 2.

As we consider a European scope for a multi-year time span during the case study, we require huge data sets with various data types and requirements to model different sectors such as electricity, transport, heating, and gas. Besides, since the models in the workflow deal with various aspects of the energy system, the required data for these models differ with respect to spatial, temporal, and technological detail. Thus, preparing the scenarios for the case study inherits challenges to collecting and consolidating huge data sets from widespread data sources. Formatting and transformation methods are required to use the data sets in the individual models, as well as non-trivial data preprocessing and transformation approaches to enable data exchange between the models. For details on the former, please refer to [24]. The latter is explained in Appendix 2: Overview of case study data.

6.2 Results

In this subsection, we report the key findings from the case study to highlight the analysis capabilities of the proposed workflow. In addition, we address the limitations of the workflow and we discuss the remedies to deal with the limitations when we face them.

6.2.1 The first layer: pathways of the energy mix

First of all, we investigated the optimal future energy mix and the investment trajectories using the scenarios in Table 1. From a manifold of results, we can summarise the key messages from the MIM modeling of the pan-European energy pathways from 2020 to 2050 as (see Table 3 for details of the scenarios):

  • Carbon Neutrality of the pan-European energy system can be reached in time within feasible additional costs. Although, the emergence of the potential pathways depends on the policy and technological constraints, which are assumed for each scenario.

  • Nearly 60% of emission reductions can be done by tech developments.

  • The future energy mix significantly changes with the assumption used for the max potential limit set for small PV installations, i.e., whether a conservative or a progressive limit is used. The conservative limit, results in an energy system manifestation in which nuclear power plants (NPPs), large central generation units, and large central batteries are required beyond 2040 (see scenario “Central World” in Fig. 8). However, for the progressive limit, representing nearly doubled potential for small PV, significantly more PV can assert themselves and predominately as small decentral PV systems accompanied by home batteries (see scenario “Decentral Prosumer” in Fig. 8). Then (almost) no NPPs survive, and installations of central power generation plants and large batteries are reduced to the minimum level. From Fig. 8, we also see that the transition of the energy system along a pathway as described by the DP scenario will be significantly cheaper.

  • In all scenarios RES will steadily increase their installed capacity. From 2040 wind power gets dominant accompanied by PV supporting the decarbonization of electricity.

  • The increasing economic pressure from RES leads to a fast phase-out of oil power plants, followed by coal power plants till 2040. GPPs will be reduced to a fraction of today, but still, some installations will be needed. Beyond 2040 these remaining power plants take in more and more the role of reserve power plants and “peakers” usually running at low FLHs.

  • The situation for NPPs depends on the scenario.

  • The shift to eMobility proved to be a robust trend and gets the preferred choice for roadside transport in almost all scenarios as presented in Fig. 9. The figure also depicts that when fuel cell electric vehicles (FCEVs) are artificially promoted at least to a minimum level of 30% \({\text {H}}_2\), a disruption of the roadside Mobility sector towards more hydrogen usage can be triggered. Then, in 2040+ significantly higher shares of FCEV occur. But this scenario is correlated with significantly increased total pathway costs.

  • P2H, and for space heating especially heat pumps, get the dominant technology class for heating. In 2050 electric heating alone adds about + 50% to today’s electricity demand. This is accompanied by district heating, which may increase from 13% to 20% of supply, mainly by collecting co-generated heat and waste heat from (industry) processes, or supplied by large heat pumps. Electric central process heating is accompanied by heating from biomass sources and solar heat, and from 2040 some large heating devices use hydrogen fuels. Initially, biomass is exhausted by the demands from process heating, leaving almost none for decentral heating.

  • High amounts of installed electric batteries (usually > 600 GW in 2050 across Europe) facilitate the implementation of RES and help to ensure the security of supply. The analysis results are presented in Fig. 10. Additional batteries from eMobility and heat storage units (which are not shown in Fig. 10) via P2H may provide further flexibility, helping to stabilize the energy system. Beyond 2040 P2G provides further storage options, reaching significant contributions in 2050 and with regional distribution supporting the implementation of RES (see Fig. 11). In the context of the development of distribution grids, we acknowledge that the incorporation of this information may have a substantial impact on the required investments. We also recognize that conducting this evaluation requires substantial computational efforts. Yet, the successful implementation of this computationally intense evaluation in [3] demonstrates that this evaluation can be integrated into our proposed workflow.

  • In 2050 at least three sectors, electricity, heating, and mobility, compete for the limited biosources. It is won by process heating and mobility, demanding green synthetic fuels for marine and inland navigation, and aviation. Usage in decentral heating and electricity generation will be reduced instead. In 2050 roughly 20% of biomass must be imported from extra-Europe.

  • P2G does not get significant before 2040 (see Fig. 11). The green hydrogen and synthetic green methane are significantly fed into the gas grid replacing natural gas, which then leads to energy-specific nominal mean shares of  30% hydrogen in the gas mix.

  • Due to the trend of “decarbonization by electrification” total electricity generation increases  180% by 2050, if compared with today’s level.

Fig. 8
figure 8

Generation mix electricity for CN CW and CN DP scenarios

Full size image
Fig. 9
figure 9

Transition of roadside mobility-shares in transport volume [G pkm resp. G tkm]

Full size image
Fig. 10
figure 10

Regional developments of storage facilities: a CN DP-2040, b CN DP-2050, c: CN CW-2050

Full size image
Fig. 11
figure 11

Regional developments for P2G: a: CN DP-2040, b: CN DP-2050

Full size image

6.2.2 Second layer: operational and market implications

To analyze the base scenario results further in terms of the operational context, we focus on the year 2040. We first model detailed heating supply for the countries using the bottom-up modeling approach in Sect. 4.1 using socioeconomic data in high spatial resolution (for further details on the spatial resolution, see Table 4). Hence, we meet the higher spatial resolution requirement of the heating sector due to local demand and supply structures.

The heat supply technologies are allocated according to registered demand based on the spatial data. The resulting heat demand distribution for Germany and France is illustrated in Fig. 12. Our results indicate that local heating demand can be supplied by a mix of decentral technologies and central power plants according to local availability.

Fig. 12
figure 12

Spatial distribution of heating demand and location of district and process heating supply by power plants

Full size image

DMEM/EUC is used in the case study to generate hourly demand and generation schedules for CWE countries by European electricity market simulation. In general, due to the high capacity of renewable energies in the scenario framework, the electricity generation in the case study is mainly driven by renewable feed-in, especially wind onshore and PV in the summer. Conventional power plants are primarily used in the case of heating obligations. For exemplary schedules, please refer to [23]. Using these results, we can evaluate the available flexibility by country, and compare different countries. For example in the case of the German schedules, our results in the case study show that electricity generation is mainly dominated by volatile feed-in of wind onshore and PV. In some hours, there are high surpluses. Thus, flexibility such as flexible P2H or P2G applications, demand-side management, or international electricity exchange is required for compensation. Ultimately, there is also the possibility of market-based curtailment. In contrast, the electricity generation mix in France has less renewable feed-in as a significant proportion comes from NPPs.

The model also enables us to evaluate the impacts of the energy mix on electricity prices. For instance, the high share and volatility of renewable energy are also reflected in the resulting electricity prices for Germany. To elaborate further on electricity prices, we analyze the impacts of market setting. We compare the base scenario with another operational mode modeling integration of decentral P2H and power-to-mobility units in market simulation. We observe that this modification increases the prices by about 20% compared to the base scenario. Besides comparing the resulting schedules of P2H in the two cases, we see that the modifications change the regular course of the supplied heat demand by adding several peaks. This reveals that market integration of small decentral suppliers and consumers, e.g. P2H, increases market efficiency due to price incentives.

Further compensation through international electricity exchange, which, however, is one driver of high grid utilization.

As we analyze the pan-European electricity grid, we observe that there is a high exchange capacity due to expanded inter-connector lines in the transmission system. The high share of renewable energy sources and their local allocation leads to a strong use of these capacities. To evaluate the cross-border exchange potential between the European countries, we use a copper-plate setting in the market simulation as an alternative operational mode to the base scenario. This alternative setting differs from the base scenario such that in the former there are no restrictions on the cross-market electricity exchange. The aggregated import and export flows within the base scenario compared to the copper-plate scenario are presented in Fig. 13. In the former, a high export surplus of Germany can be observed. This surplus is spread throughout all neighboring market areas. Significant imports are made in Poland and Italy so that cross-market flows in these directions can be seen. Whereas, in the latter, the exchange capacity between two neighboring market areas is assumed to be unlimited. Thus, almost all exchange flows increase compared to the former. This result shows that increasing exchange capacities lead to more efficient use of the renewable generation capacity.

Fig. 13
figure 13

Annual aggregated exchange flows (in TWh): a Results of the base scenario. b Results of the copper plate scenario

Full size image

6.2.3 Third layer: congestion management and integration with gas networks-electricity

The high aggregated commercial exchange flows lead to high \(N-1\) line utilization in the power flow simulation. In panel (a) of Fig. 14, the maximum \((N-1)\) line utilization in Germany and France that is the result of the one-year simulation is presented. In Germany, high overloads can be observed in the north–south direction. These are driven by the allocation of renewable capacities within Germany. Moreover, commercial exchange flows from north to south strengthen this effect. Central battery storages lead to additional overloads in particular. Those storages are allocated to nodes with high load. This does not imply a grid-optimized allocation. The operation of power flow control devices such as high voltage direct current and phase shifting transformers is optimized in the following grid operation simulation under consideration of the \(N-1\) case. Within the power flow calculation, a pre-optimization of those devices based on the N-0 case as an initial operation mode is assumed.

The congestion in France can be mainly explained by high exchange flows. Especially overloads in the south are caused by high exports to Spain.

An exemplary critical grid utilization case of the base scenario, where the congestion in Germany is mainly driven by high feed-in from wind onshore, is illustrated in panel (b) of Fig. 14. For this case, the feasibility of replacing a power-to-hydrogen electrolyzer with 2000 MW to solve the congestion is also analyzed by the GNO in the case study. The results are presented in Sect. 6.2.4.

Fig. 14
figure 14

\((N-1)\) line utilization in Germany and France. a: Maximum hour, b: exemplary hour for P2G analysis

Full size image

Based on \((N-1)\) grid utilization, the redispatch for France and Germany is determined via congestion management simulations. The congestion management model determines cost-minimal actions to remediate bottlenecks in order to obtain an \((N-1)\) secure grid state [34]. The cumulative schedule deviation due to congestion management of individual power plants is shown in Fig. 15. Consequently, redispatch costs show necessity of further grid expansion.

Fig. 15
figure 15

Location of redispatch volumes of congestion management simulation

Full size image

6.2.4 Third layer: congestion management and integration with gas networks-gas

The resulting daily supply/demand time series by temporal disaggregation of the ENTSOG forecast data merged with the market simulation results from DMEM are presented in [23]. Using these per-country-defined time series, we find the cross-border gas exchange at the pan-European level by applying the second step in Sect. 5.2. In Fig. 16, an exemplary result is presented for the capacity utilization of the gas connections that carry gas from other countries to Germany in the scenarios involving the base market simulation data (for the rest, the reader is referred to [23]). Since Germany is in the center of Europe and connected to several other countries, the results are also used as a feasibility analysis of the pan-European cross-border energy exchange.

Fig. 16
figure 16

Capacity utilization of pipelines that provide gas to Germany for the base data set. a Peak demand hour, maximum supply forecast. b Peak demand hour, minimum supply forecast. c Minimum demand hour, maximum supply forecast. d Minimum demand hour, minimum supply forecast

Full size image

Analyzing the results of the third step in Sect. 5.2, we report that the gas amount that is displaced, i.e., redispatched to a node belonging to another region than that is given by the market simulation results, or not dispatched at all, not necessarily implies infeasibility of the market simulation results. The quality of the network topology data set highly affects this result since the inconsistencies between the different data sources used to generate the scenarios manifest themselves in the result of the second step. For example, since we use the forecasts from ENTSOG, a pipeline to be added in the upcoming years may be missing in the network topology data although it is taken into account in the high-level capacities in the first step. For Germany, migration from lower-calorific value gas (L-gas) to higher-calorific value gas (H-gas) also affects the consistency of the data sets in the second and third steps of scenario generation. This change implies replacing the source of the entering gas, i.e., the gas supply to Germany from the Netherlands is expected to decrease while a new LNG terminal is planned for Germany to replace the gas supply. The data regarding this change is only partially available from public resources making the gas network data prone to error. Besides, the pipeline capacities in the second model may be overestimated depending on the available data and assumptions to fill the missing data.

On the other hand, \({\text {H}}_2\) in-feed from the artificial P2G in eastern Germany in the proof-of-the-concept scenario is infeasible with the existing network topology, since there is only one entry node in the acceptable vicinity of the installed P2G facility and the amount of \({\text {H}}_2\) from this entry node exceeds the 10% ratio. However, there are underground storage facilities connected to this entry node. Hence, the optimality of replacing a P2G facility resulting from MIM and DMEM does not imply the technical feasibility to feed the \({\text {H}}_2\) to the gas network. In the rest of the analysis, we assume the existence of the \({\text {H}}_2\) storage facility and investigate the feasibility of injecting the energy-equivalent amount of natural gas into the network instead of \({\text {H}}_2\).

For none of the generated scenarios, we could conclude that the generated gas in-/out-flow scenario is feasible. In this particular case study, our analysis heavily relies on open data for the gas transport network. Additionally, the input data sets are sourced from various data sources, introducing a potential risk of data inconsistency and highlighting the importance of data quality in our results. To address this concern and provide a robust workflow, we introduce tools for diagnosing and correcting data errors in Sect. 5.

In Sect. 5.4, we present examples of data errors that were identified and corrected by leveraging insights from mathematical models and other data sources. Consequently, here in this section, we solely focus on reporting the results from the final iteration, acknowledging that further correction of encountered data errors is beyond the scope of this case study and paper.

However, to gain a deeper understanding of the infeasibility issues identified by the third-layer analysis, we offer mathematical modeling tools for scenario evaluation. This includes the slack formulation of the gas network optimization model and the minimum irreducible infeasible subsystem methodology, as outlined in Sect. 5.3.

By incorporating these tools, we aim to provide a comprehensive analysis and enhance our understanding of the nature of infeasibility observed during the third-layer analysis.

For instance, we report that the infeasibility in the case study may be related to the similar geographical region of the network, more precisely, the part of the network that transports L-gas by the time the analysis was done as shown in Fig. 17. This region is planned to be completely converted to H-gas in 2040. In order to fix the infeasibility, the slack formulation eliminates the gas exiting from the nodes connected to this region and assigns the gas to the most appropriate nodes of the network. But these nodes may not be geographically close to the nodes where the demand is eliminated as the slack formulation is not aware of the geographical location (e.g., the distance between the nodes) of the nodes. On the other hand, when we evaluate the minimum irreducible infeasible subsystem solutions with the updated gas in-/out-flow scenarios, we see that the infeasibility is mainly related to the Russian gas entering Germany. In this part of the network, the North Stream 2 is a planned connection with Russia and it is connected with a planned pipeline, EUGAL, to Germany, regarding the gas supply data provided by TYNDP 2020 [41] that we use for our case study (see Sect. 3). Besides, the planned compressor station at Radeland on EUGAL has not been included in the compressor station data used in the analysis as we did not have enough data for modeling the compressor station by the time this analysis was performed. For a more detailed discussion of the results, the reader is referred to [23].

Fig. 17
figure 17

The geographical location of non-zero slacks results (in yellow) from the GNO slack formulation minimizing minimum vertex slack. a Results from PdMax-B Scenario. b Results from PdMin-B Scenario

Full size image

Consequently, we observe that despite our effort for improving the network data, the complex interplay between the feasibility of scenarios and projected network design makes a quantitative analysis of the flexibility options introduced by the gas infrastructure difficult. While we can come up with suggestions from slack analysis, more detailed insights depend on an automated process to diagnose errors in the data and correct them.

7 Discussion

In this paper, we have proposed a workflow for a holistic evaluation of decarbonization pathways and their operational implications. This workflow with the interaction of the MIM, DMEM, and GNO models is demonstrated in a case study. Besides, the open data used in the case study was published as three data sets for the use of the energy research community. Their total number of unique downloads have already exceeded 220 [25, 48, 49].

We analyzed several scenarios using MIM to achieve COP21 targets and carbon neutrality in a multimodal approach involving investments and operational dispatch of more than 100 technology types to show up possible pathways for energy transition and analyze the impacts of sector coupling and decarbonizing of the individual sectors.

After disaggregating results from the MIM to high spatial resolution, we applied DMEM to check the results from the aggregated level on a high spatial resolution for market and transmission grid operation for the focus year 2040. We were able to identify bottlenecks in the grid and adjust the operational schedules of electricity generation units and consumers. We also identified potential locations for P2G maximizing grid flexibility.

We proposed a method to integrate the gas grid into the electricity grid by a physical flow-based GNO model. With the pan-European gas demand and supply, we checked the operational schedules of electricity and heat generation from gas. We demonstrated how to check potential challenges from P2G generated hydrogen infeed to the gas grid, and how to check the gas network’s capability as an alternative energy transport network to the electric grid.

With the case study, we also presented two-way interaction of the models, such as the infeasibility feedback provided by GNO to the upper-stream models.

As a result, we show that the proposed method enabled an integrated analysis of the European energy system, with a macro-economic analysis of 33 countries and interactions of more than 100 technologies in MIM for 2020–2050. We disaggregated the results of 2040 for operational analysis with a high spatial resolution with DMEM, i.e., the heating supply of CWE countries is modeled at the postal code level with more than 170.000 spatial entities for 2040. The market simulation included 1602 power plant units, 891 storage units, and 8870 virtual power plants. Last but not least, we analyzed the DMEM results first at the pan-European level involving 491 inter-country gas connections, and then performed a detailed analysis of the German gas network using a physical flow-based gas network optimization model with a detailed gas transport network. Because of the complexity induced by such a large scope of the analysis, it is not computationally feasible to run all models for all years in the whole planning horizon. Thus, smart trade-offs are needed to analyze sensitivities and scenarios, such as by focusing on appropriate uncertainties at each analysis step, and extending the analysis down in the model chain only if it would affect the results of the downstream models. In this way, this workflow can also help analyze the impact of potential changes in the political situation on the grid expansion plans, and alternative courses of action. Besides, the proposed workflow has the advantage of modeling the sequential real-life processes thanks to the detailed analysis models in the third layer preceding the market simulation in the second layer.

Nevertheless, we suffered from a lack of data and inconsistencies in input data quality for the gas network analysis. From the gas network modeling point of view, our results show that a capacitated-network flow-based model not accounting for the non-linearities in real-life such as pressure loss in the pipelines and the active components like compressor stations of a particular gas grid, can result in false positives in terms of feasibility of gas in-/out-flow scenarios as they can overestimate the pipeline capacity. We also show that from an EU perspective, having a consistent data set for gas transport network topology is key to enabling a practically relevant alignment of energy transport via electric transmission grid and gas network models. Future projects should include work on improving the data quality.

Last but not least, in the gas grid analysis, the capacities of North Stream and North Stream 2 are uncertain in light of the current political environment. For the case study, we use the available data published before February 2022 so that our analysis includes the related capacities as they had been planned. We do not make any amendments as changes in the expansion planning reflecting the geopolitical environment are highly uncertain. However, we promote the methodology rather than the results since we can use the proposed method to evaluate any alternative connections or expansion plans, which may arise. Nonetheless, the case study results demonstrate the use of the tools proposed by the workflow for evaluating third-layer analysis, like, distinguishing data errors and scenario infeasibility. Besides, the case study reports cases that these tools enabled correcting data errors if enough information is available. We emphasize the recalibration of data through the consolidation of updated data sets is beyond the scope of this paper and the case study. However, the demonstration of our proposed workflow’s capability to detect infeasibilities based on network topology provides compelling evidence supporting the effectiveness of the third-layer analysis when utilized with high-quality data.