On New Thinking and Designs of Electricity Markets

In recent years increasing shares of variable renewable energy sources (RES) have changed the structure of electricity markets in several countries. The core objective of this paper is to provide insights into the conditions necessary to bring about a more democratic and sustainable electricity system by integrating even larger quantities of variable RES. Our major finding is that a market-based approach would ensure that competitive forces rather than governmental interferences – such as capacity mechanisms – shape the future of the electricity markets. This transition towards a competitive and sustainable future electricity system will be based on an approach of “new thinking” which requires a paradigm shift in the whole electricity system. This includes switching to a more flexible and smarter concept allowing a greater scope for demand participation, storage options and other flexibility measures.


Introduction
For a long time the electricity system has been determined by the generators. Until the mid-1990s, and in many countries even longer, large generation companies, which were oft en highly vertically integrated, dominated the electricity system. Th is was supported by the assumption of existing economies-of-scale. Huge power plants, mainly nuclear and hard coal, were constructed along the lines "the bigger, the cheaper". Th is strategy was accompanied by high growth rates in electricity consumption.
Over the course of time these patterns have changed. At fi rst, growth rates fell from 10% per year in the 1960s to about 5% in the 1980s and 90s and to almost zero (in some OECD countries) in recent years. Th at is to say, today, there is less cake to be shared between generators, especially, given the preference for renewable generation.
Th e fi rst signs of criticism of such a generation-focused paradigm emerged already in the 1970s. Lovins (1978) was one of the fi rst to predict three major developments: (i) that future electricity consumption rates would decrease; (ii) that decentralized generation mainly from PV systems would increase, and (iii) that the importance of demand-side issues would grow. In addition to this, with the liberalization of electricity markets the picture began to change. Th e core objective of liberalization was the introduction of competition in generation in order to harvest the full benefi ts of electricity supply for both citizens and industry. Due to huge excess capacities aft er the fi rst phase of liberalisation, the principle of "prices equal short-term marginal costs" in spot markets emerged. It is important to note that in this first phase of liberalized markets the old "one-waythinking" still prevailed,. which was characterized by the fact that the generators were at the core of the system and of the thinking of the policy makers, see Fig. 1. In recent years, mainly due to the increase of variable renewables the capacity (factors) of the conventional plants has decreased leading to losses in revenues for their owners. This has resulted in growing concerns that most of these plants will be shut down thereby leading to decreases in supply reliability. Consequently, energy-only markets* have been questioned and calls for capacity payments have been launched.
The core intention of this paper is to serve as a primer for introducing truly competitive, democratic and sustainable electricity systems in every country worldwide. It is motivated by the current discussion on how to integrate large shares of variable RES but the basic intention goes beyond that. The aim is to show how to attain real competition in electricity markets, including all dimensions such as generation and storage as well as demand-side options.
In addition, the European Commission has set ambitious targets for increasing the share of electricity from renewable energy sources (RES), e.g. EC (2009). Indeed, in the EU-28, in recent years electricity generation from variable sources such as wind and solar has increased dramatically, with Germany, Spain, Italy leading. In the EU-28, between 1997 and 2016 "new" renewables (excluding hydro) mainly from wind grew from less than 1% to about 16%. For 2030, the EU has set further goals of a share of 27% (compared to about 14% in 2016) energy from RES. This target is for all uses -heating, electricity and transport. Consequently, also electricity generation from RES will grow, as documented in the National Renewable Energy Action Plans (NREAPs), however, it is not clear to which absolute level. Another major motivation for this paper is to show what is needed in order to integrate these higher quantities into the electricity system.
These increasing shares of variable RES have especially in Germany changed the usual pattern of electricity markets in Western Europe. Yet, variable RES-E do not provide electricity simultanously with demand. It is important to note, that almost all other generation technologies do not either. The fact that these must run capacities are offered at Zero costs over a large time per year have led to the argumentation that fossil plants like Combined-cycled gas turbines (CCGT) or coal power stations become economically less attractive because of the lower fullload hours per year. This argument has led to the call for capacity mechanisms (CM) in addition to the current "energy-only" markets. The idea is that specific owners of a flexible power plant should be paid for holding the plant ready for operation.
Due to these developments, currently, the whole electricity system is at a crucial crossing. On the one hand, the way to a sustainable electricity system based mainly on RES could be paved in the next years. In this context we emphasize especially the considerable price decreases of PV which has brought this technology close to cost-effectiveness on household level, see . On the other hand, there are forces which try to retain the old centralized fossil and nuclear-based generation planned economies. Capacity mechanisms (e.g. in France and England) should help to freeze this anachronistic pattern.
The core objectives of this paper are: (i) to explain how a truly competitive market-based electricity system can be brought about in the future without continous governmental interferences; (ii) to explain why capacity payments do not contribute to such a system but rather preserve the present system and (iii) to show that generators will no longer be the heart of the system but rather balancing groups and the suppliers.
A specific intention of this paper is to bring together all important aspects for heading towards a sustainable as well as competitive future electricity system. It considers technical options and aspects of market design and applies it to a further increase of RES in the electricity system. Moreover, it links the concept of residual load to price signals from the wholesale markets, the relevance of flexibility measures on the demand-side as well as demand response due to these price signals.

How prices in electricity spot markets come about
To analyze the impact of variable RES on the prices in wholesale electricity markets it is first important to understand the current market rules and market structures, see Auer/Haas (2016). Of key relevance is to understand how prices in European electricity markets currently come about. In this context it is important to look at the historical dynamics. The liberalization process in Europe started in the late 1980s in the UK and gradually migrated to continental Europe with the implementation of the EU-directive on Common Rules for the Internal Electricity Market (EC, 1997). One of the major features of the liberalized electricity markets was that the pricing regimes changed. In former regulated markets, prices were established by setting a regulated tariff, which was calculated by dividing the total costs of electricity provided by the number of kWhs sold -with some differences between different groups of customers. The major change that took place after liberalization was that prices on the wholesale electricity markets were now expected to reflect the marginal costs of electricity generation. Since then the price formation is mainly based on a fundamental approach where the intersection of a merit order curve on the supply-side and the demand curves results in the corresponding market price at every point-of-time, see Haas et al (2013b) and Fig. 2.

Fig. 2
Merit order supply curve with and without additional PV capacities at on-peak time of a bright summer day with short term marginal costs for conventional capacities Th e typical historical pattern of electricity generation in the Western Central European electricity market consisted since decades of conventional fossil, nuclear and hydro capacities. Since the late 1990s in western central Europe, most of the time nuclear contributed the largest share, followed by fossil and hydro. Non-hydro renewables were not a signifi cant factor until recently. However, since 2013 renewable electricity contributes the largest quantity in the EU-28. At the time when liberalization started huge already depreciated excess capacities existed in Europe. Th is led to the expectation that prices will (always) refl ect the short-term marginal costs (STMC) as illustrated in Fig. 2.

The impact of variable RES on prices in electricity spot markets
Between 2011 and 2016 remarkable decreases in day-ahead prices at the Western European power exchanges were observed see Fig. 3. Th e major reason for this decline in day-ahead prices was the increase of variable RES with zero short-term marginal costs. It is the remarkable rise Th is increase of renewables has started to impact spot prices, trading patterns and the dispatch of conventional generation by about 2011. Th e explanation is simple. Assume e.g. a sunny day with ample solar generation. Th en the supply curve is shift ed to the right as schematically shown in Fig 2, which essentially pushes nuclear and fossil fueled generation "out of the market", Haas et al 2013b.
Th is impact of variable RES on electricity prices is already known since volatile hydro power was used for electricity generation. Th e best example is the Nordic market, mainly Norway and Sweden, where since decades almost only technologies with Zero short-term marginal costs meet the whole supply. Since about 2007-2010 -in Denmark already earlier -there was experience with temporarily high wind in the systems, see e.g. Nicolosi (2010). In recent years increasing generation from photovoltaic systems was added to the production portfolios, mainly in Germany, Italy and Spain, and has contributed to temporarily very low -sometimes even negative -prices.

4
The end of the myth of base load The core question is, what the impact of the aggregate of various variable RES on the wholesale electricity markets is. Aside from the above-described effects, variable RES will also influence the costs at which fossil generation -especially natural gas -is offered. The reason is that they would lead to much lower fullloadhours, e.g. only 1000 instead of 6000 h/yr before. Yet, the revenues earned from these hours must cover both the fixed and variable costs, see also Haas et al (2013a). Hence, in a market with large shares of renewable energy sources the role of conventional capacities will change see e.g. Nielsen et al (2011). This leads to the following categories of presumed "problems": (i) Prices decrease to Zero or become even negative at a number of days; (ii) a lack in contribution margin to fixed costs for conventional flexible power plants. However, it is not yet clear, on how many days very high and on how many days very low (or negative) prices will prevail and how high or how low these prices will be.
Of further relevance in this context is how the price spread in European markets will evolve in the future as larger amounts of PV, solar thermal and wind generation are added to the network. The consequence for electricity prices are shown in Fig. 4 where a hypothetical scenario with high levels of generation from wind, PV and run-of-river hydro plants over a week in summer are depicted using synthetic hourly data for an average year in Austria. The figure leads to significant volatilities in electricity market prices with total costs charged for conventional capacitiesblack solid line -within very short-term time intervals.

Fig. 4
Development of variable RES from wind, PV and run-of-river hydro plants over a week in summer on an hourly base in comparison to demand and resulting electricity market prices with total costs charged for conventional capacities (Source: own analysis, adapted from Auer /Haas 2016) Our method of approach is based on the following principles: (i) Most relevant is the coverage of residual load which is the diff erence between fi nal electricity demand and generation provided by non-fl exible electricity generation from variable RES as well as coal and nuclear power plants, see Fig. 5; this is modeled on an hourly base over a calendar year based on historical RES electricity generation; (ii) Deduction of available conventional and backup capacities including must-run; (iii) fl exibility on the demand side based on consumer behavior incl. fl exibility instrument such as storage etc.; (iv) hourly electricity prices equal to short-term marginal costs and scarcity as well as excess pricing. For the residual load shown in Fig. 5 a price pattern as described in Fig. 4 may emerge. Hence, in the long run the impact of variable RES on the price spread is that it will increase. Th e intuitive explanation is that when renewables are plentiful, say during windy or sunny periods, the prices will be extremely low, approaching zero or possibly going negative, while at other times -when demand is high and renewables are scarce -prices can be much higher due to strategic bidding by fossil generators exercising market power. While Fig. 5 shows the concept of residual load over a week, Fig, 6 shows the corresponding graph over a year classifi ed by magnitude in decreasing order. In Fig. 6 the classifi ed residual load curve over a year in the case of high shares of variable renewables is described including the relevant areas for the discussion. Th e crucial areas in this load duration curve are on the top left and on the bottom right. In the circle on the top left the question is how to cover under shortage on these hours, in the circle on the bottom right the question is how to use this excess generation of electricity.
For both areas there are in principle two options: • By regulated capacity payments ? or • By competition between supply-side and demand-side technologies and behaviour (incl. Storages, grid and other fl exibility options)?
Important remarks: Flexibility measures will contribute in a competitive way to reduce these price spikes and consequently the price spreads and lead to new equilibria between supply and demand! As an example in Fig. 6 the profi le of residual load in Austria 2013 and the development in a scenario up to 2030 with a much higher share of variable renewables is described. Th e major fi nding of Fig. 6 is that the duration curve of the residual load profi le will become steeper and that the number of hours with excess generation will become higher. Th is eff ect will lead straightforward to higher price spreads and will also increase the attractiveness of storage, fl exible peaking units and other fl exibility options.

The core problems of regulated capacity payments
If the price pattern described above in Fig. 4 is not accepted by politicians another option are capacity payments. The fact that the renewable must run capacities are offered at Zero costs over a large time per year have led to the argumentation that fossile plants like CCGT or coal power stations become economically less attractive because of the lower fullload hours per year. This argument has led to the call for "capacity" payments in addition to the current "energy-only" markets. The idea is that specific owners of a flexible power plant should be paid for holding the plant ready for operation. The major reason, why at least currently there is no need for centralized CP in Europe that there are still many other options in the market, which we think are by far not yet exhausted. However, to exhaust these options some dogmas has to be changed. Especially the historically prevailing and still existing definition of supply security -that every demand has to be met at every point-of-time regardless of what are the costs -has to be revised in a way that compares the costs of (all) supply-side and demand -side options as well as customers WTP for capacity depending on time.
The major open questions regarding centralized capacity payments are, see Haas (2014): • Which quantity of capacity should get payments and where? • How to split in existing and new capacity? • How to tune with grid extention? Every grid extention has undoubtedly an impact of necessary capacities in a specific area • Who would plan? On national or international level?
There are three core problems regarding CP: 1. All regulatory capacity payments for power plants destort the EOM and lead to wrong price signals for all other options 2. Price peaks at times of scarce resource should revive the markets and lead to effective competition 3. We should strive to retain system resource adequacy by ensuring correct price signals and without capacity payment 4. Every capacity payment reduces the shares of the variable renewables

6
A market design approach for supply security One major argument for the call for centralized CM is to retain supply security in the electricity system. The historical (anachronistic) definition of supply security is: At every point-of-time every demand has to be met regardless of the costs. In this context it is important to note that supply security is an energy economic term. It is different from technical system reliability.
The core problem is that so far world-wide the demand-side has been neglected widely with respect to contributing to an equilibrium of demand and supply in electricity markets. Major exceptions are: (i) in the 1980s and 1990s in the U.S., Sweden, Denmark and other advanced countries DSM-measures have attracted attention. After the liberalization of the electricity markets most of these programmes disappeared. (ii) In Denmark -the leading country for integrating variable renewables especially wind -has integrated a lot of power-to-heat technologies, that now play an important role in energy markets.
The major reason for this ignorance of the demand-side is that in times of regulated monopolies every demand could be met due to significant excess capacities. And still in the liberalized markets huge excess capacities remained. This aspect -to develop the impact of demand-side and customers WTP -is essentialy for a real electricity market and it is actually regardless of the aspect of an integration of larger shares of RES.
Such a market-based approach would take into account customers willingness-to-pay (WTP). The equilibrium between demand and supply would come about at lower capacities. It is also important to note that at points-of-time where WTP is lowest, e.g. in the evening, the marginal costs (MC) of providing capacity could be highest, see also Auer/Haas (2016).

7
Flexibility: The key term of the future Our major findings for integrating large quantities of variable RES-E into the electricity system by using market-based principles and how, straightforward, a sustainable electricity system could work, are that the following conditions have to be fulfilled, see also EC (2015): • Of core relevance for integrating larger shares of RES-E in a competitive way is a pricing system in revised energy-only markets where the prices signal provide information on scarcity or excess capacities at every point-of-time (at least at quarters of an hour); • Another important issue is that the demand-side market is developed, see above. So far consumers have never been asked what the value of capacity is for them and what they are willing to pay for specifi c quantities of capacity. An important analysis in this context has been conducted by Praktiknjo (2013). He clearly identifi es two fi ndings: (i) there is a quite diff erent WTP between diff erent groups of customers; (ii) it is very unlikely that generating electricity is always cheaper than saving capacity.

Fig. 7
Dimensions of electricity markets • More fl exibility in the organization of the market is required: To better integrate electricity from RES in the market the time intervals in markets should be reduced (more emphasis on intraday markets, shorter trading intervals (from hours to ¼ hours); shorter ahead leading times for market clearing and forecasting of electricity generation from variable RES); • Most important to balance variations in residual load is to implement an optimal portfolio of fl exibility options which already exists today. A very comprehensive review of energy system fl exibility measures to enable high levels of variable renewable electricity is provided in Lund (2015). Currently these potentials are not fully harvested due to low economic incentives, see next chapter. Th e most important fl exibility options to balance variations in residual load are, see Fig. 7: • short-term and long-term storages such as batteries, hydro storages, or chemical storages like hydrogen or methane; • Technical demand-side management measures conducted by utilities like cycling, load management, e.g. of cooling systems; • Demand response due to price signals mainly from large customers to price changes, time-of-use pricing; • Transmission grid extention leads in principle to flatter load and flatter generation profiles; • Smart grids: They allow variations in frequency (upwards and downwards regulation) and switch of voltage levels and contribute in this context to load balancing • Balancing groups will play a key role in this new concept. These are the entities which finally have to balance generation, flexibilities and demand options.

New vs old thinking: Further development of the wholesale electricity market design
Regardless of the issue of increasing quantities of variable RES in the electricity system there are some measures to be introduced which would improve the wholesale market structure and competitiveness basically. In addition to a revised EOM these are: • more flexibility in the organization of the market is required; • shorter ahead leading times for market clearing and forecasting of RES-E production; • long term contracts (futures, forwards) should be made available even for longer time periods than 6 years if the market needs it.
Finally we state that the transition towards a competitive and sustainable future electricity system will be based on the following principle of "new thinking", which is to accept a paradigm shift of the whole electricity system -including switching from an inflexible and one-way system where variable load is met with changes in generation to a more flexible and smarter system allowing two-way electricity flows -to our understanding -a greater scope for demand participation by consumers needs to be included. In addition, suppliers (or balancing groups) are the most important part of the whole energy service providing chain, see also Fig. 8.

Fig. 8
New thinking in electricity markets: a supply-oriented bidirectional system with very high fl exibility As indicated in Fig. 8 in future decentralized PV systems along with decentral battery storages may play an important role. Th e astonishing changes in the solar industry epitomize the over-all way PV is heading to. (WNISR 2015): "Th ere seems to be a general recognition that the fall in production costs of RE technologies, particularly of PV, coupled with the expected falling costs of electricity storage will accelerate the transformation of the power sector." And the IEA, which has been tradionally skeptical with respect to RES states in the WEO (2017): "PV is on track to become the cheapest source of new electricity in most countries world-wide".

Conclusions
Th e major conclusions of this analysis are: • Th e key to a sustainable competitive electricity system is the full exhaustion of fl exibility options based on correct price signals in the wholesale as well as in the retail market. Currently on both levels the market does not yet provide proper price signals to trigger fl exibility options (e.g. technical demand-side management, economic demand-response due to price signals as well as shortterm and long-term storage options) which would balance the residual load profi le more eff ectively.
• It is not possible to force variable RES into the system by means of technical planning. Proper financial incentives are necessary. Correct price signals are crucial in a revised energy-only market along with scarcity and excess pricing signals; the only "negative" aspect of a market without a capacity component would be that, at least in the short run, prices higher or lower than short-term marginal costs may occur temporarily. After some time the market would learn to benefit from these higher costs and also from the very low costs at times when RES are abundant. A reasonable price spread would emerge providing incentives for different market participants to benefit. • Regarding market design, more flexibility in the organization of the market is required: To better integrate RES-E in the market, time intervals in markets should be reduced (i.e. more emphasis on intraday markets, shorter trading intervals -from hours to quarters hours; faster market clearing and shorter forecasting times regarding wind and solar).
In conclusion, This transition towards a competitive and sustainable future electricity system will be based on an approach of "new thinking" which requires a paradigm shift in the whole electricity system where no longer the generators are the centre but the balancing groups respectively the supply companies. This includes switching to a more flexible and smarter concept allowing a greater scope for demand participation, storage options and other flexibility measures.
Finally we state is that the evolution of such a creative system integrating variable RES in Western Europe may also serve as a model for RES-based electricity supply systems world-wide.