Keywords

1 Introduction

Energy market restructuring and liberalisation over the past thirty years has produced mixed results. Wholesale market design and competition has expanded and matured, while retail competition has remained static. Industrialised countries have observed both positive outcomes and market failures, while developing countries have had limited success in restructuring energy systems. Forming a global picture of the liberalisation progress is difficult, as countries launched from differing starting points, and advance at different speeds, with continuous changes introduced due to political unrest and financial crises. Energy policies tend to be multi-dimensional which, in turn, makes benchmarking and evaluating success of unbundling and liberalisation a challenging endeavour.

Energy policies are translated into regulatory frameworks to support policy goals. These goals include, but are not limited to, security of supply, increased competition by encouraging new participants through open transparent market structures, and environmental sustainability programmes. In order to analyse the results of liberalisation, an understanding of the extent to which stages of liberalisation have actually been implemented in the energy sector and remain in place is needed. The three stages in effective liberalisation include unbundling, launch of a wholesale market, and support for a retail market.

Liberalisation has evolved from different frameworks. The United States took an approach of cautious relaxation of regulatory control over prices and close observation of the potential problems inherent in participant market power. Europe, on the other hand, created markets under a new competitive framework that crosses member state boundaries. Most other countries approached liberalisation in a stepwise fashion by observing the performance of each stage of market restructuring. OECD countries have been reforming energy product market regulations, improving and adapting regulatory techniques to changing market and technological conditions. This is crucial to management of the competitive and non-competitive market segments that exist in the natural gas and electricity markets.

Government energy policies pushed for deregulation to improve sector efficiencies, while large energy consumers, such as industrial and commercial sectors, pushed to access cheaper pricing. In liberalising policy, states intended to encourage the competition needed to reduce costs by unbundling vertically integrated monopolies, effectively separating generation from transmission and distribution. To pass these expected savings on to consumers requires a sufficient number of competitors that have access to an open wholesale market. Eliminating guaranteed fixed price contracts should encourage private investment to participate in an efficient manner to add generation capacity based on market signals. However, private investment financing depends not just on an expected revenue stream but also on the degree of confidence that can be attached to those revenues. Increased renewable energy participation in electricity generation and change in price formation patterns have increased investors’ uncertainty in the past decade. In most countries, the existing wholesale market structures were designed to suit the technical, cost, and operating characteristics of fossil fuel plants. Thus, the advent of large volumes of zero marginal cost generation, a significant part of which may be intermittent or inflexible or have other characteristics very different from conventional thermal plants, will require substantial revisions of existing marketing structure to ensure system stability and to encourage private investment (Sioshansi 2013).

Regulators have worked hard to transfer efficiency gains in transmission and distribution into lower consumer prices (Newberry 2002) and some markets that initially offered lower retail prices. But most retail consumer costs increased. This was due to ongoing infrastructure costs in the delivery of energy and for electricity consumers specifically, because end-user energy billing absorbed the costs of renewable generation incentives, to encourage private investment, promised by governments. Germany is a prime example of the success of regulation in supporting renewable development, but the ultimate costs for these developer incentives were socialised into retail infrastructure bills. In a frustrating dichotomy, average wholesale prices have fallen and retail prices have risen (Hannesson 2019) (Fig. 24.1).

Fig. 24.1
figure 1

Historical German retail price and wholesale market prices. (Source: Author’s graph, retail prices include taxes, levies, and fees. Data from Federal Statistical Office and European Energy Exchange)

Full size image

The past two decades have seen an increased frequency of electric power shortages in both the developing and developed world. Power shortages seldom have a single or the same cause. However, a typical pattern begins with underinvestment or very rapid demand growth that degrades reserve margins below acceptable reliability levels, and then unusual combinations of market fundamentals, such as adverse weather, fuel supply, or plant availability, will create a crisis (Heffner et al. 2010). These shortages have occurred in both liberalised market structures and traditional regulated monopolistic regimes. While liberalisation of energy markets and the resulting new regulatory structure have not delivered the results economic policy makers expected, market efficiencies and product delivery seem no worse than prior to liberalisation.

This chapter will focus on a comprehensive review of the alternative restructuring approaches used to break up vertically integrated monopolies and is presented along with the effects of generation deregulation on competition, the efficiency of system operations, and the impact on wholesale and consumer prices. It will highlight key historical moments in policy liberalisation. The chapter will concentrate on the electricity sector with brief reference to the natural gas experience and the importance of natural gas market liberalisation for electricity markets. Price risk resulting from unbundling and opportunities for hedging in the physical and financial markets will be discussed. The California energy crisis will be used to highlight the intersection of market failures in the state’s first attempt at liberalisation and establishment of a new regulatory framework. Structural economic factors of the energy sector relating to policy and regulation will be discussed. Both developed and developing countries have achieved success of policies for renewable energy investment and more mixed results through environmental cap-and-trade programmes. The chapter will conclude by describing expected future challenges to regulatory market structure in the form of increasingly intermittent supply, decentralised generation, and changing consumer demand patterns.

2 Energy as a Network Industry

Network industry privatisation commenced with the first wave of neoliberalism, under the political stewardship of Ronald Reagan and Margaret Thatcher during the 1980s. The European Commission (EC) took up Thatcher’s privatisation programme at the end of the 1980s, introducing mandated deregulation and privatisation to remove state control and eliminate the concept of the public good. The norm in the United States model is to have private ownership of unbundled network utilities combined with regulation oversight. OECD countries other than the United States have liberalised energy utilities without the use of privatisation (Ugur 2007). The EC along with OECD governments had successfully liberalised the telecommunications and air transport sectors, offering proof that a fully liberalised environment would unleash competitive forces. Technological advancement provided the foundation and capability for market unbundling and vertical separation. New governance and regulatory techniques and an increase in openness to international challenges and competition accelerated liberalisation and privatisation (Belloc and Nicita 2011).

Electricity and natural gas are network industries and have similar characteristics to other network industries such as telecom and rail transport. Network industries are traditionally characterised as natural monopolies that were originally under the control of public ownership. Natural monopolies exist when economies of scale are significant in that a single firm can produce total business output at a lower unit cost and thus more efficiently than multiple firms. This situation gives rise to a potential conflict between cost efficiency and competition. Energy markets typically exhibit structural characteristics that allow incumbent firms to exercise market domination, which restricts competition and potentially generates higher prices and reduced consumer protection. Thus, competition cannot be relied upon to provide the socially optimal outcome. Some form of government intervention in these network industries is required to support quality of service and price protections for consumers (Gonenc et al. 2000).

Energy industry unbundling has resulted in generation, transmission, and distribution as independent segments. History has shown that generation is an excellent candidate for liberalisation and privatisation. Technology through investment channels have delivered more efficient and smaller scale energy production to an open transparent competitive market. Much like the energy industry generally, energy transmission and transportation infrastructure is considered a natural monopoly. There has never been a serious attempt to liberalise this segment through the enticement of private investment. These networks have high fixed costs, long approval processes and construction timelines, and little opportunity for long-term contracts, which provides little attraction to private investors. Thus, transmission assets are either public or privately owned with regulatory oversight to ensure system reliability and fair market pricing to participants.

3 Unbundling Vertical Monopolies and Enforcing Competition Through Regulation

Regulation is driven not only by normative considerations, such as reducing and controlling rent-seeking behaviour, but also by the impetus to determine the optimal organisation of the system it regulates. This positive theory of regulation treats the existence and forms of regulation as a response to demands of politicians and interest groups. Government ideology significantly influences energy regulation (Chang and Berdiev 2011). A government’s degree of fragmentation, political strength, and authority determine its ability to implement regulation of the energy sector. Private investors lack confidence that changes in rules will not affect their investment if the sustainability of the regulation design is not clear, and thus they are wary of investing in energy in countries with signs of institutional weakness.

In OECD countries, network industry liberalisation in the 1990s resulted in the emergence of national- and subnational-level regulators to advocate for unbundling, open access, and the creation of market-driven signals to encourage competition. Competing regulatory bodies at differing levels of government as well as competitive and monopolistic design frameworks for generation, transportation, and distribution systems created confusion for potential investors. Although these regulatory groups were intended to coordinate and harmonise rules, inefficiencies in competing regulatory strategies emerged.

For mature network sectors, liberalisation policy started with unbundling the value chain into separate components and then determined which ones are potentially competitive or monopolistic. The competitive activities were liberalised via reduction of the state’s role and the introduction of a competition framework to produce economic efficiencies. Competition improved the focus on consumer experience and added consumer choice, which drove new product and service innovation.

Unbundling creates price risk. Producers and generators need to sell their commodity at market prices. Regulators did not fully understand the consequences of the market price risks or the application of forward contracts for hedging. The volume and complexity of linear and non-linear derivatives have increased as energy trading markets have matured making market monitoring challenging. Regulators need to maintain currency in trading strategies, to identify opportunities for further market efficiencies, and to monitor and punish market manipulation behaviours.

Public response to environmental degradation contributed to government intervention and climate change regulations in the energy sector. Global climate change may well be the environmental externality governments had the most difficulty regulating (Bazilian et al. 2011). The only proven means of boosting environmental protection, in an efficient manner, is economic incentives. The introduction of quantity-based mechanisms, such as tradable permits, has created market-harnessing controls aimed at policy goals (Stavins 2010). The largest market for tradable permits is the European Emissions Trading Scheme (ETS) for carbon emissions, which covers all medium- and large-scale industries within the EU borders.

States have embraced several approaches to regulation to ensure fair pricing in the energy sector: direct command and control, incentive based (which includes cost of service), rate of return and price caps, and market-based controls (which provide a structural form in the market to influence competition). Public ownership was historically run as a cost of service regulation, but both consumers and politicians criticised it because of the lack of incentives to reduce costs or address operating inefficiencies. Rate of return regulation has also been criticised for having poor incentives to develop efficiencies for lower costs to the end consumer. Price cap mechanisms have had more support and perceived efficiencies, since the energy producer will only improve financial performance, by lowering their costs.

Energy regulation evolved to protect both consumer and investor interests by setting prices or rates that were considered ‘just and reasonable’.Footnote 1 Prices were set at levels in response to consumer and political pressures and provided acceptable returns on investment for private and public integrated utilities. The US Federal Energy Regulatory Commission (FERC) relies on market forces, rather than regulation, to support competition and to achieve just and reasonable wholesale prices. While this avoids a host of distortions and inefficiencies, FERC’s record suggests the mere possibility of a market failure is insufficient to ensure good outcomes for consumers.

Comparing the outcomes of observed regulatory regimes provides information as to what policies are superior. Four model frameworks for the energy sector exist, three of which represent stages of liberalisation. These four illustrate the choices that governments make in deciding how to run the energy industry (Table 24.1).

Table 24.1 Energy sector paradigms and competition
Full size table

The first model is franchise monopolies, the traditional regulated utility scenario, where no competition exists at the generation or transmission levels. One company, which can be a public or private entity, is allowed to produce energy and deliver it to end consumers. Franchise monopolies have no liberalisation.

The second model is the single buyer structure. In this model, generation is fragmented into multiple legally separated companies to ensure none dominates the market and competition is possible. The transmission grid is not deregulated, meaning companies do not have open access, because the purchasing agent designated to represent all consumers is the only possible user of the grid. The companies have long-term contracts so that they have revenue certainty to obtain and support generator financing arrangements. This approach is considered an interim step in the road to full liberalisation, with unbundling of only generation, as the government deregulates generation and acts as an agent to provide stability while the generation market structure changes take place.

The third model is the efficient launch of wholesale energy competition with open access to pipeline and transmission systems. As with the single buyer structure, there are multiple competing energy-independent producers, but in this structure, there are also multiple competing buyers. No market power exists for the producer or consumer side of the market. Transmission has been unbundled from distribution, allowing for the design of an open access platform for all participants with an energy market structure in place, either by a bilateral trading market or by a centralised pool. This market structure is commonplace in the United States, the UK, and Europe.

The final model is complete liberalisation, which extends wholesale market and transmission access to include retail competition. It provides an effective retail market, where a retail market framework is implemented and where customers can choose their service providers. Implementation of this model involves a delay in the benefit to the end consumer in the wholesale and retail markets, since competition takes time to develop. The concept relies on consumer choices which increase transaction costs involved in managing the improvement of metering and the wide assortment of consumer contract structures.

Unbundling of a vertically integrated monopoly involves the functional and structural separation of different segments of the energy production and transmission systems (Sen et al. 2016). A simple approach to unbundling focuses on functional unbundling, which entails creating a separate legal business unit with separate accounting. Structural unbundling is more complex and requires separation of ownership of the grid from the generation and the designation of an independent system operator (ISO) or independent transmission operator (ITO). The ISO structure involves ownership unbundling of the system operator (SO), who runs the grid, from the transmission operator (TO), while the TO owns the transmission assets. In jurisdictions that have an ITO structure, regulators have left some flexibility for incumbent utilities such that transmission assets remain within the vertically integrated utility (Nardi 2012).

The energy sector is not fully liberalised in any country in the world, although most regions where unbundling has occurred have seen the wholesale market migration to an open competitive framework. Regulation in the gas market, particularly in continental Europe, has taken time to find an acceptable equilibrium that tackles import security and competitive concerns. Thus, electricity provides the best evidence on the consequences of unbundling and the sharpest test for regulatory frameworks.

4 Natural Gas Unbundling and Liberalisation

Unbundling of the natural gas industry decentralises the industry along horizontal and vertical lines, lowering barriers to entry, which resulted in the arrival of new participants to the market. Natural gas liberalisation not only benefits the wholesale natural gas market, but also opens access to transportation pipelines and storage for buyers. Regulators do not set natural gas prices directly. Instead, they support fair natural gas pricing by increasing competition using two types of market trading models: bilateral trading, which is popular in the United States, and centralised pool coordination, present in the UK. Both models provide a platform for streamlined trading and promote efficient wholesale pricing and minimise transaction costs.

The speed of natural gas market liberalisation has varied across the globe due to political forces and incumbent market structure. National energy policy for natural gas generally has three objectives: enable competition, security of supply, and environmental sustainability. Natural gas regulations were not initially designed with the expectation of a static supply-demand equilibrium, but instead with knowledge that a mature, growing electricity market is increasingly dependent upon gas-based generation.

Countries have taken two different approaches to the unbundling of natural gas utilities. The simple approach requires a separate legal entity and the structural approach involves ownership unbundling. The UK and Portugal have adopted ownership unbundling, while France and Germany have employed lighter legal unbundling rules (Growitsch and Stronzik 2014). In the United States, most states have legal unbundling of distribution in the natural gas chain and transmission remains tied to distribution (Ascari 2011). The standard argument about ownership unbundling indicates that, because it leads to a large number of independent producers, market discrimination is unlikely, while higher transaction costs could lead to a loss of economics of scope and lower efficiency. It may lead to forestalled investment and therefore security issues (Brandão et al. 2016). Recent research on the European Internal Energy Market have found that liberalisation of natural gas markets has produced only negligible reduction in wholesale and retail gas prices and that ownership unbundling had no impact on natural gas end-user prices (Growitsch and Stronzik 2014). Historically, gas suppliers, particularly outside the United States, hold long-term importing contracts, which are sometimes linked to oil pricing mechanisms, with take-or-pay clauses resulting in liberalisation policies for entry and market segmentation without benefits for consumers. In order to foster gas-to-gas competition, the development of liquid, transparent wholesale exchanges at market hubs is necessary to allow for development of short-term market transactions.

After the production segment liberalisation, regulation of the natural gas industry focused on pipeline transportation and distribution, segments that have monopolistic characteristics (Juris 1998). Although the market power of pipeline companies affects the transportation market, resale of transportation contracts introduces competition in this market and facilitates the efficient allocation of these contracts. In markets with transportation constraints, namely, gas importing countries such as Italy and Portugal, regulatory reforms have not helped foster competition. New investments are both subsidised and exempted from third-party access regulation for financial reasons, but importantly history has shown that a lack of ownership unbundling can negatively affect the incentive to invest (Cavaliere 2007).

Evidence has shown that during the deregulation of the electricity sector, liberalised coal generators benefited from lower costs as they were able to shop around for better coal prices. Natural gas-fired generators did not see a similar cost savings. Liberalised natural gas market trading already has liquid, transparent price signals that provide the best price availability to all wholesale buyers, such as natural gas-fired generators. Coal markets were at that time and continue today to be more opaque.

5 Electricity Market Liberalisation

Electricity market design aims to define rules and incentives that lead to an efficient functioning competitive market. For the past decade, market success has been achieved in OECD countries as wholesale pricing is near marginal cost. Power plants are more fuel efficient and system operators are able to optimally dispatch power plants to lower costs over larger geographic areas. Economists and policy makers use these and other factors to evaluate success and obstacles in electricity market unbundling: restructuring impact on wholesale competition, system operations, generation investment, retail competition, and environmental performance.

Competition in wholesale electricity markets can be described as somewhat competitive with mature systems around trading and price settlement. Wholesale prices in the UK, Australia, Norway (Nordpool), and, despite notable isolated failures of competition, US electricity markets (e.g. the Electricity Reliability Council of Texas (ERCOT) and the Pennsylvania, Jersey, and Maryland (PJM) Regional Transmission Operator (RTO)) are now reasonably competitive in the short run (Newberry 2002). A primary driver of this competitive performance has been the extent and magnitude of forward commitments, through contracts and vertical integration, between generation firms and retail providers.

California’s market liberalisation experience (2000–2001) was an unquestioned failure and most of the original initiatives were ultimately reversed (Carmona et al. 2012). The debacle showed that poor market design and circular regulatory and political intervention produce unsatisfactory outcomes when generation capacity is tight, particularly if energy shortages are unexpected. California had both inadequate legal frameworks and inadequate production capacity. Illustrating the importance of forward commitments, utilities were prevented from establishing proper risk management programmes, all in the name of encouraging participation in short-term markets. Price caps intended to protect energy buyers from egregiousness behaviour actually resulted in energy delivery out of state during critical in-state shortages.

California Electricity Crisis (2000–2001)

California’s first foray into liberalisation began in March 1998 with the launch of the wholesale power market along with consumer choice. Starting in 2000, the power exchange experienced high wholesale electricity settlement prices and intermittent power shortages. The three investor-owned utilities (PG&E, Socal Edison, SDG&E) began to have severe financial problems. By February 2001, after a series of rolling blackouts, the state government stepped in and purchased electricity on behalf of PG&E and Socal Edison, effectively ending market deregulation.

In order to support market growth and benefit from expected lower wholesale prices, utilities were not permitted to sign long-term contracts, instead were required to purchase electricity for end customers through the short-term power market exchange. This resulted in unhedged customer sales exposed to higher short-term price volatility. Chronic underinvestment in generation meant that supply did not keep pace with a robust, price inelastic, demand pattern in the years leading up to liberalisation. Imports from out of state were restricted in 2000 and 2001 due to very dry hydro conditions in the region. Complicating matters, electricity was being exported out of California to realise higher prices than in-state price cap limitations. State power shortage emergencies escalated through 2000, and there were thirty-eight Stage 3 levels in the first three months of 2001, which indicated that less than 1.5% reserve margin available and resulted in several rolling blackouts during the winter of 2001. Wholesale prices increased to price cap levels for prolonged periods, resulting in average daily prices of almost 400 USD/MWh in December 2000 compared to approximately 30 USD/MWh average in December 1999. Independent and out-of-state generators stopped selling to the three utilities from fear of non-payment. In order to stabilise the market, retail prices were reluctantly increased, which impacted end consumers. PG&E ultimately filed for bankruptcy in April 2001.

(Data sourced from US Energy Information Administration website, Subsequent Events California Energy Crisis Report)

The market for transmission and the market for energy are inherently intertwined in electricity system operations. The RTO should not be separated from the ISO or power pool (Hogan 1995). The ISO is responsible for grid stability through congestion management, balancing, ancillary services, and transmission usage. Integration of these functions with wholesale transmission leads to market success. The market has not facilitated transmission investment (De La Torre et al. 2008). This is in part because forward-looking congestion rents are an inadequate means of cost recovery for lumpy transmission investments. Efficient new transmission lines often eliminate the congestion rents that would otherwise motivate the investment. One way regulators can create market mechanisms to value transmission is to provide financial transmission rights (FTR) contracts. It is difficult to forecast future value of FTRs over long periods because of the challenges in accounting for changes to the generation resources and transmission system over time.

The attention to system reliability is not necessarily at odds with increasing competition and the improved economic efficiency it brings to wholesale power markets. Reliability and economic efficiency can be compatible if regulators, policy makers, and industry leaders work together to strengthen and maintain the institutions and rules that will protect these goals. In recent years, efforts at the grid level have largely been focused on dealing with issues such as renewable intermittency, congestion, load shifting, and bidirectional flow in distribution networks. In the future, technological innovation at the grid edge will facilitate the development of markets for distributed resources, service-oriented business models, and active distribution grid management (IRENA 2019). Regulators should emphasise investment incentives and innovation, not short-run operational efficiencies.

Generation investment from private sources has been robust for renewable energies where a subsidy mechanism is present. Growth in thermal dispatch generation has limited success because forward energy markets do not provide profitable price signals to entice investment and construction. Capacity markets in the long term are still uncertain as revenues from short-term energy and ancillary service markets have, most of the time, been insufficient to recover the full average cost of power plant construction and operations. The day-ahead, intra-day, and occasionally the forward market show negative prices because of production and transmission congestion dynamics. While negative prices accelerate system operations quickly in order to realign supply-demand equilibrium in the generation stack, in the long run they do not aid generation investment analysis.

While wholesale markets have clearly benefited from unbundling, retail electricity prices have remained static, or in many cases have increased, due to higher costs to support renewable energy development and integration. Retail electricity prices are more tightly linked to natural gas wholesale prices in regions that have restructured generation away from monopoly frameworks. Residential customers exhibit choice frictions, leaving most residential customers purchasing energy from distribution providers of last resort.

Emission cap-and-trade programmes are most effective when combined with competitive electricity markets. Well-connected wholesale markets have the potential to exacerbate the potential of pollution leakage in settings where pollution regulations only apply to a subset of the generators in the market. Regulated utilities and independent power producer (IPPs) respond to pollution cap-and-trade programmes differently; regulated firms invest in capital-intensive abatement technologies, while IPPs pursue lower cost abatement options. The European Union Emission Trading Scheme launched in January 2005 is preparing to enter the fourth phase, in January 2021, representing further reductions to industrial emission levels. This programme experienced growing pains, including market theft, tax evasion, and lack of coordination by member states (Stern 2009). Another example of an environmental regulation cap-and-trade programme using tradable permits was the successful reduction of SOx and NOx levels from coal-fired generation plants in the Los Angeles basin during the 1990s.

Electricity market manipulation enforcement actions have moved from the conventional analysis of generator market power abuse in day-ahead and real-time physical markets to material allegations of sustained cross-product price manipulation in forward financial markets. Electricity forwards and associated derivative product pricing with complex embedded optionality, occasionally with limited market transparency, and high expected implied volatility term structures, are challenging to price. Therefore, it is difficult to develop and apply forward market analytical frameworks and models for comparison to market price action. States have begun to use an adaptation of cross-product manipulation models from cash-settled financial markets, which provides a real demonstration under uncertainty and asymmetric information (Prete et al. 2019).

6 Unbundling Markets in Developing Countries

The foremost objective of governments of developing countries in utility reform differs from that of governments of developed countries. While the latter focus on improving existing energy structure efficiencies, developing countries have insufficient public funding to address high growth demand and weak infrastructure and, therefore, chronic supply shortages. Financing of energy infrastructure investment can, then, be considered fundamentally a macroeconomic problem. As a result, reforms prioritise the attraction of investment for generation capacity expansion. One financial implication of this is that funding through external capital channels by means of loans or debt adds to the country’s foreign liabilities and complicates its debt management. Consequently, in developing countries, motivation for privatisation has been driven to expand central systems, develop decentralised solutions for remote populations, and choose an energy growth trajectory that avoids the fossil fuel trap. Public-private partnerships (PPPs) that include development banks and institutional investors are not unusual in project financing. Such arrangements place development banks, which are local and regional in nature, on a participating partnership footing with a proactive agency role as initiators and coordinators (Arezki et al. 2017).

Selecting and applying the most appropriate institutional framework of regulation is a major challenge for developing and transition economies. Developing countries tend to take a stepwise approach to unbundling in order to determine foreign investor interest at each stage of change. With little financial support or expected private investment, countries such as Vietnam, Thailand, and Malaysia have been reluctant to open a full liberalisation process, instead maintaining unbundled generation model paradigm with IPP participation (Hall and Nguyen 2017). Brazil and the Philippines are the only developing countries that have followed the liberalisation process through from unbundling generation to the creation of a retail market with strong and ongoing government regulatory direction (Santiago and Roxas 2010).

Countries with limited regulatory capacity rely on multisector agencies that consolidate scarce regulatory resources (Estache 1997). Suboptimal energy network services can impede growth. This raises the question as to whether an unregulated private monopoly will make the investments necessary to offer the quality of service appropriate to a country’s changing needs over time. Implementation of regulatory reforms must occur upfront and prior to privatisation and should be as unambiguous as possible to ensure optimal transparency. History has shown that investor response is stronger under government guarantees. Establishing strong rules securitised in law can open channels of foreign investment for governments with low creditability and an inadequate track records, which is common in countries in sub-Sahara Africa. This can provide stability for foreign investors, although it compromises regulatory flexibility to fine-tune market support mechanisms.

Developing countries struggle to provide the regulator the independence that domestic and foreign investors seek. Most struggle with weak regulatory institutions and changing government ideology. In actual practice, they have little ability to ensure independence of the energy regulator. Hungary’s experience highlights the problems that arise when regulatory agencies change the rules or renege on pre-privatisation promises that guaranteed foreigners fixed real return on investments (Slay and Capelik 1998).

Latin America does provide liberalisation and privatisation success stories. Chile was the first to reorganise its electricity market, embarking on deregulation and privatising its national power system in the early 1980s. It focused, first, on vertical unbundling of the electricity supply chain. Cross-ownership and conflicts of interest initially hindered the development of a more competitive generation market. However, its well-designed regulatory structure that included power generation paid on a cost-based formula, unit dispatch order based on marginal costs, a power trading system between generators to manage customer contracts, and separation of large customers to allow them to freely transact in the wholesale market produced success. Transmission and distribution systems were maintained as monopoly entities with regulated price cap mechanisms. Government regulators were hands-on at the right times, supporting construction of critical interconnectors between the thermal-based generation region in the north and the renewable energy-dominated south region, to support energy transition (Pollitt 2008). Chile also avoided any significant changes in government policy over an extended period of time, and this allowed the market to stabilise and mature (Pollitt 2004). Argentina and Columbia liberalised their markets soon after Chile did, fragmenting generation capacity into many companies, allowing for divestitures and inflow of international capital, to alleviate stress on the governments’ financing of outstanding debt (Joskow 1998; Lalor and García 1996).

Brazil, the Philippines, and South Africa have not fared as well. Energy crises in these countries have resulted in frequent brownouts or blackouts due to unanticipated demand growth, and a failure to meet that demand indicated a lack of effective regulatory oversight, forecast planning, and lack of incentives for public or private sector expansion of capacity. In South Africa, the regulatory framework failed to motivate private sector participation (Castellano et al. 2015). In South Africa, as in California, market deregulation of the vertically integrated system led to a market failure and energy crisis, in this case in 2008. Regulators took a stepwise approach to deregulation which resulted in a miscalculation of the structure and size of the market.

India witnessed failure of market development coordination between national- and state-level natural gas and electricity sectors. In keeping with the global trend shifting from coal to natural gas thermal systems, India has experienced strong growth in natural gas-fired generation and hence demand for the natural gas commodity. The market structure for natural gas did not develop at a parallel speed with expected electricity generation demand, resulting in a shortage of natural gas for power generation. With the expected arrival of further natural gas pipeline capacity set for several years from now, many investors have reclassified generation plants as stranded assets. Regulators did not understand that liquefied natural gas, which is traded in a global market, would not be suitable for the country’s power generators due to the delivery cost structures (Worrall et al. 2018). This case highlights the importance of harmonised interaction between policy and regulations of the intertwined natural gas and electricity markets.

Africa as a continent is only beginning to liberalise. Several countries are pursuing the creation of renewable technologies connected to localised, rather than centralised grids to obtain high electrification rates (Clark et al. 2017). Fee-for-service methods of financing decentralised supplies rest on business models where local agents, operating concessions granted by a central authority, supply installations and maintenance and recover costs from subsidies and fee-for-service customer payments. Since the long-run financial sustainability of this model is not yet known, it is too early to say that it can be successful and have a significant positive impact on electrification rates. Botswana recently opened its market to private generation investment, while the incumbent utility remains a vertically integrated monopoly. It and other countries in Africa may move directly to an agile energy system.

As a group, few developing economies have functioning energy markets. Those that do have both government capacity and policy conviction. Unbundling of state-owned enterprises with an eye to privatisation has resulted in improved system performance, but a lack of planning for competitive markets remains (Sen et al. 2016). In fact, energy sector liberalisation and market design in developing economies have largely failed over the last fifteen years. Most rural populations in developing countries remain without electrification (Marandu et al. 2010). While the push to liberalise energy sectors continues in policy discussions, many countries are not moving forward with unbundling and liberalisation, as they have seen the developed world’s struggles and minimal success.

7 Energy Market Transition and Innovation

The process of globalisation which integrates world economies into a single system has supported environmental and climate change regulation of the energy industry. Two of the United Nations Sustainability Development Goals make direct reference to energy: access to clean energy sources for all populations and energy sector initiatives to further drive low-carbon development investment. The 2015 UN Climate Change conference, held in Paris, was the inflection point for nations agreeing on targets to reduce carbon intensity in support of climate change directives. Government support via research investment and incentive programmes, such as subsidies or tax benefits for renewable energy technology development, had started to pay off. We are now in the midst of an accelerating energy transition. Public policies to encourage the development and adoption of renewable energy technologies are essential, because low-carbon performance is not visible to most consumers and carbon is not priced in the broader global market. A wide range of factors, including volatile fossil fuel costs, falling renewable energy capital costs, energy security, climate change mitigation, and the avoidance of externalities attributed to fossil fuel energy, drive the trend to renewables. Governments spend relatively modest amounts on renewable energy research, compared to other government-sponsored programmes, but these government incentives are essential for continued market growth.

Renewable energy growth is a policy and regulatory success story that does not necessarily require unbundling of incumbent utilities or the presence of a retail or wholesale market structure. Policy which focuses on renewable energy has led to an increase in IPP investment. This growth has directly correlated to availability of government subsidies, such as feed-in tariffs or guaranteed long-term power purchase agreements (PPAs), not a competitive market sending price signals to encourage new renewable generation investment. If a country or region cannot liberalise its market, it can employ competitive bidding for supplying renewable generation resources under PPAs in an auction. This adds important renewable resources to the network, but there is little price competition in the wholesale market afterwards. This pattern is changing as the levelised cost of wind and solar technologies has decreased. Policy makers and regulators are encouraging private investors to take the wholesale market price risk, which may make it difficult to secure financing, as there is no guaranteed fixed-price revenue stream. Institutional investor participation in low-carbon energy projects may increase their attractiveness to a growing class of potential savers who value environmental benefits and act as a hedge against the climate change risk, such as stranded fossil fuel assets to which other investments in their portfolio may be exposed (Shishlov et al. 2016).

Regulatory structures in place today were designed for the initiation of market unbundling and liberalisation, an era characterised by centralised resources, unidirectional power flows, and inelastic price demand. The transition to an increasingly higher percentage of renewable resources in the generation stack, combined with a recent emergence of distributed energy resources (DERs), is challenging the status quo of today’s regulatory market structure. In the longer term, it is important that the measures introduced to encourage low-carbon investment are flexible enough to accommodate future market changes and new technologies. In light of the decentralisation of the power sector, regulators and policy makers must be technology agnostic and carefully reconsider how industry structures at the distribution level affect system planning, coordination, and operation as well as competition, market development, and cost efficiency.

The speed of energy technology innovation is only just coming to light as long-term historical data sets become available. The support of renewable energies through regulatory incentives over the past two decades has proven successful in reducing carbon intensity of energy portfolios. The growth of modern renewable energies, which include wind, solar, geothermal, and modern biofuel technologies, has increased global renewable electricity supply from 18% in 2000 to 26% in 2018, according to the International Energy Agency. In tandem with the positive news on renewable energy growth, the unique properties of variable generation bring new benefits to the system in addition to new challenges. There is concern that intermittent energy delivery, which contributes to increased market clearing price volatility, has created obstacles for efficient market operations. Most modern wind and solar photovoltaic plants have fast and precise control capability, typically exceeding that of most thermal generators (Milligan and Kirby 2010). These additional benefits of variable generation may not be accessible under existing market rules, but suitable adjustment of market designs could make it possible for generators and system operators to harness these benefits.

Lower wholesale prices of electricity have been routinely observed due to structural changes of supply with the low operating costs of primarily wind power. These lower wholesale prices are not translating into lower pricing for the retail end consumers. Subsidies are the most likely reason the price of retail electricity has risen, as the percentage of wind and solar generation has increased. There are two reasons for this retail price dynamic. First, there is often a gap between the feed-in tariff rate and the market-clearing price, such that the renewable generator receives a higher price than the market-clearing level. Second, electricity distributors are purchasing renewable electricity at a high price via direct long-term contracts under mandatory portfolio standard programmes. Renewable portfolio standard (RPS) programmes are largest and perhaps the most popular climate policy in the United States. Between 2009 and 2019, renewables’ share from RPS programmes increased by 4.2%, such that the energy component of unbundled retail pricing increased by 17% (Greenstone et al. 2019). These cost estimates significantly exceed the marginal operational costs of renewables and likely reflect costs that renewables impose on the generation system, including those associated with their intermittency, higher transmission costs, and any stranded asset costs assigned to ratepayers.

Interestingly, the price of coal- and natural gas-derived electricity, ceteris paribus, has risen during low-production periods for renewables. This situation has been observed in both the United States and Europe, and more recently in the UK, and appears because fossil fuel generators have seen a reduction in revenue opportunities. Bid prices to cover variable costs and an attempt to cover some portion of fixed costs have arisen to match the change in plant operating dynamics (Greenstone et al. 2019). In some countries, Germany in particular, taxes and retail costs to support renewable energy have increased substantially (Morey and Kirsch 2014).

Transforming an electricity grid to a cleaner, smarter, and more flexible system means capturing the value from resources at the distribution edge of the network. DERs are directly connected to low- or medium-voltage distribution systems. They include both distributed generation units (e.g. fuel cells, micro-turbines, and small-scale photovoltaic) and storage units (such as batteries and electric vehicles; Akorede et al. 2010). Building regulatory models to accommodate higher levels of renewable energy means DERs must be able to seamlessly integrate with utility-scale resources and wholesale markets. Policy makers and regulators face a complex challenge to price DERs (the current approach is net metering) and incorporate them into multi-party market transactions. The implications of DER ownership and aggregation in the market competition model are complex. Market mechanisms for coordinating vertically and horizontally disaggregated participants require updating and improving distribution-level price signals. Electricity tariff structures need to evolve to capture marginal costs and benefits of DER participants to the end consumers.

Energy regulators must take into consideration the fast-changing combination of different forms of energy, which are increasingly derived from low-carbon sources, on diverse scales and from distinct business models. Proper incentives via effective regulatory frameworks to measure progress towards low-carbon goals will be required, so that analysts and policy makers can identify top-performing technologies and policies. In this way, the carbon intensity of technologies can be compared and evaluated against performance targets, depending on particular global or regional concerns such as emissions. Energy regulation is entering uncharted territory with the new requirements for data privacy and security as demand management programmes, such as smart grids, are disseminated.

8 Conclusion

Technology failures, short-lived market participants, and imperfect policy choices with misalignment of regulatory instruments have complicated energy transformations throughout history. There have been successes with unbundling of energy utility monopolies, which included active investor participation, maturing of wholesale markets with a tendency to lower prices, and high level growth in renewable energy.

There remain several challenges for study and improvement. So far, benefit to the retail consumer has been elusive. Large energy consumers can access lower prices through wholesale market participation, but retail consumers’ energy bills continue to climb. While the arrival of renewable energies to the electricity grid has been lauded, system operation finds managing grid stability challenging with large swings in intermittent energy availability. Approaches to wholesale price formation and short-term system optimisation that the industry have employed for the past two decades appear unsustainable beyond the next decade. Emerging market countries continue to struggle with access to the private capital needed to succeed in market liberalisation and the ability to increase consumer access to energy services.

Regulation requires robustness to manage complications that arise from changes in market structure, such as the potential shift from the utility, as the traditional supplier of last resort model. The challenge is to establish a stable and politically sustainable regulatory toolbox that would combine the efficient benefits of competition, taking into account management of risks and the necessary sustained investment. Regulators must be actively engaged in all aspects of the market via monitoring progress to ensure regulatory design of market structure remains relevant and acts efficiently by taking a greater role in risk monitoring to avoid market manipulation.

Governments need to recognise the explosion in low-carbon energy technology development and support initiatives with targeted programmes. Should the current momentum with renewables continue, it may well be enough to establish serious in-roads with decarbonising global energy supply within the timelines of the UN sustainable development goals.