Bridging the Energy and Meteorology Information Gap

This chapter discusses the information gaps relating to the type, level of accuracy and frequency of delivery of specific weather and climate information, and what extra information is required by the energy sector in the coming years. It is argued that ongoing technical and scientific interaction between weather and climate service providers and the energy sector, supported by input from the information and communication technologies, can help bridge these gaps. This will help the users in the energy sector to both understand and respond appropriately to the available weather and climate information. Focusing on the linkages between weather-, climate- and energy-related information and data, the chapter draws attention to barriers to data sharing, benefits of overcoming the barriers and strategies to enhance data-sharing arrangements between the weather, climate and energy communities.


IntroductIon
Rise in global energy use has been modest over the past several years with growth rates of 1.1% and 1.0% in 2014 and 2015, respectively, and lower than its 10-year average of 1.9% (see International Energy Agency 2016; BP Global 2016). However, triggered by income and population growth, global energy use is expected to increase over the next several decades, particularly in emerging and developing economies. Weather and climate information is needed to efficiently plan, manage and operate energy services on very diverse space and time scales. Hence, weather and climate information is crucial, given the expected increase in global energy use in the coming decades and the interdependence of weather, climate and energy production and use.
World energy demand is estimated to increase by 48% from 2012 to 2040 (US Energy Information Administration 2016). Much of this increase in demand is expected among the developing non-OECD economies. Strong economic growth and expanding population in these economies will be the key drivers of rising energy use. Non-OECD energy demand is projected to rise by 71% from 2012 to 2040. In contrast, in the more mature energy-consuming and slower-growing OECD economies, total energy use is estimated to increase by only 18% from 2012 to 2040.
Given the estimated rise in world energy demand in the next several decades, the usefulness of weather and climate information for the energy sector will continue to be important and even increase. Hence, there is a growing demand for weather and climate information in the energy sector across many regions of the world. The practice of delivering weather and climate information requires sustained interactions with users along the energy supply chain. Both public and private sector meteorological service providers help to meet the weather and climate data requirements of the energy sector across the world.
A survey of clients of private meteorological service providers has revealed that the three main factors driving customer demand were the accuracy of their forecasts, assistance in operationalizing the forecasts and the availability of one-on-one consultation (Mandel and Noyes 2013).
The ability of weather and climate information providers to meet the growing requirements of the stakeholders along the energy supply chain has a range of gaps. For example, one of these gaps relate to the type, level of accuracy and frequency of delivery of specific weather and climate information, and what additional information is required by the energy industry in the near term and into the future. These gaps are not surprising in light of the complex interplay among the commercial, environmental, social and economic considerations involved in energy supply and the changing balance of energy technologies between fossil fuels, renewables and nuclear in different regions. Continuing technical and scientific interaction between weather and climate service providers and the energy industry (supplemented by input from other relevant sectors such as the information, communication and technologies [ICT] and the relevant regulators) can help bridge these gaps. This will enable the users in the energy sectors to both understand and respond appropriately to the available weather and climate information.
In this book, a range of gaps that hinder or slow down a more effective integration of weather and climate information in the energy sector business/decision making are covered. These gaps relate to, for example, (a) the increasing need for improvement in relevant weather forecast quality, (b) the growing demand for location specific and/or user specific and more targeted meteorological model outputs, (c) the continuing need for greater partnership between the energy and meteorological communities and (e) the emerging requirement to address data-sharing needs. In this chapter, these gaps are introduced and particular attention is focussed on data-sharing gaps in more detail.

Forecast Improvements
Meteorological forecast improvements are an ongoing issue influenced by both supply-and demand-side factors. On the supply side, the accuracy of forecasts has been improving steadily over time. It is important to recognize that forecast improvements will depend on both improved model predictions and improved forecast formulation and delivery. Continuing advancements in technology and expertise in collecting observations, processing, analysing, making model predictions, formulating forecasts and disseminating them are some of the key supply-side factors enabling forecast improvements in relevant variables. Chapters 6, 7, 8, 9 and 10 of this book highlight various state-of-the-art methodological issues associated with short range, medium and extended range forecast improvements.
On the demand side, for example, from the perspective of the energy sector, the need for better and improved meteorological forecasts can arise due to a range of reasons, for example: to better manage energy production and distribution risks associated with weather and climate variability; to efficiently operate energy markets and pricing; to undertake energy market regulatory compliance; to better plan and undertake the operation and maintenance of energy generation plants; to safeguard energy system assets from climate change impacts; and to develop risk management and adaptation planning. Several examples of these cases are covered in Chaps. 4, 10, 11 and 12 in this book.
According to the American Meteorological Society (AMS) (2015), opportunities for increasing forecast skill at all time ranges will need further research, close global cooperation and coordination, improved observations of the atmosphere, ocean and land surface, and the incorporation of these observations into numerical models. The avenues to improved model predictions include higher spatial resolution, more powerful supercomputers, wider use and improvement of model ensembles, the development of data mining and visualization methods that enable forecasters to make better use of model guidance and collaborative forecast development activities among operational forecasters and researchers.

targeted model outputs
As highlighted in other chapters (see Chaps. 4 and 6,7,8,9,10,11 and 12) in this book, location and/or user-specific higher resolution/downscaled/ targeted meteorological model outputs can help address risks to the energy sector from, for example, extreme weather events, changes in water availability, unusual seasonal temperatures and rising sea levels. Hence, targeted spatial analysis and improved forecasts of mesoscale weather events are important for both short-and long-term energy system management.
Such targeted model outputs are also relevant in the context of rising share of renewables (e.g. solar, wind and hydro power) and the dependence of these renewable energy systems on weather and climate variability. Additional weather observations at strategic locations as determined by quantitative models are important in this context.
In regions where the share of renewables in the energy mix is expanding, relevant model outputs/forecasts targeted towards specific user groups such as transmission system operators (TSOs) would benefit considerably in integrating renewable electricity to the grid as also discussed in Chap. 5. Targeted weather decision support products also help system load forecasting, enhanced efficiency in pricing for hourly and bulk markets and energy market trading in general. Examples of the development of specific user targeted online interactive tools that can allow users to assess how energy production and demand will change in response to climatic factors in certain regions are provided in Chaps. 9, 10 and 12.

enhanced partnershIps
Over the past several years, there has been an increasing incidence of research dialogues between energy and meteorological specialists, analysts and practitioners at domestic, regional and international levels. This promising development requires further enhancement in the form of regular bidirectional communications relating to issues such as improvements in current meteorological products and services used by the energy sector, new services and product requirements and generation, and skills and training needs of the users of relevant meteorological information. As reported in other chapters (see Chaps. 4, 5 and 13) of this book, specific user group oriented (e.g. wind energy or solar energy sector focussed) dialogues/workshops/meetings have enabled effective bidirectional interactions between, for example, meteorological experts and energy practitioners at technical, managerial and decision-/policy-making levels. The enhancement of the bidirectional interaction between energy and meteorology sectors will be underpinned by several factors. These include rising demand for meteorological services by the energy sector, continuing innovations in meteorological service development and provision, increasing availability of cost-effective digital technologies for service delivery, growing requests for development of codes, standards and guidance for meteorological information and emerging need to establish mutual trust among all stakeholders given that confidentiality issues can prevent energy firms from sharing specific meteorological requirements for operational practices in open (see Chaps. 4, 5 and 13).

data sharIng
The rest of this chapter expands more on data-related issues, firstly, because of their critical role and secondly, to provide specific indications about the factors associated with them. The linkages between weather, climate and energy are based on the fact that variation in weather and climatic conditions across short, medium, and long timescales can affect all energy sources and energy needs. Many public and private sector meteorological information providers fulfil the weather and climate data needs of the energy suppliers and users across many regions. In certain cases, some of these providers supply tailored weather and climate data to particular energy companies. Broadly speaking, an important challenge here is how to improve the interfacing of weather and climate data and information with key supply and demand-side activities of the energy industry. In this regard, sharing of relevant data is one area which is critical for enhancing the synergies within and between weather, climate and energy spheres. For example, improvement in data sharing within the energy industry could have benefits in terms of improving energy production/demand monitoring and forecasting. Also, enhanced sharing of weather and climate data could help minimize energy sector's vulnerability to extreme weather events, enable more cost-effective integration of renewable sources of energy and enhance energy supply and consumption strategies (American Meteorological Society 2012). Kusiak (2015) argues that lack of data sharing in the renewable energy industry, for example, is hindering technical progress and limiting opportunities for improving the efficiency of energy markets. He points out that optimizing the supply of renewable energy requires data on device performance, energy output and weather predictions, seconds to days in advance. At present, large amount of these data is gathered by participants such as turbine manufacturers, operators and utility companies along the energy supply chain.
Pfenninger (2017) argues that energy research needs to catch up with the open-software and open-data movements. He points out several reasons why energy models and data are not openly available. They include business confidentiality, concerns over the security of critical infrastructure, a desire to avoid exposure and scrutiny, worries about data being misrepresented or taken out of context and a lack of time and resources. Some competing energy companies also gather weather and climate data that they perceive as sensitive proprietary information. Hence, the amount of 'big' data within the meteorological sector and energy industry is rapidly growing. There is a growing demand in the energy sector to share its data openly so that better solutions for providing energy in a sustainable and cost-effective manner can be designed and implemented.
Often some private sector data is difficult for anyone outside to access without data-sharing agreements and non-disclosure arrangements. In this context, there are lessons to be learnt in relation to maintaining data confidentiality and security from other sectors such as commerce and health-care organizations (see Kusiak 2015). In general, non-disclosure agreements outlining the specifics of data sharing and results dissemination are used in data-intensive projects.

Barriers to Data Sharing
Meteorological and energy data-sharing arrangements are likely to be influenced by competition in service provision, commercialization of data, technical difficulties in data management, metadata problems, national security considerations and government data policies.
Better use of some data is often hindered due to data accessibility. Barriers to data sharing are generally based on the belief that it poses significant risks, from either side of the sectors. These risks may relate to identification of the businesses within datasets, misuse of the data resulting from misunderstanding of its quality or meaning, inappropriate exposure of commercially sensitive data and information and reputational damage due to release of information about the data custodians. Often lack of trust is also a key barrier to sharing data. Trust between data custodian and user or between custodians is essential in all circumstances. Many options, reflecting the nature of working relationships, can be used to build and retain trust as opportunities to access and share data expand (see Productivity Commission 2016).

Benefits of Data Sharing
Public sector meteorological service providers participate in regional, national and international data-sharing initiatives and obligations. They would not be able to perform many of their functions without the data exchange arrangements of the members of the World Meteorological Organization (WMO), which cover the public, private and research sectors of WMO member countries. The WMO provides an international framework through which its member countries coordinate the collection and exchange of information on the state of the global atmosphere, ocean and inland waters. The framework also supports the provision of essential meteorological and related services in all individual countries. The international exchange of essential data and products is free of charge under the provisions of Resolution 40 of the 12th Congress of the WMO. But Resolution 40 also places some restrictions on the commercial use of these data. Some shared meteorological data is only available under certain restrictions from the owner or provider. These restrictions may include a limited ability to use that data in situations other than those prescribed, or to charge for products/services derived from that data (see Australian Bureau of Meteorology 2016).
Increased accessibility of relevant meteorological data would likely enhance the forecast skill (and the underlying research and analysis) and facilitate integration of a wide range of energy resources. Sharing of data, while still safeguarding the commercial interests of individual dataproviding private-sector companies, has considerable potential to benefit the entire energy sector. Data sharing and supply between meteorological service providers and energy industries require that arrangements are established, covering the types of data required, their frequency of delivery, the reliability of the service and targets for quality. This will require policies to promote sharing of data relating to meteorological services and energy demand among the research, forecast and operations communities; safeguards to protect the commercial interests of private-sector companies that share proprietary data and an enhanced data collection and qualitycontrol capability for weather and climate observations (American Meteorological Society 2012).
In recent years, some public sector and some large search entities have supported a move away from releasing data under restrictive licenses to releasing data under more permissive 'Creative Commons' licenses that allow the data to be reused. Under the 'Creative Commons' licences, in general, others are allowed to use and distribute content as long as they credit the copyright holder. A 'Creative Commons' licence provides a simple standardized way for companies and institutions to share their work with others on flexible terms without infringing copyright. It allows users to reuse, remix and share the content legally. Offering one's work under a 'Creative Commons' licence does not mean giving up copyright. It means permitting users to make use of the material in various ways and under certain conditions (Productivity Commission 2016).

Enhancing the Data-Sharing Arrangements
Path to open-access data and hence data sharing could involve at least four basic elements: agreement to participation in a cooperative datasharing regime, awareness of the problems and of the potential benefits of data sharing, formulation of data-sharing protocols and governance structures and development of data-and-knowledge sharing platforms (see Kusiak 2015). Contreras and Reichman (2015) point out that to achieve widespread sharing of data, intellectual property, data privacy, national security and other legal and policy obstacles must be addressed. They observe four basic structural arrangements for scientific data pools (this may be applicable to weather, climate and energy sector-related data also) along a continuum ranging from the most to the least centralized. These include fully centralized, intermediate distributed, fully distributed and non-commons arrangements. According to them despite limited resources to link data repositories technically, there are advantages to fostering legal interoperability of data among distributed data repositories or custodians. To achieve this across different data users, rules for data access and usage must be compatible with each other, must comply with laws and regulations of relevant entities/jurisdictions and must address rights of ownership and control granted to data generators/custodians. The most straightforward path to legal interoperability is to contribute data to the public domain and waive all future rights to control it or to have data shared under standardized 'Creative Commons' licenses that have been widely used.
By using a risk-based approach to data access, custodians of data, whether public or private, could clarify and manage the nature of data risks. Risk could be assessed based on both the likelihood and probable consequence of data breaches. Where the potential implications of data breaches are non-trivial but likelihood is remote, custodians of data can still share or release, with mitigation strategies adopted as required. Also, access to the data needs to be carefully managed where the likelihood of breach and its consequence are considered high (see Productivity Commission 2016).
It is important to recognize that the private sector collects, stores and uses a vast amount of data (including weather, climate and energy sector data) and is almost certainly now the dominant controller of data in most economies. For example, retail energy utility sector generally has a small number of large firms, or even a single firm, serving a regional or national market. These utilities increasingly have the capacity to gather vast amount of detailed data on consumer energy use via sophisticated metering technologies such as 'smart meters'.
According to the Productivity Commission (2016), government intervention in support of data sharing or release may be warranted in certain circumstances such as insufficient business-to-business sharing and/or insufficient data/information released for the community benefit. Insufficient business-to-business data sharing may reflect monopoly holdings of data and misuse of market power. Having access to large quantities of data can give a company-particularly a large, vertically integrated business-a degree of market power. Such market power could be used to deter new entrants to a particular market. It is important to recognize that there are several mechanisms that allow business-to-business data sharing including bilateral commercial arrangements. It may still be the case that much wider access to some data could deliver greater public or community benefit. An important factor in considering these issues is the delivery of net benefits for the public while preserving commercial incentives to collect and add value to relevant data. Any mechanisms to increase access to privately held data would, however, need to be premised on a clear articulation of net benefits to the community and a demonstration that access to the relevant data is not able to be secured through other means including through existing private sector data marketplaces and platforms (see Productivity Commission 2016).
The European Union has mandated open access to electricity-market data, resulting in the creation of the ENTSO-E Transparency Platform to hold it. This highlights the fact that there are valid arguments for the creation of national energy-data agencies to coordinate the collection and archiving of a range of important data (see Pfenninger 2017).
Knowing the value of meteorological or energy data being shared is a key factor which can help assess the potential returns from adding value to such data by, for example, tailoring them for specific uses.
There are various approaches for valuing or pricing meteorological (see World Meteorological Organization 2015) or energy data. They range from free provision and marginal cost pricing to commercial pricing. The preferred approach will depend on user demand (willingness to pay for specific data) and the capability of the data supplier to act commercially. Most public sector meteorological data are currently available on an open access basis to enable full and free reuse. This can occur, for example, under an open access licence (such as Creative Commons Licence).
Where data is shared with other parties, the value placed on it will be determined by several factors: the availability of alternatives, the need for further processing for use, potential uses to which the data can be put and strategic leverage attached to the data (see Productivity Commission 2016).
reFerences American Meteorological Society. (2012). The energy sector and earth observations, sciences and services. A policy statement of the American Meteorological Society. Adopted by the American Meteorological Society Council, 20 September. Retrieved from https://www.ametsoc.org/ams/index.cfm/aboutams/ams-statements/statements-of-the-ams-in-force/the-energy-sector-and-earthobservations-sciences-and-services/ Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated. The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.