Introduction

Abstract

This chapter introduces the context and motivation for this book as well as some of the best ways to use it.

This chapter introduces the context and motivation for this book as well as some of the best ways to use it. It focuses on:

• Why is forecasting needed to help support the future energy system, especially at the low voltage level.

• Why is this book needed.

• Description of the contents and how to read it.

• What to include in a semester long course.

1.1 Motivation

Mitigation of the climate crisis by meeting global carbon reduction targets is going to require dramatic changes to how we generate and use energy. Global leaders and the energy sector must act with urgency if we are to avoid the most catastrophic outcomes of climate change.

Fig. 1.1

Plot by Our World in Data licensed under CC BY 4.0

Proportion of electricity produced per year broken down by source.

Carbon reduction will mean a continuous shift towards the decarbonisation of major sectors and infrastructures, especially in heating, industry and transport. Many of these applications will require electrification with, for example, heating moving from gas boilers to electric heat pumps, and transport moving from petrol combustion engines to electric vehicles.¹

This electrification is increasingly generated from renewable energy sources, such as wind, solar, wave, hydro and geothermal (See Fig. 1.1). The change in how energy is used and generated has two main consequences. First, larger and more volatile demand consumption behaviour due to new low carbon appliances (electric vehicle charging at home), and second, an electricity supply which has much more uncertainty due to the dependence on less predictable and intermittent weather behaviour.

These changes will effect the security of the electricity supply since it increases the chances the energy generated may not match the demand of the end consumers, which in turn could result in blackouts or damage to the electricity networks. These are going to present significant challenges for governments to “ensure access to affordable, reliable, sustainable and modern energy for all” which is one of the 17 Sustainable Development Goals established by the United Nations General Assembly in 2015.² To mitigate the effects of electrification several solutions are being developed including:

• 1. Control of Storage: storage devices can “shift” and “smooth” demand by charging during times of low demand, and discharging during periods of high demand. This can help reduce the effects of increasing demand but can also ensure optimal utilisation of renewable energy at time periods when they are needed most. Most renewable generation is dependent on weather conditions which means energy from renewable sources may be generated when it is least needed. Storing this generation can ensure that clean energy is used to meet the high demand instead of energy with higher carbon intensity from the grid.

• 2. Demand Side Response: If there is insufficient generation to match the demand, devices can be turned off to balance the network. For example, heat pumps could be turned off for a short period, reducing demand whilst ensuring heating comfort is retained. Similarly if there is too much generation demand can be turned on.

• 3. Co-ordinated EV Charging: If Electric Vehicle (EV) uptake is sufficiently high then there is a likelihood that local networks will be under excessive strain since households will choose to charge at similar periods (for example, plugging them in after arriving home from work, or charging over night so the vehicle is ready for use in the morning). One way to alleviate this effect is to co-ordinate their charging so that fewer vehicles are drawing energy from the grid simultaneously.

• 4. Local Electricity Markets: With more and more distributed generation, and an increase in controllable assets with two-way communications it creates opportunities for smart grids and localised energy markets where demand and generation can be traded. Energy can be utilised close to where it is generated and this also reduces the losses through transmitting energy over large distances.

All of these applications require varying degrees of foresight of the demand or generation in order to operate optimally and ensure minimal disruption and costs to consumers. For example, consider a storage device with the objective to maximally charge using energy generated from renewables, and to use the stored energy to reduce the peak energy usage. This requires accurate estimates of both the future demand and generation to schedule the appropriate charging and discharging of the device. These estimates are produced by so-called load forecasts, the topic of this book!

1.2 Demand Forecasting for LV Systems

As described above accurate forecasting is a vital tool for a whole range of applications including optimising energy management systems (such as storage devices), redistributing demand, peak demand reduction and electrical infrastructure development (i.e. determining where and what assets to install). Unfortunately, load forecasting at the LV level is a complex task compared to forecasting at the national level or system level. The small number of consumers connected to the LV substations mean that the demand is relatively irregular and volatile. Feeders on low voltage substations often consist of between 1 and 100 consumers and they may be residential, commercial, or a mix of both. Further to this there is street furniture such as lighting which can have a non-trivial effect on the overall demand at the LV level.

Fig. 1.2

Constructed using data from the CER Smart Metering Project—Electricity Customer Behaviour Trial, 2009–2010 [1]

Examples of half hourly demand profiles for a week for a a single household, b the aggregation of 40 households, c the aggregation of 540 households.

This volatility is apparent when comparing the half hourly demand over a week for different aggregations of households. This is shown in Fig. 1.2 for a single consumer, an aggregation of 40 consumers (the size of a typical LV secondary feeder) and finally, 540 consumers. At the highest aggregation the profiles are relatively smooth with the demand very similar from one day to the next. In contrast the single household is very volatile and irregular from day to day. LV feeders will typically be connected to around 40 consumers, and it is clear that although they are more regular than a single household, they still have significant levels of volatility and uncertainty. This book will present a case study in Chap. 14 comparing forecasts on feeders of a range of sizes, showing how aggregation links to accuracy. This variety in demand behaviour also means there is no one-size-fits-all forecast model which will forecast accurately across all LV substations, unlike systems-level or national-level forecasting.

In this book, forecast models will be used to predict the demand as both point estimates and probabilistic estimates. The latter is essential to deal with the uncertainty inherent in LV level demand and ensure the optimal performance is obtained for the applications mentioned above and in this book.

1.3 Why Do We Need This Book

As outlined above, forecasting is an essential requirement to support the changing dynamics of the electricity network, and the emerging applications and opportunities. However, the unique challenges associated to low voltage demand, in particular, the increased volatility and irregularity, will require knowledge of advanced techniques and models which are not necessarily required for the more regular demand at the system or national level.

Unfortunately knowledge and experience in these areas are relatively sparse due to the lack of available data, and the recency of the area itself. Monitoring of demand is generally reserved for higher voltage levels of the network and smart meters have only been rolled out from the 2010s onwards. There is therefore a urgent requirement for these indispensable skills to enable the growing needs of a low carbon economy.

Forecasts are the fundamental component to a vast array of applications in the smart grid and low voltage level. This book will therefore enable researchers in a vast array of fields from optimisation, control theory, power systems engineering, and uncertainty quantification. Due to the array of techniques presented, it will also be useful as a reference to more experienced forecasters who may wish to utilise further approaches and models.

Several applications are illustrated in both the case study (Chap. 14) and the further examples (Chap. 15). These are based upon the authors’ research experience using real world data to demonstrated the techniques presented. The case study in particular shows all the difficulties that can be found with processing, analysing and interpreting of real measured data, and utilising them within state-of-the-art forecast models. This is in contrast to other books which often use toy examples, which although illustrative, perhaps do not demonstrate the sometimes subtle complications which are prevalent in real systems and data.

1.4 Aims and Overview of This Book

This book aims to present a comprehensive guide to all the components necessary to develop accurate and effective load forecasts for low voltage electricity networks. The behaviour of electricity demand has changed with new energy sources such as energy storage and renewable energies sources which aim to supply the increase in energy demand. New energy control technologies have been shown to reduce energy costs, emissions, and peak demand, and load forecasting characteristics provide the opportunity to investigate the benefits of optimal control strategies. While not providing exhaustive details on all approaches, this book’s aim is to provide a unique overview of the most commonly applied techniques and methods for load forecasting at the LV level. It covers the basic time series forecasting concepts and introduces both, common statistical and machine learning methods. The reader is referred to more detailed literature on each of the methods in the Appendix.

Before developing precise forecast techniques and models, a practitioner will require understanding fundamental concepts in energy systems, statistics and machine learning. For these reasons, Chaps. 2, 3 and 4 serve as knowledge foundation for the rest of the book and any topic which requires more information will be elaborated on in later chapters. Chapter 2 briefly describes the important features of a electricity distribution network, including those attributes which make LV level forecasting much more challenging than system level. Chapter 3 introduces core concepts in probability and statistics. Many of the models and theory for time series forecasting are framed within a statistical framework, in particular when the variables’ underlying uncertainty needs to be utilised, this is especially true when implementing probabilistic forecasts for probabilistic forecasting (Chap. 11). The final primer is Chap. 4 and describes the main definitions and concepts in machine learning, of which predictive analytics is a major subset. Many chapters in this book are on different areas of machine learning and build on this primer.

Chapter 5 focuses on the core definitions, descriptions and concepts of time series, and time series forecasting. From this chapter the reader should be able to understand the required components in order to frame a well-defined time series forecast problem. Note that while we motivate this book for load forecasting, the concepts presented are not necessarily limited to be applied in forecasting electric load. Most of the concepts introduced are applicable also in other similar time series forecasting tasks.

The next three chapters develop the tools and techniques required to develop these forecast models. This begins with Chap. 6 which shows how to analyse and understand the relationships within and related to the load data time series. This includes preparation techniques, such as identifying anomalous values, and feature engineering techniques which can help you understand what variables are the main drivers of the load. These relationships will determine the types of models the forecaster will eventually choose as well as the main input variables.

An accurate forecast model is not possible without having a proper evaluation method. This is the topic of Chap. 7 which introduces a whole host of error measures and skill scores for both point and probabilistic forecast models. It also introduces some simple checks which can be performed to help improve the accuracy of the model. The next chapter (Chap. 8) describes how to appropriately train and select forecast models. Here one of the most important concepts in machine learning is introduced, namely “bias-variance trade-off”. This is vital to ensure models can generalise to new, unseen data and hence capture the true behavioural patterns in the load time series. The chapter considers a number of techniques, in particular regularisation, which helps enable this generalisation.

With the main tools and techniques introduced, the next three chapters describe a plethora of forecasting methodologies which have been developed for producing successful point and probabilistic load forecasts. Most of these models utilise the patterns found in the historical demand data since most LV demand has regular features in past behaviour. Chapters 9 and 10 focus on point forecasts using statistical and machine learning based models respectively. These separate categories have different advantages and disadvantages but are both useful to enable methods which have the full potential of descriptive power and high performance. Chapter 11 introduces a range of probabilistic forecasts. These are essential for many distribution level applications because of the relatively high uncertainty in the demand behaviour. In LV systems in particular, the demand is typically quite volatile, hence it is likely that probabilistic methods are going to be increasingly required for future smart grid applications. Since probabilistic forecasts can be defined in several ways, this chapter gives many different methods for doing this. By the end of the three main model chapters, the reader should have a solid understanding of the best methods in time series forecasting, and when to apply them. An overview of the models explored in this book are given in Fig. 1.3. The whole forecasting process, from data analysis to evaluation, as well as tips on how to choose the most appropriate model, are described in Chap. 12.

Fig. 1.3

Some of the main models explored in this book for both point and probabilistic forecasts. Also represented are additional techniques which are applied in LV load forecasting

Time series forecasting is a vibrant area of development and research. Many of these techniques are likely to be applied more and more in real world applications. For this reason, some of the more advanced topics are described (briefly) in Chap. 13. This includes improving accuracy through combining models, and how to test whether two forecasts are statistically different in accuracy from one another.

Given the model, techniques and tools, the final two chapters of this book are devoted to applying these techniques in forecast experiments and real LV level applications. Chapter 14 gives a full forecasting case study applied to the demand data of 100 real-world low voltage level feeders. In this chapter, it is demonstrated how to analyse the demand data, develop several forecast models and test them. These forecasts illustrate some of the challenges and difficulties that come with real demand data. Chapter 15 then considers a number of applications where these forecasts can be applied, most prominently featured is a battery storage scheduling problem.

It is worth noting, that this book also serves as a relatively thorough introduction to data science in general, albeit focusing on a specific domain. It includes all the features and good practice in machine learning, or more generally artificial intelligence, such as cross-validation, model evaluation, benchmarking etc. as well as a whole host of different techniques including decision trees, neural networks, linear regression and generalised additive models. Hence by studying this book you can become highly knowledgeable in generating machine learning models, in particular for time-series forecasting based applications.

The primary use case of this book is as an introduction to the short term load forecasting for undergraduate or graduate students. However, this book can also serve as an introduction for interested readers from industry, such as data scientists, statisticians or analysts working in utilities, system operators, or other companies in the energy industry. The main case study presented in this book comes from peer-reviewed papers published by the authors, and highlight the heavily applied focus of this topic, hence we hope this book can be a useful reference to those interested in applying the methods in real-world applications.

1.5 How to Read This Book

The best way to read this book mainly depends on the experience of the reader. If you are a statistician, data scientist or probabilist with experience in modelling, then the reader may want to focus on the applications and the case study and only refer back to the technical chapters when needed. In contrast, if the reader is an engineer or an expert in energy systems, but with minimal knowledge of data science then the focus should be on the earlier chapters, specifically the Chaps. 3–7. In either case, the reader should familiarise themselves with the notation and can skim through parts of these chapters based on their background. Figure 1.4 shows the links and dependencies between the chapters.

The models described in the three main Chaps. 9–11 focus on point and probabilistic forecast models. Those with a statistics background will feel most comfortable with Chap. 9 which looks at many traditional time series forecast models such as linear regression and ARIMA, but also has more recent popular techniques such as generalised additive models. These are highly interpretable models and can be useful when trying to evaluate the performance of the forecast.

Those with a computer science or more traditional machine learning background will better understand the techniques in Chap. 10 which considers many methods such as deep learning, random forest and support vector regression. These models are not so interpretable but are often high performing. The more complicated and computational intensive probabilistic forecasts are introduced in Chap. 11 and include a mix of both more statistical and machine learning models. The statistical primer (Chap. 3) and the probabilistic error measures section of Chap. 7 are essential for this topic.

There are a lot of methods in Chaps. 9–11. The reader of course could read the entire chapter and get a solid overview of the wide variety of methods available for load forecasting. However, the reader may also wish to focus on point forecasts (traditional or more modern machine learning) or probabilistic, or may wish to mix-and-match a few methods from each. For a semester course then the choice of methods would ideally be based on those which are utilised in the case study from Chap. 14. This ensures a full forecast experiment can be demonstrated. The next section below outlines one possible delivery of a single semester course.

Machine learning can only be grasped by doing. For this reason each chapter finishes with a few questions on the contents and in the appendix there is also a full walkthrough which can guide the reader through the entire process from data collection, data cleaning, model selection and evaluation. Also linked is a notebook (https://github.com/low-voltage-loadforecasting/book-case-study) which demonstrates some of the techniques applied within a python environment. This will hopefully illustrate how to apply the analysis and modelling described in this book and is outlined in the Case Study, Sect. 14.3.

Fig. 1.4

Figure illustrates direct dependencies between the various chapters in the book. Note that Chap. 12 summarises the load forecasting process and therefore integrates most of the method and technical sections

1.6 Note for a Semester Delivery Course

There are a few prerequisites for learning the material presented in this book. Students should be familiar with basic calculus, and properties of basic statistic and probabilities. It is not recommended to cover the whole book in one semester course unless students already have knowledge of probability theory and the main concepts in machine learning.

This book introduces one full case study in Chap. 14 on point and probabilistic forecasts for low voltage residential feeders. For this reason, the most suitable focus for a semester course will be to introduce the main concepts and models which can demonstrate the case study. To gain hands on experience, students should develop and test their own models and for these reason lists of open data are given in the appendix. To assist with this, the authors have developed a python notebook which steps through some of the concepts and is available at https://github.com/low-voltage-loadforecasting/book-case-study. The context is described in Sect. 14.3 but the reader should try and develop their own models in addition to those illustrated in the notebook.

To cover the case study, a semester course will require covering the main concepts in data analysis, model selection and evaluation, but will only require a selection of models from Chaps. 9 and 11. In other words the semester will only focus on some statistical and probabilistic forecast models. The following sections would therefore make for a coherent introductory course:

 

Chapter 2::

The main LV network context and motivation for the case study is presented in this chapter. If these concepts are not taught in other courses then a brief overview of this chapter should be included. At the very least the section on “features of distribution networks”.

Chapter 3::

This is a primer on essential statistical and probabilistic principles and it is recommended that it is presented in full if not covered in other modules.

Chapter 4::

This has useful core concepts for machine learning and the sections on supervised learning and optimisation are useful if there is available time.

Chapter 5::

This section introduces the main definitions and descriptions of time series forecast and hence is necessary to present in full. It also introduces the notation throughout the rest of the book.

Chapter 6::

This chapter should be taught in full as it introduces the main data analysis techniques used in the case study. A reduced version of this chapter can be taught if there is other modules which teach time series data analysis.

Chapter 7::

The point error measures should be taught, as should the CRPS from the probabilistic error measures. The residual checks section should also be included if time.

Chapter 8::

Section 8.1 on general principles should be taught in full but only the sections on least squares and information criteria are required from Sect. 8.2.

Chapter 9::

Only particular methods are required to be covered in this section, this includes the benchmarks (Sect. 9.1), exponential smoothing (Sect. 9.2), and multiple linear regression (Sect. 9.3).

Chapter 11::

From this chapter teach Sect. 11.4 on quantile regression and the section on ensemble methods (Sect. 11.6).

Chapter 14::

The case study should be worked through in full, but can be split into the different components and introduced with the relevant parts from each of the previous chapters.

Assessment:

There are some questions at the end of each chapter. These can be worked through to support the teaching of the material. There is also a guided walkthrough in the Appendix C, Instead of presenting the case study this could be worked through or set as a coursework challenge. A list of open data is also given in Appendix D.4 which students could use to develop and test their own methods.