Keywords

4.1 Introduction

In the coming years, digitalisation is set to revolutionise energy infrastructure (Kang et al., 2023). Broadly, digitalisation denotes the increasing integration of information and communication technologies (ICTs) across various sectors of the economy. This transformation is driven by advancements in data processing and analytics, Machine Learning (ML), and Artificial Intelligence (AI). Central to this transformation is the confluence of data, AI/ML, and the Internet of Things (IoT). The affordability of sensors, coupled with expanded data storage capabilities, has spurred rapid advancements in analytical techniques to better forecast energy demand as well as predict outages (Potdar et al., 2018). The smart grid represents a transformation in power system operations, driven by integration of renewable energy, deployment of advanced sensors and communication systems, active consumer participation, and increased digitalisation (Dileep, 2020). However, conventional optimisation and control techniques struggle to manage the complexity, dynamics, and uncertainty inherent in modern smart grid operations. In fact, traditional model-based methods rely on accurate system models and knowledge of parameters, which are challenging in complex, stochastic environments (Glavic, 2019). This has motivated growing interest in AI and ML techniques for smart grid applications.

Historically, the energy sector has been a pioneer in adopting technological innovations. For instance, during the 1970s, power utilities were early adopters of technologies that bolstered grid management (Gross et al., 2018). Similarly, oil and gas companies have consistently integrated innovative digital tools to simulate exploration assets and curtail maintenance costs. The energy sector’s adaptability and forward-thinking approach have positioned it to harness the full potential of digital advancements. A significant portion of the potential for digitalisation in the energy sector stems from its capacity to synchronise energy demand and supply more effectively (Baidya, 2021). The real-time data relay capabilities of the IoT can substantially minimise energy wastage, thereby curtailing carbon emissions and helping to mitigate climate change.

This chapter examines applications of deep learning (DL) and reinforcement learning (RL) across major smart grid operations (domains including optimal dispatch, electricity markets, and emerging areas like cybersecurity and privacy). For each area, key papers are analysed to provide an overview of implementations, results and limitations. Challenges and future directions are also discussed. The review illustrates that while DL shows immense potential, further research is needed to address issues like cybersecurity, scalability, and stability before large-scale deployment. Overall, DL models represent an important innovation for realising the vision of efficient, reliable, and resilient smart grid operations. The remainder of this chapter is structured as follows: Sect. 4.2 provides a brief description of the smart grid and DL and RL; Sect. 4.3 provides a description of DL applied to the batteries and the smart vehicle grid; Sect. 4.4 examines DL and RL in the context of cybersecurity while Sect. 4.5 provides some concluding remarks.

4.2 The Smart Grid and Deep Learning

The smart grid represents a significant advancement in contemporary energy management. The integration of affordable sensors and monitoring devices has significantly improved the grid's ability to monitor and adjust processes. This gives operators the tools to analyse and leverage data from sensors throughout the grid. As such, the smart grid is able to minimise losses during energy transmission and distribution, thereby improving resource utilisation and overall system efficiency (Wang et al., 2023). The smart grid also improves grid reliability. In real-time, it can respond to disruptions and outages. This is particularly important due to the increased use of renewable energy sources such as solar and wind (Wang et al., 2023). It effectively manages the intermittent nature of these resources, balancing supply and demand, storing surplus energy, and ensuring grid stability. This is instrumental in achieving a cleaner and more sustainable energy future. However, conventional modelling, optimisation, and control techniques encounter substantial challenges in managing the massive amount of data that comes from the smart grid. As such, AI and ML have emerged as crucial components in advancing the smart grid (Massaoudi et al., 2021). AI in the energy space primarily refers to the creation of algorithms capable of performing tasks that traditionally demanded human intelligence, such as real-time monitoring, fault detection, and load forecasting (Cheng & Tao, 2019). ML, a subset of AI, empowers machines to learn from data and adapt without explicit programming, making it particularly valuable for the smart grid. By processing vast amounts of data from various sensors and sources, these models can optimise the grid's operation, reducing transmission losses and improving resource allocation. Additionally, they facilitate real-time monitoring, enabling rapid detection and response to grid disruptions, ultimately minimising downtime, and ensuring uninterrupted power supply.

Two particularly important types of ML have emerged as useful for the smart grid: DL and RL (Zhang et al., 2018). Both fall under the broader category of ML and came from the development of multi-layer neural networks. While DL can encompass a broader range of applications, the term is commonly associated with neural networks with a large number of layers. In RL, the core elements consist of an individual, an overall environment, rewards or pay-outs, and actions. The goal within RL is to optimise the accumulated rewards through a sequence of actions depending upon how the environment changes. Both types of learning have been studied in the academic literature for a while but have only recently been applied to energy sector. Deep Reinforcement Learning (DRL) combines DL and RL, leveraging neural networks for perception and RL for sequential decision-making (Arulkumaran et al., 2017). This enables DRL agents to learn control policies directly from data through interactions with the smart grid, without requiring an explicit system model (Cao et al., 2020).

Models, such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks, excel at forecasting tasks due to their ability to process sequential data and learn from it. In integrated energy systems, accurate demand forecasting is crucial. The fluctuating nature of renewable energy sources like solar and wind presents a significant challenge to their integration into the energy mix. DL can mitigate this issue by analysing consumption trends and predicting generation patterns by combining different data sources (e.g., weather forecasts and commodity prices), thus enabling grid operators to balance intermittent renewable resources with natural gas, coal, and other hydrocarbons (Wang et al., 2019). This balance is critical for maintaining grid stability and ensuring a constant energy supply. Additionally, DL can optimise energy storage systems, deciding when to store excess energy and when to release it back into the grid, based on predictive models that take into account future energy generation and consumption. Yang et al. (2021) used an improved Deep Deterministic Policy Gradient (DDPG) framework for lowering operation costs. Zhou et al. (2020) introduced DRL strategy for the economic dispatch of combined heat and power. The improved DRL algorithm (i.e., distributed proximal policy optimisation or DPPO) demonstrated better performance in handling a variety of operating situations compared to conventional methods, all while providing real-time optimal control strategies.

DRL also offers opportunities for enhancing demand-side management by providing systems that can learn and adapt to dynamic energy consumption patterns (Lissa et al., 2021). In demand-side management, DRL agents are trained to optimise energy usage within a grid or a local system by considering real-time variables such as current demand, pricing, and the availability of renewable energy sources. The technology can manage the operation of interconnected devices and systems, from residential HVAC (heating, ventilation, and air conditioning) units to industrial machinery, adjusting their operation to align with changes in the price as well as changes in the source of the energy being used. This capability ensures that energy consumption is not only more economical but also more responsive to the intermittent nature of renewable energy sources. As such, emissions may fall if individuals or manufacturers can adjust production depending on the type of electricity used. For example, Zhong et al. (2021) applied DRL to dynamically optimise incentives for electric heating integration and found cost savings for users and companies, increased wind power consumption, and a more intelligent system for regenerative electric heating by considering user behaviour and differences.

Maintaining the equilibrium of electricity supply and demand is a pivotal role of automatic generation control (AGC), which modulates the power output from various generators. The synergy of ML and AI with such devices equips AGC systems with the foresight and agility to more effectively fine-tune the interplay between generation and demand, thereby bolstering the grid's flexibility and operational efficiency. DRL has been leveraged for AGC to enhance the tracking of unpredictable renewable energy sources and to augment system adaptability (Vijayshankar et al., 2021). Li et al. (2020) employed hierarchical multi-agent DRL to showcase its capability to adjust to fluctuating scenarios and to perform economic optimisations. Despite these advancements, challenges like system instability, hyperparameter sensitivity, and the complexity of sample handling persist.

4.3 Deep Learning, Batteries, and Stabilising the Smart Grid

Batteries will play an integral role in smart grid stability and emissions reductions. Batteries are not merely storage devices; they are the cornerstone of a sustainable, efficient, and reliable power system (Chang et al., 2018). Their role in facilitating the transition to a low-carbon future is becoming increasingly apparent, marking them as indispensable tools in achieving global environmental goals. The future of energy is inextricably linked to the advancement of battery technology, heralding a new era of greener power and more sustainable living. Over the past decade, the cost of lithium-ion batteries dramatically fell which significantly changed the economics of energy storage and electric vehicles (EVs). Since 1991, the price of lithium-ion batteries has dropped by approximately 97% (Ziegler & Trancik, 2021). This steep decrease is largely attributed to improved manufacturing processes, larger production facilities, and advancements in the chemistry and design of the batteries themselves, which have increased energy density and prolonged lifespan (Ziegler & Trancik, 2021). Their ability to store surplus energy from renewable sources like wind and solar is invaluable in mitigating the inherent intermittency of renewables. Moreover, DL and RL learning ensures a more consistent and reliable power supply, crucial for maintaining grid stability. By storing energy during periods of low demand and releasing it during peak consumption times, batteries effectively manage load balancing (Muralitharan et al., 2016). This process, known as peak shaving, reduces the burden on the grid and lessens the dependency on carbon-intensive, peaking power plants, which are typically activated during high demand periods. Moreover, the integration of batteries into smart grids leads to more efficient grid operations. Modern smart grids, equipped with advanced battery storage systems, optimise the use of renewable resources and minimise dependence on outdated, less efficient power generation facilities. Beyond grid stabilisation, batteries are instrumental in the broader context of emission reduction. They optimise power plant operations by reducing the need for plants to run in less efficient, more emissive standby modes. Batteries enable power plants to operate more steadily and efficiently, thus diminishing greenhouse gas (GHG) emissions (Jafari et al., 2022). Batteries also support the growth of Distributed Energy Resources (DERs), such as residential solar panels. By storing energy generated locally, these batteries reduce transmission losses and reliance on centralised power generation, which is often more carbon-intensive. This localised energy production and storage model enhances the efficiency of the power system and contributes to emission reduction.

In the transportation sector, batteries are key to the electrification of vehicles, which is a major avenue for cutting down emissions. EVs not only contribute to cleaner air but may also serve as dynamic energy storage units that can supply power back to the grid when needed. This Vehicle-to-Grid (V2G) capability allows EVs to act as mobile energy reservoirs, further stabilising the grid and promoting the use of renewable energy (Theissler et al., 2021). DL algorithms are also transformative for the energy sector in the realm of EVs, especially for battery management and monitoring. By processing historical battery performance data, DL models can detect signs of battery degradation, thus enabling pre-emptive maintenance actions to be scheduled (Theissler et al., 2021). This pre-emptive model helps in devising intelligent battery management systems; these then dynamically change charging protocols to safeguard battery health while concurrently meeting the energy demands of EV owners. DL contributes to the enhancement of state-of-charge and state-of-health estimation models (Tian et al., 2021). These models are great at forecasting the dependable range of EVs and are instrumental in extending the overall lifespan of the battery. The accuracy of these predictive models is critical, as they directly influence the trust that users place in the EV's operational reliability. Moreover, DL models can integrate environmental variables, such as temperature fluctuations, to refine the battery management process. In fact, temperature is a salient factor that significantly impacts battery performance, efficiency, and safety. Extreme cold can hinder battery chemical reactions, leading to reduced range and slower charging rates, while excessive heat can accelerate battery degradation and pose safety risks (Jaguemont et al., 2016). Thus, DL models can anticipate and adjust to temperature-related battery performance variations, thereby optimising charging strategies and operational guidance according to real-time and forecasted weather conditions (Koohfar et al., 2023). RL presents opportunities to incentivise owners to properly maintain their vehicles. RL agents can be trained to maximise long-term rewards like improved safety and reliability. Owners can receive cost savings or other benefits for proactively maintaining their vehicle based on diagnostic alerts. This positive feedback loop ensures owners prioritise maintenance, vehicles operate optimally, and costs are reduced for manufacturers who avoid warranty claims. RL models may also get smarter by incorporating maintenance data, refining alert triggers and personalised incentives to shape driver behaviour.

City planners can estimate how increased EV adoption will strain the electrical grid under different charging behaviours (Deb, 2021). Utilities can identify locations likely to require grid upgrades to meet new EV load. With computational scenario modelling, DL provides the necessary intelligence to scale infrastructure appropriately. It also aids macro-level energy management and renewable integration by revealing charging patterns. Intelligently expanding charging infrastructure relies heavily on DL (Tuchnitz et al., 2021). High-dimensional spatial datasets describing vehicle populations, existing stations, power grid capacity, and land use can be utilised to determine ideal new charging locations. DL algorithms can pinpoint placement that maximises accessibility and utilisation based on current EV owner charging habits derived from surveys and public data. Compatible sites can be proposed at parking garages, retail centres, and other high-traffic locations where drivers tend to stop for 20 minutes or longer. DL may ultimately provide a way to implement a data-driven approach for strategic infrastructure growth, ensuring charger availability keeps pace with EV adoption. DL also presents ample opportunities to enhance electric vehicle (EV) infrastructure through data-driven modelling and optimisation (Deb, 2021). A key application is creating accurate models of EV energy consumption based on driving conditions. Again, by analysing historical data, DL algorithms can learn to predict future energy needs during a planned trip based on inputs like road type and condition, traffic patterns, driving style, and weather. Models can be personalised by learning from an individual driver’s past trips to account for variations in acceleration, braking, and speed. With granular energy consumption forecasts, DL provides a major improvement over simplistic range estimation that relies on battery size alone allowing EV drivers to better (and more accurately) plan routes and charging stops.

Finally, DL enables robust V2G systems whereby EVs bi-directionally transmit power between their batteries and the grid (Vadi et al., 2019). DL optimises the timing and volume of energy flow in either direction. By analysing usage patterns, a DL model can predict upcoming charging demand during peak times. EVs can then be incentivised to delay charging by a few hours to ease grid strain, or discharge energy back to the grid if requested. Meanwhile, during periods of excess renewable generation, EVs can absorb surplus clean energy to charge batteries. This avoids curtailing sustainable power and uses EVs as dynamic storage assets. DL can combine historical data with real-time grid and vehicle signals to orchestrate V2G energy transfer. This balancing act reduces grid volatility introduced by variable renewable sources, benefiting all ratepayers. It also compensates EV owners for energy services that support the overall system. An RL agent can monitor factors like electricity prices, renewable energy availability, and individual user patterns to determine optimal charging. By receiving feedback on outcomes like minimising costs and maximising battery lifespan, the system learns when and how much to charge each vehicle. This personalised charging ensures efficient energy use while satisfying individual mobility needs. Additionally, RL enables intelligent demand response systems, where EVs interact with the grid to balance supply and demand. The RL agent learns strategies for charging or discharging vehicles in response to real-time grid conditions. For instance, EVs can soak up excess renewable energy during sunny middays when solar production peaks. Later in the evening when electricity demand spikes, those same vehicles can discharge power back to relieve grid strain. By optimising bi-directional energy flow, RL helps stabilise an electrical grid incorporating more variable wind and solar generation while compensating EV owners. At a broader level, RL can optimise traffic signals in real-time to improve EV efficiency and reduce emissions. An RL agent controlling traffic lights learns adaptive signalling strategies based on traffic conditions. This dynamic approach reduces congestion and keeps vehicles moving at steadier speeds compared to fixed timing plans. Maintaining consistent speed enhances an EV’s energy efficiency, as frequent starts and stops drain more battery charge. Smoother traffic flow also diminishes brake wear and emissions. Additionally, optimising traffic flow allows existing charging infrastructure to support more EVs.

Battery technologies, pivotal in enhancing grid stability and powering EVs, offer notable environmental benefits but also face certain challenges. On the upside, they enable the integration of intermittent renewable energy sources into the grid, facilitating a stable, continuous energy supply and thus reducing reliance on fossil fuels. This integration is instrumental in lowering GHG emissions, both in the energy sector and in transportation, as EVs replace traditional, emission-heavy vehicles. Batteries also promote energy efficiency by allowing for energy storage during low-demand periods and usage during peak times, which diminishes the need for carbon-intensive peaking power plants. However, these advantages come with challenges, including the environmental impact of battery production and disposal, which involves resource-intensive processes and potential issues with recycling and waste management. There is also the concern of sourcing raw materials, often linked to ecological and human rights issues. Moreover, the lifespan and energy density of batteries are areas requiring ongoing technological advancements to ensure long-term sustainability and practicality. Therefore, while battery technologies are central to a more sustainable future in grid management and transportation, addressing these production, disposal, and material sourcing challenges is essential for maximising their environmental benefits.

4.4 Cybersecurity and the Smart Grid

As we move towards smart grids, the critical issue of cybersecurity emerges prominently. Cybersecurity is crucial for ensuring the environmental sustainability of our energy systems, as threats can significantly hinder the adoption and efficiency of smart grids. This indirectly impacts our ability to integrate renewable resources and reduce emissions. A notable example is the Russian cyberattack on Ukraine’s electricity grid, which illustrates the potential for widespread disruption in critical energy infrastructure.Footnote 1 The integration of renewable resources and the proliferation of IoT devices into the smart grid have significantly enhanced the efficiency and reliability of energy distribution and consumption, but they also introduce complex cybersecurity challenges (Kimani et al., 2019; Gunduz et al., 2020). The threats range from data breaches and privacy violations to coordinated attacks on energy infrastructure, potentially causing widespread disruptions. For example, the Colonial Pipeline hack, which occurred in May 2021, was a significant cyberattack that targeted one of the largest pipeline operators in the United States (Hobbs, 2021; Tsvetanov & Slaria, 2021). The pipeline carries gasoline, diesel, and jet fuel along a 5,500-mile route from the Gulf Coast to the New York metropolitan area. The perpetrators deployed ransomware that successfully infiltrated and encrypted the pipeline's computer systems (Dudley & Golden, 2021). This did not just threaten data integrity; it held the company's operational capability at ransom, demanding a significant payment in cryptocurrency to provide the decryption key necessary for recovery. Colonial Pipeline took decisive action to halt all pipeline operations, triggering a supply shock across the Eastern United States and leading to fuel shortages, panic buying, and heightened public anxiety about energy security. The US Government declared a state of emergency to ensure the continuation of fuel deliveries. The incident inflicted significant economic damage and underscored the urgent necessity for more robust cybersecurity defences and strategies tailored to the unique challenges of the energy sector.

DL and RL have become imperative for enhancing cybersecurity in this context. DL models are well-equipped to identify complex patterns that could signify cybersecurity threats (Dixit & Silakari, 2021). In fact, DL algorithms can process and analyse the data points generated by smart grids and identify potential attacks before they manage to breach the system. For example, Convolutional Neural Networks (CNNs) can be trained on network data to recognise the signatures of malware or intrusion attempts, while Recurrent Neural Networks (RNNs) can monitor system logs for suspicious activities over time (Wang et al., 2019). Moreover, DL models can be used for anomaly detection, learning the normal operational patterns of an energy system and then flagging deviations that may indicate a cyber threat. This capability is crucial for early detection, allowing for immediate containment and mitigation of potential breaches.

RL is particularly suited to help cybersecurity where the threat landscape is dynamic, and the attackers continually evolve their strategies (Nguyen and Reddi 2021). By simulating cyberattack scenarios on the smart grid, RL algorithms are trained to recognise patterns of intrusion and react in real-time to neutralise threats. This simulation-based learning allows the algorithms to experience a wide range of attack vectors, ensuring a comprehensive defence strategy. In energy systems, where infrastructure resilience is critical, RL’s ability to adapt to rapid changes is invaluable. During an attack such as a Distributed Denial of Service (DDoS), RL can efficiently manage resources and re-route traffic to ensure minimal disruption. Over time, as the RL algorithm encounters more attacks, its strategy becomes more refined and robust, thereby enhancing the overall security of the system.

The implementation of DL and RL in securing the smart grid comes with its set of challenges. One of the primary concerns is the demand for large volumes of high-quality training data, which can be difficult to procure, especially in scenarios simulating sophisticated cyberattacks. The computational intensity required for training and running these advanced models also poses logistical and financial challenges. Additionally, there is the risk of adversarial ML, where attackers may intentionally feed misleading data to corrupt the learning process. The opaque nature of these models, often referred to as ‘black boxes’, complicates the understanding of their decision-making processes. This lack of transparency can be a significant hurdle in sectors like cybersecurity, where trust and accountability are paramount. To address this, the development of explainable AI/ML tools is crucial. Ensuring that these systems adhere to ethical guidelines and regulations is essential to maintain public trust and to safeguard against the misuse of technology. While DL and RL offer transformative potential for cybersecurity in the energy sector, realising this potential requires navigating technical complexities, ethical considerations, and the need for explainable and trustworthy AI systems. As these technologies continue to mature, their integration into the cybersecurity infrastructure will play a pivotal role in securing the future of energy systems against the ever-evolving landscape of cyber threats.

4.5 Conclusion

The digital transformation underway in the energy sector holds immense potential to enhance efficiency, sustainability, and resilience. Integral to this evolution is the integration of AI and ML, underpinned by proliferating data and advanced analytics. The convergence of these technologies unlocks new capabilities that were previously unattainable. A good example of this potential is the smart grid, which leverages real-time data and intelligent algorithms to optimise generation, transmission, and distribution. Another pivotal application relates to EVs, where AI can improve battery management and charging patterns. But thin data in nascent areas like predictive maintenance necessitates careful training to avoid problems. As with smart grids, transparency and ethics are vital to steer AI towards the public good. The path forward must also address data availability and quality, as training robust models requires vast datasets. Public–private partnerships could help overcome proprietary barriers to data sharing and sharing computing power and energy demands also warrant consideration given AI’s intense computational needs.

As outlined, DL and RL enable myriad grid enhancements spanning forecasting, control, and cybersecurity. However, substantial obstacles remain before large-scale adoption. Ensuring the safety and stability of AI-based systems is paramount, as failure could trigger cascading blackouts. Rigorous testing and validation are critical. The opacity of complex neural networks also engenders concerns about accountability and ethics. Developing explainable AI models to elucidate the rationale behind autonomous decisions will be crucial for stakeholders’ trust.

As underscored by the Colonial Pipeline attack, cyber threats represent the dark side of connectivity. AI-powered defence systems show promise, but underestimating how nefarious actors may use AI adversary invites failure. Adversarial ML could corrupt training data or exploit blind spots in models. Ultimately there are no silver bullets in cybersecurity. Overall, while AI enables step-changes in the energy sector, it is not a panacea. Technology is only one piece of the puzzle. Realising a sustainable energy future requires holistic thinking across policy, business models, culture, and infrastructure. AI should augment human capabilities, not supplant them. AI is a powerful tool, but not a replacement for human ingenuity, ethics, and leadership. Moving forward, striking the right balance between innovation and regulation will be crucial. Effective governance can steer AI towards the public good while giving it space to evolve responsibly. Beyond technology, truly sustainable energy demands integrating social science, especially economics, and humanities perspectives into solution design. A shared vision for the future and willingness to adapt will determine if AI lifts the energy sector to new heights or leads it astray.