Skip to main content

Advertisement

SpringerLink
Log in

Power management in hybrid electric vehicles using deep recurrent reinforcement learning

  • Original Paper
  • Published:
Electrical Engineering Aims and scope Submit manuscript

Abstract

A power management framework for hybrid electric vehicles (HEVs) is proposed based on deep reinforcement learning (DRL) with a Long Short-Term Memory (LSTM) network to minimize the fuel consumption through determining the power distribution between the two propulsion sources, the internal combustion engine (ICE) and the electric motor (EM). DRL is effective for handling the high-dimensional state and action spaces in the HEV power management problem, and the LSTM structure leverages temporal dependencies of input information, providing internal state predictions automatically without introducing extra state variables. This technique is entirely online, meaning that the framework is constructed in real time during the training phase, independently of a prior knowledge of driving cycles. The learned information stored in the LSTM network is utilized efficiently, and the computational speed is enhanced by making multiple predictions simultaneously in each step. Simulation over various driving cycles demonstrates the efficacy of the proposed framework in fuel economy improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Availability of data and material

Not applicable.

References

  1. (2003) Advisor documentation. National Renewable Energy Laboratory. http://adv-vehicle-sim.sourceforge.net/advisor_doc.html. Accessed 28 Feb 2018

  2. Baumann BM, Washington G, Glenn BC, Rizzoni G (2000) Mechatronic design and control of hybrid electric vehicles. IEEE/ASME Trans Mechatron 5(1):58–72

    Article  Google Scholar 

  3. Chen Z, Mi CC, Xu J, Gong X, You C (2014) Energy management for a power-split plug-in hybrid electric vehicle based on dynamic programming and neural networks. IEEE Trans Veh Technol 63(4):1567–1580

    Article  Google Scholar 

  4. Gong Q, Li Y, Peng Z (2009) Power management of plug-in hybrid electric vehicles using neural network based trip modeling. In: American control conference, 2009. ACC’09. IEEE, pp 4601–4606

  5. Hausknecht M, Stone P (2015) Deep recurrent q-learning for partially observable mdps. CoRR, abs/150706527

  6. He X, Parten M, Maxwell T (2005) Energy management strategies for a hybrid electric vehicle. In: 2005 IEEE conference vehicle power and propulsion. IEEE, pp 390–394

  7. Jeon S, Jo S, Park Y, Lee J (2002) Multi-mode driving control of a parallel hybrid electric vehicle using driving pattern recognition. J Dyn Syst Meas Control 124(1):141–149

    Article  Google Scholar 

  8. Kermani S, Delprat S, Guerra T, Trigui R (2009) Predictive control for hev energy management: experimental results. In: Vehicle power and propulsion conference, (2009) VPPC’09, IEEE. IEEE, pp 364–369

  9. Kirschbaum F, Back M, Hart M (2002) Determination of the fuel-optimal trajectory for a vehicle along a known route. IFAC Proc Vol 35(1):235–239

    Article  Google Scholar 

  10. Lample G, Chaplot DS (2017) Playing fps games with deep reinforcement learning. In: AAAI, pp 2140–2146

  11. Lin CC, Peng H, Grizzle JW, Kang JM (2003) Power management strategy for a parallel hybrid electric truck. IEEE Trans Control Syst Technol 11(6):839–849

    Article  Google Scholar 

  12. Lin CC, Peng H, Grizzle J (2004) A stochastic control strategy for hybrid electric vehicles. In: American control conference, 2004. Proceedings of the 2004. IEEE, vol 5, pp 4710–4715

  13. Lin X, Wang Y, Bogdan P, Chang N, Pedram M (2014) Reinforcement learning based power management for hybrid electric vehicles. In: Proceedings of the 2014 IEEE/ACM international conference on computer-aided design. IEEE Press, pp 32–38

  14. Liu C, Murphey YL (2014) (2014) Power management for plug-in hybrid electric vehicles using reinforcement learning with trip information. In: Transportation electrification conference and expo (ITEC), IEEE. IEEE, pp 1–6

  15. Liu T, Hu X, Li SE, Cao D (2017) Reinforcement learning optimized look-ahead energy management of a parallel hybrid electric vehicle. IEEE/ASME Trans Mechatron 22(4):1497–1507

    Article  Google Scholar 

  16. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529

    Article  Google Scholar 

  17. Moreno J, Ortúzar ME, Dixon JW (2006) Energy-management system for a hybrid electric vehicle, using ultracapacitors and neural networks. IEEE Trans Ind Electron 53(2):614–623

    Article  Google Scholar 

  18. Musardo C, Rizzoni G, Guezennec Y, Staccia B (2005) A-ecms: an adaptive algorithm for hybrid electric vehicle energy management. Eur J Control 11(4–5):509–524

    Article  MathSciNet  Google Scholar 

  19. Ng KS, Moo CS, Chen YP, Hsieh YC (2009) Enhanced coulomb counting method for estimating state-of-charge and state-of-health of lithium-ion batteries. Appl Energy 86(9):1506–1511

    Article  Google Scholar 

  20. Onori SRG, Serrao L (2016) HEV modeling. Springer, London

    Book  Google Scholar 

  21. Paganelli G, Tateno M, Brahma A, Rizzoni G, Guezennec Y (2001) Control development for a hybrid-electric sport-utility vehicle: strategy, implementation and field test results. In: american control conference, 2001. Proceedings of the 2001, IEEE, vol 6, pp 5064–5069

  22. Paganelli G, Delprat S, Guerra TM, Rimaux J, Santin JJ (2002) Equivalent consumption minimization strategy for parallel hybrid powertrains. In: Vehicular technology conference, 2002. VTC Spring 2002. IEEE 55th, IEEE, vol 4, pp 2076–2081

  23. Pu J, Yin C (2007) Optimal control of fuel economy in parallel hybrid electric vehicles. Proc Inst Mech Eng Part D J Automob Eng 221(9):1097–1106

    Article  Google Scholar 

  24. Qi X, Wu G, Boriboonsomsin K, Barth MJ, Gonder J (2016) Data-driven reinforcement learning-based real-time energy management system for plug-in hybrid electric vehicles. Transp Res Rec J Transp Res Board 2572:1–8

    Article  Google Scholar 

  25. Salman M, Schouten NJ, Kheir NA (2000) Control strategies for parallel hybrid vehicles. In: American control conference, 2000. Proceedings of the 2000. IEEE, vol 1, pp 524–528

  26. Schouten NJ, Salman MA, Kheir NA (2002) Fuzzy logic control for parallel hybrid vehicles. IEEE Trans Control Syst Technol 10(3):460–468

    Article  Google Scholar 

  27. Sciarretta A, Back M, Guzzella L (2004) Optimal control of parallel hybrid electric vehicles. IEEE Trans Control Syst Technol 12(3):352–363

    Article  Google Scholar 

  28. Sundström O, Guzzella L, Soltic P (2008) Optimal hybridization in two parallel hybrid electric vehicles using dynamic programming. IFAC Proc Vol 41(2):4642–4647

    Article  Google Scholar 

  29. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  30. Tian X, Cai Y, Sun X, Zhu Z, Xu Y (2019a) An adaptive ecms with driving style recognition for energy optimization of parallel hybrid electric buses. Energy 189(116):151

    Google Scholar 

  31. Tian X, He R, Sun X, Cai Y, Xu Y (2019b) An anfis-based ecms for energy optimization of parallel hybrid electric bus. IEEE Trans Veh Technol 69(2):1473–1483

    Article  Google Scholar 

  32. Tie SF, Tan CW (2013) A review of energy sources and energy management system in electric vehicles. Renew Sustain Energy Rev 20:82–102

    Article  Google Scholar 

  33. Xiong R, Cao J, Yu Q (2018) Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle. Appl Energy 211:538–548

    Article  Google Scholar 

  34. Yan F, Wang J, Huang K (2012) Hybrid electric vehicle model predictive control torque-split strategy incorporating engine transient characteristics. IEEE Trans Veh Technol 61(6):2458–2467

    Article  Google Scholar 

  35. Yue S, Wang Y, Xie Q, Zhu D, Pedram M, Chang N (2014) Model-free learning-based online management of hybrid electrical energy storage systems in electric vehicles. In: Industrial electronics society, IECON 2014-40th annual conference of the IEEE. IEEE, pp 3142–3148

  36. Zhao P, Wang Y, Chang N, Zhu Q, Lin X (2018) A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles. In: Design automation conference (ASP-DAC), 2018 23rd Asia and South Pacific, IEEE

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, 02115, US

    Mengshu Sun, Pu Zhao & Xue Lin

Authors
  1. Mengshu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xue Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mengshu Sun.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Code availability

Not applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

A preliminary version of this paper was presented at the 23rd Asia and South Pacific Design Automation Conference (ASP-DAC 2018), Jeju Island, Korea [36]

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, M., Zhao, P. & Lin, X. Power management in hybrid electric vehicles using deep recurrent reinforcement learning. Electr Eng 104, 1459–1471 (2022). https://doi.org/10.1007/s00202-021-01401-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00202-021-01401-7

Keywords

Search

Navigation