Skip to main content
SpringerLink
Log in

An Improved Soft Actor-Critic-Based Energy Management Strategy of Fuel Cell Hybrid Vehicles with a Nonlinear Fuel Cell Degradation Model

  • Regular Paper
  • Published:
International Journal of Precision Engineering and Manufacturing-Green Technology Aims and scope Submit manuscript

Abstract

With the rapid development of artificial intelligence, deep reinforcement learning (DRL)-based energy management strategies (EMSs) have become an important research direction for hybrid electric vehicles recently, which still face some problems such as fragile convergence characteristics, slower convergence speed, and unsatisfactory optimization effects. In this research, a novel DRL algorithm, i.e. an improved soft actor-critic (ISAC) algorithm is applied to the EMS of a fuel cell hybrid vehicle (FCHV), in which the priority experience replay (PER) and emphasizing recent experience (ERE) methods are adopted to improve the convergence performance of the algorithm and to enhance the FCHV fuel economy. In addition, the fuel cell durability is also considered in the proposed EMS based on a nonlinear fuel cell degradation model while considering the fuel economy. Results indicate that the FCHV fuel consumption of the proposed EMS is decreased by 7.87%, 2.79%, and 2.44% compared to that of the deep deterministic policy gradient (DDPG)-based, the twin delayed deep deterministic policy gradient (TD3)-based, and the SAC-based EMSs respectively while the fuel consumption gap to the dynamic programming-based EMS is narrowed to 2.37% by the proposed EMS. Moreover, the proposed EMS presents the best training performance considering both the convergence speed and stability, and the convergence speed of the proposed EMS is increased by an average of 47.89% compared to that of the other DRL-based EMSs. Furthermore, the fuel cell durability is improved by more than 95% using the proposed EMS compared to that of the EMS without considering the fuel cell degradation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Chang, X., Ma, T., & Wu, R. (2019). Impact of urban development on residents’ public transportation travel energy consumption in China: An analysis of hydrogen fuel cell vehicles alternatives. International Journal of Hydrogen Energy. https://doi.org/10.1016/j.ijhydene.2018.09.099

    Article  Google Scholar 

  2. Gao, J., et al. (2019). Challenges and developments of automotive fuel cell hybrid power system and control. Science China Information Sciences. https://doi.org/10.1007/s11432-018-9690-y

    Article  Google Scholar 

  3. Liu, Z., Hao, H., Cheng, X., & Zhao, F. (2018). Critical issues of energy efficient and new energy vehicles development in China. Energy Policy, 115, 92–97. https://doi.org/10.1016/J.ENPOL.2018.01.006

    Article  Google Scholar 

  4. Showers, S. O., & Raji, A. K. (2022). State-of-the-art review of fuel cell hybrid electric vehicle energy management systems. AIMS Energy, 10(3), 458–485. https://doi.org/10.3934/energy.2022023

    Article  Google Scholar 

  5. Zhou, S., Chen, Z., Huang, D., & Lin, T. (2021). Model prediction and rule based energy management strategy for a plug-in hybrid electric vehicle with hybrid energy storage system. IEEE Transactions on Power Electronics, 36(5), 5926–5940. https://doi.org/10.1109/TPEL.2020.3028154

    Article  Google Scholar 

  6. Bagwe, R. M., Byerly, A., Cipriano, E., Santos, D., & Ben-Miled, Z. (2019). Adaptive rule-based energy management strategy for a parallel HEV. Energies. https://doi.org/10.3390/en12234472

    Article  Google Scholar 

  7. He, H., Wang, X., Chen, J., & Wang, Y.-X. (2020). Regenerative fuel cell-battery-supercapacitor hybrid power system modeling and improved rule-based energy management for vehicle application. Journal of Energy Engineering, 146(6), 04020060. https://doi.org/10.1061/(asce)ey.1943-7897.0000708

    Article  Google Scholar 

  8. Liu, Y., Gao, J., Qin, D., Zhang, Y., & Lei, Z. (2018). Rule-corrected energy management strategy for hybrid electric vehicles based on operation-mode prediction. Journal of Cleaner Production, 188, 796–806. https://doi.org/10.1016/J.JCLEPRO.2018.04.024

    Article  Google Scholar 

  9. Polverino, P., Arsie, I., & Pianese, C. (2021). Optimal energy management for hybrid electric vehicles based on dynamic programming and receding horizon. Energies. https://doi.org/10.3390/en14123502

    Article  Google Scholar 

  10. Li, X., Wang, Y., Yang, D., & Chen, Z. (2019). Adaptive energy management strategy for fuel cell/battery hybrid vehicles using Pontryagin’s Minimal Principle. Journal of Power Sources, 440, 227105. https://doi.org/10.1016/J.JPOWSOUR.2019.227105

    Article  Google Scholar 

  11. Wang, Y., Wu, Z., Chen, Y., Xia, A., Guo, C., & Tang, Z. (2018). Research on energy optimization control strategy of the hybrid electric vehicle based on Pontryagin’s minimum principle. Computers & Electrical Engineering, 72, 203–213. https://doi.org/10.1016/J.COMPELECENG.2018.09.018

    Article  Google Scholar 

  12. Yang, Y., Pei, H., Hu, X., Liu, Y., Hou, C., & Cao, D. (2019). Fuel economy optimization of power split hybrid vehicles: A rapid dynamic programming approach. Energy, 166, 929–938. https://doi.org/10.1016/J.ENERGY.2018.10.149

    Article  Google Scholar 

  13. Liu, B., Li, L., Member, S., Wang, X., & Cheng, S. (2018). Hybrid electric vehicle downshifting strategy based on stochastic dynamic programming during regenerative braking process. IEEE Transactions on Vehicular Technology. https://doi.org/10.1109/TVT.2018.2815518

    Article  Google Scholar 

  14. Yuan, Z., Teng, L., Fengchun, S., & Peng, H. (2013). Comparative study of dynamic programming and pontryagin’s minimum principle on energy management for a parallel hybrid electric vehicle. Energies, 6, 2305–2318. https://doi.org/10.3390/en6042305

    Article  Google Scholar 

  15. Kim, N., Cha, S., & Peng, H. (2011). Optimal control of hybrid electric vehicles based on pontryagin’s minimum principle. IEEE Transactions on Control Systems Technology. https://doi.org/10.1109/TCST.2010.2061232

    Article  Google Scholar 

  16. Jeoung, H., Lee, W., Park, D., & Kim, N. (2022). Data-driven analysis of the correlation of future information and costates for pmp-based energy management strategy of hybrid electric vehicle. International Journal of Precision Engineering and Manufacturing-Green Technology, 9(3), 873–883. https://doi.org/10.1007/s40684-021-00400-0

    Article  Google Scholar 

  17. Li, W., Ye, J., Cui, Y., Kim, N., Cha, S. W., & Zheng, C. (2022). A speedy reinforcement learning-based energy management strategy for fuel cell hybrid vehicles considering fuel cell system lifetime. International Journal of Precision Engineering and Manufacturing—Green Technology, 9(3), 859–872. https://doi.org/10.1007/s40684-021-00379-8

    Article  Google Scholar 

  18. Bo, L., Han, L., Xiang, C., Liu, H., & Ma, T. (2022). A Q-learning fuzzy inference system based online energy management strategy for off-road hybrid electric vehicles. Energy, 252, 123976. https://doi.org/10.1016/J.ENERGY.2022.123976

    Article  Google Scholar 

  19. Chen, Z., Gu, H., Shen, S., & Shen, J. (2022). Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning. Energy, 245, 123182. https://doi.org/10.1016/J.ENERGY.2022.123182

    Article  Google Scholar 

  20. Xu, B., Shi, J., Li, S., Li, H., & Wang, Z. (2021). Energy consumption and battery aging minimization using a Q-learning strategy for a battery/ultracapacitor electric vehicle. Energy, 229, 120705. https://doi.org/10.1016/J.ENERGY.2021.120705

    Article  Google Scholar 

  21. Xu, B., et al. (2022). Q-learning-based supervisory control adaptability investigation for hybrid electric vehicles. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2021.3062179

    Article  Google Scholar 

  22. Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  23. Li, W., et al. (2021). Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles. Journal of Energy Storage, 36, 102355. https://doi.org/10.1016/J.EST.2021.102355

    Article  Google Scholar 

  24. Wu, J., He, H., Peng, J., Li, Y., & Li, Z. (2018). Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Applied Energy, 222, 799–811. https://doi.org/10.1016/J.APENERGY.2018.03.104

    Article  Google Scholar 

  25. Qi, X., Luo, Y., Wu, G., Boriboonsomsin, K., & Barth, M. (2019). Deep reinforcement learning enabled self-learning control for energy efficient driving. Transportation Research Part C: Emerging Technologies, 99, 67–81. https://doi.org/10.1016/J.TRC.2018.12.018

    Article  Google Scholar 

  26. Zheng, C., Li, W., Li, W., Xu, K., Peng, L., & Cha, S. W. (2022). A deep reinforcement learning-based energy management strategy for fuel cell hybrid buses. International Journal of Precision Engineering and Manufacturing-Green Technology, 9(3), 885–897. https://doi.org/10.1007/s40684-021-00403-x

    Article  Google Scholar 

  27. Lillicrap, T. P., et al. Continuous control with deep reinforcement learning. Retrieved from https://goo.gl/J4PIAz

  28. Lillicrap, T. P., et al. (2016). Continuous control with deep reinforcement learning. Retrieved from https://goo.gl/J4PIAz

  29. Ma, Z., Huo, Q., Zhang, T., Hao, J., & Wang, W. (2021). Deep deterministic policy gradient based energy management strategy for hybrid electric tracked vehicle with online updating mechanism. IEEE Access, 99, 1–1.

    Google Scholar 

  30. Lian, R., Tan, H., Peng, J., Li, Q., & Wu, Y. (2020). Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management. IEEE Transactions on Vehicular Technology. https://doi.org/10.1109/TVT.2020.2999263

    Article  Google Scholar 

  31. Lian, R., Peng, J., Wu, Y., Tan, H., & Zhang, H. (2020). Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle. Energy, 197, 117297. https://doi.org/10.1016/J.ENERGY.2020.117297

    Article  Google Scholar 

  32. Fujimoto, S., van Hoof, H., & Meger, D. (2018). Addressing function approximation error in actor-critic methods. Retrieved from https://github.com/

  33. Huang, R., He, H., Zhao, X., Wang, Y., & Li, M. (2022). Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm. Applied Energy, 321, 119353. https://doi.org/10.1016/J.APENERGY.2022.119353

    Article  Google Scholar 

  34. Zhou, J., Xue, S., Xue, Y., Liao, Y., Liu, J., & Zhao, W. (2021). A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy, 224, 120118. https://doi.org/10.1016/J.ENERGY.2021.120118

    Article  Google Scholar 

  35. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic algorithms and applications. ArXiv E-Prints, earXiv:1812.05905v1. https://doi.org/10.48550/arXiv.1812.05905

  36. Xu, D., Cui, Y., Ye, J., Cha, S. W., Li, A., & Zheng, C. (2022). A soft actor-critic-based energy management strategy for electric vehicles with hybrid energy storage systems. Journal of Power Sources, 524, 231099. https://doi.org/10.1016/J.JPOWSOUR.2022.231099

    Article  Google Scholar 

  37. Sun, W., Zou, Y., Zhang, X., Guo, N., Zhang, B., & Du, G. (2022). High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning. Energy. https://doi.org/10.1016/J.ENERGY.2022.124806

    Article  Google Scholar 

  38. Xiao, B., Yang, W., Wu, J., Walker, P. D., & Zhang, N. (2022). Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle. Energy, 253, 124105. https://doi.org/10.1016/J.ENERGY.2022.124105

    Article  Google Scholar 

  39. Wang, C., & Ross, K. (2019). Boosting soft actor-critic: emphasizing recent experience without forgetting the past. ArXiv E-Prints, earXiv:1906.04009. https://doi.org/10.48550/arXiv.1906.04009

  40. Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized Experience Replay. ArXiv E-Prints, earXiv:1511.05952. https://doi.org/10.48550/arXiv.1511.05952

  41. Pei, P., Chen, D., Wu, Z., & Ren, P. (2019). Nonlinear methods for evaluating and online predicting the lifetime of fuel cells. Applied Energy, 254, 113730. https://doi.org/10.1016/J.APENERGY.2019.113730

    Article  Google Scholar 

  42. Pei, P., & Chen, H. (2014). Main factors affecting the lifetime of Proton Exchange Membrane fuel cells in vehicle applications: A review. Applied Energy, 125, 60–75. https://doi.org/10.1016/J.APENERGY.2014.03.048

    Article  Google Scholar 

  43. Pei, P., Chang, Q., & Tang, T. (2008). A quick evaluating method for automotive fuel cell lifetime. International Journal of Hydrogen Energy, 33(14), 3829–3836. https://doi.org/10.1016/J.IJHYDENE.2008.04.048

    Article  Google Scholar 

  44. Chen, H., Pei, P., & Song, M. (2015). Lifetime prediction and the economic lifetime of Proton Exchange Membrane fuel cells. Applied Energy, 142, 154–163. https://doi.org/10.1016/J.APENERGY.2014.12.062

    Article  Google Scholar 

  45. Rousseau, A., et al. (2014). Electric drive vehicle development and evaluation using system simulation. IFAC Proceedings Volumes, 47(3), 7886–7891. https://doi.org/10.3182/20140824-6-ZA-1003.02832

    Article  Google Scholar 

  46. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., & Levine, S. (2019). Soft actor-critic algorithms and applications. ArXiv E-Prints, earXiv:1812.05905v2. https://doi.org/10.48550/arXiv.1812.05905

Download references

Acknowledgements

This research was supported by Shenzhen Science and Technology Innovation Commission (Grant No. KQJSCX20180330170047681, JCYJ20210324115800002), Department of Science and Technology of Guangdong Province (Grant No. 2021A0505030056, 2021A0505050005), CAS PIFI program (2017VCA0032, 2021VEB0001), CAS International Partnership Program (321GJHZ2022057MI), and Shenzhen Pengcheng Program.

Funding

Shenzhen Science and Technology Innovation Commission, KQJSCX20180330170047681, Chunhua Zheng, JCYJ20210324115800002, Chunhua Zheng, Department of Science and Technology of Guangdong Province, 2021A0505030056, Chunhua Zheng, 2021A0505050005, Chunhua Zheng, CAS PIFI program, 2017VCA0032,2021VEB0001, CAS International Partnership Program, 321GJHZ2022057MI, Chunhua Zheng, Shenzhen Pengcheng Program.

Author information

Authors and Affiliations

  1. Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, 1068 Xueyuan Avenue, Shenzhen University Town, Shenzhen, 518055, China

    Dongfang Zhang, Yunduan Cui, Yao Xiao, Shengxiang Fu & Chunhua Zheng

  2. School of Software Engineering, East China Normal University, 3663 Zhongshanbei Road, Shanghai, 200062, China

    Dongfang Zhang & Hongyan Mao

  3. School of Mechanical Engineering, Shenyang University of Technology, No.111, Shenliao West Road, Economic & Technological Development Zone, Shenyang, 110870, China

    Shengxiang Fu

  4. School of Mechanical and Aerospace Engineering, Seoul National University, San 56-1, Daehak-Dong, Gwanak-Gu, Seoul, 151742, South Korea

    Suk Won Cha

  5. Department of Mechanical Engineering, Hanyang University, 55 Hanyangdeahak-Ro, Sangnok-Gu, Ansan, Gyeonggi-Do, 15588, South Korea

    Namwook Kim

Authors
  1. Dongfang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunduan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengxiang Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Suk Won Cha
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Namwook Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hongyan Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chunhua Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunhua Zheng.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial or non-financial interests that are directly or indirectly related to the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Cui, Y., Xiao, Y. et al. An Improved Soft Actor-Critic-Based Energy Management Strategy of Fuel Cell Hybrid Vehicles with a Nonlinear Fuel Cell Degradation Model. Int. J. of Precis. Eng. and Manuf.-Green Tech. 11, 183–202 (2024). https://doi.org/10.1007/s40684-023-00547-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40684-023-00547-y

Keywords

Search

Navigation