Skip to main content
SpringerLink
Log in

Self-learning Decision and Control for Highly Automated Vehicles

  • Chapter
  • First Online:
AI-enabled Technologies for Autonomous and Connected Vehicles

  • 1020 Accesses

Abstract

The decision and control module plays a key role for autonomous driving, which is responsible for generating appropriate control commands that navigate the autonomous vehicles safely and efficiently. Existing decision and control modules for automated vehicles are mainly using a rule-based hand-engineered approach. Although working well in a number of specialized scenarios, such method shows its limitation when dealing with highly automated driving tasks such as dense urban scenarios. Recent advances in artificial intelligence have inspired a line of works about self-learning based decision and control, which enable self-reinforcement of the control policy to potentially super-human performance. In this chapter, we will introduce how to appropriately apply such techniques to automated vehicles. The chapter will begin with the motivations and basics, followed by the key challenges and recent achievements of self-learning decision and control for automated vehicles, focusing on the following key aspects: scalability, performance, interpretability, mixed-model, and emergency handling.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bojarski M, Yeres P, Choromanska A, Choromanski K, Firner B, Jackel L, Muller U (2017) Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911

  2. Chen J, Li SE, Tomizuka M (2021) Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Trans Intell Transp Syst

    Google Scholar 

  3. Chen J, Wang Z, Tomizuka M (2018) Deep hierarchical reinforcement learning for autonomous driving with distinct behaviors. In: 2018 IEEE intelligent vehicles symposium (IV). IEEE, pp 1239–1244

    Google Scholar 

  4. Chen J, Yuan B, Tomizuka M (2019) Model-free deep reinforcement learning for urban autonomous driving. In: 2019 IEEE intelligent transportation systems conference (ITSC). IEEE, pp 2765–2771

    Google Scholar 

  5. Duan J, Guan Y, Li SE, Ren Y, Sun Q, Cheng B (2021) Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. IEEE Trans Neural Netw Learn Syst

    Google Scholar 

  6. Duan J, Li SE, Guan Y, Sun Q, Cheng B (2020) Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data. IET Intell Transp Syst 14(5):297–305

    Google Scholar 

  7. Duan J, Yu D, Li SE, Wang W, Ren Y, Lin Z, Cheng B (2021) Fixed-dimensional and permutation invariant state representation of autonomous driving. arXiv preprint arXiv:2105.11299

  8. Emuna R, Borowsky A, Biess A (2020) Deep reinforcement learning for human-like driving policies in collision avoidance tasks of self-driving cars. arXiv preprint arXiv:2006.04218

  9. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning. PMLR, pp 1126–1135

    Google Scholar 

  10. Fujimoto S, van Hoof H, Meger D (2018) Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477

  11. Gu Z, Yang Y, Duan J, Li SE, Chen J, Cao W, Zheng S (2021) Belief state separated reinforcement learning for autonomous vehicle decision making under uncertainty. In: 2021 IEEE 24th international conference on intelligent transportation systems (ITSC), pp 1–7

    Google Scholar 

  12. Guan Y, Li SE, Duan J, Li J, Ren Y, Sun Q, Cheng B (2019) Direct and indirect reinforcement learning. Int J Intell Syst

    Google Scholar 

  13. Guan Y, Li SE, Duan J, Wang W, Cheng B (2018) Markov probabilistic decision making of self-driving cars in highway with random traffic flow: a simulation study. J Intell Connected Veh

    Google Scholar 

  14. Guan Y, Ren Y, Li SE, Sun Q, Luo L, Li K (2020) Centralized cooperation for connected and automated vehicles at intersections by proximal policy optimization. IEEE Trans Veh Technol 69(11):12597–12608

    Google Scholar 

  15. Guan Y, Ren Y, Sun Q, Li SE, Ma H, Duan J, Dai Y, Cheng B (2021) Integrated decision and control: towards interpretable and computationally efficient driving intelligence. arXiv preprint arXiv:2103.10290

  16. Guo J, Kurup U, Shah Mohak (2019) Is it safe to drive? An overview of factors, metrics, and datasets for driveability assessment in autonomous driving. IEEE Trans Intell Transp Syst 21(8):3135–3151

    Article  Google Scholar 

  17. Haarnoja T, Tang H, Abbeel P, Levine S (2017) Reinforcement learning with deep energy-based policies. In: Proceedings of the 34th international conference on machine learning, vol 70, pp 1352–1361. (JMLR-organization)

    Google Scholar 

  18. Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290

  19. Haarnoja T, Zhou A, Hartikainen K, Tucker G, Ha S, Tan J, Kumar V, Zhu H, Gupta A, Abbeel P et al (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

  20. Hafner D, Lillicrap T, Fischer I, Villegas R, Ha D, Lee H, Davidson J (2018) Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551

  21. Hafner D, Lillicrap T, Fischer I, Villegas R, Ha D, Lee H, Davidson J (2019) Learning latent dynamics for planning from pixels. In: International conference on machine learning, pp 2555–2565. PMLR

    Google Scholar 

  22. Hou L, Xin L, Li SE, Cheng B, Wang W (2019) Interactive trajectory prediction of surrounding road users for autonomous driving using structural-LSTM network. IEEE Trans Intell Transp Syst 21(11):4615–4625

    Google Scholar 

  23. Kahn G, Villaflor A, Pong V, Abbeel P, Levine S (2017) Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182

  24. Kim J, Canny J (2017) Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference on computer vision, pp. 2942–2950

    Google Scholar 

  25. Kong Y, Guan Y, Duan J, Li SE, Sun Q, Nie B (2021) Decision-making under on-ramp merge scenarios by distributional soft actor-critic algorithm. arXiv preprint arXiv:2103.04535

  26. Krishnan RG, Shalit U, Sontag D (2015) Deep kalman filters. arXiv preprint arXiv:1511.05121

  27. Lee AX, Nagabandi A, Abbeel P, Levine S (2019) Stochastic latent actor-critic: deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953

  28. Levine S (2018) Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv preprint arXiv:1805.00909

  29. Li G, Li SE, Cheng B, Green P (2017) Estimation of driving style in naturalistic highway traffic using maneuver transition probabilities. Transp Res Part C Emerg Technol 74:113–125

    Google Scholar 

  30. Li Shengbo, Li Keqiang, Rajamani Rajesh, Wang Jianqiang (2010) Model predictive multi-objective vehicular adaptive cruise control. IEEE Trans Control Syst Technol 19(3):556–566

    Article  Google Scholar 

  31. Li SE (2020) Reinforcement learning and control. Tsinghua University: Lecture Notes. http://www.idlab-tsinghua.com/thulab/labweb/publications.html

  32. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

  33. Lu X-Y, Wang J, Li SE, Zheng Y (2014) Multiple-vehicle longitudinal collision mitigation by coordinated brake control. Math Probl Eng 2014

    Google Scholar 

  34. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529

    Google Scholar 

  35. Mu Y, Li SE, Liu C, Sun Q, Nie B, Cheng B, Peng B (2020) Mixed reinforcement learning with additive stochastic uncertainty. arXiv preprint arXiv:2003.00848

  36. Yao Mu, Baiyu Peng, Ziqing Gu, Shengbo Eben Li, Chang Liu, Bingbing Nie, Jianfeng Zheng, and Bo Zhang. Mixed reinforcement learning for efficient policy optimization in stochastic environments. In: 2020 20th international conference on control, automation and systems (ICCAS). IEEE, pp 1212–1219

    Google Scholar 

  37. Peng B, Mu Y, Duan J, Guan Y, Li SE, Chen J (2021) Separated proportional-integral lagrangian for chance constrained reinforcement learning. arXiv preprint arXiv:2102.08539

  38. Peng B, Mu Y, Guan Y, Li SE, Yin Y, Chen J (2020) Model-based actor-critic with chance constraint for stochastic system. arXiv preprint arXiv:2012.10716 2020

  39. Ren Y, Duan J, Li SE, Guan Y, Sun Q (2020) Improving generalization of reinforcement learning with minimax distributional soft actor-critic. In: 2020 IEEE 23rd international conference on intelligent transportation systems (ITSC). IEEE, pp 1–6

    Google Scholar 

  40. Schulman J, Levine S, Abbeel P, Jordan M, Moritz P (2015) Trust region policy optimization. In: International conference on machine learning, pages 1889–1897

    Google Scholar 

  41. Shengbo LI, Yang G, Lian HOU, Hongbo GAO , Jingliang DUAN , Shuang LIANG , WANG Yu, CHENG Bo, LI Keqiang, REN Wei et al (2019) Key technique of deep neural network and its applications in autonomous driving. J Autom Saf Energy 10(2):119

    Google Scholar 

  42. Sutton RS, Szepesvári C, Geramifard A, Bowling MP (2012) Dyna-style planning with linear function approximation and prioritized sweeping. arXiv preprint arXiv:1206.3285

  43. Urmson C, Anhalt J, Bagnell D, Baker C, Bittner R, Clark MN, Dolan J, Duggins D, Galatali T, Geyer C et al (2008) Autonomous driving in urban environments: boss and the urban challenge. J Field Robot 25(8):425–466

    Google Scholar 

  44. Wen L, Duan J, Li SE, Xu S, Peng H (2020) Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In: 2020 IEEE 23rd international conference on intelligent transportation systems (ITSC). IEEE, pp 1–7

    Google Scholar 

  45. Xin L, Kong Y, Li SE, Chen J, Guan Y, Tomizuka M, Cheng B (2021) Enable faster and smoother spatio-temporal trajectory planning for autonomous vehicles in constrained dynamic environment. Proc Inst Mech Eng Part D J Autom Eng 235(4):1101–1112

    Google Scholar 

  46. Yin Y, Li SE, Li K, Yang J, Ma F (2020) Self-learning drift control of automated vehicles beyond handling limit after rear-end collision. Transp Saf Environ 2(2):97–105

    Google Scholar 

  47. Zhang F, Gonzales J, Li SE, Borrelli F, Li K (2018) Drift control for cornering maneuver of autonomous vehicles. Mechatronics 54:167–174

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, 100084, China

    Jianyu Chen

  2. School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

    Jingliang Duan, Yang Guan, Qi Sun & Shengbo Eben Li

  3. Department of Mechanical Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Yuming Yin

Authors
  1. Jianyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingliang Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qi Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuming Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shengbo Eben Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shengbo Eben Li .

Editor information

Editors and Affiliations

  1. College of Engineering and Computer Science, University of Michigan–Dearborn, Dearborn, MI, USA

    Yi Lu Murphey

  2. Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA

    Ilya Kolmanovsky

  3. College of Engineering and Computer Science, University of Michigan–Dearborn, Dearborn, MI, USA

    Paul Watta

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chen, J., Duan, J., Guan, Y., Sun, Q., Yin, Y., Li, S.E. (2023). Self-learning Decision and Control for Highly Automated Vehicles. In: Murphey, Y.L., Kolmanovsky, I., Watta, P. (eds) AI-enabled Technologies for Autonomous and Connected Vehicles. Lecture Notes in Intelligent Transportation and Infrastructure. Springer, Cham. https://doi.org/10.1007/978-3-031-06780-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06780-8_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06779-2

  • Online ISBN: 978-3-031-06780-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation