Skip to main content
SpringerLink
Log in

Deep Reinforcement Learning Enabled Decision-Making for Autonomous Driving at Intersections

  • Published:
Automotive Innovation Aims and scope Submit manuscript

Abstract

Road intersection is one of the most complex and accident-prone traffic scenarios, so it’s challenging for autonomous vehicles (AVs) to make safe and efficient decisions at the intersections. Most of the related studies focus on the solution to a single scenario or only guarantee safety without considering driving efficiency. To address these problems, this study proposed a deep reinforcement learning enabled decision-making framework for AVs to drive through intersections automatically, safely and efficiently. The mapping relationship between traffic images and vehicle operations was obtained by an end-to-end decision-making framework established by convolutional neural networks. Traffic images collected at two timesteps were used to calculate the relative velocity between vehicles. Markov decision process was employed to model the interaction between AVs and other vehicles, and the deep Q-network algorithm was utilized to obtain the optimal driving policy regarding safety and efficiency. To verify the effectiveness of the proposed decision-making framework, the top three accident-prone crossing path crash scenarios at intersections were simulated, when different initial vehicle states were adopted for better generalization capability. The results showed that the developed method could make AVs drive safely and efficiently through intersections in all of the tested scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

AV:

Autonomous vehicle

DQN:

Deep Q-network

DRL:

Deep reinforcement learning

LTAP/LD:

Left turn across path-lateral direction

LTAP/OD:

Left turn across path-opposite direction

MDP:

Markov decision process

OV:

Other vehicle

SCP:

Straight crossing path

V2I:

Vehicle-to-infrastructure

V2V:

Vehicle-to-vehicle

References

  1. Tay, R.: A random parameters probit model of urban and rural intersection crashes. Accid. Anal. Prev. 84, 38–40 (2015)

    Article  Google Scholar 

  2. Li, G.F., Wang, Y., Zhu, F.P., et al.: Drivers’ visual scanning behavior at signalized and unsignalized intersections: a naturalistic driving study in China. J. Saf. Res. 71, 219–229 (2019)

    Article  Google Scholar 

  3. Werneke, J., Vollrath, M.: How do environmental characteristics at intersections change in their relevance for drivers before entering an intersection: analysis of drivers’ gaze and driving behavior in a driving simulator study. Cogn. Technol. Work 16(2), 157–169 (2014)

    Article  Google Scholar 

  4. Lemonnier, S., Brémond, R., Baccino, T.: Gaze behavior when approaching an intersection: dwell time distribution and comparison with a quantitative prediction. Transp. Res. Part F Traffic Psychol. Behav. 35, 60–74 (2015)

    Article  Google Scholar 

  5. NHTSA: Traffic Safety Facts 2017 Data (DOT HS 812 806, updated September 2019). National Highway Traffic Safety Administration, Washington, DC (2019). https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812806. Accessed 14 Feb 2020

  6. Zhang, G.G., Yau, K.K.W., Chen, G.H.: Risk factors associated with traffic violations and accident severity in China. Accid. Anal. Prev. 59, 18–25 (2013)

    Article  Google Scholar 

  7. Ronald, J.: V2V/V2I Communications for Improved Road Safety and Efficiency. SAE, San Diego (2012)

    Book  Google Scholar 

  8. Li, G.F., Yang, Y.F., Qu, X.D.: Deep learning approaches on pedestrian detection in hazy weather. IEEE Trans. Ind. Electron. (2019). https://doi.org/10.1109/TIE.2019.2945295

    Article  Google Scholar 

  9. Dooley, D., McGinley, B., Hughes, C., et al.: A blind-zone detection method using a rear-mounted fisheye camera with combination of vehicle detection methods. IEEE Trans. Intell. Transp. Syst. 17(1), 264–278 (2015)

    Article  Google Scholar 

  10. Li, S.E.B., Li, G.F., Yu, J.Y., et al.: Kalman filter-based tracking of moving objects using linear ultrasonic sensor array for road vehicles. Mech. Syst. Signal Proc. 98, 173–189 (2018)

    Article  Google Scholar 

  11. Li, G.F., Li, S.E.B., Zou, R.B., et al.: Detection of road traffic participants using cost-effective arrayed ultrasonic sensors in low-speed traffic situations. Mech. Syst. Signal Proc. 132, 535–545 (2019)

    Article  Google Scholar 

  12. Noh, S.: Decision-making framework for autonomous driving at road intersections: safeguarding against collision, overly conservative behavior, and violation vehicles. IEEE Trans. Ind. Electron. 66(4), 3275–3286 (2018)

    Article  Google Scholar 

  13. Najm, W., Smith, J.D., Smith, D.L.: Analysis of crossing path crashes. John A. Volpe National Transportation Systems Center (US). No. DOT-VNTSC-NHTSA-01-03 (2001)

  14. Brannstrom, M., Coelingh, E., Sjoberg, J.: Model-based threat assessment for avoiding arbitrary vehicle collisions. IEEE Trans. Intell. Transp. Syst. 11(3), 658–669 (2010)

    Article  Google Scholar 

  15. Li, G.F., Li, S.E., Cheng, B., et al.: Estimation of driving style in naturalistic highway traffic using maneuver transition probabilities. Transp. Res. Part C Emerg. Technol. 74, 113–125 (2017)

    Article  Google Scholar 

  16. Polychronopoulos, A., Tsogas, M., Amditis, A., et al.: Sensor fusion for predicting vehicles’ path for collision avoidance systems. IEEE Trans. Intell. Transp. Syst. 8(3), 549–562 (2007)

    Article  Google Scholar 

  17. Campos, G.R., Runarsson, A.H., Granum, F., et al.: Collision avoidance at intersections: a probabilistic threat-assessment and decision-making system for safety interventions. Paper presented at the 17th International IEEE Conference on Intelligent Transportation Systems, Qingdao, China, 8–11 October 2014

  18. Jansson, J.: Collision avoidance theory with application to automotive collision mitigation. Dissertation, Linköping University (2005)

  19. Noh, S., Han, W.Y.: Collision avoidance in on-road environment for autonomous driving. Paper presented at the 14th International Conference on Control, Automation and Systems, Seoul, South Korea, 22–25 October 2014

  20. Naranjo, J.E., Gonzalez, C., Garcia, R., et al.: Lane-change fuzzy control in autonomous vehicles for the overtaking maneuver. IEEE Trans. Intell. Transp. Syst. 9(3), 438–450 (2008)

    Article  Google Scholar 

  21. Hubmann, C., Schulz, J., Becker, M., et al.: Automated driving in uncertain environments: planning with interaction and uncertain maneuver prediction. IEEE Trans. Intell. Veh. 3(1), 5–17 (2018)

    Article  Google Scholar 

  22. Schubert, R.: Evaluating the utility of driving: toward automated decision making under uncertainty. IEEE Trans. Intell. Transp. Syst. 9(3), 354–364 (2011)

    Google Scholar 

  23. Jansson, J., Gustafsson, F.: A framework and automotive application of collision avoidance decision making. Automatica 44(9), 2347–2351 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  24. Armand, A., Filliat, D., Ibanez-Guzman, J.: Modelling stop intersection approaches using Gaussian processes. Paper presented at the 16th International IEEE Conference on Intelligent Transportation Systems, The Hague, Netherlands, 6–9 October 2013

  25. Huang, R., Liang, H., Zhao, P., et al.: Intent-estimation-and motion-model-based collision avoidance method for autonomous vehicles in urban environments. Appl. Sci. 7(5), 457 (2017)

    Article  Google Scholar 

  26. Notomista, G., Botsch, M.: A machine learning approach for the segmentation of driving maneuvers and its application in autonomous parking. J. Artif. Intell. Soft Comput. Res. 7(4), 243–255 (2017)

    Article  Google Scholar 

  27. Hussein, A., Gaber, M.M., Elyan, E., et al.: Imitation learning: a survey of learning methods. ACM Comput. Surv. 50(2), 1–35 (2017)

    Article  Google Scholar 

  28. Bojarski, M., Yeres, P., Choromanska, A., et al.: Explaining how a deep neural network trained with end-to-end learning steers a car (2017). arXiv preprint arXiv:1704.07911

  29. Mo, C.M., Li, Y.N., Zheng, L.: Simulation and analysis on overtaking safety assistance system based on vehicle-to-vehicle communication. Automot. Innov. 1(2), 158–166 (2018)

    Article  Google Scholar 

  30. Hafner, M.R., Cunningham, D., Caminiti, L., et al.: Cooperative collision avoidance at intersections: algorithms and experiments. IEEE Trans. Intell. Transp. Syst. 14(3), 1162–1175 (2013)

    Article  Google Scholar 

  31. Rios-Torres, J., Malikopoulos, A.A.: A survey on the coordination of connected and automated vehicles at intersections and merging at highway on-ramps. IEEE Intell. Transp. Syst. Mag. 18(5), 1066–1077 (2017)

    Article  Google Scholar 

  32. Lee, J., Park, B.: Development and evaluation of a cooperative vehicle intersection control algorithm under the connected vehicles environment. IEEE Trans. Intell. Transp. Syst. 13(1), 81–90 (2012)

    Article  Google Scholar 

  33. Luo, Y.G., Yang, G., Xu, M.C., et al.: Cooperative lane-change maneuver for multiple automated vehicles on a highway. Automot. Innov. 2(3), 157–168 (2019)

    Article  Google Scholar 

  34. Zong, X.P., Xu, G.Y., Yu, G.Z., et al.: Obstacle avoidance for self-driving vehicle with reinforcement learning. SAE Int. J. Passeng. Cars Electron. Electr. Syst. 11(1), 30–39 (2017)

    Article  Google Scholar 

  35. Kendall, A., Hawke, J., Janz, D., et al.: Learning to drive in a day. Paper presented at the 2019 International Conference on Robotics and Automation, Montreal, QC, Canada, 20–24 May (2019)

  36. Zhu, M.X., Wang, X.S., Wang, Y.H.: Human-like autonomous car-following model with deep reinforcement learning. Transp. Res. Part C Emerg. Technol. 97, 348–368 (2018)

    Article  Google Scholar 

  37. Ye, Y.J., Zhang, X.H., Sun, J.: Automated vehicle’s behavior decision making using deep reinforcement learning and high-fidelity simulation environment. Transp. Res. Part C Emerg. Technol. 107, 155–170 (2019)

    Article  Google Scholar 

  38. Zhou, M.F., Yu, Y., Qu, X.B.: Development of an efficient driving strategy for connected and automated vehicles at signalized intersections: a reinforcement learning approach. IEEE Trans. Intell. Transp. Syst. 21(1), 433–443 (2020)

    Article  Google Scholar 

  39. Qi, X.W., Luo, Y.D., Wu, G.Y., et al.: Deep reinforcement learning enabled self-learning control for energy efficient driving. Transp. Res. Part C Emerg. Technol. 99, 67–81 (2019)

    Article  Google Scholar 

  40. Bellman, R.: A Markovian decision process. J. Math. Mech. 6(5), 679–684 (1957)

    MathSciNet  MATH  Google Scholar 

  41. Arulkumaran, K., Deisenroth, M.P., Brundage, M., et al.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017)

    Article  Google Scholar 

  42. Dolcetta, I.C., Ishii, H.: Approximate solutions of the Bellman equation of deterministic control theory. Appl. Math. Optim. 11(1), 161–181 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  43. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  44. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  45. Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between epsilon-greedy and softmax. Paper presented at the 34th Annual German conference on Advances in artificial intelligence, Heidelberg, Berlin, 4–7 October (2011)

  46. Wisch, M., Hellmann, A., Lerner, M., et al.: Car-to-car accidents at intersections in Europe and identification of use cases for the test and assessment of respective active vehicle safety systems. Paper presented at 26th International Technical Conference on the Enhanced Safety of Vehicles, Eindhoven, Netherlands, 10–13 June (2019)

  47. Kusano, K.D., Gabler, H.C.: Target population for intersection advanced driver assistance systems in the U.S. SAE Int. J. Transp. Saf. 3(1), 1–16 (2015)

    Article  Google Scholar 

  48. Arikere, A., Yang, D.R., Klomp, M.: Optimal motion control for collision avoidance at left turn across path/opposite direction intersection scenarios using electric propulsion. Veh. Syst. Dyn. 57(5), 637–664 (2019)

    Article  Google Scholar 

  49. Sander, U., Lubbe, N., Pietzsch, S.: Intersection AEB implementation strategies for left turn across path crashes. Traffic Inj. Prev. 20(sup1), S119–S125 (2019)

    Article  Google Scholar 

  50. Ulrich, S., Nils, L.: The potential of clustering methods to define intersection test scenarios: assessing real-life performance of AEB. Accid. Anal. Prev. 113, 1–11 (2018)

    Article  Google Scholar 

  51. Dosovitskiy, A., Ros, G., Codevilla, F., et al.: CARLA: an open urban driving simulator (2017). arXiv preprint arXiv:1711.03938

  52. Eidehall, A., Petersson, L.: Statistical threat assessment for general road scenes using Monte Carlo sampling. IEEE Trans. Intell. Transp. Syst. 9(1), 137–147 (2008)

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (Grant No. 51805332), the Young Elite Scientists Sponsorship Program funded by the China Society of Automotive Engineers, the Natural Science Foundation of Guangdong Province (Grant No. 2018A030310532), and the Shenzhen Fundamental Research Fund (Grant No. JCYJ20190808142613246).

Author information

Authors and Affiliations

  1. Institute of Human Factors and Ergonomics, College of Mechatronics and Control Engineering, Shenzhen University, Shenzhen, 518060, China

    Guofa Li, Shenglong Li & Xingda Qu

  2. Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, N2L 3G1, Canada

    Guofa Li & Dongpu Cao

  3. Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA

    Shen Li

  4. School of Mechanical Engineering, Beijing Institute of Technology, Beijing, 100081, China

    Yechen Qin

  5. State Key Laboratory of Automotive Safety and Energy, School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

    Bo Cheng

Authors
  1. Guofa Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shenglong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yechen Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dongpu Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xingda Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bo Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shen Li.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding authors state that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Li, S., Li, S. et al. Deep Reinforcement Learning Enabled Decision-Making for Autonomous Driving at Intersections. Automot. Innov. 3, 374–385 (2020). https://doi.org/10.1007/s42154-020-00113-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42154-020-00113-1

Keywords

Search

Navigation