Skip to main content
SpringerLink
Log in

Graph-based multi agent reinforcement learning for on-ramp merging in mixed traffic

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The application of Deep Reinforcement Learning (DRL) has significantly impacted the development of autonomous driving technology in the field of intelligent transportation. However, in mixed traffic scenarios involving both human-driven vehicles (HDVs) and connected and autonomous vehicles (CAVs), challenges arise, particularly concerning information sharing and collaborative control among multiple intelligent agents using DRL. To address this issue, we propose a novel framework, namely Spatial-Temporal Deep Reinforcement Learning (ST-DRL), that enables collaborative control among multiple CAVs in mixed traffic scenarios. Initially, the traffic states involving multiple agents are constructed as graph-formatted data, which is then sequential created to represent continuous time intervals. With the data representation, interactive behaviors and dynamic characteristics among multiple intelligent agents are implicitly captured. Subsequently, to better represent the spatial relationships between vehicles, a graph enabling network is utilize to encode the vehicle states, which can contribute to the improvement of information sharing efficiency among multiple intelligent agents. Additionally, a spatial-temporal feature fusion network module is designed, which integrates graph convolutional networks (GCN) and gated recurrent units (GRU). It can effectively fuse independent spatial-temporal features and further enhance collaborative control performance. Through extensive experiments conducted in the SUMO traffic simulator and comparison with baseline methods, it is demonstrated that the ST-DRL framework achieves higher success rates in mixed traffic scenarios and exhibits better trade-offs between safety and efficiency. The analysis of the results indicates that ST-DRL has increased the success rate of the task by \(15.6\%\) compared to the baseline method, while reducing model training and task completion times by \(26.6\%\) respectively.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Chen L, Li Y, Huang C, Li B, Xing Y, Tian D, Li L, Hu Z, Na X, Li Z et al (2022) Milestones in autonomous driving and intelligent vehicles: survey of surveys. IEEE Trans Intell Vehicles 8(2):1046–1056

    Article  Google Scholar 

  2. Gao C, Wang G, Shi W, Wang Z, Chen Y (2021) Autonomous driving security: state of the art and challenges. IEEE Internet Things J 9(10):7572–7595

    Article  Google Scholar 

  3. Qiu W, Ting Q, Shuyou Y, Hongyan G, Hong C (2015) Autonomous vehicle longitudinal following control based on model predictive control. In: 2015 34th Chinese Control Conference (CCC), IEEE, pp 8126–8131

  4. Han G, Fu W, Wang W, Wu Z (2017) The lateral tracking control for the intelligent vehicle based on adaptive pid neural network. Sensors 17(6):1244

    Article  Google Scholar 

  5. Hosseinzadeh M, Sinopoli B, Kolmanovsky I, Baruah S (2022) Implementing optimization-based control tasks in cyber-physical systems with limited computing capacity. In: 2022 2nd International workshop on computation-aware algorithmic design for cyber-physical systems (CAADCPS), IEEE pp 15–16

  6. Kiran BR, Sobh I, Talpaert V, Mannion P, Al Sallab AA, Yogamani S, Pérez P (2021) Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst 23(6):4909–4926

    Article  Google Scholar 

  7. Lee D-H, Chen K-L, Liou K-H, Liu C-L, Liu J-L (2021) Deep learning and control algorithms of direct perception for autonomous driving. Appl Intell 51(1):237–247

    Article  Google Scholar 

  8. Nakka SKS, Chalaki B, Malikopoulos AA (2022) A multi-agent deep reinforcement learning coordination framework for connected and automated vehicles at merging roadways. In: 2022 American Control Conference (ACC), IEEE, pp 3297–3302

  9. Chen S, Dong J, Ha P, Li Y, Labi S (2021) Graph neural network and reinforcement learning for multi-agent cooperative control of connected autonomous vehicles. Comput Aided Civ Infrastruct Eng 36(7):838–857

    Article  Google Scholar 

  10. Zhou M, Luo J, Villella J, Yang Y, Rusu D, Miao J, Zhang W, Alban M, Fadakar I, Chen Z et al (2020) Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv:2010.09776

  11. Wang H, Xie X, Zhou L (2023) Transform networks for cooperative multi-agent deep reinforcement learning. Appl Intell 53(8):9261–9269

    Article  Google Scholar 

  12. Munikoti S, Agarwal D, Das L, Halappanavar M, Natarajan B (2022) Challenges and opportunities in deep reinforcement learning with graph neural networks: a comprehensive review of algorithms and applications. arXiv:2206.07922

  13. Singh D, Srivastava R (2022) Graph neural network with rnns based trajectory prediction of dynamic agents for autonomous vehicle. Appl Intell 52(11):12801–12816

    Article  Google Scholar 

  14. Kendall A, Hawke J, Janz D, Mazur P, Reda D, Allen J-M, Lam V-D, Bewley A, Shah A (2019) Learning to drive in a day. In: 2019 International conference on robotics and automation (ICRA), IEEE, pp 8248–8254

  15. Chen J, Yuan B, Tomizuka M (2019) Model-free deep reinforcement learning for urban autonomous driving. In: 2019 IEEE Intelligent transportation systems conference (ITSC), IEEE, pp 2765–2771

  16. Yan Z, Kreidieh AR, Vinitsky E, Bayen AM, Wu C (2022) Unified automatic control of vehicular systems with reinforcement learning. IEEE Trans Autom Sci Eng

  17. Dinneweth J, Boubezoul A, Mandiau R, Espié S (2022) Multi-agent reinforcement learning for autonomous vehicles: a survey. Autonomous Intell Syst 2(1):27

    Article  Google Scholar 

  18. Yin Q, Yu T, Shen S, Yang J, Zhao M, Ni W, Huang K, Liang B, Wang L (2024) Distributed deep reinforcement learning: a survey and a multi-player multi-agent learning toolbox. Mach Intell Res 1–20

  19. Zhou W, Chen D, Yan J, Li Z, Yin H, Ge W (2022) Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic. Autonomous Intell Syst 2(1):5

    Article  Google Scholar 

  20. Bhalla S, Ganapathi Subramanian S, Crowley M (2020) Deep multi agent reinforcement learning for autonomous driving. In: Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ottawa, ON, Canada, May 13–15, 2020, Proceedings, Springer, pp 67–78

  21. Palanisamy P (2020). Multi-agent connected autonomous driving using deep reinforcement learning. In: 2020 International joint conference on neural networks (IJCNN), IEEE, pp 1–7

  22. Han Z, Liang Y, Ohkura K (2023) Developing multi-agent adversarial environment using reinforcement learning and imitation learning. Artif Life Robot 28(4):703–709

    Article  Google Scholar 

  23. Canese L, Cardarilli GC, Di Nunzio L, Fazzolari R, Giardino D, Re M, Spanò S (2021) Multi-agent reinforcement learning: a review of challenges and applications. Appl Sci 11(11):4948

    Article  Google Scholar 

  24. Ye L, Wang Z, Chen X, Wang J, Wu K, Lu K (2021) Gsan: Graph self-attention network for learning spatial-temporal interaction representation in autonomous driving. IEEE Internet Things J 9(12):9190–9204

    Article  Google Scholar 

  25. Wang J, Shi T, Wu Y, Miranda-Moreno L, Sun L (2020) Multi-agent graph reinforcement learning for connected automated driving. In: Proceedings of the 37th international conference on machine learning (ICML), pp 1–6

  26. Wang S, Fujii H, Yoshimura S (2022) Generating merging strategies for connected autonomous vehicles based on spatiotemporal information extraction module and deep reinforcement learning. Phys A Stat Mech Appl 607:128172

    Article  Google Scholar 

  27. Peng Y, Tan G, Si H, Li J (2022) Drl-gat-sa: Deep reinforcement learning for autonomous driving planning based on graph attention networks and simplex architecture. J Syst Archit 126:102505

    Article  Google Scholar 

  28. Zhu J, Easa S, Gao K (2022) Merging control strategies of connected and autonomous vehicles at freeway on-ramps: a comprehensive review. J Intell Connect Veh 5(2):99–111

  29. Wang P, Chan C-Y (2017) Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge. In: 2017 IEEE 20th International conference on intelligent transportation systems (ITSC), IEEE, pp 1–6

  30. Ren T, Xie Y, Jiang L (2020) Cooperative highway work zone merge control based on reinforcement learning in a connected and automated environment. Transp Res Rec 2674(10):363–374

    Article  Google Scholar 

  31. Schester L, Ortiz LE (2021) Automated driving highway traffic merging using deep multi-agent reinforcement learning in continuous state-action spaces. In: 2021 IEEE Intelligent vehicles symposium (IV), IEEE, pp. 280–287

  32. He C, Sun D, Zhao M, Zhao H (2021) Cooperative group control strategy in the on-ramp area for connected and automated vehicles under mixed traffic environment. In: 2021 China Automation Congress (CAC), pp 7943–7948. https://doi.org/10.1109/CAC53003.2021.9728034

  33. Chen D, Hajidavalloo MR, Li Z, Chen K, Wang Y, Jiang L, Wang Y (2023) Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. IEEE Trans Intell Transp Syst

  34. Krajzewicz D, Hertkorn G, Rössel C, Wagner P (2002) Sumo (simulation of urban mobility)-an open-source traffic simulation. In: Proceedings of the 4th Middle East symposium on simulation and modelling (MESM20002), pp 183–187

  35. Bellman R (1957) A markovian decision process. J Math Mech 679–684

  36. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555

  37. Jia J, Xing X, Chang DE (2022) Gru-attention based td3 network for mobile robot navigation. In: 2022 22nd International conference on control, automation and systems (ICCAS), IEEE, pp 1642–1647

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant (62373325, 6190334), in part by the Zhejiang Provincial Natural Science Foundation under Grant (LY21F030016), and in part by the Fundamental Research Funds for the Central Universities, CHD (No.300102343503).

Author information

Author notes

  1. Dongwei Xu and Biao Zhang contributed equally to this work.

Authors and Affiliations

  1. Department of Institute of Cyberspace Security, Zhejiang University of Technology, 288 Liuhe Road, Xihu District, Hangzhou, 311121, Zhejiang, China

    Dongwei Xu & Haifeng Guo

  2. Information Systems in GS1 China and Oriental Speedy Code Tech, BeiJing, 100011, China

    Biao Zhang

  3. Department of College of Information Engineering, Zhejiang University of Technology, 288 Liuhe Road, Xihu District, Hangzhou, 311121, Zhejiang, China

    Qingwei Qiu

  4. College of Metropolitan Transportation, Beijing University of Technology, 100 Pingyuan Village, Boya Road, Chaoyang District, Beijing, 100021, China

    Haijian Li

  5. Key Laboratory of Transport Industry of Management, Chang’an University, Xian, 710064, Shanxi, China

    Baojie Wang

Authors
  1. Dongwei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Biao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingwei Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haijian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haifeng Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Baojie Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baojie Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, D., Zhang, B., Qiu, Q. et al. Graph-based multi agent reinforcement learning for on-ramp merging in mixed traffic. Appl Intell (2024). https://doi.org/10.1007/s10489-024-05478-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05478-y

Keywords

Search

Navigation