Skip to main content
SpringerLink
Log in

Co-operative Multi-agent Twin Delayed DDPG for Robust Phase Duration Optimization of Large Road Networks

  • Conference paper
  • First Online:
Agents and Artificial Intelligence (ICAART 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13786))

Included in the following conference series:

  • 236 Accesses

Abstract

Large road networks overflowing with vehicles have called for increased traffic congestion, the impact of which is felt on an everyday basis and across different dimensions like decreased traveller satisfaction, increased fuel usage and increased air pollution among many other troubles. Improved traffic control strategies that can self-learn to adapt their decisioning in response to dynamic changes in the traffic flows and are capable of mitigating overall network congestion as opposed to localized congestion at intersections, are of great importance in mitigating traffic congestion. Traffic control strategies which were rule-based or historical-demand based were over-simplified and could not scale to large real-world road networks. To effectively control traffic congestion at scale, the need for co-operation and communication between the different intersections of a large road network is crucial. Multi-agent reinforcement learning methods are an apt choice for traffic signal control of large scale road networks as they can learn to perform predictive control actions that will reduce overall network congestion dynamically at scale. In this paper, we extend the work done in [24] to traffic signal timing (green phase duration) control using Multi-agent Twin Delayed Deep Deterministic Policy Gradients (MATD3) on large scale real-world road networks. The solution strategy was exposed to simulations of different road networks and time-varying traffic flows. The experimental results showed that our strategy is robust to the different kinds of road networks and vehicular traffic flows, and consistently outperformed its adaptive and rule-based counterparts by significantly reducing the average vehicular delay and queue length.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdulhai, B., Pringle, R., Karakoulas, G.J.: Reinforcement learning for true adaptive traffic signal control. J. Transp. Eng. 129(3) (2003)

    Google Scholar 

  2. Ackermann, J., Gabler, V., Osa, T., Sugiyama, M.: Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint arXiv:1910.01465 (2019)

  3. Balaji, P., Srinivasan, D.: Type-2 Fuzzy Logic based urban traffic management. Eng. Appl. Artif. Intell. 24(1), 12–22 (2011). https://doi.org/10.1016/j.engappai.2010.08.007. https://www.sciencedirect.com/science/article/pii/S0952197610001624

  4. Bellman, R.: A Markovian decision process. J. Math. Mech. 679–684 (1957)

    Google Scholar 

  5. Chu, T., Wang, J., Codecà, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2019)

    Article  Google Scholar 

  6. Dunne, M.C., Potts, R.B.: Algorithm for traffic control. Oper. Res. 12(6), 870–881 (1964)

    Article  MATH  Google Scholar 

  7. Gazis, D.C.: Optimum control of a system of oversaturated intersections. Oper. Res. 12(6), 815–831 (1964)

    Article  MATH  Google Scholar 

  8. Ge, H., Song, Y., Wu, C., Ren, J., Tan, G.: Cooperative deep q-learning with q-value transfer for multi-intersection signal control. IEEE Access 7, 40797–40809 (2019)

    Article  Google Scholar 

  9. Genders, W., Razavi, S.: Using a deep reinforcement learning agent for traffic signal control. arXiv preprint arXiv:1611.01142 (2016)

  10. Head, L., Mirchandani, P., Shelby, S.: The Rhodes prototype: a description and some results (1998)

    Google Scholar 

  11. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  12. Prabuchandran, K.J., Hemanth Kumar, A.N., Bhatnagar, S.: Multi-agent reinforcement learning for traffic signal control. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 2529–2534 (2014). https://doi.org/10.1109/ITSC.2014.6958095

  13. Prashanth, L.A., Bhatnagar, S.: Threshold tuning using stochastic optimization for graded signal control. IEEE Trans. Veh. Technol. 61(9), 3865–3880 (2012). https://doi.org/10.1109/TVT.2012.2209904

    Article  Google Scholar 

  14. Prashanth, L.A., Bhatnagar, S.: Reinforcement learning with function approximation for traffic signal control. IEEE Trans. Intell. Transp. Syst. 12(2), 412–421 (2011). https://doi.org/10.1109/TITS.2010.2091408

    Article  Google Scholar 

  15. Li, L., Lv, Y., Wang, F.Y.: Traffic signal timing via deep reinforcement learning. IEEE/CAA J. Automatica Sinica 3(3), 247–254 (2016)

    Article  Google Scholar 

  16. Liang, X., Du, X., Wang, G., Han, Z.: Deep reinforcement learning for traffic light control in vehicular networks. arXiv preprint arXiv:1803.11115 (2018)

  17. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

  18. Lowe, R., Wu, Y.I., Tamar, A., Harb, J., Pieter Abbeel, O., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  19. Lowrie, P.: Scats: Sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic (1990)

    Google Scholar 

  20. Mauro, V., Di Taranto, C.: Utopia. IFAC Proc. Volumes 23(2), 245–252 (1990). https://doi.org/10.1016/S1474-6670(17)52678-6. https://www.sciencedirect.com/science/article/pii/S1474667017526786. iFAC/IFIP/IFORS Symposium on Control, Computers, Communications in Transportation, Paris, France, 19–21 September

  21. Prashanth, L.A., Bhatnagar, S.: Reinforcement learning with average cost for adaptive control of traffic lights at intersections. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 1640–1645 (2011). https://doi.org/10.1109/ITSC.2011.6082823

  22. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)

    MATH  Google Scholar 

  23. Riccardo, R., Massimiliano, G.: An empirical analysis of vehicle time headways on rural two-lane two-way roads. Procedia Soc. Behav. Sci. 54, 865–874 (2012)

    Article  Google Scholar 

  24. Shanmugasundaram, P., Bhatnagar, S.: Robust traffic signal timing control using multiagent twin delayed deep deterministic policy gradients. In: ICAART (2), pp. 477–485 (2022)

    Google Scholar 

  25. Shanmugasundaram, P., Sinha, A.: Intelligent traffic control using double deep q networks for time-varying traffic flows. In: 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN), pp. 64–69 (2021). https://doi.org/10.1109/SPIN52536.2021.9565961

  26. Smith, M.: Traffic control and route-choice; a simple example. Transp. Res. Part B: Methodol. 13(4), 289–294 (1979)

    Article  Google Scholar 

  27. Tampuu, A., et al.: Multiagent cooperation and competition with deep reinforcement learning. CoRR abs/1511.08779 (2015). http://arxiv.org/abs/1511.08779

  28. Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shiv Nadar University, Greater Noida, Uttar Pradesh, India

    Priya Shanmugasundaram

  2. Indian Institute of Science, Bangalore, Karnataka, India

    Shalabh Bhatnagar

Authors
  1. Priya Shanmugasundaram
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shalabh Bhatnagar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Priya Shanmugasundaram .

Editor information

Editors and Affiliations

  1. LIACC/FEUP, Porto, Portugal

    Ana Paula Rocha

  2. ICREA, Institute of Evolutionary Biology, Barcelona, Spain

    Luc Steels

  3. Leiden Institute of Advanced Computer Science, Leiden, The Netherlands

    Jaap van den Herik

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shanmugasundaram, P., Bhatnagar, S. (2022). Co-operative Multi-agent Twin Delayed DDPG for Robust Phase Duration Optimization of Large Road Networks. In: Rocha, A.P., Steels, L., van den Herik, J. (eds) Agents and Artificial Intelligence. ICAART 2022. Lecture Notes in Computer Science(), vol 13786. Springer, Cham. https://doi.org/10.1007/978-3-031-22953-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22953-4_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22952-7

  • Online ISBN: 978-3-031-22953-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation