Abstract
A traffic signal is a fundamental part of the traffic control system to reduce congestion and enhance safety. Since the inception of motorized vehicles, traffic signal controllers are put in place to coordinate and maintain traffic flow. With the number of vehicles on the road increasing exponentially, it is imperative to innovate new traffic control frameworks to cope with the high-density traffic demand. In this regard, recent advances in machine/deep learning have enabled significant progress towards reducing congestion using reinforcement learning for traffic signal control. However, most of these works are still not ready for deployment due to assumptions of perfect knowledge of the traffic environment. In reality, congestion detection or prediction systems are at best able to approximate the traffic state with significant noise. In this work, we propose a robust training framework for reinforcement learning agents that can handle such noisy approximation of the traffic states. Specifically, we show that by carefully adding synthetic perturbations to the state space, such as the queue length during training, the reinforcement learning agents can be robustified. Conceptually, our approach is similar to adversarial training schemes and can lead to successful deployment of reinforcement learning agent-based traffic signal controllers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arel I, Liu C, Urbanik T, Kohls AG (2010) Reinforcement learning-based multi-agent system for network traffic signal control. IET Intell Trans Syst 4(2):128–135
Barceló J, Casas J (2005) Dynamic network simulation with aimsun. In: Simulation approaches in transportation analysis, pp. 57–98. Springer
Bellman R (1966) Dynamic programming. Science 153(3731):34–37
Chakraborty P, Adu-Gyamfi YO, Poddar S, Ahsani V, Sharma A, Sarkar S (2018) Traffic congestion detection from camera images using deep convolution neural networks. Trans Res Record 2672(45):222–231
Chu T, Wang J, Codecà L, Li Z (2019) Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans Intell Trans Syst 21(3):1086–1095
Cohen JM, Rosenfeld E, Kolter JZ (2019) Certified adversarial robustness via randomized smoothing. CoRR abs/1902.02918. arXiv:abs/1902.02918
Esfandiari Y, Ebrahimi K, Balu A, Elia N, Vaidya U, Sarkar S (2019) A saddle-point dynamical system approach for robust deep learning. arXiv preprint arXiv:1910.08623
Group P (2019) Vissim. http://vision-traffic.ptvgroup.com/en-us/products/ptv-vissim/
Havens A, Jiang Z, Sarkar S (2018) Online robust policy learning in the presence of unknown adversaries. In: Advances in neural information processing systems 9916–9926
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778
Joshi A, Mukherjee A, Sarkar S, Hegde C (2019) Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In: Proceedings of the IEEE international conference on computer vision 4773–4783
Koonce P, Rodegerdts L, Lee K, Quayle S, Beaird S, Braud C, Bonneson J, Tarnoff P, Urbanik T (2010) Traffic Signal Timing Manual: Chapter 5 - Office of Operations http://ops.fhwa.dot.gov/publications/fhwahop08024/chapter5.htm
Koonce P, Rodegerdts L, Lee K, Quayle S, Beaird S, Braud C, Bonneson J, Tarnoff P, Urbanik T (2010) Traffic signal timing manual: Chapter 6 - Detectors. http://ops.fhwa.dot.gov/publications/fhwahop08024/chapter5.htm
Krajzewicz D, Hertkorn G, Rössel C, Wagner P (2002) Sumo (simulation of urban mobility)-an open-source traffic simulation. In: Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002), 183–187
Kurakin A, Goodfellow I, Bengio S (2016) Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236
Lasley P (2019) 2019 urban mobility report
Lee XY, Ghadai S, Tan KL, Hegde C, Sarkar S (2020) Spatiotemporally constrained action space attacks on deep reinforcement learning agents. In: AAAI, 4577–4584
Li B, Cheng W, Li L (2018) Real-time prediction of lane-based queue lengths for signalized intersections. J Adv Trans 2018:
Li L, Lv Y, Wang FY (2016) Traffic signal timing via deep reinforcement learning. IEEE/CAA J Automatica Sinica 3(3):247–254
Li X, Ouyang Y (2012) Reliable traffic sensor deployment under probabilistic disruptions and generalized surveillance effectiveness measures. Oper Res 60(5):1183–1198
Liang X, Du X, Wang G, Han Z (2019) A deep reinforcement learning network for traffic light cycle control. IEEE Trans Vehicular Technol 68(2):1243–1253
Lin Y, Dai X, Li L, Wang FY (2018) An efficient deep reinforcement learning model for urban traffic control. arXiv preprint arXiv:1808.01876
Liu C, Zhao M, Sharma A, Sarkar S (2019) Traffic dynamics exploration and incident detection using spatiotemporal graphical modeling. J Big Data Anal Trans 1(1):37–55
Liu HX, Wu X, Ma W, Hu H (2009) Real-time queue length estimation for congested signalized intersections. Trans Res Part C 17(4):412–427
Liu M, Deng J, Xu M, Zhang X, Wang W (2017) Cooperative deep reinforcement learning for tra ic signal control. In: The 7th international workshop on Urban computing (UrbComp 2018)
Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2017) Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083
Miller AJ (1963) Settings for fixed-cycle traffic signals. J Oper Res Soc 14(4):373–386
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533
Morimoto J, Doya K (2005) Robust reinforcement learning. Neural Comput 17(2):335–359
Mukherjee A, Joshi A, Hegde C, Sarkar S. Semantic domain adaptation for deep classifiers via gan-based data augmentation
Van der Pol E, Oliehoek FA (2016) Coordinated deep reinforcement learners for traffic light control. In: Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016)
Rajagopal R, Varaiya PP (2007) Health of California’s loop detector system. University of., California PATH Program, Institute of Transportation Studies
Rodrigues F, Azevedo CL (2019) Towards robust deep reinforcement learning for traffic signal control: Demand surges, incidents and sensor failures. In: 2019 IEEE intelligent transportation systems conference (ITSC), pp. 3559–3566. IEEE
Roess RP, Prassas ES, McShane WR (2004) Traffic engineering. Pearson/Prentice Hall
Sharma A, Bullock DM, Bonneson JA (2007) Input-output and hybrid techniques for real-time prediction of delay and maximum queue length at signalized intersections. Trans Res Record 2035(1):69–80
Tan KL, Esfandiari Y, Lee XY, Sarkar S, et al (2020) Robustifying reinforcement learning agents via action space adversarial training. In: 2020 American control conference (ACC), pp. 3959–3964. IEEE
Tan KL, Poddar S, Sarkar S, Sharma A (2019) Deep reinforcement learning for adaptive traffic signal control. In: ASME 2019 dynamic systems and control conference. American Society of Mechanical Engineers Digital Collection
Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Thirtieth AAAI conference on artificial intelligence
Varaiya P (2013) Max pressure control of a network of signalized intersections. Trans Res Part C 36:177–195
Webster F (1958) Traffic signal settings, road research technical paper no. 39. Road Research Laboratory
Wei H, Chen C, Wu K, Zheng G, Yu Z, Gayah V, Li Z (2019) Deep reinforcement learning for traffic signal control along arterials
Wei H, Zheng G, Yao H, Li Z (2018) Intellilight: A reinforcement learning approach for intelligent traffic light control. In: Proceedings of the 24th ACM SIGKDD International conference on knowledge discovery & data mining, 2496–2505
Wiering M (2000) Multi-agent reinforcement learning for traffic light control. In: Machine learning: proceedings of the seventeenth international conference (ICML’2000), 1151–1158
Wu L, Liu C, Huang T, Sharma A, Sarkar S (2017) Traffic sensor health monitoring using spatiotemporal graphical modeling. In: Proceedings of the 2nd ACM SIGKDD workshop on machine learning for prognostics and health management, 13–17
Acknowledgements
This work was supported in part by NSF grant CNS-1845969. We would like to thank Subhadipto Poddar for his insights in general traffic intersection knowledge.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
Anuj Sharma works for ETALYC Inc. that works in the area of adaptive signal control.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tan, K.L., Sharma, A. & Sarkar, S. Robust Deep Reinforcement Learning for Traffic Signal Control. J. Big Data Anal. Transp. 2, 263–274 (2020). https://doi.org/10.1007/s42421-020-00029-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42421-020-00029-6