Abstract
We study approaches to use (real-time) data, communicated between cars and infrastructure, to improve and to optimize traffic flow in the future and, thereby, to support holistic, efficient and sustainable mobility solutions. To set up virtual traffic environments ranging from artificial scenarios up to complex real world road networks, we use microscopic traffic models and traffic simulation software SUMO. In particular, we apply a reinforcement learning approach, in order to teach controllers (agents) to guide certain vehicles or to control infrastructural guidance systems, such as traffic lights. With real-time information obtained from other vehicles, the agent iteratively learns to improve the traffic flow by repetitive observation and algorithmic optimization. For the RL approach, we consider different control policies including widely used neural nets but also Linear Models and Radial Basis Function Networks. Finally, we compare our RL controller with other control approaches and analyse the robustness of the RL traffic light controller, especially under extreme scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arel, I., Liu, C., Urbanik, T., Kohls, A.: Reinforcement learning-based multi-agent system for network traffic signal control. IET Intel. Transp. Syst. 4, 128–135 (2010). https://doi.org/10.1049/iet-its.2009.0070
Baumgart, U., Burger, M.: A reinforcement learning approach for traffic control. In: Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems - VEHITS, pp. 133–141. INSTICC, SciTePress (2021). https://doi.org/10.5220/0010448501330141
Belletti, F., Haziza, D., Gomes, G., Bayen, A.: Expert level control of ramp metering based on multi-task deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. 19(4), 1198–1207 (2017). https://doi.org/10.1109/TITS.2017.2725912
Bhattacharyya, R.P., Senanayake, R., Brown, K., Kochenderfer, M.J.: Online parameter estimation for human driver behavior prediction. In: 2020 American Control Conference (ACC), pp. 301–306 (2020). https://doi.org/10.23919/ACC45564.2020.9147924
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer-Verlag, New York (2006). https://doi.org/10.1108/03684920710743466
Burger, M., Goettlich, S., Jung, T.: Derivation of second order traffic flow models with time delays. Netw. Heterogen. Media 14, 265–288 (2019). https://doi.org/10.3934/nhm.2019011
De Schutter, B., De Moor, B.: Optimal traffic light control for a single intersection. Eur. J. Control. 4(3), 260–276 (1998). https://doi.org/10.1016/S0947-3580(98)70119-0
Duan, Y., Chen, X., Houthooft, R., Schulman, J., Abbeel, P.: Benchmarking deep reinforcement learning for continuous control. In: Proceedings of The 33rd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 48, pp. 1329–1338 (2016). https://doi.org/10.5555/3045390.3045531
Feinberg, E., Shwartz, A.: Handbook of Markov Decision Processes: Methods and Applications. Springer, US (2002). https://doi.org/10.1007/978-1-4615-0805-2
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., Pineau, J.: An introduction to deep reinforcement learning. Found. Trends® Mach. Learn. 11(3–4), 219–354 (2018). https://doi.org/10.1561/2200000071
Helbing, D.: Verkehrsdynamik. Springer-Verlag Berlin Heidelberg (1997). https://doi.org/10.1007/978-3-319-78695-7
Kalashnikov, D., et al.: QT-Opt: scalable deep reinforcement learning for vision-based robotic manipulation. In: Proceedings of The 2nd Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 87, pp. 651–673 (2018)
Kessels, F.: Traffic Flow Modelling. Springer International Publishing (2019). https://doi.org/10.1007/978-3-642-59063-4
Lighthill, M.J., Whitham, G.B.: On kinematic waves II. A theory of traffic flow on long crowded roads. Proc. R. Soc. Lond. A. 229, 317–345 (1955). https://doi.org/10.1098/rspa.1955.0089
Lopez, P.A., et al.: Microscopic traffic simulation using SUMO. In: The 21st IEEE International Conference on Intelligent Transportation Systems (2018). https://doi.org/10.1109/ITSC.2018.8569938
Lu, X.Y., Varaiya, P., Horowitz, R., Su, D., Shladover, S.E.: Novel freeway traffic control with variable speed limit and coordinated ramp metering. Transp. Res. Rec. J. Transp. Res. Board 2229(1), 55–65 (2011). https://doi.org/10.3141/2229-07
Mania, H., Guy, A., Recht, B.: Simple random search provides a competitive approach to reinforcement learning. arXiv: 1803.07055 (2018)
McNeil, D.R.: A solution to the fixed-cycle traffic light problem for compound poisson arrivals. J. Appl. Probab. 5(3), 624–635 (1968). https://doi.org/10.2307/3211926
Murphy, K.P.: Machine Learning: A Probabilistic Perspective. The MIT Press (2012)
OpenStreetMap (2021). https://www.openstreetmap.org. Accessed 22 Oct 2021
Orosz, G.: Connected cruise control: modelling, delay effects, and nonlinear behaviour. Veh. Syst. Dyn. 54(8), 1147–1176 (2016). https://doi.org/10.1080/00423114.2016.1193209
Orosz, G., Wilson, R.E., Stépán, G.: Traffic jams: dynamics and control. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 368(1928), 4455–4479 (2010). https://doi.org/10.1098/rsta.2010.0205
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1st (edn) (1994). https://doi.org/10.1002/9780470316887
Rajeswaran, A., et al.: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In: Proceedings of Robotics: Science and Systems (RSS) (2018). https://doi.org/10.15607/RSS.2018.XIV.049
Silver, D., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140–1144 (2018). https://doi.org/10.1126/science.aar6404
Stern, R.E., et al.: Dissipation of stop-and-go waves via control of autonomous vehicles: field experiments. Transp. Res. Part C Emerg. Technol. 89, 205–221 (2018). https://doi.org/10.1016/j.trc.2018.02.005
Sugiyama, Y., et al.: Traffic jams without bottlenecks—experimental evidence for the physical mechanism of the formation of a jam. New J. Phys. 10(3), 033001 (2008). https://doi.org/10.1088/1367-2630/10/3/033001
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. Adaptive Computation and Machine Learning, MIT Press (2018)
The MathWorks Inc: Reinforcement Learning Toolbox. Matlab R2020b. Natick, Massachusetts, United State (2021). https://www.mathworks.com/help/reinforcement-learning/
Treiber, M., Kesting, A.: Traffic Flow Dynamics. Springer-Verlag, Berlin Heidelberg (2013). https://doi.org/10.1007/978-3-642-32460-4
Vinitsky, E., et al.: Benchmarks for reinforcement learning in mixed-autonomy traffic. In: Proceedings of The 2nd Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 87, pp. 399–409 (2018)
Wang, J., Zheng, Y., Xu, Q., Wang, J., Li, K.: Controllability analysis and optimal control of mixed traffic flow with human-driven and autonomous vehicles. IEEE Trans. Intell. Transp. Syst. 22(12), 1–15 (2020). https://doi.org/10.1109/TITS.2020.3002965
Wiering, M.: Multi-agent reinforcement leraning for traffic light control. In: Proceedings of the Seventeenth International Conference on Machine Learning, pp. 1151–1158. ICML 2000 (2000)
Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., Bayen, A.M.: Flow: Architecture and Benchmarking for Reinforcement Learning in Traffic Control. arXiv: 1710.05465 (2017)
Zheng, Y., Wang, J., Li, K.: Smoothing traffic flow via control of autonomous vehicles. IEEE Internet Things J. 7(5), 3882–3896 (2020). https://doi.org/10.1109/JIOT.2020.2966506
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Baumgart, U., Burger, M. (2022). Optimal Control of Traffic Flow Based on Reinforcement Learning. In: Klein, C., Jarke, M., Helfert, M., Berns, K., Gusikhin, O. (eds) Smart Cities, Green Technologies, and Intelligent Transport Systems. VEHITS SMARTGREENS 2021 2021. Communications in Computer and Information Science, vol 1612. Springer, Cham. https://doi.org/10.1007/978-3-031-17098-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-17098-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-17097-3
Online ISBN: 978-3-031-17098-0
eBook Packages: Computer ScienceComputer Science (R0)