Hierarchical control of traffic signals using Q-learning with tile coding
- 667 Downloads
Multi-agent systems are rapidly growing as powerful tools for Intelligent Transportation Systems (ITS). It is desirable that traffic signals control, as a part of ITS, is performed in a distributed model. Therefore agent-based technologies can be efficiently used for traffic signals control. For traffic networks which are composed of multiple intersections, distributed control achieves better results in comparison to centralized methods. Hierarchical structures are useful to decompose the network into multiple sub-networks and provide a mechanism for distributed control of the traffic signals.
In this paper, a two-level hierarchical control of traffic signals based on Q-learning is presented. Traffic signal controllers, located at intersections, can be seen as autonomous agents in the first level (at the bottom of the hierarchy) which use Q-learning to learn a control policy. The network is divided into some regions where an agent is assigned to control each region at the second level (top of the hierarchy). Due to the combinational explosion in the number of states and actions, i.e. features, the use of Q-learning is impractical. Therefore, in the top level, tile coding is used as a linear function approximation method.
A network composed of 9 intersections arranged in a 3×3 grid is used for the simulation. Experimental results show that the proposed hierarchical control improves the Q-learning efficiency of the bottom level agents. The impact of the parameters used in tile coding is also analyzed.
KeywordsMulti-agent systems Hierarchical control Traffic signals Q-learning Tile coding
The first author would like to thank Research Institute for Information and Communication Technology—ITRC (Tehran, Iran) for their supports. Ana L.C. Bazzan is partially supported by CNPq.
- 1.Li H, Li Z, White RT, Wu X (2013) A real-time transportation prediction system. Int J Appl Intell, published online Google Scholar
- 4.Bielli M, Ambrosino G, Boero M (1994) Artificial intelligence applications to traffic engineering. VSP, Vermont Google Scholar
- 10.Cai C, Yang Z (2007) Study on urban traffic management based on multi-agent system. In: Proceedings of the sixth international conference on machine learning and cybernetics. IEEE, Hong Kong, pp 25–29 Google Scholar
- 11.Chen C, Li Z (2012) A hierarchical networked urban traffic signal control system based on multi-agent. In: 9th IEEE international conference on networking, sensing and control (ICNSC). IEEE, New York, pp 28–33 Google Scholar
- 14.Grégoire P, Desjardins C, Laumônier J, Chaib-draa B (2007) Urban traffic control based on learning agents. In: Intelligent transportation systems conference. IEEE, New York, pp 916–921 Google Scholar
- 15.Weiring M (2000) Multi-agent reinforcement learning for traffic light control. In: Proceedings of the seventh international conference on machine learning, pp 1151–1158 Google Scholar
- 16.Steingröver M, Schouten R, Peelen S, Nijhuis E, Bakker B (2005) Reinforcement learning of traffic light controllers adapting to traffic congestion. In: Proceedings of the 17th Belgium-Netherlands conference on artificial intelligence (BNAIC 2005), Citeseer, 2005, pp 216–223 Google Scholar
- 17.Silva BBCd, Basso EW, Bazzan ALC, Engel PM (2006) Improving reinforcement learning with context detection. In: Proceedings of the 5th international joint conference on autonomous agents and multiagent systems (AAMAS 2006), Hakodate, Japan. ACM Press, New York, pp 811–812. Available online: www.inf.ufrgs.br/maslab/pergamus/pubs/Silva+2006.pdf Google Scholar
- 18.Wen K, Qu S, Zhang Y (2008) A stochastic adaptive control model for isolated intersections. In: Proceedings of the 2007 IEEE international conference on robotics and biomimetics. Sanya, China. IEEE, New York, pp 2256–2260 Google Scholar
- 22.Vien NA, Wolfgang E, Chung TC (2013) Learning via human feedback in continuous state and action spaces. Int J Appl Intell, published online Google Scholar
- 24.Sutton R, Barto A (1998) Reinforcement learning—an introduction. MIT Press, Cambridge Google Scholar
- 26.Reynolds S (2002) Reinforcement learning with exploration. PhD dissertation, School of Computer Science, The University of Birmingham, Birmingham Google Scholar
- 27.Sutton R (1996) Generalization in reinforcement learning: successful examples using sparse coarse coding. Adv Neural Inf Process Syst 8:1038–1044 Google Scholar
- 28.Haykin S (2002) Adaptive filter theory. Prentice-Hall information and system sciences series Google Scholar