Applied Intelligence

, Volume 40, Issue 2, pp 201–213

Hierarchical control of traffic signals using Q-learning with tile coding

  • Monireh Abdoos
  • Nasser Mozayani
  • Ana L. C. Bazzan

DOI: 10.1007/s10489-013-0455-3

Cite this article as:
Abdoos, M., Mozayani, N. & Bazzan, A.L.C. Appl Intell (2014) 40: 201. doi:10.1007/s10489-013-0455-3


Multi-agent systems are rapidly growing as powerful tools for Intelligent Transportation Systems (ITS). It is desirable that traffic signals control, as a part of ITS, is performed in a distributed model. Therefore agent-based technologies can be efficiently used for traffic signals control. For traffic networks which are composed of multiple intersections, distributed control achieves better results in comparison to centralized methods. Hierarchical structures are useful to decompose the network into multiple sub-networks and provide a mechanism for distributed control of the traffic signals.

In this paper, a two-level hierarchical control of traffic signals based on Q-learning is presented. Traffic signal controllers, located at intersections, can be seen as autonomous agents in the first level (at the bottom of the hierarchy) which use Q-learning to learn a control policy. The network is divided into some regions where an agent is assigned to control each region at the second level (top of the hierarchy). Due to the combinational explosion in the number of states and actions, i.e. features, the use of Q-learning is impractical. Therefore, in the top level, tile coding is used as a linear function approximation method.

A network composed of 9 intersections arranged in a 3×3 grid is used for the simulation. Experimental results show that the proposed hierarchical control improves the Q-learning efficiency of the bottom level agents. The impact of the parameters used in tile coding is also analyzed.


Multi-agent systems Hierarchical control Traffic signals Q-learning Tile coding 

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Monireh Abdoos
    • 1
  • Nasser Mozayani
    • 1
  • Ana L. C. Bazzan
    • 2
  1. 1.School of Computer EngineeringIran University of Science and TechnologyTehranIran
  2. 2.Institute of InformaticsFederal University of Rio Grande do SulPorto AlegreBrazil

Personalised recommendations