Abstract
In this paper we introduce a Multi-agent system that uses Reinforcement Learning (RL) techniques to learn local navigational behaviors to simulate virtual pedestrian groups. The aim of the paper is to study empirically the validity of RL to learn agent-based navigation controllers and their transfer capabilities when they are used in simulation environments with a higher number of agents than in the learned scenario. Two RL algorithms which use Vector Quantization (VQ) as the generalization method for the space state are presented. Both strategies are focused on obtaining a good vector quantizier that generalizes adequately the state space of the agents. We empirically state the convergence of both methods in our navigational Multi-agent learning domain. Besides, we use validation tools of pedestrian models to analyze the simulation results in the context of pedestrian dynamics. The simulations carried out, scaling up the number of agents in our environment (a closed room with a door through which the agents have to leave), have revealed that the basic characteristics of pedestrian movements have been learned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Agre, P., Chapman, D.: Pengi: An implementation of a theory of activity. In: Proceedings of the Sixth National Conference on Artificial Intelligence, pp. 268–272. Morgan Kaufmann (1987)
Chinrungrueng, C., Sequin, C.: Optimal adaptive k-means algorithm with dynamic adjustment of learning rate. IEEE Transactions on Neural Networks 6(1), 157–169 (1995)
Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 746–752. AAAI Press (1998)
Fernández, F., Borrajo, D.: Two steps reinforcement learning. International Journal of Intelligent Systems 23(2), 213–245 (2008)
Fernández, F., García, J., Veloso, M.: Probabilistic policy reuse for inter-task transfer learning. Robotics and Autonomous Systems 58(7), 866–871 (2010)
García, J., López-Bueno, I., Fernández, F., Borrajo, D.: A Comparative Study of Discretization Approaches for State Space Generalization in the Keepaway Soccer Task. In: Reinforcement Learning: Algorithms, Implementations and Aplications. Nova Science Publishers (2010)
Hebing, D., Molnár, P.: Social force model for pedestrian dynamics. Physics Review E, 4282–4286 (1995)
Johansson, A., Helbing, D., Shukla, P.K.: Specification of the social force pedestrian model by evolutionary adjustment to video tracking data. Advances in Complex Systems 10(2), 271–288 (2007)
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Int. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Busoniu, R.B.L., Schutter, B.D.: A comprehensive survey of multi-agent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews 38, 156–172 (2008)
Mataric, M.J.: Learning to behave socially. In: From Animals to Animats: International Conference on Simulation of Adaptive Behavior, pp. 453–462. MIT Press (1994)
Nakayama, A., Sugiyama, Y., Hasebe, K.: Instability of pedestrian flow and phase structure in a two–dimensional optimal velocity model. In: Pedestrian and Evacuation Dynamics 2005, pp. 321–332. Springer, Heidelberg (2007)
Schadschneider, A., Klingsch, W., Klüpfel, H., Kretz, T., Rogsch, C., Seyfried, A.: Evacuation dynamics: Empirical results, modeling and applications. In: Encyclopedia of Complexity and Systems Science, pp. 3142–3176 (2009)
Sen, S., Sekaran, M.: Multiagent Coordination with Learning Classifier Systems. In: Weiss, G., Sen, S. (eds.) IJCAI-WS 1995. LNCS, vol. 1042, pp. 218–233. Springer, Heidelberg (1996)
Seyfried, A., Passon, O., Steffen, B., Boltes, M., Rupprecht, T., Klingsch, W.: New insights into pedestrian flow through bottlenecks. Transportation Science 43(3), 395–406 (2009)
Sutton, R.S.: Learning to predict by the methods of temporal differences. In: Machine Learning, pp. 9–44. Kluwer Academic Publishers (1988)
Taylor, M.E., Stone, P.: Behavior transfer for value-function-based reinforcement learning. In: The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems (July 2005)
Torrey, L.: Crowd simulation via multi-agent reinforcement learning. In: Proceedings of the Sixth AAAI Conference On Artificial Intelligence and Interactive Digital Entertainment. AAAI Press, Menlo Park (2010)
Whitehead, S.D., Ballard, D.H.: Learning to perceive and act by trial and error. Machine Learning, 45–83 (1991)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Martinez-Gil, F., Lozano, M., Fernández, F. (2012). Multi-agent Reinforcement Learning for Simulating Pedestrian Navigation. In: Vrancx, P., Knudson, M., Grześ, M. (eds) Adaptive and Learning Agents. ALA 2011. Lecture Notes in Computer Science(), vol 7113. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28499-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-28499-1_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-28498-4
Online ISBN: 978-3-642-28499-1
eBook Packages: Computer ScienceComputer Science (R0)