Applying Reinforcement Learning in Formation Control of Agents

  • Vali DerhamiEmail author
  • Yusef Momeni
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 616)


This paper proposes a new Reinforcement Learning (RL) algorithm for formation of agents in regular geometric forms. Due to curse of dimensionality problem, applying RL algorithms in formation problems cannot present suitable performance. Moreover, since the state space in formation problem is large, this leads to long learning time. Here, a multi-agent fuzzy reinforcement learning algorithm is presented that is an extension of fuzzy actor-critic reinforcement learning in a multi-agent environment. The final action for each agent is generated by a zero order T-S fuzzy system. In conventional fuzzy actor-critic RL, there are several candidate actions for consequence of each fuzzy rule and aim of learning is finding the best action among these discrete candidate actions. Here, using the proposed linear interpolation, a continuous action selection for determining the best action for each fuzzy rule is presented. The simulation results show the proposed method can improve the learning speed and action quality.



This research was financially supported by the Center of Excellence for Robust and Intelligence Systems (CERIS) of Yazd University.


  1. 1.
    Bik, J.J.C.M., Visser, P.N.A.M., Jennrich, O.: LISA satellite formation control. Adv. Space Res. 40, 25–34 (2007)CrossRefGoogle Scholar
  2. 2.
    Breivik, M.: Topics in guided motion control of marine vehicles. Ph.D. thesis, Norwegian University of Science and Technology (2010)Google Scholar
  3. 3.
    Chen, G., Cao, W., Chen, X., Wu, M.: Multi-agent q-learning with joint state value approximation. In: Proceedings of the 30th Chinese Control Conference, p. 48784882 (2011)Google Scholar
  4. 4.
    Chen, X., Chen, G., Cao, W., Wu, M.: Cooperative learning with joint state value approximation for multi-agent systems. J. Control Theory Appl. 11, 149–155 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Derhami, V., Majd, V.J., Ahmadabadi, M.N.: Fuzzy sarsa learning and the proof of existence of Its stationary points. Asian J. Control 10, 535–549 (2008)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Dierks, T., Jagannathan, S.: Neural network output feedback control of robot formations. IEEE Trans. Syst., Man, Cybern. Part B 40, pp. 383-399 (2010)Google Scholar
  7. 7.
    Feddema, J., Lewis, C., Schoenwald, D.: Decentralized control of cooperative robotic vehicles: theory and application. IEEE Trans. Robot. Autom 18(5), 852864 (2002)CrossRefGoogle Scholar
  8. 8.
    Franco, F.E., Waissman, V.J., Garca, L.J.: Learning the filling policy of a biodegradation process by fuzzy actorcritic learning methodology. Adv. Artif. Intell. 5317, 243–253 (2008)Google Scholar
  9. 9.
    Izzo, D., Pettazzi, L.: Autonomous and distributed motion planning for satellite swarm. J. Guid. Control Dyn. 30, 449459 (2005)Google Scholar
  10. 10.
    Macdonald, E.A.: Multi-robot assignment and formation control. M.Sc. thesis, Georgia Institute of Technology (2011)Google Scholar
  11. 11.
    Panait, L., Luke, S.: Cooperative multi-agent learning: the state of the art. Auton. Agents Multi-agent Syst. 11, 387–434 (2005)CrossRefGoogle Scholar
  12. 12.
    Sanz, Y., de Lope, J., Martn, J.A.H.: Applying reinforcement learing to multi-robot team coordination. In: Corchado, E., Abraham, A., Pedrycz, W. (eds.) HAIS 2008. LNCS, vol. 5271, pp. 625632. Springer, Heidelberg (2008)Google Scholar
  13. 13.
    Zuo, G., Han, J., Han, G.: Multi-robot formation control using reinforcement learning, advances in swarm Intelligence. vol. 6145, pp. 667–674. Springer, Berlin (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of ECEYazd UniversityYazdIran

Personalised recommendations