Ant Colony Learning Algorithm for Optimal Control

  • Jelmer Marinus van Ast
  • Robert Babuška
  • Bart De Schutter
Part of the Studies in Computational Intelligence book series (SCI, volume 281)


Ant colony optimization (ACO) is an optimization heuristic for solving combinatorial optimization problems and is inspired by the swarming behavior of foraging ants. ACO has been successfully applied in various domains, such as routing and scheduling. In particular, the agents, called ants here, are very efficient at sampling the problem space and quickly finding good solutions. Motivated by the advantages of ACO in combinatorial optimization, we develop a novel framework for finding optimal control policies that we call Ant Colony Learning (ACL). In ACL, the ants all work together to collectively learn optimal control policies for any given control problem for a system with nonlinear dynamics. In this chapter, we discuss the ACL framework and its implementation with crisp and fuzzy partitioning of the state space. We demonstrate the use of both versions in the control problem of two-dimensional navigation in an environment with variable damping and discuss their performance.


Membership Function Optimal Control Problem Pheromone Trail Terminal Vertex Pheromone Level 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Dorigo, M., Blum, C.: Ant colony optimization theory: a survey. Theoretical Computer Science 344(2-3), 243–278 (2005)zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Colorni, A., Dorigo, M., Maniezzo, V.: Distributed optimization by ant colonies. In: Varela, F.J., Bourgine, P. (eds.) Towards a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life, pp. 134–142. MIT Press, Cambridge (1992)Google Scholar
  3. 3.
    Dorigo, M., Stützle, T.: Ant Colony Optimization. The MIT Press, Cambridge (2004)zbMATHGoogle Scholar
  4. 4.
    Sim, K.M., Sun, W.H.: Ant colony optimization for routing and load-balancing: survey and new directions. IEEE Transactions on Systems, Man, Cybernetics, Part A 33(5), 560–572 (2003)CrossRefGoogle Scholar
  5. 5.
    Huang, R.H., Yang, C.L.: Ant colony system for job shop scheduling with time windows. International Journal of Advanced Manufacturing Technology 39(1-2), 151–157 (2008)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Alaykran, K., Engin, O., Dyen, A.: Using ant colony optimization to solve hybrid flow shop scheduling problems. International Journal of Advanced Manufacturing Technology 35(5-6), 541–550 (2007)CrossRefGoogle Scholar
  7. 7.
    Fan, X., Luo, X., Yi, S., Yang, S., Zhang, H.: Optimal path planning for mobile robots based on intensified ant colony optimization algorithm. In: Proceedings of the IEEE International Conference on Robotics, Intelligent Systems and Signal Processing (RISSP 2003), Changsha, Hunan, China, October 2003, pp. 131–136 (2003)Google Scholar
  8. 8.
    Wang, J., Osagie, E., Thulasiraman, P., Thulasiram, R.K.: HOPNET: A hybrid ant colony optimization routing algorithm for mobile ad hoc network. Ad Hoc Networks 7(4), 690–705 (2009)CrossRefGoogle Scholar
  9. 9.
    Purnamadjaja, A.H., Russell, R.A.: Pheromone communication in a robot swarm: necrophoric bee behaviour and its replication. Robotica 23(6), 731–742 (2005)CrossRefGoogle Scholar
  10. 10.
    Fox, B., Xiang, W., Lee, H.P.: Industrial applications of the ant colony optimization algorithm. International Journal of Advanced Manufacturing Technology 31(7-8), 805–814 (2007)CrossRefGoogle Scholar
  11. 11.
    Bianchi, L., Dorigo, M., Gambardella, L.M., Gutjahr, W.J.: Metaheuristics in stochastic combinatorial optimization: a survey. IDSIA, Manno, Switzerland, Tech. Rep. 08 (March 2006)Google Scholar
  12. 12.
    van Ast, J.M., Babuška, R., De Schutter, B.: Novel ant colony optimization approach to optimal control. International Journal of Intelligent Computing and Cybernetics 2(3), 414–434 (2009)zbMATHCrossRefGoogle Scholar
  13. 13.
    van Ast, J.M., Babuška, R., De Schutter, B.: Fuzzy ant colony optimization for optimal control. In: Proceedings of the American Control Conference (ACC 2009), Saint Louis, MO, USA, June 2009, pp. 1003–1008 (2009)Google Scholar
  14. 14.
    Socha, K., Blum, C.: An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training. Neural Computing & Applications 16(3), 235–247 (2007)CrossRefGoogle Scholar
  15. 15.
    Socha, K., Dorigo, M.: Ant colony optimization for continuous domains. European Journal of Operational Research 185(3), 1155–1173 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Bilchev, G., Parmee, I.C.: The ant colony metaphor for searching continuous design spaces. In: Fogarty, T. (ed.) AISB-WS 1995. LNCS, vol. 993, pp. 25–39. Springer, Heidelberg (1995)Google Scholar
  17. 17.
    Tsutsui, S., Pelikan, M., Ghosh, A.: Performance of aggregation pheromone system on unimodal and multimodal problems. In: Proceedings of the 2005 Congress on Evolutionary Computation (CEC 2005), Edinburgh, Scotland, September 2005, pp. 880–887 (2005)Google Scholar
  18. 18.
    Korosec, P., Silc, J., Oblak, K., Kosel, F.: The differential ant-stigmergy algorithm: an experimental evaluation and a real-world application. In: Proceedings of the 2007 Congress on Evolutionary Computation (CEC 2007), Singapore, September 2007, pp. 157–164 (2007)Google Scholar
  19. 19.
    Birattari, M., Caro, G.D., Dorigo, M.: Toward the formal foundation of Ant Programming. In: Proceedings of the International Workshop on Ant Algorithms (ANTS 2002), pp. 188–201. Springer, Brussels (2002)Google Scholar
  20. 20.
    Gambardella, L.M., Dorigo, M.: Ant-Q: A reinforcement learning approach to the traveling salesman problem. In: Prieditis, A., Russell, S. (eds.) Machine Learning: Proceedings of the Twelfth International Conference on Machine Learning, pp. 252–260. Morgan Kaufmann Publishers, San Francisco (1995)Google Scholar
  21. 21.
    Casillas, J., Cordn, O., Herrera, F.: Learning fuzzy rule-based systems using ant colony optimization algorithms. In: Proceedings of the ANTS 2000. From Ant Colonies to Artificial Ants: Second International Workshop on Ant Algorithms, Brussels, Belgium, September 2000, pp. 13–21 (2000)Google Scholar
  22. 22.
    Zhao, B., Li, S.: Design of a fuzzy logic controller by ant colony algorithm with application to an inverted pendulum system. In: Proceedings of the IEEE International Conference on Systems, Man, Cybernetics, Taipei, Taiwan, October 2006, pp. 3790–3794 (2006)Google Scholar
  23. 23.
    Zhu, W., Chen, J., Zhu, B.: Optimal design of fuzzy controller based on ant colony algorithms. In: Proceedings of the IEEE International Conference on Mechatronics and Automation, Luoyang, China, June 2006, pp. 1603–1607 (2006)Google Scholar
  24. 24.
    Buşoniu, L., Ernst, D., De Schutter, B., Babuška, R.: Continuous-state reinforcement learning with fuzzy approximation. In: Tuyls, K., Nowe, A., Guessoum, Z., Kudenko, D. (eds.) ALAMAS 2005, ALAMAS 2006, and ALAMAS 2007. LNCS (LNAI), vol. 4865, pp. 27–43. Springer, Heidelberg (2008)Google Scholar
  25. 25.
    Dorigo, M., Maniezzo, V., Colorni, A.: Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics, Part B 26(1), 29–41 (1996)CrossRefGoogle Scholar
  26. 26.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  27. 27.
    Dorigo, M., Gambardella, L.: Ant Colony System: a cooperative learning approach to the traveling salesman problem. IEEE Transactions on Evolutionary Computation 1(1), 53–66 (1997)CrossRefGoogle Scholar
  28. 28.
    Stützle, T., Hoos, U.: MAX MIN Ant System. Journal of Future Generation Computer Systems 16, 889–914 (2000)CrossRefGoogle Scholar
  29. 29.
    Åström, K.J., Wittenmark, B.: Computer Controlled Systems—Theory and Design. Prentice-Hall, Englewood Cliffs (1990)Google Scholar
  30. 30.
    Watkins, C., Dayan, P.: Q-learning. Machine Learning 8(3-4), 279–292 (1992)zbMATHCrossRefGoogle Scholar
  31. 31.
    Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Cambridge University, Tech. Rep. CUED/F-INFENG/TR166 (1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Jelmer Marinus van Ast
    • 1
  • Robert Babuška
    • 1
  • Bart De Schutter
    • 2
  1. 1.Delft Center for Systems and ControlDelft University of TechnologyDelftThe Netherlands
  2. 2.Delft Center for Systems and Control & Marine and Transport TechnologyDelft University of TechnologyDelftThe Netherlands

Personalised recommendations