Skip to main content
SpringerLink
Log in

Fitness function shaping in multiagent cooperative coevolutionary algorithms

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

Coevolution is a promising approach to evolve teams of agents which must cooperate to achieve some system objective. However, in many coevolutionary approaches, credit assignment is often subjective and context dependent, as the fitness of an individual agent strongly depends on the actions of the agents with which it collaborates. In order to alleviate this problem, we introduce a cooperative coevolutionary algorithm which biases the evolutionary search as well as shapes agent fitness functions to promote behavior that benefits the system-level performance. More specifically, we bias the search using a hall of fame approximation of optimal collaborators, and shape the agent fitness using the difference evaluation function. Our results show that shaping agent fitness with the difference evaluation improves system performance by up to 50 %, and adding an additional fitness bias improves performance by up to 75 % in our experiments. Finally, an analysis of system performance as a function of computational cost demonstrates that this algorithm makes extremely efficient use of computational resources, having a higher performance as a function of computational cost than any other algorithm tested.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Note that the counterfactual is not strictly necessary. We introduce it to keep the number of agents the same in both terms of the difference evaluation, so the intuition of inter-agent distances remains the same.

References

  1. Agogino, A., & Tumer, K. (2004). Efficient evaluation functions for multi-rover systems. In The genetic and evolutionary computation conference (pp. 1–12).

  2. Agogino, A., & Tumer, K. (2008). Efficient evaluation functions for evolving coordination. Evolutionary Computation, 16, 257–288.

    Article  Google Scholar 

  3. Agogino, A., & Tumer, K. (2008). Regulating air traffic flow with coupled agents. In Proceedings of the seventh international joint conference on autonomous agents and multiagent systems. Estoril, Portugal.

  4. Agogino, A. K., & Tumer, K. (2008). Analyzing and visualizing multiagent rewards in dynamic and stochastic environments. Journal of Autonomous Agents and Multi Agent Systems, 17(2), 320–338.

    Article  Google Scholar 

  5. Agogino, A. K., & Tumer, K. (2012). A multiagent approach to managing air traffic flow. Autonomous Agents and MultiAgent Systems, 24, 1–25.

    Article  Google Scholar 

  6. Back, T., & Schewefel, H. (1993). An overview of evolutionary algorithms for parameter optimization. Evolutionary Computation, 1(1), 1–23.

    Article  Google Scholar 

  7. Barrett, S., Stone, P., & Kraus, S. (2011). Empirical evaluation of ad hoc teamwork in the pursuit domain. In Proceedings of 11th international conference on autonomous agents and multiagent systems (AAMAS).

  8. Barros, R., Basgalupp, M., Freitas, A., & de Carvalho, A. (2014). Evolutionary design of decision-tree algorithms tailored to microarray gene expression data sets. IEEE Transactions on Evolutionary Computation, 18(6), 873–892. doi:10.1109/TEVC.2013.2291813.

    Article  Google Scholar 

  9. Bucci, A., & Pollack, J. (2005). On identifying global optima in cooperative coevolution. In Proceedings of the 2005 conference on genetic and evolutionary computation, GECCO ’05 (pp. 539–544). New York: ACM. doi:10.1145/1068009.1068098.

  10. Bucci, A., & Pollack, J. B. (2007). Thoughts on solution concepts. In Proceedings of the genetic and evolutionary computation conference—GECCO-2007. New York: ACM.

  11. Chandra, R., Frean, M., & Zhang, M. (2011). A memetic framework for cooperative coevolution of recurrent neural networks. In The 2011 international joint conference on neural networks (IJCNN) (pp. 673–680). doi:10.1109/IJCNN.2011.6033286.

  12. Chen, C., & Chen, W. (2013). Cooperatively coevolving differential evolution for compensatory neural fuzzy networks. In 2013 International conference on fuzzy theory and its applications (iFUZZY) (pp. 264–267). doi:10.1109/iFuzzy.2013.6825447.

  13. Chong, S., Tino, P., & Yao, X. (2009). Relationship between generalization and diversity in coevolutionary learning. IEEE Transactions on Computational Intelligence and AI in Games, 1(3), 214–232.

    Article  Google Scholar 

  14. Wang, Ch., & Gao, J. (2012). High-dimensional waveform inversion with cooperative coevolutionary differential evolution algorithm. IEEE Geoscience and Remote Sensing Letters, 9(2), 297–301. doi:10.1109/LGRS.2011.2166532.

    Article  Google Scholar 

  15. Colby, M., Curran, W., & Tumer, K. (2015). Approximating difference evaluations with local information (extended abstract). In Proceedings of the fourteenth international joint conference on autonomous agents and multiagent systems.

  16. Colby, M., & Tumer, K. (2012). Shaping fitness functions for coevolving cooperative multiagent systems. In AAMAS ’12: Proceedings of the 11th international conference on autonomous agents and multiagent systems (pp. 425–432). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.

  17. Colby, M., & Tumer, K. (2015). An evolutionary game theoretic analysis of difference evaluation functions. In Proceedings of the 2015 genetic and evolutionary computation conference (GECCO ’15).

  18. Darwen, P., & Yao, X. (1997). Speciation as automatic categorical modularization. IEEE Transactions on Evolutionary Computation, 1(2), 101–108.

    Article  Google Scholar 

  19. Dong, X., Yu, H., Ouyang, D., Cai, D., Ye, Y., & Zhang, Y. (2010). Cooperative coevolutionary genetic algorithms to find optimal elimination orderings for bayesian networks. In 2010 IEEE fifth international conference on bio-inspired computing: Theories and applications (BIC-TA) (pp. 1388–1394). doi:10.1109/BICTA.2010.5645605.

  20. Einarsson, G., Runarsson, T., Stefansson, G. (2014). A competitive coevolution scheme inspired by de. In 2014 IEEE symposium on differential evolution (SDE) (pp. 1–8). doi:10.1109/SDE.2014.7031529.

  21. Ficici, S. (2005). Monotonic solution concepts in coevolution. In Proceedings of the 2005 conference on genetic and evolutionary computation, GECCO ’05 (pp. 499–506). New York: ACM. doi:10.1145/1068009.1068093.

  22. Ficici, S., Melnik, O., & Pollack, J. B. (2005). A game-theoretic and dynamical-systems analysis of selection methods in coevolution. IEEE Transactions on Evolutionary Computation, 9, 580–602.

    Article  Google Scholar 

  23. Fogel, D. (1994). An introduction to simulated evolutionary optimization. IEEE Transactions on Neural Networks, 5(1), 3–14. doi:10.1109/72.265956.

    Article  Google Scholar 

  24. Garcia-Pedrajas, N., Hervas-Martinez, C., & Ortiz-Boyer, D. (2005). Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Transactions on Evolutionary Computation, 9(3), 271–302. doi:10.1109/TEVC.2005.844158.

    Article  Google Scholar 

  25. Haynes, T. D., Schoenefeld, D. A., & Wainwright, R. L. (1996). Type inheritance in strongly typed genetic programming. In P. J. Angeline & K. E. Kinnear (Eds.), Advances in genetic programming 2, Chapter 18 (pp. 359–376). Cambridge: MIT Press.

    Google Scholar 

  26. Hennes, D., Tuyls, K., & Rauterberg, M. (2009). State-coupled replicator dynamics. In Proceedings of the eighth international joint conference on autonomous agents and multiagent systems.

  27. Hoen, P. J., & Jong, E. D. D. (2004). Evolutionary multi-agent systems. In Proceedings of the 8th international conference on parallel problem solving from nature PPSN-04 (pp. 872–881).

  28. Huaxian, L., Yanbo, Z., Kaiquan, C., & Qingge, P. (2011). Route network flow assignment in the new generation of aviation by cooperative co-evolution. In 2011 IEEE 5th international conference on cybernetics and intelligent systems (CIS) (pp. 175–180). doi:10.1109/ICCIS.2011.6070323.

  29. Kira, Z., & Schultz, A. (2006). Continuous and embedded learning for multi-agent systems. In 2006 IEEE/RSJ international conference on intelligent robots and systems (pp. 3184–3190). doi:10.1109/IROS.2006.282343.

  30. Knudson, M., & Tumer, K. (2010). Coevolution of heterogeneous multi-robot teams. In Proceedings of the genetic and evolutionary computation conference, Portland, OR.

  31. Lee, S., Kim, H., & Myung, H. (2013). Cooperative coevolution-based model predictive control for multi-robot formation. In 2013 IEEE international conference on robotics and automation (ICRA) (pp. 1890–1895). doi:10.1109/ICRA.2013.6630827.

  32. Li, B., Lin, T., Liao, L., & Fan, C. (2008). Genetic algorithm based on multipopulation competitive coevolution. In IEEE congress on evolutionary computation, 2008. CEC 2008. (IEEE world congress on computational intelligence) (pp. 225–228). doi:10.1109/CEC.2008.4630803

  33. Li, X., & Yao, X. (2012). Cooperatively coevolving particle swarms for large scale optimization. IEEE Transactions on Evolutionary Computation, 16(2), 210–224. doi:10.1109/TEVC.2011.2112662.

    Article  Google Scholar 

  34. Mei, Y., Li, X., & Yao, X. (2014). Cooperative coevolution with route distance grouping for large-scale capacitated arc routing problems. IEEE Transactions on Evolutionary Computation, 18(3), 435–449. doi:10.1109/TEVC.2013.2281503.

    Article  Google Scholar 

  35. Nguyen, M., Abbass, H., & McKay, R. (2008). Analysis of ccme: Coevolutionary dynamics, automatic problem decomposition, and regularization. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 38(1), 100–109. doi:10.1109/TSMCC.2007.905847.

    Article  Google Scholar 

  36. Panait, L., Luke, S., & Wiegand, R. (2006). Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions on Evolutionary Computation, 10(6), 629–645. doi:10.1109/TEVC.2006.880330.

    Article  Google Scholar 

  37. Panait, L., Tuyls, K., Luke, S. (2008). Theoretical advantages of lenient learners: An evolutionary game theoretic perspective. The Journal of Machine Learning Research, 9, 423–457. http://dl.acm.org/citation.cfm?id=1390681.1390694

  38. Potter, M. A., & De Jong, K. A. (1995). Evolving neural networks with collaborative species. In Computer simulation conference.

  39. Rawal, A., Rajagopalan, P., & Miikkulainen, R. (2010). Constructing competitive and cooperative agent behavior using coevolution. In 2010 IEEE symposium on computational intelligence and games (CIG) (pp. 107–114). doi:10.1109/ITW.2010.5593366.

  40. Rodriguez, F., Garcia-Martinez, C., & Lozano, M. (2012). Hybrid metaheuristics based on evolutionary algorithms and simulated annealing: Taxonomy, comparison, and synergy test. IEEE Transactions on Evolutionary Computation, 16(6), 787–800. doi:10.1109/TEVC.2012.2182773.

    Article  Google Scholar 

  41. Rosin, C., & Belew, R. (1996). New methods for competitive coevolution. Evolutionary Computation, 5, 1–29.

    Article  Google Scholar 

  42. Sofge, D., De Jong, K., & Schultz, A. (2002). A blended population approach to cooperative coevolution for decomposition of complex problems. In Proceedings of the 2002 congress on evolutionary computation, 2002. CEC ’02 (Vol. 1, pp. 413–418). doi:10.1109/CEC.2002.1006270.

  43. Soule, T., & Heckendorn, R. B. (2010). A developmental approach to evolving scalable hierarchies for multi-agent swarms. In Proceedings of the 12th annual conference on genetic and evolutionary compuation—GECCO-2010. New York: ACM.

  44. Stone, P., & Veloso, M. (1997). Multiagent systems: A survey from a machine learning perspective. Autonomous Robots, 8, 345–383.

    Article  Google Scholar 

  45. Tan, T. G., Teo, J., & Lau, H. (2007). Competitive coevolution with k-random opponents for pareto multiobjective optimization. In Third international conference on natural computation, 2007. ICNC 2007 (Vol. 4, pp. 63–67). doi:10.1109/ICNC.2007.309.

  46. Tumer, K., Welch, Z., & Agogino, A. (2008). Aligning social welfare and agent preferences to alleviate traffic congestion. In Proceedings of the seventh international joint conference on autonomous agents and multiagent systems, Estoril, Portugal.

  47. Tumer, K., & Wolpert, D. (2004). A survey of collectives. In K. Tumer & D. Wolpert (Eds.), Collectives and the design of complex systems (pp. 1–42). New York: Springer.

    Chapter  Google Scholar 

  48. Tuyls, K., & Westra, R. (2009). Replicator dynamics in discrete and continuous strategy spaces. In A. M. Uhrmacher & D. Weyns (Eds.), Multi-agent systems: Simulation and applications. Boca Raton: CRC Press.

    Google Scholar 

  49. Wei, F., Wang, Y., & Huo, Y. (2013). Smoothing and auxiliary functions based cooperative coevolution for global optimization. In 2013 IEEE congress on evolutionary computation (CEC) (pp. 2736–2741). doi:10.1109/CEC.2013.6557900

  50. Wiegand, R. (2003). An analysis of cooperative coevolutionary algorithms. Ph.D. Thesis, George Mason University.

  51. Wiegand, R. P., Jong, K. A. D., & Liles, W. C. (2002). Modeling variation in cooperative coevolution using evolutionary game theory. In FOGA

  52. Wolpert, D. H., & Tumer, K. (2002). Optimal payoff functions for members of collectives. In F. Schweitzer (Ed.), Modeling complexity in economic and social systems, Chapter 20 (pp. 355–369). Singapore: World Scientific.

    Chapter  Google Scholar 

Download references

Acknowledgments

This work was partially supported under National Energy Technology Laboratory Grant Number DE-FE0011403.

Author information

Authors and Affiliations

  1. Oregon State University, Corvallis, OR, USA

    Mitchell Colby & Kagan Tumer

Authors
  1. Mitchell Colby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kagan Tumer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mitchell Colby.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Funding

This study was partially funded by the Department of Energy, National Energy Technology Laboratory, grant number DE-FE0011403.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Colby, M., Tumer, K. Fitness function shaping in multiagent cooperative coevolutionary algorithms. Auton Agent Multi-Agent Syst 31, 179–206 (2017). https://doi.org/10.1007/s10458-015-9318-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10458-015-9318-0

Keywords

Search

Navigation