Skip to main content
SpringerLink
Log in

Evolutionary testing of autonomous software agents

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

A system built in terms of autonomous software agents may require even greater correctness assurance than one that is merely reacting to the immediate control of its users. Agents make substantial decisions for themselves, so thorough testing is an important consideration. However, autonomy also makes testing harder; by their nature, autonomous agents may react in different ways to the same inputs over time, because, for instance they have changeable goals and knowledge. For this reason, we argue that testing of autonomous agents requires a procedure that caters for a wide range of test case contexts, and that can search for the most demanding of these test cases, even when they are not apparent to the agents’ developers. In this paper, we address this problem, introducing and evaluating an approach to testing autonomous agents that uses evolutionary optimisation to generate demanding test cases. We propose a methodology to derive objective (fitness) functions that drive evolutionary algorithms, and evaluate the overall approach with two simulated autonomous agents. The obtained results show that our approach is effective in finding good test cases automatically.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adamopoulos, K., Harman, M., & Hierons, R. M. (2004). How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Proceedings of the 2004 conference on Genetic and evolutionary computation (GECCO’04) (Vol. 3103/2004, pp. 1338–1349). Seattle, Washington, USA: Springer. http://www.springerlink.com/content/77akkyyrjcdm400h/.

  2. Arcuri, A., White, D. R., & Yao, X. (2008). Multi-objective improvement of software using co-evolution and smart seeding. In Proceedings of the 7th international conference on Simulated evolution and learning (SEAL’08) (pp. 61–70). Melbourne: Springer. doi:10.1007/978-3-540-89694-4_7.

  3. Arcuri, A., & Yao, X. (2008). A novel co-evolutionary approach to automatic software bug fixing. In Proceedings of the IEEE congress on Evolutionary computation (CEC’08) (pp. 162–168). Hongkong: IEEE. doi:10.1109/CEC.2008.4630793.

  4. Botía, J. A., López-Acosta, A., & Gómez-Skarmeta, A. F. (2004). ACLAnalyser: A tool for debugging multi-agent systems. In ECAI (pp. 967–968). Valencia, Spain.

  5. Bratman M. E. (1987) Intentions, plans and practical reason. Harvard University Press, Cambridge, MA

    Google Scholar 

  6. Bresciani P., Giorgini P., Giunchiglia F., Mylopoulos J., Perini A. (2004) Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems 8(3): 203–236

    Article  Google Scholar 

  7. Bühler O., Wegener J. (2008) Evolutionary functional testing. Computers and Operations Research 35(10): 3144–3160. doi:10.1016/j.cor.2007.01.015

    Article  Google Scholar 

  8. Castelfranchi, C., & Falcone, R. (2003). Agent autonomy, chap.: From automaticity to autonomy: The frontier of artificial agents, pp. 103–136. In H. Hexmoor et al. (Eds.), Proceedings of the 8th international working conference on Source code analysis and manipulation (SCAM’08) (pp. 249–258). Beijing, China: IEEE. doi:10.1109/SCAM.2008.36.

  9. Coelho, R., Kulesza, U., von Staa, A., & Lucena, C. (2006). Unit testing in multi-agent systems using mock agents and aspects. In SELMAS’06: Proceedings of the 2006 international workshop on Software engineering for large-scale multi-agent systems (pp. 83–90). New York, NY: ACM Press. doi:10.1145/1138063.1138079.

  10. Dardenne, A., van Lamsweerde, A., & Fickas, S. (1993). Goal-directed requirements acquisition. Science of Computer Programming, 20(1–2), 3–50. citeseer.ist.psu.edu/dardenne93goaldirected.html.

    Google Scholar 

  11. Deb K. D., Pratap A., Agarwal S., Meyarivan T. (2002) A fast and elitist multiobjective genetic algorithm : NSGA-II. IEEE Transactions on Evolutionary Computation 6(2): 182–197. doi:10.1109/4235.996017

    Article  Google Scholar 

  12. Dikenelli, O., Erdur, R. C., & Gumus, O. (2005). Seagent: A platform for developing semantic web based multi agent systems. In AAMAS’05: Proceedings of the fourth international joint conference on Autonomous Agents and Multiagent Systems (pp. 1271–1272). New York, NY: ACM Press. doi:10.1145/1082473.1082728.

  13. Engstrm, E., & Runeson, P. (2010). A qualitative survey of regression testing practices. In M. Ali Babar, M. Vierimaa, & M. Oivo (Eds.), Product-focused software process improvement. Lecture notes in Computer Science (Vol. 6156, pp. 3–16). Berlin: Springer.

  14. Gamma, E., & Beck, K. (2000). JUnit: A Regression Testing Framework. http://www.junit.org.

  15. Georgeff, M. P., & Ingrand, F. F. (1989). Decision-making in an embedded reasoning system. In IJCAI (pp. 972–978), Detroit, MI.

  16. Harman, M. (2007). The current state and future of search based software engineering. In L. Briand & A. Wolf (Eds.), IEEE international conference on Software Engineering (ICSE 2007), Future of Software Engineering (pp. 342–357). Los Alamitos, CA: IEEE Computer Society Press.

  17. Hexmoor H., Castelfranchi C., Falcone R., Luck M., D’Inverno M., Munroe S., Barber K. S., Gamba I., Martin C. E., Braynov S., Boella G., Cohen R., Fleming M., Franklin S., McCauley L. (2003) Agent Autonomy. Springer, Berlin

    Book  MATH  Google Scholar 

  18. Jia, Y., & Harman, M. (2008). Constructing subtle faults using higher order mutation testing. In Proceedings of the 8th international working conference on Source Code Analysis and Manipulation (SCAM’08) (pp. 249–258). Beijing: IEEE. doi:10.1109/SCAM.2008.36.

  19. Lam D. N., Barber K. S. (2005) Programming multi-agent systems, Chap: Debugging agent behavior in an implemented agent system. Springer, Berlin, pp 104–125

    Google Scholar 

  20. Lamsweerde, A. V. (2001). Goal-oriented requirements engineering: A guided tour. In IEEE international conference on Requirements Engineering (p. 249). Toronto, Canada. doi:10.1109/ISRE.2001.948567.

  21. McMinn, P., & Holcombe, M. (2003). The state problem for evolutionary testing. In Proceedings of the genetic and evolutionary computation conference (pp. 2488–2498). Berlin: Springer.

  22. Mylopoulos J., Chung L., Yu E. S. K. (1999) From object-oriented to goal-oriented requirements analysis. Commun. ACM 42(1): 31–37

    Article  Google Scholar 

  23. Nguyen, C. D., Perini, A., & Tonella, P. (2007). Automated continuous testing of multi-agent systems. In The fifth European workshop on Multi-agent systems. Hammamet, Tunisia.

  24. Nguyen, C.D., Perini, A., Tonella, P., Miles, S., Harman, M., & Luck, M. (2009). Evolutionary testing of autonomous software agents. In 8th International joint conference on Autonomous Agents and Multiagent Systems (AAMAS 2009) (pp. 521–528). Budapest, Hungary.

  25. Núñez M., Rodríguez I., Rubio F. (2005) Specification and testing of autonomous agents in e-commerce systems. Software Testing, Verification and Reliability 15(4): 211–233. doi:10.1002/stvr.v15:4

    Article  Google Scholar 

  26. Pokahr, A., Braubach, L., & Lamersdorf, W. (2005). Jadex: A BDI reasoning engine, chap.: Multi-agent programming. Kluwer. http://vsis-www.informatik.uni-hamburg.de/projects/jadex.

  27. Rodrigues, L. F., de Carvalho, G. R., de Barros Paes, R., & de Lucena, C. J. P. (2005). Towards an integration test architecture for open mas. In 1st workshop on Software Engineering for agent-oriented systems/SBES. Uberlândia-MG, Brazil.

  28. TILAB: Java agent development framework. http://jade.tilab.com(2002).

  29. Tiryaki, A. M., Öztuna, S., Dikenelli, O., & Erdur, R. C. (2006). Sunit: A unit testing framework for test driven development of multi-agent systems. In Agent-oriented software engineering VII, 7th international workshop, AOSE 2006 (pp. 156–173). Berlin: Springer.

  30. Wegener J. (2005) Stochastic algorithms: Foundations and applications, chap.: Evolutionary testing techniques. Springer, Berlin, pp 82–94

    Book  Google Scholar 

  31. Weyns D., Omicini A., Odell J. (2007) Environment as a first class abstraction in multiagent systems. Autonomous Agents and Multi-Agent Systems 14(1): 5–30

    Article  Google Scholar 

  32. White, D. R. (2009). Genetic programming for low-resource systems. Ph.D. thesis, University of York, UK. http://www.cs.york.ac.uk/~drw/papers/thesis/drwthesis.pdf.

  33. Yoo, S., & Harman, M. (2010). Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability. doi:10.1002/stvr.430.

Download references

Author information

Authors and Affiliations

  1. Fondazione Bruno Kessler, Via Sommarive, 18, 38050, Trento, Italy

    Cu D. Nguyen, Anna Perini & Paolo Tonella

  2. Department of Informatics, King’s College London, Strand, London, WC2R 2LS, UK

    Simon Miles & Michael Luck

  3. Department of Computer Science, University College London, Malet Place, London, WC1E 6BT, UK

    Mark Harman

Authors
  1. Cu D. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Miles
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anna Perini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paolo Tonella
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark Harman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael Luck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cu D. Nguyen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nguyen, C.D., Miles, S., Perini, A. et al. Evolutionary testing of autonomous software agents. Auton Agent Multi-Agent Syst 25, 260–283 (2012). https://doi.org/10.1007/s10458-011-9175-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10458-011-9175-4

Keywords

Search

Navigation