Skip to main content
SpringerLink
Log in

A Retrospective Analysis of Grey Literature for AI-Supported Test Automation

  • Conference paper
  • First Online:
Quality of Information and Communications Technology (QUATIC 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1871))

Abstract

This paper provides the results of a retrospective analysis conducted on a survey of the grey literature about the perception of practitioners on the integration of artificial intelligence (AI) algorithms into Test Automation (TA) practices.

Our study involved the examination of 231 sources, including blogs, user manuals, and posts. Our primary goals were to: (a) assess the generalizability of existing taxonomies about the usage of AI for TA, (b) investigate and understand the relationships between TA problems and AI-based solutions, and (c) systematically map out the existing AI-based tools that offer AI-enhanced solutions.

Our analysis yielded several interesting results. Firstly, we assessed a high degree of generalization of the existing taxonomies. Secondly, we identified TA problems that can be addressed using AI-enhanced solutions integrated into existing tools. Thirdly, we found that some TA problems require broader solutions that involve multiple software testing phases simultaneously, such as test generation and maintenance. Fourthly, we discovered that certain solutions are being investigated but are not supported by existing AI-based tools. Finally, we observed that there are tools that supports different phases of TA and may have a broader outreach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.functionize.com.

  2. 2.

    https://applitools.com.

  3. 3.

    https://www.mabl.com.

  4. 4.

    https://www.testim.io.

  5. 5.

    https://www.test.ai.

  6. 6.

    https://www.appvance.ai.

References

  1. Bajammal, M., Stocco, A., Mazinanian, D., Mesbah, A.: A survey on the use of computer vision to improve software engineering tasks. TSE 48(5), 1722–1742 (2020)

    Google Scholar 

  2. Camara, B., Silva, M., Endo, A., Vergilio, S.: On the use of test smells for prediction of flaky tests. In: SAST 2021, pp. 46–54. Association for Computing Machinery (2021)

    Google Scholar 

  3. Choudhary, S.R., Zhao, D., Versee, H., Orso, A.: WATER: web application test repair. In: Proceedings of 1st International Workshop on End-to-End Test Script Engineering, ETSE 2011, pp. 24–29. ACM (2011)

    Google Scholar 

  4. Feng, Y., Jones, J.A., Chen, Z., Fang, C.: Multi-objective test report prioritization using image understanding. In: Proceedings of 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, pp. 202–213. ACM, New York (2016)

    Google Scholar 

  5. García, B., Gallego, M., Gortázar, F., Munoz-Organero, M.: A survey of the selenium ecosystem. Electronics 9, 1067 (2020)

    Article  Google Scholar 

  6. Garousi, V., Felderer, M., Mäntylä, M.V.: Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. IST 106, 101–121 (2019)

    Google Scholar 

  7. Gyimesi, P., et al.: BugJS: a benchmark of javascript bugs. In: Proceedings of 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019, p. 12. IEEE (2019)

    Google Scholar 

  8. Gyimesi, P., et al.: BugJS: a benchmark and taxonomy of javascript bugs. Softw. Test. Verification Reliab. 31(4), e1751 (2020)

    Article  Google Scholar 

  9. Plotly Inc.: Collaborative data science (2015). https://plot.ly

  10. Jha, N., Popli, R.: Artificial intelligence for software testing-perspectives and practices. In: CCICT 2021, pp. 377–382 (2021)

    Google Scholar 

  11. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

    Google Scholar 

  12. Leger, G., Barragan, M.J.: Mixed-signal test automation: are we there yet? In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5 (2018)

    Google Scholar 

  13. Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Approaches and tools for automated end-to-end web testing. In: Advances in Computers, vol. 101, pp. 193–237 (2016)

    Google Scholar 

  14. Leotta, M., Stocco, A., Ricca, F., Tonella, P.: Automated migration of DOM-based to visual web tests. In: Proceedings of 30th Symposium on Applied Computing, SAC 2015, pp. 775–782. ACM (2015)

    Google Scholar 

  15. Leotta, M., Stocco, A., Ricca, F., Tonella, P.: PESTO: automated migration of DOM-based web tests towards the visual approach. Softw. Test. Verification Reliab. 28(4), e1665 (2018)

    Article  Google Scholar 

  16. Lima, R., da Cruz, A.M.R., Ribeiro, J.: Artificial intelligence applied to software testing: a literature review. In: 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6 (2020)

    Google Scholar 

  17. Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pp. 643–653. Association for Computing Machinery, New York (2014)

    Google Scholar 

  18. Mahajan, S., Halfond, W.G.J.: Detection and localization of HTML presentation failures using computer vision-based techniques. In: Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, pp. 1–10 (2015)

    Google Scholar 

  19. McKinney, W., et al.: Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference, Austin, TX, vol. 445, pp. 51–56 (2010)

    Google Scholar 

  20. Memon, A., Banerjee, I., Nagarajan, A.: What test oracle should i use for effective GUI testing? In: 18th IEEE International Conference on Automated Software Engineering, 2003 Proceedings, pp. 164–173 (2003)

    Google Scholar 

  21. Mesbah, A., van Deursen, A., Lenselink, S.: Crawling ajax-based web applications through dynamic analysis of user interface state changes. ACM Trans. Web 6(1), 1–30 (2012)

    Article  Google Scholar 

  22. Phuc Nguyen, D., Maag, S.: Codeless web testing using Selenium and machine learning. In: ICSOFT 2020: 15th International Conference on Software Technologies, ICSOFT 2020, pp. 51–60. ScitePress, Online, France (2020)

    Google Scholar 

  23. Qian, J., Ma, Y., Lin, C., Chen, L.: Accelerating OCR-based widget localization for test automation of GUI applications. Association for Computing Machinery (2023)

    Google Scholar 

  24. Replication Package (2023). https://github.com/riccaF/quatic2023-replication-package-material/

  25. Ricca, F., Marchetto, A., Stocco, A.: AI-based test automation: a grey literature analysis. In: Proceedings of 14th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021. Springer, Cham (2021)

    Google Scholar 

  26. Ricca, F., Stocco, A.: Web test automation: insights from the grey literature. In: Bureš, T., et al. (eds.) SOFSEM 2021. LNCS, vol. 12607, pp. 472–485. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67731-2_35

    Chapter  Google Scholar 

  27. Schmidt, M.: The sankey diagram in energy and material flow management. J. Ind. Ecol. 12(2), 173–185 (2008)

    Article  Google Scholar 

  28. Stocco, A., Leotta, M., Ricca, F., Tonella, P.: PESTO: a tool for migrating DOM-based to visual web tests. In: Proceedings of 14th International Working Conference on Source Code Analysis and Manipulation, SCAM 2014, pp. 65–70. IEEE Computer Society (2014)

    Google Scholar 

  29. Stocco, A., Leotta, M., Ricca, F., Tonella, P.: Why creating web page objects manually if it can be done automatically? In: Proceedings of 10th International Workshop on Automation of Software Test, AST 2015, pp. 70–74. IEEE/ACM (2015)

    Google Scholar 

  30. Stocco, A., Leotta, M., Ricca, F., Tonella, P.: Clustering-aided page object generation for web testing. In: Bozzon, A., Cudre-Maroux, P., Pautasso, C. (eds.) ICWE 2016. LNCS, vol. 9671, pp. 132–151. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-38791-8_8

    Chapter  Google Scholar 

  31. Stocco, A., Leotta, M., Ricca, F., Tonella, P.: APOGEN: automatic page object generator for web testing. Software Qual. J. 25(3), 1007–1039 (2017)

    Article  Google Scholar 

  32. Stocco, A., Willi, A., Starace, L.L.L., Biagiola, M., Tonella, P.: Neural embeddings for web testing. arXiv:2306.07400 (2023)

  33. Stocco, A., Yandrapally, R., Mesbah, A.: Visual web test repair. In: Proceedings of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018. ACM (2018)

    Google Scholar 

  34. Triou, E., Abbas, Z., Kothapalle, S.: Declarative testing: a paradigm for testing software applications. In: 2009 Sixth International Conference on Information Technology: New Generations, pp. 769–773 (2009)

    Google Scholar 

  35. Trudova., A., Dolezel., M., Buchalcevova., A.: Artificial intelligence in software test automation: a systematic literature review. In: Proceedings of the ENASE, pp. 181–192. INSTICC, SciTePress (2020)

    Google Scholar 

  36. Vos, T.E.J., Aho, P., Pastor Ricos, F., Rodriguez-Valdes, O., Mulders, A.: Testar - scriptless testing through graphical user interface. Softw. Test. Verification Reliab. 31(3), e1771 (2021)

    Google Scholar 

  37. Walia, R.: Application of machine learning for GUI test automation. In: 2022 XXVIII International Conference on Information, Communication and Automation Technologies (ICAT), pp. 1–6 (2022)

    Google Scholar 

  38. Yadav, V., Botchway, R.K., Senkerik, R., Kominkova, Z.O.: Robot testing from a machine learning perspective. In: 2021 International Conference on Electrical, Computer and Energy Technologies (ICECET), pp. 1–4 (2021)

    Google Scholar 

  39. Zhang, C., Cheng, H., Tang, E., Chen, X., Bu, L., Li, X.: Sketch-guided GUI test generation for mobile applications. In: Proceedings of ASE 2017, pp. 38–43 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Università degli Studi di Genova, Genoa, Italy

    Filippo Ricca

  2. University of Trento, Trento, Italy

    Alessandro Marchetto

  3. Technical University of Munich, Munich, Germany

    Andrea Stocco

  4. fortiss GmbH, Munich, Germany

    Andrea Stocco

Authors
  1. Filippo Ricca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Marchetto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Stocco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filippo Ricca .

Editor information

Editors and Affiliations

  1. University of Aveiro, Aveiro, Portugal

    José Maria Fernandes

  2. Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme H. Travassos

  3. University of Oulu, Oulu, Finland

    Valentina Lenarduzzi

  4. University of Oulu, Oulu, Finland

    Xiaozhou Li

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ricca, F., Marchetto, A., Stocco, A. (2023). A Retrospective Analysis of Grey Literature for AI-Supported Test Automation. In: Fernandes, J.M., Travassos, G.H., Lenarduzzi, V., Li, X. (eds) Quality of Information and Communications Technology. QUATIC 2023. Communications in Computer and Information Science, vol 1871. Springer, Cham. https://doi.org/10.1007/978-3-031-43703-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43703-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43702-1

  • Online ISBN: 978-3-031-43703-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation