Using Surveys and Web-Scraping to Select Tools for Software Testing Consultancy

  • Päivi Raulamo-JurvanenEmail author
  • Kari Kakkonen
  • Mika Mäntylä
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10027)


We analyzed findings from data collected utilizing surveys and Web-scraping, to support Knowit Oy, a software testing consultation company, in the process of selecting the right tools for software testing & test automation. We conducted two surveys (2013 & 2016) among (mostly Finnish) software professionals to acquire criteria and a list of tools used for software testing in industry. Considering all our data sources Selenium was the most popular pure tool, while Robot Framework was the most referenced tool (latter survey). According to the surveys Jenkins and Sikuli have the highest increase in popularity (or familiarity). Top referred criteria for selection were usability, functionality, maintainability and available support for a tool. While Knowit considers it best to utilize traditional surveys, Web-scraping is seen as cost effective support for such instruments. To get comprehensive picture and to gain knowledge of the tools in markets multiple sources should be used.


Test automation Software testing tool Software test automation tool Tool support Selection criteria 



Our thanks to all respondents for their contribution to this survey.


  1. 1.
    Capgemini Consulting: World Quality Report 2015–2016. (2015).
  2. 2.
    Fewster, M., Graham, D.: Software Test Automation: Effective Use of Test Execution Tools. ACM Press/Addison-Wesley Publishing Co (1999)Google Scholar
  3. 3.
    Garousi, V., Zhi, J.: A survey of software testing practices in Canada. J. Syst. Softw. 86(5), 1354–1376 (2013)CrossRefGoogle Scholar
  4. 4.
    Hussain, S., Wang, Z., Toure, I. K., Diop, A.: Web service testing tools: A comparative study. arXiv preprint arXiv:1306.4063 (2013)
  5. 5.
    ISTQB (International Software Testing Qualifications Board): ISTQB® Worldwide Software Testing Practices Report 2015–2016 (2016).
  6. 6.
    Kaur, H., Gupta, G.: Comparative study of automated testing tools: selenium, quick test professional and testcomplete. Int. J. Eng. Res. Appl. 3(5), 1739–1743 (2013)Google Scholar
  7. 7.
    Kaur, M., Kumari, R.: Comparative study of automated testing tools: testcomplete and quicktest pro. Int. J. Comput. Appl. 24(1), 1–7 (2011)Google Scholar
  8. 8.
    Michael, J.B., Bossuyt, B.J., Snyder, B.B.: Metrics for measuring the effectiveness of software-testing tools. In: 13th International Symposium on Software Reliability Engineering, pp. 117–128. IEEE (2003)Google Scholar
  9. 9.
    Ng, S., Murnane, T., Reed, K., Grant, D., Chen, T.: A preliminary survey on software testing practices in Australia. In: 2004 Australian Software Engineering Conference, pp. 116–125. IEEE (2004)Google Scholar
  10. 10.
    Pan, A., Raposo, J., Álvarez, M., Hidalgo, J., Viña, Á.: Semi-automatic wrapper generation for commercial web sources. In: Rolland, C., Brinkkemper, S., Saeki, M. (eds.). IFIP, vol. 103, pp. 265–283. Springer, Heidelberg (2002). doi: 10.1007/978-0-387-35614-3_16 CrossRefGoogle Scholar
  11. 11.
    Petersen, K., Wohlin, C.: Context in industrial software engineering research. In: 3rd International Symposium on Empirical Software Engineering and Measurement, pp. 401–404. IEEE (2009)Google Scholar
  12. 12.
    Portillo-Rodríguez, J., Vizcaíno, A., Piattini, M., Beecham, S.: Tools used in global software engineering: a systematic mapping review. Inf. Softw. Technol. 54(7), 663–685 (2012)CrossRefGoogle Scholar
  13. 13.
    Poston, R.M., Sexton, M.P.: Evaluating and selecting testing tools. Software 9(3), 33–42, IEEE (1992)Google Scholar
  14. 14.
    Surowiecki, J.: The Wisdom of Crowds. Anchor, New York (2005)Google Scholar
  15. 15.
    Tassey, G.: The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, RTI Project, 7007.011 (2002)Google Scholar
  16. 16.
    Tiitinen, M., Kakkonen, K.: Software Testing Tools Research in Finland, Knowit Oy (2013).
  17. 17.
  18. 18.
  19. 19.
    Yehezkel, S.: Test Automation Survey (2016).

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Päivi Raulamo-Jurvanen
    • 1
    Email author
  • Kari Kakkonen
    • 2
  • Mika Mäntylä
    • 1
  1. 1.M3S (M-Group)ITEE University of OuluOuluFinland
  2. 2.Knowit OyHelsinkiFinland

Personalised recommendations