Skip to main content

Trade-off between automated and manual software testing

International Journal of System Assurance Engineering and Management volume 2, pages 114–125 (2011)Cite this article

Abstract

The study explores the current state of test automation in software testing organizations by focusing on the views and observations of managers, testers and developers in each organization. The case study was conducted in selected organizational units that develop and test technical software for industrial automation or telecommunication domains. The data was collected with 41 theme-based interviews in each unit. The interview data was analyzed qualitatively by using the grounded theory research method. It was found that although test automation was viewed as beneficial, it was not utilized widely in the companies. The main benefits of test automation were quality improvement, the possibility to execute more tests in less time and fluent reuse of testware. The major disadvantages were the costs associated with developing test automation especially in dynamic customized environments. Such issues as properties of tested products, attitudes of employees, resource limitations, and customers influenced the level of test automation in the case organizations.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  • Atlas (2009) ATLAS.ti—The knowledge workbench. Scientific Software Development, http://www.atlasti.com Acessed 29 July 2009

  • Bach J (1997) Test automation snake oil. In: Proceedings of 14th international conference and exposition on testing computer

  • Berner S, Weber R, Keller RK (2005) Observations and lessons learned from automated testing. In: Proceedings of the 27th international conference on Software engineering, St. Louis, Mo, USA, 571–579. doi:10.1145/1062455.1062556

  • Bertolino A (2007) Software testing research: achievements, challenges, dreams, future of software engineering. IEEE Computer Society, pp 85–103. doi: 10.1109/FOSE.2007.25

  • Blackburn M, Busser R, Nauman A (2004) Why model-based test automation is different and what you should know to get started.In: International conference on practical software quality and testing

  • Briand LC (2007) A critical analysis of empirical research in software testing. In: First international symposium on empirical software engineering and measurement Madrid, Spain: IEEE Computer Society. doi: 10.1109/ESEM.2007.40

  • Carter CR, Dresner M (2001) Purchasing’s role in environmental management: Cross-functional development of grounded theory. J Supply Chain Manag 37:12–28

    Article  Google Scholar 

  • Denzin NK (1978) The research act: a theoretical introduction to sociological methods. McGraw Hill, New York

    Google Scholar 

  • Dustin E, Rashka J, Paul J (1999) Automated software testing: introduction management, and performance. Addison-Wesley, Boston

    Google Scholar 

  • Eisenhardt KM (1989) Building theories from case study research. Acad Manag Rev 14:532–550

    Google Scholar 

  • EU (2003) SME Definition. European Commission

  • Fewster M (2001) Common mistakes in test automation.In: Fall test automation conference, Boston

  • Glaser B, Strauss AL (1967) The discovery of grounded theory: strategies for qualitative research. Aldine, Chicago

    Google Scholar 

  • Hartman A, Katara M, Paradkar A (2007) Domain specific approaches to software test automation. In: Proceedings of 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, Dubrovnik, 621–622, 2007. doi: 10.1145/1287624.1287730

  • ISO/IEC (2002) ISO/IEC 15504-1 Information Technology—Process Assessment—Part 1: Concepts and Vocabulary

  • Jones C (1994) Economics of software reuse. IEEE Computer 27:106–107. doi:10.1109/2.299437

    Article  Google Scholar 

  • Kaner C (1997) Improving the maintainability of automated test suites. Softw QA 4(4):28

    Google Scholar 

  • Kaner C (2000) Architectures of test automation. In: Software testing, analysis and review conference (Star) West, San Jose

  • Karhu K, Taipale O, Smolander K (2007) Outsourcing and knowledge management in software testing. In: Proceedings of the 11th international conference on evaluation and assessment in software engineering (EASE), Keele University, Staffordshire, UK, BCS, 2–3 April 2007

  • Karhu K, Repo T, Taipale O, Smolander, K (2009) Empirical observations on software testing automation. In: Proceedings of the 2nd international conference on software testing, verification and validation, Denver, CO, USA. doi: 10.1109/ICST.2009.16

  • Kit E (1995) Software testing in the real world: improving the process. Reading. Addison-Wesley, Boston

    Google Scholar 

  • Klein HK, Myers MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Q 23:67–94. doi:10.2307/249410

    Article  Google Scholar 

  • Mantere T (2003) Automatic software testing by genetic algorithms. Dissertation, Universitas Wasaensis, Vaasa, vol. 112

  • Miles MB, Huberman AM (1994) Qualitative data analysis. SAGE Publications, Thousand Oaks

    Google Scholar 

  • Ng SP, Murmane T, Reed K, Grant D, Chen TY (2004) A preliminary survey on software testing practices in Australia. In: Proceedings of 2004 Australian software engineering conference, Melbourne, Australia, 116–125

  • Pare′ G, Elam JJ (1997) Using case study research to build theories of IT implementation.In: IFIP TC8 WG international conference on information systems and qualitative research, Philadelphia, USA

  • Persson C, Yilmaztürk N (2004) Establishment of automated regression testing at ABB: industrial experience report on ‘Avoiding the Pitfalls’.In: the 19th international conference on automated software engineering (ASE’04): IEEE Computer Society. doi: 10.1109/ASE.2004.1342729

  • Ramler R, Wolfmaier K (2006) Economic perspectives in test automation: balancing automated and manual testing with opportunity cost. AST’06 Shanghai. ACM, China. doi:10.1145/1138929.1138946

    Google Scholar 

  • Robson C (2002) Real world research, 2nd edn. Blackwell Publishing, Oxford

    Google Scholar 

  • Santos-Neto P, Resende R, Pádua C (2007) Requirements for information systems model-based testing. In: Proceedings of the 2007 ACM symposium on applied computing, Seoul, Korea, 1409–1415. doi: 10.1145/1244002.1244306

  • Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25:557–572. doi:10.1109/32.799955

    Article  Google Scholar 

  • Shaw M (2003) Writing good software engineering research papers. In: Proceedings of the 25th international conference on software engineering (ICSE’ 03), Portland, pp 726–736

  • Sjöberg DIK, Dybå T, Jorgensen M (2007) The future of empirical methods in software engineering research. FOSE’07: 2007 Future of Software Engineering, pp. 358–378. doi: 10.1109/FOSE.2007.30

  • Smolander K, Rossi M, Purao S (2008) Software architectures: blueprint literature, language or decision. Eur J Inf Syst 17:575–588

    Article  Google Scholar 

  • Sommerville I (1995) Software engineering. Addison Wesley, Essex

    Google Scholar 

  • Strauss A, Corbin J (1990) Basics of qualitative research: grounded theory procedures and techniques. SAGE Publications, Newbury Park

    Google Scholar 

  • Taipale O, Smolander K, Kälviäinen H (2005) Finding and ranking research directions for software testing. In: European software process improvement and innovation conference, Budapest, Hungary, pp 39–48. doi: 10.1145/336512.336532

  • Taipale O, Smolander K, Kälviäinen H (2006a) A survey on software testing.In: The 6th international SPICE conference on software process improvement and capability dEtermination (SPICE’ 2006), Luxembourg

  • Taipale O, Smolander K, Kälviäinen H (2006b) Factors affecting software testing time schedule. In: Australian software engineering conference, Sydney. doi: 10.1109/ASWEC.2006.27

  • Taipale O, Smolander K, Kälviäinen H (2006c) Cost reduction and quality improvement in software testing.In: Software quality management conference, Southampton, UK

  • Taipale O, Karhu K, Smolander K (2007) Observing software testing practice from the viewpoint of organizations and knowledge management. In: Proceedings of the 1st international symposium on empirical software engineering and measurement (ESEM), 2007, Madrid, Spain, IEEE. doi: 10.1109/ESEM.2007.18

  • Tassey G (2002) The economic impacts of inadequate infrastructure for software testing. US national institute of standards and technology report, RTI Project Number 7007.011

  • Torkar R, Mankefors S (2003) A survey on testing and reuse. In: IEEE international conference on software-science, technology and engineering (SwSTE’03), Herzlia, Israel

Download references

Acknowledgment

This study was supported by the ESPA-project (http://www.soberit.hut.fi/espa), funded by the Finnish Funding Agency for Technology and Innovation, and by the companies mentioned in the project pages.

Author information

Authors and Affiliations

  1. Laboratory of Software Engineering, Lappeenranta University of Technology, 20, 53851, Lappeenranta, Finland

    Ossi Taipale, Jussi Kasurinen & Kari Smolander

  2. Bitten Oy, Hämeentie 32 B 68, 00530, Helsinki, Finland

    Katja Karhu

Authors
  1. Ossi Taipale
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jussi Kasurinen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Katja Karhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kari Smolander
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ossi Taipale.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Taipale, O., Kasurinen, J., Karhu, K. et al. Trade-off between automated and manual software testing. Int J Syst Assur Eng Manag 2, 114–125 (2011). https://doi.org/10.1007/s13198-011-0065-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13198-011-0065-6

Keywords

  • Software testing
  • Test automation
  • Manual testing
  • Benefits and drawbacks of test automation
  • Qualitative research
  • Empirical study