Skip to main content

Injecting software faults in Python applications

The OpenStack case study

Abstract

Software fault injection techniques have been largely used as means for evaluating the dependability of systems in presence of certain types of faults. Despite the large diversity of tools offering the possibility of emulating the presence of software faults, there is little practical support for emulating the presence of software faults in Python applications, which are increasingly being used to support business critical cloud services. In this paper, we present FIT4Python, a tool for injecting software faults in Python code and then use it, in a mutation testing campaign, to analyse the effectiveness of OpenStack’s test suite against new probable software faults. We begin by analysing the types of faults affecting Nova Compute, the core component of OpenStack. We use our tool to emulate the presence of new faults in Nova Compute API to understand how well OpenStack’s battery of unit, functional, and integration tests cover these new, but probable, situations. Results show clear limitations in the effectiveness of OpenStack developers’ test suites, with many cases of injected faults passing undetected through all three types of tests and that nearly half of the analysed problems could be detected with trivial changes or additions to the unit tests.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Listing 1
Listing 2
Listing 3
Fig. 3

References

  1. Agnelo J, Laranjeiro N, Bernardino J (2020) Using orthogonal defect classification to characterize nosql database defects. J Syst Softw 159:110,451. https://doi.org/10.1016/j.jss.2019.110451. https://www.sciencedirect.com/science/article/pii/S0164121219302250

    Article  Google Scholar 

  2. Arlat J, Crouzet Y, Karlsson J, Folkesson P, Fuchs E, Leber GH (2003) Comparison of physical and software-implemented fault injection techniques. IEEE Trans Comput 52(9):1115–1133. https://doi.org/10.1109/TC.2003.1228509

    Article  Google Scholar 

  3. AS SN (2021) Cosmic ray. https://pypi.org/project/cosmic-ray, original-date: 2015-04-18T07:44:21Z

  4. Barton JH, Czeck EW, Segall ZZ, Siewiorek DP (1990) Fault injection experiments using fiat. IEEE Trans Comput 39(4):575–582. https://doi.org/10.1109/12.54853

    Article  Google Scholar 

  5. Byoungju C, Mathur AP (1993) High-performance mutation testing. J Syst Softw 20(2):135–152. https://doi.org/10.1016/0164-1212(93)90005-I. http://www.sciencedirect.com/science/article/pii/016412129390005I

    Article  Google Scholar 

  6. Chillarege R (1996) Orthogonal defect classification. In: Lyu MR (ed) Handbook of software reliability engineering. IEEE CS Press, chap 9, pp 359–399

  7. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. https://doi.org/10.1177/001316446002000104

    Article  Google Scholar 

  8. Cotroneo D, De Simone L, Liguori P, Natella R, Bidokhti N (2019) How bad can a bug get? an empirical analysis of software failures in the OpenStack cloud computing platform. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC/FSE 2019. ACM, New York, pp 200-211. https://doi.org/10.1145/3338906.3338916

  9. Cotroneo D, Simone LD, Liguori P, Natella R (2020) ProFIPy: programmable software fault injection as-a-service. In: 2020 50th annual IEEE/IFIP international conference on dependable systems and networks (DSN). pp 364–372. https://doi.org/10.1109/DSN48063.2020.00052

  10. Delgado-Pérez P, Segura S, Medina-Bulo I (2017) Assessment of C++ object-oriented mutation operators: A selective mutation approach. Software Testing Verification Reliab 27(4-5):e1630. https://doi.org/10.1002/stvr.1630

    Article  Google Scholar 

  11. Denisov A, Pankevich S (2018) Mull it over: mutation testing based on LLVM. In: 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 25–31. https://doi.org/10.1109/ICSTW.2018.00024

  12. Derezińska A (2013) A quality estimation of mutation clustering in c# programs. In: Zamojski W, Mazurkiewicz J, Sugier J, Walkowiak T, Kacprzyk J (eds) New results in dependability and computer systems, advances in Intelligent Systems and Computing. Springer International Publishing, Heidelberg, pp 119–129. https://doi.org/10.1007/978-3-319-00945-2_11

  13. Durães JA, Madeira HS (2006) Emulation of software faults: A field data study and a practical approach. IEEE Trans Softw Eng 32(11)

  14. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Statist 32(2):407–499. https://doi.org/10.1214/009053604000000067

    MathSciNet  Article  Google Scholar 

  15. Fonseca J, Vieira M, Madeira H (2014) Evaluation of web security mechanisms using vulnerability attack injection. IEEE Trans Dependable Secur Comput 11(5):440–453. https://doi.org/10.1109/TDSC.2013.45

    Article  Google Scholar 

  16. Fraser G, Arcuri A (2015) Achieving scalable mutation-based generation of whole test suites. https://doi.org/10.1007/s10664-013-9299-z, vol 20, pp 783–812

  17. Grady RB (1992) Practical software metrics for project management and process improvement. Prentice-Hall, Inc., Upper Saddle River

    Google Scholar 

  18. Hałas K (2021) MutPy: Mutation testing tool for Python 3.x. https://pypi.org/project/mutpy

  19. Hajdu A, Ivaki N, Kocsis I, Klenik A, Gönczy L, Laranjeiro N, Madeira H, Pataricza A (2020) Using fault injection to assess blockchain systems in presence of faulty smart contracts. https://doi.org/10.1109/ACCESS.2020.3032239, vol 8, pp 190,760–190,783

  20. Hovmöller A (2021) Mutmut: mutation testing for Python 3. https://pypi.org/project/mutmut

  21. Hsueh MC, Tsai TK, Iyer RK (1997) Fault injection techniques and tools. Computer 30(4):75–82. http://portal.acm.org/citation.cfm?id=619017.620685&coll=Portal&dl=GUIDE&CFID=74933873&CFTOKEN=73521499

    Article  Google Scholar 

  22. Hudak J, Suh BH, Siewiorek D, Segall Z (1993) Evaluation and comparison of fault-tolerant software techniques. IEEE Trans Reliab 42(2):190–204. https://doi.org/10.1109/24.229487

    Article  Google Scholar 

  23. IBM (2013) Orthogonal defect classification v 5.2 for software design and code. https://researcher.watson.ibm.com/researcher/files/us-pasanth/ODC-5-2.pdf

  24. IEEE (2010) IEEE standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993) pp 1?23. https://doi.org/10.1109/IEEESTD.2010.5399061

  25. Just R, Schweiggert F (2015) Higher accuracy and lower run time: efficient mutation analysis using non-redundant mutation operators. Software Testing Verification Reliab 25(5-7):490–507. https://doi.org/10.1002/stvr.1561

    Article  Google Scholar 

  26. Kanoun K, Spainhower L (2008) Dependability benchmarking for computer systems. Wiley-IEEE Computer Society Press, Washington

    Book  Google Scholar 

  27. Kao W, Iyer RK, Tang D (1993) Fine: A fault injection and monitoring environment for tracing the unix system behavior under faults. IEEE Trans Softw Eng 19(11):1105–1118. https://doi.org/10.1109/32.256857

    Article  Google Scholar 

  28. Kepner E (2021) Mutatest. https://pypi.org/project/mutatest/, original-date: 2018-12-22T15:04:53Z

  29. Kintis M, Papadakis M, Jia Y, Malevris N, Traon YL, Harman M (2018) Detecting trivial mutant equivalences via compiler optimisations. https://doi.org/10.1109/TSE.2017.2684805. https://www.computer.org/csdl/journal/ts/2018/04/07882714/13rRUxBa5ty, publisher: IEEE Computer Society, vol 44, pp 308–333

  30. Koopman P, DeVale J (2000) The exception handling effectiveness of posix operating systems. IEEE Trans Softw Eng 26(9):837–848. https://doi.org/10.1109/32.877845

    Article  Google Scholar 

  31. Kropp NP, Koopman PJ, Siewiorek DP (1998) Automated robustness testing of off-the-shelf software components. In: Proceedings of the the twenty-eighth annual international symposium on fault-tolerant computing, FTCS ’98. IEEE Computer Society, Washington, pp 230–. http://dl.acm.org/citation.cfm?id=795671.796919

  32. Kurtz B, Ammann P, Offutt J, Delamaro ME, Kurtz M, Gökçe N (2016) Analyzing the validity of selective mutation with dominator mutants. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. Association for Computing Machinery, New York, pp 571-582. https://doi.org/10.1145/2950290.2950322

  33. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1):159–174. https://doi.org/10.2307/2529310. https://www.jstor.org/stable/2529310

    Article  Google Scholar 

  34. Laranjeiro N, Agnelo J, Bernardino J (2021) A Systematic Review on Software Robustness Assessment. ACM Comput Surv 54(4):89:1–89:65. https://doi.org/10.1145/3448977

    Article  Google Scholar 

  35. Li N, West M, Escalona A, Durelli VHS (2015) Mutation testing in practice using Ruby. In: 2015 IEEE eighth international conference on software testing, verification and validation workshops (ICSTW). IEEE Computer Society. pp 1–6. https://doi.org/10.1109/ICSTW.2015.7107453. https://www.computer.org/csdl/proceedings-article/icstw/2015/07107453/12OmNvUaNme

  36. Lima JAP, Vergilio SR (2018) Search-Based Higher Order Mutation Testing: A Mapping Study. In: Proceedings of the III Brazilian symposium on systematic and automated software testing, SAST ’18. Association for Computing Machinery, New York, pp 87–96. https://doi.org/10.1145/3266003.3266013.

  37. Mahmood A, Andrews DM, McCluskey EJ (1984) Executable assertions and flight software. In: Digital avionics systems conference, american institute of aeronautics and astronautics. pp 346–351. https://doi.org/10.2514/6.1984-2726.

  38. Marques H, Laranjeiro N, Bernardino J (2021) Injecting software faults in Python applications: The OpenStack case study - supplemental material. https://doi.org/10.5281/zenodo.4581121

  39. Martins E, Rubira C, Leme N (2002) Jaca: a reflective fault injection tool based on patterns. In: Proceedings international conference on dependable systems and networks. pp 483–487. https://doi.org/10.1109/DSN.2002.1028934

  40. McCabe T (1976) A complexity measure. IEEE Transactions on Software Engineering SE-2(4):308–320. https://doi.org/10.1109/TSE.1976.233837

    MathSciNet  Article  Google Scholar 

  41. Miller BP, Koski D, Pheow C, Maganty LV, Murthy R, Natarajan A, Steidl J (1995) Fuzz revisited: A re-examination of the reliability of unix utilities and services. Tech. rep., University of Wisconsin-Madison Department of Computer Sciences

  42. Namin AS, Andrews J, Murdoch D (2008) Sufficient mutation operators for measuring test effectiveness. In: 2008 ACM/IEEE 30th international conference on software engineering. pp 351–360. https://doi.org/10.1145/1368088.1368136

  43. Natella R (2011) Achieving representative faultloads in software fault injection. PhD Thesis, Università degli Studi di Napoli Federico II, Napoli, Italy. https://doi.org/10.6092/UNINA/FEDOA/8833. http://www.fedoa.unina.it/8833/

  44. Natella R, Cotroneo D, Madeira HS (2016a) Assessing dependability with software fault injection: a survey. https://doi.org/10.1145/2841425, vol 48, pp 44:1–44:55

  45. Natella R, Cotroneo D, Madeira HS (2016b) Assessing dependability with software fault injection: a survey. https://doi.org/10.1145/2841425, vol 48, pp 1–55

  46. Ng W, Chen P (2001) The design and verification of the Rio file cache. IEEE Trans Comput 50(4):322–337. https://doi.org/10.1109/12.919278

    Article  Google Scholar 

  47. OpenStack (2021) Conceptual architecture. https://docs.openstack.org/install-guide/get-started-conceptual-architecture.html

  48. Petrović G, Ivanković M (2018) State of mutation testing at google. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18. Association for Computing Machinery, New York, pp 163–171. https://doi.org/10.1145/3183519.3183521.

  49. Pizzoleto AV, Ferrari FC, Offutt J, Fernandes L, Ribeiro M (2019) A systematic literature review of techniques and metrics to reduce the cost of mutation testing. J Syst Softw 157:110,388. https://doi.org/10.1016/j.jss.2019.07.100. http://www.sciencedirect.com/science/article/pii/S0164121219301554

    Article  Google Scholar 

  50. Praphamontripong U, Offutt J (2017) Finding redundancy in web mutation operators. In: 2017 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 134–142. https://doi.org/10.1109/ICSTW.2017.30

  51. Praphamontripong U, Offutt J (2017) Finding redundancy in web mutation operators. In: 2017 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 134–142. https://doi.org/10.1109/ICSTW.2017.30

  52. Sanches BP, Basso T, Moraes R (2011) J-SWFIT: a java software fault injection tool. In: 2011 5th Latin-American symposium on dependable computing. pp 106–115. https://doi.org/10.1109/LADC.2011.20

  53. Tiobe (2021) TIOBE Index. https://www.tiobe.com/tiobe-index/

  54. Wang B, Xiong Y, Shi Y, Zhang L, Hao D (2017) Faster mutation analysis via equivalence modulo states. In: Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2017. Association for Computing Machinery, New York, pp 295–306. https://doi.org/10.1145/3092703.3092714

  55. Wei-Lun K, Iyer RK (1994) Define: a distributed fault injection and monitoring environment. In: Proceedings of IEEE workshop on fault-tolerant parallel and distributed systems. pp 252–259. https://doi.org/10.1109/FTPDS.1994.494497

  56. Wilson PF, Dell L, Anderson G (1993) Root cause analysis: A tool for total quality management. ASQ Quality Press, Milwaukee

    Google Scholar 

  57. Wood A (1996) Software reliability growth models. Technical Report 96.1, Tandem Computers

  58. Woodward M (1993) Mutation testing—its origin and evolution. Inf Softw Technol 35(3):163–169. https://doi.org/10.1016/0950-5849(93)90053-6. http://www.sciencedirect.com/science/article/pii/0950584993900536

    Article  Google Scholar 

  59. Zhang L, Gligoric M, Marinov D, Khurshid S (2013) Operator-based and random mutant selection: Better together. In: 2013 28th IEEE/ACM international conference on automated software engineering (ASE). pp 92–102. https://doi.org/10.1109/ASE.2013.6693070

  60. Zhang L, Marinov D, Khurshid S (2013) Faster mutation testing inspired by test prioritization and reduction. In: Proceedings of the 2013 international symposium on software testing and analysis, ISSTA 2013. Association for Computing Machinery, New York, pp 235-245. https://doi.org/10.1145/2483760.2483782

Download references

Acknowledgments

This work has been supported by national funds through the FCT - Foundation for Science and Technology, I.P., within the scope of the project CISUC - UID/CEC/00326/2020 and by European Social Fund, through the Regional Operational Program Centro 2020; by project VALU3S - Verification and Validation of Automated Systems’ Safety and Security, H2020-EU.2.1.1.7. - ECSEL, Grant agreement ID: 876852; and by project AIDA: Adaptive, Intelligent and Distributed Assurance Platform project, co-financed by the European Regional Development Fund (ERDF) through the Operational Program for Competitiveness and Internationalization – COMPETE 2020 (POCI-01-0247-FEDER-045907) and by the Portuguese Foundation for Science and Technology under CMU Portugal Program. We also would like to acknowledge the National Distributed Computing Infrastructure (INCD) for providing access to computing resources.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Nuno Laranjeiro.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Paolo Tonella

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Marques, H., Laranjeiro, N. & Bernardino, J. Injecting software faults in Python applications. Empir Software Eng 27, 20 (2022). https://doi.org/10.1007/s10664-021-10047-9

Download citation

Keywords

  • Software faults
  • Bug reports
  • Orthogonal defect classification
  • Fault injection
  • Mutation testing
  • Dependability evaluation