Abstract
Software fault injection techniques have been largely used as means for evaluating the dependability of systems in presence of certain types of faults. Despite the large diversity of tools offering the possibility of emulating the presence of software faults, there is little practical support for emulating the presence of software faults in Python applications, which are increasingly being used to support business critical cloud services. In this paper, we present FIT4Python, a tool for injecting software faults in Python code and then use it, in a mutation testing campaign, to analyse the effectiveness of OpenStack’s test suite against new probable software faults. We begin by analysing the types of faults affecting Nova Compute, the core component of OpenStack. We use our tool to emulate the presence of new faults in Nova Compute API to understand how well OpenStack’s battery of unit, functional, and integration tests cover these new, but probable, situations. Results show clear limitations in the effectiveness of OpenStack developers’ test suites, with many cases of injected faults passing undetected through all three types of tests and that nearly half of the analysed problems could be detected with trivial changes or additions to the unit tests.
Similar content being viewed by others
References
Agnelo J, Laranjeiro N, Bernardino J (2020) Using orthogonal defect classification to characterize nosql database defects. J Syst Softw 159:110,451. https://doi.org/10.1016/j.jss.2019.110451. https://www.sciencedirect.com/science/article/pii/S0164121219302250
Arlat J, Crouzet Y, Karlsson J, Folkesson P, Fuchs E, Leber GH (2003) Comparison of physical and software-implemented fault injection techniques. IEEE Trans Comput 52(9):1115–1133. https://doi.org/10.1109/TC.2003.1228509
AS SN (2021) Cosmic ray. https://pypi.org/project/cosmic-ray, original-date: 2015-04-18T07:44:21Z
Barton JH, Czeck EW, Segall ZZ, Siewiorek DP (1990) Fault injection experiments using fiat. IEEE Trans Comput 39(4):575–582. https://doi.org/10.1109/12.54853
Byoungju C, Mathur AP (1993) High-performance mutation testing. J Syst Softw 20(2):135–152. https://doi.org/10.1016/0164-1212(93)90005-I. http://www.sciencedirect.com/science/article/pii/016412129390005I
Chillarege R (1996) Orthogonal defect classification. In: Lyu MR (ed) Handbook of software reliability engineering. IEEE CS Press, chap 9, pp 359–399
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. https://doi.org/10.1177/001316446002000104
Cotroneo D, De Simone L, Liguori P, Natella R, Bidokhti N (2019) How bad can a bug get? an empirical analysis of software failures in the OpenStack cloud computing platform. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC/FSE 2019. ACM, New York, pp 200-211. https://doi.org/10.1145/3338906.3338916
Cotroneo D, Simone LD, Liguori P, Natella R (2020) ProFIPy: programmable software fault injection as-a-service. In: 2020 50th annual IEEE/IFIP international conference on dependable systems and networks (DSN). pp 364–372. https://doi.org/10.1109/DSN48063.2020.00052
Delgado-Pérez P, Segura S, Medina-Bulo I (2017) Assessment of C++ object-oriented mutation operators: A selective mutation approach. Software Testing Verification Reliab 27(4-5):e1630. https://doi.org/10.1002/stvr.1630
Denisov A, Pankevich S (2018) Mull it over: mutation testing based on LLVM. In: 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 25–31. https://doi.org/10.1109/ICSTW.2018.00024
Derezińska A (2013) A quality estimation of mutation clustering in c# programs. In: Zamojski W, Mazurkiewicz J, Sugier J, Walkowiak T, Kacprzyk J (eds) New results in dependability and computer systems, advances in Intelligent Systems and Computing. Springer International Publishing, Heidelberg, pp 119–129. https://doi.org/10.1007/978-3-319-00945-2_11
Durães JA, Madeira HS (2006) Emulation of software faults: A field data study and a practical approach. IEEE Trans Softw Eng 32(11)
Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Statist 32(2):407–499. https://doi.org/10.1214/009053604000000067
Fonseca J, Vieira M, Madeira H (2014) Evaluation of web security mechanisms using vulnerability attack injection. IEEE Trans Dependable Secur Comput 11(5):440–453. https://doi.org/10.1109/TDSC.2013.45
Fraser G, Arcuri A (2015) Achieving scalable mutation-based generation of whole test suites. https://doi.org/10.1007/s10664-013-9299-z, vol 20, pp 783–812
Grady RB (1992) Practical software metrics for project management and process improvement. Prentice-Hall, Inc., Upper Saddle River
Hałas K (2021) MutPy: Mutation testing tool for Python 3.x. https://pypi.org/project/mutpy
Hajdu A, Ivaki N, Kocsis I, Klenik A, Gönczy L, Laranjeiro N, Madeira H, Pataricza A (2020) Using fault injection to assess blockchain systems in presence of faulty smart contracts. https://doi.org/10.1109/ACCESS.2020.3032239, vol 8, pp 190,760–190,783
Hovmöller A (2021) Mutmut: mutation testing for Python 3. https://pypi.org/project/mutmut
Hsueh MC, Tsai TK, Iyer RK (1997) Fault injection techniques and tools. Computer 30(4):75–82. http://portal.acm.org/citation.cfm?id=619017.620685&coll=Portal&dl=GUIDE&CFID=74933873&CFTOKEN=73521499
Hudak J, Suh BH, Siewiorek D, Segall Z (1993) Evaluation and comparison of fault-tolerant software techniques. IEEE Trans Reliab 42(2):190–204. https://doi.org/10.1109/24.229487
IBM (2013) Orthogonal defect classification v 5.2 for software design and code. https://researcher.watson.ibm.com/researcher/files/us-pasanth/ODC-5-2.pdf
IEEE (2010) IEEE standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993) pp 1?23. https://doi.org/10.1109/IEEESTD.2010.5399061
Just R, Schweiggert F (2015) Higher accuracy and lower run time: efficient mutation analysis using non-redundant mutation operators. Software Testing Verification Reliab 25(5-7):490–507. https://doi.org/10.1002/stvr.1561
Kanoun K, Spainhower L (2008) Dependability benchmarking for computer systems. Wiley-IEEE Computer Society Press, Washington
Kao W, Iyer RK, Tang D (1993) Fine: A fault injection and monitoring environment for tracing the unix system behavior under faults. IEEE Trans Softw Eng 19(11):1105–1118. https://doi.org/10.1109/32.256857
Kepner E (2021) Mutatest. https://pypi.org/project/mutatest/, original-date: 2018-12-22T15:04:53Z
Kintis M, Papadakis M, Jia Y, Malevris N, Traon YL, Harman M (2018) Detecting trivial mutant equivalences via compiler optimisations. https://doi.org/10.1109/TSE.2017.2684805. https://www.computer.org/csdl/journal/ts/2018/04/07882714/13rRUxBa5ty, publisher: IEEE Computer Society, vol 44, pp 308–333
Koopman P, DeVale J (2000) The exception handling effectiveness of posix operating systems. IEEE Trans Softw Eng 26(9):837–848. https://doi.org/10.1109/32.877845
Kropp NP, Koopman PJ, Siewiorek DP (1998) Automated robustness testing of off-the-shelf software components. In: Proceedings of the the twenty-eighth annual international symposium on fault-tolerant computing, FTCS ’98. IEEE Computer Society, Washington, pp 230–. http://dl.acm.org/citation.cfm?id=795671.796919
Kurtz B, Ammann P, Offutt J, Delamaro ME, Kurtz M, Gökçe N (2016) Analyzing the validity of selective mutation with dominator mutants. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. Association for Computing Machinery, New York, pp 571-582. https://doi.org/10.1145/2950290.2950322
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1):159–174. https://doi.org/10.2307/2529310. https://www.jstor.org/stable/2529310
Laranjeiro N, Agnelo J, Bernardino J (2021) A Systematic Review on Software Robustness Assessment. ACM Comput Surv 54(4):89:1–89:65. https://doi.org/10.1145/3448977
Li N, West M, Escalona A, Durelli VHS (2015) Mutation testing in practice using Ruby. In: 2015 IEEE eighth international conference on software testing, verification and validation workshops (ICSTW). IEEE Computer Society. pp 1–6. https://doi.org/10.1109/ICSTW.2015.7107453. https://www.computer.org/csdl/proceedings-article/icstw/2015/07107453/12OmNvUaNme
Lima JAP, Vergilio SR (2018) Search-Based Higher Order Mutation Testing: A Mapping Study. In: Proceedings of the III Brazilian symposium on systematic and automated software testing, SAST ’18. Association for Computing Machinery, New York, pp 87–96. https://doi.org/10.1145/3266003.3266013.
Mahmood A, Andrews DM, McCluskey EJ (1984) Executable assertions and flight software. In: Digital avionics systems conference, american institute of aeronautics and astronautics. pp 346–351. https://doi.org/10.2514/6.1984-2726.
Marques H, Laranjeiro N, Bernardino J (2021) Injecting software faults in Python applications: The OpenStack case study - supplemental material. https://doi.org/10.5281/zenodo.4581121
Martins E, Rubira C, Leme N (2002) Jaca: a reflective fault injection tool based on patterns. In: Proceedings international conference on dependable systems and networks. pp 483–487. https://doi.org/10.1109/DSN.2002.1028934
McCabe T (1976) A complexity measure. IEEE Transactions on Software Engineering SE-2(4):308–320. https://doi.org/10.1109/TSE.1976.233837
Miller BP, Koski D, Pheow C, Maganty LV, Murthy R, Natarajan A, Steidl J (1995) Fuzz revisited: A re-examination of the reliability of unix utilities and services. Tech. rep., University of Wisconsin-Madison Department of Computer Sciences
Namin AS, Andrews J, Murdoch D (2008) Sufficient mutation operators for measuring test effectiveness. In: 2008 ACM/IEEE 30th international conference on software engineering. pp 351–360. https://doi.org/10.1145/1368088.1368136
Natella R (2011) Achieving representative faultloads in software fault injection. PhD Thesis, Università degli Studi di Napoli Federico II, Napoli, Italy. https://doi.org/10.6092/UNINA/FEDOA/8833. http://www.fedoa.unina.it/8833/
Natella R, Cotroneo D, Madeira HS (2016a) Assessing dependability with software fault injection: a survey. https://doi.org/10.1145/2841425, vol 48, pp 44:1–44:55
Natella R, Cotroneo D, Madeira HS (2016b) Assessing dependability with software fault injection: a survey. https://doi.org/10.1145/2841425, vol 48, pp 1–55
Ng W, Chen P (2001) The design and verification of the Rio file cache. IEEE Trans Comput 50(4):322–337. https://doi.org/10.1109/12.919278
OpenStack (2021) Conceptual architecture. https://docs.openstack.org/install-guide/get-started-conceptual-architecture.html
Petrović G, Ivanković M (2018) State of mutation testing at google. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18. Association for Computing Machinery, New York, pp 163–171. https://doi.org/10.1145/3183519.3183521.
Pizzoleto AV, Ferrari FC, Offutt J, Fernandes L, Ribeiro M (2019) A systematic literature review of techniques and metrics to reduce the cost of mutation testing. J Syst Softw 157:110,388. https://doi.org/10.1016/j.jss.2019.07.100. http://www.sciencedirect.com/science/article/pii/S0164121219301554
Praphamontripong U, Offutt J (2017) Finding redundancy in web mutation operators. In: 2017 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 134–142. https://doi.org/10.1109/ICSTW.2017.30
Praphamontripong U, Offutt J (2017) Finding redundancy in web mutation operators. In: 2017 IEEE international conference on software testing, verification and validation workshops (ICSTW). pp 134–142. https://doi.org/10.1109/ICSTW.2017.30
Sanches BP, Basso T, Moraes R (2011) J-SWFIT: a java software fault injection tool. In: 2011 5th Latin-American symposium on dependable computing. pp 106–115. https://doi.org/10.1109/LADC.2011.20
Tiobe (2021) TIOBE Index. https://www.tiobe.com/tiobe-index/
Wang B, Xiong Y, Shi Y, Zhang L, Hao D (2017) Faster mutation analysis via equivalence modulo states. In: Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2017. Association for Computing Machinery, New York, pp 295–306. https://doi.org/10.1145/3092703.3092714
Wei-Lun K, Iyer RK (1994) Define: a distributed fault injection and monitoring environment. In: Proceedings of IEEE workshop on fault-tolerant parallel and distributed systems. pp 252–259. https://doi.org/10.1109/FTPDS.1994.494497
Wilson PF, Dell L, Anderson G (1993) Root cause analysis: A tool for total quality management. ASQ Quality Press, Milwaukee
Wood A (1996) Software reliability growth models. Technical Report 96.1, Tandem Computers
Woodward M (1993) Mutation testing—its origin and evolution. Inf Softw Technol 35(3):163–169. https://doi.org/10.1016/0950-5849(93)90053-6. http://www.sciencedirect.com/science/article/pii/0950584993900536
Zhang L, Gligoric M, Marinov D, Khurshid S (2013) Operator-based and random mutant selection: Better together. In: 2013 28th IEEE/ACM international conference on automated software engineering (ASE). pp 92–102. https://doi.org/10.1109/ASE.2013.6693070
Zhang L, Marinov D, Khurshid S (2013) Faster mutation testing inspired by test prioritization and reduction. In: Proceedings of the 2013 international symposium on software testing and analysis, ISSTA 2013. Association for Computing Machinery, New York, pp 235-245. https://doi.org/10.1145/2483760.2483782
Acknowledgments
This work has been supported by national funds through the FCT - Foundation for Science and Technology, I.P., within the scope of the project CISUC - UID/CEC/00326/2020 and by European Social Fund, through the Regional Operational Program Centro 2020; by project VALU3S - Verification and Validation of Automated Systems’ Safety and Security, H2020-EU.2.1.1.7. - ECSEL, Grant agreement ID: 876852; and by project AIDA: Adaptive, Intelligent and Distributed Assurance Platform project, co-financed by the European Regional Development Fund (ERDF) through the Operational Program for Competitiveness and Internationalization – COMPETE 2020 (POCI-01-0247-FEDER-045907) and by the Portuguese Foundation for Science and Technology under CMU Portugal Program. We also would like to acknowledge the National Distributed Computing Infrastructure (INCD) for providing access to computing resources.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Paolo Tonella
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Marques, H., Laranjeiro, N. & Bernardino, J. Injecting software faults in Python applications. Empir Software Eng 27, 20 (2022). https://doi.org/10.1007/s10664-021-10047-9
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-021-10047-9