A comprehensive study of pseudo-tested methods

Abstract

Pseudo-tested methods are defined as follows: they are covered by the test suite, yet no test case fails when the method body is removed, i.e., when all the effects of this method are suppressed. This intriguing concept was coined in 2016, by Niedermayr and colleagues, who showed that such methods are systematically present, even in well-tested projects with high statement coverage. This work presents a novel analysis of pseudo-tested methods. First, we run a replication of Niedermayr’s study with 28K+ methods, enhancing its external validity thanks to the use of new tools and new study subjects. Second, we perform a systematic characterization of these methods, both quantitatively and qualitatively with an extensive manual analysis of 101 pseudo-tested methods. The first part of the study confirms Niedermayr’s results: pseudo-tested methods exist in all our subjects. Our in-depth characterization of pseudo-tested methods leads to two key insights: pseudo-tested methods are significantly less tested than the other methods; yet, for most of them, the developers would not pay the testing price to fix this situation. This calls for future work on targeted test generation to specify those pseudo-tested methods without spending developer time.

This is a preview of subscription content, log in to check access.

Listing 1
Listing 2
Fig. 1
Listing 3
Fig. 2
Listing 4
Listing 5
Listing 6
Listing 7
Listing 8
Listing 9
Listing 10
Listing 11
Listing 12
Listing 13
Listing 14

Notes

  1. 1.

    Compared to Niedermayr et al. (2016), we add two new transformations, one to return null and another to return an empty array. These additions allow to expand the scope of methods to be analyzed.

  2. 2.

    https://github.com/STAMP-project/pitest-descartes

  3. 3.

    https://github.com/AlDanial/cloc

  4. 4.

    The computation of the Pearson coefficient and the Wilcoxon test were performed using the features of the R language.

  5. 5.

    http://cobertura.github.io/cobertura/

  6. 6.

    The violin plot for pseudo-tested methods of commons-cli and jopt-simple are not displayed, as they have too few methods in this category.

  7. 7.

    https://github.com/jhy/jsoup/blob/35e80a779b7908ddcd41a6a7df5f21b30bf999d2/src/main/java/org/jsoup/nodes/Entities.java#L295

  8. 8.

    https://github.com/aws/aws-sdk-java/pull/1437

  9. 9.

    https://github.com/apache/commons-collections/pull/36

  10. 10.

    https://github.com/apache/commons-codec/pull/13

  11. 11.

    https://github.com/apache/commons-io/pull/61

  12. 12.

    https://github.com/STAMP-project/descartes-experiments/blob/6f8a9c7c111a1da5794622652eae5327d0571ef1/direct-communications.md

  13. 13.

    https://github.com/INRIA/spoon/blob/fd878bc71b73fc1da82356eaa6578f760c70f0de/src/main/java/spoon/reflect/visitor/DefaultJavaPrettyPrinter.java#L479

  14. 14.

    https://github.com/INRIA/spoon/issues/1818

  15. 15.

    https://github.com/apache/flink/blob/740f711c4ec9c4b7cdefd01c9f64857c345a68a1/flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java#L86

  16. 16.

    https://gitlab.ow2.org/sat4j/sat4j/blob/09e9173e400ea6c1794354ca54c36607c53391ff/org.sat4j.core/src/main/java/org/sat4j/tools/xplain/Xplain.java#L214

  17. 17.

    https://gitlab.ow2.org/sat4j/sat4j/blob/09e9173e400ea6c1794354ca54c36607c53391ff/org.sat4j.core/src/main/java/org/sat4j/minisat/core/Solver.java#L384

  18. 18.

    https://gitlab.ow2.org/sat4j/sat4j/commit/afab137a4c1a54219f3990713b4647ff84b8bfea

  19. 19.

    https://gitlab.ow2.org/sat4j/sat4j/commit/46291e4d15a654477bd17b0ce905926d24e042ca

  20. 20.

    https://github.com/INRIA/spoon/blob/fd878bc71b73fc1da82356eaa6578f760c70f0de/src/main/java/spoon/support/reflect/eval/VisitorPartialEvaluator.java#L515

References

  1. Andrews JH, Briand LC, Labiche Y (2005) Is Mutation an Appropriate Tool for Testing Experiments? In: Proceedings of the 27th International Conference on Software engineering, ICSE ’05. ACM, New York, pp 402–411

  2. Androutsopoulos K, Clark D, Dan H, Hierons RM, Harman M (2014) An analysis of the relationship between conditional entropy and failed error propagation in software testing. In: Proceedings of the 36th international conference on software engineering. ACM, pp 573–583

  3. Coles H, Laurent T, Henard C, Papadakis M, Ventresque A (2016) PIT: A Practical Mutation Testing Tool for Java (Demo). In: Proceedings of the 25th International Symposium on Software Testing and analysis, ISSTA 2016. ACM, New York, pp 449–452

  4. Daran M, Thévenod-Fosse P (1996) Software Error Analysis: A Real Case Study Involving Real Faults And Mutations. In: Proceedings of the 1996 ACM SIGSOFT International Symposium on Software Testing and analysis, ISSTA ’96. ACM, New York, pp 158–171

  5. Delahaye M, Bousquet LD (2013) A Comparison of Mutation Analysis Tools for Java. In: 2013 13th International, Conference on Quality Software, pp 187–195

  6. Delamaro ME, Offutt J, Ammann P (2014) Designing deletion mutation operators. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp 11–20

  7. DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection Help for the practicing programmer. Computer 11(4):34–41

    Article  Google Scholar 

  8. Deng L, Offutt J, Li N (2013) Empirical evaluation of the statement deletion mutation operator. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp 84–93

  9. Durelli VHS, Souza NMD, Delamaro ME (2017) Are deletion mutants easier to identify manually? In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp 149–158

  10. Gopinath R, Ahmed I, Alipour MA, Jensen C, Groce A (2017) Does choice of mutation tool matter? Softw Qual J 25(3):871–920

    Article  Google Scholar 

  11. Gopinath R, Jensen C, Groce A (2014) Mutations: How close are they to real faults?. In: 2014 IEEE 25th international symposium on Software reliability engineering (ISSRE). IEEE, pp 189–200

  12. Gourlay JS (1983) A mathematical framework for the investigation of testing. IEEE Transactions on software engineering SE-9(6):686–709

    Article  MATH  Google Scholar 

  13. Jahangirova G, Clark D, Harman M, Tonella P (2016) Test oracle assessment and improvement. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, pp 247–258

  14. Just R, Schweiggert F, Kapfhammer GM (2011) MAJOR An efficient and extensible tool for mutation analysis in a Java compiler. In: Proceedings of the International Conference on Automated Software Engineering (ASE), pp 612–615

  15. Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are Mutants a Valid Substitute for Real Faults in Software Testing? In: Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software engineering, FSE 2014. ACM, New York, pp 654–665

  16. Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49(11-12):1073–1086

    Article  Google Scholar 

  17. Kintis M, Papadakis M, Papadopoulos A, Valvis E, Malevris N (2016) Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In: 2016 IEEE, 16th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 147–156

  18. Laurent T, Papadakis M, Kintis M, Henard C, Traon YL, Ventresque A (2017) Assessing and Improving the Mutation Testing Practice of PIT. In: 2017 IEEE, International Conference on Software Testing, Verification and Validation (ICST), pp 430–435

  19. Niedermayr R, Juergens E, Wagner S (2016) Will my tests tell me if I break this code?. In: Proceedings of the International Workshop on Continuous Software Evolution and delivery. ACM Press, New York, pp 23–29

  20. Petrovic G, Ivankovic M (2018) State of mutation testing at google. In: Proceedings of the 40th International Conference on Software Engineering 2017 (SEIP)

  21. Schuler D, Zeller A (2013) Checked coverage: an indicator for oracle quality. Softw Test Verification Reliab 23(7):531–551

    Article  Google Scholar 

  22. Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in Empirical Software Engineering. Empir Softw Eng 13(2):211–218

    Article  Google Scholar 

  23. Staats M, Whalen MW, Heimdahl MP (2011) Programs, tests, and oracles: the foundations of testing revisited. In: Proceedings of the 33rd international conference on software engineering. ACM, pp 391–400

  24. Untch RH (2009) On reduced neighborhood mutation analysis using a single mutagenic operator. In: Proceedings of the 47th Annual Southeast Regional Conference, ACM-SE 47. ACM, New York, pp 71:1–71:4

  25. Vera-Pérez OL, Monperrus M, Baudry B (2018) Descartes: A pitest engine to detect pseudo-tested methods. In: Proceedings of the 2018 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE ’18), pp 908–911

Download references

Acknowledgments

We would like to acknowledge the invaluable help and feedback provided by the development teams of authzforce, spoon and pdfbox. We also express our appreciation to Simon Urli, Daniel Le Berre, Arnaud Blouin, Marko Ivanković, Goran Petrovic and Andy Zaidman for their feedback and their very accurate suggestions. This work has been partially supported by the EU Project STAMP ICT-16-10 No.731529 and by the Wallenberg Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Oscar Luis Vera-Pérez.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Paolo Tonella

Appendix A: Source code for the Study Subjects

Appendix A: Source code for the Study Subjects

Legend

Column “Project” lists the projects included in the present study. Column “URL” contains links to the available source code. Column “Commit ID” contains the SHA-1 hash identifying the commit with the source code state that was used in this study.

Project URL/Commit ID
authzforce https://github.com/authzforce/core.git
  81ae56671bc343eabf2bc99ee0c51ba6ae28d649
aws-sdk-java https://github.com/aws/aws-sdk-java
  b5ae6ce44f4b5053a9a0255c9648f3073fafcf55
commons-cli https://github.com/apache/commons-cli
  c246bd419ee0efccd9a96f9d33486617d5d38a56
commons-codec https://github.com/apache/commons-codec
  e9da3d16ae67f2940a0bbdf982ecec19a0481981
commons-collections https://github.com/apache/commons-collections
  db189926f7415b9866e76cd8123e40c09c1cc67e
commons-io https://github.com/apache/commons-io
  e36d53170875d26d59ca94bd376bf40bc5690ee6
commons-lang https://github.com/apache/commons-lang
  e8f924f51be5bc8bcd583ea96e5ef25f9b2ca72a
flink-core https://github.com/apache/flink/tree/master/flink-core
  740f711c4ec9c4b7cdefd01c9f64857c345a68a1
gson https://github.com/google/gson
  c3d17e39f1cb6ec41496e639ab42f7e7cca3b465
jaxen https://github.com/jaxen-xpath/jaxen
  a8bd80599fd4d1c9aa1248d3276198535a30bfc5
jfreechart https://github.com/jfree/jfreechart
  a7156d4595ff7f6a7c8dac50625295c284b86732
jgit https://github.com/eclipse/jgit
  1513a5632dcaf8c6e2d6998427087e11ba35566d
joda-time https://github.com/JodaOrg/joda-time
  6ad133837a4c4f8199d00a05c3c16267dbf6deb8
jopt-simple https://github.com/jopt-simple/jopt-simple
  b38b70d1e7685766ab400d8b57ef9ca9c010e0bb
jsoup https://github.com/jhy/jsoup
  35e80a779b7908ddcd41a6a7df5f21b30bf999d2
pdfbox https://github.com/apache/pdfbox
  09e9173e400ea6c1794354ca54c36607c53391ff
sat4j-core https://gitlab.ow2.org/sat4j/sat4j/tree/master/org.sat4j.core
  1a0127645bf98b768ee3628076d0246596dd15eb
scifio https://github.com/scifio/scifio
  2760af6982ad18aab400e9cd99b9f63ef2495333
spoon https://github.com/INRIA/spoon
  fd878bc71b73fc1da82356eaa6578f760c70f0de
urbanairship https://github.com/urbanairship/java-library
  aafc049cc1cd3971c62a3dfc1d72cfe61160f32c
xwiki-rendering https://github.com/xwiki/xwiki-rendering
  cb3c444fb743e073eefbac2b44351a6166d94ac1

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Vera-Pérez, O.L., Danglot, B., Monperrus, M. et al. A comprehensive study of pseudo-tested methods. Empir Software Eng 24, 1195–1225 (2019). https://doi.org/10.1007/s10664-018-9653-2

Download citation

Keywords

  • Software testing
  • Software developers
  • Pseudo-tested methods
  • Test quality
  • Program analysis