Empirical Software Engineering

, Volume 23, Issue 4, pp 2086–2119 | Cite as

Correctness attraction: a study of stability of software behavior under runtime perturbation

  • Benjamin DanglotEmail author
  • Philippe Preux
  • Benoit Baudry
  • Martin Monperrus


Can the execution of software be perturbed without breaking the correctness of the output? In this paper, we devise a protocol to answer this question from a novel perspective. In an experimental study, we observe that many perturbations do not break the correctness in ten subject programs. We call this phenomenon “correctness attraction”. The uniqueness of this protocol is that it considers a systematic exploration of the perturbation space as well as perfect oracles to determine the correctness of the output. To this extent, our findings on the stability of software under execution perturbations have a level of validity that has never been reported before in the scarce related work. A qualitative manual analysis enables us to set up the first taxonomy ever of the reasons behind correctness attraction.


Perturbation analysis Software correctness Empirical study 



This work was partially supported by the EU Project STAMP ICT-16-10 No.731529, CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020, and the French Ministry of Higher Education and Research. We also wishes to acknowledge the continual support of Inria, and PP acknowledges the stimulating environment provided by the SequeL Inria project-team.


  1. Barr E, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525CrossRefGoogle Scholar
  2. Baudry B, Monperrus M (2015) The multiple facets of software diversity: recent developments in year 2000 and beyond. ACM Comput Surv 1–26Google Scholar
  3. Dijkstra EW (1988) On the cruelty of really teaching computing scienceGoogle Scholar
  4. Eggert PR, Parker DS (2005) Perturbing and evaluating numerical programs without recompilation—the Wonglediff way. Softw Pract Exper 35(4):313–322CrossRefGoogle Scholar
  5. Khoo W M (2013) Decompilation as search. University of Cambridge, PhD thesisGoogle Scholar
  6. Li X, Yeung D (2007) Application-level correctness and its impact on fault tolerance. In: 2007 IEEE 13th International symposium on high performance computer architecture, pp 181–192Google Scholar
  7. Mittal S (2016) A survey of techniques for approximate computing. ACM Comput Surv 48(4):62,1–62,33MathSciNetGoogle Scholar
  8. Morell L, Murrill B, Rand R (1997) Perturbation analysis of computer programs. In: Proceedings of the 12th annual conference on computer assurance, 1997. COMPASS ’97 Are we making progress towards computer assurance?, pp 77–87Google Scholar
  9. Pawlak R, Monperrus M, Petitprez N, Noguera C, Seinturier L (2015) Spoon: a library for implementing analyses and transformations of java source code. Softw Pract Exper 46:1155–1179CrossRefGoogle Scholar
  10. Rinard M, Cadar C, Nguyen HH (2005) Exploring the acceptability envelope. In: Companion to the 20th Annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA ’05. New York, p ACMGoogle Scholar
  11. Roy P, Ray R, Wang C, Wong WF (2014) Asac: automatic sensitivity analysis for approximate computing. SIGPLAN Not 49(5):95–104CrossRefGoogle Scholar
  12. Sedgewick R (1978) Implementing quicksort programs. Commun ACM 21(10):847–857CrossRefzbMATHGoogle Scholar
  13. Tallam S, Tian C, Gupta R, Zhang X (2008) Avoiding program failures through safe execution perturbations. In: Proceedings of the 2008 32Nd Annual IEEE international computer software and applications conference, COMPSAC ’08. IEEE Computer Society, Washington, DC, pp 152–159Google Scholar
  14. Tang E, Barr E, Li X, Su Z (2010) Perturbing numerical calculations for statistical analysis of floating-point program (in)stability. In: Proceedings of the 19th International symposium on software testing and analysis, ISSTA ’10. ACM, New York, pp 131–142Google Scholar
  15. Wang N, Fertig M, Patel S (2003) Y-branches: when you come to a fork in the road, take it. In: 12th International conference on parallel architectures and compilation techniques, pp 56–66Google Scholar
  16. Welch TA (1984) A technique for high-performance data compression. Computer 17(6):8–19CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Benjamin Danglot
    • 1
    Email author
  • Philippe Preux
    • 1
    • 2
  • Benoit Baudry
    • 3
  • Martin Monperrus
    • 4
  1. 1.Inria Lille - Nord EuropeInriaVilleneuve-d’AscqFrance
  2. 2.Université LilleVilleneuve-d’AscqFrance
  3. 3.Inria Rennes - Bretagne AtlantiqueInria RennesRennesFrance
  4. 4.Université de Lille 1 - Sciences et TechnologiesKTH Royal Institute of TechnologyStockholmSuède

Personalised recommendations