Correctness attraction: a study of stability of software behavior under runtime perturbation
Can the execution of software be perturbed without breaking the correctness of the output? In this paper, we devise a protocol to answer this question from a novel perspective. In an experimental study, we observe that many perturbations do not break the correctness in ten subject programs. We call this phenomenon “correctness attraction”. The uniqueness of this protocol is that it considers a systematic exploration of the perturbation space as well as perfect oracles to determine the correctness of the output. To this extent, our findings on the stability of software under execution perturbations have a level of validity that has never been reported before in the scarce related work. A qualitative manual analysis enables us to set up the first taxonomy ever of the reasons behind correctness attraction.
KeywordsPerturbation analysis Software correctness Empirical study
This work was partially supported by the EU Project STAMP ICT-16-10 No.731529, CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020, and the French Ministry of Higher Education and Research. We also wishes to acknowledge the continual support of Inria, and PP acknowledges the stimulating environment provided by the SequeL Inria project-team.
- Baudry B, Monperrus M (2015) The multiple facets of software diversity: recent developments in year 2000 and beyond. ACM Comput Surv 1–26Google Scholar
- Dijkstra EW (1988) On the cruelty of really teaching computing scienceGoogle Scholar
- Khoo W M (2013) Decompilation as search. University of Cambridge, PhD thesisGoogle Scholar
- Li X, Yeung D (2007) Application-level correctness and its impact on fault tolerance. In: 2007 IEEE 13th International symposium on high performance computer architecture, pp 181–192Google Scholar
- Morell L, Murrill B, Rand R (1997) Perturbation analysis of computer programs. In: Proceedings of the 12th annual conference on computer assurance, 1997. COMPASS ’97 Are we making progress towards computer assurance?, pp 77–87Google Scholar
- Rinard M, Cadar C, Nguyen HH (2005) Exploring the acceptability envelope. In: Companion to the 20th Annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA ’05. New York, p ACMGoogle Scholar
- Tallam S, Tian C, Gupta R, Zhang X (2008) Avoiding program failures through safe execution perturbations. In: Proceedings of the 2008 32Nd Annual IEEE international computer software and applications conference, COMPSAC ’08. IEEE Computer Society, Washington, DC, pp 152–159Google Scholar
- Tang E, Barr E, Li X, Su Z (2010) Perturbing numerical calculations for statistical analysis of floating-point program (in)stability. In: Proceedings of the 19th International symposium on software testing and analysis, ISSTA ’10. ACM, New York, pp 131–142Google Scholar
- Wang N, Fertig M, Patel S (2003) Y-branches: when you come to a fork in the road, take it. In: 12th International conference on parallel architectures and compilation techniques, pp 56–66Google Scholar