State Coverage: Software Validation Metrics beyond Code Coverage
- 1.4k Downloads
Currently, testing is still the most important approach to reduce the amount of software defects. Software quality metrics help to prioritize where additional testing is necessary by measuring the quality of the code. Most approaches to estimate whether some unit of code is sufficiently tested are based on code coverage, which measures what code fragments are exercised by the test suite. Unfortunately, code coverage does not measure to what extent the test suite checks the intended functionality.
We propose state coverage, a metric that measures the ratio of state updates that are read by assertions with respect to the total number of state updates, and we present efficient algorithms to measure state coverage. Like code coverage, state coverage is simple to understand and we show that it is effective to measure and easy to aggregate. During a preliminary evaluation on several open-source libraries, state coverage helped to identify multiple unchecked properties and detect several bugs.
Keywordsstate coverage test adequacy metric test oracle
Unable to display preview. Download preview PDF.
- 1.Barnett, G., Del Tongo, L.: Data Structures and Algorithms: Annotated Reference with Examples. NETSlackers (2008)Google Scholar
- 2.Barnett, G., Del Tongo, L.: Data structures and algorithms, dsa (2008), http://dsa.codeplex.com/
- 4.Chang, J., Richardson, D.J., Sankar, S.: Structural specification-based testing with adl. In: Proceedings of the 1996 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 1996, New York, NY, USA, pp. 62–70 (1996)Google Scholar
- 7.de Halleux, J.: Quickgraph: A 100% c# graph library with graphviz support (2007), http://www.codeproject.com/KB/miscctrl/quickgraph.aspx
- 9.Fähndrich, M., Barnett, M., Logozzo, F.: Embedded contract languages. In: SAC 2010: Proceedings of the 2010 ACM Symposium on Applied Computing, New York, NY, USA, pp. 2103–2110 (2010)Google Scholar
- 11.Heimdahl, M.P., George, D., Weber, R.: Specification test coverage adequacy criteria = specification test generation inadequacy criteria? In: IEEE International Symposium on High-Assurance Systems Engineering, pp. 178–186 (2004)Google Scholar
- 14.Koster, K., Kao, D.: State coverage: a structural test adequacy criterion for behavior checking. In: The 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers, ESEC-FSE Companion 2007, New York, NY, USA, pp. 541–544 (2007)Google Scholar
- 15.Kudrjavets, G., Nagappan, N., Ball, T.: Assessing the relationship between software assertions and faults: An empirical investigation. In: ISSRE 2006: Proceedings of the 17th International Symposium on Software Reliability Engineering, pp. 204–212. IEEE Computer Society, Washington, DC, USA (2006)CrossRefGoogle Scholar
- 17.N.I. of Standards and technology. The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3 (2002)Google Scholar
- 18.Osherove, R.: The Art of Unit Testing with examples in .NET. Manning Publications Co. (2009)Google Scholar
- 22.Song, Y., Thummalapenta, S., Xie, T.: Unitplus: assisting developer testing in eclipse. In: Eclipse 2007: Proceedings of the 2007 OOPSLA Workshop on Eclipse Technology Exchange, New York, NY, USA, pp. 26–30 (2007)Google Scholar
- 25.Vanoverberghe, D., de Halleux, J., Tillmann, N., Piessens, F.: State coverage: Software validation metrics beyond code coverage - extended version (2011), http://www.cs.kuleuven.be/publicaties/rapporten/cw/CW610.abs.html