Abstract
Structural code coverage criteria have been studied since the early seventies, and now they are well supported by commercial and open-source tools and are commonly embedded in several advanced industrial processes. Most industrial applications still refer to simple criteria, like statement and branch coverage, and consider complex criteria, like modified condition decision coverage, only rarely and often driven by the requirements of certification agencies. The industrial value of structural criteria is limited by the difficulty of achieving high coverage, due to both the complexity of deriving test cases that execute specific uncovered elements and the presence of many infeasible elements in the code. In this paper, we propose a technique that both generates test cases that execute yet uncovered branches and identifies infeasible branches that can be eliminated from the computation of the branch coverage. In this way, we can increase branch coverage to closely approximate full coverage, thus improving its industrial value. The algorithm combines symbolic analysis, abstraction refinement, and a novel technique named coarsening, to execute unexplored branches, identify infeasible ones, and mitigate the state space explosion problem. In the paper, we present the technique and illustrate its effectiveness through a set of experimental results obtained with a prototype implementation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Notice that both the problem of covering all feasible elements and the problem of revealing all infeasible elements of a program are undecidable in general, and thus our approach may not terminate on some programs. When this is the case, our approach stops after a timeout and reports the elements that have been neither covered nor identified as infeasible.
Function calc _ week takes two parameters, l _ time (a date) and week _ behavior (the week counting options), and returns the corresponding week of the year (an integer value between 0 and 53). The parameter week _ behavior is interpreted as a sequence of bits that indicate the day that starts the week (either Sunday or Monday), the baseline to count weeks (either 0 or 1), and the reference standard for the date representation (ISO standard 8601:1988 or not), respectively. In the context of a specific application, calc _ week is typically used by passing a fixed constant value of week _ behavior to all calls. This may cause some of the branches of week _ behavior to be infeasible within a specific application, as it happens in general when reusable libraries are integrated in systems that use only subsets of their functionalities. For further details on this example, we refer the interested readers to Baluda et al. (2010).
Arc-B counts the static branches after the Cil pre-compilation that unrolls decisions with multiple conditions as a cascade of single condition decisions, and performs simple code optimizations based on constant propagation. For calc _ week , this determines slightly different counts of static branches across the different specializations of the program.
Gcov is a test coverage program included in the Gnu development suite. Gcov can track the branches and statements covered when executing a C program, provided that this has been compiled with specific instrumentation options.
References
Ball, T. (2003). Abstraction-guided test generation: A case study. Tech. Rep. MSR-TR-2003-86, Microsoft Research.
Ball, T. (2004). A theory of predicate-complete test coverage and generation. In Proceedings of the third international symposium on formal methods for components and objects (FMCO 2004), Lecture Notes in Computer Science, Springer Berlin / Heidelberg (Vol. 3657, pp. 1–22).
Ball, T., Cook, B., Levin, V., & Rajamani, S. K. (2004). SLAM and static driver verifier: Technology transfer of formal methods inside Microsoft. In Proceedings of the 4th international conference on integrated formal methods (IFM 2004), Springer (pp. 1–20).
Baluda, M., Braione, P., Denaro, G., & Pezzè, M. (2010). Structural coverage of feasible code. In Proceedings of the fifth international workshop on automation of software test (AST 2010).
Beckman, N. E., Nori, A. V., Rajamani, S. K., Simmons, R. J., Tetali, S. D., & Thakur, A. V. (2010). Proofs from tests. IEEE Transactions on Software Engineering, 36(4), 495–508
Beyer, D., Chlipala, A. J., Henzinger, T. A., Jhala, R., & Majumdar, R. (2004). Generating tests from counterexamples. In Proceedings of the 26th international conference on software engineering (ICSE ’04), IEEE Computer Society (pp. 326–335).
Beyer, D., Henzinger, T., Jhala, R., & Majumdar, R. (2007). The software model checker BLAST. International Journal on Software Tools for Technology Transfer (STTT), 9(5–6), 505–525 doi:10.1007/s10009-007-0044-z.
Boyer, R. S., Elspas, B., Levitt, K. N. (1975). SELECT—a formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Notices 10, 234–245.
Bucur, S., Ureche, V., Zamfir, C., & Candea, G. (2011). Parallel symbolic execution for automated real-world software testing. In Proceedings of EuroSys 2011, ACM.
Burnim, J., & Sen, K. (2008). Heuristics for scalable dynamic test generation. In Proceedings of the 23rd IEEE/ACM international conference on automated software engineering (ASE 2008) (pp. 443–446).
Cadar, C., Ganesh, V., Pawlowski, P. M., Dill, D. L., & Engler, D. R. (2006). EXE: Automatically generating inputs of death. In Proceedings of the 13th ACM conference on computer and communications security (CCS ’06), (pp. 322–335). ACM, New York, NY, USA. doi:10.1145/1180405.1180445.
Cadar, C., Dunbar, D., & Engler, D. (2008). KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX symposium on operating systems design and implementation (OSDI 2008).
Callahan, J., Schneider, F., & Easterbrook, S. (1996). Automated software testing using model-checking. In: Proceedings of the 1996 SPIN workshop (SPIN 1996). Also WVU Technical Report NASA-IVV-96-022.,http://www.citeseer.ist.psu.edu/article/callahan96automated.html.
Chipounov, V., Kuznetsov, V., & Candea, G. (2011). S2E: A platform for in-vivo multi-path analysis of software systems. In Proceedings of the 16th international conference on architectural support for programming languages and operating systems (ASPLOS 2011) (pp. 265–278), ACM. doi:10.1145/1950365.1950396.
Clarke, L. A. (1976). A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, 2, 215–222.
Csallner, C., & Smaragdakis, Y. (2005). Check’n’crash: Combining static checking and testing. In Proceedings of the 27th international conference on software engineering (ICSE 2005) (pp. 422–431).
Csallner, C., Smaragdakis, Y., & Xie, T. (2008). DSD-crasher: A hybrid analysis tool for bug finding. ACM Transactions on Software Engineering and Methodology (TOSEM) 17(2), 1–37.
Do, H., Elbaum, S. G., & Rothermel, G. (2005). Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering: An International Journal 10(4), 405–435.
Ferguson, R., & Korel, B. (1996). The chaining approach for software test data generation. ACM Transactions on Software Engineering and Methodology 5, 63–86.
Frankl, P. G., & Weyuker, E. J. (1988). An applicable family of data flow testing criteria. IEEE Transactions on Software Engineering 14(10),1483–1498 doi:10.1109/32.6194.
Fraser, G., Wotawa, F., & Ammann, P. E. (2009). Testing with model checkers: A survey. Software Testing, Verification and Reliability 19(3), 215–261.
Godefroid, P., Klarlund, N., & Sen, K. (2005). DART: Directed automated random testing. In Proceedings of the ACM SIGPLAN 2005 conference on programming language design and implementation (PLDI 2005) (pp. 213–223).
Godefroid, P., Levin, M. Y., & Molnar, D. (2008). Automated whitebox fuzz testing. In Proceedings of the 16th annual network and distributed system security symposium (NDSS 2008) (pp. 151–166). doi:10.1.1.129.5914.
Gulavani, B. S., Henzinger, T. A., Kannan, Y., Nori, A. V., & Rajamani, S. K. (2006). Synergy: A new algorithm for property checking. In Proceedings of the 14th ACM SIGSOFT symposium on foundations of software engineering (FSE-14) (pp. 117–127).
King, J. C. (1976). Symbolic execution and program testing. Communications of the ACM 19(7), 385–394.
Korel, B. (1992). Dynamic method for software test data generation. Software Testing, Verification and Reliability 2(4), 203–213.
Majumdar R., & Sen, K. (2007) Hybrid concolic testing. In Proceedings of the 29th international conference on software engineering (pp. 416–426). IEEE Computer Society.
Namin, A. S., & Andrews, J. H. (2009). The influence of size and coverage on test suite effectiveness. In Proceedings of the eighteenth international symposium on software testing and analysis (issta ’09) (pp. 57–68). ACM, New York, NY, USA.
Pezzè, M., & Young, M. (2007). Software testing and analysis: process, principles and techniques. Wiley, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/047145593.
RTCA, Inc. (1993). Document RTCA/DO-178B. U.S. Department of Transportation, Federal Aviation Administration, Washington, D.C.
Sen, K., Marinov, D., & Agha, G. (2005). CUTE: A concolic unit testing engine for C. In Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering (ESEC/FSE-13) (pp. 263–272).
Staats, M., & Pǎsǎreanu, C. S. (2010). Parallel symbolic execution for structural test generation. In Proceedings of the 19th international symposium on software testing and analysis (ISSTA 2010) (pp. 183–194). ACM. doi:10.1145/1831708.1831732.
Visser, W., Pǎsǎreanu, C. S., & Khurshid, S. (2004). Test input generation with Java PathFinder. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004) (pp. 97–107). ACM.
Weyuker, E. J. (1988) The evaluation of program-based software test data adequacy criteria. Communications of the ACM, 31(6), 668–675, doi:10.1145/62959.62963.
Xie, T., Tillmann, N., de Halleux. P., & Schulte, W. (2009). Fitness-guided path exploration in dynamic symbolic execution. In Proceedings of the 39th annual IEEE/IFIP international conference on dependable systems and networks (DSN 2009) (pp. 359–368). URL http://www.csc.ncsu.edu/faculty/xie/publications/dsn09-fitnex.pd.
Yates, D. F., & Malevris, N. (1989). Reducing the effects of infeasible paths in branch testing. In Proceedings of the ACM SIGSOFT ’89 third symposium on software testing, analysis, and verification (pp. 48–54). ACM.
Acknowledgments
This work is partially supported by the European Community under the call FP7-ICT-2009-5—project PINCETTE 257647 and by the Swiss National Science Foundation AVATAR project—SNF grant nr. 200021_132666
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper is an extended version of Baluda et al. (2010).
Rights and permissions
About this article
Cite this article
Baluda, M., Braione, P., Denaro, G. et al. Enhancing structural software coverage by incrementally computing branch executability. Software Qual J 19, 725–751 (2011). https://doi.org/10.1007/s11219-011-9150-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-011-9150-y