Abstract
Malfunction of safety-critical systems may cause damage to people and the environment. Software within those systems is rigorously designed and verified according to domain specific guidance, such as ISO26262 for automotive safety. This paper describes academic and industrial co-operation in tool development to support one of the most stringent of the requirements — achieving full code coverage in requirements-driven testing. We present a verification workflow supported by a tool that integrates the coverage measurement tool RapiCover with the test-vector generator FShell. The tool assists closing the coverage gap by providing the engineer with test vectors that help in debugging coverage-related code quality issues and creating new test cases, as well as justifying the presence of unreachable parts of the code in order to finally achieve full effective coverage according to the required criteria. We illustrate the tool’s practical utility on automotive industry benchmarks. It generates 8\(\times \) more MC/DC coverage than random search.
The research leading to these results has received funding from the ARTEMIS Joint Undertaking under grant agreement number 295311 “VeTeSS”.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
RVS is licensed software. An evaluation version can be requested from http://www.rapitasystems.com. The licensing policy disallows anonymous licenses. To compensate for this, we provide a video showing the plug-in here: http://www.cprover.org/coverage-closure/rvs-fshell-demo.mp4.
- 2.
- 3.
Available from: http://forsyte.at/software/fshell.
- 4.
The code for these benchmarks was provided by the respective companies under a GPL-like license and can be downloaded here: http://www.cprover.org/coverage-closure/nfm-package.zip.
- 5.
Provided by Centro Ricerche Fiat.
- 6.
We chose length 5 because it seemed a good compromise between increasing coverage and keeping test execution times short for these case studies: on the e-Shift case study, adding 100 test vectors of length 5 increased coverage by 1.1 %; 100 test vectors of length 10 increased it by only 1.3 % while test execution times would double and only half as many test vectors could be explored.
- 7.
The msg benchmark achieved 100 loop unwindings in 3 hours, compared to 37, 6 and 58 unwindings for airbag, eshift and vtec in 8 days.
- 8.
Atego. “ARINC 653 & Virtualization Solutions Architectures and Partitioning”, Safety-Critical Tools Seminar, April 2012.
References
ISO26262 road vehicles – functional safety, Part 6: Product development at the software level, Annex B: Model-based development (2011)
Biere, A., Cimatti, A., Clarke, E., Zhu, Y.: Symbolic model checking without BDDs. In: Cleaveland, W.R. (ed.) TACAS 1999. LNCS, vol. 1579, p. 193. Springer, Heidelberg (1999)
Clarke, E., Kroening, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 168–176. Springer, Heidelberg (2004)
Dupuy, A., Leveson, N.: An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software. In: Digital Avionics Systems Conference, vol. 1, pp. 1B6/1–1B6/7 (2000)
Fraser, G., Wotawa, F., Ammann, P.: Testing with model checkers: a survey. Softw. Test., Verification Reliab. 19(3), 215–261 (2009)
Ghani, K., Clark, J.A.: Automatic test data generation for multiple condition and MCDC coverage. In: ICSEA, pp. 152–157 (2009)
Holzer, A., Schallhart, C., Tautschnig, M., Veith, H.: FShell: systematic test case generation for dynamic analysis and measurement. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 209–213. Springer, Heidelberg (2008)
Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. Trans. Software Eng. 37(5), 649–678 (2011)
Kandl, S., Kirner, R.: Error detection rate of MC/DC for a case study from the automotive domain. In: Min, S.L., Pettit, R., Puschner, P., Ungerer, T. (eds.) SEUS 2010. LNCS, vol. 6399, pp. 131–142. Springer, Heidelberg (2010)
Kroening, D., Strichman, O.: Efficient computation of recurrence diameters. In: Zuck, L.D., Attie, P.C., Cortesi, A., Mukhopadhyay, S. (eds.) VMCAI 2003. LNCS, vol. 2575, pp. 298–309. Springer, Heidelberg (2002)
Rierson, L.: Developing Safety-Critical Software: A Practical Guide for Aviation Software and DO-178C Compliance. CRC Press, Boca Raton (2013). Chapter 14.3 Potential Risks of Model-Based Development and Verification
Schrammel, P., Melham, T., Kroening, D.: Chaining test cases for reactive system testing. In: Yenigün, H., Yilmaz, C., Ulrich, A. (eds.) ICTSS 2013. LNCS, vol. 8254, pp. 133–148. Springer, Heidelberg (2013)
Sheeran, M., Singh, S., Stålmarck, G.: Checking safety properties using induction and a SAT-solver. In: Johnson, S.D., Hunt Jr., W.A. (eds.) FMCAD 2000. LNCS, vol. 1954, pp. 108–125. Springer, Heidelberg (2000)
Tallam, S., Gupta, N.: A concept analysis inspired greedy algorithm for test suite minimization. In: PASTE, pp. 35–42 (2005)
Zamli, K.Z., Al-Sewari, A.A., Hassin, M.H.M.: On test case generation satisfying the MC/DC criterion. Int. J. Adv. Soft Comput. Appl. 5(3) (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Nellis, A., Kesseli, P., Conmy, P.R., Kroening, D., Schrammel, P., Tautschnig, M. (2016). Assisted Coverage Closure. In: Rayadurgam, S., Tkachuk, O. (eds) NASA Formal Methods. NFM 2016. Lecture Notes in Computer Science(), vol 9690. Springer, Cham. https://doi.org/10.1007/978-3-319-40648-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-40648-0_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-40647-3
Online ISBN: 978-3-319-40648-0
eBook Packages: Computer ScienceComputer Science (R0)