Efficient and Trustworthy Tool Qualification for Model-Based Testing Tools

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7641)


The application of test automation tools in a safety-critical context requires so-called tool qualification according to the applicable standards. The objective of this qualification is to justify that verification steps automated by the tool will not lead to faulty systems under test to be accepted as fit for purpose. In this paper we review the tool qualification requirements of the standards ISO 26262 (automotive domain) and the new RTCA DO-178C (avionic domain) and propose a general approach on how to qualify model-based testing tools according to these standards in an efficient and at the same time reliable way. Our approach relies on a lightweight error detection mechanism based on the idea of replaying test executions against the model. We further show how the error detection capabilities can be integrated into a convincing argument for tool qualification, going through the necessary verification activities step-by-step. We highlight the key steps for the RT-Tester Model-Based Test Generator, which is used in test campaigns in the automotive, railway and avionic domains. The approach avoids having to qualify several complex components present in model-based testing tools, such as code generators for test procedures and constraint solving algorithms for test data elaboration.


  1. 1.
    Blackburn, M.R., Busser, R.D.: T-VEC: A Tool for Developing Critical Systems. In: Compass, pp. 237–249. IEEE Computer Society Press (1996)Google Scholar
  2. 2.
    Carver, R.H., Tai, K.C.: Replay and Testing for Concurrent Programs. IEEE Software 8(2), 66–74 (1991)CrossRefGoogle Scholar
  3. 3.
    Cousot, P., Cousot, R.: Abstract Interpretation: A Unified Lattice model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In: POPL, pp. 238–252. ACM Press (1977)Google Scholar
  4. 4.
    França, R.B., Favre-Felix, D., Leroy, X., Pantel, M., Souyris, J.: Towards Formally Verified Optimizing Compilation in Flight Control Software. In: PPES. OASICS, vol. 18, pp. 5–9–68. Schloss Dagstuhl (2011)Google Scholar
  5. 5.
    Hillebrand, J., Reichenpfader, P., Mandic, I., Siegl, H., Peer, C.: Establishing Confidence in the Usage of Software Tools in Context of ISO 26262. In: Flammini, F., Bologna, S., Vittorini, V. (eds.) SAFECOMP 2011. LNCS, vol. 6894, pp. 257–269. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    International Organization for Standardization. ISO 26262 - Road Vehicles - Functional Safety - Part 8: Supporting Processes. ICS 43.040.10 (2009)Google Scholar
  7. 7.
    Klein, G., Andronick, J., Elphinstone, K., Heiser, G., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., Sewell, T., Tuch, H., Winwood, S.: seL4: Formal Verification of an Operating-System Kernel. Commun. ACM 53(6), 107–115 (2010)Google Scholar
  8. 8.
    Kroening, D., Strichman, O.: Decision Procedures. Springer (2008)Google Scholar
  9. 9.
    Leroy, X.: Formal Verification of a Realistic Compiler. Commun. ACM 52(7), 107–115 (2009)zbMATHGoogle Scholar
  10. 10.
    Löding, H., Peleska, J.: Timed Moore Automata: Test Data Generation and Model Checking. In: ICST, pp. 449–458. IEEE Computer Society (2010)Google Scholar
  11. 11.
    Myreen, M.O.: Verified Just-in-Time Compiler on x86. In: POPL, pp. 107–118. ACM (2010)Google Scholar
  12. 12.
    Peleska, J., Vorobev, E., Lapschies, F.: Automated Test Case Generation with SMT-Solving and Abstract Interpretation. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 298–312. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    Randimbivololona, F., Souyris, J., Baudin, P., Pacalet, A., Raguideau, J., Schoen, D.: Applying Formal Proof Techniques to Avionics Software: A Pragmatic Approach. In: Wing, J.M., Woodcock, J., Davies, J. (eds.) FM 1999. LNCS, vol. 1709, pp. 1798–1815. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  14. 14.
    Regehr, J.: The Future of Compiler Correctness (2010),
  15. 15.
    RTCA SC-167/EUROCAE WG-12. Software Considerations in Airborne Systems and Equipment Certification. Number RTCA/DO-178B. RTCA, Inc., 1140 Connecticut Avenue, N.W., Suite 1020, Washington, D.C. 20036 (December 1992)Google Scholar
  16. 16.
    RTCA SC-205/EUROCAE WG-71. Software Considerations in Airborne Systems and Equipment Certification. Number RTCA/DO-178C. RTCA, Inc., 1140 Connecticut Avenue, N.W., Suite 1020, Washington, D.C. 20036 (December 2011)Google Scholar
  17. 17.
    RTCA SC-205/EUROCAE WG-71. Software Tool Qualification Considerations. Number RTCA/DO-330. RTCA, Inc. (December 2011)Google Scholar
  18. 18.
    Souyris, J., Wiels, V., Delmas, D., Delseny, H.: Formal Verification of Avionics Software Products. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 532–546. Springer, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  1. 1.Verified Systems International GmbHBremenGermany
  2. 2.Department of Mathematics and Computer ScienceUniversity of BremenGermany

Personalised recommendations