Formal Methods in System Design

, Volume 25, Issue 2, pp 167–198

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software


  • Guillaume Brat
    • Kestrel TechnologyNASA Ames Research Center
  • Doron Drusinsky
    • Time-Rover
  • Dimitra Giannakopoulou
    • RIACSNASA Ames Research Center
  • Allen Goldberg
    • Kestrel TechnologyNASA Ames Research Center
  • Klaus Havelund
    • Kestrel TechnologyNASA Ames Research Center
  • Mike Lowry
    • NASA Ames Research Center
  • Corina Pasareanu
    • Kestrel TechnologyNASA Ames Research Center
  • Arnaud Venet
    • Kestrel TechnologyNASA Ames Research Center
  • Willem Visser
    • RIACSNASA Ames Research Center
  • Rich Washington
    • RIACSNASA Ames Research Center

DOI: 10.1023/B:FORM.0000040027.28662.a4

Cite this article as:
Brat, G., Drusinsky, D., Giannakopoulou, D. et al. Formal Methods in System Design (2004) 25: 167. doi:10.1023/B:FORM.0000040027.28662.a4


We report on a study to determine the maturity of different verification and validation technologies (V&V) applied to a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover controller. What makes this study unique is that it is the first (to the best of our knowledge) controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example, where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used, as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results cannot be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It confirmed our belief that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore, the results of the experiment inspired a novel framework for testing the next generation of the Rover.

model checking testing static analysis runtime analysis mars flight software

Copyright information

© Kluwer Academic Publishers 2004