Formal Methods in System Design

, Volume 25, Issue 2–3, pp 167–198 | Cite as

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

  • Guillaume Brat
  • Doron Drusinsky
  • Dimitra Giannakopoulou
  • Allen Goldberg
  • Klaus Havelund
  • Mike Lowry
  • Corina Pasareanu
  • Arnaud Venet
  • Willem Visser
  • Rich Washington
Article

Abstract

We report on a study to determine the maturity of different verification and validation technologies (V&V) applied to a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover controller. What makes this study unique is that it is the first (to the best of our knowledge) controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example, where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used, as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results cannot be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It confirmed our belief that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore, the results of the experiment inspired a novel framework for testing the next generation of the Rover.

model checking testing static analysis runtime analysis mars flight software 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    J.–R. Abrial, E. Borger, and H. Langmaack, “Formal methods for industrial applications: Specifying and programming the steam boiler control,” in LNCS, Vol. 1165, Springer–Verlag, 1996.Google Scholar
  2. 2.
    C. Artho, “Finding faults in multi–threaded programs,” Master's, thesis, Institute of Computer Systems, Federal Institute of Technology, Zurich/Austin, 2001.Google Scholar
  3. 3.
    C. Artho, D. Drusinsky, A. Goldberg, K. Havelund, M. Lowry, C. Pasareanu, G. Roşu, and W. Visser, “Experiments with test case generation and runtime analysis,” in E. Börger, A. Gargantini, and E. Riccobene (Eds.), Abstract State Machines (ASM'03), Lecture Notes in Computer Science, Springer, 2003, pp. 87–107.Google Scholar
  4. 4.
    G.S. Avrunin, J.C. Corbett, M.B. Dwyer, C.S. Pasareanu, and S.F. Siegel, “Comparing finite–state verification techniques for concurrent software,” Technical Report UM–CS–1999–069, Department of Computer Science, University of Massachusetts at Amherst, USA, 1999.Google Scholar
  5. 5.
    B. Boehm and D. Port, “Defect and fault seeding in dependability benchmarking,” in Proc. of the DSN Workshop on Dependability Benchmarking, June 2002.Google Scholar
  6. 6.
    A.T. Chamillard, L.A. Clarke, and G.S. Avrunin, “An empirical comparison of static concurrency analysis techniques,” TR 96–84, Department of Computer Science, University of Massachusetts, 1997.Google Scholar
  7. 7.
    B.P. Collins and C.J. Nix, “The use of software engineering, including the Z notation, in the development of CICS,” Quality Assurance, Vol. 14, No. 2, pp. 103–110, 1988.Google Scholar
  8. 8.
    J.C. Corbett, “Evaluating deadlock detection methods for concurrent software,” IEEE Trans. Softw. Eng., Vol. 22, No. 3, pp. 161–179, 1996.Google Scholar
  9. 9.
    C. Drew and M. Hardman, Designing and Conducting Behavioral Research,Pergamon General Psychology Series, 1985.Google Scholar
  10. 10.
    D. Drusinsky, “The temporal Rover and the ATG Rover,” in SPIN Model Checking and Software Verification, Vol. 1885 of LNCS, Springer, 2000, pp. 323–330.Google Scholar
  11. 11.
    S. Duri, U. Buy, R. Devarapalli, and S.M. Shatz, “Application and experimental evaluation of state space reduction methods for deadlock analysis in ada,” ACM Trans. Softw. Eng. Meth., Vol. 3, No. 4, pp. 340–380, 1994.Google Scholar
  12. 12.
    A. Groce and W. Visser, “Model checking Java programs using structural heuristics,” in Proceedings of the 2002 International Symposium on Software Testing and Analysis (ISSTA), ACM Press, July 2002.Google Scholar
  13. 13.
    K. Havelund and G. Roşu, “Monitoring Java programs with Java PathExplorer,” in Proceedings of Runtime Verification (RV'01), Vol. 55 of Electronic Notes in Theoretical Computer Science, Elsevier Science, 2001.Google Scholar
  14. 14.
    PolySpace, http://www.polyspace.com.Google Scholar
  15. 15.
    S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson, “Eraser: A dynamic data race detector for multithreaded programs,” ACM Transactions on Computer Systems, Vol. 15, No. 4, pp. 391–411, 1997.Google Scholar
  16. 16.
    W. Visser, K. Havelund, G. Brat, and S.–J. Park. “Model checking programs,” in Proc. of the 15th IEEE International Conference on Automated Software Engineering, Grenoble, France, Sept. 2000.Google Scholar
  17. 17.
    W. Visser, K. Havelund, G. Brat, S.–J. Park, and F. Lerda, “Model checking programs,” Automated Software Engineering Journal, Vol. 10, No. 2, 2003.Google Scholar
  18. 18.
    R. Washington, K. Golden, and J. Bresina, “Plan execution, monitoring, and adaptation for planetary rovers,” Electronic Transactions on Artificial Intelligence, Vol. 4, No. A, pp. 3–21, 2000. http://www.ep.liu.se/ej/etai/2000/004/.Google Scholar
  19. 19.
    J.C. Widmaier, C. Smidts, and X. Huang, “Producing more reliable software: Mature software engineering process vs. state–of–the–art technology,” in Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, ACM Press, June 2000, pp. 87–94.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Guillaume Brat
    • 1
  • Doron Drusinsky
    • 2
  • Dimitra Giannakopoulou
    • 3
  • Allen Goldberg
    • 1
  • Klaus Havelund
    • 4
  • Mike Lowry
    • 5
  • Corina Pasareanu
    • 1
  • Arnaud Venet
    • 1
  • Willem Visser
    • 6
  • Rich Washington
    • 3
  1. 1.Kestrel TechnologyNASA Ames Research CenterMoffett FieldUSA
  2. 2.Time-RoverCupertinoUSA
  3. 3.RIACSNASA Ames Research CenterMoffett FieldUSA
  4. 4.Kestrel TechnologyNASA Ames Research CenterMoffett FieldUSA
  5. 5.NASA Ames Research CenterMoffett FieldUSA
  6. 6.RIACSNASA Ames Research CenterMoffett FieldUSA

Personalised recommendations