Experimental Evaluation of Verification and Validation Tools on Martian Rover Software
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
We report on a study to determine the maturity of different verification and validation technologies (V&V) applied to a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover controller. What makes this study unique is that it is the first (to the best of our knowledge) controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example, where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used, as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results cannot be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It confirmed our belief that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore, the results of the experiment inspired a novel framework for testing the next generation of the Rover.
- J.–R. Abrial, E. Borger, and H. Langmaack, “Formal methods for industrial applications: Specifying and programming the steam boiler control,” in LNCS, Vol. 1165, Springer–Verlag, 1996.
- C. Artho, “Finding faults in multi–threaded programs,” Master's, thesis, Institute of Computer Systems, Federal Institute of Technology, Zurich/Austin, 2001.
- C. Artho, D. Drusinsky, A. Goldberg, K. Havelund, M. Lowry, C. Pasareanu, G. Roşu, and W. Visser, “Experiments with test case generation and runtime analysis,” in E. Börger, A. Gargantini, and E. Riccobene (Eds.), Abstract State Machines (ASM'03), Lecture Notes in Computer Science, Springer, 2003, pp. 87–107.
- G.S. Avrunin, J.C. Corbett, M.B. Dwyer, C.S. Pasareanu, and S.F. Siegel, “Comparing finite–state verification techniques for concurrent software,” Technical Report UM–CS–1999–069, Department of Computer Science, University of Massachusetts at Amherst, USA, 1999.
- B. Boehm and D. Port, “Defect and fault seeding in dependability benchmarking,” in Proc. of the DSN Workshop on Dependability Benchmarking, June 2002.
- A.T. Chamillard, L.A. Clarke, and G.S. Avrunin, “An empirical comparison of static concurrency analysis techniques,” TR 96–84, Department of Computer Science, University of Massachusetts, 1997.
- B.P. Collins and C.J. Nix, “The use of software engineering, including the Z notation, in the development of CICS,” Quality Assurance, Vol. 14, No. 2, pp. 103–110, 1988.
- J.C. Corbett, “Evaluating deadlock detection methods for concurrent software,” IEEE Trans. Softw. Eng., Vol. 22, No. 3, pp. 161–179, 1996.
- C. Drew and M. Hardman, Designing and Conducting Behavioral Research,Pergamon General Psychology Series, 1985.
- D. Drusinsky, “The temporal Rover and the ATG Rover,” in SPIN Model Checking and Software Verification, Vol. 1885 of LNCS, Springer, 2000, pp. 323–330.
- S. Duri, U. Buy, R. Devarapalli, and S.M. Shatz, “Application and experimental evaluation of state space reduction methods for deadlock analysis in ada,” ACM Trans. Softw. Eng. Meth., Vol. 3, No. 4, pp. 340–380, 1994.
- A. Groce and W. Visser, “Model checking Java programs using structural heuristics,” in Proceedings of the 2002 International Symposium on Software Testing and Analysis (ISSTA), ACM Press, July 2002.
- K. Havelund and G. Roşu, “Monitoring Java programs with Java PathExplorer,” in Proceedings of Runtime Verification (RV'01), Vol. 55 of Electronic Notes in Theoretical Computer Science, Elsevier Science, 2001.
- PolySpace, http://www.polyspace.com.
- S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson, “Eraser: A dynamic data race detector for multithreaded programs,” ACM Transactions on Computer Systems, Vol. 15, No. 4, pp. 391–411, 1997.
- W. Visser, K. Havelund, G. Brat, and S.–J. Park. “Model checking programs,” in Proc. of the 15th IEEE International Conference on Automated Software Engineering, Grenoble, France, Sept. 2000.
- W. Visser, K. Havelund, G. Brat, S.–J. Park, and F. Lerda, “Model checking programs,” Automated Software Engineering Journal, Vol. 10, No. 2, 2003.
- R. Washington, K. Golden, and J. Bresina, “Plan execution, monitoring, and adaptation for planetary rovers,” Electronic Transactions on Artificial Intelligence, Vol. 4, No. A, pp. 3–21, 2000. http://www.ep.liu.se/ej/etai/2000/004/.
- J.C. Widmaier, C. Smidts, and X. Huang, “Producing more reliable software: Mature software engineering process vs. state–of–the–art technology,” in Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, ACM Press, June 2000, pp. 87–94.
- Experimental Evaluation of Verification and Validation Tools on Martian Rover Software
Formal Methods in System Design
Volume 25, Issue 2-3 , pp 167-198
- Cover Date
- Print ISSN
- Online ISSN
- Kluwer Academic Publishers
- Additional Links
- model checking
- static analysis
- runtime analysis
- mars flight software
- Industry Sectors
- Author Affiliations
- 1. Kestrel Technology, NASA Ames Research Center, Moffett Field, CA, 94035, USA
- 2. Time-Rover, 11425 Charsan Ln., Cupertino, CA, 95014, USA
- 3. RIACS, NASA Ames Research Center, Moffett Field, CA, 94035, USA
- 4. Kestrel Technology, NASA Ames Research Center, Moffett Field, CA, 94035, USA
- 5. NASA Ames Research Center, Moffett Field, CA, 94035, USA
- 6. RIACS, NASA Ames Research Center, Moffett Field, CA, 94035, USA