Sanity Checks in Formal Verification

  • Orna Kupferman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4137)


One of the advantages of temporal-logic model-checking tools is their ability to accompany a negative answer to the correctness query by a counterexample to the satisfaction of the specification in the system. On the other hand, when the answer to the correctness query is positive, most model-checking tools provide no additional information. In the last few years there has been growing awareness to the importance of suspecting the system or the specification of containing an error also in the case model checking succeeds. The main justification of such suspects are possible errors in the modeling of the system or of the specification. The goal of sanity checks is to detect such errors by further automatic reasoning. Two leading sanity checks are vacuity and coverage. In vacuity, the goal is to detect cases where the system satisfies the specification in some unintended trivial way. In coverage, the goal is to increase the exhaustiveness of the specification by detecting components of the system that do not play a role in verification process. For both checks, the challenge is to define vacuity and coverage formally, develop algorithms for detecting vacuous satisfaction and low coverage, and suggest methods for returning to the user helpful information. We survey existing work on vacuity and coverage and argue that, in many aspects, the two checks are essentially the same: both are based on repeating the verification process on some mutant input. In vacuity, mutations are in the specifications, whereas in coverage, mutations are in the system. This observation enables us to adopt work done in the context of vacuity to coverage, and vise versa.


Model Check Temporal Logic Linear Temporal Logic Atomic Proposition Kripke Structure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [AFF+02]
    Armoni, R., Fix, L., Flaisher, A., Gerth, R., Ginsburg, B., Kanza, T., Landver, A., Mador-Haim, S., Singerman, E., Tiemeyer, A., Vardi, M.Y., Zbar, Y.: The ForSpec temporal logic: A new temporal property-specification logic. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 296–296. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  2. [AFF+03]
    Armoni, R., Fix, L., Flaisher, A., Grumberg, O., Piterman, N., Tiemeyer, A., Vardi, M., Y.: Enhanced vacuity detection in linear temporal logic. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 368–380. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. [BB94]
    Beatty, D., Bryant, R.: Formally verifying a microprocessor using a simulation methodology. In: Proc. 31st Design Automation Conference, pp. 596–602. IEEE Computer Society Press, Los Alamitos (1994)Google Scholar
  4. [BBE+01]
    Beer, I., Ben-David, S., Eisner, C., Fisman, D., Gringauze, A., Rodeh, Y.: The temporal logic Sugar. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 363–367. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  5. [BBER01]
    Beer, I., Ben-David, S., Eisner, C., Rodeh, Y.: Efficient detection of vacuity in ACTL formulas. Formal Methods in System Design 18(2), 141–162 (2001)CrossRefzbMATHGoogle Scholar
  6. [BCC+99]
    Biere, A., Cimatti, A., Clarke, E.M., Fujita, M., Zhu, Y.: Symbolic model checking using SAT procedures instead of BDDs. In: Proc. 36th Design Automation Conference, pp. 317–320. IEEE Computer Society Press, Los Alamitos (1999)Google Scholar
  7. [BCM+92]
    Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. Information and Computation 98(2), 142–170 (1992)CrossRefMathSciNetzbMATHGoogle Scholar
  8. [BF00]
    Bening, L., Foster, H.: Principles of verifiable RTL design – a functional coding style supporting verification processes. Kluwer Academic Publishers, Dordrecht (2000)zbMATHGoogle Scholar
  9. [BFG+05]
    Bustan, D., Flaisher, A., Grumberg, O., Kupferman, O., Vardi, M.Y.: Regular Vacuity. In: Borrione, D., Paul, W. (eds.) CHARME 2005. LNCS, vol. 3725, pp. 191–206. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. [BG01]
    Bruns, G., Godefroid, P.: Temporal logic query checking. In: Proceedings of the 16th Annual IEEE Symposium on Logic in Computer Science (LICS 2001), June 16–19, 2001, pp. 409–420. IEEE Computer Society, Los Alamitos (2001)CrossRefGoogle Scholar
  11. [BG04]
    Godefroid, P., Bruns, G.: Model Checking with Multi-valued Logics. In: Díaz, J., Karhumäki, J., Lepistö, A., Sannella, D. (eds.) ICALP 2004. LNCS, vol. 3142, pp. 281–293. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  12. [CG04a]
    Chechik, M., Gurfinkel, A.: Extending extended vacuity. In: Hu, A.J., Martin, A.K. (eds.) FMCAD 2004. LNCS, vol. 3312, pp. 306–321. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. [CG04b]
    Chechik, M., Gurfinkel, A.: How Vacuous Is Vacuous? In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 451–466. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  14. [CGL93]
    Clarke, E.M., Grumberg, O., Long, D.: Verification tools for finite-state concurrent systems. In: de Bakker, J.W., de Roever, W.-P., Rozenberg, G. (eds.) REX 1993. LNCS, vol. 803, pp. 124–175. Springer, Heidelberg (1994)Google Scholar
  15. [CGMZ95]
    Clarke, E.M., Grumberg, O., McMillan, K.L., Zhao, X.: Efficient generation of counterexamples and witnesses in symbolic model checking. In: Proc. 32nd Design Automation Conference, pp. 427–432. IEEE Computer Society Press, Los Alamitos (1995)CrossRefGoogle Scholar
  16. [Cha00]
    Chan, W.: Temporal-logic queries. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 450–463. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  17. [Cho03]
    Chockler, H.: Coverage metrics for model checking. PhD thesis, Hebrew University, Jerusalem, Israel (2003)Google Scholar
  18. [CK02]
    Chockler, H., Kupferman, O.: Coverage of implementations by simulating specifications. In: Baeza-Yates, R.A., Montanari, U., Santoro, N. (eds.) Proceedings of 2nd IFIP International Conference on Theoretical Computer Science, Montreal, Canada, August 2002. IFIP Conference Proceedings, vol. 223, pp. 409–421. Kluwer Academic Publishers, Dordrecht (2002)Google Scholar
  19. [CKKV01]
    Chockler, H., Kupferman, O., Kurshan, R.P., Vardi, M.Y.: A practical approach to coverage in model checking. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 66–78. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  20. [CKV01]
    Chockler, H., Kupferman, O., Vardi, M.Y.: Coverage metrics for temporal logic model checking. In: Margaria, T., Yi, W. (eds.) TACAS 2001. LNCS, vol. 2031, pp. 528–542. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  21. [CKV03]
    Vardi, M.Y., Kupferman, O., Chockler, H.: Coverage Metrics for Formal Verification. In: Geist, D., Tronci, E. (eds.) CHARME 2003. LNCS, vol. 2860, pp. 111–125. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  22. [Dil98]
    Dill, D.L.: What’s between simulation and formal verification? In: Proc. 35st Design Automation Conference, pp. 328–329. IEEE Computer Society Press, Los Alamitos (1998)Google Scholar
  23. [HHK96]
    Hardin, R.H., Har’el, Z., Kurshan, R.P.: COSPAN. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 423–427. Springer, Heidelberg (1996)Google Scholar
  24. [HKHZ99]
    Hoskote, Y., Kam, T., Ho, P.-H., Zhao, X.: Coverage estimation for symbolic model checking. In: Proc. 36th Design automation conference, pp. 300–305 (1999)Google Scholar
  25. [KGG99]
    Katz, S., Grumberg, O., Geist, D.: Have I Written Enough Properties? - a method of comparison between specification and implementation. In: Pierre, L., Kropf, T. (eds.) CHARME 1999. LNCS, vol. 1703, pp. 280–297. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  26. [Kur98]
    Kurshan, R.P.: FormalCheck User’s Manual. Cadence Design, Inc. (1998)Google Scholar
  27. [KV03]
    Kupferman, O., Vardi, M.Y.: Vacuity detection in temporal model checking. Journal on Software Tools For Technology Transfer 4(2), 224–233 (2003)CrossRefGoogle Scholar
  28. [Nam01]
    Namjoshi, K.S.: Certifying Model Checkers. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 2–13. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  29. [Nam04]
    Namjoshi, K.S.: An efficiently checkable, proof-based formulation of vacuity in model checking. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 57–69. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  30. [Pel01]
    Peled, D.: Software Reliability Methods. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  31. [PS02]
    Purandare, M., Somenzi, F.: Vacuum cleaning CTL formulae. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 485–499. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  32. [TK01]
    Tasiran, S., Keutzer, K.: Coverage metrics for functional validation of hardware designs. IEEE Design and Test of Computers 18(4), 36–45 (2001)CrossRefGoogle Scholar
  33. [Ver03]
    Verisity. Surecove’s code coverage technology (2003),
  34. [VW94]
    Vardi, M.Y., Wolper, P.: Reasoning about infinite computations. Information and Computation 115(1), 1–37 (1994)CrossRefMathSciNetzbMATHGoogle Scholar
  35. [ZHM97]
    Zhu, H., Hall, P.V., May, J.R.: Software unit test coverage and adequacy. ACM Computing Surveys 29(4), 366–427 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Orna Kupferman
    • 1
  1. 1.School of Engineering and Computer ScienceHebrew UniversityJerusalemIsrael

Personalised recommendations