Sanity Checks in Formal Verification
One of the advantages of temporal-logic model-checking tools is their ability to accompany a negative answer to the correctness query by a counterexample to the satisfaction of the specification in the system. On the other hand, when the answer to the correctness query is positive, most model-checking tools provide no additional information. In the last few years there has been growing awareness to the importance of suspecting the system or the specification of containing an error also in the case model checking succeeds. The main justification of such suspects are possible errors in the modeling of the system or of the specification. The goal of sanity checks is to detect such errors by further automatic reasoning. Two leading sanity checks are vacuity and coverage. In vacuity, the goal is to detect cases where the system satisfies the specification in some unintended trivial way. In coverage, the goal is to increase the exhaustiveness of the specification by detecting components of the system that do not play a role in verification process. For both checks, the challenge is to define vacuity and coverage formally, develop algorithms for detecting vacuous satisfaction and low coverage, and suggest methods for returning to the user helpful information. We survey existing work on vacuity and coverage and argue that, in many aspects, the two checks are essentially the same: both are based on repeating the verification process on some mutant input. In vacuity, mutations are in the specifications, whereas in coverage, mutations are in the system. This observation enables us to adopt work done in the context of vacuity to coverage, and vise versa.
KeywordsModel Check Temporal Logic Linear Temporal Logic Atomic Proposition Kripke Structure
Unable to display preview. Download preview PDF.
- [AFF+02]Armoni, R., Fix, L., Flaisher, A., Gerth, R., Ginsburg, B., Kanza, T., Landver, A., Mador-Haim, S., Singerman, E., Tiemeyer, A., Vardi, M.Y., Zbar, Y.: The ForSpec temporal logic: A new temporal property-specification logic. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 296–296. Springer, Heidelberg (2002)CrossRefGoogle Scholar
- [BB94]Beatty, D., Bryant, R.: Formally verifying a microprocessor using a simulation methodology. In: Proc. 31st Design Automation Conference, pp. 596–602. IEEE Computer Society Press, Los Alamitos (1994)Google Scholar
- [BCC+99]Biere, A., Cimatti, A., Clarke, E.M., Fujita, M., Zhu, Y.: Symbolic model checking using SAT procedures instead of BDDs. In: Proc. 36th Design Automation Conference, pp. 317–320. IEEE Computer Society Press, Los Alamitos (1999)Google Scholar
- [CGL93]Clarke, E.M., Grumberg, O., Long, D.: Verification tools for finite-state concurrent systems. In: de Bakker, J.W., de Roever, W.-P., Rozenberg, G. (eds.) REX 1993. LNCS, vol. 803, pp. 124–175. Springer, Heidelberg (1994)Google Scholar
- [Cho03]Chockler, H.: Coverage metrics for model checking. PhD thesis, Hebrew University, Jerusalem, Israel (2003)Google Scholar
- [CK02]Chockler, H., Kupferman, O.: Coverage of implementations by simulating specifications. In: Baeza-Yates, R.A., Montanari, U., Santoro, N. (eds.) Proceedings of 2nd IFIP International Conference on Theoretical Computer Science, Montreal, Canada, August 2002. IFIP Conference Proceedings, vol. 223, pp. 409–421. Kluwer Academic Publishers, Dordrecht (2002)Google Scholar
- [Dil98]Dill, D.L.: What’s between simulation and formal verification? In: Proc. 35st Design Automation Conference, pp. 328–329. IEEE Computer Society Press, Los Alamitos (1998)Google Scholar
- [HHK96]Hardin, R.H., Har’el, Z., Kurshan, R.P.: COSPAN. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 423–427. Springer, Heidelberg (1996)Google Scholar
- [HKHZ99]Hoskote, Y., Kam, T., Ho, P.-H., Zhao, X.: Coverage estimation for symbolic model checking. In: Proc. 36th Design automation conference, pp. 300–305 (1999)Google Scholar
- [Kur98]Kurshan, R.P.: FormalCheck User’s Manual. Cadence Design, Inc. (1998)Google Scholar
- [Ver03]Verisity. Surecove’s code coverage technology (2003), http://www.verisity.com/products/surecov.html