Coverage metrics for formal verification

Special section on Recent Advances in Hardware Verification

Abstract

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is and whether it really covers all the behaviors of the system. The challenge of making the verification process as exhaustive as possible is even more crucial in simulation-based verification, where the infeasible task of checking all input sequences is replaced by checking a test suite consisting of a finite subset of them. It is very important to measure the exhaustiveness of the test suite, and indeed there has been extensive research in the simulation-based verification community on coverage metrics, which provide such a measure. It turns out that no single measure can be absolute, leading to the development of numerous coverage metrics whose usage is determined by industrial verification methodologies. On the other hand, prior research of coverage in formal verification has focused solely on state-based coverage. In this paper we adapt the work done on coverage in simulation-based verification to the formal-verification setting in order to obtain new coverage metrics. Thus, for each of the metrics used in simulation-based verification, we present a corresponding metric that is suitable for the setting of formal verification and describe an algorithmic way to check it.

Keywords

Formal verification Model checking Coverage metrics Algorithms 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Armoni, R., Fix, L., Flaisher, A., Grumberg, O., Piterman, N., Vardi, M.Y.: Enhanced vacuity detection for linear temporal logic. In: Proceedings of the 15th International Conference on Computer-Aided Verification. Berlin, Heidelberg, New York: Springer 2003Google Scholar
  2. 2.
    Armoni, R., Fix, L., Gerth, R., Ginsburg, B., Kanza, T., Landver, A., Mador-Haim, S., Tiemeyer, A., Singerman, E., Vardi, M.Y., Zbar, Y.: The ForSpec temporal language: a new temporal property-specification language. In: Procedings of the 8th Internationl Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’02). Lecture Notes in Computer Science, vol. 2280, pp. 296–311. Berlin, Heidelberg, New York: Springer 2002Google Scholar
  3. 3.
    Beatty, D., Bryant, R.: Formally verifying a microprocessor using a simulation methodology. In: Proceedings of the 31st Design Automation Conference, pp. 596–602. IEEE Press, New York 1994Google Scholar
  4. 4.
    Beer, I., Ben-David, S., Eisner, C., Rodeh, Y.: Efficient detection of vacuity in ACTL formulas. Formal Methods Syst. Des. 18(2), 141–162 (2001)CrossRefGoogle Scholar
  5. 5.
    Bening, L., Foster, H.: Principles of Verifiable RTL Design – a Functional Coding Style Supporting Verification Processes. Kluwer, Dordrecht 2000MATHGoogle Scholar
  6. 6.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on java predicates. In: Proceedings of ACM/SIGSOFT International Symposium on Software Testing and Analysys (ISSTA), pp. 123–133. Rome, Italy (2002)Google Scholar
  7. 7.
    Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. Inf. Comput. 98(2), 142–170(1992)CrossRefMathSciNetGoogle Scholar
  8. 8.
    Büchi, J.R.: On a decision method in restricted second order arithmetic. In: Proceedings of the International Congress on Logic, Methods and Philosophical Science 1960, pp. 1–12. Stanford: Stanford University Press 1962Google Scholar
  9. 9.
    Budd, T.A.: Mutation analysis: ideas, examples, problems, and prospects. In: Proceedings of the Conference on Software Testing, North Holland (1981)Google Scholar
  10. 10.
    Budd, T.A., Angluin, D.: Two notions of correctness and their relation to testing. Acta Inf 18, 31–45 (1982)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Cadence. Assertion-based verification. http://www.cadence.com (2003)
  12. 12.
    Chockler, H.: Coverage Metrics for Model Checking. Ph.D. thesis, Hebrew University, Jerusalem, Israel (2003)Google Scholar
  13. 13.
    Chockler, H., Kupferman, O.: Coverage of implementations by simulating specifications. In: Baeza-Yates, R.A., Montanari, U., Santoro, N. (eds.): Proceedings of the 2nd IFIP International Conference on Theoretical Computer Science, Proceedings of the IFIP Conference, Montreal, vol. 223, pp. 409–421. Kluwer, Dordrecht 2002Google Scholar
  14. 14.
    Chockler, H., Kupferman, O., Kurshan, R.P., Vardi, M.Y.: A practical approach to coverage in model checking. In: Proceedings of the 13th International Conference on Computer-Aided Verification. Lecture Notes in Computer Science, vol. 2102, pp. 66–78. Berlin, Heidelberg, New York: Springer 2001Google Scholar
  15. 15.
    Chockler, H., Kupferman, O., Vardi, M.Y.: Coverage metrics for temporal logic model checking. In: Tools and Algorithms for the Construction and Analysis of Systems. Lecture Notes in Computer Science, vol. 2031, pp. 528–542. Berlin, Heidelberg, New York: Springer 2001Google Scholar
  16. 16.
    Clarke, E.M., Grumberg, O., McMillan, K.L., Zhao, X.: Efficient generation of counterexamples and witnesses in symbolic model checking. In: Proceedings of the 32nd Conference on Design Automation, pp. 427–432. IEEE Press, New York 1995CrossRefGoogle Scholar
  17. 17.
    Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge, MA 1999Google Scholar
  18. 18.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–43 (1978)Google Scholar
  19. 19.
    Dill, D.L.: What’s between simulation and formal verification? In: Proceedings of the 35st Conference on Design Automation, pp. 328–329. IEEE Press, New York 1998Google Scholar
  20. 20.
    French, R.S., Lam, M.S., Levitt, J.R., Olukotun, K.: A general method for compiling event-driven simulations. In: Proceedings of the Conference on Design Automation, pp. 151–156 (1995)Google Scholar
  21. 21.
    Grumberg, O., Long, D.E.: Model checking and modular verification. ACM Trans. Programm. Lang. Syst. 16(3), 843–871 (1994)CrossRefGoogle Scholar
  22. 22.
    Hardin, R.H., Har’el, Z., Kurshan, R.P.: COSPAN. In: Proceedings of the 8th International Conference on Computer-Aided Verification. Lecture Notes in Computer Science, vol. 1102, pp. 423–427. Berlin, Heidelberg, New York: Springer 1996Google Scholar
  23. 23.
    Ho, R.C., Horowitz, M.A.: Validation coverage analysis for complex digital designs. In: Proceedings of the International Conference on Computer-Aided Design, pp. 146–151 (1996)Google Scholar
  24. 24.
    Hoskote, Y.V.: Formal techniques for verification of synchronous sequential circuits. Ph.D. thesis, University of Texas at Austin (1995)Google Scholar
  25. 25.
    Holzmann, G.J.: The model checker SPIN. IEEE Trans. Softw. Eng. 23(5), 279–295 (1997)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Hoskote, Y., Kam, T., Ho, P.-H., Zhao, X.: Coverage estimation for symbolic model checking. In: Proceedings of the 36th Conference on Design Automation, pp. 300–305 (1999)Google Scholar
  27. 27.
    Kantrowitz, M., Noack, L.: I’m done simulating: now what? verification coverage analysis and correctness checking of the dec chip 21164 alpha microprocessor. In: Proceedings of the Conference on Design Automation, pp. 325–330 (1996)Google Scholar
  28. 28.
    Katz, S.: Techniques for increasing coverage of formal verification. M.Sc. thesis, The Technion, Israel (2001)Google Scholar
  29. 29.
    Katz, S., Geist, D., Grumberg, O.: “Have I written enough properties ?” a method of comparison between specification and implementation. In: Proceedings of the 10th Advanced Research Working Conference on Correct Hardware Design and Verification Methods. Lecture Notes in Computer Science, vol. 1703, pp. 280–297. Berlin, Heidelberg, New York: Springer 1999Google Scholar
  30. 30.
    Kupferman, O., Vardim, M.Y.: Vacuity detection in temporal model checking. J. Softw. Tools Technol. Transfer 4(2), 224–233 (2003)CrossRefGoogle Scholar
  31. 31.
    Kurshan, R.P.: Formal verification in a commercial setting. In: Proceedings of the Conference on Design Automation, vol. 34, pp. 258–262 (1997)Google Scholar
  32. 32.
    Kurshan, R.P.: FormalCheck User’s Manual. San Jose: Cadence Design 1998Google Scholar
  33. 33.
    Manna, Z., Pnueli, A: The Temporal Logic of Reactive and Concurrent Systems: Specification. Berlin, Heidelberg, New York: Springer 1992Google Scholar
  34. 34.
    Marick, B.: How to misuse code coverage. In: Proceedings of the 16th International Conference on Testing Computer Software (1999)Google Scholar
  35. 35.
    Peled, D.: Software Reliability Methods. Berlin, Heidelberg, New York: Springer 2001MATHGoogle Scholar
  36. 36.
    Purandare, M., Somenzi, F.: Vacuum cleaning CTL formulae. In: Proceedings of the 14th Conference on Computer-Aided Verification. Lecture Notes in Computer Science, vol. 2404, pp. 485–499. Berlin, Heidelberg, New York: Springer 2002Google Scholar
  37. 37.
    Shen, J., Abraham, J.A.: An rtl abstraction technique for processor microarchitecture validation and test generation. J. Electron. Test. 16(1–2), 67–81 (1999)Google Scholar
  38. 38.
    Tasiran, S., Keutzer, K.: Coverage metrics for functional validation of hardware designs. IEEE Des. Test Comput. 18(4), 36–45 (2001)CrossRefGoogle Scholar
  39. 39.
    Ur, S., Ziv., A.: Off-the-shelf vs. custom made coverage models, which is the one for you? In: Proceedings of the International Conference on Software Testing, Analysis, and Review (1998)Google Scholar
  40. 40.
    Vardi, M.Y., Wolper, P.: An automata-theoretic approach to automatic program verification. In: Proceedings of the 1st Symposium on Logic in Computer Science, pp. 332–344. Cambridge, MA (1986)Google Scholar
  41. 41.
    Vardi, M.Y., Wolper, P.: Reasoning about infinite computations. Inf. Comput. 115(1), 1–37 (1994)CrossRefMathSciNetGoogle Scholar
  42. 42.
    Verisity. Surecove’s code coverage technology. http://www.verisity.com/products/surecov.html (2003)
  43. 43.
    Zhu, H., Hall, P.V., May, J.R.: Software unit test coverage and adequacy. ACM Comput. Surv. 29(4), 366–427 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  1. 1.IBM Research LabsMount CarmelHaifaIsrael
  2. 2.School of Engineering and Computer ScienceHebrew UniversityJerusalemIsrael
  3. 3.Department of Computer ScienceRice UniversityHoustonUSA

Personalised recommendations