Testing Library Specifications by Verifying Conformance Tests

  • Joseph R. Kiniry
  • Daniel M. Zimmerman
  • Ralph Hyland
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7305)


Formal specifications of standard libraries are necessary when statically verifying software that uses those libraries. Library specifications must be both correct, accurately reflecting library behavior, and useful, describing library behavior in sufficient detail to allow static verification of client programs. Specification and verification researchers regularly face the question of whether the library specifications we use are correct and useful, and we have collectively provided no good answers. Over the past few years we have created and refined a software engineering process, which we call the Formal CTD Process (FCTD ), to address this problem. Although FCTD is primarily targeted toward those who write Java libraries (or specifications for existing Java libraries) using the Java Modeling Language (JML), its techniques are broadly applicable. The key to FCTD is its novel usage of library conformance test suites. Rather than executing the conformance tests, FCTD uses them to measure the correctness and utility of specifications through static verification. FCTD is beginning to see significant use within the JML community and is the cornerstone process of the JML Spec-a-thons, meetings that bring JML researchers and practitioners together for intensive specification writing sessions. This article describes the Formal CTD Process, its use in small case studies, and its broad application to the standard Java class library.


Test Suite Unit Test System Under Test Java Modeling Language Conformance Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beust, C., Suleiman, H.: Next Generation Java Testing. Addison–Wesley Publishing Company (2007)Google Scholar
  2. 2.
    Burdy, L., Cheon, Y., Cok, D., Ernst, M., Kiniry, J., Leavens, G.T., Leino, K.R.M., Poll, E.: An overview of JML tools and applications. International Journal on Software Tools for Technology Transfer (February 2005)Google Scholar
  3. 3.
    Chalin, P., Robby, et al.: JMLEclipse: An Eclipse-based JML specification and verification environment (2011),
  4. 4.
    Cheon, Y., Leavens, G.T.: A Simple and Practical Approach to Unit Testing: The JML and JUnit Way. In: Magnusson, B. (ed.) ECOOP 2002. LNCS, vol. 2374, pp. 231–255. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  5. 5.
    Cok, D.R., et al.: OpenJML (2011),
  6. 6.
    Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., Xiao, C.: The Daikon system for dynamic detection of likely invariants. Science of Computer Programming 69(1-3), 35–45 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Flanagan, C., Leino, K.R.M.: Houdini, an Annotation Assistant for ESC/Java. In: Oliveira, J.N., Zave, P. (eds.) FME 2001. LNCS, vol. 2021, pp. 500–517. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  8. 8.
    Flanagan, C., Leino, K.R.M., Lillibridge, M., Nelson, G., Saxe, J.B., Stata, R.: Extended static checking for Java. ACM SIGPLAN Notices 37(5), 234–245 (2002)CrossRefGoogle Scholar
  9. 9.
    Free Software Foundation, Inc.: GNU Classpath (2011),
  10. 10.
    Gamma, E., Beck, K.: JUnit: A regression testing framework (2011),
  11. 11.
    Hyland, R.: A Process for the Specification of Core JDK Classes. Master’s thesis, University College Dublin (April 2010)Google Scholar
  12. 12.
    Janota, M., Grigore, R., Moskal, M.: Reachability analysis for annotated code. In: 6th International Workshop on the Specification and Verification of Component-based Systems (SAVCBS 2007), Dubrovnik, Croatia (September 2007)Google Scholar
  13. 13.
    Kiniry, J.R., Cochran, D., Tierney, P.: A verification-centric realization of e-voting. In: International Workshop on Electronic Voting Technologies (EVT 2007), Boston, Massachusetts (2007)Google Scholar
  14. 14.
    Cok, D.R., Kiniry, J.R.: ESC/Java2: Uniting eSC/Java and JML. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, pp. 108–128. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Kiniry, J.R., Zimmerman, D.M.: Secret Ninja Formal Methods. In: Cuellar, J., Sere, K. (eds.) FM 2008. LNCS, vol. 5014, pp. 214–228. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Leavens, G.T., Cheon, Y., Clifton, C., Ruby, C., Cok, D.R.: How the Design of JML Accommodates Both Runtime Assertion Checking and Formal Verification. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2002. LNCS, vol. 2852, pp. 262–284. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  17. 17.
    Meyer, B.: Object-Oriented Software Construction, 2nd edn. Prentice-Hall, Inc. (1988)Google Scholar
  18. 18.
    Oracle Corporation: JT Harness project (2011),
  19. 19.
    Oracle Corporation: OpenJDK (2011),
  20. 20.
    Oracle Corporation: OpenJDK community TCK license agreement (2011),
  21. 21.
    The Apache Software Foundation: Apache Harmony - Open Source Java SE (2011),
  22. 22.
    The Eclipse Foundation: The Eclipse project (2011),
  23. 23.
    Zhu, H., Hall, P.A.V., May, J.H.R.: Software unit test coverage and adequacy. ACM Computing Surveys 29(4), 366–427 (1997)CrossRefGoogle Scholar
  24. 24.
    Zimmerman, D.M., Nagmoti, R.: JMLUnit: The Next Generation. In: Beckert, B., Marché, C. (eds.) FoVeOOS 2010. LNCS, vol. 6528, pp. 183–197. Springer, Heidelberg (2011)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Joseph R. Kiniry
    • 1
  • Daniel M. Zimmerman
    • 2
  • Ralph Hyland
    • 3
  1. 1.IT University of CopenhagenDenmark
  2. 2.University of Washington TacomaUSA
  3. 3.University College DublinIreland

Personalised recommendations