Advertisement

Abstract

Recent years have seen increasing success in building large formal proof developments using interactive theorem provers (ITPs). Some proofs have involved many authors, years of effort, and resulted in large, complex interdependent sets of proof “source code” files. Developing these in the first place, and maintaining and extending them afterwards, is a considerable challenge. It has prompted the idea of Proof Engineering as a new sub-field, to find methods and tools to help. It is natural to try to borrow ideas from Software Engineering for this.

In this paper we investigate the idea of defining proof metrics by analogy with software metrics. We seek metrics that may help to monitor and compare formal proof developments, which might be used to guide good practice, locate likely problem areas, or suggest refactorings. Starting from metrics that have been proposed for object-oriented design, we define analogues for formal proofs. We show that our metrics enjoy reasonable properties, and we demonstrate their behaviour with some practical experiments, showing changes over time as proof developments evolve, and making comparisons across between different ITPs.

Keywords

Formal Proof Theorem Statement Software Metrics Proof Check Proof Script 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

We’re grateful to colleagues Iain Whiteside, Ajitha Rajan and the DReaM group at Edinburgh for discussions. The referees provided useful remarks. We acknowledge financial support from grants from UK EPSRC (EP/J001058/1) and the Austrian Science Fund (P26201). For tools and data, please visit http://homepages.inf.ed.ac.uk/da/proofmetrics/.

References

  1. 1.
    Alama, J., Mamane, L., Urban, J.: Dependencies in formal mathematics: applications and extraction for coq and mizar. In: Jeuring, J., Campbell, J.A., Carette, J., Dos Reis, G., Sojka, P., Wenzel, M., Sorge, V. (eds.) CICM 2012. LNCS, vol. 7362, pp. 1–16. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  2. 2.
    Al-Dallal, J., Briand, L.C.: A precise method-method interaction-based cohesion metric for object-oriented classes. ACM Trans. Softw. Eng. Methodol. 21(2), 8:1–8:34 (2012)CrossRefGoogle Scholar
  3. 3.
    Bancerek, G., Rudnicki, P.: A compendium of continuous lattices in MIZAR. J. Autom. Reasoning 29(3–4), 189–224 (2002)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Basili, V.R., Briand, L.C., Melo, W.L.: A validation of object-oriented design metrics as quality indicators. IEEE Trans. Softw. Eng. 22(10), 751–761 (1996)CrossRefGoogle Scholar
  5. 5.
    Blanchette, J.C., Haslbeck, M., Matichuk, D., Nipkow, T.: Mining the archive of formal proofs. In: Kerber, M., Carette, J., Kaliszyk, C., Rabe, F., Sorge, V. (eds.) CICM 2015. LNCS, vol. 9150, pp. 3–17. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  6. 6.
    Bourke, T., Daum, M., Klein, G., Kolanski, R.: Challenges and experiences in managing large-scale proofs. In: Jeuring, J., Campbell, J.A., Carette, J., Dos Reis, G., Sojka, P., Wenzel, M., Sorge, V. (eds.) CICM 2012. LNCS, vol. 7362, pp. 32–48. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  7. 7.
    Chidamber, S.R., Kemerer, C.F.: A metrics suite for object oriented design. IEEE Trans. Softw. Eng. 20(6), 476–493 (1994)CrossRefGoogle Scholar
  8. 8.
    Demeyer, S., Ducasse, S.: Metrics, do they really help? In: Malenfant, J. (ed.) Proceedings LMO 1999 (Languages et Models a Objets), pp. 69–82 (1999)Google Scholar
  9. 9.
    Demeyer, S., Ducasse, S., Nierstrasz, O.: Finding refactorings via change metrics. In: Object-Oriented Programming Systems, Languages & Applications, OOPSLA 2000, pp. 166–177 (2000)Google Scholar
  10. 10.
    Gonthier, G., Mathematics, E.: The odd order theorem proof. In: Principles of Programming Languages, POPL 2013, pp. 1–2. ACM (2013)Google Scholar
  11. 11.
    Gonthier, G., Asperti, A., Avigad, J., Bertot, Y., Cohen, C., Garillot, F., Le Roux, S., Mahboubi, A., O’Connor, R., Ould Biha, S., Pasca, I., Rideau, L., Solovyev, A., Tassi, E., Théry, L.: A machine-checked proof of the odd order theorem. In: Blazy, S., Paulin-Mohring, C., Pichardie, D. (eds.) ITP 2013. LNCS, vol. 7998, pp. 163–179. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  12. 12.
    Hales, T.C.: The jordan curve theorem, formally and informally. Am. Math. Mon. 114(10), 882–894 (2007)MathSciNetMATHGoogle Scholar
  13. 13.
    Hales, T.C., et al.: A formal proof of the Kepler conjecture. In: CoRR abs/1501.02155 (2015)Google Scholar
  14. 14.
    Hales, T.C., et al.: A revision of the proof of the kepler conjecture. Discrete Comput. Geom. 44(1), 1–34 (2010)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Harrison, J.: The HOL light theory of euclidean space. J. Autom. Reasoning 50(2), 173–190 (2013)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Hölzl, J., Heller, A.: Three chapters of measure theory in Isabelle/HOL. In: van Eekelen, M., Geuvers, H., Schmaltz, J., Wiedijk, F. (eds.) ITP 2011. LNCS, vol. 6898, pp. 135–151. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Jeffery, R.D., et al.: An empirical research agenda for understanding formal methods productivity. Inf. Softw. Technol. 60, 102–112 (2015)CrossRefGoogle Scholar
  18. 18.
    Kaliszyk, C., Urban, J.: Learning-assisted automated reasoning with flyspeck. J. Autom. Reasoning 53(2), 173–213 (2014)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Klein, G.: Proof engineering considered essential. In: Jones, C., Pihlajasaari, P., Sun, J. (eds.) FM 2014. LNCS, vol. 8442, pp. 16–21. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  20. 20.
    Klein, G., et al.: seL4: Formal verification of an OS kernel. In: Symposium on Operating Systems Principles SOSP, pp. 207–220. ACM (2009)Google Scholar
  21. 21.
    Kühlwein, D., Blanchette, J.C., Kaliszyk, C., Urban, J.: MaSh: Machine learning for sledgehammer. In: Blazy, S., Paulin-Mohring, C., Pichardie, D. (eds.) ITP 2013. LNCS, vol. 7998, pp. 35–50. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  22. 22.
    Marcus, A., Poshyvanyk, D.: The conceptual cohesion of classes. In: IEEE International Conference on Software Maintenance, ICSM 2005, pp. 133–142 (2005)Google Scholar
  23. 23.
    Matichuk, D., et al.: Empirical study towards a leading indicator for cost of formal software verification. In: International Conference on Software Engineering, ICSE 2015, pp. 722–732 (2015)Google Scholar
  24. 24.
    Pąk, K.: Automated improving of proof legibility in the mizar system. In: Watt, S.M., Davenport, J.H., Sexton, A.P., Sojka, P., Urban, J. (eds.) CICM 2014. LNCS, vol. 8543, pp. 373–387. Springer, Heidelberg (2014)Google Scholar
  25. 25.
    Paulson, L.C.: The inductive approach to verifying cryptographic protocols. J. Comput. Secur. 6(1–2), 85–128 (1998)CrossRefGoogle Scholar
  26. 26.
    Pons, O., Bertot, Y., Rideau, L.: Notions of dependency in proof assistants. In: User Interfaces for Theorem Provers (UITP) (1998)Google Scholar
  27. 27.
    Simon, F., Steinbruckner, F., Lewerentz, C.: Metrics based refactoring. In: Software Maintenance and Reengineering, CSMR. 2001, pp. 30–38 (2001)Google Scholar
  28. 28.
    Sutcliffe, G.: The TPTP world – Infrastructure for automated reasoning. In: Clarke, E.M., Voronkov, A. (eds.) LPAR-16 2010. LNCS, vol. 6355, pp. 1–12. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  29. 29.
    Pąk, K.: Automated improving of proof legibility in the mizar system. In: Watt, S.M., Davenport, J.H., Sexton, A.P., Sojka, P., Urban, J. (eds.) CICM 2014. LNCS, vol. 8543, pp. 373–387. Springer, Heidelberg (2014)Google Scholar
  30. 30.
    Urban, J., Sutcliffe, G.: ATP cross-verification of the mizar MPTP challenge problems. In: Dershowitz, N., Voronkov, A. (eds.) LPAR 2007. LNCS (LNAI), vol. 4790, pp. 546–560. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  31. 31.
    Weyuker, E.J.: Evaluating software complexity measures. IEEE Trans. Software Eng. 14(9), 1357–1365 (1988)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.LFCS, School of InformaticsUniversity of EdinburghEdinburghScotland, UK
  2. 2.University of InnsbruckInnsbruckAustria

Personalised recommendations