Journal of Automated Reasoning

, Volume 52, Issue 2, pp 191–213 | Cite as

Premise Selection for Mathematics by Corpus Analysis and Kernel Methods

  • Jesse Alama
  • Tom Heskes
  • Daniel Kühlwein
  • Evgeni Tsivtsivadze
  • Josef UrbanEmail author


Smart premise selection is essential when using automated reasoning as a tool for large-theory formal proof development. This work develops learning-based premise selection in two ways. First, a fine-grained dependency analysis of existing high-level formal mathematical proofs is used to build a large knowledge base of proof dependencies, providing precise data for ATP-based re-verification and for training premise selection algorithms. Second, a new machine learning algorithm for premise selection based on kernel methods is proposed and implemented. To evaluate the impact of both techniques, a benchmark consisting of 2078 large-theory mathematical problems is constructed, extending the older MPTP Challenge benchmark. The combined effect of the techniques results in a 50 % improvement on the benchmark over the state-of-the-art Vampire/SInE system for automated reasoning in large theories.


Automated reasoning in large theories Machine learning Premise selection Automated theorem proving 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alama, J.: Formal proofs and refutations. Ph.D. thesis, Stanford University (2009)Google Scholar
  2. 2.
    Alama, J., Brink, K., Mamane, L., Urban, J.: Large formal wikis: Issues and solutions. In: Davenport, J.H., Farmer, W.M., Urban, J., Rabe, F. (eds.) Calculemus/MKM. Lecture Notes in Computer Science, vol. 6824, pp. 133–148. Springer (2011)Google Scholar
  3. 3.
    Alama, J., Kühlwein, D., Urban, J.: Automated and human proofs in general mathematics: an initial comparison. In: Bjørner, N., Voronkov, A. (eds.) LPAR. Lecture Notes in Computer Science, vol. 7180, pp. 37–45. Springer (2012)Google Scholar
  4. 4.
    Alama, J., Mamane, L., Urban, J.: Dependencies in formal mathematics: applications and extraction for Coq and Mizar. In: Jeuring, J., Campbell, J.A., Carette, J., Reis, G.D., Sojka, P., Wenzel, M., Sorge, V. (eds.) AISC/MKM/Calculemus. Lecture Notes in Computer Science, vol. 7362, pp. 1–16. Springer (2012)Google Scholar
  5. 5.
    Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950)Google Scholar
  6. 6.
    Bertot, Y., Castéran, P.: Interactive theorem proving and program development. Coq’Art: the calculus of inductive constructions. In: Texts in Theoretical Computer Science. Springer (2004)Google Scholar
  7. 7.
    Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Secaucus (2006)Google Scholar
  8. 8.
    Blanchette, J.C., Bulwahn, L., Nipkow, T.: Automatic proof and disproof in Isabelle/HOL. In: Tinelli, C., Sofronie-Stokkermans, V. (eds.) FroCos. Lecture Notes in Computer Science, vol. 6989, pp. 12–27. Springer (2011)Google Scholar
  9. 9.
    Carlson, A., Cumby, C., Rosen, J., Roth, D.: The SNoW learning architecture. Tech. Rep. UIUCDCS-R-99-2101, UIUC Computer Science Department (1999).
  10. 10.
    Davis, M.: Obvious logical inferences. In: Hayes, P.J. (ed.) IJCAI, pp. 530–531. Kaufmann (1981)Google Scholar
  11. 11.
    Grabowski, A., Korniłowicz, A., Naumowicz, A.: Mizar in a nutshell. J. Formaliz. Reason. 3(2), 153–245 (2010)zbMATHMathSciNetGoogle Scholar
  12. 12.
    Harrison, J.: HOL light: A tutorial introduction. In: Srivas, M.K., Camilleri, A.J. (eds.) FMCAD. Lecture Notes in Computer Science, vol. 1166, pp. 265–269. Springer (1996)Google Scholar
  13. 13.
    Harrison, J., Slind, K., Arthan, R.: HOL. In: Wiedijk, F. (ed.) The Seventeen Provers of the World. Lecture Notes in Computer Science, vol. 3600, pp. 11–19. Springer (2006)Google Scholar
  14. 14.
    Hoder, K., Voronkov, A.: Sine qua non for large theory reasoning. In: Bjørner, N., Sofronie-Stokkermans, V. (eds.) CADE. Lecture Notes in Computer Science, vol. 6803, pp. 299–314. Springer (2011)Google Scholar
  15. 15.
    Meng, J., Paulson, L.C.: Translating higher-order clauses to first-order clauses. J. Autom. Reason. 40(1), 35–60 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    de Moura, L.M., Bjørner, N.: Z3: An efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS. Lecture Notes in Computer Science, vol. 4963, pp. 337–340. Springer (2008)Google Scholar
  17. 17.
    Nipkow, T., Paulson, L.C., Wenzel, M.: Isabelle/HOL—A proof assistant for higher-order logic. Lecture Notes in Computer Science, vol. 2283. Springer (2002)Google Scholar
  18. 18.
    Paulson, L.C., Susanto, K.W.: Source-level proof reconstruction for interactive theorem proving. In: Schneider, K., Brandt, J. (eds.) TPHOLs. Lecture Notes in Computer Science, vol. 4732, pp. 232–245. Springer (2007)Google Scholar
  19. 19.
    Pease, A., Sutcliffe, G.: First order reasoning on a large ontology. In: Sutcliffe, G., Urban, J., Schulz, S. (eds.) Proceedings of the CADE-21 Workshop on Empirically Successful Automated Reasoning in Large Theories, Bremen, Germany, 17th July 2007. CEUR Workshop Proceedings, vol. 257. (2007)Google Scholar
  20. 20.
    Riazanov, A., Voronkov, A.: The design and implementation of VAMPIRE. AI Commun. 15(2–3), 91–110 (2002)zbMATHGoogle Scholar
  21. 21.
    Richard, M.D., Lippmann, R.P.: Neural network classifiers estimate Bayesian a posteriori probabilities. Neural Comput. 3(4), 461–483 (2010)CrossRefGoogle Scholar
  22. 22.
    Rifkin, R., Yeo, G., Poggio, T.: Regularized least-squares classification. In: Suykens, J., Horvath, G., Basu, S., Micchelli, C., Vandewalle, J. (eds.) Advances in Learning Theory: Methods, Model and Applications, pp. 131–154. IOS Press, Amsterdam (2003)Google Scholar
  23. 23.
    Rudnicki, P.: Obvious inferences. J. Autom. Reason. 3(4), 383–393 (1987)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Schoelkopf, B., Herbrich, R., Williamson, R., Smola, A.J.: A generalized representer theorem. In: Helmbold, D., Williamson, R. (eds.) Proceedings of the 14th Annual Conference on Computational Learning Theory, pp. 416–426. Berlin, Germany (2001)Google Scholar
  25. 25.
    Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2001)Google Scholar
  26. 26.
    Schulz, S.: E - A brainiac theorem prover. AI Commun. 15(2–3), 111–126 (2002)zbMATHGoogle Scholar
  27. 27.
    Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: primal estimated sub-gradient solver for SVM. Math. Program. 127(1), 3–30 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  28. 28.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, New York (2004)CrossRefGoogle Scholar
  29. 29.
    Simpson, S.G.: Subsystems of Second Order Arithmetic, 2nd edn. Perspectives in Mathematical Logic. Springer (2009)Google Scholar
  30. 30.
    Solovay, R.: AC and strongly inaccessible cardinals. Available on the Foundations of Mathematics archives at (2008)
  31. 31.
    Tarski, A.: On well-ordered subsets of any set. Fundam. Math. 32, 176–183 (1939)Google Scholar
  32. 32.
    Trybulec, A.: Tarski Grothendieck set theory. Formaliz. Math. 1(1), 9–11 (1990)Google Scholar
  33. 33.
    Tsivtsivadze, E., Pahikkala, T., Boberg, J., Salakoski, T., Heskes, T.: Co-regularized least-squares for label ranking. In: Fürnkranz, J., Hüllermeier, E. (eds.) Preference Learning, pp. 107–123. Springer, Berlin (2011)Google Scholar
  34. 34.
    Urban, J.: MPTP—motivation, implementation, first experiments. J. Autom. Reason. 33(3–4), 319–339 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Urban, J.: MPTP 0.2: design, implementation, and initial experiments. J. Autom. Reason. 37(1–2), 21–43 (2006)zbMATHGoogle Scholar
  36. 36.
    Urban, J., Hoder, K., Voronkov, A.: Evaluation of automated theorem proving on the Mizar Mathematical Library. In: Fukuda, K., van der Hoeven, J., Joswig, M., Takayama, N. (eds.) ICMS. Lecture Notes in Computer Science, vol. 6327, pp. 155–166. Springer (2010)Google Scholar
  37. 37.
    Urban, J., Rudnicki, P., Sutcliffe, G.: ATP and presentation service for Mizar formalizations. J. Autom. Reason. 50, 229–241 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  38. 38.
    Urban, J., Sutcliffe, G.: Automated reasoning and presentation support for formalizing mathematics in Mizar. In: Autexier, S., Calmet, J., Delahaye, D., Ion, P.D.F., Rideau, L., Rioboo, R., Sexton, A.P. (eds.) AISC/MKM/Calculemus. Lecture Notes in Computer Science, vol. 6167, pp. 132–146. Springer (2010)Google Scholar
  39. 39.
    Urban, J., Sutcliffe, G., Pudlák, P., Vyskocil, J.: MaLARea SG1- machine learner for automated reasoning with semantic guidance. In: Armando, A., Baumgartner, P., Dowek, G. (eds.) IJCAR. Lecture Notes in Computer Science, vol. 5195, pp. 441–456. Springer (2008)Google Scholar
  40. 40.
    Weidenbach, C., Dimova, D., Fietzke, A., Kumar, R., Suda, M., Wischnewski, P.: SPASS version 3.5. In: Schmidt, R.A. (ed.) CADE. LNCS, vol. 5663, pp. 140–145. Springer (2009)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Jesse Alama
    • 1
  • Tom Heskes
    • 2
  • Daniel Kühlwein
    • 2
  • Evgeni Tsivtsivadze
    • 2
  • Josef Urban
    • 2
    Email author
  1. 1.Center for Artificial IntelligenceNew University of LisbonLisbonPortugal
  2. 2.Intelligent Systems, Institute for Computing and Information SciencesRadboud UniversityNijmegenNetherlands

Personalised recommendations