Abstract
Running a competition for automated theorem proving (ATP) systems is a difficult and arguable venture. However, the potential benefits of such an event by far outweigh the controversial aspects. The motivations for running the CADE-13 ATP System Competition were to contribute to the evaluation of ATP systems, to stimulate ATP research and system development, and to expose ATP systems to researchers both within and outside the ATP community. This article identifies and discusses the issues that determine the nature of such a competition. Choices and motivated decisions for the CADE-13 competition, with respect to the issues, are given.
Similar content being viewed by others
References
Astrachan, O., Khera, V. and Kotz, D.: The Internet programming contest: A report and philosophy, in Proc. of the 24th SIGCSE Technical Conference, SIGCSE Bulletin, ACM Press, 1993, pp. 48–52.
Astrachan, O. L. and Loveland, D. W.: Measuring the performance of automated theorem provers, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994, pp. 37–41.
Boyer, R. S. and Moore, J. S.: A theorem prover for a computational logic, in Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 1–15.
Buro, M. and Kleine-Büning, H.: Report on a SAT Competition. Technical Report FB 17, Mathematik/Informatik, Universität Paderborn, Paderborn, Germany, 1992.
Chen, E., Istrail, S., Johnson, D., Kececioglu, J., Messing, J., Nadeau, J., Pevzner, P., Rice, P., Vingron, M. and Waterman, M.: The 4th DIMACS Interantional Algorithm Implementation Challenge. URL http://dimacs.rutgers.edu/challenge.html, 1994.
Comer, J., Rinewalt, R., Ryan, P. and Poucher, W. B.: The ACM scholastic programming contest - 1977 to 1990, Proceedings of the 21st SIGCSE Technical Symposium on Computer Science Education, SIGCSE Bulletin 22(1) (1990), 256.
Constable, R., Allen, S., Bromly, H., Cleaveland, W., Cremer, J., Harper, R., Howe, D., Knoblock, T., Mendler, N., Panangaden, P., Sasaki, J. and Smith, S.: Implementing Mathematics with the Nuprl Proof Development System, Prentice-Hall, 1986.
Demoen, B.: The 2nd annual Prolog programming contest, The Newsletter for the Association for Logic Programming 9(2) (1996), 3–4.
Dunker, U.: Search Space and Proof Complexity of Theorem Proving Strategies, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994.
Farmer, W., Guttman, J. and Thayer, J.: IMPS: An interactive mathematical proof system, in Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 653–654.
Gordon, M. and Melham, T.: Introduction to HOL, a Theorem Proving Environment for Higher Order Logic, Cambridge University Press, 1993.
Hanks, S., Pollack, M. E. and Cohen, P. R.: Benchmarks, test beds, controlled experimentation and the design of agent architectures, AI Magazine 14(4) (1993), 17–42.
Heisel, M., Reif, W. and Stephan, W.: Tactical theorem proving in program verification, in M. Stickel (ed.), Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 117–131.
Huang, X., Kerber, M., Kohlhase, M., Melis, E., Nesmith, D., Richts, J. and Siekmann, J.: Ω-MKRP: A proof development environment, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 788–792.
Kerber, M.: How to prove higher order theorems in first order logic, in Proc. of the 12th International Joint Conference on Artificial Intelligence, IJCAI, 1991, pp. 137–142.
Kopec, D., Newborn, M. and Valvo, M.: The 22nd annual ACM international computer chess championship, Communications of the ACM 35(11) (1992), 100–110.
Lee, S.-J. and Plaisted, D. A.: Eliminating duplication with the hyper-linking strategy, Journal of Automated Reasoning 9(1) (1992), 25–42.
Letz, R.: First-Order Calculi and Proof Procedures for Automated Deduction, PhD thesis, Technical University of Munich, Munich, Germany, 1993.
McCune, W. W.: Otter 3.0 Reference Manual and Guide. Technical Report ANL-94/6, Argonne National Laboratory, Argonne, Illinois, 1994.
Michie, D., Muggleton, D., Page, D. and Srinivasan, A.: 3 Inductive Learning Competitions. URL ftp://ftp.comlab.ox.ac.uk/pub/Packages/ILP, 1994.
Ohlbach, H. J.: Semantics based translation methods for modal logics, Journal of Logic and Computation 1(5) (1991), 691–746.
Overbeek, R.: The CADE-11 competitions: A personal view, Journal of Automated Reasoning 11(3) (1993), 315–316.
Owre, S., Rushby, J. M. and Shankar, N.: PVS: A prototype verification system, in D. Kapur (ed.), Proc. of the 11th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 607, Springer-Verlag, 1992, pp. 748–752.
Paulson, L. C.: Isabelle: The next 700 theorem provers, Logic and Computer Science (1990), 361–386.
Pelletier, F. J.: The philosophy of automated theorem proving, in J. Mylopolous and R. Reiter (eds), Proc. of the 12th International Joint Conference on Artificial Intelligence, IJCAI, 1991, pp. 1039–1045.
Plaisted, D. A.: The search efficiency of theorem proving strategies, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 57–71.
Portoraro, F. D.: Symlog: Automated advice in Fitch-style proof construction, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 802–806.
Prehofer, C. and Slind, K.: Theorem proving in interactive verification systems, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994.
G. Sutcliffe and C. B. Suttner: ATP System Results for the TPTP Problem Library (up to TPTP v1.1.3), 1995. Technical Report 95/16, Department of Computer Science, James Cook University, Townsville, Australia; Technical Report AR-95-06, Institut für Informatik, Technische Universität München, Munich, Germany.
Sutcliffe, G. and Suttner, C. B.: Special issue: The CADE-13 ATP system competition, Journal of Automated Reasoning 18(2) (1997).
Sutcliffe, G. and Suttner, C. B.: The results of the CADE-13 ATP system competition, Journal of Automated Reasoning 18(2) (1997).
Sutcliffe, G., Suttner, C. B. and Yemenis, T.: The TPTP problem library, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 252–266.
Suttner, C. B. and Sutcliffe, G.: The TPTP Problem Library (TPTP v1.2.1), 1996. Technical Report AR-96-02, Institut für Informatik, Technische Universität München, Munich, Germany; Technical Report 96/09, Department of Computer Science, James Cook University, Townsville, Australia.
Wadsworth, H. M.: Handbook of Statistical Methods for Engineers and Scientists, McGraw-Hill, 1990.
Wang, T. C. and Goldberg, A.: RVF: An automated formal verification system, in D. Kapur (ed.), Proc. of the 11th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 607, Springer-Verlag, 1992, pp. 735–739.
Wos, L., Winker, S., McCune, W. W., Overbeek, R., Lusk, E. and Stevens, R.: Automated reasoning contributes to mathematics and logic, in M. Stickel (ed.), Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 485–499.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Suttner, C., Sutcliffe, G. The Design of the CADE-13 ATP System Competition. Journal of Automated Reasoning 18, 139–162 (1997). https://doi.org/10.1023/A:1005802523220
Issue Date:
DOI: https://doi.org/10.1023/A:1005802523220