The competitive nature of most algorithmic experimentation is a source of problems that are all too familiar to the research community. It is hard to make fair comparisons between algorithms and to assemble realistic test problems. Competitive testing tells us which algorithm is faster but not why. Because it requires polished code, it consumes time and energy that could be better spent doing more experiments. This article argues that a more scientific approach of controlled experimentation, similar to that used in other empirical sciences, avoids or alleviates these problems. We have confused research and development; competitive testing is suited only for the latter.
Key Wordscomputational testing benchmark problems
Unable to display preview. Download preview PDF.
- Böhm, H. (1992). Report on a SAT competition. Technical report no. 110, Universität Paderborn, Germany.Google Scholar
- Chandru, V., Coullard, C.R., Hammer, P.L., Montanez, M., and Sun, X. (1990). On renamable Horn and generalired Horn functions. InAnnals of Mathematics and Artificial Intelligence, Basel: Baltzer AG.Google Scholar
- Cheeseman, P., Kanefsky, B., and Taylor, W.M. (1991). Where the really hard problems are. InProceedings of the International Joint Conference on Artificial Intelligence, ICAI91, Sydney, Springer-Verlag (pp. 331–337).Google Scholar
- Crawford, J.M., and Auton, L.D. (1993). Experimental results on the crossover point in satisfiability problems. InProceedings of the Eleventh National Conference on Artificial Intelligence, AAAI93, Washington, D.C., MIT Press (pp. 21–27).Google Scholar
- Gent, I.P., and Walsh, T. (1994). The SAT phase transition. In A.G. Cohn (Ed.),Proceedings of the Eleventh European Conference on Artificial Intelligence, ECAI94 (pp. 105–109).Google Scholar
- Hooker, J.N., and Vinay, V. (Forthcoming). Branching rules for satisfiability.Journal of Automated Reasoning.Google Scholar
- Larrabee, T., and Tsujii, Y. (1993). Evidence for a satisfiability threshold for random 3cnf formulas. In H. Hirsch et al. (Eds.),Proceedings of the Spring Symposium on Artificial Intelligence and NP-Hard Problems (pp. 112–118). Stanford, CA.Google Scholar
- McGeoch, C.C. (Forthcoming). Toward an experimental method for algorithm simulations.ORSA Journal on Computing.Google Scholar
- Mitchell, D., Selman, B., and Levesque, H. (1992). Hard and easy distributions of SAT problems. InProceedings, Tenth National Conference on Artificial Intelligence, AAAI92 (pp. 459–465). Cambridge, MA: MIT Press.Google Scholar
- Trick, J., and Johnson, D.S. (Eds.). (1995).Second DIMACS Challenge: Cliques, Coloring and Satisfiability. Series in Discrete Mathematics and Theoretical Computer Science. Providence, RI: American Mathematical Society.Google Scholar