Selecting Problems for Algorithm Evaluation

  • Andrew V. Goldberg
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1668)


In this paper we address the issue of developing test sets for computational evaluation of algorithms. We discuss both test families for comparing several algorithms and selecting one to use in an application, and test families for predicting algorithm performance in practice.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    W. Barr, B. Golden, J. Kelly, M. Resende, and W. Stewart. Designing and Reporting on Computational Experiments with Heuristic Methods. J. Heuristics, 1:9–32, 1995.MATHCrossRefGoogle Scholar
  2. 2.
    R. E. Bellman. On a Routing Problem. Quart. Appl. Math., 16:87–90, 1958.MathSciNetMATHGoogle Scholar
  3. 3.
    D. P. Bertsekas and P. Tseng. Relaxation Methods for Minimum Cost Ordinary and Generalized Network Flow Problems. Oper. Res., 36:93–114, 1988.MATHMathSciNetGoogle Scholar
  4. 4.
    C. S. Chekuri, A. V. Goldberg D. R. Karger, M. S. Levine, and C. Stein. Experimental Study of Minimum Cut Algorithms. In Proc. 8th ACM-SIAM Symposium on Discrete Algorithms, pages 324–333, 1997.Google Scholar
  5. 5.
    J. Cheriyan and S. N. Maheshwari. Analysis of Preflow Push Algorithms for Maximum Network Flow. SIAM J. Comput., 18:1057–1086, 1989.MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    B. V. Cherkassky and A. V. Goldberg. On Implementing Push-Relabel Method for the Maximum Flow Problem. Algorithmica, 19:390–410, 1997.MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    B. V. Cherkassky, A. V. Goldberg, and T. Radzik. Shortest Paths Algorithms: Theory and Experimental Evaluation. Math. Prog., 73:129–174, 1996.MathSciNetGoogle Scholar
  8. 8.
    E. W. Dijkstra. A Note on Two Problems in Connexion with Graphs. Numer. Math., 1:269–271, 1959.MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    N. Eiron, M. Rodeh, and I. Steinwarts. Matrix Multiplication: A Case Study of Algorithm Engineering. In Proc. 2-nd Workshop on Algorithm Engineering, Saarbruken, Germany, pages 98–109, 1998.Google Scholar
  10. 10.
    R.E. Erickson, C.L. Monma, and A.F. Veinott Jr. Send-and-split method for minimum-concave-cost network flows. Math. of Oper. Res., 12:634–664, 1979.MathSciNetGoogle Scholar
  11. 11.
    L. R. Ford, Jr. and D. R. Fulkerson. Flows in Networks. Princeton Univ. Press, Princeton, NJ, 1962.MATHGoogle Scholar
  12. 12.
    A. V. Goldberg and M. Kharitonov. On Implementing Scaling Push-Relabel Algorithms for the Minimum-Cost Flow Problem. In D. S. Johnson and C. C. McGeoch, editors, Network Flows and Matching: First DIMACS Implementation Challenge (Refereed Proceedings), pages 157–198. AMS, 1993.Google Scholar
  13. 13.
    A. V. Goldberg and B. M. E. Moret. Combinatorial Algorithms Test Sets (CATS): The ACM/EATCS Platform for Experimental Research. In Proc. 10th ACM-SIAM Symposium on Discrete Algorithms, pages S913–S914, 1999.Google Scholar
  14. 14.
    M. D. Grigoriadis and Y. Hsu. The Rutgers Minimum Cost Network Flow Subroutines. SIGMAP Bulletin of the ACM, 26:17–18, 1979.Google Scholar
  15. 15.
    D. Gusfield. University of california at davis. personal communication. 1996.Google Scholar
  16. 16.
    D. B. Johnson. How to do Experiments. Unpublished manuscript, AT&T Laboratories, 1996.Google Scholar
  17. 17.
    D. Klingman, A. Napier, and J. Stutz. Netgen: A Program for Generating Large Scale Capacitated Assignment, Transportation, and Minimum Cost Flow Network Problems. Management Science, 20:814–821, 1974.MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    D. Knuth. The Stanford GraphBase. ACM Press, New York, NY. and Addison-Wesley, Reading, MA, 1993.Google Scholar
  19. 19.
    A. LaMarca and R. E. Ladner. The Influence of Caches on the Performance of Heaps. ACM Jour. of Experimental Algorithmics, 1, 1996.Google Scholar
  20. 20.
    T. Leong, P. Shor, and C. Stein. Implementation of a Combinatorial Multicommodity Flow Algorithm. In D. S. Johnson and C. C. McGeoch, editors, Network Flows and Matching: First DIMACS Implementation Challenge (Refereed Proceedings), pages 387–406. AMS, 1993.Google Scholar
  21. 21.
    M. S. Levine. Experimental Study of Minimum Cut Algorithms. Technical Report MIT-LCS-TR-719, MIT Lab for Computer Science, 1997.Google Scholar
  22. 22.
    B. Ju. Levit and B. N. Livshits. Nelineinye Setevye Transportnye Zadachi. Transport, Moscow, 1972. In Russian.Google Scholar
  23. 23.
    C. McGeoch. Towards an Experimental Method for Algorithm Simulation. INFORMS J. Comp., 8:1–15, 1996.MATHGoogle Scholar
  24. 24.
    A. A. Middleton. Numerical Result for the Ground-State Interface in Random Medium. Phys. Rev. E, 52:R3337–R3340, 1995.CrossRefGoogle Scholar
  25. 25.
    E. F. Moore. The Shortest Path Through a Maze. In Proc. of the Int. Symp. On the Theory of Switching, pages 285–292. Harvard University Press, 1959.Google Scholar
  26. 26.
    B. M. E. Moret and H. D. Shapiro. An Empirical Assessment of Algorithms for Constructing a Minimum Spanning Tree. In DIMACS Monographs in Discrete Mathematics and Theoretical Computer Science 15, pages 99–117, 1994.Google Scholar
  27. 27.
    B. M.E. Moret and H. D. Shapiro. On Minimizing a Set of Tests. SIAM J. Sceintific & Statistical Comp., 6:983–1003, 1985.CrossRefGoogle Scholar
  28. 28.
    A. T. Ogielski. Integer Optimization and Zero-Temperature Fixed Point in Ising Random-Field Systems. Physical Review Lett., 57:1251–1254, 1986.CrossRefGoogle Scholar
  29. 29.
    S. Pallottino. Shortest-Path Methods: Complexity, Interrelations and New Propositions. Networks, 14:257–267, 1984.MATHCrossRefGoogle Scholar
  30. 30.
    U. Pape. Implementation and Efficiency of Moore Algorithms for the Shortest Root Problem. Math. Prog., 7:212–222, 1974.MATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    S. Roy and I. Cox. A Maximum-Flow Formulation to the Stereo Correspondence Problem. In Proc. Int. Conf. on Computer Vision (ICCV’ 98), Bombay, India, pages 492–499, 1998.Google Scholar
  32. 32.
    F. B. Zhan and C. E. Noon. Shortest Path Algorithms: An Evaluation using Real Road Networks. Transp. Sci., 32:65–73, 1998.MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Andrew V. Goldberg
    • 1
  1. 1.InterTrust STAR Lab.SunnyvaleUSA

Personalised recommendations