Constraints

, Volume 9, Issue 1, pp 5–34 | Cite as

On Benchmarking Constraint Logic Programming Platforms. Response to Fernandez and Hill's “A Comparative Study of Eight Constraint Programming Languages over the Boolean and Finite Domains”

  • Mark Wallace
  • Joachim Schimpf
  • Kish Shen
  • Warwick Harvey
Article

Abstract

The comparative study published in this journal by Fernandez and Hill benchmarked some constraint programming systems on a set of well-known puzzles. The current article examines the positive and negative aspects of this kind of benchmarking.

The article analyses some pitfalls in benchmarking, recalling previous published results from benchmarking different kinds of software, and explores some issues in comparative benchmarking of CLP systems.

A benchmarking exercise should cover a broad set of representative problems and a broad set of programming constructs. This can be achieved using two kinds of benchmarking: Applications Benchmarking and Unit Testing. The article reports the authors' experiences with these two kinds of benchmarking in the context of the CHIC2 Esprit project. The benchmarks were used to unit test different features of the CLP system ECLiPSe and to compare application development with different high-level constraint platforms.

The conclusion is that, in deciding which system to use on a new application, it is less useful to compare standard features of CLP systems, than to compare their relevant functionalities.

constraint logic programming CLP Benchmarking ECLiPSe Unit Testing Finite Domain solver performance 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bachelu, A., Baptiste, P., Varnier C., Boucher, E., & Legeard B. (1997). Multi-criteria comparison between algorithmic, constraint logic and specific constraint programming on a real scheduling problem. In Proc. Practical Application of Constraint Technology, pages 47–63.Google Scholar
  2. 2.
    BASF, CORE, Dash, Procter and Gamble, Lisbon University, Peugeot, Barbot, Loria, & COSYTEC. (2002). Large scale integrated supply chain optimisation software, www.dash.co.uk/liscosweb/.Google Scholar
  3. 3.
    Beck, J. C., Davenport, A. J., & Fox, M. S. Five pitfalls of empirical scheduling reasearch. In Proc. Principles and Practice of Constraint Programming CP97, pages 390–404, Springer.Google Scholar
  4. 4.
    Chic-2 home page, www.icparc.ic.ac.uk/chic2/, 1999.Google Scholar
  5. 5.
    Carlson, M., Ottosson, G., & Carlson, B. (1997). An open-ended finite domain constraint solver. In PLILP'97, Vol. 1292 of LNCs, pages 191–206.Google Scholar
  6. 6.
    Dash Optimization. (2001). XPRESS-MP, www.dash.co.uk/.Google Scholar
  7. 7.
    Diaz, D., & Codognet, P. (1993). A minimal extension of the wam for clp(FD). In Proc. 10th International Conference on Logic Programming, pages 774–790.Google Scholar
  8. 8.
    El Sakkout, H., & Wallace, M. (2000). Probe backtrack search for minimal pertubation in dynamic scheduling. Constraints, 5(4): 359–388.Google Scholar
  9. 9.
    Eremin, A., & Wallace, M. (2001). Hybrid benders decomposition algorithms in constraint logic programming. In Proc. CP'2001.Google Scholar
  10. 10.
    Fernández, A. (1997). Self-referential quizzes, URL: www.lcc.uma.es/~afdez/srq/index.html.Google Scholar
  11. 11.
    Fernández, A., & Hill, P. M. (2000). A comparative study of eight constraint programming languages over the Boolean and finite domains. Journal of Constraints, 5: 275–301.Google Scholar
  12. 12.
    Gray, J., ed. (1991). The Benchmark Handbook. Data Management Systems. Morgan Kaufmann.Google Scholar
  13. 13.
    Haygood, R. (1989). A prolog benchmark suite for aquarius. Technical Report CSD-89-809, University of California, Berkeley.Google Scholar
  14. 14.
    Henz, M. (1996). Don't be puzzled! In Proceedings of the Workshop on Constraint Programming Applications, in conjunction with the Second International Conference on principles and Practice of Constraint Programming (CP96), Cambridge, Massachussetts, USA (August).Google Scholar
  15. 15.
    Hajian, M., El-Sakkout, H., Wallace, M., Lever, J., & Richards, E. B. (1998). Towards a closer integration of finite domain propagation and simplex-based methods. Annals of Operations Research, 81: 421–431.Google Scholar
  16. 16.
    Hooker, J. N. (1996). Testing heuristics: We have it all wrong. Journal of Heuristics, 1: 33–42.Google Scholar
  17. 17.
    ILOG. (2001) CPLEX, www.ilog.com/products/cplex/.Google Scholar
  18. 18.
    Kernighan, B., & Van Wyk, C. (1998). Timing trials, or the trials of timing: Experiments with scripting and user-interface languages. Software Practise and Experience, 28(8): 819–843, cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html.Google Scholar
  19. 19.
    McClain, D. (1999). A comparison of programming languages for scientific processing, URL: www. azstarnet.com/~dmcclain/LanguageStudy.html.Google Scholar
  20. 20.
    Mackworth, A. K., & Freuder, E. C. (1985). The complexity of some polynomial network consistency algorithms for constraint satisfaction problems. Artificial Intelligence, 25(1): 65–74.Google Scholar
  21. 21.
    Müller, T., and Würtz, J. Interfacing propagators with a concurrent constraint language. In PLILP'96, pages 195–206.Google Scholar
  22. 22.
    Nadel, B. (1989). Constraint satisfaction algorithms. Computational Intelligence, 5: 188–224.Google Scholar
  23. 23.
    Ovako, Air Liquide, IQSOFT, PrologIA, & SICS. (1998). Trial application using constraint programming in industrial manufacturing. ESPRIT Project No 23365, URL: www.sics.se/col/projects/tacit/description.html.Google Scholar
  24. 24.
    Pereira, F. (1987). AI expert prolog benchmarks, URL: www.icparc.ic.ac.uk/~mgw/aiexpertbench.txt.Google Scholar
  25. 25.
    Refalo, P. (1999). Tight cooperation and its application in piecewise linear optimization. In CP'99, Vol. 1913 of LNCS, pages 373–389.Google Scholar
  26. 26.
    Rodosek, R., & Wallace, M. G. (1998). A generic model and hybrid algorithm for hoist scheduling problems. In Proceedings of the 4th International Conference on Principles and Practice of Constraint Programming, pages 385-399, Pisa.Google Scholar
  27. 27.
    Rodosek, R., Wallace, M., & Hajian, M. (1999). A new approach to integrating mixed integer programming with constraint logic programming. Annals of Operations Research, 86: 63–87.Google Scholar
  28. 28.
    Van Hentenryck, P. (1989). Constraint Satisfaction in Logic Programming. Logic Programming. MIT Press.Google Scholar
  29. 29.
    Waclena, K. (1997). My programming language crisis, URL: www.lib.uchicago.edu/keith/crisis/.Google Scholar
  30. 30.
    Williams, H. P. (1993). Model Building in Mathematical Programming. John Wiley and Sons.Google Scholar
  31. 31.
    Wallace, M., Novello, S., & Schimpf, J. (1997). Eclipse-a platform for constraint programming. ICL Systems Journal, 12(1): 159–200.Google Scholar
  32. 32.
    Wallace, M., & Schimpf, J. (2002). Finding the right hybrid algorithm-a combinatorial meta-problem. Annals of Mathematics and Artificial Intelligence, Special issue on Large Scale Combinatorial Optimisation and Constraints, to appear.Google Scholar
  33. 33.
    Zhou, N.-F. (1998). A high-level intermediate language and the algorithms for compiling finite-domain constraints. In Proc. Joint International Conference on Logic Programming, pages 70–84. MIT Press.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Mark Wallace
    • 1
  • Joachim Schimpf
    • 1
  • Kish Shen
    • 1
  • Warwick Harvey
    • 1
  1. 1.IC-Parc, William Penney LaboratoryImperial CollegeLondonUK

Personalised recommendations