Mixed Models for the Analysis of Optimization Algorithms

  • Marco ChiarandiniEmail author
  • Yuri GoegebeurEmail author


We review linear statistical models for the analysis of computational experiments on optimization algorithms. The models offer the mathematical framework to separate the effects of algorithmic components and instance features included in the analysis. We regard test instances as drawn from a population and we focus our interest not on those single instances but on the whole population. Hence, instances are treated as a random factor. Overall these experimental designs lead to mixed effects linear models. We present both the theory to justify these models and a computational example in which we analyze and comment on several possible experimental designs. The example is a component-wise analysis of local search algorithms for the 2-edge-connectivity augmentation problem. We use standard statistical software to perform the analysis and report the R commands. Data sets and the analysis in SAS are available in an online compendium.


Optimization Algorithm Local Search Local Search Algorithm Instance Class Marginal Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bang-Jensen J, Chiarandini M, Morling P (2009) A computational investigation of heuristic algorithms for 2-edge-connectivity augmentation.Networks (In print.)Google Scholar
  2. Barr R, Golden B, Kelly J, Resende M, Stewart W (1995) Designing and reporting on computational experiments with heuristic methods.Journal of Heuristics 1(1):9–32zbMATHCrossRefGoogle Scholar
  3. Bates D (2007) Personal CommunicationGoogle Scholar
  4. Bates D, Maechler M, Dai B (2008) lme4:Linear mixed-effects models using S4 classes. URL, R package version 0.999375-28
  5. Birattari M (2004) On the estimation of the expected performance of a metaheuristic on a class of instances. How many instances, how many runs? Tech. Rep. TR/IRIDIA/2004-01, IRIDIA, Université Libre de Bruxelles, Brussels, BelgiumGoogle Scholar
  6. Bondy J, Murty U (2008) Graph Theory. Graduate Texts in Mathematics, Vol. 244,SpringerGoogle Scholar
  7. Chiarandini M (2005) Stochastic local search methods for highly constrained combinatorial optimisation problems. PhD thesis, Computer Science Department, Darmstadt University of Technology, Darmstadt, GermanyGoogle Scholar
  8. Coffin M, Saltzman MJ (2000)Statistical analysis of computational tests of algorithms and heuristics. INFORMS Journal on Computing 12(1):24–44zbMATHCrossRefGoogle Scholar
  9. Conforti M, Galluccio A, Proietti G (2004) Edge-connectivity augmentation and network matrices. In:Workshop on Graph-Theoretic Concepts in Computer Science, Springer, Lecture Notes in Computer Science, vol 3353, pp 355–364CrossRefGoogle Scholar
  10. Cormen T, Leiserson C, Rivest R (2001) Introduction to Algorithms, 2nd edn.MIT presszbMATHGoogle Scholar
  11. Czarn A, MacNish C, Vijayan K, Turlach B, Gupta R (2004) Statistical exploratory analysis of genetic algorithms. Evolutionary Computation, IEEE Transactions on 8(4):405–421CrossRefGoogle Scholar
  12. Fox J (2002) Linear mixed models. Appendix to An R and S-PLUS Companion to Applied Regression,URL contrib/Fox-Companion/appendix-mixed-models.pdf
  13. Glover F, Kochenberger G (eds) (2002) Handbook of Metaheuristics, International Series in Operations Research & Management Science, vol 57. Kluwer Academic, Norwell, MAGoogle Scholar
  14. Hooker JN (1996) Testing heuristics: We have it all wrong. Journal of Heuristics 1:32–42Google Scholar
  15. Johnson R, Wichern D (2007) Applied Multivariate Statistical Analysis, 6th edn.Prentice-Hall InternationalzbMATHGoogle Scholar
  16. MKutner MH, Nachtsheim CJ, Neter J, Li W (2005) Applied Linear Statistical Models, 5th edn. McGraw-HillGoogle Scholar
  17. Lehmann E (2003) Theory of point estimation.SpringerGoogle Scholar
  18. Lehmann E, Romano J (2008) Testing statistical hypothesis.SpringerGoogle Scholar
  19. Lenth RV (2006) Java applets for power and sample size [computer software]. Retrieved 29 January 2009 from˜rlenth/ Power
  20. Lin BW, Rardin RL (1979) Controlled experimental design for statistical comparison of integer programming algorithms. Management Science 25(12):1258–1271CrossRefMathSciNetGoogle Scholar
  21. McGeoch CC (1996) Toward an experimental method for algorithm simulation. INFORMS Journal on Computing 8(1):1–15, this journal issue contains also commentaries by Pierre L’Ecuyer, James B. Orlin and Douglas R. Shier, and a rejoinder by C. McGeochzbMATHCrossRefMathSciNetGoogle Scholar
  22. Michiels W, Aarts E, Korst J (2007) Theoretical Aspects of Local Search. Monographs in Theoretical Computer Science, An EATCS Series, SpringerGoogle Scholar
  23. Molenberghs G, Verbeke G (eds) (1997)Linear Mixed Models in Practice - A SASOriented Approach. >SpringerGoogle Scholar
  24. Molenberghs G, Verbeke G (2005) Models for Discrete Longitudinal Data. SpringerzbMATHGoogle Scholar
  25. Montgomery DC (2005) Design and Analysis of Experiments, 6th edn. WileyzbMATHGoogle Scholar
  26. Pinheiro J, Bates D, DebRoy S, Sarkar D, the R Core team (2008) nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1–89Google Scholar
  27. Pinheiro JC, Bates DM (2000) Mixed-Effects Models in S and S-Plus. SpringerzbMATHCrossRefGoogle Scholar
  28. R Development Core Team (2008) R: A Language and Environment for StatisticalComputing. R Foundation for Statistical Computing, Vienna, Austria, URL
  29. Rardin RL, Uzsoy R (2001) Experimental evaluation of heuristic optimization algorithms: A tutorial. Journal of Heuristics 7(3):261–304zbMATHCrossRefGoogle Scholar
  30. Rousseeuw PJ (1984) Least median of squares regression. Journal of the American Statistical Association 79(388):871–880, URL stable/2288718 SAS Institute Inc. (2007) SAS online documentation: Parameterization of mixed models., retrieved on 2009-05-24
  31. Stram D, Lee J (1994) Variance components testing in the longitudinal mixed effects model. Biometrics 50(4):1171–1177zbMATHCrossRefGoogle Scholar
  32. Stram D, Lee J (1995) Correction to ’Variance components testing in the longitudinal mixed effects model’. Biometrics 51(3):1196Google Scholar
  33. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1(1):67–82CrossRefGoogle Scholar
  34. Zanakis SH (1977) Heuristic 0-1 linear programming: An experimental comparison of three methods. Management Science 24(1):91–104zbMATHCrossRefMathSciNetGoogle Scholar
  35. Zemel E (1981) Measuring the quality of approximate solutions to zero-one programming problems. Mathematics of Operations Research 6(3):319–332zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  1. 1.Department of Mathematics and Computer ScienceUniversity of Southern DenmarkOdenseDenmark
  2. 2.Research Group Quantitative Psychology and Individual DifferencesK.U.LeuvenBelgium

Personalised recommendations