Inferential Performance Assessment of Stochastic Optimisers and the Attainment Function

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1993)


The performance of stochastic optimisers can be assessed experimentally on given problems by performing multiple optimisation runs, and analysing the results. Since an optimiser may be viewed as an estimator for the (Pareto) minimum of a (vector) function, stochastic optimiser performance is discussed in the light of the criteria applicable to more usual statistical estimators. Multiobjective optimisers are shown to deviate considerably from standard point estimators, and to require special statistical methodology. The attainment function is formulated, and related results from random closed-set theory are presented, which cast the attainment function as a mean-like measure for the outcomes of multiobjective optimisers. Finally, a covariance-measure is defined, which should bring additional insight into the stochastic behaviour of multiobjective optimisers. Computational issues and directions for further work are discussed at the end of the paper.


Pareto Front Stochastic Behaviour Objective Vector Multiobjective Genetic Algorithm Empirical Cumulative Distribution Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Cressie, N. A. C. (1993). Statistics for Spatial Data. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, New York, revised edition.Google Scholar
  2. Embrechts, P., Klüppelberg, C., and Mikosch, T. (1997). Modelling Extremal Events. Springer-Verlag, Berlin.zbMATHGoogle Scholar
  3. Fonseca, C. M. and Fleming, P. J. (1995). An overview of evolutionary algorithms in multiobjective optimization. Evolutionary Computation, 3(1):1–16.CrossRefGoogle Scholar
  4. Fonseca, C. M. and Fleming, P. J. (1996). On the performance assessment and comparison of stochastic multiobjective optimizers. In Voigt, H.-M., Ebeling, W., Rechenberg, I., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature-PPSN IV, number 1141 in Lecture Notes in Computer Science, pages 584–593. Springer Verlag, BeCrossRefGoogle Scholar
  5. Good, P. I. (2000). Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer Series in Statistics. Springer Verlag, New York, 2nd edition.zbMATHGoogle Scholar
  6. Goutsias, J. (1998). Modeling random shapes: An introduction to random closed set theory. Technical Report JHU/ECE 90-12, Department of Electrical and Computer Engineering, Image Analysis and Comunnications Laboratory, The John Hopkins University, Baltimore, MD 21218.Google Scholar
  7. Hoos, H. and Stutzle, T. (1998). Evaluating Las Vegas algorithms-pitfalls and remedies. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pages 238–245.Google Scholar
  8. Justel, A., Peña, D., and Zamar, R. (1997). A multivariate Kolmogorov-Smirnov test of goodness of fit. Statistics and Probability Letters, 35:251–259.zbMATHCrossRefMathSciNetGoogle Scholar
  9. Kendall, D. G. (1974). Foundations of a theory of random sets. In Harding, E. F. and Kendall, D. G., editors, Stochastic Geometry. A Tribute to the Memory of Rollo Davidson, pages 322–376. John Wiley & Sons, New York.Google Scholar
  10. Knowles, J. D. and Corne, D. W. (2000). Approximating the nondominated front using the Pareto Archived Evolution Strategy. IEEE Transactions on Evolutionary Computation, 8(2):149–172.Google Scholar
  11. Lockhart, R. and Stephens, M. (1994). Estimation and tests of fit for the three-parameter Weibull distribution. Journal of the Royal Statistical Society, Series B, 56(3):491–500.zbMATHMathSciNetGoogle Scholar
  12. Matheron, G. (1975). Random Sets and Integral Geometry. John Wiley & Sons, New York.zbMATHGoogle Scholar
  13. Mood, A. M., Graybill, F. A., and Boes, D. C. (1974). Introduction to the Theory of Statistics. McGraw-Hill Series in Probability and Statistics. McGraw-Hill Book Company, Singapore, 3rd edition.zbMATHGoogle Scholar
  14. Shaw, K. J., Fonseca, C. M., Nortcliffe, A. L., Thompson, M., Love, J., and Fleming, P. J. (1999). Assessing the performance of multiobjective genetic algorithms for optimization of a batch process scheduling problem. In Proceedings of the Congress on Evolutionary Computation (CEC99), volume 1, pages 37–45, Washington DC.Google Scholar
  15. Smith, R. (1987). Estimating tails of probability distributions. The Annals of Statistics, 15(3):1174–1207.zbMATHMathSciNetCrossRefGoogle Scholar
  16. Stoyan, D. (1998). Random sets: Models and statistics. International Statistical Review, 66:10–27.CrossRefGoogle Scholar
  17. Van Veldhuizen, D. and Lamont, G. B. (2000). On measuring multiobjective evolutionary algorithm performance. In Proceedings of the 2000 Congress on Evolutionary Computation, pages 204–211.Google Scholar
  18. Witting, H. (1985). Mathematische Statistik I. B. G. Teubner, Stuttgart.zbMATHGoogle Scholar
  19. Zitzler, E. (1999). Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications. PhD thesis, Computer Engineering and Net-works Laboratory, Swiss Federal Institute of Technology Zurich.Google Scholar
  20. Zitzler, E., Deb, K., and Thiele, L. (1999). Comparison of multiobjective evolutionary algorithms: Empirical results (Revised version). Technical Report 70, Computer Engineering and Networks Laboratory, Swiss Federal Institute of Technology Zurich.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  1. 1.Instituto de Sistemas e Robótica CoimbraPortugal
  2. 2.ADEEC, UCEUniversidade do Algarve FaroPortugal
  3. 3.Departamento de MatemáticaUniversidade de Aveiro AveiroPortugal

Personalised recommendations