Abstract
The performance of stochastic optimisers can be assessed experimentally on given problems by performing multiple optimisation runs, and analysing the results. Since an optimiser may be viewed as an estimator for the (Pareto) minimum of a (vector) function, stochastic optimiser performance is discussed in the light of the criteria applicable to more usual statistical estimators. Multiobjective optimisers are shown to deviate considerably from standard point estimators, and to require special statistical methodology. The attainment function is formulated, and related results from random closed-set theory are presented, which cast the attainment function as a mean-like measure for the outcomes of multiobjective optimisers. Finally, a covariance-measure is defined, which should bring additional insight into the stochastic behaviour of multiobjective optimisers. Computational issues and directions for further work are discussed at the end of the paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cressie, N. A. C. (1993). Statistics for Spatial Data. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, New York, revised edition.
Embrechts, P., Klüppelberg, C., and Mikosch, T. (1997). Modelling Extremal Events. Springer-Verlag, Berlin.
Fonseca, C. M. and Fleming, P. J. (1995). An overview of evolutionary algorithms in multiobjective optimization. Evolutionary Computation, 3(1):1–16.
Fonseca, C. M. and Fleming, P. J. (1996). On the performance assessment and comparison of stochastic multiobjective optimizers. In Voigt, H.-M., Ebeling, W., Rechenberg, I., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature-PPSN IV, number 1141 in Lecture Notes in Computer Science, pages 584–593. Springer Verlag, Be
Good, P. I. (2000). Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer Series in Statistics. Springer Verlag, New York, 2nd edition.
Goutsias, J. (1998). Modeling random shapes: An introduction to random closed set theory. Technical Report JHU/ECE 90-12, Department of Electrical and Computer Engineering, Image Analysis and Comunnications Laboratory, The John Hopkins University, Baltimore, MD 21218.
Hoos, H. and Stutzle, T. (1998). Evaluating Las Vegas algorithms-pitfalls and remedies. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pages 238–245.
Justel, A., Peña, D., and Zamar, R. (1997). A multivariate Kolmogorov-Smirnov test of goodness of fit. Statistics and Probability Letters, 35:251–259.
Kendall, D. G. (1974). Foundations of a theory of random sets. In Harding, E. F. and Kendall, D. G., editors, Stochastic Geometry. A Tribute to the Memory of Rollo Davidson, pages 322–376. John Wiley & Sons, New York.
Knowles, J. D. and Corne, D. W. (2000). Approximating the nondominated front using the Pareto Archived Evolution Strategy. IEEE Transactions on Evolutionary Computation, 8(2):149–172.
Lockhart, R. and Stephens, M. (1994). Estimation and tests of fit for the three-parameter Weibull distribution. Journal of the Royal Statistical Society, Series B, 56(3):491–500.
Matheron, G. (1975). Random Sets and Integral Geometry. John Wiley & Sons, New York.
Mood, A. M., Graybill, F. A., and Boes, D. C. (1974). Introduction to the Theory of Statistics. McGraw-Hill Series in Probability and Statistics. McGraw-Hill Book Company, Singapore, 3rd edition.
Shaw, K. J., Fonseca, C. M., Nortcliffe, A. L., Thompson, M., Love, J., and Fleming, P. J. (1999). Assessing the performance of multiobjective genetic algorithms for optimization of a batch process scheduling problem. In Proceedings of the Congress on Evolutionary Computation (CEC99), volume 1, pages 37–45, Washington DC.
Smith, R. (1987). Estimating tails of probability distributions. The Annals of Statistics, 15(3):1174–1207.
Stoyan, D. (1998). Random sets: Models and statistics. International Statistical Review, 66:10–27.
Van Veldhuizen, D. and Lamont, G. B. (2000). On measuring multiobjective evolutionary algorithm performance. In Proceedings of the 2000 Congress on Evolutionary Computation, pages 204–211.
Witting, H. (1985). Mathematische Statistik I. B. G. Teubner, Stuttgart.
Zitzler, E. (1999). Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications. PhD thesis, Computer Engineering and Net-works Laboratory, Swiss Federal Institute of Technology Zurich.
Zitzler, E., Deb, K., and Thiele, L. (1999). Comparison of multiobjective evolutionary algorithms: Empirical results (Revised version). Technical Report 70, Computer Engineering and Networks Laboratory, Swiss Federal Institute of Technology Zurich.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grunert da Fonseca, V., Fonseca, C.M., Hall, A.O. (2001). Inferential Performance Assessment of Stochastic Optimisers and the Attainment Function. In: Zitzler, E., Thiele, L., Deb, K., Coello Coello, C.A., Corne, D. (eds) Evolutionary Multi-Criterion Optimization. EMO 2001. Lecture Notes in Computer Science, vol 1993. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44719-9_15
Download citation
DOI: https://doi.org/10.1007/3-540-44719-9_15
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-41745-3
Online ISBN: 978-3-540-44719-1
eBook Packages: Springer Book Archive