Journal of Heuristics

, 15:617 | Cite as

A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization

  • Salvador García
  • Daniel Molina
  • Manuel Lozano
  • Francisco Herrera
Article

Abstract

In recent years, there has been a growing interest for the experimental analysis in the field of evolutionary algorithms. It is noticeable due to the existence of numerous papers which analyze and propose different types of problems, such as the basis for experimental comparisons of algorithms, proposals of different methodologies in comparison or proposals of use of different statistical techniques in algorithms’ comparison.

In this paper, we focus our study on the use of statistical techniques in the analysis of evolutionary algorithms’ behaviour over optimization problems. A study about the required conditions for statistical analysis of the results is presented by using some models of evolutionary algorithms for real-coding optimization. This study is conducted in two ways: single-problem analysis and multiple-problem analysis. The results obtained state that a parametric statistical analysis could not be appropriate specially when we deal with multiple-problem results. In multiple-problem analysis, we propose the use of non-parametric statistical tests given that they are less restrictive than parametric ones and they can be used over small size samples of results. As a case study, we analyze the published results for the algorithms presented in the CEC’2005 Special Session on Real Parameter Optimization by using non-parametric test procedures.

Keywords

Statistical analysis of experiments Evolutionary algorithms Parametric tests Non-parametric tests 

References

  1. Abramowitz, M.: Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables. Dover, New York (1974) Google Scholar
  2. Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1769–1776 (2005a) Google Scholar
  3. Auger, A., Hansen, N.: Performance evaluation of an advanced local search evolutionary algorithm. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1777–1784 (2005b) Google Scholar
  4. Ballester, P.J., Stephenson, J., Carter, J.N., Gallagher, K.: Real-parameter optimization performance study on the CEC-2005 benchmark with SPC-PNX. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 498–505 (2005) Google Scholar
  5. Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: The New Experimentalism. Springer, New York (2006) MATHGoogle Scholar
  6. Czarn, A., MacNish, C., Vijayan, K., Turlach, R., Gupta, R.: Statistical exploratory analysis of genetic algorithms. IEEE Trans. Evol. Comput. 8(4), 405–421 (2004) CrossRefGoogle Scholar
  7. Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006) MathSciNetGoogle Scholar
  8. Gallagher, M., Yuan, B.: A general-purpose tunable landscape generator. IEEE Trans. Evol. Comput. 10(5), 590–603 (2006) CrossRefGoogle Scholar
  9. García, S., Molina, D., Lozano, M., Herrera, F.: An experimental study on the use of non-parametric tests for analyzing the behaviour of evolutionary algorithms in optimization problems. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 275–285 (2007) (in Spanish) Google Scholar
  10. García-Martínez, C., Lozano, M.: Hybrid real-coded genetic algorithms with female and male differentiation. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 896–903 (2005) Google Scholar
  11. Hansen, N.: (2005). Compilation of results on the CEC benchmark function set. Tech. Report, Institute of Computational Science, ETH Zurich, Switzerland. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/compareresults.pdf
  12. Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988) MATHCrossRefMathSciNetGoogle Scholar
  13. Holm, S.: A simple sequentially rejective multiple test procedure. Scandinavian J. Statist. 6, 65–70 (1979) MATHMathSciNetGoogle Scholar
  14. Hooker, J.: Testing heuristics: we have it all wrong. J. Heuristics 1(1), 33–42 (1997) CrossRefMathSciNetGoogle Scholar
  15. Iman, R.L., Davenport, J.M.: Approximations of the critical region of the Friedman statistic. Commun. Stat. 18, 571–595 (1980) CrossRefGoogle Scholar
  16. Kramer, O.: An experimental analysis of evolution strategies and particle swarm optimisers using design of experiments. In: Proceedings of the Genetic and Evolutionary Computation Conference 2007 (GECCO’2007), pp. 674–681 (2007) Google Scholar
  17. Liang, J.J., Suganthan, P.N.: Dynamic multi-swarm particle swarm optimizer with local search. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 522–528 (2005) Google Scholar
  18. Molina, D., Herrera, F., Lozano, M.: Adaptive local search parameters for real-coded memetic algorithms. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 888–895 (2005) Google Scholar
  19. Moreno-Pérez, J.A., Campos-Rodríguez, C., Laguna, M.: On the comparison of metaheuristics through non-parametric statistical techniques. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 286–293 (2007) (in Spanish) Google Scholar
  20. Morse, D.T.: Minsize2: a computer program for determining effect size and minimum sample size for statistical significance for univariate, multivariate, and nonparametric tests. Educ. Psychol. Meas. 59(3), 518–531 (1999) CrossRefGoogle Scholar
  21. Noether, G.E.: Sample size determination for some common nonparametric tests. J. Am. Stat. Assoc. 82(398), 645–647 (1987) MATHCrossRefMathSciNetGoogle Scholar
  22. Ortiz-Boyer, D., Hervás-Martínez, C., García-Pedrajas, N.: Improving crossover operators for real-coded genetic algorithms using virtual parents. J. Heuristics 13, 265–314 (2007) CrossRefGoogle Scholar
  23. Ozcelik, B., Erzurumlu, T.: Comparison of the warpage optimization in the plastic injection molding using ANOVA, neural network model and genetic algorithm. J. Mater. Process. Technol. 171(3), 437–445 (2006) CrossRefGoogle Scholar
  24. Patel, J.K., Read, C.B.: Handbook of the Normal Distribution. Dekker, New York (1982) MATHGoogle Scholar
  25. Pošík, P.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 872–879 (2005) Google Scholar
  26. Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1785–1791 (2005) Google Scholar
  27. Rojas, I., González, J., Pomares, H., Merelo, J.J., Castillo, P.A., Romero, G.: Statistical analysis of the main parameters involved in the design of a genetic algorithm. IEEE Trans. Syst. Man Cybern. Part C 32(1), 31–37 (2002) CrossRefGoogle Scholar
  28. Rônkkônen, J., Kukkonen, S., Price, K.V.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 506–513 (2005) Google Scholar
  29. Shaffer, J.P.: Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561–584 (1995) CrossRefGoogle Scholar
  30. Sinha, A., Tiwari, S., Deb, K.: A population-based, steady-state procedure for real-parameter optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 514–521 (2005) Google Scholar
  31. Suganthan, P.N., Hansen, N., Liang, J.J., Deb, K., Chen, Y.P., Auger, A., Tiwari, S.: (2005). Problem definitions and evaluation criteria for the CEC 2005 Special Session on Real Parameter Optimization. Tech. Report, Nanyang Technological University. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/Tech-Report-May-30-05.pdf
  32. Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, Boca Raton (2003) Google Scholar
  33. Whitley, D.L., Beveridge, R., Graves, C., Mathias, K.E.: Test driving three 1995 genetic algorithms: new test functions and geometric matching. J. Heuristics 1(1), 77–104 (1995) MATHCrossRefGoogle Scholar
  34. Whitley, D.L., Rana, S., Dzubera, J., Mathias, K.E.: Evaluating evolutionary algorithms. Artif. Intell. 85(1–2), 245–276 (1996) CrossRefGoogle Scholar
  35. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997) CrossRefGoogle Scholar
  36. Wright, S.P.: Adjusted p-values for simultaneous inference. Biometrics 48, 1005–1013 (1992) CrossRefGoogle Scholar
  37. Yuan, B., Gallagher, M.: On building a principled framework for evaluating and testing evolutionary algorithms: a continuous landscape generator. In: Proceedings of the 2003 Congress on Evolutionary Computation (CEC2003), pp. 451–458 (2003) Google Scholar
  38. Yuan, B., Gallagher, M.: Experimental results for the Special Session on Real-Parameter Optimization at CEC 2005: a simple, continuous EDA. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1792–1799 (2005) Google Scholar
  39. Zar, J.H.: Biostatistical Analysis. Prentice Hall, Englewood Cliffs (1999) Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Salvador García
    • 1
  • Daniel Molina
    • 2
  • Manuel Lozano
    • 1
  • Francisco Herrera
    • 1
  1. 1.Department of Computer Science and Artificial IntelligenceUniversity of GranadaGranadaSpain
  2. 2.Department of Computer EngineeringUniversity of CádizCádizSpain

Personalised recommendations