Analysing and characterising optimization problems using length scale

Abstract

Analysis of optimization problem landscapes is fundamental in the understanding and characterisation of problems and the subsequent practical performance of algorithms. In this paper, a general framework is developed for characterising black-box optimization problems based on length scale, which measures the change in objective function with respect to the distance between candidate solution pairs. Both discrete and continuous problems can be analysed using the framework, however, in this paper, we focus on continuous optimization. Length scale analysis aims to efficiently and effectively utilise the information available in black-box optimization. Analytical properties regarding length scale are discussed and illustrated using simple example problems. A rigorous sampling methodology is developed and demonstrated to improve upon current practice. The framework is applied to the black-box optimization benchmarking problem set, and shows greater ability to discriminate between the problems in comparison to seven well-known landscape analysis techniques. Dimensionality reduction and clustering techniques are applied comparatively to an ensemble of the seven techniques and the length scale information to gain insight into the relationships between problems. A fundamental summary of length scale information is an estimate of its probability density function, which is shown to capture salient structural characteristics of the benchmark problems.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional spaces. In: Proceedings of the 8th international conference on database theory. Springer, London, pp 420–434

  2. Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (2010) Experimental methods for the analysis of optimization algorithms. Springer, New York

    Google Scholar 

  3. Beliakov G (2006) Interpolation of lipschitz functions. J Comput Appl Math 196(1):20–44

    MathSciNet  Article  MATH  Google Scholar 

  4. Beyer KS, Goldstein J, Ramakrishnan R, Shaft U (1999) When is ”nearest neighbor” meaningful? In: Proceedings of the 7th international conference on database theory. Springer, London, pp 217–235

  5. Borenstein Y, Poli R (2006) Kolmogorov complexity, optimization and hardness. In: IEEE congress on evolutionary computation (CEC 2006), pp 112–119

  6. Cheeseman P, Kanefsky B, Taylor W (1991) Where the really hard problems are. In: Proceedings of 12th international joint conference on AI, Morgan Kauffman, pp 331–337

  7. Collard P, Vérel S, Clergue M (2004) Local search heuristics: fitness cloud versus fitness landscape. In: The 2004 European conference on artificial intelligence, IOS Press, pp 973–974

  8. Cover TM, Thomas JA (1991) Elements of information theory. Wiley, New York

    Google Scholar 

  9. Fletcher R (1987) Practical methods of optimization, 2nd edn. Wiley, Hoboken

    Google Scholar 

  10. Forrester A, Sóbester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, Hoboken

    Google Scholar 

  11. Gallagher M (2000) Multi-layer perceptron error surfaces: visualization, structure and modelling. PhD thesis, Department of Computer Science and Electrical Engineering, University of Queensland

  12. Gallagher M (2001) Fitness distance correlation of neural network error surfaces: a scalable, continuous optimization problem. In: Raedt LD, Flach P (eds) European conference on machine learning, Singapore, Lecture notes in artificial intelligence, vol 2167, pp 157–166

  13. Gallagher M, Downs T, Wood I (2002) Empirical evidence for ultrametric structure in multi-layer perceptron error surfaces. Neural Process Lett 16(2):177–186

    Article  MATH  Google Scholar 

  14. Gent I, Walsh T (1996) The TSP phase transition. Artif Intell 88(1–2):349–358

    MathSciNet  Article  MATH  Google Scholar 

  15. Grinstead CM, Snell JL (2012) Introduction to probability. American Mathematical Society, Providence

    Google Scholar 

  16. Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182

    MATH  Google Scholar 

  17. Hansen N (2000) Invariance, self-adaptation and correlated mutations in evolution strategies. In: Schoenauer et al M (ed) Parallel problem solving from nature—PPSN VI. Lecture notes in computer science, vol 1917, Springer, pp 355–364

  18. Hansen N, Finck S, Ros R, Auger A (2010) Real-parameter black-box optimization benchmarking 2010: noiseless functions definitions. Technical Report, RR-6829, INRIA

  19. Horst R, Tuy H (1996) Global optimization: deterministic approaches. Springer, New Yotk

    Google Scholar 

  20. Hutter F, Hamadi Y, Hoos H, Leyton-Brown K (2006) Performance prediction and automated tuning of randomized and parametric algorithms. In: Benhamou F (ed) Principles and practice of constraint programming. Lecture notes in computer science, vol 4204, Springer, pp 213–228

  21. Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9(1):3–12

    Article  Google Scholar 

  22. Jones T, Forrest S (1995) Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In: Proceedings of the 6th international conference on genetic algorithms. Morgan Kaufmann, San Francisco, CA, pp 184–192

  23. Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86

    MathSciNet  Article  MATH  Google Scholar 

  24. Lee JA, Verleysen M (2007) Nonlinear dimensionality reduction. Springer, New York

    Google Scholar 

  25. Lunacek M, Whitley D (2006) The dispersion metric and the CMA evolution strategy. In: Proceedings of the 8th annual conference on genetic and evolutionary computation. ACM, New York, pp 477–484

  26. Macready W, Wolpert D (1996) What makes an optimization problem hard. Complexity 5:40–46

    MathSciNet  Article  Google Scholar 

  27. Malan K, Engelbrecht A (2009) Quantifying ruggedness of continuous landscapes using entropy. In: IEEE congress on evolutionary computation, pp 1440–1447

  28. Mersmann O, Bischl B, Trautmann H, Preuss M, Weihs C, Rudolph G (2011) Exploratory landscape analysis. In: Proceedings of the 13th annual conference on genetic and evolutionary computation. ACM, New York, pp 829–836

  29. Morgan R, Gallagher M (2012) Length scale for characterising continuous optimization problems. In: Coello et al CAC (ed) Parallel problem solving from nature—PPSN XII. Lecture notes in computer science, vol 7491, Springer, pp 407–416

  30. Morgan R, Gallagher M (2014) Sampling techniques and distance metrics in high dimensional continuous landscape analysis: limitations and improvements. IEEE Trans Evol Comput 18(3):456–461

    Article  Google Scholar 

  31. Muñoz MA, Kirley M, Halgamuge S (2012a) A meta-learning prediction model of algorithm performance for continuous optimization problems. In: Coello et al CAC (ed) Parallel problem solving from nature—PPSN XII. Lecture notes in computer science, vol 7491, Springer, pp 226–235

  32. Muñoz MA, Kirley M, Halgamuge SK (2012b) Landscape characterization of numerical optimization problems using biased scattered data. In: IEEE congress on evolutionary computation, pp 1180–1187

  33. Müller C, Baumgartner B, Sbalzarini I (2009) Particle swarm CMA evolution strategy for the optimization of multi-funnel landscapes. In: IEEE congress on evolutionary computation, pp 2685–2692

  34. Müller CL, Sbalzarini IF (2011) Global characterization of the CEC 2005 fitness landscapes using fitness-distance analysis. In: Proceedings of the 2011 international conference on applications of evolutionary computation. vol Part I. Springer, Berlin, Heidelberg, pp 294–303

  35. Overton M (2001) Numerical computing with IEEE floating point arithmetic. Cambridge University Press, Cambridge

    Google Scholar 

  36. Pitzer E, Affenzeller M (2012) A comprehensive survey on fitness landscape analysis. In: Fodor J, Klempous R, Suárez Araujo C (eds) Recent advances in intelligent engineering systems, studies in computational intelligence. Springer, New York, pp 161–191

    Google Scholar 

  37. Pitzer E, Affenzeller M, Beham A, Wagner S (2012) Comprehensive and automatic fitness landscape analysis using heuristiclab. In: Moreno-Díaz R, Pichler F, Quesada-Arencibia A (eds) Computer aided systems theory–EUROCAST 2011. Lecture notes in computer science, vol 6927, Springer, pp 424–431

  38. Reidys C, Stadler P (2002) Combinatorial landscapes. SIAM Rev 44(1):3–54

    MathSciNet  Article  MATH  Google Scholar 

  39. Rice JR (1976) The algorithm selection problem. Adv Comput 15:65–118

    Article  Google Scholar 

  40. Ridge E, Kudenko D (2007) An analysis of problem difficulty for a class of optimisation heuristics. In: Proceedings of the 7th european conference on evolutionary computation in combinatorial optimization, Springer, pp 198–209

  41. Rosé H, Ebeling W, Asselmeyer T (1996) The density of states—a measure of the difficulty of optimisation problems. In: Voigt et al HM (ed) Parallel problem solving from nature PPSN IV. Lecture notes in computer science, vol 1141, Springer, pp 208–217

  42. Rosen K (1999) Handbook of discrete and combinatorial mathematics, 2nd edn., Discrete mathematics and its applicationsTaylor & Francis, Routledge

    Google Scholar 

  43. Sergeyev YD, Kvasov DE (2010) Lipschitz global optimization. Wiley Encycl Oper Res Manag Sci 4:2812–2828

    Google Scholar 

  44. Sheather S, Jones M (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B (Methodological) 53:683–690

    MathSciNet  MATH  Google Scholar 

  45. Shlesinger MF, West BJ, Klafter J (1987) Lévy dynamics of enhanced diffusion: application to turbulence. Phys Rev Lett 58:1100–1103

    MathSciNet  Article  Google Scholar 

  46. Smith T, Husbands P, O’Shea M (2002) Fitness landscapes and evolvability. Evol Comput 10(1):1–34

    Article  Google Scholar 

  47. Smith-Miles K (2008) Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput Surv 41(1):1–25

    Article  Google Scholar 

  48. Smith-Miles K, Lopes L (2011) Measuring instance difficulty for combinatorial optimization problems. Comput Oper Res 39(5):875–889

    MathSciNet  Article  MATH  Google Scholar 

  49. Smith-Miles K, Tan TT (2012) Measuring algorithm footprints in instance space. In: IEEE congress on evolutionary computation (CEC), pp 1–8

  50. Solla SA, Sorkin GB, White SR (1986) Configuration space analysis for optimization problems. In: et al EB (ed) Disordered systems and biological organization, NATO ASI Series, vol F20, Springer, Berlin, New York, pp 283–293

  51. Stadler PF (1996) Landscapes and their correlation functions. J Math Chem 20:1–45

    MathSciNet  Article  MATH  Google Scholar 

  52. Stadler PF (2002) Fitness landscapes. In: Lässig M, Valleriani A (eds) Biological evolution and statistical physics. Lecture notes in physics, vol 585, Springer, pp 183–204

  53. Stadler PF, Schnabl W (1992) The landscape of the traveling salesman problem. Phys Lett A 161(4):337–344

    MathSciNet  Article  MATH  Google Scholar 

  54. Steer K, Wirth A, Halgamuge S (2008) Information theoretic classification of problems for metaheuristics. In: Li et al X (ed) Simulated evolution and learning, Lecture notes in computer science, vol 5361, Springer, pp 319–328

  55. Strongin R (1973) On the convergence of an algorithm for finding a global extremum. Eng Cybern 11:549–555

    MathSciNet  Google Scholar 

  56. Talbi E (2009) Metaheuristics: from design to implementation., Wiley series on parallel and distributed computingWiley, Hoboken

    Google Scholar 

  57. Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11):2579–2605

    MATH  Google Scholar 

  58. van Hemert J (2005) Property analysis of symmetric travelling salesman problem instances acquired through evolution. Evol Comput Comb Optim 3448:122–131

    MATH  Google Scholar 

  59. Vassilev VK, Fogarty TC, Miller JF (2000) Information characteristics and the structure of landscapes. Evol Comput 8:31–60

    Article  Google Scholar 

  60. Wang Y, Li B (2008) Understand behavior and performance of real coded optimization algorithms via nk-linkage model. In: IEEE world congress on computational intelligence, pp 801–808

  61. Weinberger E (1990) Correlated and uncorrelated fitness landscapes and how to tell the difference. Biol Cybern 63:325–336

    Article  MATH  Google Scholar 

  62. Whitley D, Watson JP (2005) Complexity theory and the no free lunch theorem. In: Search methodologies, Springer, pp 317–339

  63. Whitley D, Lunacek M, Sokolov A (2006) Comparing the niches of CMA-ES, CHC and pattern search using diverse benchmarks. In: Runarsson et al TP (ed) Parallel problem solving from nature—PPSN IX. Lecture notes in computer science, vol 4193, Springer, pp 988–997

  64. Wood GR, Zhang BP (1996) Estimation of the Lipschitz constant of a function. J Glob Optim 8:91–103

    MathSciNet  Article  MATH  Google Scholar 

  65. Zhang W (2004) Phase transitions and backbones of the asymmetric traveling salesman problem. J Artif Intell Res 21(1):471–497

    MathSciNet  MATH  Google Scholar 

  66. Zhang W, Korf RE (1996) A study of complexity transitions on the asymmetric traveling salesman problem. Artif Intell 81(1–2):223–239

    MathSciNet  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Rachael Morgan.

Ethics declarations

Conflicts of interest

None.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Morgan, R., Gallagher, M. Analysing and characterising optimization problems using length scale. Soft Comput 21, 1735–1752 (2017). https://doi.org/10.1007/s00500-015-1878-z

Download citation

Keywords

  • Length scale
  • Fitness landscape analysis
  • Black-box optimization
  • Search space diagnostics