Applied Intelligence

, Volume 47, Issue 2, pp 319–330 | Cite as

Adaptive pattern search for large-scale optimization

  • Vincent Gardeux
  • Mahamed G. H. Omran
  • Rachid Chelouah
  • Patrick Siarry
  • Fred Glover
Article
  • 155 Downloads

Abstract

The emergence of high-dimensional data requires the design of new optimization methods. Indeed, conventional optimization methods require improvements, hybridization, or parameter tuning in order to operate in spaces of high dimensions. In this paper, we present a new adaptive variant of a pattern search algorithm to solve global optimization problems exhibiting such a character. The proposed method has no parameters visible to the user and the default settings, determined by almost no a priori experimentation, are highly robust on the tested datasets. The algorithm is evaluated and compared with 11 state-of-the-art methods on 20 benchmark functions of 1000 dimensions from the CEC’2010 competition. The results show that this approach obtains good performances compared to the other methods tested.

Keywords

Pattern search Scatter search Optimization Continuous High-dimension Large-scale Adaptive methods 

Supplementary material

10489_2017_901_MOESM1_ESM.xls (621 kb)
(XLS 621 KB)
10489_2017_901_MOESM2_ESM.xlsx (43 kb)
(XLSX 42.6 KB)
10489_2017_901_MOESM3_ESM.pdf (180 kb)
(PDF 179 KB)

References

  1. 1.
    Olariu S, Zomaya AY (2005) Handbook of bioinspired algorithms and applications. Chapman & Hall/CRC, LondonCrossRefMATHGoogle Scholar
  2. 2.
    Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of IEEE international conference on neural networks. PerthGoogle Scholar
  3. 3.
    Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., p 372Google Scholar
  4. 4.
    Bellman R (1957) Dynamic programming. Princeton University Press, PrincetonMATHGoogle Scholar
  5. 5.
    Lee EK (2007) Large-scale optimization-based classification models in medicine and biology. Ann Biomed Eng 35(6):1095–1 1109CrossRefGoogle Scholar
  6. 6.
    Nasiri JA et al (2009) High dimensional problem optimization using distributed multi-agent PSO. In: Third UKSim European symposium on computer modeling and simulation, 2009. EMS ’09Google Scholar
  7. 7.
    Larranaga P et al (2006) Machine learning in bioinformatics. Brief Bioinform 7(1):86–112MathSciNetCrossRefGoogle Scholar
  8. 8.
    Levitsky V et al (2007) Effective transcription factor binding site prediction using a combination of optimization, a genetic algorithm and discriminant analysis to capture distant interactions. BMC Bioinform 8(481):1–20Google Scholar
  9. 9.
    Saeys Y, Inza I, Larrañaga P (2007) A review of feature selection techniques in bioinformatics. Bioinformatics 23(19):2507–2517CrossRefGoogle Scholar
  10. 10.
    Ghalwash MF et al (2016) Structured feature selection using coordinate descent optimization. BMC Bioinform 17:158CrossRefGoogle Scholar
  11. 11.
    Blanco R, Larrañaga P (2001) Selection of highly accurate genes for cancer classification by estimation of distribution algorithms. in: Workshop of Bayesian models in medicine. AIME 2001. 1–4 July. CascaisGoogle Scholar
  12. 12.
    Saeys Y et al (2004) Feature selection for splice site prediction: a new method using EDA-based feature ranking. BMC Bioinform 5(64):1–11Google Scholar
  13. 13.
    Armananzas R et al (2008) A review of estimation of distribution algorithms in bioinformatics. BioData Mining 1(6):1–12Google Scholar
  14. 14.
    Dittrich M et al (2008) Identifying functional modules in protein-protein interaction networks: an integrated exact approach. Bioinformatics 24(13):I223–I231CrossRefGoogle Scholar
  15. 15.
    Xiao X et al (2003) Gene clustering using self-organizing maps and particle swarm optimization. In: Parallel and distributed processing symposium, 22–26 April. IEEE Computer SocietyGoogle Scholar
  16. 16.
    Maulik U, Bandyopadhyay S, Mukhopadhyay A (2011) Multiobjective genetic algorithms for clustering: applications in data mining and bioinformatics. Springer Science & Business MediaGoogle Scholar
  17. 17.
    Gardeux V et al (2013) Optimization for feature selection in DNA microarrays. In: Heuristics: theory and applications. Nova PublishersGoogle Scholar
  18. 18.
    Handl J, Kell D, Knowles J (2007) Multiobjective optimization in bioinformatics and computational biology. IEEE/ACM Trans Comput Biol Bioinform 4(2):279–292CrossRefGoogle Scholar
  19. 19.
    Shan S, Wang GG (2010) Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multidiscip Optim 41(2):219–241MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Regis R (2013) An initialization strategy for high-dimensional surrogate-based expensive black-box optimization. In: Zuluaga LF, Terlaky T (eds) Modeling and optimization: theory and applications. Springer, New York, pp 51–85CrossRefGoogle Scholar
  21. 21.
    Hvattum LM, Glover F (2009) Finding local optima of high-dimensional functions using direct search methods. Eur J Oper Res 195(1):31–45MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    LaTorre A, Muelas S, Pena JM (2011) A MOS-based dynamic memetic differential evolution algorithm for continuous optimization: a scalability test. Soft Comput 15(11):2187–2199CrossRefGoogle Scholar
  23. 23.
    Wang H, Wu ZJ, Rahnamayan S (2011) Enhanced opposition-based differential evolution for solving high-dimensional continuous optimization problems. Soft Comput 15(11):2127–2140CrossRefGoogle Scholar
  24. 24.
    Yang ZY, Tang K, Yao X (2011) Scalability of generalized adaptive differential evolution for large-scale continuous optimization. Soft Comput 15(11):2141–2155CrossRefGoogle Scholar
  25. 25.
    Zhao S-Z, Suganthan PN, Das S (2010) Self-adaptive differential evolution with modified multi-trajectory search for CEC’2010 large scale optimization. In: Swarm, evolutionary, and memetic computing. Springer, Berlin, pp 1–10Google Scholar
  26. 26.
    Hedar A-R, Ali A (2012) Tabu search with multi-level neighborhood structures for high dimensional problems. Appl Intell 37(2):189–206CrossRefGoogle Scholar
  27. 27.
    Stanarevic N (2012) Hybridizing artificial bee colony (ABC) algorithm with differential evolution for large scale optimization problems. Int J Math Comput Simul 6(1):194–202Google Scholar
  28. 28.
    You X (2010) Differential evolution with a new mutation operator for solving high dimensional continuous optimization problems. J Comput Inf Syst 6(9):3033–3039Google Scholar
  29. 29.
    Ros R, Hansen N (2008) A simple modification in CMA-ES achieving linear time and space complexity. In: Rudolph G et al (eds) Parallel problem solving from nature—PPSN X. Springer, Berlin, pp 296–305CrossRefGoogle Scholar
  30. 30.
    Liao T, Montes de Oca MA (2011) Tuning parameters across mixed dimensional instances: a performance scalability study of Sep-G-CMA-ES. In: Proceedings of the 13th annual conference companion on genetic and evolutionary computation. ACM, Dublin, pp 703–706Google Scholar
  31. 31.
    Montes de Oca MA, Aydın D, Stützle T (2011) An incremental particle swarm for large-scale continuous optimization problems: an example of tuning-in-the-loop (re)design of optimization algorithms. Soft Comput 15 (11):2233–2255CrossRefGoogle Scholar
  32. 32.
    Masegosa AD, Pelta DA, Verdegay JL (2013) A centralised cooperative strategy for continuous optimisation: the influence of cooperation in performance and behaviour. Inf Sci 219(0):73–92MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Li X, Yao X (2012) Cooperatively coevolving particle swarms for large scale optimization. IEEE Trans Evol Comput 16(2):210–224MathSciNetCrossRefGoogle Scholar
  34. 34.
    Li X et al (2015) Editorial for the special issue of Information Sciences Journal (ISJ) on “Nature-inspired algorithms for large scale global optimization”. Inf Sci 316:437–439CrossRefGoogle Scholar
  35. 35.
    Tsurkov V (2001) Large-scale optimization. Applied optimization. Springer USGoogle Scholar
  36. 36.
    Liu L, Shao L, Li X (2015) Evolutionary compact embedding for large-scale image classification. Inf Sci 316:567–581CrossRefGoogle Scholar
  37. 37.
    Miranda V, Martins J, Palma V (2014) Optimizing large scale problems with metaheuristics in a reduced space mapped by autoencoders-application to the wind-hydro coordination. IEEE Trans Power Syst 29(6):3078–3085CrossRefGoogle Scholar
  38. 38.
    LaTorre A, Muelas S, Pena J (2015) A comprehensive comparison of large scale global optimizers. Inf Sci 316:517–549CrossRefGoogle Scholar
  39. 39.
    Gardeux V et al (2009) Unidimensional search for solving continuous high-dimensional optimization problems. In: Ninth international conference on intelligent systems design and applications. ISDA ’09. November 30–December 2, 2009. IEEE Computer Society, PisaGoogle Scholar
  40. 40.
    Yang X-S, Koziel S (2011) Computational optimization and applications in engineering and industry, vol 359. Springer Science & Business MediaGoogle Scholar
  41. 41.
    Conn AR, Scheinberg K, Vicente LN (2009) Introduction to derivative-free optimization, vol 8. SIAM, PhiladelphiaCrossRefMATHGoogle Scholar
  42. 42.
    Torczon V (1997) On the convergence of pattern search algorithms. SIAM J Optim 7(1):1–25MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    Kolda TG, Lewis RM, Torczon V (2003) Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev 45(3):385–482MathSciNetCrossRefMATHGoogle Scholar
  44. 44.
    Hooke R, Jeeves TA (1961) “Direct search” solution of numerical and statistical problems. J ACM 8 (2):212–229CrossRefMATHGoogle Scholar
  45. 45.
    Lozano M, Molina D, Herrera F (2011) Editorial scalability of evolutionary algorithms and other metaheuristics for large-scale continuous optimization problems. Soft Comput 15(11):2085–2087CrossRefGoogle Scholar
  46. 46.
    Glover F et al (1998) A template for scatter search and path relinking. In: Hao J-K (ed) Artificial evolution. Springer, Berlin, pp 1–51Google Scholar
  47. 47.
    Glover F (1995) Tabu thresholding: improved search by nonmonotonic trajectories. INFORMS J Comput 7 (4):426–442CrossRefMATHGoogle Scholar
  48. 48.
    Gardeux V et al (2011) EM323: a line search based algorithm for solving high-dimensional continuous non-linear optimization problems. Soft Comput 15(11):2275–2285CrossRefGoogle Scholar
  49. 49.
    Kirkpatrick S, Gelatt CD Jr, Vecchi MP (1983) Optimization by simulated annealing. Science 220 (4598):671–680MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Omidvar MN, Li X, Yao X (2010) Cooperative co-evolution with delta grouping for large scale non-separable function optimization. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  51. 51.
    Tang K et al (2010) Benchmark functions for the CEC’2010 special session and competition on large scale global optimization. In: Nature inspired computation and applications laboratory, USTC, China: http://nical.ustc.edu.cn/cec10ss.php
  52. 52.
    Yang Z, Tang K, Yao X (2008) Large scale evolutionary optimization using cooperative coevolution. Inf Sci 178(15):2985–2999MathSciNetCrossRefMATHGoogle Scholar
  53. 53.
    Yang Z, Tang K, Yao X (2008) Multilevel cooperative coevolution for large scale optimization. In: IEEE congress on evolutionary computation (CEC 2008). June 1–6. IEEE Computer Society, Hong KongGoogle Scholar
  54. 54.
    Korosec P, Tashkova K, Silc J (2010) The differential Ant-Stigmergy Algorithm for large-scale global optimization. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  55. 55.
    Wang H et al (2010) Sequential DE enhanced by neighborhood search for large scale global optimization. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  56. 56.
    Wang Y, Li B (2010) Two-stage based ensemble optimization for large-scale global optimization. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  57. 57.
    Molina D, Lozano M, Herrera F (2010) MA-SW-Chains: memetic algorithm based on local search chains for large scale continuous global optimization. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  58. 58.
    Brest J et al (2010) Large scale global optimization using self-adaptive differential evolution algorithm. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  59. 59.
    Zhao S-Z, Suganthan PN, Das S (2010) Dynamic multi-swarm particle swarm optimizer with sub-regional harmony search. In: IEEE congress on evolutionary computation (CEC 2010). 18–23 July. IEEE Computer Society, BarcelonaGoogle Scholar
  60. 60.
    Brest J et al (2008) High-dimensional real-parameter optimization using self-adaptive differential evolution algorithm with population size reduction. In: IEEE congress on evolutionary computation (CEC 2008). June 1–6. IEEE Computer Society, Hong KongGoogle Scholar
  61. 61.
    Potter MA, Jong KAD (1994) A cooperative coevolutionary approach to function optimization. In: Proceedings of the international conference on evolutionary computation. The third conference on parallel problem solving from nature: parallel problem solving from nature. Springer, pp 249–257Google Scholar
  62. 62.
    Dorigo M, Birattari M (2010) Ant colony optimization. In: Encyclopedia of machine learning. Springer, pp 36–39Google Scholar
  63. 63.
    Kennedy J (2010) Particle swarm optimization. In: Encyclopedia of machine learning. Springer, pp 760–766Google Scholar
  64. 64.
    Vaz AIF, Vicente LN (2007) A particle swarm pattern search method for bound constrained global optimization. J Glob Optim 39(2):197–219MathSciNetCrossRefMATHGoogle Scholar
  65. 65.
    Dolan ED, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91(2):201–213MathSciNetCrossRefMATHGoogle Scholar
  66. 66.
    Ren Y, Wu Y (2013) An efficient algorithm for high-dimensional function optimization. Soft Comput 17 (6):995–1004CrossRefGoogle Scholar
  67. 67.
    García S et al (2009) A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 special session on real parameter optimization. J Heuristics 15(6):617–644CrossRefMATHGoogle Scholar
  68. 68.
    Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate—a practical and powerful approach to multiple testing. J R Stat Soc Ser B Methodol 57(1):289–300MathSciNetMATHGoogle Scholar
  69. 69.
    Dass P et al (2015) Hybridisation of classical unidimensional search with ABC to improve exploitation capability. Int J Artif Intell Soft Comput 5(2):151–164CrossRefGoogle Scholar
  70. 70.
    Jadon S, Bansal J, Tiwari R (2016) Escalated convergent artificial bee colony. J Exp Theor Artif Intell 28(1–2):181–200CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Vincent Gardeux
    • 1
  • Mahamed G. H. Omran
    • 2
  • Rachid Chelouah
    • 1
  • Patrick Siarry
    • 3
  • Fred Glover
    • 4
  1. 1.Department of Computer ScienceEISTI Engineering SchoolCergyFrance
  2. 2.Department of Computer ScienceGulf University for Science & TechnologyKuwait CityKuwait
  3. 3.LiSSi LaboratoryUniversity of Paris-Est CreteilCreteilFrance
  4. 4.Leeds School of BusinessUniversity of ColoradoBoulderUSA

Personalised recommendations