Skip to main content
SpringerLink
Log in

No Free Lunch Theorem: A Review

  • Chapter
  • First Online:
Approximation and Optimization

Part of the book series: Springer Optimization and Its Applications ((SOIA,volume 145))

Abstract

The “No Free Lunch” theorem states that, averaged over all optimization problems, without re-sampling, all optimization algorithms perform equally well. Optimization, search, and supervised learning are the areas that have benefited more from this important theoretical concept. Formulation of the initial No Free Lunch theorem, very soon, gave rise to a number of research works which resulted in a suite of theorems that define an entire research field with significant results in other scientific areas where successfully exploring a search space is an essential and critical task. The objective of this paper is to go through the main research efforts that contributed to this research field, reveal the main issues, and disclose those points that are helpful in understanding the hypotheses, the restrictions, or even the inability of applying No Free Lunch theorems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Source: Google Scholar.

References

  1. Al-Rifaie, M.M., Bishop, J.M.: Swarmic paintings and colour attention. In: International Conference on Evolutionary and Biologically Inspired Music and Art, pp. 97–108. Springer, Berlin (2013)

    Chapter  Google Scholar 

  2. Al-Rifaie, M.M., Bishop, J.M., Caines, S.: Creativity and autonomy in swarm intelligence systems. Cogn. Comput. 4(3), 320–331 (2012)

    Article  Google Scholar 

  3. Amari, S., Murata, N., Muller, K.R., Finke, M., Yang, H.H.: Asymptotic statistical theory of overtraining and cross-validation. IEEE Trans. Neural Netw. 8(5), 985–996 (1997)

    Article  Google Scholar 

  4. Auger, A., Teytaud, O.: Continuous lunches are free plus the design of optimal optimization algorithms. Algorithmica 57(1), 121–146 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Auger, A., Schoenauer, M., Teytaud, O.: Local and global order 3/2 convergence of a surrogate evolutionary algorithm. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 857–864. ACM, New York (2005)

    Google Scholar 

  6. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)

    MATH  Google Scholar 

  7. Cataltepe, Z., Abu-Mostafa, Y.S., Magdon-Ismail, M.: No free lunch for early stopping. Neural Comput. 11(4), 995–1009 (1999)

    Article  Google Scholar 

  8. Ciucu, F., Schmitt, J.: Perspectives on network calculus: no free lunch, but still good value. ACM SIGCOMM Comput. Commun. Rev. 42(4), 311–322 (2012)

    Article  Google Scholar 

  9. Corne, D., Knowles, J.: Some multiobjective optimizers are better than others. In: IEEE Congress on Evolutionary Computation (CEC 2003), vol. 4, pp. 2506–2512. IEEE, Piscataway (2003)

    Google Scholar 

  10. Dembski, W.A.: No Free Lunch: Why Specified Complexity Cannot be Purchased Without Intelligence. Rowman & Littlefield, Langham (2006).

    MATH  Google Scholar 

  11. Dorigo, M., Birattari, M.: Ant colony optimization. In: Encyclopedia of Machine Learning, pp. 36–39. Springer, Boston (2011)

    Google Scholar 

  12. Drettakis, G., Roussou, M., Reche, A., Tsingos, N.: Design and evaluation of a real-world virtual environment for architecture and urban planning. Presence Teleop. Virt. 16(3), 318–332 (2007)

    Article  Google Scholar 

  13. Droste, S., Jansen, T., Wegener, I.: Optimization with randomized search heuristics – the (A)NFL theorem, realistic scenarios, and difficult functions. Theor. Comput. Sci. 287(1), 131–144 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  14. Eberhart, R., Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the IEEE Sixth International Symposium on Micro Machine and Human Science, 1995, MHS’95, pp. 39–43. IEEE, Piscataway (1995)

    Google Scholar 

  15. Epitropakis, M.G., Plagianakos, V.P., Vrahatis, M.N.: Evolutionary adaptation of the differential evolution control parameters. In: Proceedings of the IEEE Congress on Evolutionary Computation, 2009, CEC’09, pp. 1359–1366. IEEE, Piscataway (2009)

    Google Scholar 

  16. Epitropakis, M.G., Tasoulis, D.K., Pavlidis, N.G., Plagianakos, V.P., Vrahatis, M.N.: Enhancing differential evolution utilizing proximity-based mutation operators. IEEE Trans. Evol. Comput. 15(1), 99–119 (2011)

    Article  Google Scholar 

  17. Epitropakis, M.G., Plagianakos, V.P., Vrahatis, M.N.: Evolving cognitive and social experience in particle swarm optimization through differential evolution: a hybrid approach. Inf. Sci. 216, 50–92 (2012)

    Article  Google Scholar 

  18. Ficici, S.G.: Solution Concepts in Coevolutionary Algorithms. PhD thesis, Brandeis University Waltham, Waltham (2004)

    Google Scholar 

  19. Floudas, C.A., Pardalos, P.M.: Encyclopedia of Optimization. Springer Science & Business Media B.V., Dordrecht (2008)

    MATH  Google Scholar 

  20. Georgiou, V.L., Malefaki, S., Parsopoulos, K.E., Alevizos, Ph.D., Vrahatis, M.N.: Expeditive extensions of evolutionary Bayesian probabilistic neural networks. In: Third International Conference on Learning and Intelligent Optimization (LION3 2009). Lecture Notes in Computer Science, vol. 5851, pp. 30–44. Springer, Berlin (2009)

    Google Scholar 

  21. Gómez, D., Rojas, A.: An empirical overview of the no free lunch theorem and its effect on real-world machine learning classification. Neural Comput. 28(1), 216–228 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  22. Goutte, C.: Note on free lunches and cross-validation. Neural Comput. 9(6), 1245–1249 (1997)

    Article  Google Scholar 

  23. Griffiths, E.G., Orponen, P.: Optimization, block designs and no free lunch theorems. Inf. Process. Lett. 94(2), 55–61 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  24. Ho, Y.C.: The no free lunch theorem and the human-machine interface. IEEE Control. Syst. 19(3), 8–10 (1999)

    Article  Google Scholar 

  25. Hopkins, D.A., Thomas, M.: Neural network and regression methods demonstrated in the design optimization of a subsonic aircraft. Structural Mechanics and Dynamics Branch 2002 Annual Report, p. 25 (2003)

    Google Scholar 

  26. Hume, D. (Introduction by Mossner, E.C.): A Treatise of Human Nature. Classics Series. Penguin Books Limited, London (1986)

    Google Scholar 

  27. Hume, D.: A Treatise of Human Nature. The Floating Press Ltd., Auckland (2009). First published in 1740

    Google Scholar 

  28. Karaboga, D., Basturk, B.: A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. J. Glob. Optim. 39(3), 459–471 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  29. Kimbrough, S.O., Koehler, G.J., Lu, M., Wood, D.H.: On a feasible–infeasible two-population (FI-2Pop) genetic algorithm for constrained optimization: distance tracing and no free lunch. Eur. J. Oper. Res. 190(2), 310–327 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  30. Kleijnen, J.P.C.: Sensitivity analysis of simulation experiments: regression analysis and statistical design. Math. Comput. Simul. 34(3–4), 297–315 (1992)

    Article  MATH  Google Scholar 

  31. Kocsis, L., Szepesvari, C.: Bandit-based Monte-Carlo planning. In: European Conference on Machine Learning (ECML 2006). Lecture Notes in Computer Science, vol. 4212, pp. 282–293. Springer, Berlin (2006)

    Google Scholar 

  32. Köppen M.: Some technical remarks on the proof of the no free lunch theorem. In: Proceedings of the Fifth Joint Conference on Information Sciences (JCIS), vol. 1, pp. 1020–1024. Atlantic City (2000)

    Google Scholar 

  33. Köppen, M., Wolpert, D.H., Macready, W.G.: Remarks on a recent paper on the “No Free Lunch” theorems. IEEE Trans. Evol. Comput. 5(3), 295–296 (2001)

    Article  Google Scholar 

  34. Laskari, E.C., Parsopoulos, K.E., Vrahatis, M.N.: Utilizing evolutionary operators in global optimization with dynamic search trajectories. Numer. Algorithms 34(2–4), 393–403 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  35. Macready, W.G., Wolpert, D.H.: What makes an optimization problem hard? Complexity 1(5), 40–46 (1996)

    Article  MathSciNet  Google Scholar 

  36. Marshall, J.A.R., Hinton, T.G.: Beyond no free lunch: Realistic algorithms for arbitrary problem classes. In: IEEE Congress on Evolutionary Computation, pp. 1–6. IEEE, Piscataway (2010)

    Google Scholar 

  37. Orr, H.A.: Review of no free lunch by William A Dembski. Boston Review. Available on-line at http://bostonreview.net/BR27, 3 (2002)

  38. Parsopoulos, K.E., Vrahatis, M.N.: Recent approaches to global optimization problems through particle swarm optimization. Nat. Comput. 1(2–3), 235–306 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  39. Parsopoulos, K.E., Vrahatis, M.N.: On the computation of all global minimizers through particle swarm optimization. IEEE Trans. Evol. Comput. 8(3), 211–224 (2004)

    Article  Google Scholar 

  40. Parsopoulos, K.E., Vrahatis, M.N.: Parameter selection and adaptation in unified particle swarm optimization. Math. Comput. Model. 46(1–2), 198–213 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  41. Parsopoulos, K.E., Vrahatis, M.N.: Particle Swarm Optimization and Intelligence: Advances and Applications. Information Science Publishing (IGI Global), Hershey (2010)

    Book  Google Scholar 

  42. Perakh, M.: The No Free Lunch Theorems and Their Application to Evolutionary Algorithms (2003)

    Google Scholar 

  43. Petalas, Y.G., Parsopoulos, K.E., Vrahatis, M.N.: Memetic particle swarm optimization. Ann. Oper. Res. 156(1), 99–127 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  44. Poli, R., Graff, M.: There is a free lunch for hyper-heuristics, genetic programming and computer scientists. In: Proceedings of the 12th European Conference on Genetic Programming, EuroGP ’09, pp. 195–207. Springer, Berlin (2009)

    Google Scholar 

  45. Poli, R., Graff, M., McPhee, N.F.: Free lunches for function and program induction. In: Proceedings of the Tenth ACM SIGEVO Workshop on Foundations of Genetic Algorithms, FOGA ’09, pp. 183–194. ACM, New York (2009)

    Google Scholar 

  46. Rivals, I., Personnaz, L.: On cross validation for model selection. Neural Comput. 11(4), 863–870 (1999)

    Article  Google Scholar 

  47. Rosenberg, L.B.: Human swarms, a real-time paradigm for collective intelligence. Collective Intelligence (2015)

    Google Scholar 

  48. Schumacher, C., Vose, M.D., Whitley, L.D.: The no free lunch and problem description length. In: Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation, pp. 565–570. Morgan Kaufmann Publishers Inc., Burlington (2001)

    Google Scholar 

  49. Service, T.C., Tauritz, D.R.: A no-free-lunch framework for coevolution. In: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, pp. 371–378. ACM, Piscataway (2008)

    Google Scholar 

  50. Sotiropoulos, D.G., Stavropoulos, E.C., Vrahatis, M.N.: A new hybrid genetic algorithm for global optimization. Nonlinear Anal. Theory Methods Appl. 30(7), 4529–4538 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  51. Teytaud, O., Flory, S.: Upper confidence trees with short term partial information. In: European Conference on the Applications of Evolutionary Computation, pp. 153–162. Springer, Berlin (2011)

    Chapter  Google Scholar 

  52. Thalmann, D.: Crowd Simulation. Wiley Online Library (2007)

    Google Scholar 

  53. Van Grieken, M.: Optimisation pour l’apprentissage et apprentissage pour l’optimisation. PhD thesis, Université Paul Sabatier-Toulouse III, Toulouse (2004)

    Google Scholar 

  54. Vanaret, C., Gallard, F., Martins, J.: On the consequences of the “No Free Lunch” theorem for optimization on the choice of an appropriate MDO architecture. In: 18th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, pp. 3148 (2017)

    Google Scholar 

  55. Wolpert, D.H.: On the connection between in-sample testing and generalization error. Complex Syst. 6(1), 47–94 (1992)

    MathSciNet  MATH  Google Scholar 

  56. Wolpert, D.H.: The lack of a priori distinctions between learning algorithms. Neural Comput. 8(7), 1341–1390 (1996)

    Article  Google Scholar 

  57. Wolpert, D.H.: The existence of a priori distinctions between learning algorithms. Neural Comput. 8(7), 1391–1420 (1996)

    Article  Google Scholar 

  58. Wolpert, D.H.: The supervised learning no-free-lunch theorems. In: Soft Computing and Industry, pp. 25–42. Springer, London (2002)

    Chapter  Google Scholar 

  59. Wolpert, D.H.: The Supervised Learning No-Free-Lunch Theorems, pp. 25–42. Springer, London (2002)

    Chapter  Google Scholar 

  60. Wolpert, D.H.: What the no free lunch theorems really mean; how to improve search algorithms. SFI working paper: 2012–10-017. Santa Fe Institute, Santa Fe (2012)

    Google Scholar 

  61. Wolpert, D.H., Macready, W.G.: No Free Lunch Theorems for Search. Tech. Rep. SFI-TR-95-02-010. Santa Fe Institute, Santa Fe (1995)

    Google Scholar 

  62. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    Article  Google Scholar 

  63. Wolpert, D. H., Macready, W.G.: Coevolutionary free lunches. IEEE Trans. Evol. Comput. 9(6), 721–735 (2005)

    Article  Google Scholar 

  64. Yang, X.S.: Firefly algorithm, stochastic test functions and design optimization. Int. J. Bio-Inspired Comput. 2(2), 78–84 (2010)

    Article  Google Scholar 

  65. Yang, X.S.: A new metaheuristic bat-inspired algorithm. In: Nature Inspired Cooperative Strategies for Optimization (NICSO 2010), pp. 65–74. Springer, Berlin (2010)

    Chapter  Google Scholar 

  66. Yang, X.S.: Swarm-based metaheuristic algorithms and no-free-lunch theorems. In: Theory and New Applications of Swarm Intelligence. InTech, London (2012)

    Google Scholar 

  67. Yang, X.S., Deb, S.: Cuckoo search via Lévy flights. In: Proceedings of the World Congress on Nature & Biologically Inspired Computing, 2009, NaBIC 2009. pp. 210–214. IEEE, Piscataway (2009)

    Google Scholar 

  68. Zhu, H., Rohwer, R.: No free lunch for cross-validation. Neural Comput. 8(7), 1421–1426 (1996)

    Article  Google Scholar 

Download references

Acknowledgements

S.-A. N. Alexandropoulos is supported by Greece and the European Union (European Social Fund-ESF) through the Operational Programme “Human Resources Development, Education and Lifelong Learning” in the context of the project “Strengthening Human Resources Research Potential via Doctorate Research” (MIS-5000432), implemented by the State Scholarships Foundation (IKY). P. M. Pardalos is supported by the Paul and Heidi Brown Preeminent Professorship at ISE (University of Florida, USA), and a Humboldt Research Award (Germany).

Author information

Authors and Affiliations

  1. Department of Informatics and Telecommunications, University of Ioannina, Arta, Greece

    Stavros P. Adam

  2. Computational Intelligence Laboratory – CILab, Department of Mathematics, University of Patras, Patras, Greece

    Stavros P. Adam, Stamatios-Aggelos N. Alexandropoulos & Michael N. Vrahatis

  3. Department of Industrial & Systems Engineering, University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

Authors
  1. Stavros P. Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stamatios-Aggelos N. Alexandropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Panos M. Pardalos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael N. Vrahatis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael N. Vrahatis .

Editor information

Editors and Affiliations

  1. Department of Economics, University of Athens, Athens, Greece

    Ioannis C. Demetriou

  2. Department of Industrial & Systems Engineering, University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Adam, S.P., Alexandropoulos, SA.N., Pardalos, P.M., Vrahatis, M.N. (2019). No Free Lunch Theorem: A Review. In: Demetriou, I., Pardalos, P. (eds) Approximation and Optimization . Springer Optimization and Its Applications, vol 145. Springer, Cham. https://doi.org/10.1007/978-3-030-12767-1_5

Download citation

Publish with us

Policies and ethics

Search

Navigation