Skip to main content
Log in

How experimental algorithmics can benefit from Mayo’s extensions to Neyman–Pearson theory of testing

  • Published:
Synthese Aims and scope Submit manuscript

Abstract

Although theoretical results for several algorithms in many application domains were presented during the last decades, not all algorithms can be analyzed fully theoretically. Experimentation is necessary. The analysis of algorithms should follow the same principles and standards of other empirical sciences. This article focuses on stochastic search algorithms, such as evolutionary algorithms or particle swarm optimization. Stochastic search algorithms tackle hard real-world optimization problems, e.g., problems from chemical engineering, airfoil optimization, or bio-informatics, where classical methods from mathematical optimization fail. Nowadays statistical tools that are able to cope with problems like small sample sizes, non-normal distributions, noisy results, etc. are developed for the analysis of algorithms. Although there are adequate tools to discuss the statistical significance of experimental data, statistical significance is not scientifically meaningful per se. It is necessary to bridge the gap between the statistical significance of an experimental result and its scientific meaning. We will propose some ideas on how to accomplish this task based on Mayo’s learning model (NPT*).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bartz-Beielstein T. (2006). Experimental research in evolutionary computation—The new experimentalism. Berlin, Heidelberg, New York: Springer.

    Google Scholar 

  • Bartz-Beielstein, T., & Preuss, M. (2004). CEC tutorial on experimental research in evolutionary computation. In IEEE Congress on Evolutionary Computation, Tutorial Program. Tutorials given at CEC in 2004 and 2005.

  • Bartz-Beielstein, T., & Preuss, M. (2005). GECCO tutorial on experimental research in evolutionary computation. In 2005 Genetic and Evolutionary Computation Conference, Tutorial Program. Tutorials given at GECCO in 2005, 2006, and 2007.

  • Bartz-Beielstein, T., & Preuss, M. (2006a). Considerations of budget allocation for sequential parameter pptimization (SPO). In Workshop on Empirical Methods for the Analysis of Algorithms (EMAA), Tutorial Program. Held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN IX) (pp. 35–40).

  • Bartz-Beielstein, T., & Preuss, M. (2006b). Sequential parameter optimization (SPO) and the role of tuning in experimental analysis. In Workshop on Empirical Methods for the Analysis of Algorithms (EMAA). Held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN IX) (pp. 5–6). Invited talk

  • Clerc, M., & Kennedy, J. (2006). Standard PSO version 2006. http://www.particleswarm.info/Standard_PSO_2006.c. Cited 11 August 2007.

  • Gregoire, T. (2001). Biometry in the 21st Century: Whither statistical inference? (Invited Keynote). Proceedings of the Forest Biometry and Information Science Conference held at the University of Greenwich, June 2001. http://cms1.gre.ac.uk/conferences/iufro/proceedings/gregoire.pdf. Cited 19 May 2004.

  • Kleijnen J.P.C. (1987). Statistical tools for simulation practitioners. New York, NY: Marcel Dekker.

    Google Scholar 

  • Mayo D.G. (1983). An objective theory of statistical testing. Synthese 57: 297–340

    Article  Google Scholar 

  • Mayo D.G. (1996). Error and the growth of experimental knowledge. Chicago, IL: The University of Chicago Press.

    Google Scholar 

  • Mayo D.G., Spanos A. (2004). Methodology in practice: Statistical misspecification testing. Philosophy of Science 71: 1007–1025

    Article  Google Scholar 

  • Mayo D.G., Spanos A. (2006). Severe testing as a basic concept in a Neyman–Pearson philosophy of induction. British Journal for the Philosophy of Science 57: 323–357

    Article  Google Scholar 

  • Montgomery D.C. (2001). Design and analysis of experiments (5th ed.). New York, NY: Wiley.

    Google Scholar 

  • Morrison D.E., Henkel R.E. (eds) (1970). The significance test controversy—A reader. London, UK: Butterworths.

    Google Scholar 

  • Wolpert D., Macready W. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1(1): 67–82

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas Bartz-Beielstein.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bartz-Beielstein, T. How experimental algorithmics can benefit from Mayo’s extensions to Neyman–Pearson theory of testing. Synthese 163, 385–396 (2008). https://doi.org/10.1007/s11229-007-9297-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11229-007-9297-z

Keywords

Navigation