Stopping rules as experimental design

  • Samuel C. FletcherEmail author
Paper in General Philosophy of Science
Part of the following topical collections:
  1. EPSA17: Selected papers from the biannual conference in Exeter


A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” (SRP) states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound.


Stopping rules Optional stopping Likelihood principle Statistical evidence Experimental design Statistical testing 



Thanks to Greg Gandenberger, Kasey Genin, Jonathan Livengood, Dan Malinksy, Conor Mayo-Wilson, Jan Sprenger, and an anonymous referee for comments on a previous version, and audiences at Minnesota, Munich, Bologna (SILFS2017), Edinburgh (BSPS2017), and Exeter (EPSA2017) for their insightful comments. Part of this work was completed with the support of a European Commission Marie Curie Fellowship (PIIF-GA-2013-628533).


  1. Azjen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs: Prentice-Hall.Google Scholar
  2. Azjen, I. (1985). From intentions to actions: A theory of planned behavior. In Kuhl, J, & Beckmann, J (Eds.) Action control (pp. 11–39). Berlin: Springer.Google Scholar
  3. Azjen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211.CrossRefGoogle Scholar
  4. Backe, A. (1999). The likelihood principle and the reliability of experiments. Philosophy of Science, 66(Proceedings), S354–S361.CrossRefGoogle Scholar
  5. Berger, J.O., & Wolpert, R.L. (1988). The likelihood principle, 2nd edn. Hayward: Institute of Mathematical Statistics.Google Scholar
  6. Berry, S., & Viele, K. (2008). A note on hypothesis testing with random sample sizes and its relationship with Bayes factors. Journal of Data Science, 6, 75–87.Google Scholar
  7. Birnbaum, A. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 57(298), 269–306.CrossRefGoogle Scholar
  8. Carnap, R. (1950). Empiricism, semantics, and ontology. Revue Internationale de Philosophie, 4(11), 20–40.Google Scholar
  9. Cohen, A.J. (2010). A conceptual and (preliminary) normative exploration of waste. Social Philosophy and Policy, 27(2), 233–273.CrossRefGoogle Scholar
  10. Cox, D.R. (1958). Some problems connected with statistical inference. Annals of Mathematical Statistics, 29, 357–372.CrossRefGoogle Scholar
  11. Edwards, W., Lindman, H., Savage, L.J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70(3), 193–242.CrossRefGoogle Scholar
  12. Fishbein, M., & Azjen, I. (1975). Belief, attitude, intention, and behavior: An introduction theory and research. Reading: Addison-Wesley.Google Scholar
  13. Fishbein, M., & Azjen, I. (2011). Predicting and changing behavior: The reasoned action approach. New York: Psychology Press.CrossRefGoogle Scholar
  14. Fletcher, G.P. (1998). Basic concepts of criminal law. New York: Oxford University Press.Google Scholar
  15. Franklin, A. (1994). How to avoid the experimenters’ regress. Studies in History and Philosophy of Science, 25(3), 463–491.CrossRefGoogle Scholar
  16. Franklin, A. (2010). Gravity waves and neutrinos: The later work of Joseph Weber. Perspectives on Science, 18(2), 119–151.CrossRefGoogle Scholar
  17. Gandenberger, G. (2015). Differences among noninformative stopping rules are often relevant to Bayesian decisions. arXiv:1707.00214.
  18. Gillies, D. (1990). Bayesianism versus falsificationism. Ratio (New Series), III (1), 82–98.Google Scholar
  19. Hacking, I. (1965). The logic of statistical inference. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  20. Howson, C., & Urbach, P. (2006). Scientific reasoning: The Bayesian approach, 3rd edn. Chicago: Open Court.Google Scholar
  21. Huber, F. (2018). Confirmation theory. In: The internet encyclopedia of philosophy. n.d. Accessed 30 Mar.Google Scholar
  22. Kelly, T. (2016). Evidence. In Zalta, E.N. (Ed.) The Stanford encyclopedia of philosophy. Winter 2016 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar
  23. Koike, D.A. (1989). Pragmatic competence and adult L2 acquisition: Speech acts in interlanguage. The Modern Language Journal, 73(3), 279–289.CrossRefGoogle Scholar
  24. Livengood, J. (2019). Counting experiments. Philosophical Studies, 176(1), 175–195.CrossRefGoogle Scholar
  25. Malinksy, D. (2015). Hypothesis testing, “Dutch book” arguments, and risk. Philosophy of Science, 82(5), 917–929.CrossRefGoogle Scholar
  26. Mayo, D.G. (1996). Error and the growth of experimental knowledge. Chicago: University of Chicago Press.CrossRefGoogle Scholar
  27. Mayo, D.G., & Kruse, M. (2001). Principles of inference and their consequences. In Corfield, D., & Williamson, J. (Eds.) Foundations of Bayesianism (pp. 381–403). Dordrecht: Kluwer.Google Scholar
  28. Raiffa, H., & Shlaifer, R. (1961). Applied statistical decision theory. Boston: Harvard University.Google Scholar
  29. Reiss, J., & Sprenger, J. (2017). Scientific objectivity. In Zalta, E.N. (Ed.) The Stanford encyclopedia of philosophy. Winter 2017 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar
  30. Romeijn, J.-W. (2017). Philosophy of statistics. In Zalta, E.N. (Ed.) The Stanford encyclopedia of philosophy. Spring 2017 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar
  31. Savage, L.J. (1962). The foundations of statistical inference: A discussion. London: Methuen.Google Scholar
  32. Schervish, M.J., Seidenfeld, T., Kadane, J.B. (2002). A rate of incoherence applied to fixed-level testing. Philosophy of Science, 69(Proceedings), S248–S264.CrossRefGoogle Scholar
  33. Shapin, S. (2010). Never pure: Historical studies of science as if it was produced by people with bodies, situated in time, space, culture, and society, and struggling for credibility and authority. Baltimore: Johns Hopkins University Press.Google Scholar
  34. Siegmund, D. (1985). Sequential analysis: Tests and confidence intervals. New York: Springer.CrossRefGoogle Scholar
  35. Southwood, N. (2016). Does “ought” imply “feasible”? Philosophy and Public Affairs, 44(1), 7–45.CrossRefGoogle Scholar
  36. Sprenger, J. (2009). Evidence and experimental design in sequential trials. Philosophy of Science, 76(5), 637–649.CrossRefGoogle Scholar
  37. Steel, D. (2003). A Bayesian way to make stopping rules matter. Synthese, 58, 213–227.Google Scholar
  38. Steele, K. (2013). Persistent experimenters, stopping rules, and statistical inference. Erkenntnis, 78, 937–961.CrossRefGoogle Scholar
  39. Wald, A. (1947). Sequential analysis. New York: Wiley.Google Scholar
  40. Whitehead, J. (1997). The design and analysis of sequential clinical trials, 2nd edn. New York: Wiley.Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of PhilosophyUniversity of Minnesota, Twin CitiesMinneapolisUSA

Personalised recommendations