# Stopping rules as experimental design

**Part of the following topical collections:**

## Abstract

A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” (SRP) states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound.

## Keywords

Stopping rules Optional stopping Likelihood principle Statistical evidence Experimental design Statistical testing## Notes

### Acknowledgements

Thanks to Greg Gandenberger, Kasey Genin, Jonathan Livengood, Dan Malinksy, Conor Mayo-Wilson, Jan Sprenger, and an anonymous referee for comments on a previous version, and audiences at Minnesota, Munich, Bologna (SILFS2017), Edinburgh (BSPS2017), and Exeter (EPSA2017) for their insightful comments. Part of this work was completed with the support of a European Commission Marie Curie Fellowship (PIIF-GA-2013-628533).

## References

- Azjen, I., & Fishbein, M. (1980).
*Understanding attitudes and predicting social behavior*. Englewood Cliffs: Prentice-Hall.Google Scholar - Azjen, I. (1985). From intentions to actions: A theory of planned behavior. In Kuhl, J, & Beckmann, J (Eds.)
*Action control*(pp. 11–39). Berlin: Springer.Google Scholar - Azjen, I. (1991). The theory of planned behavior.
*Organizational Behavior and Human Decision Processes*,*50*(2), 179–211.CrossRefGoogle Scholar - Backe, A. (1999). The likelihood principle and the reliability of experiments.
*Philosophy of Science*,*66*(Proceedings), S354–S361.CrossRefGoogle Scholar - Berger, J.O., & Wolpert, R.L. (1988).
*The likelihood principle*, 2nd edn. Hayward: Institute of Mathematical Statistics.Google Scholar - Berry, S., & Viele, K. (2008). A note on hypothesis testing with random sample sizes and its relationship with Bayes factors.
*Journal of Data Science*,*6*, 75–87.Google Scholar - Birnbaum, A. (1962). On the foundations of statistical inference.
*Journal of the American Statistical Association*,*57*(298), 269–306.CrossRefGoogle Scholar - Carnap, R. (1950). Empiricism, semantics, and ontology.
*Revue Internationale de Philosophie*,*4*(11), 20–40.Google Scholar - Cohen, A.J. (2010). A conceptual and (preliminary) normative exploration of waste.
*Social Philosophy and Policy*,*27*(2), 233–273.CrossRefGoogle Scholar - Cox, D.R. (1958). Some problems connected with statistical inference.
*Annals of Mathematical Statistics*,*29*, 357–372.CrossRefGoogle Scholar - Edwards, W., Lindman, H., Savage, L.J. (1963). Bayesian statistical inference for psychological research.
*Psychological Review*,*70*(3), 193–242.CrossRefGoogle Scholar - Fishbein, M., & Azjen, I. (1975).
*Belief, attitude, intention, and behavior: An introduction theory and research*. Reading: Addison-Wesley.Google Scholar - Fishbein, M., & Azjen, I. (2011).
*Predicting and changing behavior: The reasoned action approach*. New York: Psychology Press.CrossRefGoogle Scholar - Fletcher, G.P. (1998).
*Basic concepts of criminal law*. New York: Oxford University Press.Google Scholar - Franklin, A. (1994). How to avoid the experimenters’ regress.
*Studies in History and Philosophy of Science*,*25*(3), 463–491.CrossRefGoogle Scholar - Franklin, A. (2010). Gravity waves and neutrinos: The later work of Joseph Weber.
*Perspectives on Science*,*18*(2), 119–151.CrossRefGoogle Scholar - Gandenberger, G. (2015). Differences among noninformative stopping rules are often relevant to Bayesian decisions. arXiv:1707.00214.
- Gillies, D. (1990). Bayesianism versus falsificationism.
*Ratio (New Series)*,*III*(1), 82–98.Google Scholar - Hacking, I. (1965).
*The logic of statistical inference*. Cambridge: Cambridge University Press.CrossRefGoogle Scholar - Howson, C., & Urbach, P. (2006).
*Scientific reasoning: The Bayesian approach*, 3rd edn. Chicago: Open Court.Google Scholar - Huber, F. (2018). Confirmation theory. In:
*The internet encyclopedia of philosophy*. n.d. Accessed 30 Mar.Google Scholar - Kelly, T. (2016). Evidence. In Zalta, E.N. (Ed.)
*The Stanford encyclopedia of philosophy*. Winter 2016 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar - Koike, D.A. (1989). Pragmatic competence and adult L2 acquisition: Speech acts in interlanguage.
*The Modern Language Journal*,*73*(3), 279–289.CrossRefGoogle Scholar - Livengood, J. (2019). Counting experiments.
*Philosophical Studies*,*176*(1), 175–195.CrossRefGoogle Scholar - Malinksy, D. (2015). Hypothesis testing, “Dutch book” arguments, and risk.
*Philosophy of Science*,*82*(5), 917–929.CrossRefGoogle Scholar - Mayo, D.G. (1996).
*Error and the growth of experimental knowledge*. Chicago: University of Chicago Press.CrossRefGoogle Scholar - Mayo, D.G., & Kruse, M. (2001). Principles of inference and their consequences. In Corfield, D., & Williamson, J. (Eds.)
*Foundations of Bayesianism*(pp. 381–403). Dordrecht: Kluwer.Google Scholar - Raiffa, H., & Shlaifer, R. (1961).
*Applied statistical decision theory*. Boston: Harvard University.Google Scholar - Reiss, J., & Sprenger, J. (2017). Scientific objectivity. In Zalta, E.N. (Ed.)
*The Stanford encyclopedia of philosophy*. Winter 2017 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar - Romeijn, J.-W. (2017). Philosophy of statistics. In Zalta, E.N. (Ed.)
*The Stanford encyclopedia of philosophy*. Spring 2017 edition. Stanford University: Metaphysics Research Laboratory.Google Scholar - Savage, L.J. (1962).
*The foundations of statistical inference: A discussion*. London: Methuen.Google Scholar - Schervish, M.J., Seidenfeld, T., Kadane, J.B. (2002). A rate of incoherence applied to fixed-level testing.
*Philosophy of Science*,*69*(Proceedings), S248–S264.CrossRefGoogle Scholar - Shapin, S. (2010).
*Never pure: Historical studies of science as if it was produced by people with bodies, situated in time, space, culture, and society, and struggling for credibility and authority*. Baltimore: Johns Hopkins University Press.Google Scholar - Siegmund, D. (1985).
*Sequential analysis: Tests and confidence intervals*. New York: Springer.CrossRefGoogle Scholar - Southwood, N. (2016). Does “ought” imply “feasible”?
*Philosophy and Public Affairs*,*44*(1), 7–45.CrossRefGoogle Scholar - Sprenger, J. (2009). Evidence and experimental design in sequential trials.
*Philosophy of Science*,*76*(5), 637–649.CrossRefGoogle Scholar - Steel, D. (2003). A Bayesian way to make stopping rules matter.
*Synthese*,*58*, 213–227.Google Scholar - Steele, K. (2013). Persistent experimenters, stopping rules, and statistical inference.
*Erkenntnis*,*78*, 937–961.CrossRefGoogle Scholar - Wald, A. (1947).
*Sequential analysis*. New York: Wiley.Google Scholar - Whitehead, J. (1997).
*The design and analysis of sequential clinical trials*, 2nd edn. New York: Wiley.Google Scholar