A better stopping rule for conventional statistical tests

  • Robert W. FrickEmail author


The goal of some research studies is to demonstrate the existence of an effect. Statistical testing, withp less than .05, is one criterion for establishing the existence of this effect. In this situation, the fixedsample stopping rule, in which the number of subjects is determined in advance, is impractical and inefficient. This article presents a sequential stopping rule that is practical and about 30% more efficient: Once a minimum number of subjects is tested, stop withp less than .01 or greater than .36; otherwise, keep testing. This procedure keeps alpha at .05 and can be adjusted to fit researchers’ needs and inclinations.


Null Hypothesis Monte Carlo Simulation Sequential Probability Ratio Test High Criterion Multiple Statistical Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Arghami, N. R., &Billard, L. (1982). A modification of a truncated partial sequential procedure.Biometrika,69, 613–618.CrossRefGoogle Scholar
  2. Arghami, N. R., &Billard, L. (1991). A partial sequentialt-test.Sequential Analysis,10, 181–197.CrossRefGoogle Scholar
  3. Armitage, P. (1957). Restricted sequential procedures.Biometrika,44, 9–26.Google Scholar
  4. Billard, L., &Vagholkar, M. K. (1969). A sequential procedure for testing a null hypothesis against a two-sided alternative hypothesis.Journal of the Royal Statistical Society B,31, 285–294.Google Scholar
  5. Boneau, C. A. (1960). The effects of violations of assumptions underlying thet test.Psychological Bulletin,57, 49–64.CrossRefPubMedGoogle Scholar
  6. DeMets, D. L., &Lan, K. K. G. (1984). An overview of sequential methods and their application in clinical trials.Communications in Statistics: Theory & Methods,13, 2315–2338.CrossRefGoogle Scholar
  7. Doll, R. (1982). Clinical trials: Retrospect and prospect.Statistics in Medicine,1, 337–344.CrossRefPubMedGoogle Scholar
  8. Edwards, W., Lindman, H., &Savage, L. (1963). Bayesian statistical inference for psychological research.Psychological Review,70, 193–242.CrossRefGoogle Scholar
  9. Eiting, M. H. (1991). Sequential reliability tests.Applied Psychological Measurement,15, 193–205.CrossRefGoogle Scholar
  10. Fiske, D. W., &Jones, L. V. (1954). Sequential analysis in psychological research.Psychological Bulletin,51, 264–275.CrossRefPubMedGoogle Scholar
  11. Frick, R. W. (1995). Accepting the null hypothesis.Memory & Cognition,23, 132–138.CrossRefGoogle Scholar
  12. Frick, R. W. (1996). The appropriate use of null hypothesis testing.Psychological Methods,1, 379–390.CrossRefGoogle Scholar
  13. Goodman, S. N., &Royall, R. (1988). Evidence and scientific research.American Journal of Public Health,78, 1568–1574.CrossRefPubMedGoogle Scholar
  14. Haybittle, J. L. (1971). Repeated assessment of results in clinical trials of cancer treatment.British Journal of Radiology,44, 793–797.CrossRefPubMedGoogle Scholar
  15. Hoffman, H. S. (1992). An application of sequential analysis to observerbased psychophysics.Infant Behavior & Development,15, 271–277.CrossRefGoogle Scholar
  16. Hwang, I. K. (1992). Overview of the development of sequential procedures. In K. E. Bruce (Ed.),Biopharmaceutical sequential statistical application (pp. 3–18). New York: Dekker.Google Scholar
  17. Jennison, C., &Turnbull, B. W. (1991). Group sequential tests and repeated confidence intervals. In M. Ghosh & P. K. Sen (Eds.),Handbook of sequential analysis (pp. 283–311). New York: Dekker.Google Scholar
  18. Lan, K. K. G., DeMets, D. L., &Halperin, M. (1984). More flexible sequential and non-sequential designs in long-term clinical trials.Communication in Statistics: Theory & Methods,13, 2339–2353.CrossRefGoogle Scholar
  19. Lan, K. K. G., Simon, R., &Halperin, M. (1982). Stochastically curtailed tests in long-term clinical trials.Communication in Statistics: Sequential Analysis,1, 207–219.CrossRefGoogle Scholar
  20. Linn, R. L., Rock, D. A., &Cleary, T. A. (1969). The development and evaluation of several programmed testing methods.Educational & Psychological Measurement,29, 129–146.CrossRefGoogle Scholar
  21. Linn, R. L., Rock, D. A., &Cleary, T. A. (1972). Sequential testing for dichotomous decisions.Educational & Psychological Measurement,32, 85–95.CrossRefGoogle Scholar
  22. Neyman, J. (1950).First course in probability and statistics. New York: Holt.Google Scholar
  23. O’Brien, P. C., &Fleming, T. R. (1979). A multiple testing procedure for clinical trials.Biometrics,35, 549–556.CrossRefPubMedGoogle Scholar
  24. Park, C. (1992). An approximation method for the characteristics of the sequential probability ratio test.Sequential Analysis,11, 55–72.CrossRefGoogle Scholar
  25. Peto, R., Pike, P., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., &Smith, P. G. (1976). Design and analysis of randomized clinical trials requiring prolonged observation of each patient.British Journal of Cancer,35, 585–611.CrossRefGoogle Scholar
  26. Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials.Biometrika,64, 191–199.CrossRefGoogle Scholar
  27. Pocock, S. J. (1992). When to stop a clinical trial.British Medical Journal,305, 235–240.CrossRefPubMedGoogle Scholar
  28. Proschan, M. A., Follmann, D. A., &Waclawiw, M. A. (1992). Effects of assumption violations on Type I error rate in group sequential monitoring.Biometrics,48, 1131–1143.CrossRefGoogle Scholar
  29. Siegmund, D. (1985).Sequential analysis: Tests and confidence intervals. New York: Springer-Verlag.Google Scholar
  30. Slud, E., &Wei, L. J. (1982). Two sample repeated significance tests based on the modified Wilcoxon statistic.Journal of the American Statistical Association,77, 862–868.CrossRefGoogle Scholar
  31. Sobel, M., &Wald, A. (1949). A sequential decision procedure for choosing one of three hypotheses concerning the unknown mean of a normal distribution.Annals of Mathematical Statistics,20, 502–522.CrossRefGoogle Scholar
  32. Sterling, T. D., Rosenbaum, W. L., &Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa.American Statistician,49, 108–112.CrossRefGoogle Scholar
  33. Wald, W. (1947).Sequential analysis. New York: Dover.Google Scholar
  34. Weiss, L. (1953). Testing one simple hypothesis against another.Annals of Mathematical Statistics,24, 273–281.CrossRefGoogle Scholar
  35. Whitehead, J., &Brunier, H. (1990). The double triangular test: A sequential test for the two-sided alternative with early stopping under the null hypothesis.Sequential Analysis,9, 117–136.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc 1998

Authors and Affiliations

  1. 1.Department of PsychologySUNY at Stony BrookStony Brook

Personalised recommendations