A better stopping rule for conventional statistical tests

Abstract

The goal of some research studies is to demonstrate the existence of an effect. Statistical testing, withp less than .05, is one criterion for establishing the existence of this effect. In this situation, the fixedsample stopping rule, in which the number of subjects is determined in advance, is impractical and inefficient. This article presents a sequential stopping rule that is practical and about 30% more efficient: Once a minimum number of subjects is tested, stop withp less than .01 or greater than .36; otherwise, keep testing. This procedure keeps alpha at .05 and can be adjusted to fit researchers’ needs and inclinations.

References

  1. Arghami, N. R., &Billard, L. (1982). A modification of a truncated partial sequential procedure.Biometrika,69, 613–618.

    Article  Google Scholar 

  2. Arghami, N. R., &Billard, L. (1991). A partial sequentialt-test.Sequential Analysis,10, 181–197.

    Article  Google Scholar 

  3. Armitage, P. (1957). Restricted sequential procedures.Biometrika,44, 9–26.

    Google Scholar 

  4. Billard, L., &Vagholkar, M. K. (1969). A sequential procedure for testing a null hypothesis against a two-sided alternative hypothesis.Journal of the Royal Statistical Society B,31, 285–294.

    Google Scholar 

  5. Boneau, C. A. (1960). The effects of violations of assumptions underlying thet test.Psychological Bulletin,57, 49–64.

    Article  PubMed  Google Scholar 

  6. DeMets, D. L., &Lan, K. K. G. (1984). An overview of sequential methods and their application in clinical trials.Communications in Statistics: Theory & Methods,13, 2315–2338.

    Article  Google Scholar 

  7. Doll, R. (1982). Clinical trials: Retrospect and prospect.Statistics in Medicine,1, 337–344.

    Article  PubMed  Google Scholar 

  8. Edwards, W., Lindman, H., &Savage, L. (1963). Bayesian statistical inference for psychological research.Psychological Review,70, 193–242.

    Article  Google Scholar 

  9. Eiting, M. H. (1991). Sequential reliability tests.Applied Psychological Measurement,15, 193–205.

    Article  Google Scholar 

  10. Fiske, D. W., &Jones, L. V. (1954). Sequential analysis in psychological research.Psychological Bulletin,51, 264–275.

    Article  PubMed  Google Scholar 

  11. Frick, R. W. (1995). Accepting the null hypothesis.Memory & Cognition,23, 132–138.

    Article  Google Scholar 

  12. Frick, R. W. (1996). The appropriate use of null hypothesis testing.Psychological Methods,1, 379–390.

    Article  Google Scholar 

  13. Goodman, S. N., &Royall, R. (1988). Evidence and scientific research.American Journal of Public Health,78, 1568–1574.

    Article  PubMed  Google Scholar 

  14. Haybittle, J. L. (1971). Repeated assessment of results in clinical trials of cancer treatment.British Journal of Radiology,44, 793–797.

    Article  PubMed  Google Scholar 

  15. Hoffman, H. S. (1992). An application of sequential analysis to observerbased psychophysics.Infant Behavior & Development,15, 271–277.

    Article  Google Scholar 

  16. Hwang, I. K. (1992). Overview of the development of sequential procedures. In K. E. Bruce (Ed.),Biopharmaceutical sequential statistical application (pp. 3–18). New York: Dekker.

    Google Scholar 

  17. Jennison, C., &Turnbull, B. W. (1991). Group sequential tests and repeated confidence intervals. In M. Ghosh & P. K. Sen (Eds.),Handbook of sequential analysis (pp. 283–311). New York: Dekker.

    Google Scholar 

  18. Lan, K. K. G., DeMets, D. L., &Halperin, M. (1984). More flexible sequential and non-sequential designs in long-term clinical trials.Communication in Statistics: Theory & Methods,13, 2339–2353.

    Article  Google Scholar 

  19. Lan, K. K. G., Simon, R., &Halperin, M. (1982). Stochastically curtailed tests in long-term clinical trials.Communication in Statistics: Sequential Analysis,1, 207–219.

    Article  Google Scholar 

  20. Linn, R. L., Rock, D. A., &Cleary, T. A. (1969). The development and evaluation of several programmed testing methods.Educational & Psychological Measurement,29, 129–146.

    Article  Google Scholar 

  21. Linn, R. L., Rock, D. A., &Cleary, T. A. (1972). Sequential testing for dichotomous decisions.Educational & Psychological Measurement,32, 85–95.

    Article  Google Scholar 

  22. Neyman, J. (1950).First course in probability and statistics. New York: Holt.

    Google Scholar 

  23. O’Brien, P. C., &Fleming, T. R. (1979). A multiple testing procedure for clinical trials.Biometrics,35, 549–556.

    Article  PubMed  Google Scholar 

  24. Park, C. (1992). An approximation method for the characteristics of the sequential probability ratio test.Sequential Analysis,11, 55–72.

    Article  Google Scholar 

  25. Peto, R., Pike, P., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., &Smith, P. G. (1976). Design and analysis of randomized clinical trials requiring prolonged observation of each patient.British Journal of Cancer,35, 585–611.

    Article  Google Scholar 

  26. Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials.Biometrika,64, 191–199.

    Article  Google Scholar 

  27. Pocock, S. J. (1992). When to stop a clinical trial.British Medical Journal,305, 235–240.

    Article  PubMed  Google Scholar 

  28. Proschan, M. A., Follmann, D. A., &Waclawiw, M. A. (1992). Effects of assumption violations on Type I error rate in group sequential monitoring.Biometrics,48, 1131–1143.

    Article  Google Scholar 

  29. Siegmund, D. (1985).Sequential analysis: Tests and confidence intervals. New York: Springer-Verlag.

    Google Scholar 

  30. Slud, E., &Wei, L. J. (1982). Two sample repeated significance tests based on the modified Wilcoxon statistic.Journal of the American Statistical Association,77, 862–868.

    Article  Google Scholar 

  31. Sobel, M., &Wald, A. (1949). A sequential decision procedure for choosing one of three hypotheses concerning the unknown mean of a normal distribution.Annals of Mathematical Statistics,20, 502–522.

    Article  Google Scholar 

  32. Sterling, T. D., Rosenbaum, W. L., &Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa.American Statistician,49, 108–112.

    Article  Google Scholar 

  33. Wald, W. (1947).Sequential analysis. New York: Dover.

    Google Scholar 

  34. Weiss, L. (1953). Testing one simple hypothesis against another.Annals of Mathematical Statistics,24, 273–281.

    Article  Google Scholar 

  35. Whitehead, J., &Brunier, H. (1990). The double triangular test: A sequential test for the two-sided alternative with early stopping under the null hypothesis.Sequential Analysis,9, 117–136.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Robert W. Frick.

Additional information

I thank Ira Bernstein, Peter Dixon, Arthur Samuel, and anonymous reviewers for their suggestions.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Frick, R.W. A better stopping rule for conventional statistical tests. Behavior Research Methods, Instruments, & Computers 30, 690–697 (1998). https://doi.org/10.3758/BF03209488

Download citation

Keywords

  • Null Hypothesis
  • Monte Carlo Simulation
  • Sequential Probability Ratio Test
  • High Criterion
  • Multiple Statistical Test