Abstract
The goal of some research studies is to demonstrate the existence of an effect. Statistical testing, withp less than .05, is one criterion for establishing the existence of this effect. In this situation, the fixedsample stopping rule, in which the number of subjects is determined in advance, is impractical and inefficient. This article presents a sequential stopping rule that is practical and about 30% more efficient: Once a minimum number of subjects is tested, stop withp less than .01 or greater than .36; otherwise, keep testing. This procedure keeps alpha at .05 and can be adjusted to fit researchers’ needs and inclinations.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Arghami, N. R., &Billard, L. (1982). A modification of a truncated partial sequential procedure.Biometrika,69, 613–618.
Arghami, N. R., &Billard, L. (1991). A partial sequentialt-test.Sequential Analysis,10, 181–197.
Armitage, P. (1957). Restricted sequential procedures.Biometrika,44, 9–26.
Billard, L., &Vagholkar, M. K. (1969). A sequential procedure for testing a null hypothesis against a two-sided alternative hypothesis.Journal of the Royal Statistical Society B,31, 285–294.
Boneau, C. A. (1960). The effects of violations of assumptions underlying thet test.Psychological Bulletin,57, 49–64.
DeMets, D. L., &Lan, K. K. G. (1984). An overview of sequential methods and their application in clinical trials.Communications in Statistics: Theory & Methods,13, 2315–2338.
Doll, R. (1982). Clinical trials: Retrospect and prospect.Statistics in Medicine,1, 337–344.
Edwards, W., Lindman, H., &Savage, L. (1963). Bayesian statistical inference for psychological research.Psychological Review,70, 193–242.
Eiting, M. H. (1991). Sequential reliability tests.Applied Psychological Measurement,15, 193–205.
Fiske, D. W., &Jones, L. V. (1954). Sequential analysis in psychological research.Psychological Bulletin,51, 264–275.
Frick, R. W. (1995). Accepting the null hypothesis.Memory & Cognition,23, 132–138.
Frick, R. W. (1996). The appropriate use of null hypothesis testing.Psychological Methods,1, 379–390.
Goodman, S. N., &Royall, R. (1988). Evidence and scientific research.American Journal of Public Health,78, 1568–1574.
Haybittle, J. L. (1971). Repeated assessment of results in clinical trials of cancer treatment.British Journal of Radiology,44, 793–797.
Hoffman, H. S. (1992). An application of sequential analysis to observerbased psychophysics.Infant Behavior & Development,15, 271–277.
Hwang, I. K. (1992). Overview of the development of sequential procedures. In K. E. Bruce (Ed.),Biopharmaceutical sequential statistical application (pp. 3–18). New York: Dekker.
Jennison, C., &Turnbull, B. W. (1991). Group sequential tests and repeated confidence intervals. In M. Ghosh & P. K. Sen (Eds.),Handbook of sequential analysis (pp. 283–311). New York: Dekker.
Lan, K. K. G., DeMets, D. L., &Halperin, M. (1984). More flexible sequential and non-sequential designs in long-term clinical trials.Communication in Statistics: Theory & Methods,13, 2339–2353.
Lan, K. K. G., Simon, R., &Halperin, M. (1982). Stochastically curtailed tests in long-term clinical trials.Communication in Statistics: Sequential Analysis,1, 207–219.
Linn, R. L., Rock, D. A., &Cleary, T. A. (1969). The development and evaluation of several programmed testing methods.Educational & Psychological Measurement,29, 129–146.
Linn, R. L., Rock, D. A., &Cleary, T. A. (1972). Sequential testing for dichotomous decisions.Educational & Psychological Measurement,32, 85–95.
Neyman, J. (1950).First course in probability and statistics. New York: Holt.
O’Brien, P. C., &Fleming, T. R. (1979). A multiple testing procedure for clinical trials.Biometrics,35, 549–556.
Park, C. (1992). An approximation method for the characteristics of the sequential probability ratio test.Sequential Analysis,11, 55–72.
Peto, R., Pike, P., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., &Smith, P. G. (1976). Design and analysis of randomized clinical trials requiring prolonged observation of each patient.British Journal of Cancer,35, 585–611.
Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials.Biometrika,64, 191–199.
Pocock, S. J. (1992). When to stop a clinical trial.British Medical Journal,305, 235–240.
Proschan, M. A., Follmann, D. A., &Waclawiw, M. A. (1992). Effects of assumption violations on Type I error rate in group sequential monitoring.Biometrics,48, 1131–1143.
Siegmund, D. (1985).Sequential analysis: Tests and confidence intervals. New York: Springer-Verlag.
Slud, E., &Wei, L. J. (1982). Two sample repeated significance tests based on the modified Wilcoxon statistic.Journal of the American Statistical Association,77, 862–868.
Sobel, M., &Wald, A. (1949). A sequential decision procedure for choosing one of three hypotheses concerning the unknown mean of a normal distribution.Annals of Mathematical Statistics,20, 502–522.
Sterling, T. D., Rosenbaum, W. L., &Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa.American Statistician,49, 108–112.
Wald, W. (1947).Sequential analysis. New York: Dover.
Weiss, L. (1953). Testing one simple hypothesis against another.Annals of Mathematical Statistics,24, 273–281.
Whitehead, J., &Brunier, H. (1990). The double triangular test: A sequential test for the two-sided alternative with early stopping under the null hypothesis.Sequential Analysis,9, 117–136.
Author information
Authors and Affiliations
Corresponding author
Additional information
I thank Ira Bernstein, Peter Dixon, Arthur Samuel, and anonymous reviewers for their suggestions.
Rights and permissions
About this article
Cite this article
Frick, R.W. A better stopping rule for conventional statistical tests. Behavior Research Methods, Instruments, & Computers 30, 690–697 (1998). https://doi.org/10.3758/BF03209488
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03209488