Abstract
Chen and Dunlap (1993) added to the growing list of papers promoting the use of randomization tests in statistical testing. Their particular contribution was an SAS program that could bring computation of these tests to a wider audience. The present paper points to several problems with the presentation of Chen and Dunlap and provides solutions to these problems. It is concluded that randomization tests deserve more attention, but that they are best computed by programs written in a low-level programming language or, if using SAS on a mainframe, by using the MULTTEST procedure.
Article PDF
Avoid common mistakes on your manuscript.
References
Boik, R. J. (1987). The Fisher-Pitman permutation test: A non-robust alternative to the normal theoryF test when variances are heterogeneous.British Journal of Mathematical & Statistical Psychology,40, 26–42.
Chen, R. S., &Dunlap, W. P. (1993). SAS procedures for approximate randomization tests.Behavior Research Methods, Instruments, & Computers,25, 406–409.
Edgington, E. S. (1965). The assumption of homogeneity of variance for thet test and nonparametric tests.Journal of Psychology,59, 177–179.
Edgington, E. S. (1987).Randomization tests (2nd ed.). New York: Marcel Dekker.
Efron, B., &Tibshirani, R. J. (1993).An introduction to the bootstrap. New York: Chapman & Hall.
Keller-McNulty, S., &Higgins, J. J. (1987). Effect of tail weight and outliers on power and type-I error of robust permutation tests for location.Communications in Statistics: Simulation & Computation,16, 17–35.
Kempthorne, O., &Doerfler, T. E. (1969). The behaviour of some significance tests under experimental randomization.Biometrika,56, 231–248.
Lehmann, E. L. (1975).Nonparametrics, statistical methods based on ranks. San Francisco: Holden-Day.
Lehmann, E. L. (1986).Testing statistical hypotheses (2nd ed.). New York: Wiley.
Manly, B. F. J. (1991).Randomization and Monte Carlo methods in biology. London: Chapman & Hall.
May, R. B., &Hunter, M. A. (1988). Interpreting students’ interpretations of research.Teaching of Psychology,15, 156–158.
May, R. B.,Hunter, M. A., &Masson, M. E. J. (1993).NPStat (Version 3.7) [Computer program]. Department of Psychology, University of Victoria (Canada).
May, R. B., Masson, M. E. J., &Hunter, M. A. (1989). Randomization tests: Viable alternatives to normal curve tests.Behavior Research Methods, Instruments, & Computers,21, 482–483.
May, R. B., Masson, M. E. J., &Hunter, M. A. (1990).Applications of statistics in behavioral research. New York: Harper & Row.
Noreen, E. W. (1989).Computer-intensive methods for testing hypotheses: An introduction. New York: Wiley.
Numerical Algorithms Group (1990).The NAG Fortran Library (Mark 14) [Computer Program]. Oxford, U.K.: Author.
Onghena, P. (1992). Randomization tests for extensions and variations of ABAB single-case experimental designs: A rejoinder.Behavioral Assessment,14, 153–171.
Onghena, P. (1994).The power of randomization tests for single-case designs. Unpublished doctoral dissertation, Department of Psychology, Katholieke Universiteit, Leuven, Belgium.
Onghena, P., &Edgington, E. S. (1994). Randomization tests for restricted alternating treatments designs.Behaviour Research & Therapy,32, 783–786.
Onghena, P., &Van Damme, G. (1994). SCRT 1.1: Single-case randomization tests.Behavior Research Methods, Instruments, & Computers,26, 369.
Rasmussen, J. L. (1986). An evaluation of parametric and nonparametric tests on modified and non-modified data.British Journal of Mathematical & Statistical Psychology,39, 213–220.
SAS Institute Inc. (1992).SAS technical report P-229, SAS/STAT software: Changes and enhancements, release 6.07. Cary, NC: Author.
Simon, J. L., &Bruce, P. (1991). Resampling: A tool for everyday statistical work.Chance,4, 22–32.
Van Damme, G., &Onghena, P. (1993).Single case randomization tests (Version 1.1) [Computer program]. Department of Psychology, Katholieke Universiteit, Leuven, Belgium.
van den Brink, W. P., &van den Brink, S. G. J. (1989). A comparison of the power of thet test, Wilcoxon’s test, and the approximate permutation test for the two-sample location problem.British Journal of Mathematical & Statistical Psychology,42, 183–189.
Westfall, P. H., &Young, S. S. (1993).Resampling-based multiple testing: Examples and methods for p-value adjustment. New York: Wiley.
Author information
Authors and Affiliations
Corresponding author
Additional information
The authors wish to thank Marc Brysbaert, Stef Decoene, Luc Delbeke, Eugene S. Edgington, and William P. Dunlap for their helpful comments on an earlier version of the manuscript. The first author is Research Assistant of the National Fund for Scientific Research of Belgium.
Rights and permissions
About this article
Cite this article
Onghena, P., May, R.B. Pitfalls in computing and interpreting randomization testp values: A commentary on Chen and Dunlap. Behavior Research Methods, Instruments, & Computers 27, 408–411 (1995). https://doi.org/10.3758/BF03200438
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03200438