Skip to main content
SpringerLink
Log in

Permutation Statistical Methods

  • Chapter
  • First Online:
The Measurement of Association

  • 830 Accesses

Abstract

This chapter provides an introduction to two models of statistical inference—the population model and the permutation model—and the three main approaches to permutation statistical methods—exact, moment approximation, and Monte Carlo resampling-approximation. Advantages of permutation statistical methods are described and recursion techniques are described and illustrated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For a brief overview of the development of permutation statistical methods, see a 2011 article in Wiley Indisciplinary Reviews: Computational Statistics by Berry , Johnston , and Mielke [9]. For a comprehensive history of the development of permutation statistical methods, see Berry , Johnston , and Mielke A Chronicle of Permutation Statistical Methods: 1920–2000, and Beyond [10].

  2. 2.

    In the literature, the terms “permutation” and “randomization” are often used interchangeably [39, p. 910].

  3. 3.

    The Mehta –Patel network enumeration algorithm cleverly circumvents the need to completely enumerate all possible arrangements of the data, yet still provides an exact probability value [105, 106].

  4. 4.

    For discussions of permutation methods applied to tests of differences, see Permutation Statistical Methods: An Integrated Approach by Berry, Mielke, and Johnston [14].

  5. 5.

    Note that, for these data, the sample standard deviations, s 1 = 7.4081 and s 2 = 7.0321, are very similar.

  6. 6.

    Also see a 1958 article in The Journal of the American Statistical Association by Chung and Fraser on “Randomization tests for a multivariate two-sample problem” [24].

  7. 7.

    Maxim Mersenne (1588–1648) was a Parisian monk, music theorist, and mathematician. Mersenne was the first to observe that if 2n − 1 was a prime number, then n must also be a prime number, but that the converse was not necessarily true. The Mersenne Twister pseudorandom number generator is named in his honor.

  8. 8.

    Again, note that the sample standard deviations, s 1 = 7.6408 and s 2 = 6.1270, are very similar.

  9. 9.

    In mathematics, the gamma function Γ(n) may be thought of as an extension of the factorial function to real and complex number arguments. If n is a positive integer, Γ(n) = (n − 1)! and n! =  Γ(n + 1).

  10. 10.

    Note that large values of F correspond to small values of δ.

  11. 11.

    If random sampling from a population has been accomplished, permutation tests can then provide inferences to the specified population.

  12. 12.

    In November 2015 Higgins reported that response rates for telephone surveys had fallen to less than 10% in 2015, from more than 80% in 1970 [65, p. 30].

  13. 13.

    There were only 48 states in 1936. Alaska and Hawaii were added in 1959.

  14. 14.

    Emphasis in the original.

  15. 15.

    Emphasis in the original.

  16. 16.

    Emphasis in the original.

  17. 17.

    Emphasis in the original.

  18. 18.

    Emphasis in the original.

  19. 19.

    Emphasis in the original.

  20. 20.

    The 1973 Feinstein article was the 23rd in a series of informative summary articles on statistical methods for clinical researchers published in Clinical Pharmacology and Therapeutics. A collection of 29 of the articles written by Feinstein is available in Clinical Biostatistics where this article was retitled “Permutation tests and ‘statistical significance’ ” [40].

  21. 21.

    Here, Feinstein utilized permutation tests as the gold standard against which to evaluate classical tests, referencing a 1963 article by McHugh [104] and 1966 articles by Baker and Collier [3], and Edgington [33].

  22. 22.

    In this second advantage, Feinstein clearly described the data-dependent nature of permutation tests, anticipating by many years later research on permutation methods.

  23. 23.

    In the literature of mathematical statistics there are examples of distributions where a non-parametric test that “throws away information” is clearly superior to a parametric test; see, for example, articles by Festinger in 1946 [41], Pitman in 1948 [124], Whitney in 1948 [144], and van den Brink and van den Brink in 1989 [141].

  24. 24.

    Emphasis in the original.

  25. 25.

    See also an informative and engaging 2012 article on this topic by Megan Higgs in American Scientist [66].

  26. 26.

    One petaflops indicates a quadrillion operations per second, or a 1 with 15 zeroes following it.

  27. 27.

    In keeping with convention, “permutation methods” is used throughout this book.

  28. 28.

    Gray code, after Frank Gray, or reflected binary code (RBC), is an encoding of numbers such that adjacent numbers have a single digit differing by 1.

  29. 29.

    A recursive process is one in which items are defined in terms of items of similar kind. Using a recursive relation, a class of items can be constructed from one or a few initial values (a base) and a small number of relationships (rules). For example, given the base, F 0 = 0 and F 1 = F 2 = 1, the Fibonacci series {0,  1,  1,  2,  3,  5,  8,  13,  21, …} can be constructed by the recursive rule F n = F n−1 + F n−2 for n > 2.

  30. 30.

    Attribution of the series is generally given to James Stirling, but more likely was first determined by Abraham de Moivre [116, p. 25].

  31. 31.

    The letter F for the analysis of variance (variance-ratio) test statistic was introduced in 1934 by George Snedecor at Iowa State University, much to the displeasure of R.A. Fisher [131, p. 15]. Prior to 1934 the test statistic was indicated by z, the letter originally assigned to it by Fisher .

References

  1. Altman, D.G., Bland, J.M.: Measurement in medicine: The analysis of method comparison studies. Statistician 32, 307–317 (1983)

    Article  Google Scholar 

  2. Bakeman, R., Robinson, B.F., Quera, V.: Testing sequential association: Estimating exact p values using sampled permutations. Psychol. Methods 1, 4–15 (1996)

    Article  Google Scholar 

  3. Baker, F.B., Collier, Jr., R.O.: Some empirical results on variance ratios under permutation in the completely randomized design. J. Am. Stat. Assoc. 61, 813–820 (1966)

    Article  Google Scholar 

  4. Barboza, D., Markoff, J.: Power in numbers: China aims for high-tech primacy. NY Times 161, D2–D3 (6 Dec 2011)

    Google Scholar 

  5. Barnard, G.A.: 2 × 2 tables. A note on E. S. Pearson’s paper. Biometrika 34, 168–169 (1947)

    Google Scholar 

  6. Bartlett, M.S.: Properties of sufficiency and statistical tests. P. Roy. Soc. Lond. A Mat. 160, 268–282 (1937)

    Article  MATH  Google Scholar 

  7. Berkson, J.: Some difficulties of interpretation encountered in the application of the chi-square test. J. Am. Stat. Assoc. 33, 526–536 (1938)

    Article  MATH  Google Scholar 

  8. Bernardin, H.J., Beatty, R.W.: Performance Appraisal: Assessing Human Behavior at Work. Kent, Boston (1984)

    Google Scholar 

  9. Berry, K.J., Johnston, J.E., Mielke, P.W.: Permutation methods. Comput. Stat. 3, 527–542 (2011)

    Article  Google Scholar 

  10. Berry, K.J., Johnston, J.E., Mielke, P.W.: A Chronicle of Permutation Statistical Methods: 1920–2000 and Beyond. Springer–Verlag, Cham, CH (2014)

    Chapter  MATH  Google Scholar 

  11. Berry, K.J., Mielke, P.W.: Subroutines for computing exact chi-square and Fisher’s exact probability tests. Educ. Psychol. Meas. 45, 153–159 (1985)

    Article  Google Scholar 

  12. Berry, K.J., Mielke, P.W.: Exact chi-square and Fisher’s exact probability test for 3 by 2 cross-classification tables. Educ. Psychol. Meas. 47, 631–636 (1987)

    Article  Google Scholar 

  13. Berry, K.J., Mielke, P.W.: Monte Carlo comparisons of the asymptotic chi-square and likelihood-ratio tests with the nonasymptotic chi-square test for sparse R by C tables. Psychol. Bull. 103, 256–264 (1988)

    Article  Google Scholar 

  14. Berry, K.J., Mielke, P.W., Johnston, J.E.: Permutation Statistical Methods: An Integrated Approach. Springer–Verlag, Cham, CH (2016)

    Google Scholar 

  15. Biondini, M.E., Mielke, P.W., Berry, K.J.: Data-dependent permutation techniques for the analysis of ecological data. Vegetatio 75, 161–168 (1988). [The name of the journal was changed to Plant Ecology in 1997]

    Google Scholar 

  16. Bradbury, I.: Analysis of variance versus randomization—A comparison. Brit. J. Math. Stat. Psy. 40, 177–187 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  17. Bradley, I.: Analysis of variance versus randomization tests—a comparison. Brit. J. Math. Stat. Psy. 40, 177–187 (1987)

    Article  MathSciNet  Google Scholar 

  18. Bradley, J.V.: Distribution-Free Statistical Tests. Prentice–Hall, Englewood Cliffs, NJ (1968)

    Google Scholar 

  19. Bradley, J.V.: A common situation conducive to bizarre distribution shapes. Am. Stat. 31, 147–150 (1977)

    Google Scholar 

  20. Brillinger, D.R., Jones, L.V., Tukey, J.W.: The role of statistics in weather resources management. Tech. Rep. II, Weather Modification Advisory Board, United States Department of Commerce, Washington, DC (1978)

    Google Scholar 

  21. Bross, I.D.J.: Is there an increased risk? Fed. Proc. 13, 815–819 (1954)

    Google Scholar 

  22. Bryson, M.C.: The Literary Digest poll: Making of a statistical myth. Am. Stat. 30, 184–185 (1976)

    Google Scholar 

  23. Chen, R.S., Dunlap, W.P.: SAS procedures for approximate randomization tests. Beh. Res. Meth. Ins. C 25, 406–409 (1993)

    Article  Google Scholar 

  24. Chung, J.H., Fraser, D.A.S.: Randomization tests for a multivariate two-sample problem. J. Am. Stat. Assoc. 53, 729–735 (1958)

    Article  MATH  Google Scholar 

  25. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)

    Article  Google Scholar 

  26. Cowles, M.: Statistics in Psychology: An Historical Perspective, 2nd edn. Lawrence Erlbaum, Mahwah, NJ (2001)

    MATH  Google Scholar 

  27. Curran-Everett, D.: Explorations in statistics: Standard deviations and standard errors. Adv. Physiol. Educ. 32, 203–208 (2008)

    Article  Google Scholar 

  28. Dawson, R.B.: A simplified expression for the variance of the χ 2 function on a contingency table. Biometrika 41, 280 (1954)

    MathSciNet  Google Scholar 

  29. Draper, D., Hodges, J.S., Mallows, C.L., Pregibon, D.: Exchangeability and data analysis. J. R. Stat. Soc. A Stat. 156, 9–37 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  30. Dwass, M.: Modified randomization tests for nonparametric hypotheses. Ann. Math. Stat. 28, 181–187 (1957)

    Article  MathSciNet  MATH  Google Scholar 

  31. Eden, T., Yates, F.: On the validity of Fisher’s z test when applied to an actual example of non-normal data. J. Agric. Sci. 23, 6–17 (1933)

    Article  Google Scholar 

  32. Edgington, E.S.: Randomization tests. J. Psychol. 57, 445–449 (1964)

    Article  Google Scholar 

  33. Edgington, E.S.: Statistical inference and nonrandom samples. Psychol. Bull. 66, 485–487 (1966)

    Article  Google Scholar 

  34. Edgington, E.S.: Approximate randomization tests. J. Psychol. 72, 143–149 (1969)

    Article  Google Scholar 

  35. Edgington, E.S.: Statistical Inference: The Distribution-free Approach. McGraw–Hill, New York (1969)

    Google Scholar 

  36. Edgington, E.S.: Randomization Tests. Marcel Dekker, New York (1980)

    MATH  Google Scholar 

  37. Edgington, E.S., Onghena, P.: Randomization Tests, 4th edn. Chapman & Hall/CRC, Boca Raton, FL (2007)

    Google Scholar 

  38. Editorial: Save the census. NY Times 166, A18 (17 July 2017)

    Google Scholar 

  39. Feinstein, A.R.: Clinical Biostatistics XXIII: The role of randomization in sampling, testing, allocation, and credulous idolatry (Part 2). Clin. Pharmacol. Ther. 14, 898–915 (1973)

    Article  Google Scholar 

  40. Feinstein, A.R.: Clinical Biostatistics. C. V. Mosby, St. Louis (1977)

    Article  Google Scholar 

  41. Festinger, L.: The significance of differences between means without reference to the frequency distribution function. Psychometrika 11, 97–105 (1946)

    Article  MathSciNet  MATH  Google Scholar 

  42. Fisher, R.A.: Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh (1925)

    MATH  Google Scholar 

  43. Fisher, R.A.: The Design of Experiments. Oliver and Boyd, Edinburgh (1935)

    Google Scholar 

  44. Fisher, R.A.: The logic of inductive inference (with discussion). J. R. Stat. Soc. 98, 39–82 (1935)

    Article  MATH  Google Scholar 

  45. Fox, J.A., Tracy, P.E.: Randomized Response: A Method for Sensitive Surveys. Sage, Beverly Hills, CA (1986)

    Book  Google Scholar 

  46. Frick, R.W.: Interpreting statistical testing: Process and propensity, not population and random sampling. Beh. Res. Meth. Ins. C 30, 527–535 (1998)

    Article  Google Scholar 

  47. Friedman, M.: A comparison of alternative tests of significance for the problem of m rankings. Ann. Math. Stat. 11, 86–92 (1940)

    Article  MathSciNet  MATH  Google Scholar 

  48. Galton, F.: Statistics by intercomparison, with remarks on the law of frequency of error. Philos. Mag. 4 49(322), 33–46 (1875)

    Google Scholar 

  49. Geary, R.C.: Some properties of correlation and regression in a limited universe. Metron 7, 83–119 (1927)

    MATH  Google Scholar 

  50. Geary, R.C.: Testing for normality. Biometrika 34, 209–242 (1947)

    Article  MathSciNet  MATH  Google Scholar 

  51. Gelman, A., Goel, S., Rivers, D., Rothschild, D.: The mythical swing voter. Quart. J. Pol. Sci. 11, 103–130 (2016)

    Article  Google Scholar 

  52. Good, I.J.: Further comments concerning the lady tasting tea or beer: P-values and restricted randomization. J. Stat. Comput. Simul. 40, 263–267 (1992)

    Article  Google Scholar 

  53. Good, P.I.: Permutation, Parametric and Bootstrap Tests of Hypotheses. Springer–Verlag, New York (1994)

    Book  MATH  Google Scholar 

  54. Good, P.I.: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer–Verlag, New York (1994)

    Book  MATH  Google Scholar 

  55. Good, P.I.: Resampling Methods: A Practical Guide to Data Analysis. Birkhäuser, Boston (1999)

    Book  MATH  Google Scholar 

  56. Good, P.I.: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses, 2nd edn. Springer–Verlag, New York (2000)

    Book  MATH  Google Scholar 

  57. Haldane, J.B.S.: The exact value of the moments of the distribution of χ 2, used as a test of goodness of fit, when expectations are small. Biometrika 29, 133–143 (1937). [Correction: Biometrika 31, 220 (1939)]

    Article  MATH  Google Scholar 

  58. Haldane, J.B.S.: The mean and variance of χ 2, when used as a test of goodness of fit, when expectations are small. Biometrika 31, 346–355 (1940)

    MathSciNet  MATH  Google Scholar 

  59. Havlicek, L.L., Peterson, N.L.: Robustness of the t test: A guide for researchers on effect of violations of assumptions. Psych. Rep. 34, 1095–1114 (1974)

    Article  Google Scholar 

  60. Hayes, A.F.: Permustat: Randomization tests for the Macintosh. Beh. Res. Meth. Ins. C 28, 473–475 (1996)

    Article  Google Scholar 

  61. Hayes, A.F.: Permutation test is not distribution-free: Testing H 0: ρ = 0. Psychol. Method 1, 184–198 (1996)

    Article  Google Scholar 

  62. Hayes, A.F.: Randomization tests and the equality of variance assumption when comparing group means. Anim. Behav. 59, 653–656 (2000)

    Article  Google Scholar 

  63. Hays, W.L.: Statistics. Hold, Rinehart and Winston, New York (1988)

    Google Scholar 

  64. Henley, S.: Nonparametric Geostatistics. Applied Science, London (1981)

    Book  MATH  Google Scholar 

  65. Higgins, T.: The polling industry cuts the cord. Bloomberg Businessweek November 23–29, 30 (2015)

    Google Scholar 

  66. Higgs, M.D.: Do we really need the S-word? Am. Sci. 101, 6–8 (2013). http://www.americanscientist.org/issues/pub/2013/1/do-we-really-need-the-s-word (2013). Accessed 4 Jan 2013

  67. Hilbert, M.: How much information is there in the “information society”? Significance 9, 8–12 (2012)

    Article  Google Scholar 

  68. Howell, D.C.: Statistical Methods for Psychology, 6th edn. Wadsworth, Belmont, CA (2007)

    Google Scholar 

  69. Hubbard, R.: Alphabet soup: Blurring the distinctions between p’s and α’s in psychological research. Theor. Psychol. 14, 295–327 (2004)

    Article  Google Scholar 

  70. Hunter, M.A., May, R.B.: Some myths concerning parametric and nonparametric tests. Can. Psychol. 34, 384–389 (1993)

    Article  Google Scholar 

  71. Johnston, J.E., Berry, K.J., Mielke, P.W.: Permutation tests: Precision in estimating probability values. Percept. Motor Skill 105, 915–920 (2007)

    Article  Google Scholar 

  72. Kempthorne, O.: The Design and Analysis of Experiments. Wiley, New York (1952)

    MATH  Google Scholar 

  73. Kempthorne, O.: The randomization theory of experimental inference. J. Am. Stat. Assoc. 50, 946–967 (1955)

    MathSciNet  Google Scholar 

  74. Kempthorne, O.: Some aspects of experimental inference. J. Am. Stat. Assoc. 61, 11–34 (1966)

    Article  MathSciNet  Google Scholar 

  75. Kempthorne, O.: Why randomize? J. Stat. Plan. Infer. 1, 1–25 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  76. Kennedy, P.E.: Randomization tests in econometrics. J. Bus. Econ. Stat. 13, 85–94 (1995)

    MathSciNet  Google Scholar 

  77. Lachin, J.M.: Statistical properties of randomization in clinical trials. Control Clin. Trials 9, 289–311 (1988)

    Article  Google Scholar 

  78. LaFleur, B.J., Greevy, R.A.: Introduction to permutation and resampling-based hypothesis tests. J. Clin. Child Adolesc. 38, 286–294 (2009)

    Article  Google Scholar 

  79. Lange, J.: Crime as Destiny: A Study of Criminal Twins. Allen & Unwin, London (1931). [Translated by C. Haldane]

    Google Scholar 

  80. Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Wiley, New York (1986)

    Book  MATH  Google Scholar 

  81. Lehmann, E.L., Stein, C.M.: On the theory of some non-parametric hypotheses. Ann. Math. Stat. 20, 28–45 (1949)

    Article  MATH  Google Scholar 

  82. Lewis, T., Saunders, I.W., Westcott, M.: The moments of the pearson chi-squared statistic and the minimum expected value in two-way tables. Biometrika 71, 515–522 (1984). [Correction: Biometrika 76, 407 (1989)]

    Article  MathSciNet  MATH  Google Scholar 

  83. Liang, F., Liu, C., Carroll, R.J.: Stochastic approximation in Monte Carlo computation. J. Am. Stat. Assoc. 102, 305–320 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  84. Lindley, D.V., Novick, M.R.: The role of exchangeability in inference. Ann. Stat. 9, 45–58 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  85. Ludbrook, J.: Advantages of permutation (randomization) tests in clinical and experimental pharmacology and physiology. Clin. Exp. Pharmacol. P 21, 673–686 (1994)

    Article  Google Scholar 

  86. Ludbrook, J.: Issues in biomedical statistics: Comparing means by computer-intensive tests. Aust. NZ J. Surg. 65, 812–819 (1995)

    Article  Google Scholar 

  87. Ludbrook, J.: The Wilcoxon–Mann–Whitney test condemned. Brit. J. Surg. 83, 136–137 (1996)

    Article  Google Scholar 

  88. Ludbrook, J.: Statistical techniques for comparing measures and methods of measurement: A critical review. Clin. Exp. Pharmacol. P 29, 527–536 (2002)

    Article  Google Scholar 

  89. Ludbrook, J., Dudley, H.A.F.: Issues in biomedical statistics: Analyzing 2 × 2 tables of frequencies. Aust. NZ J. Surg. 64, 780–787 (1994)

    Article  Google Scholar 

  90. Ludbrook, J., Dudley, H.A.F.: Issues in biomedical statistics: Statistical inference. Aust. NZ J. Surg. 64, 630–636 (1994)

    Article  Google Scholar 

  91. Ludbrook, J., Dudley, H.A.F.: Why permutation tests are superior to t and F tests in biomedical research. Am. Stat. 52, 127–132 (1998)

    Google Scholar 

  92. Ludbrook, J., Dudley, H.A.F.: Discussion of “Why permutation tests are superior to t and F tests in biomedical research” by J. Ludbrook and H.A.F. Dudley. Am. Stat. 54, 87 (2000)

    Google Scholar 

  93. Lyons, D.: In race for fastest computer, China outpaces U.S. Newsweek 158, 57–59 (5 Dec 2011)

    Google Scholar 

  94. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology. Chapman & Hall, London (1991)

    Book  MATH  Google Scholar 

  95. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology, 2nd edn. Chapman & Hall, London (1997)

    MATH  Google Scholar 

  96. Manly, B.F.J.: Randomization, Bootstrap and Monte Carlo Methods in Biology, 3rd edn. Chapman & Hall/CRC, Boca Raton, FL (2007)

    Google Scholar 

  97. Manly, B.F.J., Francis, R.I.C.: Analysis of variance by randomization when variances are unequal. Aust. NZ J. Stat. 41, 411–429 (1999)

    Article  MATH  Google Scholar 

  98. Marascuilo, L.A., McSweeney: Nonparametric and Distribution-free methods in the Social Sciences. Brooks–Cole, Monterey, CA (1977)

    Google Scholar 

  99. Matsumoto, M., Nishimura, T.: Mersenne twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM T Model Comput. S 8, 3–30 (1998)

    Article  MATH  Google Scholar 

  100. Matthews, R.: Beautiful, but dangerous. Significance 13, 30–31 (2016)

    Article  Google Scholar 

  101. Matthews, R.: Chancing It: The Laws of Chance and How They Can Work for You. Profile Books, London (2016)

    Google Scholar 

  102. Maxim, P.S.: Quantitative Research Methods in the Social Sciences. Oxford, New York (1999)

    Google Scholar 

  103. May, R.B., Hunter, M.A.: Some advantages of permutation tests. Can. Psychol. 34, 401–407 (1993)

    Article  Google Scholar 

  104. McHugh, R.B.: Comment on “Scales and statistics: Parametric and nonparametric” by N.H. Anderson. Psychol. Bull. 60, 350–355 (1963)

    Google Scholar 

  105. Mehta, C.R., Patel, N.R.: Algorithm 643: FEXACT. A FORTRAN subroutine for Fisher’s exact test on unordered r × c contingency tables. ACM T Math. Software 12, 154–161 (1986)

    MATH  Google Scholar 

  106. Mehta, C.R., Patel, N.R.: A hybrid algorithm for Fisher’s exact test in unordered r × c contingency tables. Commun. Stat. Theor. M 15, 387–403 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  107. Micceri, T.: The unicorn, the normal curve, and other improbable creatures. Psychol. Bull. 105, 156–166 (1989)

    Article  Google Scholar 

  108. Mielke, P.W.: Some exact and nonasymptotic analyses of discrete goodness-of-fit and r-way contingency tables. In: Johnson, N.L., Balakrishnan, N. (eds.) Advances in the Theory and Practice of Statistics: A Volume in Honor of Samuel Kotz, pp. 179–192. Wiley, New York (1997)

    Google Scholar 

  109. Mielke, P.W., Berry, K.J.: Non-asymptotic inferences based on the chi-square statistic for r by c contingency tables. J. Stat. Plan Infer. 12, 41–45 (1985)

    Article  Google Scholar 

  110. Mielke, P.W., Berry, K.J.: Cumulant methods for analyzing independence of r-way contingency tables and goodness-of-fit frequency data. Biometrika 75, 790–793 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  111. Mielke, P.W., Berry, K.J.: Data-dependent analyses in psychological research. Psychol. Rep. 91, 1225–1234 (2002)

    Article  Google Scholar 

  112. Mielke, P.W., Berry, K.J.: Permutation Methods: A Distance Function Approach, 2nd edn. Springer–Verlag, New York (2007)

    MATH  Google Scholar 

  113. Mielke, P.W., Berry, K.J., Johnston, J.E.: Resampling programs for multiway contingency tables with fixed marginal frequency totals. Psychol. Rep. 101, 18–24 (2007)

    Article  Google Scholar 

  114. Mielke, P.W., Iyer, H.K.: Permutation techniques for analyzing multi-response data from randomized block experiments. Commun. Stat. Theor. M 11, 1427–1437 (1982)

    Article  MATH  Google Scholar 

  115. Murphy, K.R., Cleveland, J.: Understanding Performance Appraisal: Social, Organizational, and Goal-based Perspectives. Sage, Thousand Oaks, CA (1995)

    Google Scholar 

  116. Namias, V.: A simple derivatin of Stirling’s asymptotic series. Am. Math. Monthly 93, 25–29 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  117. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika 20A, 175–240 (1928)

    MATH  Google Scholar 

  118. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: Part II. Biometrika 20A, 263–294 (1928)

    MATH  Google Scholar 

  119. Nussbaum, B.D.: To ask or not to ask? It depends on the question. AmstatNews 481, 3–4 (July 2017)

    Google Scholar 

  120. Nussbaum, B.D.: Bigger isn’t always better when it comes to data. AmstatNews 479, 3–4 (May 2017)

    Google Scholar 

  121. Pitman, E.J.G.: Significance tests which may be applied to samples from any populations. Suppl. J. R. Stat. Soc. 4, 119–130 (1937)

    Article  MATH  Google Scholar 

  122. Pitman, E.J.G.: Significance tests which may be applied to samples from any populations: II. The correlation coefficient test. Suppl. J. R. Stat. Soc. 4, 225–232 (1937)

    Article  MATH  Google Scholar 

  123. Pitman, E.J.G.: Significance tests which may be applied to samples from any populations: III. The analysis of variance test. Biometrika 29, 322–335 (1938)

    MATH  Google Scholar 

  124. Pitman, E.J.G.: Lecture notes on non-parametric statistical inference (1948). [Unpublished lecture notes for a course given at Columbia University in 1948]

    Google Scholar 

  125. Quetelet, L.A.J.: Lettres à S. A. R. le Duc Régnant de Saxe–Cobourg et Gotha, sur la Théorie des Probabilitiés Appliquée aux Sciences Morales et Politiques. Hayez, Bruxelles (1846). [English translation, Letters Addressed to H.R.H. the Grand Duke of Saxe Coburg and Gotha on the Theory of Probabilities as Applied to the Moral and Political Sciences, by O.G. Downes and published by Charles & Edwin Layton, London, 1849]

    Google Scholar 

  126. Rew, H.: Francis galton. J. R. Stat. Soc. 85, 293–298 (1922)

    Article  Google Scholar 

  127. Rothschild, D., Goel, S.: If a poll’s margin of error is plus or minus 3 points, think 7. NY Times 166(57,377), A20 (6 October 2016)

    Google Scholar 

  128. Saal, F.E., Downey, R.G., Lahey, M.A.: Rating the ratings: Assessing the quality of rating data. Psychol. Bull. 88, 413–428 (1980)

    Article  Google Scholar 

  129. Scheffé, H.: The Analysis of Variance. Wiley, New York (1959)

    MATH  Google Scholar 

  130. Schmidt, F.L., Johnson, R.H.: Effect of race on peer ratings in an industrial situation. J. Appl. Psychol. 57, 237–241 (1973)

    Article  Google Scholar 

  131. Snedecor, G.W.: Calculation and Interpretation of Analysis of Variance and Covariance. Collegiate Press, Ames, IA (1934)

    Book  Google Scholar 

  132. Stigler, S.M.: The Seven Pillars of Statistical Wisdom. Harvard University Press, Cambridge, MA (2016)

    Book  MATH  Google Scholar 

  133. Still, A.W., White, A.P.: The approximate randomization test as an alternative to the F test in analysis of variance. Brit. J. Math. Stat. Psy. 34, 243–252 (1981)

    Article  Google Scholar 

  134. Thompson, D.W.: On Growth and Form: The Complete Revised Edition. Dover, New York (1992)

    Book  Google Scholar 

  135. Trachtman, J.N., Giambalvo, V., Dippner, R.S.: On the assumptions concerning the assumptions of a t test. J. Gen. Psych. 99, 107–116 (1978)

    Article  Google Scholar 

  136. Tracy, P.E., Fox, J.A.: The validity of randomized response for sensitive measurements. Am. Soc. Rev. 46, 187–200 (1981)

    Article  Google Scholar 

  137. Tukey, J.W.: Data analysis and behavioral science (1962). [Unpublished manuscript]

    Google Scholar 

  138. Tukey, J.W.: The future of data analysis. Ann. Math. Stat. 33, 1–67 (1962)

    Article  MathSciNet  MATH  Google Scholar 

  139. Tukey, J.W.: Randomization and re-randomization: The wave of the past in the future. In: Statistics in the Pharmaceutical Industry: Past, Present and Future. Philadelphia Chapter of the American Statistical Association (June 1988). [Presented at a Symposium in Honor of Joseph L. Ciminera held in June 1988 at Philadelphia, Pennsylvania]

    Google Scholar 

  140. Umesh, U.N., Peterson, R.A.: A critical evaluation of the randomized response method. Sociol. Method Res. 20, 104–138 (1991)

    Article  Google Scholar 

  141. van den Brink, W.P., van den Brink, S.G.L.: A comparison of the power of the t test, Wilcoxon’s test, and the approximate permutation test for the two-sample location problem. Brit. J. Math. Stat. Psy. 42, 183–189 (1989)

    Article  MATH  Google Scholar 

  142. Vuong, A.: A new chip off the old block. Denver Post 120, 1A, 16A (2 June 2013)

    Google Scholar 

  143. Warner, S.L.: Randomized response: A survey technique for eliminating evasive answer bias. J. Am. Stat. Assoc. 60, 63–69 (1965)

    Article  MATH  Google Scholar 

  144. Whitney, D.R.: A Comparison of the Power of Non-parametric Tests and Tests Based on the Normal Distribution Under Nonnormal Alternatives (1948). [Unpublished Ph.D. dissertation at The Ohio State University, Columbus, Ohio]

    Google Scholar 

  145. Yates, F.: Contingency tables involving small numbers and the χ 2 test. Suppl. J. R. Stat. Soc. 1, 217–235 (1934)

    Article  MATH  Google Scholar 

  146. Yu, K., Liang, F., Ciampa, J., Chatterjee, N.: Efficient p-value evaluation for resampling-based tests. Biostatistics 12, 582–593 (2011)

    Article  MATH  Google Scholar 

  147. Zelterman, D.: Goodness-of-fit tests for large sparse multinomial distributions. J. Am. Stat. Assoc. 82, 624–629 (1987)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Sociology, Colorado State University, Fort Collins, Colorado, USA

    Kenneth J. Berry

  2. Alexandria, Virginia, USA

    Janis E. Johnston

  3. Department of Statistics, Colorado State University, Fort Collins, Colorado, USA

    Paul W. Mielke Jr.

Authors
  1. Kenneth J. Berry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janis E. Johnston
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul W. Mielke Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Berry, K.J., Johnston, J.E., Mielke, P.W. (2018). Permutation Statistical Methods. In: The Measurement of Association. Springer, Cham. https://doi.org/10.1007/978-3-319-98926-6_2

Download citation

Publish with us

Policies and ethics

Search

Navigation