Abstract
The testing for an association between two categorical variables using count data is commonplace in the behavioral sciences. Here, we present evidence that influential biostatistical textbooks give contradictory and incomplete advice on good practice in the analysis of such contingency table data. We survey the statistical literature and offer guidance on such analyses. Specifically, we call for greater use of exact testing rather than tests which use an asymptotic chi-squared distribution. That is, we suggest that researchers take a conservative approach and only perform asymptotic testing where there is little doubt that it is appropriate. We recommend a specific criterion for such decision-making. Where asymptotic testing is appropriate, we recommend chi-squared over the G-test and recommend against the implementation of Yates (or any other) correction. We also provide advice on the effective use of exact testing for associations in contingency tables. Lastly, we highlight issues that need to be considered when using the commonly recommended Fisher’s exact test.
Similar content being viewed by others
References
Agresti A (1992) A survey of exact inference for contingency tables. Stat Sci 7:131–177
Agresti A (2001) Exact inference for categorical data: recent advances and continuing controversies. Stat Med 20:2709–2722
Agresti A (2007) An introduction to categorical data analysis, 2nd edn. Wiley, New York
Andres AM, Mato AS (1994) Choosing the optimal unconditioned test for comparing two independent proportions. Comput Stat Data Anal 17:555–574
Baglivo J, Oliver D, Pagano M (1988) Methods for analysis of contingency tables with large and small cell counts. J Am Stat Assoc 83:1006–1013
Bailey NJT (1995) Statistical methods in biology. Cambridge University Press, Cambridge
Barnard GA (1945) A new test for 2 × 2 tables. Nature 156:783–784
Berger RL (1994) Mehta CR & Hilton JF (1993) comment. Am Stat 48:2
Berger VW (2004) On the generation and ownership of alpha in medical studies. Control Clin Trials 25:613–619
Boschloo RB (1970) Raised conditional level of significance for the 2 × 2-table when testing the equality of two probabilities. Stat Neerl 24:1–35
Bradley DR, Bradley TD, McGrath SG, Cutcomb SD (1979) Type I error rate of the chi-square test of independence in R x C tables that have small expected frequencies. Psychol Bull 86:1290–1297
Camilli G, Hopkins KD (1978) Applicability of chi-square to 2 × 2 contingency tables with small expected cell frequencies. Psychol Bull 85:163–167
Camilli G, Hopkins KD (1979) Testing for association in 2 × 2 contingency tables with very small sample sizes. Psychol Bull 86:1011–1014
Clewer AG, Scarisbrik DH (2001) Practical statistics and experimental design for plant and crop sciences. Wiley, New York
Cochran WG (1954) Some methods for strengthening the common chi-squared tests. Biometrics 10:417–451
Conahan MA (1970) The comparative accuracy of the likelihood ratio and chi-squared as approximations to the exact multinomial test. ED.D. diss. Lehigh Univ, p 64
Cressie N, Read TRC (1989) Pearson’s chi-squared and the log-likelihood ratio statistic G2: a comparative review. Int stat rev 57:19–43
Delucchi KL (1983) The use and misuse of chi-square: Lewis & Burke revisited. Psychol Bull 94:166–176
Fisher RA (1925) Statistical methods for research workers. Oliver & Boyd, Edinburgh
Freeman GH, Halton JH (1951) Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika 38:141–149
Garamszegi LZ et al (2009) Changing philosophies and tools for statistical inferences in behavioural ecology. Behav Ecol 20:1363–1375
Haber M (1980) A comparison of some continuity corrections for the chi-squared test on 2 × 2 tables. J Am Stat Assoc 75:510–515
Haviland MG (1990) Yates’s correction for continuity and the analysis of 2 × 2 contingency-tables. Stat Med 9:363–367
Hawkins D (2005) Biomeasurement. Oxford University Press, Oxford
Hirji KF, Tan S-J, Elashoff RM (1991) A quasi-exact test for comparing 2 binomial proportions. Stat Med 10:1137–1153
Larntz K (1978) Small-sample comparisons of exact levels for chi-squared goodness-of-fit statistics. J Am Stat Assoc 73:253–263
Lawal HB, Upton GJG (1984) On the use of chi-squared as a test of independence in contingency tables with small cell expectations. Aust J Stat 26:75–85
Lawson R (2004) Small sample confidence intervals for the odds ratio. Commun Stat simul comput 33:1095–1113
Lloyd CJ (1988) Doubling the one-sided P-value in testing independence in 2 x 2 tables against a two-sided alternative. Stat Med 7:1297–1306
Lloyd CJ, Moldovan MV (2007) Unconditional efficient one-sided confidence limits for the odds ratio based on conditional likelihood. Stat Med 26:5136–5146
Lombardi CM, Hurlbert SH (2009) Misprescription and misuse of one-tailed tests. Aust Ecol 34:447–468
Lydersen S, Pradhan V, Senchaudhuri P, Laake P (2005) Comparison of exact tests for association in unordered contingency tables using standard, mid-P, and randomized test versions. J Stat Comput Simul 75:447–458
Lydersen S, Pradhan V, Senchaudhuri P, Laake P (2007) Choice of test for association in small sample unordered r x c tables. Stat Med 26:4328–4343
Lydersen S, Fagerland M, Laake P (2009) Tutorial in biostatistics: recommended tests for association in 2 × 2 tables. Stat Med 28:1159–1175
Martin Andres A, Herranz Tejedor I (1995) Is Fisher’s exact test very conservative. Comput Stat Data Anal 19:579–591
Martin Andres A, Silva Mato A, Tapia Garcia JM, Sanchez Auevedo MJ (2004) Comparing the asymptotic power of exact tests in the 2 x 2 tables. Comput Stat Data Anal 47:745–756
Mehrotra DV, Chan ISF, Berger RL (2003) A cautionary note on exact unconditional inference for a difference between two independent binomial proportions. Biometrics 59:441–450
Mehta CR, Patel NR (1983) A network algorithm for performing fisher’s exact test in r x c contingency tables. J Am Stat Assoc 78:427–434
Mehta CR, Patel NR (1986) A hybrid algorithm for Fisher’s exact test in unordered r×c contingency tables. Commun Stat Theory Methods 15:387–403
Mehta CR, Hilton JF (1993) Exact power of conditional and unconditional tests: doing beyond the 2 × 2 table. Am Stat 47:91–98
Meulepas E (1998) A two-tailed P-value for Fisher’s exact test. Biom J 40:3–10
Neuhäuser M (2004) The choice of α for one-sided tests. Drug Inf J 38:57–60
Parshall CG, Kromrey JD (1996) Tests of independence in contingency tables with small samples: a comparison of statistical power. Educ Psychol Meas 56:26–44
Quinn GP, Keough MJ (2002) Experimental design and data analysis for biologists. Cambridge University Press, Cambridge
Rice WR, Gaines SD (1994) “Heads I win, tails you lose”: testing directional alternative hypotheses in ecological and evolutionary research. Trends Ecol Evol 9:235–237
Roscoe JT, Byars JA (1971) Sample size restraints commonly imposed on the use of the chi-square statistic. J Am Stat Assoc 66:755–759
Ruxton GD, Neuhäuser M (2010) When should we use one-tailed hypothesis testing? Meth Ecol Evol 1:114–117
Senchaudhuri P, Mehta CR, Patel NR (1995) Estimating exact P-values by the method of control variates or Monte-Carlo rescue. J Am Stat Assoc 90:640–648
Sokal RR, Rohlf FJ (1995) Biometry, 3rd edn. Freeman, New York
Stephens PA, Buskirk SW, Hayward GD, del Rio CM (2005) Information theory and hypothesis testing: a call for pluralism. J Appl Ecol 42:4–12
Stephens PA, Buskirk SW, Martínez del Rio C (2007) Inference in ecology and evolution. Trends Ecol Evol 22:192–197
Suissa S, Shuster JJ (1985) Exact unconditional sample sizes for the 2 × 2 binomial trials. J R Stat Soc A 148:317–327
Thompson B (1988) Misuse of chi-square contingency table test statistics. Educ Psychol Res 8:39–49
Van der Meulen EA (2008) A nonrandomized, nonconservative version of the Fisher exact test. Commun Stat Theory Methods 37:699–708
Williams B (1993) Biostatistics. Chapman & Hall, New York
Yates F (1984) Tests of significance for 2 × 2 contingency tables. J R Stat Soc A 147:426–463
Zar JH (1999) Biostatistical analysis, 4th edn. Prentice Hall, New York
Zar JH (2008) Biostatistical analysis, 5th edn. Prentice Hall, New York
Acknowledgement
We thank the referees for the very valuable suggestions on previous versions of this article.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by L. Garamszegi
Rights and permissions
About this article
Cite this article
Ruxton, G.D., Neuhäuser, M. Good practice in testing for an association in contingency tables. Behav Ecol Sociobiol 64, 1505–1513 (2010). https://doi.org/10.1007/s00265-010-1014-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00265-010-1014-0