Abstract
Omnibus tests of significance in contingency tables use statistics of the chi-square type. When the null is rejected, residual analyses are conducted to identify cells in which observed frequencies differ significantly from expected frequencies. Residual analyses are thus conditioned on a significant omnibus test. Conditional approaches have been shown to substantially alter type I error rates in cases involving t tests conditional on the results of a test of equality of variances, or tests of regression coefficients conditional on the results of tests of heteroscedasticity. We show that residual analyses conditional on a significant omnibus test are also affected by this problem, yielding type I error rates that can be up to 6 times larger than nominal rates, depending on the size of the table and the form of the marginal distributions. We explored several unconditional approaches in search for a method that maintains the nominal type I error rate and found out that a bootstrap correction for multiple testing achieved this goal. The validity of this approach is documented for two-way contingency tables in the contexts of tests of independence, tests of homogeneity, and fitting psychometric functions. Computer code in MATLAB and R to conduct these analyses is provided as Supplementary Material.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
A cautionary note is in order here to justify the equivalence of this alternative estimate of the empirical type I error rate p of the omnibus test. By comparison with the situation in the set of simulations whose results are presented in Fig. 1, the current set used negative binomial sampling instead of binomial sampling for the estimation. This is because tables continued to be generated until T were obtained in which the condition for rejection was satisfied, as opposed to generating T tables and observing how many of them satisfied the condition. Thus, let y be the number of failures that occurred before the required x = T successes were observed. An estimate of p under negative binomial sampling (Forbes, Evans, Hastings, & Peacock, 2011, p. 142) is thus given either by x/(x + y) or by (x − 1)/(x + y − 1), but the difference between these alternative estimates is negligible when x ≥ 200,000, as implied in our simulations. We thus used x/(x + y) to estimate empirical test size.
Note that the implied multiple testing does not ensure a familywise rejection rate of 0.05, but this is not a problem here because our goal is to assess test accuracy cellwise.
References
Abdi, H. (2007). Bonferroni test. In N. J. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 104–108). Thousand Oaks: Sage. doi:10.4135/9781412952644.n60
Aggarwal, C. C. (2013). Outlier analysis. New York: Springer.
Agresti, A. (1984). Analysis of ordinal categorical data. New York: Wiley.
Agresti, A. (1990). Categorical data analysis. New York: Wiley.
Albers, W., Boon, P. C., & Kallenberg, C. M. (2000). The asymptotic behavior of tests of normal means based on a variance pre-test. Journal of Statistical Planning and Inference, 88, 47–57. doi:10.1016/S0378-3758(99)00211-6
Albertazzi, L., Da Pos, O., Canal, L., Micciolo, R., Malfatti, M., & Vescovi, M. (2013). The hue of shapes. Journal of Experimental Psychology: Human Perception and Performance, 39, 37–47. doi:10.1037/a0028816
Alcalá-Quintana, R., & García-Pérez, M. A. (2013). Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: matlab and R routines. Behavior Research Methods, 45, 972–998. doi:10.3758/s13428-013-0325-2
Berry, K. J., & Mielke, P. W., Jr. (1986). R by C chi-square analyses with small expected cell frequencies. Educational and Psychological Measurement, 46, 169–173. doi:10.1177/0013164486461018
Berry, K. J., & Mielke, P. W., Jr. (1988). Monte Carlo comparisons of the asymptotic chi-square and likelihood-ratio tests with the nonasymptotic chi-square test for sparse r × c tables. Psychological Bulletin, 103, 256–264. doi:10.1037/0033-2909.103.2.256
Bishop, Y. M. M., Fienberg, S. E., & Holland, P. W. (1975). Discrete multivariate analysis: Theory and practice. Cambridge: MIT Press.
Brown, M. B., & Fuchs, C. (1983). On maximum likelihood estimation in sparse contingency tables. Computational Statistics and Data Analysis, 1, 3–15. doi:10.1016/0167-9473(83)90059-2
Caudill, S. B. (1988). Type I errors after preliminary tests for heteroscedasticity. The Statistician, 37, 65–68. doi:10.2307/2348380
Cressie, N., & Read, T. R. C. (1984). Multinomial goodness-of-fit tests. Journal of the Royal Statistical Society Series B, 46, 440–464.
Everitt, B. S. (1992). The analysis of contingency tables (2nd ed.). London: Chapman & Hall.
Fienberg, S. E. (2005). Contingency tables and log-linear models. In K. Kempf-Leonard (Ed.), Encyclopedia of social measurement (Vol. 1, pp. 499–506). New York: Elsevier. doi:10.1016/B0-12-369398-5/00148-1
Forbes, C., Evans, M., Hastings, N., & Peacock, B. (2011). Statistical distributions (4th ed.). New York: Wiley.
Forsberg, L., Jonsson, B., & Kristiansson, U. (1999). On Type I errors after a preliminary test for heteroscedasticity. The Statistician, 48, 63–72. doi:10.1111/1467-9884.00171
García-Pérez, M. A. (1994). Parameter estimation and goodness-of-fit testing in multinomial models. British Journal of Mathematical and Statistical Psychology, 47, 247–282. doi:10.1111/j.2044-8317.1994.tb01037.x
García-Pérez, M. A. (2012). Statistical conclusion validity: Some common threats and simple remedies. Frontiers in Psychology, 3, 325. doi:10.3389/fpsyg.2012.00325
García-Pérez, M. A., & Alcalá-Quintana, R. (2012). Response errors explain the failure of independent-channels models of perception of temporal order. Frontiers in Psychology, 3, 94. doi:10.3389/fpsyg.2012.00094
García-Pérez, M. A., & Núñez-Antón, V. (2003). Cellwise residual analysis in two-way contingency tables. Educational and Psychological Measurement, 63, 825–839. doi:10.1177/0013164403251280
García-Pérez, M. A., & Núñez-Antón, V. (2009). Accuracy of power-divergence statistics for testing independence and homogeneity in two-way contingency tables. Communications in Statistics – Simulation and Computation, 38, 503–512. doi:10.1080/03610910802538351
Haberman, S. J. (1973). The analysis of residuals in cross-classified tables. Biometrics, 29, 205–220.
Hayes, A. F., & Cai, L. (2007). Further evaluating the conditional decision rule for comparing two independent means. British Journal of Mathematical and Statistical Psychology, 60, 217–244. doi:10.1348/000711005X62576
Justice, L. M., McGinty, A. S., Zucker, T., Cabell, S. Q., & Piasta, S. B. (2013). Bi-directional dynamics underlie the complexity of talk in teacher–child play-based conversations in classrooms serving at-risk pupils. Early Childhood Research Quarterly, 28, 496–508. doi:10.1016/j.ecresq.2013.02.005
Kuhnt, S. (2004). Outlier identification procedures for contingency tables using maximum likelihood and L 1 estimates. Scandinavian Journal of Statistics, 31, 431–442. doi:10.1111/j.1467-9469.2004.02_057.x
Kuhnt, S. (2010). Breakdown concepts for contingency tables. Metrika, 71, 281–294. doi:10.1007/s00184-008-0230-3
Kuhnt, S., Rapallo, F., & Rehage, A. (2014). Outlier detection in contingency tables based on minimal patterns. Statistics and Computing. doi:10.1007/s11222-013-9382-8
Lantz, B. (2013). The impact of sample non-normality on ANOVA and alternative methods. British Journal of Mathematical and Statistical Psychology, 66, 224–244. doi:10.1111/j.2044-8317.2012.02047.x
Moser, B. K., & Stevens, G. R. (1992). Homogeneity of variance in the two-sample means test. The American Statistician, 46, 19–21. doi:10.1080/00031305.1992.10475839
Nandram, B., Bhatta, D., & Bhadra, D. (2014). A likelihood ratio test of quasi-independence for sparse two-way contingency tables. Journal of Statistical Computation and Simulation. doi:10.1080/00949655.2013.815190
Ng, M., & Wilcox, R. R. (2011). A comparison of two-stage procedures for testing least-squares coefficients under heteroscedasticity. British Journal of Mathematical and Statistical Psychology, 64, 244–258. doi:10.1348/000711010X508683
Numerical Algorithms Group (1999). NAG fortran library manual, Mark 19. Oxford: Author.
Rapallo, F. (2012). Outliers and patterns of outliers in contingency tables with algebraic statistics. Scandinavian Journal of Statistics, 39, 784–797. doi:10.1111/j.1467-9469.2012.00790.x
Rasch, D., Kubinger, K. D., & Moder, K. (2011). The two-sample t test: Pre-testing its assumptions does not pay off. Statistical Papers, 52, 219–231. doi:10.1007/s00362-009-0224-x
Read, T. R. C., & Cressie, N. A. C. (1988). Goodness-of-fit statistics for discrete multivariate data. New York: Springer.
Rochon, J., & Kieser, M. (2011). A closer look at the effect of preliminary goodness-of-fit testing for normality for the one-sample t-test. British Journal of Mathematical and Statistical Psychology, 64, 410–426. doi:10.1348/0007-1102.002003
Rom, D. M. (2013). An improved Hochberg procedure for multiple tests of significance. British Journal of Mathematical and Statistical Psychology, 66, 189–196. doi:10.1111/j.2044-8317.2012.02042.x
Savardi, U., Bianchi, I., & Bertamini, M. (2010). Naïve predictions of motion and orientation in mirrors: From what we see to what expect reflections to do. Acta Psychologica, 134, 1–15. doi:10.1016/j.actpsy.2009.11.008
Schucany, W. R., & Ng, H. K. T. (2006). Preliminary goodness-of-fit tests for normality do not validate the one-sample Student t. Communications in Statistics – Theory and Methods, 35, 2275–2286. doi:10.1080/03610920600853308
Simonoff, J. S. (1988). Detecting outlying cells in two-way contingency tables via backwards-stepping. Technometrics, 30, 339–345. doi:10.1080/00401706.1988.10488407
Simonoff, J. S. (2003). Analyzing categorical data. New York: Springer.
Stevens, W. L. (1950). Fiducial limits of the parameter of a discontinuous distribution. Biometrika, 37, 117–129.
Szczepanowski, R., Traczyk, J., Wierzchoń, M., & Cleeremans, A. (2013). The perception of visual emotion: Comparing different measures of awareness. Consciousness and Cognition, 22, 212–220. doi:10.1016/j.concog.2012.12.003
Thorley, C., & Rushton-Woods, J. (2013). Blame conformity: Leading eyewitness statements can influence attributions of blame for an accident. Applied Cognitive Psychology, 27, 291–296. doi:10.1002/acp.2906
Tompkins, V., Zucker, T. A., Justice, L. M., & Binici, S. (2013). Inferential talk during teacher–child interactions in small-group play. Early Childhood Research Quarterly, 28, 424–436. doi:10.1016/j.ecresq.2012.11.001
van Eijk, R. L. J., Kohlrausch, A., Juola, J. F., & van de Par, S. (2008). Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception & Psychophysics, 70, 955–968. doi:10.3758/PP.70.6.955
Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & Psychophysics, 63, 1293–1313. doi:10.3758/BF03194544
Wickens, T. D. (1989). Multiway contingency tables analysis for the social sciences. Hillsdale: Erlbaum.
Wickens, T. D. (1998). Categorical data analysis. Annual Review of Psychology, 49, 537–558. doi:10.1146/annurev.psych.49.1.537
Wilcox, R. R. (1998). How many discoveries have been lost by ignoring modern statistical methods? American Psychologist, 53, 300–314. doi:10.1037/0003-066X.53.3.300
Zimmerman, D. W. (1996). Some properties of preliminary tests of equality of variances in the two-sample location problem. Journal of General Psychology, 123, 217–231. doi:10.1080/00221309.1996.9921274
Zimmerman, D. W. (2004). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57, 173–181. doi:10.1348/000711004849222
Zimmerman, D. W. (2011). A simple and effective decision rule for choosing a significance test to protect against non-normality. British Journal of Mathematical and Statistical Psychology, 64, 388–409. doi:10.1348/000711010X524739
Acknowledgments
This research was supported by grants PSI2009-08800 (Ministerio de Ciencia e Innovación), PSI2012-32903 (Ministerio de Economía y Competitividad), MTM2010-14913 (Ministerio de Ciencia e Innovación and FEDER), US12/09 (Universidad del País Vasco UPV/EHU), and IT-642-13 and UFI11/03 (Departamento de Educación del Gobierno Vasco - UPV/EHU Econometrics Research Group).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
ESM 1
(ZIP 411 kb)
Rights and permissions
About this article
Cite this article
García-Pérez, M.A., Núñez-Antón, V. & Alcalá-Quintana, R. Analysis of residuals in contingency tables: Another nail in the coffin of conditional approaches to significance testing. Behav Res 47, 147–161 (2015). https://doi.org/10.3758/s13428-014-0472-0
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13428-014-0472-0