Confounder detection in linear mediation models: Performance of kernel-based tests of independence

  • Wolfgang WiedermannEmail author
  • Xintong Li


It is well-known that the identification of direct and indirect effects in mediation analysis requires strong unconfoundedness assumptions. Even when the predictor is under experimental control, unconfoundedness assumptions must be imposed on the mediator–outcome relation in order to guarantee valid indirect-effect identification. Researchers are therefore advised to test for unconfoundedness when estimating mediation effects. Significance tests to evaluate unconfoundedness usually rely on an instrumental variable (IV)—that is, a variable that is nonindependent of the explanatory variable and, at the same time, independent of all exogenous factors that affect the outcome when the explanatory variable is held constant. Because IVs may be hard to come by, the present study shows that confounders of the mediator–outcome relation can be detected without making use of IVs when variables are nonnormal. We show that kernel-based tests of independence are able to detect confounding under nonnormality. Results from a simulation study are presented that suggest that these tests perform well in terms of Type I error protection and statistical power, independent of the distribution or measurement level of the confounder. A real-world data example from the Job Search Intervention Study (JOBS II) illustrates how the presented approach can be used to minimize the risk of obtaining biased indirect-effect estimates. The data requirements and role of unconfoundedness tests as diagnostic tools are discussed. A Monte Carlo–based power analysis tool for sample size planning is also provided.


Mediation analysis Confounder Exogeneity Hilbert–Schmidt independence criterion Nonnormality 


Supplementary material

13428_2019_1230_MOESM1_ESM.docx (17 kb)
ESM 1 (DOCX 17 kb)
13428_2019_1230_MOESM2_ESM.docx (2.8 mb)
ESM 2 (DOCX 2877 kb)


  1. Ames, J. A. (2013). Accuracy and precision of an effect size and its variance from a multilevel model for cluster randomized trials: A simulation study. Multivariate Behavioral Research, 48, 592–618. Google Scholar
  2. Angrist, J., & Krueger, A. (2001). Instrumental variables and the search for identification: From supply and demand to natural experiments. Journal of Economic Perspectives, 15, 69–85. Scholar
  3. Angrist, J. D., & Pischke, J. S. (2009). Mostly harmless econometrics: An empiricist’s companion. Princeton, NJ: Princeton University Press.Google Scholar
  4. Anscombe, F. J., & Glynn, W. J. (1983). Distribution of the kurtosis statistic b2 for normal samples. Biometrika, 70, 227–234. Google Scholar
  5. Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. Google Scholar
  6. Beale, E. M. L., & Mallows, C. L. (1959). Scale mixing of symmetric distributions with zero means. Annals of Mathematical Statistics, 30, 1145–1151. Google Scholar
  7. Blanca, M. J., Arnau, J., López-Montiel, D., Bono, R., & Bendayan, R. (2013). Skewness and kurtosis in real data samples. Methodology, 9, 78–84. Google Scholar
  8. Blundell, R., & Horowitz, J. L. (2007). A non-parametric test of exogeneity. Review of Economic Studies, 74, 1035–1058. Google Scholar
  9. Bound, J., Jaeger, D. A., & Baker, R. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. Journal of the American Statistical Association, 90, 443–450. Google Scholar
  10. Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144–152. Google Scholar
  11. Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism? (Don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550–558. Google Scholar
  12. Caetano, C. (2015). A test of exogeneity without instrumental variables in models with bunching. Econometrica, 83, 1581–1600. Google Scholar
  13. Cain, M. K., Zhang, Z., & Yuan, K. H. (2017). Univariate and multivariate skewness and kurtosis for measuring nonnormality: Prevalence, influence and estimation. Behavior Research Methods, 49, 1716–1735. Google Scholar
  14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  15. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.Google Scholar
  16. Collins, L. M., Schafer, J. L., & Kam, L. M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351. Google Scholar
  17. Cox, M. G., Kisbu-Sakarya, Y., Miočević, M., & MacKinnon, D. P. (2013). Sensitivity plots for confounder bias in the single mediator model. Evaluation Review, 37, 405–431. Google Scholar
  18. D’Agostino, R. B. (1971). An omnibus test of normality for moderate and large size samples. Biometrika, 58, 341–348. Google Scholar
  19. Darmois, G. (1953). Analyse generale des liaisons stochastiques [General analysis of stochastic links]. Review of the International Statistical Institute, 21, 2–8. Google Scholar
  20. de Luna, X., & Johansson, P. (2014). Testing for the unconfoundedness assumption using an instrumental assumption. Journal of Causal Inference, 2, 187–199. Google Scholar
  21. Derogatis, L. R., Lipman, R. S., Riekles, I. C., Uhlenhuth, E. H., & Covi, L., (1974). The Hopkins Symptom Checklist (HSCL). In P. Pichot (Ed.), Psychological measurements in psychopharmacology: Modern problems in pharmacopsychiatry (Vol. 7, pp. 79–110). New York, NY: Karger.Google Scholar
  22. Dodge, Y., & Yadegari, I. (2010). On direction of dependence. Metrika, 72, 139–150. Google Scholar
  23. Donald, S. G., Hsu, Y.C., & Lieli, R. P. (2014). Testing the unconfoundedness assumption via inverse probability weighted estimators of (L)ATT. Journal of Business and Economic Statistics, 32, 395–415. Google Scholar
  24. Dong, N., & Maynard, R. (2013). PowerUp! A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental studies. Journal of Research on Educational Effectiveness, 6, 24–67. Google Scholar
  25. Entner, D., Hoyer, P. O., & Spirtes, P. (2012). Statistical test for consistent estimation of causal effects in linear non-Gaussian models. Journal of Machine Learning Research: Workshop and Conference Proceedings, 22, 364–372.Google Scholar
  26. Fox, J. (2008). Applied regression analysis and generalized linear models (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  27. Frisch, R., & Waugh, F. (1933). Partial time regressions as compared with individual trends. Econometrica, 1, 387–401. Google Scholar
  28. Fritz, M. S., & MacKinnon, D. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18, 233–239. Google Scholar
  29. Fritz, M. S., Kenny, D. A., & MacKinnon, D. P. (2016). The combined effects of measurement error and omitting confounders in single-mediator models. Multivariate Behavioral Research, 51, 681–697.Google Scholar
  30. Garreau, D. (2017). Asymptotic normality of the median heuristic. arXiv preprint. arXiv:1707.07269Google Scholar
  31. Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926. Google Scholar
  32. Greenland, S., & Morgenstern, H. (2001). Confounding in health research. Annual Review of Public Health, 22, 189–212. Google Scholar
  33. Gretton, A., Fukumizu, K., Teo, C. H., Song, L., Schölkopf, B., & Smola, A. J. (2008). A kernel statistical test of independence. In J. C. Platt, D. Koller, Y. Singer, & S. T. Roweis (Eds.), Advances in neural information processing systems (Vol. 20, pp. 585–592). La Jolla, CA: Neural Information Processing Systems Foundation.Google Scholar
  34. Hausman, J. A. (1978). Specification tests in econometrics. Econometrica, 46, 1251–1271. Google Scholar
  35. Hayes, A. F., & Scharkow, M. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science, 24, 1918–1927. Google Scholar
  36. Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.Google Scholar
  37. Hyvärinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York, NY: Wiley & Sons.Google Scholar
  38. Iacobucci, D., Saldanha, N., & Deng, X. (2007). A meditation on mediation: Evidence that structural equations models perform better than regressions. Journal of Consumer Psychology, 17, 139–153. Google Scholar
  39. Imai, K., Keele, L., & Tingley, D. (2010). A general approach to causal mediation analysis. Psychological Methods, 15, 309–334. Google Scholar
  40. Imai, K., Keele, L., & Yamamoto, T. (2010). Identification, inference, and sensitivity analysis for causal mediation effects. Statistical Science, 25, 51–71. Google Scholar
  41. Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105, 765–789. Google Scholar
  42. Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2014). Commentary: Practical implications of theoretical results for causal mediation analysis. Psychological Methods, 19, 482–487. Google Scholar
  43. Jarque, C. M., & Bera, A. K. (1980). Efficient tests for normality, homoscedasticity and serial independence of regression residuals. Economics Letters, 6, 255–259. Google Scholar
  44. Judd, C. M., & Kenny, D. A. (1981). Process analysis: Estimating mediation in treatment evaluations. Evaluation Review, 5, 602–619. Google Scholar
  45. Judd, C. M., & Kenny, D. A. (2010). Data analysis in social psychology: Recent and recurring issues. In S. T. Fiske, D. T. Gilbert, & G. Lindzey (Eds.), Handbook of social psychology (5th ed., Vol. 1, pp. 115–139). New York, NY: Wiley.Google Scholar
  46. Keele, L. (2015). Causal mediation analysis: Warning! Assumptions ahead. American Journal of Evaluation, 36, 500–513. Google Scholar
  47. Kisbu-Sakarya, Y., MacKinnon, D. P., & Miočević, M. (2014). The distribution of the product explains normal theory mediation confidence interval estimation. Multivariate Behavioral Research, 49, 261–268. Google Scholar
  48. Loeys, T., Talloen, W., Goubert, L., Moerkerke, B., & Vansteelandt, S. (2016). Assessing moderated mediation in linear models requires fewer confounding assumptions than assessing mediation. British Journal of Mathematical and Statistical Psychology, 69, 352–374. Google Scholar
  49. Lovell, M. C. (2008). A simple proof of the FWL theorem. Journal of Economic Education, 39, 88–91. Google Scholar
  50. MacKinnon, D. P. (2008). Introduction to statistical mediation analysis. New York, NY: Taylor & Francis.Google Scholar
  51. MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation, confounding, and suppression effect. Prevention Science, 1, 173–181. Google Scholar
  52. MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104. Google Scholar
  53. MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99–128. Google Scholar
  54. MacKinnon, D. P., & Pirlott, A. G. (2015). Statistical approaches for enhancing causal interpretation of the M to Y relation in mediation analysis. Personality and Social Psychology Review, 19, 30–43. Google Scholar
  55. Mauro, R. (1990). Understanding L.O.V.E. (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108, 314–329. Google Scholar
  56. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166. Google Scholar
  57. Miranda, C. S., & von Zuben, F. J. (2015). Asymmetric distributions from constrained mixtures. ArXiv preprint. arXiv:1503.06429Google Scholar
  58. Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J., & Schölkopf, B. (2016). Distinguishing cause from effect using observational data: Methods and benchmarks. Journal of Machine Learning Research, 17, 1103–1204.Google Scholar
  59. Ng, M., & Lin, J. (2016). Testing for mediation effects under non-normality and heteroscedasticity: A comparison of classic and modern methods. International Journal of Quantitative Research in Education, 3, 24–40. Google Scholar
  60. Pearl, J. (2001). Direct and indirect effects. In Proceedings of the 17th Conference on Uncertainly in Artificial Intelligence (pp. 411–420). San Francisco, CA: Morgan Kaufmann.Google Scholar
  61. Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). New York, NY: Cambridge University Press.Google Scholar
  62. Pearl, J. (2012). The causal mediation formula—A guide to the assessment of pathways and mechanisms. Prevention Science, 13, 426–436. Google Scholar
  63. Pearl, J. (2014). Interpretation and identification of causal mediation. Psychological Methods, 19, 459–481. Google Scholar
  64. Pfister, N., & Peters, J. (2017). dHSIC: Independence testing via Hilbert Schmidt independence criterion (R package version 1.1). Retrieved from
  65. Pirlott, A. G., & MacKinnon, D. P. (2016). Design approaches to experimental mediation. Journal of Experimental Social Psychology, 66, 29–38. Google Scholar
  66. R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from Google Scholar
  67. Randles, R. H., Fligner, M. A., Policello, G. E., & Wolfe, D. A. (1980). An asymptotically distribution-free test for symmetry versus asymmetry. Journal of the American Statistical Association, 75, 168–172.Google Scholar
  68. Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.Google Scholar
  69. Schölkopf, B., & Smola, A. J. (2002). Learning with kernels: Support vector machines, regularization, optimization, and beyond. Cambridge, MA: MIT Press.Google Scholar
  70. Seier, E., & Bonett, D. G. (2011). A polyplot for visualizing location, spread, skewness, and kurtosis. American Statistician, 65, 258–261. Google Scholar
  71. Sen, A., & Sen, B. (2014). Testing independence and goodness-of-fit in linear models. Biometrika, 101, 927–942. Google Scholar
  72. Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52, 591–611. Google Scholar
  73. Shimizu, S., Hoyer, P. O., Hyvärinen, A., & Kerminen, A. (2006). A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7, 2003–2030.Google Scholar
  74. Shimizu, S., Inazumi, T., Sogawa, Y., Hyvärinen, A., Kawahara, Y., Washio, T., ... Bollen, K. (2011). DirectLiNGAM: A direct method for learning a linear non-Gaussian structural equation model. Journal of Machine Learning Research, 12, 1225–1248.Google Scholar
  75. Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422–445. Google Scholar
  76. Skitovich, W. P. (1953). On a property of the normal distribution. Doklady Akademii Nauk SSSR [Reports of the Academy of Sciences USSR], 89, 217–219.Google Scholar
  77. Small, D. S. (2012). Mediation analysis without sequential ignorability: Using baseline covariates interacted with random assignment as instrumental variables. Journal of Statistical Research, 46, 91–103.Google Scholar
  78. Sriperumbudur, B., Fukumizu, K., Gretton, A., Lanckriet, G. R. G., & Schölkopf, B. (2009). Kernel choice and classifiability for RKHS embeddings of probability distributions. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, & A. Culotta (eds.), Advances in neural information processing systems 22 (pp. 1750–1758). La Jolla, CA: Neural Information Processing Systems Foundation.Google Scholar
  79. Ten Have, T. R., Joffe, M. M., Lynch, K. G., Brown, G. K., Maisto, S. A., & Beck, A. T. (2007). Causal mediation analyses with rank preserving models. Biometrics, 63, 926–934. Google Scholar
  80. VanderWeele, T. J. (2010). Bias formulas for sensitivity analysis for direct and indirect effects. Epidemiology, 21, 540–551. Google Scholar
  81. VanderWeele, T. J. (2015). Explanation in causal inference: Methods for mediation and interaction. Oxford, UK: Oxford University Press.Google Scholar
  82. Vinokur, A. D., Price, R. H., & Schul, Y. (1995). Impact of the JOBS intervention on unemployed workers varying in risk for depression. American Journal of Community Psychology, 23, 39–74. Google Scholar
  83. Vinokur, A. D., & Schul, Y. (1997). Mastery and inoculation against setbacks as active ingredients in the JOBS intervention for the unemployed. Journal of Consulting and Clinical Psychology, 65, 867–877. Google Scholar
  84. White, H., & MacDonald, G. M. (1980). Some large-sample tests for nonnormality in the linear regression model. Journal of the American Statistical Association, 75, 16–28. Google Scholar
  85. Wiedermann, W., Artner, R., & von Eye, A. (2017). Heteroscedasticity as a basis of direction dependence in reversible linear regression models. Multivariate Behavioral Research, 52, 222–241. Google Scholar
  86. Wiedermann, W., Merkle, E. C., & von Eye, A. (2018). Direction of dependence in measurement error models. British Journal of Mathematical and Statistical Psychology, 71, 117–145. Google Scholar
  87. Wiedermann, W., & Li, X. (2018). Direction dependence analysis: Testing the direction of effects in linear models with an implementation in SPSS. Behavior Research Methods, 50, 1581–1601. Google Scholar
  88. Wiedermann, W., & von Eye, A. (2015). Direction dependence analysis: A confirmatory approach for testing directional theories. International Journal of Behavioral Development, 39, 570–580. Google Scholar
  89. Wooldridge, J. M. (2015). Control function methods in applied econometrics. Journal of Human Resources, 50, 420–445. Google Scholar
  90. Yuan, Y., & MacKinnon, D. P. (2014). Robust mediation analysis based on median regression. Psychological Methods, 19, 1–20. Google Scholar
  91. Zhang, Z. (2014). Monte Carlo based statistical power analysis for mediation models: Methods and software. Behavior Research Methods, 46, 1184–1198. Google Scholar
  92. Zheng, C., Atkins, D. C., Zhou, X. H., & Rhew, I. C. (2015). Causal models for mediation analysis: An introduction to structural mean models. Multivariate Behavioral Research, 50, 614–631. Google Scholar
  93. Zu, J., & Yuan, K. H. (2010). Local influence and robust procedures for mediation analysis. Multivariate Behavioral Research, 45, 1–44. Google Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Department of Educational, School, and Counseling PsychologyUniversity of MissouriColumbiaUSA

Personalised recommendations