Effect heterogeneity and variable selection for standardizing causal effects to a target population

  • Anders HuitfeldtEmail author
  • Sonja A. Swanson
  • Mats J. Stensrud
  • Etsuji Suzuki


The participants in randomized trials and other studies used for causal inference are often not representative of the populations seen by clinical decision-makers. To account for differences between populations, researchers may consider standardizing results to a target population. We discuss several different types of homogeneity conditions that are relevant for standardization: Homogeneity of effect measures, homogeneity of counterfactual outcome state transition parameters, and homogeneity of counterfactual distributions. Each of these conditions can be used to show that a particular standardization procedure will result in an unbiased estimate of the effect in the target population, given assumptions about the relevant scientific context. We compare and contrast the homogeneity conditions, in particular their implications for selection of covariates for standardization and their implications for how to compute the standardized causal effect in the target population. While some of the recently developed counterfactual approaches to generalizability rely upon homogeneity conditions that avoid many of the problems associated with traditional approaches, they often require adjustment for a large (and possibly unfeasible) set of covariates.


Methodology Effect heterogeneity Generalizability External validity Standardization Effect measures 



The authors are grateful to Dr. Issa Dahabreh and two anonymous reviewers for suggestions that greatly improved the manuscript. Any remaining errors are our own.


The authors received no specific funding for this work. Dr. Stensrud is supported by the Research Council of Norway, Grant NFR239956/F20 - Analyzing clinical health registries: Improved software and mathematics of identifiability. Dr. Swanson is supported by NWO/ZonMw Veni Grant (91617066). Dr. Suzuki is supported by Japan Society for the Promotion of Science (KAKENHI Grant Numbers JP17K17898, JP15K08776, and JP18K10104) and The Okayama Medical Foundation. Dr. Huitfeldt was supported by the Effective Altruism Hotel Blackpool during revision of the manuscript.


  1. 1.
    Weiss NS. Generalizing from the results of randomized studies of treatment: can non-randomized studies be of help? Eur J Epidemiol. 2019;34(8):715–8.CrossRefGoogle Scholar
  2. 2.
    Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology (Cambridge, Mass.). 2004;15(5):615–25.CrossRefGoogle Scholar
  3. 3.
    Suzuki E, Tsuda T, Mitsuhashi T, Mansournia MA, Yamamoto E. Errors in causal inference: an organizational schema for systematic error and random error. Ann Epidemiol. 2016;26(11):788–93.CrossRefGoogle Scholar
  4. 4.
    Huitfeldt A, Goldstein A, Swanson SA. The choice of effect measure for binary outcomes: introducing counterfactual outcome state transition parameters. Epidemiol Methods. 2018;7(1):14.Google Scholar
  5. 5.
    Dahabreh IJ, Hernán MA. Extending inferences from a randomized trial to a target population. Eur J Epidemiol. 2019;34(8):719–22.CrossRefGoogle Scholar
  6. 6.
    VanderWeele TJ. Confounding and effect modification: distribution and measure. Epidemiol Methods. 2012;1(1):55–82.Google Scholar
  7. 7.
    Greenland S. Interpretation and estimation of summary ratios under heterogeneity. Stat Med. 1982;1(3):217–27.CrossRefGoogle Scholar
  8. 8.
    Bernard S, Neville KA, Nguyen AT, Flockhart DA. Interethnic differences in genetic polymorphisms of CYP2d6 in the U.S. population: clinical implications. Oncologist. 2006;11(2):126–35.CrossRefGoogle Scholar
  9. 9.
    Huitfeldt A, Stensrud MJ, Suzuki E. On the collapsibility of measures of effect in the counterfactual causal framework. Emerg Themes Epidemiol. 2019;16(1):1–5.CrossRefGoogle Scholar
  10. 10.
    Cochran WG. The comparison of percentages in matched samples. Biometrika. 1950;37(3/4):256–66.CrossRefGoogle Scholar
  11. 11.
    Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ Br Med J. 2003;327(7414):557–60.CrossRefGoogle Scholar
  12. 12.
    Iwasaki M, Yamamoto S, Otani T, Inoue M, Hanaoka T, Sobue T, Tsugane S, Japan Public Health Center-based Prospective Study (JPHC Study) Group. Generalizability of relative risk estimates from a well-defined population to a general population. Eur J Epidemiol. 2006;21(4):253–62.CrossRefGoogle Scholar
  13. 13.
    Poole C, Shrier I, VanderWeele TJ. Is the risk difference really a more heterogeneous measure? Epidemiology. 2015;26(5):714–8.CrossRefGoogle Scholar
  14. 14.
    Spiegelman D, VanderWeele TJ. Evaluating public health interventions: 6. modeling ratios or differences? Let the data tell us. Am J Public Health. 2017;107(7):1087–91.CrossRefGoogle Scholar
  15. 15.
    The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on Methodological Standards in Pharmacoepidemiology (Revision 7). EMA/95098/2010.Google Scholar
  16. 16.
    Sheps MC. Shall we count the living of the dead? N Engl J Med. 1958;259(25):1210–4.CrossRefGoogle Scholar
  17. 17.
    Deeks JJ. Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes. Stat Med. 2002;21(11):1575–600.CrossRefGoogle Scholar
  18. 18.
    Baker R, Jackson D. A new measure of treatment effect for random-effects meta-analysis of comparative binary outcome data. 2018. ArXiv:1806.03471.
  19. 19.
    Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). New York: Wiley; 2011.Google Scholar
  20. 20.
    Glasziou PP, Irwig LM. An evidence based approach to individualising treatment. BMJ. 1995;311(7016):1356–9.CrossRefGoogle Scholar
  21. 21.
    Lesko CR, Buchanan AL, Westreich D, Edwards JK, Hudgens MG, Cole SR. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28:553–61.CrossRefGoogle Scholar
  22. 22.
    Murray EJ, Robins JM, Seage GR, Freedberg KA, Hernán MA. A comparison of agent-based models and the parametric G-formula for causal inference. Am J Epidemiol. 2017;186(2):131–42.CrossRefGoogle Scholar
  23. 23.
    Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensity scores to assess the generalizability of results from randomized trials. J R Stat Soc Ser A Stat Soc. 2001;174(2):369–86.CrossRefGoogle Scholar
  24. 24.
    Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. Am J Epidemiol. 2010;172(1):107–15.CrossRefGoogle Scholar
  25. 25.
    Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc. 1952;47(260):663–85.CrossRefGoogle Scholar
  26. 26.
    Robins JM. Association, causation, and marginal structural models. Synthese. 1999;121(1–2):151–79.CrossRefGoogle Scholar
  27. 27.
    Robins JM, Hernán MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology (Cambridge, Mass.). 2000;11(5):550–60.CrossRefGoogle Scholar
  28. 28.
    Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017;186(8):1010–4.CrossRefGoogle Scholar
  29. 29.
    Lu H, Cole SR, Hall HI, Schisterman EF, Breger TL, Edwards JK, Westreich D. Generalizing the per-protocol treatment effect: the case of ACTG A5095. Clin Trials. 2018;16(1):52–62.CrossRefGoogle Scholar
  30. 30.
    Susukida R, Crum RM, Ebnesajjad C, Stuart EA, Mojtabai R. Generalizability of findings from randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network. Addiction. 2017;112(7):1210–9.CrossRefGoogle Scholar
  31. 31.
    Buchanan AL, Hudgens MG, Cole SR, Mollan KR, Sax PE, Daar ES, Adimora AA, Eron JJ, Mugavero MJ. Generalizing evidence from randomized trials using inverse probability of sampling weights. J R Stat Soc Ser A (Stat Soc). 2018;181(4):1193–209.CrossRefGoogle Scholar
  32. 32.
    Dahabreh IJ, Robertson SE, Tchetgen Tchetgen EJ, Stuart EA, Hernán MA. Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals. Biometrics. 2018;75(2):685–94.CrossRefGoogle Scholar
  33. 33.
    Nguyen TQ, Ebnesajjad C, Cole SR, Stuart EA. Sensitivity analysis for an unobserved moderator in RCT-to-target-population generalization of treatment effects. Ann Appl Stat. 2017;11(1):225–47.CrossRefGoogle Scholar
  34. 34.
    Breskin A, Westreich D, Cole SR, Edwards JK. Using bounds to compare the strength of exchangeability assumptions for internal and external validity. Am J Epidemiol. 2019;188(7):1355–60. Scholar
  35. 35.
    Westreich D, Edwards JK, Lesko CR, Cole SR, Stuart EA. Target validity and the hierarchy of study designs. Am J Epidemiol. 2019;188(2):438–43.CrossRefGoogle Scholar
  36. 36.
    Pearl J, Bareinboim E. Transportability of causal and statistical relations: a formal approach. In: Twenty-fifth AAAI conference on artificial intelligence, August 2011.Google Scholar
  37. 37.
    Bareinboim E, Pearl J. A general algorithm for deciding transportability of experimental results. J Causal Inference. 2013;1(1):107–34.CrossRefGoogle Scholar
  38. 38.
    Pearl J, Bareinboim E. External validity: from do-calculus to transportability across populations. Stat Sci. 2014;29(4):579–95.CrossRefGoogle Scholar
  39. 39.
    Bareinboim E, Pearl J. Causal inference and the data-fusion problem. Proc Natl Acad Sci. 2016;113(27):7345–52.CrossRefGoogle Scholar
  40. 40.
    Pearl J. Causality: models, reasoning and inference. 2nd ed. Cambridge: Cambridge University Press; 2009.CrossRefGoogle Scholar
  41. 41.
    Dahabreh IJ, Robins JM, Haneuse SJ-PA, Hernán MA. Generalizing causal inferences from randomized trials: counterfactual and graphical identification, June 2019. arXiv:1906.10792 [stat].

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Norwegian Institute of Public HealthOsloNorway
  2. 2.PharmacoEpidemiology and Drug Safety Research Group, Department of Pharmacy, and PharmaTox Strategic Initiative, Faculty of Mathematics and Natural SciencesUniversity of OsloOsloNorway
  3. 3.Department of EpidemiologyErasmus MCRotterdamNetherlands
  4. 4.Department of EpidemiologyHarvard T.H. Chan School of Public HealthBostonUSA
  5. 5.Department of BiostatisticsUniversity of OsloOsloNorway
  6. 6.Department of Epidemiology, Graduate School of Medicine, Dentistry and Pharmaceutical SciencesOkayama UniversityOkayamaJapan

Personalised recommendations