Skip to main content

Advertisement

SpringerLink
Log in

Target Validity: Bringing Treatment of External Validity in Line with Internal Validity

  • Epidemiologic Methods (P Howards, Section Editor)
  • Published:
Current Epidemiology Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

“Target bias” is the difference between an estimate of association from a study sample and the causal effect in the target population of interest. It is the sum of internal and external bias. Given the extensive literature on internal validity, here, we review threats and methods to improve external validity.

Recent Findings

External bias may arise when the distribution of modifiers of the effect of treatment differs between the study sample and the target population. Methods including those based on modeling the outcome, modeling sample membership, and doubly robust methods are available, assuming data on the target population is available.

Summary

The relevance of information for making policy decisions is dependent on both the actions that were studied and the sample in which they were evaluated. Combining methods for addressing internal and external validity can improve the policy relevance of study results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. Olsen RB, Orr LL, Bell SH, Stuart EA. External validity in policy evaluations that choose sites purposively. J Policy Anal Manag. 2013;32(1):107–21.

    Google Scholar 

  2. Edwards JK, Lesko CR, Keil AP. Invited commentary: causal inference across space and time-quixotic quest, worthy goal, or both? Am J Epidemiol. 2017;186(2):143–5.

    PubMed  PubMed Central  Google Scholar 

  3. US Preventive Services Task Force United States. Office of Disease Prevention Health Promotion. Guide to clinical preventive services: report of the US Preventive Services Task Force: US Department of Health and Human Services, Office of Public Health and …; 1996.

  4. Susukida R, Crum RM, Ebnesajjad C, Stuart EA, Mojtabai R. Generalizability of findings from randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network. Addiction. 2017;112(7):1210–9.

    PubMed  PubMed Central  Google Scholar 

  5. Greenhouse JB, Kaizar EE, Kelleher K, Seltman H, Gardner W. Generalizing from clinical trial data: a case study. The risk of suicidality among pediatric antidepressant users. Stat Med. 2008;27(11):1801–13.

    PubMed  PubMed Central  Google Scholar 

  6. Weisberg HI, Hayden VC, Pontes VP. Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clinical Trials. 2009;6(2):109–18.

    PubMed  Google Scholar 

  7. •• Westreich D, Edwards JK, Lesko CR, Cole SR, Stuart EA. Target validity and the hierarchy of study designs. Am J Epidemiol. 2019;188(2):438–43 Introduces the term target validity and defines target validity both formally and conceptually.

    PubMed  Google Scholar 

  8. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin; 2001. xxi, 623 p. p.

  9. Imai K, King G, Stuart EA. Misunderstandings between experimentalists and observationalists about causal inference. J R Stat Soc a Stat. 2008;171:481–502.

    Google Scholar 

  10. Rothman KJ, Greenland S, Lash TL. Modern epidemiology. 3rd ed. Philadelphia, PA: Wolters Kluwer Health/Lippincott Williams & Wilkins; 2008. x, 758 p. p.

  11. Curno MJ, Rossi S, Hodges-Mameletzis I, Johnston R, Price MA, Heidari S. A systematic review of the inclusion (or exclusion) of women in HIV research: from clinical studies of antiretrovirals and vaccines to cure strategies. J Acquir Immune Defic Syndr. 2016;71(2):181–8.

    CAS  PubMed  Google Scholar 

  12. Geller SE, Koch AR, Roesch P, Filut A, Hallgren E, Carnes M. The more things change, the more they stay the same: a study to evaluate compliance with inclusion and assessment of women and minorities in randomized controlled trials. Acad Med. 2018;93(4):630–5.

    PubMed  PubMed Central  Google Scholar 

  13. Green BL, Maisiak R, Wang MQ, Britt MF, Ebeling N. Participation in health education, health promotion, and health research by African Americans: effects of the Tuskegee Syphilis Experiment. J Health Educ. 1997;28(4):196–201.

    Google Scholar 

  14. Susukida R, Crum RM, Stuart EA, Ebnesajjad C, Mojtabai R. Assessing sample representativeness in randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network. Addiction. 2016;111(7):1226–34.

    PubMed  PubMed Central  Google Scholar 

  15. • Breskin A, Westreich D, Cole SR, Edwards JK. Using bounds to compare the strength of exchangeability assumptions for internal and external validity. Am J Epidemiol. 2019;188(7):1355–60 Derives nonparametric bounds to compare strengths of assumptions for internal and external validity under which a TATE (causal risk difference) can be estimated from an associational risk difference.

    PubMed  PubMed Central  Google Scholar 

  16. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92.

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Lesko CR, Jacobson LP, Althoff KN, Abraham AG, Gange SJ, Moore RD, et al. Collaborative, pooled and harmonized study designs for epidemiologic research: challenges and opportunities. Int J Epidemiol. 2018;47(2):654–68.

    PubMed  PubMed Central  Google Scholar 

  18. Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. Am J Epidemiol. 2010;172(1):107–15.

    PubMed  PubMed Central  Google Scholar 

  19. Ackerman B, Schmid I, Rudolph KE, Seamans MJ, Susukida R, Mojtabai R, et al. Implementing statistical methods for generalizing randomized trial findings to a target population. Addict Behav. 2019;94:124–32.

    PubMed  Google Scholar 

  20. Buchanan AL, Hudgens MG, Cole SR, Mollan K, Sax PE, Daar ES, et al. Generalizing evidence from randomized trials using inverse probability of sampling weights. Technical Report Series 2015.

  21. • Dahabreh IJ, Robertson SE, Tchetgen EJ, Stuart EA, Hernan MA. Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals. Biometrics. 2019;75(2):685–94 Summary of methods and review of conditions for estimating a TATE from RCT data.

    PubMed  Google Scholar 

  22. Lesko CR, Buchanan AL, Westreich D, Edwards JK, Hudgens MG, Cole SR. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28(4):553–61.

    PubMed  PubMed Central  Google Scholar 

  23. Stuart EA, Ackerman B, Westreich D. Generalizability of randomized trial results to target populations: design and analysis possibilities. Res Soc Work Pract. 2018;28(5):532–7.

    PubMed  Google Scholar 

  24. Stuart EA, Bradshaw CP, Leaf PJ. Assessing the generalizability of randomized trial results to target populations. Prev Sci. 2015;16(3):475–85.

    PubMed  PubMed Central  Google Scholar 

  25. Hernán MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011;22(3):368–77.

    PubMed  PubMed Central  Google Scholar 

  26. Bareinboim E, Pearl J. A general algorithm for deciding transportability of experimental results. J Causal Inference. 2013;1(1):107–34.

    Google Scholar 

  27. Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017;186(8):1010–4.

    PubMed  PubMed Central  Google Scholar 

  28. Dahabreh IJ, Hernán MA. Extending inferences from a randomized trial to a target population. Eur J Epidemiol. 2019;34(8):719–22.

    CAS  PubMed  Google Scholar 

  29. Pearl J. Generalizing experimental findings. J Causal Inference. 2015;3(2):259–66.

    Google Scholar 

  30. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82–93.

    PubMed  Google Scholar 

  31. Rothwell PM. Commentary: external validity of results of randomized trials: disentangling a complex concept. Int J Epidemiol. 2010;39(1):94–6.

    PubMed  Google Scholar 

  32. Dekkers OM, von Elm E, Algra A, Romijn JA, Vandenbroucke JP. How to assess the external validity of therapeutic trials: a conceptual approach. Int J Epidemiol. 2010;39(1):89–94.

    CAS  PubMed  Google Scholar 

  33. Braslow JT, Duan N, Starks SL, Polo A, Bromley E, Wells KB. Generalizability of studies on mental health treatment and outcomes, 1981 to 1996. Psychiatric services (Washington, DC). 2005;56(10):1261–8.

  34. Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA - J Am Med Assoc. 2007;297(11):1233–40.

    Google Scholar 

  35. Sokka T, Pincus T. Eligibility of patients in routine care for major clinical trials of anti–tumor necrosis factor α agents in rheumatoid arthritis. Arthritis Rheum. 2003;48(2):313–8.

    CAS  PubMed  Google Scholar 

  36. Hoertel N, Le Strat Y, Blanco C, Lavaud P, Dubertret C. Generalizability of clinical trial results for generalized anxiety disorder to community samples. Depress Anxiety. 2012;29(7):614–20.

    PubMed  Google Scholar 

  37. Blanco C, Olfson M, Okuda M, Nunes EV, Liu S-M, Hasin DS. Generalizability of clinical trials for alcohol dependence to community samples. Drug Alcohol Depend. 2008;98(1–2):123–8.

    PubMed  PubMed Central  Google Scholar 

  38. Steg PG, López-Sendón J, de Sa EL, Goodman SG, Gore JM, Anderson FA, et al. External validity of clinical trials in acute myocardial infarction. Arch Intern Med. 2007;167(1):68–73.

    PubMed  Google Scholar 

  39. Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensity scores to assess the generalizability of results from randomized trials. J R Stat Soc Ser A Stat Soc. 2011;174(2):369–86.

    Google Scholar 

  40. Tipton E. How generalizable is your experiment? An index for comparing experimental samples and populations. J Educ Behav Stat. 2014;39(6):478–501.

    Google Scholar 

  41. Ebrahim S, Davey SG. Commentary: should we always deliberately be non-representative? Int J Epidemiol. 2013;42(4):1022–6.

    PubMed  Google Scholar 

  42. Richiardi L, Pizzi C, Pearce N. Commentary: representativeness is usually not necessary and often should be avoided. Int J Epidemiol. 2013;42(4):1018–22.

    PubMed  Google Scholar 

  43. Rothman K, Hatch E, Gallacher J. Representativeness is not helpful in studying heterogeneity of effects across subgroups. Int J Epidemiol. 2014;43(2):633–4.

    PubMed  Google Scholar 

  44. Rothman KJ, Gallacher JE, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42(4):1012–4.

    PubMed  PubMed Central  Google Scholar 

  45. Tipton E. Stratified sampling using cluster analysis: a sample selection strategy for improved generalizations from experiments. Eval Rev. 2013;37(2):109–39.

    PubMed  Google Scholar 

  46. Tipton E, Hedges L, Vaden-Kiernan M, Borman G, Sullivan K, Caverly S. Sample selection in randomized experiments: a new method using propensity score stratified sampling. J Res Educ Eff. 2014;7(1):114–35.

    Google Scholar 

  47. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA - J Am Med Assoc. 2003;290(12):1624–32.

    CAS  Google Scholar 

  48. Insel TR. Beyond efficacy: the STAR* D trial. Am J Psychiatr. 2006;163(1):5–7.

    PubMed  Google Scholar 

  49. Dahabreh IJ, Robertson SE, Hernan MA. On the relation between G-formula and inverse probability weighting estimators for generalizing trial results. Epidemiology. 2019.

  50. Dahabreh IJ, Hernan MA, Robertson SE, Buchanan A, Steingrimsson JA. Generalizing trial findings in nested trial designs with sub-sampling of non-randomized individuals. arXiv preprint arXiv:190206080. 2019.

  51. Robins J. A new approach to causal inference in mortality studies with sustained exposure periods - application to control of the healthy worker survivor effect. Mathe Model. 1986;7:1393–512.

    Google Scholar 

  52. Wang C, Mollan KR, Hudgens MG, Tucker JD, Zheng H, Tang W, et al. Generalisability of an online randomised controlled trial: an empirical analysis. J Epidemiol Community Health. 2018;72(2):173–8.

    PubMed  Google Scholar 

  53. Kern HL, Stuart EA, Hill J, Green DP. Assessing methods for generalizing experimental impact estimates to target populations. J Res Educ Eff. 2016;9(1):103–27.

    PubMed  PubMed Central  Google Scholar 

  54. Rudolph KE, Schmidt NM, Glymour MM, Crowder R, Galin J, Ahern J, et al. Composition or context: using transportability to understand drivers of site differences in a large-scale housing experiment. Epidemiology (Cambridge, Mass). 2018;29(2):199.

    Google Scholar 

  55. • Bonander C, Nilsson A, Bergström GM, Björk J, Strömberg U. Correcting for selective participation in cohort studies using auxiliary register data without identification of non-participants. Scand J Public Health. 2019:1403494819890784. Summary of practical issues when estimating sampling weights to generalize trial results to a target population, based on what data are available in the target population.

  56. Sato T, Matsuyama Y. Marginal structural models as a tool for standardization. Epidemiology. 2003;14(6):680–6.

    PubMed  Google Scholar 

  57. Bonander C, Nilsson A, Björk J, Bergström GM, Strömberg U. Participation weighting based on sociodemographic register data improved external validity in a population-based cohort study. J Clin Epidemiol. 2019;108:54–63.

    PubMed  Google Scholar 

  58. Stuart EA, Rhodes A. Generalizing treatment effect estimates from sample to population: a case study in the difficulties of finding sufficient data. Eval Rev. 2017;41(4):357–88.

    PubMed  Google Scholar 

  59. Nguyen TQ, Ackerman B, Schmid I, Cole SR, Stuart EA. Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: assumptions, models, effect scales, data scenarios, and implementation details. PLoS One. 2018;13(12):e0208795.

    PubMed  PubMed Central  Google Scholar 

  60. Hong JL, Jonsson Funk M, LoCasale R, Dempster SE, Cole SR, Webster-Clark M, et al. Generalizing randomized clinical trial results: implementation and challenges related to missing data in the target population. Am J Epidemiol. 2018;187(4):817–27.

    PubMed  Google Scholar 

  61. Batts K, Pemberton M, Bose J, Weimer B, Henderson L, Penne M, et al. Comparing and evaluating substance use treatment utilization estimates from the National Survey on Drug Use and Health and other data sources. CBHSQ Data Review Rockville, MD: Substance Abuse and Mental Health Services Administration (US). 2014.

  62. Ackerman B, Lesko CR, Siddique J, Susukida R, Stuart EA. Generalizing randomized trial findings to a target population using complex survey population data. arXiv preprint arXiv:200307500. 2020.

  63. Marcus SM. Assessing non-consent bias with parallel randomized and nonrandomized clinical trials. J Clin Epidemiol. 1997;50(7):823–8.

    CAS  PubMed  Google Scholar 

  64. Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology. 1999;10(1):37–48.

    CAS  PubMed  Google Scholar 

  65. Greenland S, Robins JM. Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol. 1986;15(3):413–9.

    CAS  PubMed  Google Scholar 

  66. Greenland S, Robins JM. Identifiability, exchangeability and confounding revisited. Epidemiologic perspectives & innovations : EP+I. 2009;6:4.

  67. Pressler TR, Kaizar EE. The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Stat Med. 2013;32(20):3552–68.

    PubMed  PubMed Central  Google Scholar 

  68. Lawlor DA, Tilling K, Davey SG. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45(6):1866–86.

    PubMed  Google Scholar 

  69. Hartman E, Grieve R, Ramsahai R, Sekhon JS. From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects. J R Stat Soc Series A (Statistics in Society). 2015;178(3):757–78.

    Google Scholar 

  70. Webster-Clark MA, Sanoff HK, Stürmer T, Lund SPHJL. Diagnostic assessment of assumptions for external validity: an example using data in metastatic colorectal cancer. Epidemiology (Cambridge, Mass). 2019;30(1):103.

    Google Scholar 

  71. Hazlett C. Estimating causal effects of new treatments despite self-selection: the case of experimental medical treatments. Journal of Causal Inference. 2019;7(1).

Download references

Funding

This work was supported by National Institutes of Health grants K01 AA028193 and K01 AI125087 and US Department of Education Institution of Education Sciences grant R305D150003.

Author information

Authors and Affiliations

  1. Department of Epidemiology, Johns Hopkins School of Public Health, 615 N. Wolfe St., Baltimore, MD, 21205, USA

    Catherine R. Lesko

  2. Department of Biostatistics, Johns Hopkins School of Public Health, Baltimore, MD, USA

    Benjamin Ackerman

  3. Department of Epidemiology, University of North Carolina, Chapel Hill, NC, USA

    Michael Webster-Clark & Jessie K. Edwards

Authors
  1. Catherine R. Lesko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benjamin Ackerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Webster-Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jessie K. Edwards
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Catherine R. Lesko.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Human and Animal Rights and Informed Consent

All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Epidemiologic Methods

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lesko, C.R., Ackerman, B., Webster-Clark, M. et al. Target Validity: Bringing Treatment of External Validity in Line with Internal Validity. Curr Epidemiol Rep 7, 117–124 (2020). https://doi.org/10.1007/s40471-020-00239-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40471-020-00239-0

Keywords

Search

Navigation