A systematic review of propensity score methods in the acute care surgery literature: avoiding the pitfalls and proposing a set of reporting guidelines

Original Article

Abstract

Background

Propensity score methods are techniques commonly employed in observational research to account for confounding when estimating the effects of treatments and exposures. These methods have been increasingly employed in the acute care surgery literature in an attempt to infer causality; however, the adequacy of reporting and the appropriateness of statistical analyses when using propensity score matching remain unclear.

Objectives

The goal of this systematic review is to assess the adequacy of reporting of propensity score methods, with an emphasis on propensity score matching (to assess balance and the use of appropriate statistical tests), in acute care surgery (ACS) studies and to provide suggestions for improvement for junior investigators.

Methods

We searched three databases, and other relevant literature (from January 2005 to June 2015) to identify observational studies within the ACS literature using propensity score methods (PROSPERO No: CRD42016036432). Two reviewers extracted data and assessed the quality of the studies retrieved by reviewing the adequacy of both overall reporting and of the propensity score matching methods used.

Results

A total of 49/71 (69%) of studies adequately reported propensity score methods overall. Matching was the most common propensity score method used in 46/71 (65%) studies, with 36/46 (78%) studies reporting matching methods adequately. Only 19/46 (41%) of matching studies reported the balance of baseline characteristics between treated and untreated subjects while 6/46 (13%) used correct statistical methods to assess balance. There were 35/46 (76%) of matching studies that explicitly used statistical methods appropriate for the analysis of matched data when estimating the treatment effect and its statistical significance.

Conclusion

We have proposed reporting guidelines for the use of propensity score methods in the acute care surgery literature. This is to help investigators improve the adequacy of reporting and statistical analyses when using observational data to estimate effects of treatments and exposures.

Keywords

Propensity score Matching Trauma Acute care Surgery 

Notes

Compliance with ethical standards

Conflict of interest

Victoria McCredie, Peter Austin and Tanya L. Zakrison declare that they have no conflict of interest.

Supplementary material

68_2017_786_MOESM1_ESM.docx (60 kb)
Supplementary material 1 (DOCX 59 KB)

References

  1. 1.
    Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar Behav Res. 2011;46(3):399–424.CrossRefGoogle Scholar
  2. 2.
    McMurry TL, Hu Y, Blackstone EH, Kozower BD. Propensity scores: methods, considerations, and applications in the. Journal of Thoracic Cardiovascular Surgery. J Thorac Cardiovasc Surg. 2015;150(1):14–9.CrossRefPubMedGoogle Scholar
  3. 3.
    Luo Z, Gardiner JC, Bradley CJ. Applying propensity score methods in medical research: pitfalls and prospects. Med Care Res Rev. 2010;67(5):528–54.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Austin PC. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Med Decis Mak. 2009;29(6):661–77.CrossRefGoogle Scholar
  5. 5.
    Austin PC. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med. 2008;27(12):2037–49.CrossRefPubMedGoogle Scholar
  6. 6.
    Austin PC. Primer on statistical interpretation or methods report card on propensity-score matching in the cardiology literature from 2004 to 2006: a systematic review. Circ Cardiovasc Qual Outcomes. 2008;1(1):62–7.CrossRefPubMedGoogle Scholar
  7. 7.
    Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. J Thorac Cardiovasc Surg. 2007;134(5):1128–35.CrossRefPubMedGoogle Scholar
  8. 8.
    Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. New York: Wiley; 2011.Google Scholar
  9. 9.
    Panic N, Leoncini E, de Belvis G, Ricciardi W, Boccia S. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8(12):e83138.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Gayat E, Pirracchio R, Resche-Rigon M, Mebazaa A, Mary JY, Porcher R. Propensity scores in intensive care and anaesthesiology literature: a systematic review. Intensive Care Med. 2010;36(12):1993–2003.CrossRefPubMedGoogle Scholar
  12. 12.
    Inoue J, Shiraishi A, Yoshiyuki A, Haruta K, Matsui H, Otomo Y. Resuscitative endovascular balloon occlusion of the aorta might be dangerous in patients with severe torso trauma: a propensity score analysis. J Trauma Acute Care Surg. 2016;80(4):559–67.CrossRefPubMedGoogle Scholar
  13. 13.
    Austin PC, Grootendorst P, Anderson GM. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study. Stat Med. 2007;26(4):734–53.CrossRefPubMedGoogle Scholar
  14. 14.
    Austin PC. Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharm Stat. 2011;10(2):150–61.CrossRefPubMedGoogle Scholar
  15. 15.
    Austin PC. A comparison of 12 algorithms for matching on the propensity score. Stat Med. 2014;33(6):1057–69.CrossRefPubMedGoogle Scholar
  16. 16.
    Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biom J. 2009;51(1):171–84.CrossRefPubMedGoogle Scholar
  17. 17.
    Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med. 2007;26(16):3078–94.CrossRefPubMedGoogle Scholar
  18. 18.
    Austin PC. The use of propensity score methods with survival or time-to-event outcomes: reporting measures of effect similar to those used in randomized experiments. Stat Med. 2014;33(7):1242–58.CrossRefPubMedGoogle Scholar
  19. 19.
    Morgan SL, Todd JL. A diagnostic routine for the detection of consequential heterogeneity of causal effects. Sociol Methodol. 2008;38:231–81.CrossRefGoogle Scholar
  20. 20.
    Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. 2009;28(25):3083–107.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Mitra R, Reiter JP. A comparison of two methods of estimating propensity scores after multiple imputation. Stat Methods Med Res. 2016;25(1):188–204.CrossRefPubMedGoogle Scholar
  22. 22.
    Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55.CrossRefGoogle Scholar
  23. 23.
    Rassen JA, Shelat AA, Myers J, Glynn RJ, Rothman KJ, Schneeweiss S. One-to-many propensity score matching in cohort studies. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 2):69–80.CrossRefPubMedGoogle Scholar
  24. 24.
    Austin PC, Mamdani MM, Stukel TA, Anderson GM, Tu JV. The use of the propensity score for estimating treatment effects: administrative versus clinical data. Stat Med. 2005;24(10):1563–78.CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • T. L. Zakrison
    • 3
  • P. C. Austin
    • 1
  • V. A. McCredie
    • 2
  1. 1.Institute for Clinical Evaluative SciencesTorontoCanada
  2. 2.Department of Critical Care MedicineSunnybrook Health Sciences CenterTorontoCanada
  3. 3.Department of SurgeryUniversity of Miami Miller School of MedicineMiamiUSA

Personalised recommendations