Abstract
Introduction
The ability of observational studies to draw conclusions on causal relationships between covariates and outcomes can be improved by incorporating randomly matched controls using the propensity scoring method. This procedure controls for pre-program differences between the enrolled and non-enrolled groups by reducing each participant’s set of covariates into a single score, which makes it feasible to match on what are essentially multiple variables simultaneously. This paper introduces this concept using the first year results of a congestive heart failure (CHF) disease management (DM) program as an example.
Methods
This study employed a case-control pre-post study design with controls randomly matched to patients based on the propensity score. There were 94 patients with CHF enrolled in a DM program for at least 1 year (cases), who were matched to 94 patients with CHF drawn from a health plan’s CHF population (controls). Independent variables that estimated the propensity score were pre-program: hospital admissions, emergency department (ED) visits, total costs, and risk level. Baseline (1 year prior to program commencement) and 1-year outcome variables were compared for the two groups.
Results
The results indicated that, at post-program, program participants had significantly lower hospitalization rates (p = 0.005), ED visit rates (p = 0.048), and total costs (p = 0.003) than their matched controls drawn from the CHF population.
Conclusions
Because of its simplicity and utility, propensity scoring should be considered as an alternative procedure for use with current non-experimental designs in evaluating DM program effectiveness.
Similar content being viewed by others
References
Kleijen J, Gotzche P, Kunz RA, et al. So what’s so special about randomization? In: Maynard A, Chalmers I, editors. Non-random reflections on health services research. London: BMJ Publishing, 1997: 93–106
D’Arcy Hart P. Early controlled clinical trials. BMJ 1996; 312(2): 769–82
Linden A, Adams J, Roberts N. An assessment of the total population approach for evaluating disease management program effectiveness. Dis Manag 2003; 6(2): 93–102
Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: an introduction to time series analysis. Dis Manag 2003; 6(4): 243–55
Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: an introduction to survival analysis. Dis Manag. 2004; 7(3): 180–190
Linden A, Adams J, Roberts N. Evaluation methods in disease management: determining program effectiveness. Position Paper for the Disease Management Association of America (DMAA). 2003 Oct
Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago (IL): Rand McNally, 1966
Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues for field settings. Chicago (IL): Rand McNally College Publishing Company, 1979
Shadish SR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston (MA): Houghton Mifflin, 2002
American Healthways and the John Hopkins Consensus Conference. Consensus report: standard outcome metrics and evaluation methodology for disease management programs. Dis Manag 2003; 6(3): 121–38
Dehejia RH, Wahba S. Propensity score-matching methods for non-experimental causal studies. Rev Econ and Stats 2002; 84: 151–61
Rosenbaum P, Rubin D. The central role of the propensity score in observational studies for causal effects. Biometrika 1983; 70: 41–55
Rosenbaum P, Rubin D. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am Stat 1985; 39: 33–8
Rubin D. Estimating causal effects of treatments in randomized and non-randomized studies. J Educ Psychol 1974; 66: 688–701
Rubin D. Assignment to treatment group on the basis of a covariate. J Educ Stats 1977; 2: 1–26
Cox DR. The analysis of binary data. London: Methuen, 1970
Cox DR. The analysis of multivariate binary data. Appl Stat 1972; 21: 113–20
Dehejia RH, Wahba S. Causal effects in nonexperimental studies: reevaluating the evaluation of training studies. J Am Stat Assoc 1999; 94: 1053–62
Heckman J, Ichimura J, Todd P. Matching as an econometric evaluation estimator: evidence from evaluating a job training programme. Rev Econ Stud 1997; 64: 605–54
Rubin DB, Thomas N. Matching using estimated propensity scores: relating theory to practice. Biometrics 1996; 52: 249–64
Rubin DB. Estimating causal effect from large data sets using propensity scores. Ann Intern Med 1997; 127: 757–63
Canner P. How much data should be collected in clinical trials? Stat Med 1984; 3: 423–32
Canner P. Covariate adjustment of treatment effects in clinical trials. Control Clin Trials 1991; 12: 359–66
Drake C. Effects of misspecification of the propensity score on estimators of treatment effects. Biometrics 1993; 49: 1231–6
Cochran WG. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 1968; 24: 205–13
Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc 1984; 79: 516–24
Aday L, Andersen RM. Equity in access to medical care: realized and potential. Med Care 1981; 19(12 Suppl.): 4–27
Andersen RM. Behavioral model of families: use of health services. Research Series No. 25. Chicago (IL): Center for Health Administration Studies, University of Chicago, 1968
Linden A, Adams J, Roberts N. Strengthening the case for disease management effectiveness: unhiding the hidden bias. Am J Eval. In press
Rosenbaum P. Sensitivity analysis for certain permutation tests in matched observational studies. Biometrika 1987; 74: 13–26
Fisher ES, Barton JA, Malenka DJ, et al. Overcoming potential pitfalls in the Medicare data for epidemiologic research. Am J Public Health 1990; 80: 533–46
Romano PS, Mark DH. Bias in the coding of hospital discharge data and its implication for quality assessment. Med Care 1994; 32: 81–90
Roos LL, Sharp SM, Cohen MM. Comparing clinical information with claims data: some similarities and differences. J Clin Epidemiol 1991; 44: 881–8
Acknowledgments
No sources of funding were used to assist in the preparation of this study. The authors have no conflicts of interest that are directly relevant to the content of this study.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Linden, A., Adams, J.L. & Roberts, N. Using Propensity Scores to Construct Comparable Control Groups for Disease Management Program Evaluation. Dis-Manage-Health-Outcomes 13, 107–115 (2005). https://doi.org/10.2165/00115677-200513020-00004
Published:
Issue Date:
DOI: https://doi.org/10.2165/00115677-200513020-00004