Examining the Impact of Missing Data on Propensity Score Estimation in Determining the Effectiveness of Self-Monitoring of Blood Glucose (SMBG)

  • Ralph D'AgostinoJr.
  • Wei Lang
  • Michael Walkup
  • Timothy Morgan
  • Andrew Karter


In many health services applications, research to determine the effectiveness of a particular treatment cannot be carried out using a controlled clinical trial. In settings such as these, observational studies must be used. Propensity score methods are useful tools to employ in order to balance the distribution of covariates between treatment groups and hence reduce the potential bias in treatment effect estimates in observational studies. A challenge in many health services research studies is the presence of missing data among the covariates that need to be balanced. In this paper, we compare three simple propensity models using data that examine the effectiveness of self-monitoring of blood glucose (SMBG) in reducing hemoglobin A1c in a cohort of 10,566 type 2 diabetics. The first propensity score model uses only subjects with complete case data (n=6,687), the second incorporates missing value indicators into the model, and the third fits separate propensity scores for each pattern of missing data. We compare the results of these methods and find that incorporating missing data into the propensity score model reduces the estimated effect of SMBG on hemoglobin A1c by more than 10%, although this reduction was not clinically significant. In addition, beginning with the complete data, we artificially introduce missing data using a nonignorable missing data mechanism and compare treatment effect estimates using the three propensity score methods and a simple analysis of covariance (ANCOVA) method. In these analyses, we find that the complete case analysis and the ANCOVA method both perform poorly, the missing value indicator model performs moderately well, and the pattern mixture model performs even better in estimating the original treatment effect observed in the␣complete data prior to the introduction of artificial missing data. We conclude that in observational studies one␣must not only adjust for potentially confounding variables using methods such as propensity scores, but one␣should also account for missing data in these models in order to allow for causal inference more appropriately to be applied.

missing data pattern mixture models propensity scores self-monitoring of blood glucose diabetes 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. American Diabetes Association: Clinical Practice Recommendations 1999, Diabetes Care, 22(Supplement 1), pp. S77-S78, 1999.Google Scholar
  2. Paul Allison, D, “Multiple imputation for missing data: a cautionary tale,” Sociological Methods & Research, 28, pp. 301-309, 2000.Google Scholar
  3. American Diabetes Association, Self-monitoring of blood glucose. Diabetes Care, 17, pp. 81-86, 1994.Google Scholar
  4. American Diabetes Association: Clinical Practice Recommendations, Diabetes Care, 21(Supplement 1), pp. S1-S98, 1998.Google Scholar
  5. American Diabetes Association: Clinical Practice Recommendations, Diabetes Care, 22, 1999.Google Scholar
  6. S. G. Baker, “Analyzing a randomized cancer prevention trial with a missing binary outcome, an auxiliary variable and all or none compliance,” JASA, 95, pp. 43-50, 2000.Google Scholar
  7. F. G. Barker 2nd, S. M. Chang, P. H. Gutin, M. K. Malec, M. W. McDemott, M. D. Prados and C. B. Wilson, “Survival and functional status after resection of recurrent glioblastoma multiforme,” Neurosurgery, 42, pp. 709-720, 1998.Google Scholar
  8. A. F. Conners Jr., S. A. Speroff and N. V. Dawson et al., “The effectiveness of right heart catheterization in the initial care of critically ill patients. support investigators,” Journal of the American Medical Association, 276, pp. 889-997, 1996.Google Scholar
  9. R. B. D'Agostino Jr., “Propensity score methods for bias reduction in the comparison of a treatment to a nonrandomized control group,” Statistics in Medicine, 17, pp. 2265-2281, 1998.Google Scholar
  10. R. B. D'Agostino Jr. and D. B. Rubin, “Estimating and using propensity scores with partially missing data,” JASA, 95, pp. 749-759, 2000.Google Scholar
  11. A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood estimation from incomplete data via the EM algorithm (with discussion),” Journal of the Royal Statistical Society, Ser. B 39, pp. 1-38, 1977.Google Scholar
  12. J. M. Evans, R.W. Newton, D. A. Ruta, T. M. MacDonald, R. J. Stevenson and A. D. Morris, “Frequency of blood glucose monitoring in relation to glycemic control: observational study with diabetes database,” BMJ, 319, pp. 83-86, 1999.Google Scholar
  13. A. Faas, F. G. Schellevis and J. T. Van Eijk, “The efficacy of self-monitoring of blood glucose in NIDDM subjects. A criteria-based literature review,” Diabetes Care, 20, pp. 1482-1486, 1997.Google Scholar
  14. X. S. Gu and P. R. Rosenbaum, “Comparison of multivariable matching methods; Structures, distances, and algorithms,” Journal of Computational and Graphical Statistics, 2, pp. 405-420, 1993.Google Scholar
  15. J. J. Heckman, H. Ichimura, J. Smith and P. Todd, “Sources of selection bias in evaluating social programs: an interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method,” Proceedings of the National Academy of Sciences of the United States of America, 93, pp. 13416-13420, 1996.Google Scholar
  16. J. G. Ibrahim, S. R. Lipitz and M. Chen, “Missing covariates in generalized linear models when the missing data mechanism is nonignorable,” Journal of the Royal Statistical Society, Series B 61, Part1 1, pp. 173-190, 1999. 31 D. B. Rubin, Multiple imputation for nonresponse in surveys, John Wiley & Sons, 1987.Google Scholar
  17. A. J. Karter, A. Ferrara, J. Darbinian, L. M. Ackerson and J. V. Selby, “Self-monitoring of blood glucose: language and financial barriers in a managed care population with diabetes,” Diabetes Care, 23(4), pp. 477-483, 2000.Google Scholar
  18. A. J. Karter, L. M. Ackerson, J. Darbinian, R. B. D'Agostino Jr., J. Liu and J. V. Selby, “Self-monitoring of blood glucose and glycemic control in a large managed care population: The Northern California Kaiser Permanente Diabetes Registry,” American Journal of Medicine, 111, pp. 1-9, 2001.Google Scholar
  19. E. Lieberman, A. Cohen, J. Lang, R. B. D'Agostino Jr., S. Datta and F. D. Frigoletto Jr., “The association of epidural anesthesia with cesarean delivery in nulliparas,” Obstetrics and Gynecology, 88, pp. 993-1000, 1996.Google Scholar
  20. R. J. A. Little, “Pattern-mixture models for multivariate incomplete data,” JASA, 88, pp. 125-134, 1993.Google Scholar
  21. B. W. Lytle, E. H. Blackstone, F. D. Loop, P. L. Houghtaling, J. H. Arnold, P. M. Mccarthy and D. M. Cosgrove, “Two internal thoracic artery grafts are better than one,” Journal of Thoracic and Cardiovascular Surgery, 117, pp. 855-872, 1999.Google Scholar
  22. T. L. Martin, J. V. Selby and D. Zhang, “Physician and patient prevention practices in NIDDM in a large urban managed-care organization,” Diabetes Care, 18, pp. 1124-1132, 1995.Google Scholar
  23. X. L. Meng and D. B. Rubin, “Maximum likelihood estimation via the ECM algorithm: a general framework,” Biometrika, 80, pp. 267-278, 1993.Google Scholar
  24. I. Olkin and R. F. Tate, “Multivariate correlation models with mixed discrete and continuous variables,” Annals of Mathematical Statistics, 32, pp. 448-465, 1961.Google Scholar
  25. S. S. Rich. “Analytic options for asthma genetics,” Clinical and Experimental Allergy, 28, pp. 108-110, 1998.Google Scholar
  26. James M. Robins and Wang, “Naisyin inference for imputation estimators,” Biometrika, 87, pp. 113-124, 2000.Google Scholar
  27. P. R. Rosenbaum and D. B. Rubin, “The central role of the propensity score in observational studies for causal effects,” Biometrika, 70, pp. 41-55, 1983.Google Scholar
  28. P. R. Rosenbaum and D. B. Rubin, “Reducing bias in observational studies using subclassification on the propensity score,” JASA, 79, pp. 516-524, 1984.Google Scholar
  29. P. R. Rosenbaum and D. B. Rubin, “Constructing a control group using multivariate matched sampling methods that incorporate the propensity score,” American Statistician, 39, pp. 33-38, 1985.Google Scholar
  30. D. B. Rubin, “Multiple imputations in sample surveys: A phenomenological bayesian approach to nonresponse,” ASA Proceedings of Survey Research Methods Section, pp. 20-28, 1978.Google Scholar
  31. D. B. Rubin, “Basic ideas of multiple imputation for nonresponse,” Survey Methodology 12, pp. 37-47, 1986.Google Scholar
  32. D. B. Rubin, “Practical implications of statistical inference for causal effects and the critical role of the assignment mechanism,” Biometrics, 47, pp. 1213-1234, 1991.Google Scholar
  33. D. B. Rubin, “Estimating causal effects from large data sets using propensity scores,” Annals of Internal Medicine, 127, pp. 757-763, 1997.Google Scholar
  34. D. B. Rubin and N. Thomas, “Affinely invariant matching methods with ellipsoidal distributions,” The Annals of Statistics, 20, pp. 1079-1093, 1992a.Google Scholar
  35. D. B. Rubin and N. Thomas, “Characterizing the effect of matching using linear propensity score methods with normal distributions,” Biometrika, 79, pp. 797-809, 1992b.Google Scholar
  36. D. B. Rubin and N. Thomas, “Matching using estimated propensity scores; Relating theory to practice,” Biometrics, 52, pp. 249-264, 1996.Google Scholar
  37. D. B. Rubin and N. Thomas, “Combining propensity score matching with additional adjustments for prognostic covariates,” Journal of the American Statistical Association (JASA), 95, pp. 573-586, 2000.Google Scholar
  38. J. V. Selby, B. Ettinger, B. Swain and J. B. Brown, “First 20 months' experience with use of Metformin for type 2 diabetes in a large health maintenance organization,” Diabetes Care, 22, pp. 38-44, 1999.Google Scholar
  39. J. V. Selby, G. T. Ray, D. Zhang and C. J. Colby, “Excess costs of medical care for patients with diabetes in a managed care population,” Diabetes Care, 20, pp. 1396-1402, 1997.Google Scholar
  40. W. Vach and M. Blettner, “Biased estimation of the odds ratio in case-control studies due to the use of ad hoc methods of correcting for missing values for confounding variables,” American Journal of Epidemiology, 134, pp. 895-907, 1991.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Ralph D'AgostinoJr.
    • 1
  • Wei Lang
    • 1
  • Michael Walkup
    • 1
  • Timothy Morgan
    • 1
  • Andrew Karter
    • 2
  1. 1.Department of Public Health Sciences, Section on BiostatisticsWake Forest University School of MedicineWinston-Salem
  2. 2.Kaiser PermanenteOakland

Personalised recommendations