Abstract
The quality of propensity scores is traditionally measured by assessing how well they make the distributions of covariates in the treatment and control groups match, which we refer to as “good balance”. Good balance guarantees less biased estimates of the treatment effect. However, the cost of achieving good balance is that the variance of the estimates increases due to a reduction in effective sample size, either through the introduction of propensity score weights or dropping cases when propensity score matching. In this paper, we investigate whether it is best to optimize the balance or to settle for a less than optimal balance and use double robust estimation to adjust for remaining differences. We compare treatment effect estimates from regression, propensity score weighting, and double robust estimation with varying levels of effort expended to achieve balance using data from a study about the differences in outcomes by HIV status in heterosexually active homeless men residing in Los Angeles. Because of how costly data collection efforts are for this population, it is important to find an alternative estimation method that does not reduce effective sample size as much as methods that aggressively aim to optimize balance. Results from a simulation study suggest that there are instances in which we can obtain more precise treatment effect estimates without increasing bias too much by using a combination of regression and propensity score weights that achieve a less than optimal balance. There is a bias-variance tradeoff at work in propensity score estimation; every step toward better balance usually means an increase in variance and at some point a marginal decrease in bias may not be worth the associated increase in variance.
Similar content being viewed by others
References
Bang, H., Robins, J.M.: Doubly robust estimation in missing data and causal inference models. Biometrics 61, 962–972 (2005)
Centers for Disease Control and Prevention. HIV Surveillance—United States, 1981–2008. Morbidity and Mortality Weekly Report (MMWR) 60, 689–93 (2011)
Cochran, W.G., Rubin, D.B.: Controlling bias in observational studies: a review. Sankhya Ser A 35(4), 417–446 (1973)
Elliott, M.N., Golinelli, D., Hambarsoomian, K., Perlman, J., Wenzel, S.: Sampling with field burden constraints: an application to sheltered homeless and low income housed women. Field Methods 18, 43–58 (2006)
Gelberg, L., Andersen, R.M., Leake, B.D.: The behavioral model for vulnerable populations: application to medical care use and outcomes for homeless people. Health Serv. Res. 34, 1273–1302 (2000)
Henry, R., Richardson, J.L., Stoyanoff, S., Garcia, G.P., Dorey, F., Iverson, E., et al.: HIV/AIDS health service utilization by people who have been homeless. AIDS Behav. 12, 815–821 (2008)
Hirano, K., Imbens, G.W.: Estimation of casual effects using propensity score weighting: an application to data on right heart catherterization. Health Serv. Outcomes Res. Methodol. 2, 259–278 (2001)
Kang, J.D.Y., Schafer, J.L.: Demystifying double robustness, a comparison of alternative strategies for estimating a population mean from incomplete data. Stat. Sci. 22, 523–539 (2007)
Kennedy, D.P., Wenzel, S.L., Tucker, J.S., Green Jr, H.D., Golinelli, D., Ryan, G.W., et al.: Unprotected sex of homeless women living in Los Angeles County: an investigation of the multiple levels of risk. AIDS Behav. 14, 960–973 (2010)
Lee, B.K., Lessler, J., Stuart, E.A.: Improving propensity score weighting using machine learning. Stat. Med. 29, 337–346 (2010)
Lunceford, J.K., Davidian, M.: Stratification and weighting via propensity score in estimating of casual treatment effects, a comparative study. Stat. Med. 23, 2937–2960 (2004)
McCaffrey, D., Ridgeway, G., Morral, A.: Propensity score estimation with boosted regression for evaluating adolescent substance abuse treatment. Psychol. Methods 9, 403–425 (2004)
National Alliance to End Homelessness. Fact Sheet: homelessness and HIV/AIDS. National Alliance to End Homelessness, Washington, DC (2006)
Ridgeway, G.: GBM 1.6-3 package manual. R Project (2007)
Ridgeway, G., McCaffrey, D.F.: Comment on ‘demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data’ by Kang and Schafer. Stat Sci 22, 540–543 (2007)
Rosenbaum, P.R.: The consequences of adjustment for a concomitant variable that has been affected by the treatment. J. R. Stat. Soc. Ser. A 147(5), 656–666 (1984)
Rosenbaum, P.R.: Sensitivity analysis in observational studies. Encyclopedia of Statistics in Behavioral Science. Wiley, New York (2005)
Rosenbaum, P.R., Rubin, D.: The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41–55 (1983)
Rubin, D.: The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics 29, 185–203 (1973)
Tsiatis, A.A., Davidian, M., Zhang, M., Lu, X.: Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach. Stat. Med. 25, 1–10 (2000)
Tucker, J.S., Wenzel, S.L., Golinelli, D., Zhou, A., Green Jr, H.D.: Predictors of substance abuse treatment need and receipt among homeless women. J. Subst. Abuse Treat. 40, 287–294 (2011)
U.S. Census Bureau. State and County QuickFacts: USA. Washington, DC: U.S. Census Bureau (2011). http://quickfacts.census.gov/qfd/states/00000.html. Accessed 27 Feb 2012
Wenzel, S.L.: Heterosexual HIV risk behavior in homeless men. National Institute for Child Health and Human Development (2009)
Wenzel, S.L., Green Jr, H.D., Tucker, J.S., Golinelli, D., Kennedy, D.P., Ryan, G., et al.: The social context of homeless women’s alcohol and drug use. Drug Alcohol Depend. 105, 16–23 (2009)
Acknowledgments
This research was supported by Grant R01HD059307 from the National Institute of Child Health and Human Development. The authors thank the men who shared their experiences, the service agencies in the Central City East area of Los Angeles that collaborated with us, and the RAND Survey Research Group for assistance in data collection.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Golinelli, D., Ridgeway, G., Rhoades, H. et al. Bias and variance trade-offs when combining propensity score weighting and regression: with an application to HIV status and homeless men. Health Serv Outcomes Res Method 12, 104–118 (2012). https://doi.org/10.1007/s10742-012-0090-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-012-0090-1