Benefit-of-the-doubt approaches for calculating a composite measure of quality

  • Michael Shwartz
  • James F. Burgess
  • Dan Berlowitz


Standard approaches for determining weights when calculating a composite measure of health care quality from individual quality indicators (QIs) include equal weighting, opportunity-based weights, and judgment-based weights. Benefit-of-the-doubt approaches have not been used in the health services area, though one has been used to calculate composite measures for profiling countries. Underlying these approaches is the assumption that relative performance on a set of indicators is, at least to some extent, a revealed preference by the organizational unit about the relative importance of the indicators. A benefit-of-the-doubt approach recognizes these revealed preferences by assigning higher weights to indicators on which performance is better and lower weights to indicators on which performance is poorer. We consider two benefit-of-the-doubt approaches. The first uses simple linear programming (LP) models; the second uses data envelopment analysis (DEA), the way in which the benefit-of-the-doubt approach has been previously implemented. In both cases, constraints are added to limit weight adjustments to some percentage of policy-determined baseline weights. Using both standard and benefit-of-the-doubt approaches, composite scores are calculated from data on five QIs from 32 Department of Veterans Affairs (VA) nursing homes. We examine the tradeoff between the level of allowable weight adjustment and impact on facility rankings. If weights are constrained to be within 75% of baseline weights, all approaches identify pretty much the same high performing facilities. Weights from benefit-of-the-doubt approaches, because they are able to reflect local preferences and conditions, should be attractive to facilities and, in a collaborative environment, to policy makers.


Composite measures Benefit-of-the-doubt approaches Health care quality Performance measurement Linear programming Data envelopment analysis 


  1. Agency for Healthcare Research and Quality: Prevention quality indicators overview. (2004). Accessed 4 Nov 2008
  2. Agency for Healthcare Research and Quality (AHRQ) Quality Indicators: Patient Safety Indicator Composite Measure Final Technical Report (October 2006)Google Scholar
  3. Allen, R., Athanassopoulos, A., Dyson, R.G., Thanassoulis, E.: Weights restrictions and value judgements in data envelopment analysis. Ann. Oper. Res. 73, 13–34 (1997)CrossRefGoogle Scholar
  4. Arling, G., Karon, S.L., Sainfort, F., Zimmerman, D.R., Ross, R.: Risk adjustment of nursing home quality indicators. Gerontologist 37, 757–766 (1997)PubMedGoogle Scholar
  5. Arling, G., Lewis, T., Kane, R.L., Mueller, C., Flood, S.: Improving quality assessment through multilevel modeling: the case of nursing home compare. Health Serv. Res. 42, 1177–1199 (2007)CrossRefPubMedGoogle Scholar
  6. Ash, A.S., Shwartz, M., Peköz, E.: Comparing outcomes across providers. In: Iezzoni, L.I. (ed.) Risk Adjustment for Measuring Health Care Outcomes, 3rd edn, pp. 297–333. Health Administration Press, Chicago (2003)Google Scholar
  7. Berlowitz, D.B., Ash, A.S., Brandeis, G.H., Brand, H.K., Halpern, J.L., Moskowitz, M.A.: Rating long-term care facilities on pressure ulcer development: importance of case-mix adjustment. Ann. Intern. Med. 124, 557–563 (1996)PubMedGoogle Scholar
  8. Berlowitz, D.R., Young, G.J., Hickey, E.C., Saliba, D., Mittman, B.S., Czarnowski, E., et al.: Quality improvement implementation in the nursing home. Health Serv. Res. 38, 65–83 (2003)CrossRefPubMedGoogle Scholar
  9. Berlowitz, D.B., Rosen, A.K., Wang, F., Tsilimingras, D., Tariot, P.N., Engelhardt, B., et al.: Purchasing or providing nursing home care: can quality of care data provide guidance. J. Am. Geriatr. Soc. 53, 603–608 (2005)CrossRefPubMedGoogle Scholar
  10. Braun, B.I.: The effect of nursing home quality on patient outcome. J. Am. Geriatr. Soc. 39, 329–338 (1991)PubMedGoogle Scholar
  11. Caldis, T.: Composite health plan quality scales. Health Care Financ. Rev. 28, 95–107 (2007)PubMedGoogle Scholar
  12. Carter, M.W.: Factors associated with ambulatory care-sensitive hospitalizations among nursing home residents. J Aging Health 15, 295–330 (2003)CrossRefPubMedGoogle Scholar
  13. Charnes, A., Cooper, W.W., Rhodes, E.: Measuring the efficiency of decision making units. Eur. J. Oper. Res. 2, 429–444 (1978)CrossRefGoogle Scholar
  14. Cherchye, L., Kuosmanen, T.: Benchmarking sustainable development: a synthetic meta-index approach. In: McGillivray, M., Clarke, M. (eds.) Perspectives on human development, chapter 7. United Nations University Press, Tokyo (2004)Google Scholar
  15. Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T.: An introduction to ‘benefit of the doubt’ composite indicators. Soc. Indic. Res. 82, 111–145 (2007)CrossRefGoogle Scholar
  16. Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T., Saisana, M., Saltelli, A., Liska, R., Tarantola, S.: Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index. J. Oper. Res. Soc. 59, 239–251 (2008)CrossRefGoogle Scholar
  17. Cooper, W.W., Seiford, L.M., Tone, K.: Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software. Springer Science and Business Media, LLC, New York (2007)Google Scholar
  18. Despotis, D.K.: A reassessment of the Human Development Index via data envelopment analysis. J. Oper. Res. Soc. 56, 969–980 (2005)CrossRefGoogle Scholar
  19. Edwards, J.R., Bagozzi, R.P.: On the nature and direction of relationships between constructs and measures. Psychol. Methods 5, 155–174 (2000)CrossRefPubMedGoogle Scholar
  20. Feinstein, A.R.: Multi-item “instruments” vs Virginia Apgar’s principles of clinimetrics. Arch. Intern. Med. 159, 125–128 (1999)CrossRefPubMedGoogle Scholar
  21. Gandhi, T.K., Cook, E.F., Puopolo, A.L., Burstin, H.R., Haas, J.S., Brennan, T.A.: Inconsistent report cards: assessing the comparability of various measures of the quality of ambulatory care. Med. Care 40, 155–165 (2002)CrossRefPubMedGoogle Scholar
  22. Gormley, W.T., Weimer, D.L.: Organizational Report Cards. Harvard University Press, Cambridge MA (1999)Google Scholar
  23. Institute of Medicine: Performance Measurement: Accelerating Improvement. National Academy Press, Washington, DC (2006)Google Scholar
  24. Jacobs, R., Goddard, M., Smith, P.C.: How robust are hospital ranks based on composite performance measures. Med. Care 43, 1177–1184 (2005)CrossRefPubMedGoogle Scholar
  25. Jarvis, C.B., MacKenzie, S.B., Podsakoff, P.M.: A critical review of construct indicators and measurement model misspecification in marketing and consumer research. J. Consum. Res. 30, 199–218 (2003)CrossRefGoogle Scholar
  26. Jha, A.K., Zhonghe, L., Orav, E.J., Epstein, A.M.: Care in U.S. hospitals—The Hospital Quality Alliance Program. N. Engl. J. Med. 353, 265–274 (2005)CrossRefPubMedGoogle Scholar
  27. Landrum, M.B., Bronskill, S.E., Normand, S.-L.: Analytic methods for constructing cross-sectional profiles of health care providers. Health Serv. Outcomes Res. Methodol. 1, 23–47 (2000)CrossRefGoogle Scholar
  28. Lenard, M.L., Wagner, J.M., Shimshak, D.G., Porell, F.W., Klimberg, R.K.: Evaluating the performance of nursing homes using data envelopment analysis. In: Lawrence, K.D. (ed.) Mathematical Programming, Applications of Management Science, vol. 11, pp. 89–105. Elsevier Ltd, Amsterdam (2004)Google Scholar
  29. Lied, T.R., Malsbary, R., Eisenberg, C., Ranck, J.: Combining HEDIS indicators: A new approach to measuring plan performance. Health Care Financ. Rev. 23, 117–129 (2002)PubMedGoogle Scholar
  30. Lindenauer, P.K., Remus, D., Roman, S., Rothberg, M.B., Benjamin, E.M., Ma, A., et al.: Public reporting and pay for performance in hospital quality improvement. N. Engl. J. Med. 356, 486–496 (2007)CrossRefPubMedGoogle Scholar
  31. Lovell, C.A.K.: Measuring the macroeconomic performance of the Taiwanese economy. Int. J. Prod. Econ. 39, 165–178 (1995)CrossRefGoogle Scholar
  32. Mahlberg, B., Obersteiner, M.: Remeasuring the HDI by data envelopment analysis. Interim Report IR-01-069. International Institute for Applied Systems Analysis, Laxenburg, Austria (2001)Google Scholar
  33. Mukamel, D.B.: Risk-adjusted outcome measures and quality of care in nursing homes. Med. Care 35, 367–385 (1997)CrossRefPubMedGoogle Scholar
  34. Nardo, M., Saisana, M., Saltelli, A., Tarantola S. Handbook on constructing composite indicators: methodology and user guide. Organization for Economic Co-operation and Development (OECD) Statistics Working Paper, 2005. Accessed 9 Sept 2009
  35. O’Brien, S.M., Shahian, D.M., DeLong, E.R., Normand, S.-L.T., Edwards, F.H., Ferraris, V.A., et al.: Quality measurement in adult cardiac surgery: part 2—statistical considerations in composite measure scoring and provider rating. Ann. Thorac. Surg. 83, S13–S26 (2007)CrossRefPubMedGoogle Scholar
  36. Pizer, S.D., Wang, M., Comstock, C.: Preventable hospitalization as a measure of quality of care in nursing homes. Working Paper #2003-01. Health Care Finance and Economics, VA Bedford, MA (2003)Google Scholar
  37. Porell, F., Caro, F.G., Silva, A., Monane, M.: A longitudinal analysis of nursing home outcomes. Health Serv. Res. 33, 835–865 (1998)PubMedGoogle Scholar
  38. Premier: Hospital Quality Incentive Demonstration Project. Summary of Composite Quality Scoring Methodology. (2003). Accessed 4 Nov 2008
  39. Reeves, D., Campbell, S.M., Adams, J., Shekelle, P.G., Kontopantelis, E.: Combining multiple indicators of clinical quality: an evaluation of different analytic approaches. Med. Care 45, 489–496 (2007)CrossRefPubMedGoogle Scholar
  40. Rosen, A., Wu, J., Chang, B., Berlowitz, D.R., Rakovski, C., Ash, A.S., et al.: Risk adjustment for measuring health outcomes: an application in VA long-term care. Am. J. Med. Qual. 16, 118–127 (2001)CrossRefPubMedGoogle Scholar
  41. Ryan, A.M., Burgess, J.F., Tompkins, C.P., Wallack, S.S.: The relationship between Medicare’s process of care quality measures and mortality. Inquiry 46, 274–291 (2009)PubMedGoogle Scholar
  42. Semple, J.: Constrained games for evaluating organizational performance. Eur. J. Oper. Res. 96, 103–112 (1996)CrossRefGoogle Scholar
  43. Shwartz, M., Ren, J., Peköz, E.A., Wang, X., Cohen, A.B., Restuccia, J.D.: Estimating a composite measure of hospital quality from the Hospital Compare database: differences when using a Bayesian hierarchical latent variable model versus denominator-based weights. Med. Care 46, 778–785 (2008)CrossRefPubMedGoogle Scholar
  44. Werner, R.M., Bradlow, E.T.: Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA 296, 2694–2702 (2006)CrossRefPubMedGoogle Scholar
  45. Winston, W., Albright, S.C.: Practical Management Science, 2nd edn. Duxbury, Pacific Grove, CA (2001)Google Scholar
  46. Zaslavsky, A.M., Shaul, J.A., Zaborski, L.B., Cioffi, M.J., Cleary, P.D.: Combining health plan performance indicators into simpler composite measures. Health Care Financ. Rev. 23, 101–115 (2002)PubMedGoogle Scholar
  47. Zimmerman, D.R., Karon, S.L., Arling, G., Clark, B.R., Collins, T., Ross, R., et al.: Development and testing of nursing home quality indicators. Health Care Financ. Rev. 16, 107–127 (1995)PubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Michael Shwartz
    • 1
    • 2
  • James F. Burgess
    • 1
    • 4
  • Dan Berlowitz
    • 3
    • 4
  1. 1.Center for Organization, Leadership and Management ResearchVA Boston Healthcare System (152M)BostonUSA
  2. 2.School of ManagementBoston UniversityBostonUSA
  3. 3.Center for Health Quality, Outcomes and Economic ResearchBedford VA HospitalBedfordUSA
  4. 4.School of Public HealthBoston UniversityBostonUSA

Personalised recommendations