Abstract
Standard approaches for determining weights when calculating a composite measure of health care quality from individual quality indicators (QIs) include equal weighting, opportunity-based weights, and judgment-based weights. Benefit-of-the-doubt approaches have not been used in the health services area, though one has been used to calculate composite measures for profiling countries. Underlying these approaches is the assumption that relative performance on a set of indicators is, at least to some extent, a revealed preference by the organizational unit about the relative importance of the indicators. A benefit-of-the-doubt approach recognizes these revealed preferences by assigning higher weights to indicators on which performance is better and lower weights to indicators on which performance is poorer. We consider two benefit-of-the-doubt approaches. The first uses simple linear programming (LP) models; the second uses data envelopment analysis (DEA), the way in which the benefit-of-the-doubt approach has been previously implemented. In both cases, constraints are added to limit weight adjustments to some percentage of policy-determined baseline weights. Using both standard and benefit-of-the-doubt approaches, composite scores are calculated from data on five QIs from 32 Department of Veterans Affairs (VA) nursing homes. We examine the tradeoff between the level of allowable weight adjustment and impact on facility rankings. If weights are constrained to be within 75% of baseline weights, all approaches identify pretty much the same high performing facilities. Weights from benefit-of-the-doubt approaches, because they are able to reflect local preferences and conditions, should be attractive to facilities and, in a collaborative environment, to policy makers.
Similar content being viewed by others
References
Agency for Healthcare Research and Quality: Prevention quality indicators overview. http://www.qualityindicators.ahrq.gov/pqi_overview.htm (2004). Accessed 4 Nov 2008
Agency for Healthcare Research and Quality (AHRQ) Quality Indicators: Patient Safety Indicator Composite Measure Final Technical Report (October 2006)
Allen, R., Athanassopoulos, A., Dyson, R.G., Thanassoulis, E.: Weights restrictions and value judgements in data envelopment analysis. Ann. Oper. Res. 73, 13–34 (1997)
Arling, G., Karon, S.L., Sainfort, F., Zimmerman, D.R., Ross, R.: Risk adjustment of nursing home quality indicators. Gerontologist 37, 757–766 (1997)
Arling, G., Lewis, T., Kane, R.L., Mueller, C., Flood, S.: Improving quality assessment through multilevel modeling: the case of nursing home compare. Health Serv. Res. 42, 1177–1199 (2007)
Ash, A.S., Shwartz, M., Peköz, E.: Comparing outcomes across providers. In: Iezzoni, L.I. (ed.) Risk Adjustment for Measuring Health Care Outcomes, 3rd edn, pp. 297–333. Health Administration Press, Chicago (2003)
Berlowitz, D.B., Ash, A.S., Brandeis, G.H., Brand, H.K., Halpern, J.L., Moskowitz, M.A.: Rating long-term care facilities on pressure ulcer development: importance of case-mix adjustment. Ann. Intern. Med. 124, 557–563 (1996)
Berlowitz, D.R., Young, G.J., Hickey, E.C., Saliba, D., Mittman, B.S., Czarnowski, E., et al.: Quality improvement implementation in the nursing home. Health Serv. Res. 38, 65–83 (2003)
Berlowitz, D.B., Rosen, A.K., Wang, F., Tsilimingras, D., Tariot, P.N., Engelhardt, B., et al.: Purchasing or providing nursing home care: can quality of care data provide guidance. J. Am. Geriatr. Soc. 53, 603–608 (2005)
Braun, B.I.: The effect of nursing home quality on patient outcome. J. Am. Geriatr. Soc. 39, 329–338 (1991)
Caldis, T.: Composite health plan quality scales. Health Care Financ. Rev. 28, 95–107 (2007)
Carter, M.W.: Factors associated with ambulatory care-sensitive hospitalizations among nursing home residents. J Aging Health 15, 295–330 (2003)
Charnes, A., Cooper, W.W., Rhodes, E.: Measuring the efficiency of decision making units. Eur. J. Oper. Res. 2, 429–444 (1978)
Cherchye, L., Kuosmanen, T.: Benchmarking sustainable development: a synthetic meta-index approach. In: McGillivray, M., Clarke, M. (eds.) Perspectives on human development, chapter 7. United Nations University Press, Tokyo (2004)
Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T.: An introduction to ‘benefit of the doubt’ composite indicators. Soc. Indic. Res. 82, 111–145 (2007)
Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T., Saisana, M., Saltelli, A., Liska, R., Tarantola, S.: Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index. J. Oper. Res. Soc. 59, 239–251 (2008)
Cooper, W.W., Seiford, L.M., Tone, K.: Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software. Springer Science and Business Media, LLC, New York (2007)
Despotis, D.K.: A reassessment of the Human Development Index via data envelopment analysis. J. Oper. Res. Soc. 56, 969–980 (2005)
Edwards, J.R., Bagozzi, R.P.: On the nature and direction of relationships between constructs and measures. Psychol. Methods 5, 155–174 (2000)
Feinstein, A.R.: Multi-item “instruments” vs Virginia Apgar’s principles of clinimetrics. Arch. Intern. Med. 159, 125–128 (1999)
Gandhi, T.K., Cook, E.F., Puopolo, A.L., Burstin, H.R., Haas, J.S., Brennan, T.A.: Inconsistent report cards: assessing the comparability of various measures of the quality of ambulatory care. Med. Care 40, 155–165 (2002)
Gormley, W.T., Weimer, D.L.: Organizational Report Cards. Harvard University Press, Cambridge MA (1999)
Institute of Medicine: Performance Measurement: Accelerating Improvement. National Academy Press, Washington, DC (2006)
Jacobs, R., Goddard, M., Smith, P.C.: How robust are hospital ranks based on composite performance measures. Med. Care 43, 1177–1184 (2005)
Jarvis, C.B., MacKenzie, S.B., Podsakoff, P.M.: A critical review of construct indicators and measurement model misspecification in marketing and consumer research. J. Consum. Res. 30, 199–218 (2003)
Jha, A.K., Zhonghe, L., Orav, E.J., Epstein, A.M.: Care in U.S. hospitals—The Hospital Quality Alliance Program. N. Engl. J. Med. 353, 265–274 (2005)
Landrum, M.B., Bronskill, S.E., Normand, S.-L.: Analytic methods for constructing cross-sectional profiles of health care providers. Health Serv. Outcomes Res. Methodol. 1, 23–47 (2000)
Lenard, M.L., Wagner, J.M., Shimshak, D.G., Porell, F.W., Klimberg, R.K.: Evaluating the performance of nursing homes using data envelopment analysis. In: Lawrence, K.D. (ed.) Mathematical Programming, Applications of Management Science, vol. 11, pp. 89–105. Elsevier Ltd, Amsterdam (2004)
Lied, T.R., Malsbary, R., Eisenberg, C., Ranck, J.: Combining HEDIS indicators: A new approach to measuring plan performance. Health Care Financ. Rev. 23, 117–129 (2002)
Lindenauer, P.K., Remus, D., Roman, S., Rothberg, M.B., Benjamin, E.M., Ma, A., et al.: Public reporting and pay for performance in hospital quality improvement. N. Engl. J. Med. 356, 486–496 (2007)
Lovell, C.A.K.: Measuring the macroeconomic performance of the Taiwanese economy. Int. J. Prod. Econ. 39, 165–178 (1995)
Mahlberg, B., Obersteiner, M.: Remeasuring the HDI by data envelopment analysis. Interim Report IR-01-069. International Institute for Applied Systems Analysis, Laxenburg, Austria (2001)
Mukamel, D.B.: Risk-adjusted outcome measures and quality of care in nursing homes. Med. Care 35, 367–385 (1997)
Nardo, M., Saisana, M., Saltelli, A., Tarantola S. Handbook on constructing composite indicators: methodology and user guide. Organization for Economic Co-operation and Development (OECD) Statistics Working Paper, 2005. http://www.olis.oecd.org/olis/2005doc.nsf/LinkTo/std-doc(2005)3. Accessed 9 Sept 2009
O’Brien, S.M., Shahian, D.M., DeLong, E.R., Normand, S.-L.T., Edwards, F.H., Ferraris, V.A., et al.: Quality measurement in adult cardiac surgery: part 2—statistical considerations in composite measure scoring and provider rating. Ann. Thorac. Surg. 83, S13–S26 (2007)
Pizer, S.D., Wang, M., Comstock, C.: Preventable hospitalization as a measure of quality of care in nursing homes. Working Paper #2003-01. Health Care Finance and Economics, VA Bedford, MA (2003)
Porell, F., Caro, F.G., Silva, A., Monane, M.: A longitudinal analysis of nursing home outcomes. Health Serv. Res. 33, 835–865 (1998)
Premier: Hospital Quality Incentive Demonstration Project. Summary of Composite Quality Scoring Methodology. http://www.premierinc.com/quality-safety/tools-services/p4p/hqi/resources/top-performer-summary.pdf. (2003). Accessed 4 Nov 2008
Reeves, D., Campbell, S.M., Adams, J., Shekelle, P.G., Kontopantelis, E.: Combining multiple indicators of clinical quality: an evaluation of different analytic approaches. Med. Care 45, 489–496 (2007)
Rosen, A., Wu, J., Chang, B., Berlowitz, D.R., Rakovski, C., Ash, A.S., et al.: Risk adjustment for measuring health outcomes: an application in VA long-term care. Am. J. Med. Qual. 16, 118–127 (2001)
Ryan, A.M., Burgess, J.F., Tompkins, C.P., Wallack, S.S.: The relationship between Medicare’s process of care quality measures and mortality. Inquiry 46, 274–291 (2009)
Semple, J.: Constrained games for evaluating organizational performance. Eur. J. Oper. Res. 96, 103–112 (1996)
Shwartz, M., Ren, J., Peköz, E.A., Wang, X., Cohen, A.B., Restuccia, J.D.: Estimating a composite measure of hospital quality from the Hospital Compare database: differences when using a Bayesian hierarchical latent variable model versus denominator-based weights. Med. Care 46, 778–785 (2008)
Werner, R.M., Bradlow, E.T.: Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA 296, 2694–2702 (2006)
Winston, W., Albright, S.C.: Practical Management Science, 2nd edn. Duxbury, Pacific Grove, CA (2001)
Zaslavsky, A.M., Shaul, J.A., Zaborski, L.B., Cioffi, M.J., Cleary, P.D.: Combining health plan performance indicators into simpler composite measures. Health Care Financ. Rev. 23, 101–115 (2002)
Zimmerman, D.R., Karon, S.L., Arling, G., Clark, B.R., Collins, T., Ross, R., et al.: Development and testing of nursing home quality indicators. Health Care Financ. Rev. 16, 107–127 (1995)
Acknowledgments
This study was supported by VA Health Services Research and Development Service grant IIR 06-260. Findings and conclusions do not necessarily reflect the opinions or policies of the Department of Veterans Affairs.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shwartz, M., Burgess, J.F. & Berlowitz, D. Benefit-of-the-doubt approaches for calculating a composite measure of quality. Health Serv Outcomes Res Method 9, 234–251 (2009). https://doi.org/10.1007/s10742-010-0056-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-010-0056-0