Abstract
Report cards classifying physicians into performance tiers are central to health care quality improvement initiatives. Misclassification is a concern since physicians often have small patient panels for standard performance measures. Given that report cards are used for different purposes by different stakeholders, we specify loss functions and evaluate the potential cost of misclassification for physician report card designs. Monte Carlo simulation to explore misclassification risk and cost for four illustrative physician report card designs and three loss functions representing overall misclassification, patient, and pay-for-performance program perspectives. True physician performance is simulated under a beta-binomial model with parameters yielding simulated true scores resembling previously reported estimates. Misclassification risk varied across report card designs. Overall misclassification risk increased with the number of performance tiers for our simulated scenarios. However, the relationship between misclassification cost and number of tiers was inconsistent across the loss functions. The report card with the lowest misclassification cost varied across stakeholders. Within stakeholder, the costs of a two-tier report card with a high or low hurdle (25th and 75th percentile, respectively) varied. Loss functions and report card designs are illustrative and not intended to exhaustively catalog all possibilities. Little guidance exists on misclassification costs from the patient perspective. Misclassification cost depends on how performance information will be used and by whom. Selecting the lowest-cost design for a given stakeholder could maximize the usefulness of physician performance data. Misclassification cost could guide report card design, improving the usefulness of a report card for one stakeholder without disadvantaging others .
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adams, J.L.: The Reliability of Provider Profiling: A Tutorial. TR-653-NCQA. RAND Corporation, Santa Monica (2009)
Adams, J.L., Mehrotra, A., Thomas, J.W., McGlynn, E.: Physician cost profiling-reliability and risk of misclassification. N. Engl. J. Med. 362(11), 1014–1021 (2010)
Aschwanden, C.: How to fire your doctor. Rule One: Make sure you have another one lined up first. The Washington Post. http://tinyurl.com/jqak4yr (2014). Accessed 22 April 2016
Austin, P.C.: Bayes rules for optimally using bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals. BMC Med. Res. Methodol. 8, 30 (2008)
Austin, P.C., Brunner, L.: Optimal bayesian probability levels for hospital report cards. Health Serv. Outcomes Res. Method. 8, 80–97 (2008)
Baker, L.C., Kessler, D.P., et al.: Expanding patients’ property rights in their medical records. Am. J. Health Econ. 1(1), 82–100 (2015)
Centers for Medicare & Medicaid Services: Computation of the 2016 value modifier fact sheet. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeedbackProgram/Downloads/2016-VM-Fact-Sheet.pdf (2016). Accessed 02 Feb 2016
Davis, M., Hibbard, J., Milstein, A.: Consumer tolerance for inaccuracy in physician performance ratings: one size fits none. Issue Brief (Center for Studying Health System Change) 110, 1–5 (2007)
Draper, D., Liebhaber, A., Ginsburg, P.: High-performance health plan networks: early experiences. Issue Brief (Center for Studying Health System Change) 111, 1 (2007)
Elliott, M.N., Lehrman, W., Goldstein, E., Hambarsoomian, K., Beckett, M.K., Giordano, L.: Do hospitals rank differently on HCAHPS for different patient subgroups? Med. Care Res. Rev. 67(1), 56–73 (2010)
Findlay, S.: Health Policy Brief: Physician Compare. Health Affairs (2015). Accessed 7 April 2016
Friedberg, M.W., Damberg, C.L.: A five-point checklist to help performance reports incentivize improvement and effectively guide patients. Health Aff. 31(3), 612–618 (2012)
HealthInsight: About quality data. http://utahhealthscape.org/?p=about_the_data (2016). Accessed 26 June 2016
Hibbard, J., Sofaer, S.: Best Practices in Public Reporting No. 1: How to Effectively Present Health Care Performance Data to Consumers. Agency for Healthcare Research and Quality, Rockville (2010)
Hofer, T.P., Hayward, R.A., Greenfield, S., Wagner, E.H., Kaplan, S.H., Manning, W.G.: The unreliability of individual physician report cards for assessing the costs and quality of care of a chronic disease. J. Am. Med. Assoc. 281(22), 2098–2105 (1999)
Kaplan, S.H., Griffith, J.L., Price, L.L., Pawlson, L.G., Greenfield, S.: Improving the reliability of physician performance assessment: identifying the “physician effect” on quality and creating composite measures. Med. Care 47(4), 378–387 (2009)
Lee, T.H.: Eulogy for a quality measure. N. Engl. J. Med. 357(12), 1175–1177 (2007)
Lin, R., Louis, T.A., Paddock, S.M., Ridgeway, G.: Loss function based ranking in two-stage, hierarchical models. Bayesian Anal. 1(4), 915 (2006)
Marcotte, L., Moriates, C., Milstein, A.: Professional organizations’ role in supporting physicians to improve value in health care. J. Am. Med. Assoc. 312(3), 231–232 (2014)
Medscape: Medscape physician compensation report 2015. http://www.medscape.com/features/slideshow/compensation/2015/public/overview#page=2 (2015). Accessed 24 July 2016
Office of the Patient Advocate: Scoring documentation for public reporting on CAHPS. http://reportcard.opa.ca.gov/rc/pdfs/2014-15%20Edition%20CAHPS_Final.pdf (2015). Accessed 26 June 2016
Paddock, S.M., Adams, J.L., de la Guardia, F.H.: Better-than-average and worse-than-average hospitals may not significantly differ from average hospitals: an analysis of Medicare Hospital Compare ratings. BMJ Qual. Saf. 24(2), 128–134 (2015)
Robinowitz, D.L., Dudley, R.A.: Public reporting of provider performance: can its impact be made greater? Ann. Rev. Pub. Health 27, 517–536 (2006)
Scholle, S.H., Roski, J., Adams, J.L., Dunn, D.L., Kerr, E.A., Dugan, D.P., Jensen, R.E.: Benchmarking physician performance: reliability of individual and composite measures. Am. J. Manag. Care 14(12), 833 (2008)
Shen, W., Louis, T.A.: Triple-goal estimates in two-stage hierarchical models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 60(2), 455–471 (1998)
Smith, K.A., Sussman, J.B., Bernstein, S.J., Hayward, R.A.: Improving the reliability of physician “report cards”. Med. Care 51(3), 266 (2013)
Wei, P., Allen, M.: Surgeon Scorecard. ProPublica. https://projects.propublica.org/surgeons/ (2015). Accessed 7 April 2016
Acknowledgements
Financial support for this study was provided by Grant 1 R21 HS021860 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The authors are grateful to the referees for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
de la Guardia, F.H., Hwang, J., Adams, J.L. et al. Loss function-based evaluation of physician report cards. Health Serv Outcomes Res Method 18, 96–108 (2018). https://doi.org/10.1007/s10742-018-0179-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-018-0179-2