Loss function-based evaluation of physician report cards

  • Fernando Hoces de la Guardia
  • Jessica Hwang
  • John L. Adams
  • Susan M. PaddockEmail author


Report cards classifying physicians into performance tiers are central to health care quality improvement initiatives. Misclassification is a concern since physicians often have small patient panels for standard performance measures. Given that report cards are used for different purposes by different stakeholders, we specify loss functions and evaluate the potential cost of misclassification for physician report card designs. Monte Carlo simulation to explore misclassification risk and cost for four illustrative physician report card designs and three loss functions representing overall misclassification, patient, and pay-for-performance program perspectives. True physician performance is simulated under a beta-binomial model with parameters yielding simulated true scores resembling previously reported estimates. Misclassification risk varied across report card designs. Overall misclassification risk increased with the number of performance tiers for our simulated scenarios. However, the relationship between misclassification cost and number of tiers was inconsistent across the loss functions. The report card with the lowest misclassification cost varied across stakeholders. Within stakeholder, the costs of a two-tier report card with a high or low hurdle (25th and 75th percentile, respectively) varied. Loss functions and report card designs are illustrative and not intended to exhaustively catalog all possibilities. Little guidance exists on misclassification costs from the patient perspective. Misclassification cost depends on how performance information will be used and by whom. Selecting the lowest-cost design for a given stakeholder could maximize the usefulness of physician performance data. Misclassification cost could guide report card design, improving the usefulness of a report card for one stakeholder without disadvantaging others .


Misclassification risk Report cards Quality measurement Loss function 



Financial support for this study was provided by Grant 1 R21 HS021860 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The authors are grateful to the referees for their helpful comments.


  1. Adams, J.L.: The Reliability of Provider Profiling: A Tutorial. TR-653-NCQA. RAND Corporation, Santa Monica (2009)CrossRefGoogle Scholar
  2. Adams, J.L., Mehrotra, A., Thomas, J.W., McGlynn, E.: Physician cost profiling-reliability and risk of misclassification. N. Engl. J. Med. 362(11), 1014–1021 (2010)CrossRefPubMedPubMedCentralGoogle Scholar
  3. Aschwanden, C.: How to fire your doctor. Rule One: Make sure you have another one lined up first. The Washington Post. (2014). Accessed 22 April 2016
  4. Austin, P.C.: Bayes rules for optimally using bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals. BMC Med. Res. Methodol. 8, 30 (2008)CrossRefPubMedPubMedCentralGoogle Scholar
  5. Austin, P.C., Brunner, L.: Optimal bayesian probability levels for hospital report cards. Health Serv. Outcomes Res. Method. 8, 80–97 (2008)CrossRefGoogle Scholar
  6. Baker, L.C., Kessler, D.P., et al.: Expanding patients’ property rights in their medical records. Am. J. Health Econ. 1(1), 82–100 (2015)CrossRefGoogle Scholar
  7. Centers for Medicare & Medicaid Services: Computation of the 2016 value modifier fact sheet. (2016). Accessed 02 Feb 2016
  8. Davis, M., Hibbard, J., Milstein, A.: Consumer tolerance for inaccuracy in physician performance ratings: one size fits none. Issue Brief (Center for Studying Health System Change) 110, 1–5 (2007)Google Scholar
  9. Draper, D., Liebhaber, A., Ginsburg, P.: High-performance health plan networks: early experiences. Issue Brief (Center for Studying Health System Change) 111, 1 (2007)Google Scholar
  10. Elliott, M.N., Lehrman, W., Goldstein, E., Hambarsoomian, K., Beckett, M.K., Giordano, L.: Do hospitals rank differently on HCAHPS for different patient subgroups? Med. Care Res. Rev. 67(1), 56–73 (2010)CrossRefPubMedGoogle Scholar
  11. Findlay, S.: Health Policy Brief: Physician Compare. Health Affairs (2015). Accessed 7 April 2016Google Scholar
  12. Friedberg, M.W., Damberg, C.L.: A five-point checklist to help performance reports incentivize improvement and effectively guide patients. Health Aff. 31(3), 612–618 (2012)CrossRefGoogle Scholar
  13. HealthInsight: About quality data. (2016). Accessed 26 June 2016
  14. Hibbard, J., Sofaer, S.: Best Practices in Public Reporting No. 1: How to Effectively Present Health Care Performance Data to Consumers. Agency for Healthcare Research and Quality, Rockville (2010)Google Scholar
  15. Hofer, T.P., Hayward, R.A., Greenfield, S., Wagner, E.H., Kaplan, S.H., Manning, W.G.: The unreliability of individual physician report cards for assessing the costs and quality of care of a chronic disease. J. Am. Med. Assoc. 281(22), 2098–2105 (1999)CrossRefGoogle Scholar
  16. Kaplan, S.H., Griffith, J.L., Price, L.L., Pawlson, L.G., Greenfield, S.: Improving the reliability of physician performance assessment: identifying the “physician effect” on quality and creating composite measures. Med. Care 47(4), 378–387 (2009)CrossRefPubMedGoogle Scholar
  17. Lee, T.H.: Eulogy for a quality measure. N. Engl. J. Med. 357(12), 1175–1177 (2007)CrossRefPubMedGoogle Scholar
  18. Lin, R., Louis, T.A., Paddock, S.M., Ridgeway, G.: Loss function based ranking in two-stage, hierarchical models. Bayesian Anal. 1(4), 915 (2006)CrossRefPubMedPubMedCentralGoogle Scholar
  19. Marcotte, L., Moriates, C., Milstein, A.: Professional organizations’ role in supporting physicians to improve value in health care. J. Am. Med. Assoc. 312(3), 231–232 (2014)CrossRefGoogle Scholar
  20. Medscape: Medscape physician compensation report 2015. (2015). Accessed 24 July 2016
  21. Office of the Patient Advocate: Scoring documentation for public reporting on CAHPS. (2015). Accessed 26 June 2016
  22. Paddock, S.M., Adams, J.L., de la Guardia, F.H.: Better-than-average and worse-than-average hospitals may not significantly differ from average hospitals: an analysis of Medicare Hospital Compare ratings. BMJ Qual. Saf. 24(2), 128–134 (2015)CrossRefPubMedGoogle Scholar
  23. Robinowitz, D.L., Dudley, R.A.: Public reporting of provider performance: can its impact be made greater? Ann. Rev. Pub. Health 27, 517–536 (2006)CrossRefGoogle Scholar
  24. Scholle, S.H., Roski, J., Adams, J.L., Dunn, D.L., Kerr, E.A., Dugan, D.P., Jensen, R.E.: Benchmarking physician performance: reliability of individual and composite measures. Am. J. Manag. Care 14(12), 833 (2008)PubMedPubMedCentralGoogle Scholar
  25. Shen, W., Louis, T.A.: Triple-goal estimates in two-stage hierarchical models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 60(2), 455–471 (1998)CrossRefGoogle Scholar
  26. Smith, K.A., Sussman, J.B., Bernstein, S.J., Hayward, R.A.: Improving the reliability of physician “report cards”. Med. Care 51(3), 266 (2013)CrossRefPubMedPubMedCentralGoogle Scholar
  27. Wei, P., Allen, M.: Surgeon Scorecard. ProPublica. (2015). Accessed 7 April 2016

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Fernando Hoces de la Guardia
    • 1
  • Jessica Hwang
    • 1
    • 2
  • John L. Adams
    • 1
    • 3
  • Susan M. Paddock
    • 1
    Email author
  1. 1.RAND CorporationSanta MonicaUSA
  2. 2.Statistics DepartmentStanford UniversityPalo AltoUSA
  3. 3.Kaiser PermanentePasadenaUSA

Personalised recommendations