Loss function-based evaluation of physician report cards
Report cards classifying physicians into performance tiers are central to health care quality improvement initiatives. Misclassification is a concern since physicians often have small patient panels for standard performance measures. Given that report cards are used for different purposes by different stakeholders, we specify loss functions and evaluate the potential cost of misclassification for physician report card designs. Monte Carlo simulation to explore misclassification risk and cost for four illustrative physician report card designs and three loss functions representing overall misclassification, patient, and pay-for-performance program perspectives. True physician performance is simulated under a beta-binomial model with parameters yielding simulated true scores resembling previously reported estimates. Misclassification risk varied across report card designs. Overall misclassification risk increased with the number of performance tiers for our simulated scenarios. However, the relationship between misclassification cost and number of tiers was inconsistent across the loss functions. The report card with the lowest misclassification cost varied across stakeholders. Within stakeholder, the costs of a two-tier report card with a high or low hurdle (25th and 75th percentile, respectively) varied. Loss functions and report card designs are illustrative and not intended to exhaustively catalog all possibilities. Little guidance exists on misclassification costs from the patient perspective. Misclassification cost depends on how performance information will be used and by whom. Selecting the lowest-cost design for a given stakeholder could maximize the usefulness of physician performance data. Misclassification cost could guide report card design, improving the usefulness of a report card for one stakeholder without disadvantaging others .
KeywordsMisclassification risk Report cards Quality measurement Loss function
Financial support for this study was provided by Grant 1 R21 HS021860 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The authors are grateful to the referees for their helpful comments.
- Aschwanden, C.: How to fire your doctor. Rule One: Make sure you have another one lined up first. The Washington Post. http://tinyurl.com/jqak4yr (2014). Accessed 22 April 2016
- Centers for Medicare & Medicaid Services: Computation of the 2016 value modifier fact sheet. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeedbackProgram/Downloads/2016-VM-Fact-Sheet.pdf (2016). Accessed 02 Feb 2016
- Davis, M., Hibbard, J., Milstein, A.: Consumer tolerance for inaccuracy in physician performance ratings: one size fits none. Issue Brief (Center for Studying Health System Change) 110, 1–5 (2007)Google Scholar
- Draper, D., Liebhaber, A., Ginsburg, P.: High-performance health plan networks: early experiences. Issue Brief (Center for Studying Health System Change) 111, 1 (2007)Google Scholar
- Findlay, S.: Health Policy Brief: Physician Compare. Health Affairs (2015). Accessed 7 April 2016Google Scholar
- HealthInsight: About quality data. http://utahhealthscape.org/?p=about_the_data (2016). Accessed 26 June 2016
- Hibbard, J., Sofaer, S.: Best Practices in Public Reporting No. 1: How to Effectively Present Health Care Performance Data to Consumers. Agency for Healthcare Research and Quality, Rockville (2010)Google Scholar
- Medscape: Medscape physician compensation report 2015. http://www.medscape.com/features/slideshow/compensation/2015/public/overview#page=2 (2015). Accessed 24 July 2016
- Office of the Patient Advocate: Scoring documentation for public reporting on CAHPS. http://reportcard.opa.ca.gov/rc/pdfs/2014-15%20Edition%20CAHPS_Final.pdf (2015). Accessed 26 June 2016
- Wei, P., Allen, M.: Surgeon Scorecard. ProPublica. https://projects.propublica.org/surgeons/ (2015). Accessed 7 April 2016