Skip to main content
SpringerLink
Log in

Setting a Fair Performance Standard for Physicians’ Quality of Patient Care

  • Original Research
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

Background

Assessing physicians’ clinical performance using statistically sound, evidence-based measures is challenging. Little research has focused on methodological approaches to setting performance standards to which physicians are being held accountable.

Objective

Determine if a rigorous approach for setting an objective, credible standard of minimally-acceptable performance could be used for practicing physicians caring for diabetic patients.

Design

Retrospective cohort study.

Participants

Nine hundred and fifty-seven physicians from the United States with time-limited certification in internal medicine or a subspecialty.

Main Measures

The ABIM Diabetes Practice Improvement Module was used to collect data on ten clinical and two patient experience measures. A panel of eight internists/subspecialists representing essential perspectives of clinical practice applied an adaptation of the Angoff method to judge how physicians who provide minimally-acceptable care would perform on individual measures to establish performance thresholds. Panelists then rated each measure’s relative importance and the Dunn–Rankin method was applied to establish scoring weights for the composite measure. Physician characteristics were used to support the standard-setting outcome.

Key Results

Physicians abstracted 20,131 patient charts and 18,974 patient surveys were completed. The panel established reasonable performance thresholds and importance weights, yielding a standard of 48.51 (out of 100 possible points) on the composite measure with high classification accuracy (0.98). The 38 (4%) outlier physicians who did not meet the standard had lower ratings of overall clinical competence and professional behavior/attitude from former residency program directors (p = 0.01 and p = 0.006, respectively), lower Internal Medicine certification and maintenance of certification examination scores (p = 0.005 and p < 0.001, respectively), and primarily worked as solo practitioners (p = 0.02).

Conclusions

The standard-setting method yielded a credible, defensible performance standard for diabetes care based on informed judgment that resulted in a reasonable, reproducible outcome. Our method represents one approach to identifying outlier physicians for intervention to protect patients.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Miller TP, Brennan TA, Milstein A. How can we make more progress in measuring physicians' performance to improve the value of care? Health Aff. 2009;28:1429–1437.

    Article  Google Scholar 

  2. Holmboe ES. Assessment of the practicing physician: Challenges and opportunities. J Contin Educ Health Prof. 2008;28(Suppl 1):4–10.

    Article  Google Scholar 

  3. Landon BE, Normand S-LT. Performance measurement in the small office practice: Challenges and potential solutions. Ann Intern Med. 2008;148:353–357.

    PubMed  Google Scholar 

  4. Landon BE, Normand S-LT, Blumenthal D, Daley J. Physician clinical performance assessment: Prospects and barriers. JAMA. 2003;290:1183–1189.

    Article  PubMed  CAS  Google Scholar 

  5. Scholle SH, Pawlson LG, Solberg LI, et al. Measuring practice systems for chronic illness care: Accuracy of self-reports from clinical personnel. Jt Comm J Qual Patient Saf. 2008;34:407–416.

    PubMed  Google Scholar 

  6. Kaplan SH, Griffith JL, Price LL, Pawlson LG, Greenfield S. Improving the reliability of physician performance assessment: Identifying the "physician effect" on quality and creating composite measures. Med Care. 2009;47:378–387.

    Article  PubMed  Google Scholar 

  7. Lipner RS, Weng W, Arnold GK, Duffy FD, Lynn LA, Holmboe ES. A three-part model for measuring diabetes care in physician practice. Acad Med. 2007;82(Suppl 10):S48–S52.

    Article  PubMed  Google Scholar 

  8. Weng W, Hess BJ, Lynn LA, Holmboe ES, Lipner RS. Measuring physicians’ performance in clinical practice: Reliability, classification accuracy, and validity. Eval Health Prof. 2010;33:302–320.

    Article  Google Scholar 

  9. Cizek GJ. Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, NJ: Lawrence Erlbaum; 2001.

    Google Scholar 

  10. Angoff WH. Scales, norms, and equivalent scores. In: Thorndike RL, ed. Educational Measurement. American Council on Education: Washington, DC; 1971:514–515.

    Google Scholar 

  11. Downing SA, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006;18:50–57.

    Article  PubMed  Google Scholar 

  12. Boulet JR, De Champlain AF, McKinley DW. Setting defensible performance standards on OSCEs and standardized patient examinations. Med Teach. 2003;25:245–249.

    Article  PubMed  Google Scholar 

  13. McKinley DW, Boulet JR, Hambleton RK. A work-centered approach for setting passing scores on performance-based assessments. Eval Health Prof. 2005;28:349–369.

    Article  PubMed  Google Scholar 

  14. Duffy FD, Lynn LA, Didura H, et al. Self-assessment of practice performance: Development of the ABIM Practice Improvement Module (PIMSM). J Contin Educ Health Prof. 2008;28:38–46.

    Article  PubMed  Google Scholar 

  15. American Diabetes Association. Standards of medical care in diabetes-2009. Diab Care. 2009;32(Suppl 1):S13–S61.

    Article  Google Scholar 

  16. Rittenhouse DR, Shortell SM. The patient-centered medical home. JAMA. 2009;301:2038–2040.

    Article  PubMed  CAS  Google Scholar 

  17. Dunn–Rankin P. Scaling Methods. Hillsdale, NY: Erlbaum; 1983.

    Google Scholar 

  18. Reeves D, Campbell SM, Adams J, Shekelle PG, Kontopantelis E, Roland MO. Combining multiple indicators of clinical quality: An evaluation of different analytic approaches. Med Care. 2007;45:489–496.

    Article  PubMed  Google Scholar 

  19. Mosier C. On the reliability of a weighted composite. Psychometrika. 1943;8:161–168.

    Article  Google Scholar 

  20. Clauser BE, Margolis MJ, Case SM. Testing for licensure and certification in the professions. In: Brennan RL, ed. Educational Measurement. 4th ed. Westport, CT: Praeger Publishers; 2006:701–731.

    Google Scholar 

  21. SAS Institute. Statistical Analysis System. Version 9.1. Cary, NC: SAS Institute; 2002.

  22. American Board of Internal Medicine. ABIM HIPAA Business Associate Agreement. Available at: http://www.abim.org/pdf/hipaa/hipaa_compliance.pdf. Accessed October 19, 2010.

  23. Norcini JJ, Shea JA. The credibility and comparability of standards. Appl Meas Ed. 1997;10:39–59.

    Article  Google Scholar 

  24. Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries. Arch Intern Med. 2008;168:1396–1403.

    Article  PubMed  Google Scholar 

  25. Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ. 2002;36:853–859.

    Article  PubMed  Google Scholar 

  26. Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288:3019–3026.

    Article  PubMed  Google Scholar 

  27. Eddy DM, Schlessinger L. Validation of the Archimedes diabetes model. Diab Care. 2003;26:3102–3110.

    Article  Google Scholar 

  28. Greenfield S, Kaplan SH, Kahn R, Nimomiya J, Griffith JL. Profiling care provided by different groups of physicians: Effects of patient case-mix (bias) and physician-level clustering on quality assessment results. Ann Intern Med. 2002;136:111–121.

    PubMed  Google Scholar 

  29. Holmboe ES, Meehan TP, Lynn L, Doyle P, Sherwin T, Duffy FD. Promoting physicians’ self-assessment and quality improvement: The ABIM Diabetes Practice Improvement Module. J Contin Ed Health Prof. 2006;26:109–119.

    Article  Google Scholar 

  30. Holmboe ES, Weng W, Arnold GF, et al. The comprehensive care project: Measuring physician performance in ambulatory practice. Health Serv Res. 2010;45:1912–1933.

    Google Scholar 

Download references

Contributors

We thank the standard-setting panel, and Dr. Gerald Arnold and Leslie Tucker for their help with the scoring strategy and manuscript preparation, respectively.

Funders

The ABIM Foundation funded this study.

Prior Presentations

An abstract of this study was presented at the annual AcademyHealth meeting on 28 June 2009.

Conflict of Interest

All authors are employed by the ABIM. Drs. Hess, Weng, and Lipner are co-inventors of a business method invention describing the application of the standard-setting method to practicing physicians. The invention is patent pending. Dr. Holmboe received honoraria for teaching about clinical assessment from the Uniformed Services University of the Health Sciences, the University of Kansas, and the Harvard-Macy Systems Assessment Course. Dr. Holmboe receives royalties for a textbook on assessment published by Mosby-Elsevier.

Author information

Authors and Affiliations

  1. American Board of Internal Medicine, 510 Walnut Street, Suite 1700, Philadelphia, PA, 19106, USA

    Brian J. Hess PhD, Weifeng Weng PhD, Lorna A. Lynn MD, Eric S. Holmboe MD & Rebecca S. Lipner PhD

Authors
  1. Brian J. Hess PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weifeng Weng PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lorna A. Lynn MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric S. Holmboe MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rebecca S. Lipner PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brian J. Hess PhD.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Appendix A

Process for Determining Minimum Performance Thresholds and Point Values (Scoring Weights) for Each Measure. (DOC 41 kb)

Appendix B

The Distribution of Composite Measure Scores and the Standard Representing Minimally-Acceptable Diabetes Care (N = 957 Physicians). (DOC 46 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hess, B.J., Weng, W., Lynn, L.A. et al. Setting a Fair Performance Standard for Physicians’ Quality of Patient Care. J GEN INTERN MED 26, 467–473 (2011). https://doi.org/10.1007/s11606-010-1572-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-010-1572-x

KEY WORDS

Search

Navigation