Journal of General Internal Medicine

, Volume 34, Issue 11, pp 2575–2579 | Cite as

A Comparison of Online Physician Ratings and Internal Patient-Submitted Ratings from a Large Healthcare System

  • Kanu OkikeEmail author
  • Natalie R. Uhr
  • Sherry Y. M. Shin
  • Kristal C. Xie
  • Chong Y. Kim
  • Tadashi T. Funahashi
  • Michael H. Kanter
Original research



Physician online ratings are ubiquitous and influential, but they also have their detractors. Given the lack of scientific survey methodology used in online ratings, some health systems have begun to publish their own internal patient-submitted ratings of physicians.


The purpose of this study was to compare online physician ratings with internal ratings from a large healthcare system.


Retrospective cohort study comparing online ratings with internal ratings from a large healthcare system.


Kaiser Permanente, a large integrated healthcare delivery system.


Physicians in the Southern California region of Kaiser Permanente, including all specialties with ambulatory clinic visits.

Main Measures

The primary outcome measure was correlation between online physician ratings and internal ratings from the integrated healthcare delivery system.


Of 5438 physicians who met inclusion and exclusion criteria, 4191 (77.1%) were rated both online and internally. The online ratings were based on a mean of 3.5 patient reviews, while the internal ratings were based on a mean of 119 survey returns. The overall correlation between the online and internal ratings was weak (Spearman’s rho .23), but increased with the number of reviews used to formulate each online rating.


Physician online ratings did not correlate well with internal ratings from a large integrated healthcare delivery system, although the correlation increased with the number of reviews used to formulate each online rating. Given that many consumers are not aware of the statistical issues associated with small sample sizes, we would recommend that online rating websites refrain from displaying a physician’s rating until the sample size is sufficiently large (for example, at least 15 patient reviews). However, hospitals and health systems may be able to provide better information for patients by publishing the internal ratings of their physicians.


physician online ratings physician online reviews patient satisfaction 



The authors would like to thank Mimi Hugh MS MPH and the Kaiser Permanente Southern California Member Appraisal of Physician/Provider Services (MAPPS) Committee for their assistance with this study, as well as all the physicians of the Southern California Permanente Medical Group.

Compliance with Ethical Standards

All elements of the study were approved by the Kaiser Permanente Institutional Review Board (IRB).

Conflict of Interest

The authors declare that they do not have a conflict of interest.


  1. 1.
    Atkinson S. Current status of online rating of Australian doctors. Aust J Prim Health. 2014;20(3):222–3.CrossRefGoogle Scholar
  2. 2.
    Bakhsh W, Mesfin A. Online ratings of orthopedic surgeons: analysis of 2185 reviews. Am J Orthop (Belle Mead NJ). 2014;43(8):359–63.PubMedGoogle Scholar
  3. 3.
    Black EW, Thompson LA, Saliba H, Dawson K, Black NM. An analysis of healthcare providers’ online ratings. Inform Prim Care. 2009;17(4):249–53.PubMedGoogle Scholar
  4. 4.
    Detz A, Lopez A, Sarkar U. Long-term doctor-patient relationships: patient perspective from online reviews. J Med Internet Res. 2013;15(7):e131.CrossRefGoogle Scholar
  5. 5.
    Ellimoottil C, Hart A, Greco K, Quek ML, Farooq A. Online reviews of 500 urologists. J Urol. 2013;189(6):2269–73.CrossRefGoogle Scholar
  6. 6.
    Emmert M, Meier F. An analysis of online evaluations on a physician rating website: evidence from a German public reporting instrument. J Med Internet Res. 2013;15(8):e157.CrossRefGoogle Scholar
  7. 7.
    Emmert M, Meier F, Heider AK, Durr C, Sander U. What do patients say about their physicians? an analysis of 3000 narrative comments posted on a German physician rating website. Health Policy. 2014;118(1):66–73.CrossRefGoogle Scholar
  8. 8.
    Gao GG, McCullough JS, Agarwal R, Jha AK. A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res. 2012;14(1):e38.CrossRefGoogle Scholar
  9. 9.
    Kadry B, Chu LF, Kadry B, Gammas D, Macario A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011;13(4):e95.CrossRefGoogle Scholar
  10. 10.
    Lopez A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012;27(6):685–92.CrossRefGoogle Scholar
  11. 11.
    Merrell JG, Levy BH, 3rd, Johnson DA. Patient assessments and online ratings of quality care: a “wake-up call” for providers. Am J Gastroenterol. 2013;108(11):1676–85.CrossRefGoogle Scholar
  12. 12.
    Sabin JE. Physician-rating websites. Virtual Mentor. 2013;15(11):932–6.PubMedGoogle Scholar
  13. 13.
    Segal J, Sacopulos M, Sheets V, Thurston I, Brooks K, Puccia R. Online doctor reviews: do they track surgeon volume, a proxy for quality of care? J Med Internet Res. 2012;14(2):e50.CrossRefGoogle Scholar
  14. 14.
    Wallace BC, Paul MJ, Sarkar U, Trikalinos TA, Dredze M. A large-scale quantitative analysis of latent factors and sentiment in online doctor reviews. J Am Med Inform Assoc. 2014;21(6):1098–103.CrossRefGoogle Scholar
  15. 15.
    Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014;311(7):734–5.CrossRefGoogle Scholar
  16. 16.
    Strech D. Ethical principles for physician rating sites. J Med Internet Res. 2011;13(4):e113.CrossRefGoogle Scholar
  17. 17.
    Segal J, Sacopulos MJ, Rivera DJ. Legal remedies for online defamation of physicians. J Leg Med. 2009;30(3):349–88.CrossRefGoogle Scholar
  18. 18.
    Emmert M, Sander U, Pisch F. Eight questions about physician-rating websites: a systematic review. J Med Internet Res. 2013;15(2):e24.CrossRefGoogle Scholar
  19. 19.
    Daskivich TJ, Houman J, Fuller G, Black JT, Kim HL, Spiegel B. Online physician ratings fail to predict actual performance on measures of quality, value, and peer review. J Am Med Inform Assoc. 2018;25(4):401–407.CrossRefGoogle Scholar
  20. 20.
    Okike K, Peter-Bibb TK, Xie KC, Okike ON. Association Between Physician Online Rating and Quality of Care. J Med Internet Res. 2016;18(12):e324.CrossRefGoogle Scholar
  21. 21.
    Randhawa S, Viqar A, Strother J, et al. How Do Patients Rate Their Radiation Oncologists in the Modern Era: An Analysis of Cureus. 2018;10(9):e3312.PubMedPubMedCentralGoogle Scholar
  22. 22.
    Press Ganey Associates Inc. “Patient Experience - Patient-Centered Care.” Available online at: Last accessed 26 June 2019.
  23. 23.
    Agency for Healthcare Research and Quality. “CAHPS Clinician & Group Survey.” Available online at: Last accessed 26 June 2019.
  24. 24.
    Chen J, Presson A, Zhang C, Ray D, Finlayson S, Glasgow R. Online physician review websites poorly correlate to a validated metric of patient satisfaction. J Surg Res. 2018;227:1–6.CrossRefGoogle Scholar
  25. 25.
    Ricciardi BF, Waddell BS, Nodzo SR, et al. Provider-Initiated Patient Satisfaction Reporting Yields Improved Physician Ratings Relative to Online Rating Websites. Orthopedics. 2017;40(5):304–310.CrossRefGoogle Scholar
  26. 26.
    Widmer RJ, Maurer MJ, Nayar VR, et al. Online Physician Reviews Do Not Reflect Patient Satisfaction Survey Responses. Mayo Clin Proc. 2018;93(4):453–457.CrossRefGoogle Scholar
  27. 27.
    Ryan T, Specht J, Smith S, DelGaudio JM. Does the Press Ganey Survey Correlate to Online Health Grades for a Major Academic Otolaryngology Department? Otolaryngol Head Neck Surg. 2016;155(3):411–5.CrossRefGoogle Scholar
  28. 28.
    Smith A, Anderson M. “Online Shopping and E-Commerce.” Pew Research Center, 2016 December. Available online at: . Last accessed 26 June 2019.
  29. 29.
    McKinstry B, Yang SY. Do patients care about the age of their general practitioner? A questionnaire survey in five practices. Br J Gen Pract. 1994;44(385):349–51.PubMedPubMedCentralGoogle Scholar
  30. 30.
    Kaiser Permanente. “Southern California Fast Facts.” Kaiser Permanente. 2019 March. Available online at: Last accessed 26 June 2019.
  31. 31.
    Tsugawa Y, Jena AB, Orav EJ, et al. Age and sex of surgeons and mortality of older surgical patients: observational study. BMJ. 2018;361:k1343.CrossRefGoogle Scholar
  32. 32.
    Tsugawa Y, Newhouse JP, Zaslavsky AM, Blumenthal DM, Jena AB. Physician age and outcomes in elderly patients in hospital in the US: observational study. BMJ. 2017;357:j1797.CrossRefGoogle Scholar
  33. 33.
    Lee V. Transparency and Trust - Online Patient Reviews of Physicians. N Engl J Med. 2017;376(3):197–199.CrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine 2019

Authors and Affiliations

  • Kanu Okike
    • 1
    Email author
  • Natalie R. Uhr
    • 2
  • Sherry Y. M. Shin
    • 3
  • Kristal C. Xie
    • 4
  • Chong Y. Kim
    • 5
  • Tadashi T. Funahashi
    • 5
  • Michael H. Kanter
    • 6
    • 7
  1. 1.Hawaii Permanente Medical Group Kaiser Moanalua Medical CenterHonoluluUSA
  2. 2.Harvard UniversityCambridgeUSA
  3. 3.University of California–Los AngelesLos AngelesUSA
  4. 4.Boston UniversityBostonUSA
  5. 5.Southern California Permanente Medical GroupPasadenaUSA
  6. 6.Department of Clinical ScienceKaiser Permanente, School of MedicinePasadenaUSA
  7. 7.Department of Research and EvaluationSouthern California Permanente Medical GroupPasadenaUSA

Personalised recommendations