Abstract
Provider profiling as a means to describe and compare the performance of health care professionals has gained momentum in the past decade. As a key component of pay-for-performance programs profiling has been increasingly used to identify top-performing providers. However, rigorous examination of the performance of statistical methods for profiling when used to classify top-performing providers is lacking. The objective of this study was to compare the classification accuracy of three methods for identifying providers exceeding performance thresholds and to analyze data on satisfaction with mental health care providers at Group Health Cooperative using these methods. Questionnaire data on patient satisfaction with mental health care providers at Group Health Cooperative was collected between April 2008 and January 2010. Simulated data were used to compare the classification accuracy of alternative statistical methods. We evaluated sensitivity, specificity, and root mean squared error of alternative statistical methods using simulated data. For Group Health providers, we compared agreement of alternative approaches to classification. We found that when between-provider variability in performance was low, all three methods exhibited poor classification accuracy. When used to evaluate mental health care provider performance, we found substantial uncertainty in the estimates and poor agreement across methods. Based on these findings, we recommend providing uncertainty estimates for provider rankings and caution against the use of any classification method when between-provider variability is low.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alliance of Community Health Plans: Quality incentives for network providers. http://www.achp.org/permalink/1213.html (2009). Accessed 22 Jan 2012
Austin, P.C.: The reliability and validity of Bayesian measures for hospital profiling: a Monte Carlo assessment. J. Stat. Plan. Inference 128(1), 109–122 (2005)
Austin, P.C.: Bayes rules for optimally using Bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals. BMC Med. Res. Methodol. 8, 30 (2008)
Austin, P.C., Naylor, C.D., Tu, J.V.: A comparison of a Bayesian vs. a frequentist method for profiling hospital performance. J. Eval. Clin. Pract. 7(1), 35–45 (2001)
Austin, P.C., Alter, D.A., Tu, J.V.: The use of fixed- and random-effects models for classifying hospitals as mortality outliers: a Monte Carlo assessment. Med. Decis. Making 23(6), 526–539 (2003)
Bryk, A.S., Raudenbush, S.W.: Hierarchical linear models. Sage, Newbury Park (1992)
Burgess, J.F., Christiansen, C.L., Michalak, S.E., Morris, C.N.: Medical profiling: improving standards and risk adjustments using hierarchical models. J. Health Econ. 19(3), 291–309 (2000)
Christiansen, C.L., Morris, C.N.: Improving the statistical approach to health care provider profiling. Ann. Intern. Med. 127(8 Pt 2), 764–768 (1997)
Cohen, M.E., Dimick, J.B., Bilimoria, K.Y., Ko, C.Y., Richards, K., Hall, B.L.: Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling. J. Am. Coll. Surg. 209(6), 687–693 (2009)
Crosier, M., Scott, J., Steinfeld, B.: Improving satisfaction in patients receiving mental health care: a case study. J. Behav. Health Serv. Res. 39(1), 42–54 (2012)
D’Errigo, P., Tosti, M.E., Fusco, D., Perucci, C.A., Seccareccia, F.: Use of hierarchical models to evaluate performance of cardiac surgery centres in the Italian CABG outcome study. BMC Med. Res. Methodol. 7, 29 (2007)
DeLong, E.R., Peterson, E.D., DeLong, D.M., Muhlbaier, L.H., Hackett, S., Mark, D.B.: Comparing risk-adjustment methods for provider profiling. Stat. Med. 16(23), 2645–2664 (1997)
Epstein, A.M., Lee, T.H., Hamel, M.B.: Paying physicians for high-quality care. N. Engl. J. Med. 350(4), 406–410 (2004)
Glance, L.G., Dick, A.W., Osler, T.M., Mukamel, D.B.: Accuracy of hospital report cards based on administrative data. Health Serv. Res. 41(4), 1413–1437 (2006)
Hauck, K., Rice, N., Smith, P.: The influence of health care organisations on health system performance. J. Health Serv. Res. Policy 8(2), 68–74 (2003)
Heagerty, P.J., Kurland, B.F.: Misspecified maximum likelihood estimates and generalised linear mixed models. Biometrika 88(4), 973–985 (2001)
Heidelberger, P., Welch, P.D.: Simulation run length control in the presence of an initial transient. Oper. Res. 31(6), 1109–1144 (1983)
Kak, N., Burkhalter, B., Cooper, M.-A.: Measuring the competency of healthcare providers. Qual. Assur. Proj. 2(1), 1–28 (2001)
Katon, W., Rutter, C.M., Lin, E., Simon, G., Von Korff, M., Bush, T., Walker, E., Ludman, E.: Are there detectable differences in quality of care or outcome of depression across primary care providers? Med. Care 38(6), 552–561 (2000)
Koehler, E., Brown, E., Haneuse, S.J.: On the assessment of Monte Carlo error in simulation-based statistical analyses. Am. Stat. 63(2), 155–162 (2009)
Lebow, J.L.: Consumer satisfaction with mental health treatment. Psychol. Bull. 91(2), 244–259 (1982)
Lockwood, J.R., Louis, T.A., McCaffrey, D.F.: Uncertainty in rank estimation: implications for value-added modeling accountability systems. J. Educ. Behav. Stat. 27(3), 255–270 (2002)
Marshall, M.N., Shekelle, P.G., Leatherman, S., Brook, R.H.: The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 283(14), 1866–1874 (2000)
McCulloch, C.E., Neuhaus, J.M.: Prediction of random effects in linear and generalized linear models under model misspecification. Biometrics 67(1), 270–279 (2011)
Ohlssen, D.I., Sharples, L.D., Spiegelhalter, D.J.: Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons. Stat. Med. 26(9), 2088–2112 (2007a)
Ohlssen, D.I., Sharples, L.D., Spiegelhalter, D.J.: A hierarchical modelling framework for identifying unusual performance in health care providers. J. R. Stat. Soc. Ser. A Stat. Soc. 170(4), 865–890 (2007b)
Petersen, L.A., Woodard, L.D., Urech, T., Daw, C., Sookanan, S.: Does pay-for-performance improve the quality of health care? Ann. Intern. Med. 145(4), 265–272 (2006)
Racz, M.J., Sedransk, J.: Bayesian and frequentist methods for provider profiling using risk-adjusted assessments of medical outcomes. J. Am. Stat. Assoc. 105(489), 48–58 (2010)
Rosenthal, M.B., Dudley, R.A.: Pay-for-performance: will the latest payment trend improve care? JAMA 297(7), 740–744 (2007)
Rosenthal, M.B., Frank, R.G.: What is the empirical basis for paying for quality in health care? Med. Care Res. Rev. 63(2), 135–157 (2006)
Shahian, D.M., Normand, S.-L.T.: Comparison of “risk-adjusted” hospital outcomes. Circulation 117(15), 1955–1963 (2008)
Shahian, D.M., Silverstein, T., Lovett, A.F., Wolf, R.E., Normand, S.-L.T.: Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation 115(12), 1518–1527 (2007)
Simon, G., Rutter, C., Crosier, M., Scott, J., Operskalski, B.H., Ludman, E.: Are comparisons of consumer satisfaction with providers biased by nonresponse or case-mix differences? Psychiatr. Serv. 60(1), 67–73 (2009)
Simon, G.E., Ludman, E.J.: Predictors of early dropout from psychotherapy for depression in community practice. Psychiatr. Serv. 61(7), 684–689 (2010)
Smith, P.C.: Measuring health system performance. Eur. J. Health Econ. 60(3), 145–148 (2002)
Acknowledgments
This research was supported by National Institute of Mental Health grant P20 MH068572.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ding, V.Y., Hubbard, R.A., Rutter, C.M. et al. Assessing the accuracy of profiling methods for identifying top providers: performance of mental health care providers. Health Serv Outcomes Res Method 13, 1–17 (2013). https://doi.org/10.1007/s10742-012-0099-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-012-0099-5