Abstract
Healthcare quality measures are statistics that serve to evaluate healthcare providers and identify those that need to improve their care. Before using these measures in clinical practice, developers and reviewers assess measure reliability, which describes the degree to which differences in the measure values reflect actual variation in healthcare quality, as opposed to random noise. The Inter-Unit Reliability (IUR) is a popular statistic for assessing reliability, and it describes the proportion of total variation in a measure that is attributable to between-provider variation. However, Kalbfleisch et al. (Health Services and Outcomes Research Methodology, 18, 215–225, (2018)) have argued that the IUR has a severe limitation in that some of the between-provider variation may be unrelated to quality of care. In this paper, we illustrate the practical implications of this limitation through several concrete examples. We show that certain best-practices in measure development, such as careful risk adjustment and exclusion of unstable measure values, can decrease the sample IUR value. These findings uncover potential negative consequences of discarding measures with IUR values below some arbitrary threshold.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets that support this work are not publicly available because they contain personally identifiable information. More information is available at https://mycrownweb.org.
References
Adams, J.: The reliability of provider profiling: A tutorial. Available at: http://www.rand.org/pubs/technical_reports/TR653.html (2009)
Ash, A., Fienberg, S., Louis, T., Normand, S., Stukel, T., Utts, J.: Statistical issues in assessing hospital performance. Commissioned by the Committee of Presidents of Statistical Societies for the Centers for Medicare and Medicaid Services (CMS). Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/Statistical-Issues-in-Assessing-Hospital-Performance.pdf (2012)
Aubert, O., Yoo, D., Zielinski, D., Cozzi, E., Cardillo, M., Dürr, M., Loupy, A.: COVID-19 pandemic and worldwide organ transplantation: A population-based study. Lancet Public Health 6(10), e709–e719 (2021)
Centers for Medicare and Medicaid Services. End Stage Renal Disease Quality Reporting System. Available at: https://mycrownweb.org (2022)
Chen, B., Lawson, K., Finelli, A., Saarela, O.: Causal variance decompositions for institutional comparisons in healthcare. Stat. Methods Med. Res. 29(7), 1972–1986 (2020)
Estes, J., Chen, Y., Şentürk, D., Rhee, C., Kürüm, E., You, A., Nguyen, D.: Profiling dialysis facilities for adverse recurrent events. Stat. Med. 39(9), 1374–1389 (2020)
Estes, J., Nguyen, D., Chen, Y., Dalrymple, L., Rhee, C., Kalantar-Zadeh, K., Sentürk, D.: Time-dynamic profiling with application to hospital readmission among patients on dialysis. Biometrics 74(4), 1383–1394 (2018)
He, K., Dahlerus, C., Xia, L., Li, Y., Kalbfleisch, J.: The profile inter-unit reliability. Biometrics 76(2), 654–663 (2019)
He, K., Kalbfleisch, J., Li, Y., Li, Y.: Evaluating hospital readmission rates in dialysis facilities; adjusting for hospital effects. Lifetime Data Anal. 19(4), 490–512 (2013)
He, K., Kalbfleisch, J., Yang, Y., Fei, Z.: Inter-unit reliability for nonlinear models. Stat. Med. 38(5), 844–854 (2019)
Jones, H., Spiegelhalter, D.: The identification of “unusual’’ health-care providers from a hierarchical model. Am. Stat. 65(3), 154–163 (2011)
Kalbfleisch, J., He, K., Xia, L., Li, Y.: Does the inter-unit reliability (IUR) measure reliability? Health Serv. Outcomes Res. Methodol. 18, 215–225 (2018)
Kalbfleisch, J., Wolfe, R.: On monitoring outcomes of medical providers. Stat. Biosci. 5(2), 286–302 (2013)
Liddell, F.: Simple exact analysis of the standardised mortality ratio. J. Epidemiol. Commun. Health 38(1), 85–88 (1984)
Loupy, A., Aubert, O., Reese, P., Bastien, O., Bayer, F., Jacquelinet, C.: Organ procurement and transplantation during the COVID-19 pandemic. Lancet 395(10237), e95–e96 (2020)
Myers, L., Liu, V.: The COVID-19 pandemic strikes again and again and again. JAMA Netw. Open 5(3), e221760 (2022)
National Quality Forum. Scientific methods panel July web meeting summary. Available at: https://www.qualityforum.org/Measuring_Performance/Scientific_Methods_Panel/Meetings/2021_Scientific_Methods_Panel_Meetings.aspx (2021)
Staggs, V., Cramer, E.: Reliability of pressure ulcer rates: How precisely can we differentiate among hospital units, and does the standard signal-noise reliability measure reflect this precision?. Res. Nurs. Health 39(4), 298–305 (2016)
Staggs, V., Gajewski, B.: Bayesian and frequentist approaches to assessing reliability and precision of health-care provider quality measures. Stat. Methods Med. Res. 26(3), 1341–1349 (2017)
Taylor, C., Whitaker, M., Anglin, O., Milucky, J., Patel, K., Pham, H.: COVID-NET Surveillance Team. COVID-19-associated hospitalizations among adults during SARS-CoV-2 Delta and Omicron variant predominance, by race/ethnicity and vaccination status - COVID-NET, 14 states, July 2021–January 2022. MMWR Morb Mortal Wkly Rep. 71(2), 466–473 (2022)
University of Michigan Kidney Epidemiology and Cost Center. Measure information form. Available at: https://dialysisdata.org/sites/default/files/content/ESRD_Measures/PPPW_MIF_MJF.pdf (2017)
University of Michigan Kidney Epidemiology and Cost Center. Standardized hospitalization ratio for dialysis facilities. Available at: https://dialysisdata.org/sites/default/files/content/1463_NQF%20Submission 2020.pdf (2020a)
University of Michigan Kidney Epidemiology and Cost Center. Standardized mortality ratio for dialysis facilities. Available at: https://www.dialysisdata.org/sitesdefault/files/content/0369/NQF%20Submission 2020.pdf
Varewyck, M., Goetghebeur, E., Eriksson, M., Vansteelandt, S.: On shrinkage and model extrapolation in the evaluation of clinical center performance. Biostatistics 15(4), 651–664 (2014)
Funding
This work is partially supported by the Centers for Medicare and Medicaid Services under contract 75FCMC18D0041, task order 75FCMC18F0001 and the National Institute of Diabetes and Digestive and Kidney Diseases under grant R01DK129539. The statements contained in this article are solely those of the authors and do not necessarily reflect the views or policies of the Centers for Medicare and Medicaid Services or the National Institute of Diabetes and Digestive and Kidney Diseases.
Author information
Authors and Affiliations
Contributions
All authors contributed to the conceptualization of this work, reviewed and revised the draft manuscript, and approved of the final version. NH performed the analyses in Sects. 4, 5, and 7. KJP performed the analyses in Sect. 6. NH and KH prepared the text.
Corresponding author
Ethics declarations
Competing Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hartman, N., Shahinian, V.B., Ashby, V.B. et al. Limitations of the inter-unit reliability: a set of practical examples. Health Serv Outcomes Res Method 24, 156–169 (2024). https://doi.org/10.1007/s10742-023-00307-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-023-00307-0