Skip to main content
SpringerLink
Log in

Test-specific funnel plots for healthcare provider profiling leveraging individual- and summary-level information

  • Published:
Health Services and Outcomes Research Methodology Aims and scope Submit manuscript

Abstract

In addition to applications in meta-analysis, funnel plots have emerged as an effective graphical tool for visualizing the detection of health care providers with unusual performance. Although there already exist a variety of approaches to producing funnel plots in the literature of provider profiling, limited attention has been paid to elucidating the critical relationship between funnel plots and hypothesis testing. Within the framework of generalized linear models, here we establish methodological guidelines for creating funnel plots specific to the statistical tests of interest. Moreover, we show that the test-specific funnel plots can be created merely leveraging summary statistics instead of individual-level information. This appealing feature inhibits the leak of protected health information and reduces the cost of inter-institutional data transmission. Two data examples, one for surgical patients from Michigan hospitals and the other for Medicare-certified dialysis facilities, demonstrate the applicability to different types of providers and outcomes with either individual- or summary-level information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

Data for surgical patients contain protected health information and are not available for sharing under data use agreement with the Michigan Surgical Quality Collaborative. The Medicare Dialysis Facilities data are available online at https://data.cms.gov/quality-of-care/medicare-dialysis-facilities.

References

  • Abramo, G., D’Angelo, A.C., Grilli, L.: From rankings to funnel plots: the question of accounting for uncertainty when assessing university research performance. J. Informet. 10(3), 854–862 (2016)

    Article  Google Scholar 

  • Begg, C.B., Berlin, J.A.: Publication bias: a problem in interpreting medical data. J. R. Stat. Soc. A. Stat. Soc. 151(3), 419–445 (1988)

    Article  Google Scholar 

  • Berlin, J.A., Begg, C.B., Louis, T.A.: An assessment of publication bias using a sample of published clinical trials. J. Am. Stat. Assoc. 84(406), 381–392 (1989)

    Article  Google Scholar 

  • Celentano, D.D., Szklo, M.: Gordis Epidemiology. Elsevier, Amsterdam (2019)

    Google Scholar 

  • Centers for Medicare & Medicaid Services.: End-stage renal disease quality incentive program (ESRD QIP) Calendar Year (CY) 2021 Measure Technical Specifications (2020). https://www.cms.gov/files/document/cy-2021-final-technical-specifications-20201130.pdf. Accessed 15 June 2021

  • Centers for Medicare & Medicaid Services.: Medicare Dialysis Facilities Data (2021). https://data.cms.gov/quality-of-care/medicare-dialysis-facilities. Accessed 07 Dec 2021

  • Chen, S.X., Liu, J.S.: Statistical applications of the Poisson-binomial and conditional Bernoulli distributions. Stat. Sin. 7(2), 875–892 (1997)

    Google Scholar 

  • Chen, Y., Şentürk, D., Estes, J.P., et al.: Performance characteristics of profiling methods and the impact of inadequate case-mix adjustment. Commun. Stat.-Simul. Comput. 50(6), 1854–1871 (2021)

    Article  Google Scholar 

  • DerSimonian, R., Laird, N.: Meta-analysis in clinical trials. Control. Clin. Trials 7(3), 177–188 (1986)

    Article  CAS  PubMed  Google Scholar 

  • Efron, B.: Large-scale simultaneous hypothesis testing: the choice of a null hypothesis. J. Am. Stat. Assoc. 99(465), 96–104 (2004)

    Article  Google Scholar 

  • Efron, B.: Size, power and false discovery rates. Ann. Stat. 35(4), 1351–1377 (2007)

    Article  Google Scholar 

  • Egger, M., Smith, G.D., Schneider, M., et al.: Bias in meta-analysis detected by a simple, graphical test. BMJ 315(7109), 629–634 (1997)

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Estes, J.P., Nguyen, D.V., Chen, Y., et al.: Time-dynamic profiling with application to hospital readmission among patients on dialysis. Biometrics 74(4), 1383–1394 (2018)

    Article  PubMed  PubMed Central  Google Scholar 

  • Estes, J.P., Chen, Y., Şentürk, D., et al.: Profiling dialysis facilities for adverse recurrent events. Stat. Med. 39(9), 1374–1389 (2020)

    Article  PubMed  PubMed Central  Google Scholar 

  • Gallant, A.R.: Nonlinear Statistical Models, vol. 310. Wiley, Hoboken (1987)

    Book  Google Scholar 

  • Goldstein, H., Spiegelhalter, D.J.: League tables and their limitations: statistical issues in comparisons of institutional performance. J. R. Stat. Soc. A. Stat. Soc. 159(3), 385–409 (1996)

    Article  Google Scholar 

  • Griffen, D., Callahan, C.D., Markwell, S., et al.: Application of statistical process control to physician-specific emergency department patient satisfaction scores: a novel use of the funnel plot. Acad. Emerg. Med. 19(3), 348–355 (2012)

    Article  PubMed  Google Scholar 

  • He, K., Kalbfleisch, J.D., Li, Y., et al.: Evaluating hospital readmission rates in dialysis facilities; adjusting for hospital effects. Lifetime Data Anal. 19(4), 490–512 (2013)

    Article  PubMed  Google Scholar 

  • Hong, Y.: On computing the distribution function for the Poisson binomial distribution. Comput. Stat. Data Anal. 59, 41–51 (2013)

    Article  Google Scholar 

  • Hong, Y.: poibin: the Poisson binomial distribution. https://cran.r-project.org/package=poibin, R package version 1.5 (2020)

  • Ieva, F., Paganoni, A.M.: Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag. Sci. 18(2), 166–172 (2015)

    Article  PubMed  Google Scholar 

  • Inskip, H., Beral, V., Fraser, P., et al.: Methods for age-adjustment of rates. Stat. Med. 2(4), 455–466 (1983)

    Article  CAS  PubMed  Google Scholar 

  • Jin, J., Cai, T.T.: Estimating the null and the proportion of nonnull effects in large-scale multiple comparisons. J. Am. Stat. Assoc. 102(478), 495–506 (2007)

    Article  CAS  Google Scholar 

  • Johnson, N.L., Kemp, A.W., Kotz, S.: Univariate Discrete Distributions, 3rd edn. Wiley, Hoboken (2005)

    Book  Google Scholar 

  • Kalbfleisch, J.D., He, K.: Discussion on “Time-dynamic profiling with application to hospital readmission among patients on dialysis,” by Jason P. Estes, Danh V. Nguyen, Yanjun Chen, Lorien S. Dalrymple, Connie M. Rhee, Kamyar Kalantar-Zadeh, and Damla Senturk. Biometrics 74(4):1401–1403 (2018)

  • Kalbfleisch, J.D., Wolfe, R.A.: On monitoring outcomes of medical providers. Stat. Biosci. 5(2), 286–302 (2013)

    Article  Google Scholar 

  • Kalbfleisch, J.D., He, K., Xia, L., et al.: Does the inter-unit reliability (iur) measure reliability? Health Serv. Outcomes Res. Method. 18(3), 215–225 (2018)

    Article  Google Scholar 

  • Kidney Epidemiology and Cost Center.: Report for the Standardized Hospitalization Ratio (2016). https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/ESRDQIP/Downloads/SHR-Methodology-Report.pdf. Accessed 09 Dec 2021

  • Kidney Epidemiology and Cost Center (2021) Guide to the Dialysis Facility Reports for Fiscal Year 2022: Overview, Methodology, and Interpretation. https://data.cms.gov/sites/default/files/2022-02/d09a97a0-8004-4264-97ab-441c8703caef/FY2022_DFR_Guide.pdf, accessed: 2022-03-20

  • Lee, M., Feuer, E.J., Fine, J.P.: On the analysis of discrete time competing risks data. Biometrics 74(4), 1468–1481 (2018)

    Article  PubMed  Google Scholar 

  • Light, R.J., Pillemer, D.B.: Summing Up: the Science of Reviewing Research. Harvard University Press, Cambridge (1984)

    Book  Google Scholar 

  • Manktelow, B.N., Seaton, S.E.: Specifying the probability characteristics of funnel plot control limits: an investigation of three approaches. PLoS ONE 7(9), e45, 723 (2012)

    Article  CAS  Google Scholar 

  • Manktelow, B.N., Seaton, S.E., Evans, T.A.: Funnel plot control limits to identify poorly performing healthcare providers when there is uncertainty in the value of the benchmark. Stat. Methods Med. Res. 25(6), 2670–2684 (2016)

    Article  PubMed  Google Scholar 

  • Mohammed, M.A., Deeks, J.J.: In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann. Thorac. Surg. 86(1), 348 (2008)

    Article  PubMed  Google Scholar 

  • National Institutes of Health.: NIH Genomic Data Sharing Policy (2014). https://grants.nih.gov/grants/guide/notice-files/not-od-14-124.html. Accessed: 07 Dec 2021

  • R Core Team.: R: a language and environment for statistical computing (2021). https://www.R-project.org

  • Seaton, S.E., Manktelow, B.N.: The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio. BMC Med. Res. Methodol. 12(98), 1–8 (2012)

    Google Scholar 

  • Seaton, S.E., Barker, L., Lingsma, H.F., et al.: What is the probability of detecting poorly performing hospitals using funnel plots? BMJ Qual. Saf. 22(10), 870–876 (2013)

    Article  PubMed  Google Scholar 

  • Spiegelhalter, D.J.: Funnel plots for comparing institutional performance. Stat. Med. 24(8), 1185–1202 (2005)

    Article  PubMed  Google Scholar 

  • Sterne, J.A., Becker, B.J., Egger, M.: The Funnel Plot. In: Rothstein, H.R., Sutton, A.J., Borenstein, M. (eds.) Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, chap 5, pp. 75–98. Wiley, Hoboken (2005)

    Google Scholar 

  • Sutton, A.J., Duval, S.J., Tweedie, R., et al.: Empirical assessment of effect of publication bias on meta-analyses. BMJ 320(7249), 1574–1577 (2000)

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Sutzko, D.C., Obi, A.T., Kamdar, N., et al.: Low to moderate risk non-orthopedic surgical patients do not benefit from VTE chemoprophylaxis. Ann. Surg. (2020). https://doi.org/10.1097/SLA.0000000000004646

    Article  PubMed  Google Scholar 

  • U.S. Department of Health & Human Services.: Health Information Privacy (1996). https://www.hhs.gov/hipaa/index.html. Accessed 07 Dec 2021

  • Verburg, I.W., Holman, R., Peek, N., et al.: Guidelines on constructing funnel plots for quality indicators: a case study on mortality in intensive care unit patients. Stat. Methods Med. Res. 27(11), 3350–3366 (2018)

    Article  PubMed  Google Scholar 

  • Wu, W., He, K.: ppfunnel: Create elegant funnel plots for health care provider profiling (2021). https://github.com/UM-KevinHe/ppfunnel. Accessed 09 Dec 2021

  • Wu, W., Yang, Y., Kang, J., et al.: Improving large-scale estimation and inference for profiling health care providers. Stat. Med. 41(15), 2840–2853 (2022)

    Article  PubMed  PubMed Central  Google Scholar 

  • Xia, L., He, K., Li, Y., et al.: Accounting for total variation and robustness in profiling health care providers. Biostatistics 23(1), 257–273 (2022)

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Dr. Kirsten Herold (University of Michigan) for helpful discussion and comments on the manuscript.

Funding

This work is partially supported by the National Institute of Diabetes and Digestive and Kidney Diseases (R01-DK-129539), the University of Michigan Institute for Computational Discovery & Engineering, and Blue Cross Blue Shield of Michigan.

Author information

Authors and Affiliations

  1. Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI, 48109, USA

    Wenbo Wu & Kevin He

  2. Robert Wood Johnson Medical School, Rutgers University, 125 Paterson Street, New Brunswick, NJ, 08901, USA

    Jonathan P. Kuriakose

  3. Michigan Surgical Quality Collaborative, University of Michigan, 2800 Plymouth Road, Ann Arbor, MI, 48109, USA

    Wenjing Weng & Richard E. Burney

  4. Department of Surgery, University of Michigan, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA

    Richard E. Burney

Authors
  1. Wenbo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan P. Kuriakose
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjing Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard E. Burney
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kevin He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kevin He.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 82 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, W., Kuriakose, J.P., Weng, W. et al. Test-specific funnel plots for healthcare provider profiling leveraging individual- and summary-level information. Health Serv Outcomes Res Method 23, 45–58 (2023). https://doi.org/10.1007/s10742-022-00285-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-022-00285-9

Keywords

Search

Navigation