Abstract
Objectives
Our objective was to identify hospitals with unusual mortality rates for major pancreatectomies over a period of ten years using 30-day mortality data from the French national database.
Methods
Data for all patients who underwent pancreatectomy were extracted from the national medico-economic database (Programme de Médicalisation des Systèmes d'Information). To identify quality outliers for each hospital, the observed-to-expected 30-day mortality rates were used as a quality indicator.
Results
A total of 19 494 patients underwent a major pancreatectomy in France between January 2009 and December 2018. The overall 30-day mortality rate was 4.8% (n = 944). For the 2009–2014 period, the funnel plot showed that 10 of the 176 hospitals lie outside the central 95% region and 7 lie outside the central 99.8% region. For the 2015–2018 period, out of 176 hospitals, 6 lie outside the central 95% region and 2 lie outside the central 99.8% region. The change in standardized mortality ratios between 2009–2014 and 2015–2018 testing for differences from the overall change, they were there 4 hospitals lie outside the central 95% region and 0 lie outside the central 99.8% region.
Conclusion
Over time, the improvement in hospital quality was weak. This study suggests that there is a pressing need to reorganize the supply of care for pancreatic surgery in France.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Pancreatectomy requires expertise both for the procedure and for the management of postoperative complications [1,2,3,4]. For these reasons, many countries have reorganized care by proposing a system of regionalization [5, 6]. The idea is to group the appropriate teams on the same site, resulting in an increase in the volume of activity. While numerous studies have shown that hospital volume significantly influences the risk of postoperative death, the volume of activity alone is not enough to measure hospital performance [4,5,6].
In France, despite the availability of medico-administrative data, there is a paucity of literature relative to the surgical performance of hospitals and clinics. Meanwhile, the vast majority of the French population has direct access to health care, which makes it possible to investigate the practice of pancreas surgery through an assessment of quality of care from the national medico-administrative database. To identify hospitals with standard performance, standardized mortality ratios (SMRs) have been used as performance indicators [7,8,9]. As a commonly used metric, the 30-day mortality rate captures most surgery-related deaths for pancreatic surgery [7], therefore representing a legitimate measure of surgical quality. Here, we are interested in identifying quality outlier hospitals using funnel plots [10]. Using a national database, we aimed to analyze the change over time in the performance of hospitals that perform major pancreatectomies.
The objective of this study was to use the national medico-administrative database to determine hospital performance for major pancreatectomies that deviated from the national 30-day mortality rate from as an indicator of performance from 2009–2018.
Material and methods
Data source and study population
Data for all patients who underwent pancreatectomy in France between 2009 and 2018 were extracted from the national medico-administrative database (Programme de Médicalisation des Systèmes d'Information (PMSI)). We used the 10th revision of the International Classification of Diseases (ICD-10) to identify relevant diagnosis codes in the discharge abstracts [11,12,13]. Patients were selected when the primary diagnosis was a malignant tumor (all C25 codes) or benign tumor (D136, D137, K868). The Common Classification of Medical Procedures (CCAM) was used to define relevant interventions: pancreaticoduodenectomy and total pancreatectomy.
Patient characteristics
In addition to collecting data for age and sex, we used the ICD-10 codes to identify comorbidities present at the time of hospitalization: pulmonary disease (chronic bronchitis, emphysema), heart disease (coronary artery disease, arrhythmia, chronic heart failure, valvulopathy, pulmonary embolism), peripheral vascular disease (aneurysm, peripheral vascular disease), neurological disease (stroke, neurological sequelae, dementia), liver disease, renal disease, anemia, infectious disease, hematologic disease and other treatments (neo-adjuvant chemotherapy or corticosteroid therapy). We calculated a modified Charlson Comorbidity Index (CCI) score for each patient [14].
Patient consent was not required. Ethics approval for use of this database was obtained from the French National Commission for Data protection (Commission Nationale de l’Informatique et des Libertés: No 1576793), and this study adhered to the tenets of the Declaration of Helsinki.
Hospital characteristics
Hospitals were classified as non-teaching hospitals, private for-profit institutions, private non-profit institutions or teaching hospitals. The annual volume of each establishment was estimated.
Outcome measurements
30-day mortality was defined as any death occurring within 30 days of surgery or during the same hospitalization as pancreatectomy.
Statistical methods
Risk-adjustment model
We performed univariate analyses with Chi-squared tests for binary and categorical variables and Student’s t tests for continuous variables. Logistic regression models were constructed for the 2009–2014 period using backward stepwise variable selection for comorbidities. We used a bootstrap backward procedure, as recommended by Steyerberg [15], to determine which of these factors were significantly associated with the outcome in logistic regression models. Using this approach, 1000 replicated bootstrap samples were selected from the original data. In each replicated sample, variables such as age, gender and modified Charlson Comorbidity Index score were forced into the model. Risk factors selected in at least 500 samples (50%) of the replicates were included in the model. We then included the variables for hospital characteristics (volume and type of establishment) in the model. The hospital volume was a continuous variable that was transformed into a logarithm to be included in the model.
We did the following to assess the quality of our model [15]: the area under the receiver operating characteristic (ROC) curve was used to measure the discriminatory ability [15] and the Hosmer–Lemeshow goodness-of-fit test to assess the reliability of the model.
For the 2015–2018 period, we updated the model by testing three methods described by Steyerberg [15]: calibration-in-large, recalibration and model revision.
Identification of quality outliers
For the analysis of hospital outliers over time, we grouped the years together as follows: 2009–2014 and 2015–2018. Observed-to-expected rates (O/E ratio) can be used as a quality indicator for hospitals. The O/E ratio is observed mortality divided by the expected mortality rate estimated from a logistic regression model. For each period, we constructed funnel plots to determine outliers for 30-day mortality according to Spiegelhalter’s methodology [16] To calculate an overdispersion factor for risk adjusted rates, we used a multiplicative approach with a Winsorized estimator (10th and 90th percentiles) [17]. We used funnel plots to evaluate changes in the standardized mortality ratio (SMR) between 2009–2014 and 2015–2018, and we calculated the SMR1 for 2015–2018 divided by the SMR2 for 2009–2014 [16]. The ratio (SMR2/SMR1) was then transformed into a logarithm.
Calculations were performed with STATA 14 statistical software (StataCorp, College Station, Tex) and R statistical software (http://www.r-project.org).
Results
A total of 19 494 patients underwent major pancreatectomy in France between January 1, 2009 and December 31, 2018. The 30-day mortality rate was 4.84% (n = 944). The mortality rate decreased from 2014, as shown in Fig. 1. The creation of two periods going from 2009–2014 and 2015–2018 is justified by the change in the mortality rate (Fig. 1). Patient characteristics are presented in Table 1. Mean patient age increased significantly during the second period, and comorbidities such as metabolic disease, anemia and infectious disease were also more frequent during the second period (Table 1). The modified Charlson Comorbidity Index score was higher during the second period. During the two periods, 176 hospitals performed major pancreatectomies. The median number of pancreatic resections per year was 5 for the first period and 7 for the second period (Table 1).
Risk-adjustment models
The model developed during the first period is reported in Table 1. This model had good performance with a C-index of 0.814 (Table 2). The Hosmer–Lemeshow goodness-of-fit test was nonsignificant for this model (Chi2 9.4, p < 0.3) (Table 1). For the second period, we were forced to use a revised model to obtain good performance with a C-index of 0.805 (Table 2). The Hosmer–Lemeshow goodness-of-fit test was nonsignificant for this model (Chi2 9.8, p < 0.28) (Table 2). It should be noted that the kidney disease variable had more weight in the revised model (Table 2).
Identification of quality outliers
The funnel plot for the 2009–2014 period is displayed in Fig. 2. In our case, there is no overdispersion because the Winsorized estimator was close to 1. Out of 176 hospitals, 10 were found to lie outside the central 95% region and 7 outside the central 99.8% region (Fig. 2). For the 2015–2018 period, the funnel plot is displayed in Fig. 3. To calculate an overdispersion factor for risk adjusted rates, the Winsorized estimator was 1. Out of 176 hospitals, 6 were found to lie outside the central 95% region and 2 outside the central 99.8% region (Fig. 3). Among the hospitals outliers for the 2015–2018 period, 3 were already outliers for the 2009–2014 period. Figure 4 shows the change in standardized mortality ratios between 2009–2014 and 2015–2018, testing for differences in the overall change (0.024). Out of 176 hospitals, 4 lie outside the central 95% region and 0 lie outside the central 99.8% region (Fig. 4).
Discussion
While the present study shows that there has been a decrease in mortality following pancreatectomy in recent years, the rate remains higher than what has been reported in the literature. For instance, a meta-analysis conducted in 2015 reported an estimated postoperative mortality rate of 3.1%, with a 95% CI ranging from 2.4 to 3.9% [18].
The graphical analyses using the funnel plot indicate that there has been a slight decrease in outliers over time. However, the SMR ratio shows that only 3 centers improved mortality outcomes and that one center actually worsened the quality of care. The majority of hospitals have remained close to the national benchmark without any real change in the quality of care. It is difficult to find a single explanation for the lack of improvement in the quality of care. However, the number of centers that perform major pancreatic resections is high in France compared to other countries [4, 5], and the number of centers performing pancreatic resection is also high considering the needs of the French population. A study published in 2011 by Finks et al. [4] reported a median annual volume of 16 procedures in the USA, versus only 5–7 for France.
Previous work using the French medico-administrative database has shown that hospitals performing less than 10 pancreatic resections per year have significantly higher postoperative mortality than hospitals that do more than 20 pancreatectomies per year [19]. The authors also found a linear decrease in postoperative mortality depending on volume (9.1%, 8.1% and 5.3% in low, intermediate and high volume centers, respectively) [19]. Other studies have confirmed the influence of activity volume on the quality of care provided by hospitals and surgeons [20, 21].
Some countries have implemented the centralization of complex surgeries in selected hospitals in an attempt to group together the technical skills needed to improve the quality of care [5, 6, 22]. This idea seems attractive, especially if we want to be able to measure hospital performance reliably. Compared with other countries, France offers dispersed care for major interventions such as pancreatic surgery, and most private and public hospitals will perform a pancreatic resection at least once a year. This practice could be one of the reasons that the death rate is higher in France than in other countries [3, 4, 6, 22].
There was no overdispersion of our quality indicator, which can be explained by the quality of the model used [17]. The graphical funnel plot method is a relevant tool for the evaluation of the quality of hospitals. It is rather conservative method for the detection of outliers, with a low probability of wrongly classifying a hospital beyond the limits [16]. Hospitals classified as “worse” have a standardized mortality rate beyond the upper limit of 99.8% of the funnel plot. They have an excess of mortality compared to the national average. On the other hand, establishments classified as “better” have a standardized mortality rate below the limit below 99.8% of the funnel plot. In our work, we are not interested in these establishments. The focus of this study was on teams with excess mortality. This method could easily be used in France for a number of surgical procedures by choosing the most relevant quality indicator. The funnel plots are flexible and minimize the risk of falsely ranking a hospital as an outlier.
The limitations of the study are primarily related to the quality of the data used to develop the risk-adjusted model. The quality of hospital data has improved in France over the last few years, but there is still considerable potential for miscoding. Coding practices vary greatly among institutions, resulting in higher rates of recorded comorbidities in certain hospitals and sub-coding in others. However, the data are representative of French practice. While the results of the present study are based on French data, some of the conclusions may nonetheless be generalizable to other settings.
Conclusion
Pancreatic surgery is practiced by many centers, and as a result some institutions operate on as little as one patient per year. Over time, the observed improvement in hospital quality was weak. The results of this study suggest that there is a pressing need to reorganize the supply of care for pancreatic surgery in France.
References
Cardini B, Primavesi F, Maglione M et al (2019) Outcomes following pancreatic resections- result and challenges of an Austrian university hospital compared to nationwide data and international centres. Eur Surg 51:81–89
Soreide JA, Sandvik OM, Soreide K (2016) Improving pancreas surgery over time: performance factors related to transition of care and patient volume. Int J Surg 32:116–122
Bengt L, van Rijssen BG, Koerkamp MJZ et al (2017) Nationwide prospective audit of pancreatic surgery: design, accuracy and outcomes of the dutch pancreatic cancer audit. HPB 19:919–926
Finks JF, Osborne NH, Birkmeyer JD (2011) Trends in hospital volume and operative mortality for high-risk surgery. N Engl J Med 364:2128–2137
Krautz C, Nimptsch U, Weber GF et al (2017) Effect of hospital volume on in-hospital morbidity and mortality following pancreatic surgery in Germany. Ann Surg 267:411–417
Gooiker GA, van der Geest LGM, Wouters MWJM et al (2011) Quality improvement of pancreatic surgery by centralization in the western part of the Netherlands. Ann Surg Oncol 18:1821–1829
Renzi C, Sorge C, Fusco D et al (2012) Reporting of quality indicators and improvement in hospital performance: the P.RE Val.E. regional outcome evaluation program. HSR 47:1880–1901
Glance LG, Li Y, Dick AW (2016) Quality of quality measurement. Impact of risk adjustment, hospital volume and hospital performance. Anesthesiology 125:1092–1102
Normand SLT, Shahian DM (2007) Statistical and clinical aspects of hospitals outcomes profiling. Stat Sci 22:206–226
Spiegelhalter D, Sherlaw-Johnson C, Bardsley M, Blunt I (2012) Statistical methods for healthcare regulation: rating, screening and surveillance. J R Statlist Soc A 175:1–47
World Health Organization. International statistical classification of diseases and related health problems 10th revision. http://apps.who.int/classifications/icd10/browse/2016/en Date last accessed: March 1, 2016. Date last updated: 2016
Iezzoni LI (1997) Assessing quality using administrative data. Ann Intern Med 127(8):666–674
Pages PB, Cottenet J, Mariet AS et al (2016) In-hospital mortality following lung cancer resection: nationwide administrative database. Eur Respir J 47:1809–1817
Charlson M, Szatrowski TP, Peterson J, Gold J (1994) Validation of a combined comorbidity index. J Clin Epidemiol 47:1245–1251
Steyerberg EW (2009) Clinical prediction models. A practical approach to development, validating and updating. Springer-Verlag, New York
Spiegelhalter DJ (2005) Funnel plots for comparing institutional performance. Statist Med 24:1185–1202
Verburg IWM, Holman R, Peek N, Abu-Hanna A, de Keizer NF (2018) Guidelines on constructing funnel plots for quality indicators: a case study on mortality in intensive care unit patients. Stat Methods in Med Res 27:3350–3366
Gurusamy K, Toon C, Virendrakumar B, Morris S, Davidson B (2015) Feasibility of comparing the results of pancreatic resections between surgeons: a systematic review and meta-analysis of pancreatic resections. HPB Surg. https://doi.org/10.1155/2015/896875
El Amrani M, Clement G, Lenne X et al (2012) Failure-to-rescue in patients undergoing pancreatectomy. Is hospital volume a standard for quality improvement programs? Nationwide analysis of 12333 patients. Ann Surg 268:799–807
Mehta HB, Parmar AD, Adhikari D et al (2016) Relative impact of surgeon and hospital volume on operative mortality and complications following pancreatic resection in medicare patients. J Surg Res 204:326–334
Gooiker GA, Van Gijn W, Wouters MWJM et al (2011) Systematic review and meta-analysis of the volume-outcome relationship in pancreatic surgery. Br J Surg 98:485–494
Santry H, Kao LS, Shafi S, Lottenberg L, Crandall M (2019) Pro-con debate on regionalization of emergency general surgery: controversy or common sense? Trauma Surg Acute Care Open 4:e000319. https://doi.org/10.1136/tsaco-2019-000319
Acknowledgements
The authors would like to thank Suzanne Rankin (Dijon University Hospital) for revising the manuscript. This work was supported by the Multi-organization Cancer Institute (Institut thématique multi-organismes cancer: ITMO Cancer), the Public Health Research Institute (Institut de Recherche en Santé Publique: IRESP) and the French National Institute of Health and Medical Research (Institut national de la santé et de la recherche médicale: INSERM) in the context of the 2014–2019 cancer plan.
Funding
The author(s) received no specific funding for this work.
Author information
Authors and Affiliations
Contributions
Alain Bernard was involved in conceptualization and writing—original draft preparation; Jonathan Cottenet and Anne-Sophie Mariet helped in formal analysis and methodology; Catherine Quantin contributed to validation; Serge Aho, Alexandre Doussot and Olivier Facy were involved in writing—review & editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors have declared that no competing interests exist.
Consent for publication
Patient consent for publication: Not required.
Ethical approval
All the data were completely anonymous. Patient consent was not required. Ethics approval for use of this database was obtained from the National Commission for Data protection (CNIL No 1576793), and this study adhered to the tenets of the Declaration of Helsinki.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Bernard, A., Cottenet, J., Aho, S. et al. Detecting Hospital Outliers in Post-Pancreatectomy Care Using Funnel Plots from 2009–2018 Based on Nationwide Medico-Administrative Data. World J Surg 45, 2210–2217 (2021). https://doi.org/10.1007/s00268-021-06078-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00268-021-06078-4