Detecting Hospital Outliers in Post-Pancreatectomy Care Using Funnel Plots from 2009–2018 Based on Nationwide Medico-Administrative Data

Objectives Our objective was to identify hospitals with unusual mortality rates for major pancreatectomies over a period of ten years using 30-day mortality data from the French national database. Methods Data for all patients who underwent pancreatectomy were extracted from the national medico-economic database (Programme de Médicalisation des Systèmes d'Information). To identify quality outliers for each hospital, the observed-to-expected 30-day mortality rates were used as a quality indicator. Results A total of 19 494 patients underwent a major pancreatectomy in France between January 2009 and December 2018. The overall 30-day mortality rate was 4.8% (n = 944). For the 2009–2014 period, the funnel plot showed that 10 of the 176 hospitals lie outside the central 95% region and 7 lie outside the central 99.8% region. For the 2015–2018 period, out of 176 hospitals, 6 lie outside the central 95% region and 2 lie outside the central 99.8% region. The change in standardized mortality ratios between 2009–2014 and 2015–2018 testing for differences from the overall change, they were there 4 hospitals lie outside the central 95% region and 0 lie outside the central 99.8% region. Conclusion Over time, the improvement in hospital quality was weak. This study suggests that there is a pressing need to reorganize the supply of care for pancreatic surgery in France.


Introduction
Pancreatectomy requires expertise both for the procedure and for the management of postoperative complications [1][2][3][4]. For these reasons, many countries have reorganized care by proposing a system of regionalization [5,6]. The idea is to group the appropriate teams on the same site, resulting in an increase in the volume of activity. While numerous studies have shown that hospital volume significantly influences the risk of postoperative death, the volume of activity alone is not enough to measure hospital performance [4][5][6].
In France, despite the availability of medico-administrative data, there is a paucity of literature relative to the surgical performance of hospitals and clinics. Meanwhile, the vast majority of the French population has direct access to health care, which makes it possible to investigate the practice of pancreas surgery through an assessment of quality of care from the national medico-administrative database. To identify hospitals with standard performance, standardized mortality ratios (SMRs) have been used as performance indicators [7][8][9]. As a commonly used metric, the 30-day mortality rate captures most surgery-related deaths for pancreatic surgery [7], therefore representing a legitimate measure of surgical quality. Here, we are interested in identifying quality outlier hospitals using funnel plots [10]. Using a national database, we aimed to analyze the change over time in the performance of hospitals that perform major pancreatectomies.
The objective of this study was to use the national medico-administrative database to determine hospital performance for major pancreatectomies that deviated from the national 30-day mortality rate from as an indicator of performance from 2009-2018.

Data source and study population
Data for all patients who underwent pancreatectomy in France between 2009 and 2018 were extracted from the national medico-administrative database (Programme de Me´dicalisation des Syste`mes d'Information (PMSI)). We used the 10th revision of the International Classification of Diseases (ICD-10) to identify relevant diagnosis codes in the discharge abstracts [11][12][13]. Patients were selected when the primary diagnosis was a malignant tumor (all C25 codes) or benign tumor (D136, D137, K868). The Common Classification of Medical Procedures (CCAM) was used to define relevant interventions: pancreaticoduodenectomy and total pancreatectomy.

Patient characteristics
In addition to collecting data for age and sex, we used the ICD-10 codes to identify comorbidities present at the time of hospitalization: pulmonary disease (chronic bronchitis, emphysema), heart disease (coronary artery disease, arrhythmia, chronic heart failure, valvulopathy, pulmonary embolism), peripheral vascular disease (aneurysm, peripheral vascular disease), neurological disease (stroke, neurological sequelae, dementia), liver disease, renal disease, anemia, infectious disease, hematologic disease and other treatments (neo-adjuvant chemotherapy or corticosteroid therapy). We calculated a modified Charlson Comorbidity Index (CCI) score for each patient [14].
Patient consent was not required. Ethics approval for use of this database was obtained from the French National Commission for Data protection (Commission Nationale de l'Informatique et des Liberte´s: No 1576793), and this study adhered to the tenets of the Declaration of Helsinki.

Hospital characteristics
Hospitals were classified as non-teaching hospitals, private for-profit institutions, private non-profit institutions or teaching hospitals. The annual volume of each establishment was estimated.
Outcome measurements 30-day mortality was defined as any death occurring within 30 days of surgery or during the same hospitalization as pancreatectomy.

Risk-adjustment model
We performed univariate analyses with Chi-squared tests for binary and categorical variables and Student's t tests for continuous variables. Logistic regression models were constructed for the 2009-2014 period using backward stepwise variable selection for comorbidities. We used a bootstrap backward procedure, as recommended by Steyerberg [15], to determine which of these factors were significantly associated with the outcome in logistic regression models. Using this approach, 1000 replicated bootstrap samples were selected from the original data. In each replicated sample, variables such as age, gender and modified Charlson Comorbidity Index score were forced into the model. Risk factors selected in at least 500 samples (50%) of the replicates were included in the model. We then included the variables for hospital characteristics (volume and type of establishment) in the model. The hospital volume was a continuous variable that was transformed into a logarithm to be included in the model.
We did the following to assess the quality of our model [15]: the area under the receiver operating characteristic (ROC) curve was used to measure the discriminatory ability [15] and the Hosmer-Lemeshow goodness-of-fit test to assess the reliability of the model.
For the 2015-2018 period, we updated the model by testing three methods described by Steyerberg [15]: calibration-in-large, recalibration and model revision. quality indicator for hospitals. The O/E ratio is observed mortality divided by the expected mortality rate estimated from a logistic regression model. For each period, we constructed funnel plots to determine outliers for 30-day mortality according to Spiegelhalter's methodology [16] To calculate an overdispersion factor for risk adjusted rates, we used a multiplicative approach with a Winsorized estimator (10th and 90th percentiles) [17]. We used funnel plots to evaluate changes in the standardized mortality ratio (SMR) between 2009-2014 and 2015-2018, and we calculated the SMR1 for 2015-2018 divided by the SMR2 for 2009-2014 [16]. The ratio (SMR2/SMR1) was then transformed into a logarithm. Calculations were performed with STATA 14 statistical software (StataCorp, College Station, Tex) and R statistical software (http://www.r-project.org).

Results
A total of 19 494 patients underwent major pancreatectomy in France between January 1, 2009 and December 31, 2018. The 30-day mortality rate was 4.84% (n = 944). The mortality rate decreased from 2014, as shown in Fig. 1. The creation of two periods going from 2009-2014 and 2015-2018 is justified by the change in the mortality rate ( Fig. 1). Patient characteristics are presented in Table 1. Mean patient age increased significantly during the second period, and comorbidities such as metabolic disease, anemia and infectious disease were also more frequent during the second period ( Table 1). The modified Charlson Comorbidity Index score was higher during the second period. During the two periods, 176 hospitals performed major pancreatectomies. The median number of pancreatic resections per year was 5 for the first period and 7 for the second period (Table 1).

Risk-adjustment models
The model developed during the first period is reported in Table 1. This model had good performance with a C-index of 0.814 ( Table 2). The Hosmer-Lemeshow goodness-offit test was nonsignificant for this model (Chi 2 9.4, p \ 0.3) ( Table 1). For the second period, we were forced to use a revised model to obtain good performance with a C-index of 0.805 ( Table 2). The Hosmer-Lemeshow goodness-offit test was nonsignificant for this model (Chi 2 9.8, p \ 0.28) ( Table 2). It should be noted that the kidney disease variable had more weight in the revised model ( Table 2).

Identification of quality outliers
The funnel plot for the 2009-2014 period is displayed in Fig. 2. In our case, there is no overdispersion because the Winsorized estimator was close to 1. Out of 176 hospitals, 10 were found to lie outside the central 95% region and 7 outside the central 99.8% region (Fig. 2). For the 2015-2018 period, the funnel plot is displayed in Fig. 3. To calculate an overdispersion factor for risk adjusted rates, the Winsorized estimator was 1. Out of 176 hospitals, 6 were found to lie outside the central 95% region and 2 outside the central 99.8% region (Fig. 3). Among the hospitals outliers for the 2015-2018 period, 3 were already outliers for the 2009-2014 period. Figure 4 shows the change in standardized mortality ratios between 2009-2014 and 2015-2018, testing for differences in the overall change (0.024). Out of 176 hospitals, 4 lie outside Fig. 1 Change in the 30-day mortality rate over time the central 95% region and 0 lie outside the central 99.8% region (Fig. 4).

Discussion
While the present study shows that there has been a decrease in mortality following pancreatectomy in recent years, the rate remains higher than what has been reported in the literature. For instance, a meta-analysis conducted in 2015 reported an estimated postoperative mortality rate of 3.1%, with a 95% CI ranging from 2.4 to 3.9% [18].
The graphical analyses using the funnel plot indicate that there has been a slight decrease in outliers over time.
However, the SMR ratio shows that only 3 centers improved mortality outcomes and that one center actually worsened the quality of care. The majority of hospitals have remained close to the national benchmark without any real change in the quality of care. It is difficult to find a single explanation for the lack of improvement in the quality of care. However, the number of centers that perform major pancreatic resections is high in France compared to other countries [4,5], and the number of centers performing pancreatic resection is also high considering the needs of the French population. A study published in 2011 by Finks et al. [4] reported a median annual volume of 16 procedures in the USA, versus only 5-7 for France.
Previous work using the French medico-administrative database has shown that hospitals performing less than 10 pancreatic resections per year have significantly higher postoperative mortality than hospitals that do more than 20 pancreatectomies per year [19]. The authors also found a  [19]. Other studies have confirmed the influence of activity volume on the quality of care provided by hospitals and surgeons [20,21].  Some countries have implemented the centralization of complex surgeries in selected hospitals in an attempt to group together the technical skills needed to improve the quality of care [5,6,22]. This idea seems attractive, especially if we want to be able to measure hospital performance reliably. Compared with other countries, France offers dispersed care for major interventions such as pancreatic surgery, and most private and public hospitals will perform a pancreatic resection at least once a year. This practice could be one of the reasons that the death rate is higher in France than in other countries [3,4,6,22].
There was no overdispersion of our quality indicator, which can be explained by the quality of the model used [17]. The graphical funnel plot method is a relevant tool for the evaluation of the quality of hospitals. It is rather conservative method for the detection of outliers, with a low probability of wrongly classifying a hospital beyond the limits [16]. Hospitals classified as ''worse'' have a standardized mortality rate beyond the upper limit of 99.8% of the funnel plot. They have an excess of mortality compared to the national average. On the other hand, establishments classified as ''better'' have a standardized mortality rate below the limit below 99.8% of the funnel plot. In our work, we are not interested in these establishments. The focus of this study was on teams with excess mortality. This method could easily be used in France for a number of surgical procedures by choosing the most relevant quality  indicator. The funnel plots are flexible and minimize the risk of falsely ranking a hospital as an outlier. The limitations of the study are primarily related to the quality of the data used to develop the risk-adjusted model. The quality of hospital data has improved in France over the last few years, but there is still considerable potential for miscoding. Coding practices vary greatly among institutions, resulting in higher rates of recorded comorbidities in certain hospitals and sub-coding in others. However, the data are representative of French practice. While the results of the present study are based on French data, some of the conclusions may nonetheless be generalizable to other settings.

Conclusion
Pancreatic surgery is practiced by many centers, and as a result some institutions operate on as little as one patient per year. Over time, the observed improvement in hospital quality was weak. The results of this study suggest that there is a pressing need to reorganize the supply of care for pancreatic surgery in France.
Author contributions Alain Bernard was involved in conceptualization and writing-original draft preparation; Jonathan Cottenet and Anne-Sophie Mariet helped in formal analysis and methodology; Catherine Quantin contributed to validation; Serge Aho, Alexandre Doussot and Olivier Facy were involved in writing-review & editing.
Funding The author(s) received no specific funding for this work.

Declarations
Conflict of interest The authors have declared that no competing interests exist.
Consent for publication Patient consent for publication: Not required.
Ethical approval All the data were completely anonymous. Patient consent was not required. Ethics approval for use of this database was obtained from the National Commission for Data protection (CNIL No 1576793), and this study adhered to the tenets of the Declaration of Helsinki.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.