Introduction

The case-mix adjusted mortality rate is considered essential for accurate evaluation of hospital-level outcomes [1–7]. Numerous clinical registries have incorporated risk-adjusted mortality measurements to enhance internal and external benchmarking and as drivers for quality improvement among participating institutions [8–12]. Researchers have applied risk-adjustment tools to facilitate observational research [1, 2, 13], and physicians can utilize risk-adjustment tools to anticipate the mortality risk for patients [14, 15]. In pediatric extracorporeal membrane oxygenation (ECMO) for respiratory support such a risk-adjustment tool would be of great utility in clinical and analytic applications. However, no such risk-adjustment tool exists [16].

In this study we use data from the Extracorporeal Life Support Organization (ELSO) registry, an international registry of 298 centers, to develop and internally validate the Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Ped-RESCUERS) tool. Ped-RESCUERS is designed to estimate the pre-ECMO risk of in-hospital death for children receiving respiratory ECMO support.

Materials and methods

This study was designed in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement [17]. As a retrospective analysis of deidentified data, formal consent was not required, and it was determined to be exempt from human subjects review by the Institutional Review Board of the University of Michigan Medical School.

Patient selection

We queried the ELSO registry for all pediatric patients aged 29 days to less than 18 years who received ECMO support for respiratory failure from 2009 to 2014. A priori, we chose 2009 as the start date for data accrual because advances in ECMO technology from that year onward have been reported to enhance delivery of ECMO support [18, 19]. If a patient was placed on ECMO more than once, we only considered the first ECMO run. The development dataset incorporated data among children who received pediatric respiratory ECMO between 1 January 2009 and 31 December 2012. A validation dataset was created from children who received respiratory ECMO support between 1 January 2013 and 31 December 2014. We limited our validation dataset to those with complete data for selected variables (Fig. 1).

Fig. 1
figure 1

Flow diagram for development and internal validation datasets. ECMO extracorporeal membrane oxygenation. In the developmental dataset, all children were analyzed because missing data was imputed

Full size image

Multiple imputations

We derived Ped-RESCUERS from a Bayesian logistic regression model predicting mortality. Our development dataset included variables with missing data (Online Resource 1, Table e1). We addressed missing data through multiple imputation with iterative chained equations [20–22] because logistic regression models that limit analysis to patients with complete data can lead to biased results [17, 23]. Multiple imputation uses the partial information available in the observation and data contained in other observations in the dataset to predict an observation’s missing data (additional details in Online Resource 1, Supplemental Methods) [20]. Importantly, the outcome variable of death prior to hospital discharge contained no missing observations.

Candidate variables

We adapted primary diagnostic fields from a previous publication of pediatric respiratory ECMO mortality risk factors [24] (Table 1) and defined diagnostic groups using International Classification of Disease-9-Clinical Modification (ICD-9-CM) diagnostic codes. Primary diagnoses were not considered as one variable with 15 categories, but rather as 15 present or absent dummy variables. This allowed the contribution of each diagnosis to the model to be considered individually.

Table 1 Pre-ECMO characteristics of patients in the development (2009–2012) dataset
Full size table

We recoded all other categorical variables into dummy variables. This yielded 53 candidate variables, listed in Tables 1 and 2. In addition to primary diagnoses, candidate variables included clinical data collected at most 6 h prior to ECMO, such as physiologic (the worst pre-ECMO blood gas and lowest systolic blood pressure) and therapeutic (ventilator settings and number of days of mechanical ventilation prior to ECMO) data. Other variables included the presence of pre-ECMO cardiac arrest, pre-ECMO renal failure, and any clinical comorbidities as defined by Feudtner et al. [25] (additional details in Online Resource 1, Supplemental Methods and Table e2). A priori we identified two types of interactions to consider: the interaction between ventilator settings and ventilator type and an interaction between blood pressure and age.

Table 2 Blood gas and supportive therapies prior to extracorporeal membrane oxygenation (ECMO) support in the development (2009–2012) dataset
Full size table

Model fitting

It can be difficult to compare the effect sizes (often presented as odds ratios or beta-coefficients) across dichotomous variables and continuous variables. We address this challenge by transforming all variables to a common standard through location-scale transforming before model fitting [26]. First, we center all variables at the observed mean and then divide by twice the standard deviation of the observed data. Next, we fit a multiple logistic regression model including the candidate variables. Each individual variable’s beta-coefficient (β k) is interpreted as the log odds ratio for mortality for each two-standard deviation change in measurement. The standardization allows for direct comparison of odds ratios between variables but does not necessarily translate into clinically relevant quantities. For a dichotomous variable with 50 % prevalence (and therefore twice the standard deviation equal to 1), such as gender, β k is the difference in log odds between female and male. For dichotomous variables with prevalence not equal to 50 %, β k is no longer the difference in log odds between the two values, since two standard deviations is less than 1 (additional details in Online Resource 1, Supplemental Methods).

Development of Ped-RESCUERS

We fit a single Bayesian multivariate logistic regression with outcome of mortality and 53 candidate variables. A variable was selected if there was a 75 % probability that a two-standard deviation variable increase changed a child’s likelihood of mortality by at least 2 %, from a baseline of 40 %. Statistically, this translates to having a β k > 0.08 (mortality risk factor) or β k < −0.08 (protective factor) since logit−1[0.08] = 0.42 and logit−1[−0.08] = 0.38 (additional details in Online Resource 1, Supplemental Methods).

Discrimination, calibration, and validation of Ped-RESCUERS

For development and internal validation we assessed the model discrimination using the area under curve (AUC) of the receiver operating characteristic curves, and we characterized the calibration using the Brier score [27, 28]. Validation was against those with complete data.

We conducted a sensitivity analysis in which we re-created the Ped-RESCUERS score after marking data for 64 patients with potentially incorrect data as missing and then imputing the data. Potentially incorrect data included patients with implausible ventilator settings such as a positive-end expiratory pressure on a high frequency oscillatory ventilator (HFOV) or extremely abnormal blood gas values such as an arterial partial pressure of oxygen (PaO2) less than 10 mmHg (additional details in Online Resource 1, Supplemental Methods).

Results

During 2009–2014, 2458 children aged 29 days to less than 18 years received ECMO for respiratory support, with an overall mortality of 39.8 %. The duration of ECMO was a median of 188 h (7.8 days) with an interquartile range of 104–356 h. Prior to ECMO, 49 % of children were receiving support via an HFOV and 47 % were receiving inhaled nitric oxide. The median oxygenation index (mean airway pressure × fraction of inspired oxygen × 100/PaO2) was 44, with an interquartile range of 30–60. Two-thirds of subjects were neuromuscularly blocked and 61 % received a vasoactive infusion.

The median duration of time between admission to an ECMO center and the subsequent start of ECMO support was 17 h less in those who survived compared to those who died (Table 1). There was a similar distribution of primary diagnoses and comorbidities in the development (2009–2012) and validation (2013–2014) time periods (Table 1; Online Resource 1, Table e3). Blood gas values and ventilator measurements were not substantively different between the two time periods (Table 2; Online Resource 1, Table e4). The mortality rate from 2009 to 2012 was 40.8 %, while the mortality rate was 38.0 % from 2013 to 2014.

Ped-RESCUERS development

The development dataset included 1611 children who received ECMO from 2009 to 2012. Eleven variables satisfied selection criteria and were included in the model fit to estimate the Ped-RESCUERS score (Table 3). To calculate Ped-RESCUERS for an individual child, use the β-coefficients in combination with the patient’s variable values as described in Table 4 or visit http://www.ped-rescuers.com.

Table 3 Multivariable analysis of factors associated with death prior to hospital discharge
Full size table
Table 4 Sample calculation of Ped-RESCUERS for a child with pneumonia
Full size table

The largest estimated association with mortality was pH, having an odds ratio of 0.46 (95 % credible interval (CI) 0.29–0.70). Three primary diagnoses were independently associated with mortality compared to our reference category of pneumonia. Asthma and bronchiolitis were associated with lower mortality, whereas pertussis was associated with higher mortality (Table 3). The presence of a malignancy was the only comorbidity meeting our selection criteria. Malignancy was associated with a 10 % increase in a child’s mortality risk relative to a child without malignancy (Table 3).

Among pre-ECMO support, a higher mean airway pressure, longer pre-ECMO intubation time, and pre-ECMO milirinone use were associated with an increased mortality risk. Compared to a child intubated for 2 days pre-ECMO, a person intubated for 10 days would have an associated 5 % increase in their mortality risk.

Unexpectedly, the model demonstrated that a higher arterial partial pressure of arterial carbon dioxide (PaCO2) within 6 h prior to ECMO was associated with a decreased mortality risk. In univariate analysis (Wilcoxon rank-sum test), there was no statistically significant difference between the PaCO2 of survivors and children who died (p = 0.41). The association between higher PaCO2 and lower mortality was only apparent after multivariable adjustment in the logistic regression model. In an attempt to understand what might be underlying this association, we compared the mortality rate and median PaCO2 of children with obstructive lung disease (asthma and bronchiolitis) to all other diagnoses. As expected, mortality rate was lower for children with asthma and bronchiolitis (22 %) compared to all other diagnoses (44 %). Those with asthma and bronchiolitis also had a higher median PaCO2 72 [53–95] mmHg versus 63 [48–84] mmHg.

Discrimination, calibration, and validation of Ped-RESCUERS

Ped-RESCUERS had an AUC of 0.690 in the developmental dataset (Fig. 2). From 2013 to 2014 there were 847 children who received respiratory ECMO support. Over the 11 selected variables in Ped-RESCUERS, 438 children had complete data (Fig. 1). These 438 children made up the internal validation dataset. In the validation dataset, the AUC decreased to 0.634 (95 % CI 0.595, 0.649) (Fig. 2).

Fig. 2
figure 2

Receiver operating characteristic curves. Receiver operating characteristics (ROC) curve dashed line depicts the area under the curve (AUC) for the development sample and solid line ROC curve gives the AUC for the validation data

Full size image

Figure e1 (Online Resource 1) illustrates the model’s calibration in the validation dataset by comparing the difference in expected and observed mortality. In a well-calibrated model, the plotted line should not deviate far from the line y = 0. Overall calibration, as measured by the Brier score, is 0.229 (95 % CI 0.227, 0.238). Thus, there remains variability in mortality prediction unexplained by the Ped-RESCUERS model.

Sensitivity analysis

We investigated the sensitivity of our findings to the 64 patients with unusual recorded measurements of ventilator settings and the five patients with a very low PaO2 (<10 mm Hg) by assuming these specific measurements were unobserved and re-running our entire imputation and analysis strategy. Confirming our findings, the identical set of predictor variables was selected in the sensitivity analysis, with very similar estimated associations. Because 17 patients in our validation dataset had unusual measurements, the size of the validation dataset decreased by 17 patients, from 438 to 421. The AUC, 0.635 (95 % CI 0.594, 0.651), and Brier score, 0.229 (95 % CI 0.227, 0.239), were equivalent to our primary findings.

Discussion

To our knowledge, Ped-RESCUERS is the first risk-adjustment tool created for children receiving respiratory ECMO support. This tool performs similarly in development and validation datasets. However, the discrimination of Ped-RESCUERS as currently formulated provides opportunities for improvement.

The tool is composed of three types of data including physiologic (i.e., blood gases), administrative (i.e., diagnostic), and therapeutic support (i.e., mechanical ventilator). Selected variables have face validity that resonates with routine clinical experience in ECMO support. Patients with cancer and pertussis have relatively poor outcomes, while those with asthma and bronchiolitis do well [24]. Additionally, children requiring ECMO later in their course and children with low pH and high ventilator settings also typically do more poorly [15, 24]. Previous studies have demonstrated an association between ECMO mortality and milrinone [29]. Milrinone is used for patients with cardiac dysfunction and its association with ECMO mortality may suggest that children with cardiac dysfunction and respiratory failure are less likely to survive.

Age, renal function, and measures of hypoxemia are often incorporated into severity of illness scores [15, 30–33], but were not selected in Ped-RESCUERS. The exclusion of acute renal dysfunction and measures of hypoxemia is consistent with the findings in the adult Respiratory ECMO Survival Prediction (RESP) score [15]. In Ped-RESCUERS, the ratio of the arterial partial pressure of oxygen to the fraction of inspired oxygen (PF ratio) was excluded because there is little difference between the PF ratio of those who survived versus died (Table 2). Since ECMO is able to oxygenate the blood regardless of the severity of lung disease, it may be that the most prognostic variables are those that predict lung recovery. We were unable to evaluate the predictive capacity of the ratio of the peripheral saturation of oxygen (SpO2) to FiO2 (the SF ratio) because ELSO does not record SpO2 [34].

Significantly, Ped-RESCUERS found that a higher PaCO2 was associated with a lower mortality risk, while the RESP score found that a PaCO2 greater than 75 mmHg was associated with a higher risk of mortality. We suspect two factors are underlying this difference. First, only after adjusting for pH (and the other variables) was there a statistically significant association between a higher PaCO2 and survival. The Ped-RESCUERS model suggests that among people with a similar pH, those with a higher PaCO2 tend to survive more frequently. The RESP score does not include pH, which makes the PaCO2 effects less comparable; this phenomenon is known as Simpson’s paradox [35]. Second, in the RESP score, only 1 % of the cohort had an obstructive respiratory failure (asthma). In Ped-RESCUERS, 15 % of children had asthma or bronchiolitis and these children survived twice as often and had a 15 % higher PaCO2. Consequently, the difference also may be because some of the survival benefit of asthma and bronchiolitis is being attributed to the associated high PaCO2.

Implications

We believe Ped-RESCUERS has three potential applications. First, it can be used to provide risk-adjusted internal and external benchmarking of ECMO performance quality to centers that participate in ELSO [36]. Benchmarking is needed to characterize the substantial, clinically meaningful variation in hospital-level mortality rates [37]. Second, risk-adjustment tools such as Ped-RESCUERS can enhance conduct of observational research [1]. For example, if a researcher wanted to perform an observational study comparing outcomes for children who received ECMO care under conditions of an awake state versus deep sedation, then Ped-RESCUERS measure could be used to match patients or more efficiently adjust for differences in the intervention and control arms. Third, Ped-RESCUERS could help physicians anticipate the risk of mortality for similar patient groups prior to ECMO.

Study limitations

Ped-RESCUERS cannot predict if an individual child will benefit from ECMO support, because this tool is derived from a sample of patients who all received ECMO. Ped-RESCUERS is designed to estimate the pre-ECMO mortality risk among patients who clinically are deemed to require ECMO. Additionally, Ped-RESCUERS was developed and internally validated using ELSO data. Consequently, it may not generalize to patients cared for at non-ELSO centers. Future studies should seek to test its discrimination and calibration at non-ELSO centers.

Ped-RESCUERS also has less discriminatory power than risk-adjustment tools for other clinical populations [30, 31, 38], but we believe there is an opportunity to improve the model discrimination with additional clinical data. This model does not include some physiologic measures included in other severity of illness measures [30, 31, 38] such as pupillary response [30, 31, 38], and other laboratory measures such as creatinine [30, 31], lactatemia [30, 31, 39], white blood cell count [30, 31], platelets [30, 31], and prothrombin level [30]. Models with this information are able to better predict mortality with an AUC greater than 0.85.

The discriminatory power of Ped-RESCUERS must be considered when applying the tool. For the first time, Ped-RESCUERS will enable inter-institutional risk-adjusted mortality rate comparisons, but the adjusted outcomes must be considered with the understanding that this model leaves significant variance in mortality unexplained. The unexplained variance in mortality will also impact Ped-RESCUERS applicability in research. It means the tool cannot completely adjust for differences between two groups in an observational trial. Nonetheless, Ped-RESCUERS provides a more efficient adjustment using one variable instead of 11, and this efficiency allows for a powered study with fewer patients.

A challenge of the database is that not all cases have complete data. Although most candidate variables are missing less than 10 % of observations, some are missing more frequently (Online Resource 1, Table e1). One variable, mean airway pressure, is missing in 29 % of cases. Importantly, the model performs similarly in the validation sample when only observations without missing data are tested, and we have attempted to minimize the effect of missing with multiple imputation. If we decided to exclude mean airway pressure based on its degree of missing, then we would have lost the predictive value of this variable, and excluded a clinically important factor.

Over time, pre-ECMO care has evolved. The median number of days between intubation and ECMO cannulation has decreased from 3.5 days [24] to 2 days. Additionally, in the 1990s, the median PaCO2 was 50 mmHg, by 2005 it was 60 mmHg [24], and now it is 65 mmHg. This shift in the median pre-ECMO PaCO2 demonstrates increasing permissive hypercapnia, which may be motivated by the 2000 publication demonstrating improved survival with low tidal volume ventilation in acute respiratory distress syndrome [40]. Because of these changes in pre-ECMO care and advances in ECMO care and technology, Ped-RESCUERS and any ECMO risk-adjustment tool will need recalibration with time [33].

Conclusions

Ped-RESCUERS is a promising first step in creating a pre-ECMO mortality risk estimation tool. Ped-RESCUERS does not explain all the difference in mortality risk, but it is an important incremental step that advances the risk adjustment for mortality for purposes of benchmarking and research.