Background

Acute respiratory distress syndrome (ARDS) is an acute and intense inflammatory disease process of the lungs with an associated high mortality rate of about 40% in non-COVID-19 ARDS patients [1, 2]. ARDS is a highly heterogeneous syndrome without a specific diagnostic test [3,4,5]. According to the LUNG-SAFE study, ARDS is unrecognized in more than half of patients at the time of fulfillment of ARDS criteria [1]. The current “Berlin definition” is under controversy [5,6,7,8]. The previous American-European Consensus Conference (AECC) [9] and the Berlin definitions are predominantly based on the value of the PaO2/FiO2 ratio at the time of ARDS onset [10].

A working definition of ARDS is essentially required for clinical trials, epidemiologic studies, and biological studies. Moreover, a definition of ARDS is required for clinicians to initiate treatments that would improve clinical outcomes [11], although stratification of ARDS—as defined by Berlin criteria—has been shown not very useful for assessing lung severity [8, 12]. The empirical PaO2/FiO2 cut-offs for “severity” of 100, 200, and 300 mmHg are arbitrary and poorly validated [13]. A recently published Reevaluation of Systemic Early Neuromuscular Blockade (ROSE) trial emphasized the variability of these PaO2/FiO2 cut-offs as the investigators did not enroll patients based on the PaO2/FiO2 at the time of ARDS onset, but based on a PaO2/FiO2 < 150 mmHg within the first 48-h after ARDS diagnosis [14, 15]. The PaO2/FiO2 ratio strongly depends on ventilator settings, including positive end-expiratory pressure (PEEP), inspiratory/expiratory time (I:E) ratio, and FiO2, and the requirement of a minimum PEEP of 5 cmH2O did not substantially improve Berlin prediction compared to AECC [13, 16]. Besides, Berlin definition does not account for the nonlinear relationship of PaO2 and FiO2 [17] and has a limited predictive accuracy in recent trials [18,19,20,21].

Assessment of severity in ARDS remains a challenge. The relation between oxygenation and prognosis in ARDS varies among published reports [20]. For example, the current mild ARDS category may not be significantly associated with 28-day mortality [22,23,24]. However, although stratification of severity based on Berlin criteria may be helpful to identify severe ARDS patients, it may have less significance to differentiate between mild and moderate ARDS [20]. A recent study identified two different subgroups of moderate ARDS using a 150 mmHg PaO2/FiO2 threshold and may represent a more homogeneous distribution of ARDS patients across subgroups of severity [25,26,27]. Whether ARDS outcome relates to severity of respiratory failure [28], a higher severity is a risk factor for prolonged mechanical ventilation [19]. Since PaO2/FiO2 does not account for PEEP in its calculation, reported PaO2/FiO2 provides a sense of ARDS severity without knowledge of applied PEEP levels.

The main goal of this study was proposing a novel variable [PaO2/(FiO2xPEEP)] or P/FPE for PEEP ≥ 5 cmH2O that, together with corresponding thresholds, could serve as an improved criterion to assess ARDS severity. The thresholds are 60 to 40 mmHg/cmH2O for mild, 40 to 20 for moderate, and less than 20 for severe. This new criterion adequately addressed Berlin’s definition gap in computing ARDS severity by including PEEP in the new oxygenation ratio. Increasing the PEEP level with the same FiO2 yields different PaO2 and SpO2 [29]. Thus, including PEEP in calculating the degree of oxygenation severity could be better than current Berlin definition. We examined this hypothesis by applying machine learning (ML) approaches for predicting ARDS severity over time.

Methods

Study design and patient population

Two clinical databases were used for evaluation. Data of the first 3 ICU days (considering day 1 for representative data within the first 24 h after ARDS onset, day 2 for data within 24–48 h after onset, and day 3 for data within 48–72 h after onset) (N = 2738, 1519, and 1341 patients, respectively) were extracted from a single-center database MIMIC-III (MetaVision, 2008–2012) [30]. The median length of an ICU stay (LOS) of all selected ARDS patients in MIMIC-III was 11.29 days (Q1–Q3: 7.85–17.54). Similarly, data of the first 3 ICU days after ARDS onset (N = 5153, 2981, and 2326 patients, respectively) were extracted from a multicenter database eICU (2014–2015) [31]. The median length of an ICU LOS of all selected ARDS patients in eICU was 11.72 days (6.92–18.84). All selected patients from both databases fulfilled the Berlin criteria for ARDS and were stratified into mild, moderate or severe ARDS [6] and received mechanical ventilation (MV) for > 48 h [32, 33]. Disease progression of ARDS in each database was tracked along those 3 ICU days to assess lung severity. Patients younger than 18 years were excluded.

Data extraction

Clinical data of ARDS patients were extracted from both databases (MIMIC-III and eICU) using Python 3.7, an interpreted, interactive, object-oriented, open-source programming language. The selection of clinical variables was based on previous studies [1, 19, 34,35,36,37].

MIMIC-III

MIMIC-III is a large, publicly available database including de-identified health-related data of approximately 60,000 admissions of ICU patients [30]. The input variables include baseline demographic information (age); hemodynamic parameters including mean, maximum and minimum heart rate (HR); ventilator parameters including mean, maximum and minimum respiratory rate (RR), SpO2, and PEEP. These predictors on the third ICU day after assessing lung severity, including their description (mean and 95% CI), are presented in Table 1, and Additional file 1: Tables S1 and S2. The main target variable was ARDS severity (where 0 = mild, 1 = moderate, and 2 = severe). ICU mortality (Fig. 1, Additional file 1: Figs. S1 and S2) and duration of MV were also obtained (Additional file 1: Table S3).

Table 1 Input variables and their descriptive statistics in MIMIC-III at 72-h
Full size table
Fig. 1
figure 1

Intensive Care Unit mortality at 72 h in relation to degree of lung severity in patients with acute respiratory distress syndrome included in the MIMIC-III database, according to PaO2/FiO2 ratio and P/FPE (see text for details)

Full size image

eICU

eICU is a multicenter and publicly available ICU database with high level of detail in the data about more than 200,000 ICU admissions [31]. Input variables included: baseline demographic information (age); ventilator parameters including PEEP; blood gas parameters including FiO2, PaO2, and PaCO2 (Table 2, Additional file 1: Tables S4 and S5). The main target variable was ARDS severity (where 0 = mild, 1 = moderate, and 2 = severe). ICU mortality (Fig. 2, Additional file 1: Figs. S3 and S4) and duration of MV were also obtained (Additional file 1: Table S6).

Table 2 Input variables and their descriptive statistics in eICU at 72-h
Full size table
Fig. 2
figure 2

Intensive Care Unit mortality at 72 h in relation to degree of lung severity in patients with acute respiratory distress syndrome included in the eICU database, according to PaO2/FiO2 ratio and P/FPE (see text for details)

Full size image

Experimental methods

Before starting our analysis, the thresholds of the P/FPE index (with PEEP ≥ 5) were experimentally tuned. We computed the minimum and maximum P/FPE values of the patients in the two databases, which were 2 and 60 mmHg/cmH2O, respectively. Then, several cut-offs were studied in order to determine the ones that could be more accurate in the stratification of ARDS severity. For this purpose, we tested round values (to be easily remembered by intensivists) in the range 2–60 and analyzed P/FPE index of the ARDS severity groups obtained. The partition showing a better separation of the ARDS severity groups obtained was achieved in this study for the following thresholds (with PEEP ≥ 5): 60–40 for mild, 40–20 for moderate, and < 20 for severe.

Our study is based only on ML analysis and not on the conventional statistical hypothesis testing analysis. In general, ML is an exploratory process and a current application of artificial intelligence to generate predictive models. Using this technology, there is not a one-model-fits-all solution. Precisely, there is no ML method that reaches the highest accuracy for all domains, datasets, or problem types [38]. The optimal model differs from one problem to another based on the characteristics of variables and observations. Our aim was to implement ML models capable of predicting ARDS severity over time to compare the PaO2/FiO2 ratio—as mandated by the current Berlin criteria for ARDS—with the proposed new P/FPE ratio according to the following three scenarios: (1) Scenario I: predicting ARDS severity in the 3rd ICU day using information captured in the 1st ICU day; (2) Scenario II: predicting ARDS severity in the 3rd ICU day using information captured in the 2nd ICU day; (3) Scenario III: predicting ARDS severity in the 3rd ICU day using information captured in the 1st and 2nd ICU days.

We implemented three robust supervised ML algorithms using Python 3.7. The ML algorithms were Light Gradient Boosting Machine (LightGBM) [39], Random Forest (RF) [40], and eXtreme Gradient Boosting (XGBoost) [41]. Grid search was used to identify the optimal values for their input parameters. The quality of the prediction models was computed based on a tenfold cross-validation approach. AUC and CORR (correlation between the predicted and actual values of severity level) were used to assess model performance in predicting ARDS severity as a categorical prediction. To provide a meaning to the findings, we used the classification of performance suggested by Hosmer and Lemeshow [42]: “excellent” if AUC ≥ 0.9; “good” if AUC is between 0.8 and 0.9; “fair” if AUC is between 0.7 and 0.8; “poor” if AUC is between 0.6 and 0.7; and “very poor” if AUC is below 0.6. For CORR, we used the interpretation suggested by Mukaka [43] who proposed “very high” for CORR ≥ 0.9 (positive correlation) or CORR ≤ -0.9 (negative correlation); “high” if CORR is between 0.7 and 0.9 (positive) or -0.9 and -0.7 (negative); “moderate” if CORR is between 0.5 and 0.7 (positive) or -0.7 and -0.5 (negative); “low” if CORR is between 0.3 and 0.5 (positive) or -0.5 and -0.3 (negative), and “negligible” otherwise.

Results

The findings of the three classification ML methods for the three predictive scenarios in the two databases are presented in Tables 3 and 4. Table 3 shows the quality of ML predictions for MIMIC-III, confronting the results obtained for PaO2/FiO2 (Table 3(a)) with those obtained for P/FPE (Table 3(b)). Table 4 shows the same comparative results in patients from the eICU database.

Table 3 Quality of the third ICU day severity predictive ML models for MIMIC-III
Full size table
Table 4 Quality of the third ICU day severity predictive ML models for eICU
Full size table

For MIMIC-III, the best ML severity predictive model on the third ICU day was obtained by scenario II and by P/FPE with an AUC = 0.788 and CORR = 0.566, using LightGBM algorithm. When PaO2/FiO2 is used, AUC = 0.635 and CORR = 0.19, but these performances were obtained with different algorithms. In qualitative terms, P/FPE ratio improves PaO2/FiO2 ratio from “poor” to “fair” AUC, and from “negligible” to “moderate” CORR.

For the eICU database, the results were slightly better. The best ML severity predictive model was also observed for scenario II. This finding confirms that the best approach to predict ARDS severity on the third ICU day is to consider the condition of the patient in the second ICU day after ARDS onset, rather than the first ICU day or both. For eICU data, the best AUC and CORR values are 0.873 and 0.745 for P/FPE; and 0.863 and 0.725 for PaO2/FiO2. These results are qualified as a “good” predictive accuracy and a “high” correlation.

In general, P/FPE ratio has a better behavior in the prediction of ARDS severity than PaO2/FiO2 ratio in terms of AUC and CORR. Whereas PaO2/FiO2 obtained up to 0.635 AUC and up to 0.19 CORR in MIMIC-III, the use of P/FPE reached 0.788 AUC and 0.566 CORR. This represents increments of + 0.153 AUC and + 0.376 CORR and shows the advantages of using the P/FPE ratio.

Discussion

In this large study, we propose a novel variable or formula (P/FPE) and corresponding thresholds for classifying ARDS severity. We investigated several ML methods to generate severity predictive models in almost 8,000 patients with ARDS over time after ARDS diagnosis. Our findings confirmed that the best approach to predict ARDS severity on the third ICU day is to consider the condition of the patient in the second ICU day after ARDS onset, rather than during the first ICU day as mandated by Berlin criteria.

For the MIMIC-III database, predictive models using the P/FPE ratio attained outstanding improvements in terms of AUC (15% improvement) and CORR (37.6% improvement), when compared to the previous PaO2/FiO2 models. For the eICU database, models based on P/FPE also outperformed PaO2/FiO2 predictions, with 14.8% and 2% improvements of AUC and CORR, respectively. The difference in terms of the accuracy between the two databases is remarkable regarding CORR. This is due to the fact that eICU is a multicenter ICU database with high granularity data (i.e., high level of detail in the data) for over 200,000 admissions to ICUs. By contrast, MIMIC-III is a single-center ICU database for approximately 60,000 admissions of ICU patients. Therefore, in all extracted data of the three ICU days, the number of extracted patients from eICU was greater than the number of extracted patients from MIMIC-III. Consequently, this would lead to better ML results in terms of CORR for the eICU database. Overall, the novel P/FPE ratio outperformed the PaO2/FiO2 ratio in all ML applied models and showed that predictions based on the patient condition in the second day after onset are better than predictions based on the first 24 h (7.2–13.8% AUC and 1.5–22% CORR improvements), followed by the predictions based on both the first and the second day conditions (0.1–0.3% AUC and 0.18–14% CORR improvements).

In contrast to our study, most recent studies developed ML approaches to predict the risk of ARDS in critically ill patients prior to ARDS onset [36, 44, 45], based on single-center databases [36, 45] and using one single ML algorithm [36, 44]. Consequently, their findings have serious limitations for the generalizability in the context of assessing the prediction of ARDS outcome.

This large study proposes a novel criterion to reclassify ARDS patients in terms of severity by using ML methods on an extensive amount of data from two large datasets of critically ill patients. The relatively good accuracy of P/FPE (when compared to PaO2/FiO2) in stratifying ARDS patients could allow to overcome the major clinical drawbacks of the current Berlin definition. Also, this study is implementing ML models for predicting severity over time after ARDS onset. Critically ill patients are an ideal population for clinical database investigations using machine learning algorithms because while the data from ICUs are extensive, the value of many diagnostic and therapeutic interventions remains largely unproven [46].

ARDS is considered one of the major reasons of ICU admission, and it is associated with a high hospital mortality [1]. Despite its high mortality rate and high rates of ICU utilization, ARDS remains critically misdiagnosed and globally under-diagnosed in the ICU settings [1]. Furthermore, increasing ARDS severity is associated with increased mortality rate [6]. The PaO2/FiO2 ratio categorizes ARDS patients according to the severity of their oxygenation deficit without considering the level of applied PEEP in the assessment of lung severity. The PaO2/FiO2 ratio does not appropriately show the severity of ARDS for PEEP ≥ 5. However, the application of PEEP plays a significant role in improving oxygenation. It is well established that changes in PEEP alter the PaO2/FiO2 in lung-injured patients [29]. Attempting to predict lung severity and patient outcomes based solely in PaO2/FiO2 on this basis is inherent flawed. Thus, the stratification of ARDS patients as proposed by the Berlin criteria is useless for assessing severity of lung injury and could be of no benefit for enrolling patients into therapeutic clinical trials. The P/FPE for PEEP ≥ 5 appropriately addressed Berlin’s definition gap in computing ARDS severity by including PEEP in the novel ratio. Clearly, our study showed that P/FPE thresholds improved prediction of ARDS severity. This can lead to important medical implications by accurately anticipate specific treatment for each ARDS category, which could eventually decrease ARDS mortality. In other words, P/FPE can represent a good solution for the clinical assessment of ARDS severity and as a guidance for treatment of ARDS.

Our study has several strengths. First, we have analyzed a large population of ARDS patients within their first three ICU days after onset. Second, we have described and validated our findings using both a large single-center database (MIMIC-III) and a large multicenter database (eICU). Third, we have investigated several ML predictive models for ARDS severity over time after ARDS onset. We believe that our approach is generalizable across other ARDS populations. However, we acknowledge some limitations to our study. First, our work is based on a retrospective analysis of data whose results concerning P/FPE benefits should be confirmed in further prospective studies. Second, our analysis is concerned with the evolution and stratification of patients in their third ICU day after ARDS onset. Although the first 72-h are essential in the management and progression of ARDS patients, our study lacks the assessment of a long-term outcome (e.g., ICU mortality, 60-day mortality). Third, further longitudinal studies on complete evolution of ARDS patients could help to find out new evidence(s) on the management of ARDS since our ML results achieved outstanding improvements compared to the current state, with “fair” to “good” predictions of ARDS severity [42]. Forth, one could argue that extracorporeal membrane oxygenation (ECMO) is not considered in this study. ECMO is a clinical outcome and can only temporarily sustain severe ARDS patients to bridge periods of time when oxygenation through the lungs cannot be achieved via MV. Moreover, ECMO is a constrained resource that is not available in all ICUs. Hence, for the purpose of our study, we only considered patients receiving MV for > 48 h [32, 33]. Fifth, regarding the potential consequences of using the new ratio at the bedside, further studies are needed to examine whether it could help for clinical decision making and guiding therapy. Our study opens a possibility to better define ARDS severity, as a new research area for patient care improvement.

Conclusions

This large study proposes a novel criterion based on the P/FPE formula to assess ARDS severity using ML, which is significantly better than the current Berlin criteria using baseline PaO2/FiO2. Clinically, applying the proposed new criteria for ARDS severity enables clinical care physicians to assess lung severity by involving PEEP information. Moreover, being able to better adjust the severity profiles of ARDS patients will potentially improve the selection of more adequate therapeutic regimens for each ARDS category, which could contribute to reduce ARDS mortality. However, additional studies are required in order to confirm this. In both databases (MIMIC-III and eICU) and either in Berlin or P/FPE, scenario II (assessment of oxygenation deficit after 24 h of ARDS diagnosis and routine ICU treatment) was the best severity predictive scenario. From a ML perspective, P/FPE outperformed PaO2/FiO2 in all ML models predicting ARDS severity after onset over time in all scenarios either in MIMIC-III or eICU. Accordingly, this study can serve as an example of how ML is a worth-considering technology to gain new insights in the development of ARDS predictive models which could contribute to improve ICU resource allocation and mortality reduction.