Introduction

During the worldwide Covid-19 pandemic, 15% to 20% of patients may need hospitalisation [1, 2]. Among them 5% to 10% require intensive care unit (ICU) admission for hypoxemic acute respiratory failure (ARF), and more than 50% of these severe Covid-19 ARF patients could develop acute respiratory distress syndrome (ARDS) [3].

Among different oxygenation strategies for ARF (standard or high-flow nasal oxygen therapy (HFNO), non-invasive (NIV) or invasive mechanical ventilation (MV)), HFNO has recently become the first-line treatment for hypoxemic ARF due to its physiological effects and potential benefit on patients’ outcomes [4,5,6,7]. Owing to a possible contamination of caregivers by SARS-CoV-2 aerosolization, HFNO was not recommended initially and rarely used in Covid-19 ARF [3, 8,9,10]. Later, increased knowledge and protective equipment for frontline healthcare workers [11] led to a wider HFNO use in this indication [12,13,14,15].

As with any non-invasive ventilatory management strategy, and although this has never been formally demonstrated in a prospective manner, HFNO should not delay, however, endotracheal intubation at the potential risk of worsening patients’ outcomes [16]. Thus, a simple index recorded at bedside could help clinicians to identify patients at high risk of HFNO failure and to start MV rapidly. For this purpose, ROX index, defined as the ratio of pulse oximetry/fraction of inspired oxygen relative to respiratory rate (SpO2/FiO2/RR), was proposed recently in “de novo” hypoxemic ARF [17,18,19].

Whilst HFNO allows to rapidly improve hypoxemia and decrease RR in patients if successful [5, 7], this benefit may be clinically less obvious in Covid-19 ARF patients. Indeed, because of different pathophysiological mechanisms of hypoxemia in these patients, they can be severely hypoxemic while being little or not tachypneic with relatively mild respiratory discomfort, the so-called “happy” or “silent” hypoxemia, especially at the initial phase of ARF [20, 21]. Therefore, the ROX index could be potentially less discriminant, with different cut-off values in patients with Covid-19 because of “silent hypoxemia” which is considered through the RR into the ROX index.

Consequently, we hypothesised that ROX index could be a potential useful diagnostic tool with sufficient accuracy in hypoxemic Covid-19 ARF patients. However, as suggested more recently [22,23,24,25,26], cut-off values for these patients may be different from those previously described for non-Covid-19 patients [17,18,19] and “silent hypoxemia” in Covid-19 patients [20, 21] could also impact the ROX index measurement. Therefore, we aimed to evaluate the accuracy and usefulness of ROX index to predict the risk of intubation or death in Covid-19 ARF patients initially treated with HFNO in ICU.

Methods

Study design

We conducted a multicentre cohort study during the first Covid-19 outbreak in Eastern Normandy between March 10th and May 25th, 2020. The local ethics committee approved the study (approval number E2020-40). Due to its retrospective observational design, patients’ written informed consent was not required.

Study population and HFNO strategy

We screened all consecutive adult patients referred to five ICUs, i.e., two in Rouen University hospital and three in regional community hospitals (Elbeuf, Dieppe, Le Havre), and admitted for hypoxemic ARF (need for standard O2 ≥ 6 L/min to reach a pulse oximetry (SpO2) ≥ 92%, with or without clinical signs of respiratory distress). For inclusion, patients had to have highly suspected or confirmed SARS-CoV-2 pneumonia based on CT-scan imaging and/or RT-PCR, and had to be treated with HFNO as first-line respiratory support in ICU. We excluded Covid-19 ARF patients transferred from other French regions, patients treated only with standard O2 < 6 L/min, and patients requiring immediate intubation or with do-not-intubate order at ICU admission. All patients were followed until hospital discharge or death, whichever occurred first. Some patients from Rouen University ICUs were included in a previously published study [15].

Depending on participating ICUs, HFNO was delivered either by Optiflow® (Fisher & Paykel Healthcare, Auckland, New Zealand) or Airvo2® (Fisher & Paykel Healthcare, Auckland, New Zealand) device, or through an ICU ventilator. As previously described, initial HFNO settings were the highest tolerated humidified flow rate ranging from 30 to 60L/min, and a FiO2 titrated for a SpO2 > 92% [6].

Evaluation criteria and data collection

The primary objective was to assess the accuracy of ROX index, hence we determined a cut-off value to discriminate between HFNO failure (sensitivity) and HFNO success (specificity). More precisely, HFNO failure was defined as the need for intubation or death before intubation within 28 days after ICU admission and HFNO initiation (H0), since NIV is not usually used as second-line ventilatory support in hypoxemic ARF in participating ICUs [6]. The final decision for intubation was made by the attending ICU physician, the main criteria being shared by all participating ICUs [6]. To estimate ROX index specificity, patients who did not need MV and who did not die within 28 days after H0 were classified as HFNO success. The compromise between late intubation (sensitivity) and avoiding MV (specificity) was established at H0, as well as at prominent time-points thereafter [17, 18].

To reach our secondary objective to predict HFNO failure within 28 days after H0, we retrieved the following information from patients’ electronic files: patients’ characteristics (age, sex, body mass index, comorbidities), Simplified Acute Physiologic Score II (SAPS II) [27] and Sequential Organ Failure Assessment (SOFA) [28], Covid-19 history (symptom onset, RT-PCR assay result, type and pulmonary extension of chest-X ray or CT-scan injuries), renal or hemodynamic organ failure, clinical respiratory conditions and parameters (interface and flow rate with standard O2, estimated FiO2 [29], SpO2, RR), arterial blood gas (ABG) at ICU admission before HFNO initiation. During the first 24 h after HFNO initiation, clinical respiratory conditions and parameters (HFNC flow rate, FiO2, SpO2, RR) were recorded at H0, H2-H4, H6, H12, H18, H24, as well as ABG at H2-H4, H6-H12 and H12-H24.

Furthermore, we collected data regarding time, cause and duration of intubation, respiratory conditions at ICU discharge, vital status and date of ICU and hospital discharge.

Statistical analysis

To assess ROX index accuracy, the primary endpoint was sensitivity based on the cut-off of 4.88 at H12 under HFNO as proposed previously by Roca et al. [17, 18] in non-Covid-19 ARF patients. For this purpose, sample size was determined such that the half-width of the 95% confidence interval for sensitivity should not exceed 20%. A cohort rather than a case–control design was chosen because knowledge of prevalence is needed for prediction of HFNO failure during the first 28 days after ICU admission. The receiver operating characteristic (ROC) curve visualizes ROX index accuracy. To this end, specificity for a given time-point was estimated by considering those patients as controls who received HFNO at this time and had no HFNO failure before hospital discharge or day 28 after ICU admission, whichever came first. Sensitivity was estimated correspondingly, i.e., patients who received HFNO at the given time-point and had HFNO failure subsequently. Youden’s index served to determine the cut-off, defined as the largest difference between sensitivity and 1-specificity.

To reach the second objective, the cumulative incidence function of HFNO failure was depicted because, unlike those patients who decease, not all patients need intubation. Then covariates, fixed at the time prediction starts, were screened one by one to assess their influence on the risk of HFNO failure within the first 28 days after ICU admission using logistic regression. Covariates significantly related to this risk were considered for multivariable analysis and backward selection was used to establish a parsimonious model.

Patient characteristics of at least ordinal level were described using the median with the first (Q1) and third quartile (Q3). The Kruskall–Wallis’s test was used to come to know whether there was a potential center effect. Freeman–Halton’s test and Wilcoxon’s test for independent samples were used to compare patients with to those without vasopressor at ICU admission.

Given the study’s exploratory nature, no correction for multiple testing was carried out and a p-value less than 0.05 was considered statistically significant. Analyses were performed using SAS software from SAS Institute Inc. (Cary, NC, USA).

Results

Among 218 Covid-19 hypoxemic ARF patients admitted to ICUs, 99 (45%) were treated with HFNO alone as first-line respiratory support (Fig. 1). During the 28 days following ICU admission, 50 patients (51%) remained alive without requiring intubation (HFNO success group) whereas 49 patients (49%) failed (HFNO failure group), including 3 patients who died without prior intubation and 46 who were intubated. Among the latter, 12 died before ICU discharge (Fig. 1). Table 1 shows patients’ characteristics at ICU admission and HFNO initiation for the overall population and according to HFNO success and failure groups.

Fig. 1
figure 1

Flowchart of the study and HFNO outcome ICU intensive care unit, ARF acute respiratory failure, HFNO high-flow nasal oxygen therapy

Full size image
Table 1 Patients’ characteristics and conditions at ICU admission and at discharge according to HFNO outcome
Full size table

As regards ROX index accuracy at H0, the area under the ROC curve (AUROC) was 0.70, representing a relative acceptable discrimination (see Additional file 1: Figure S1), and Youden’s index yielded 6.20 as cut-off value (Table 2). For this cut-off, sensitivity was 63% [95%CI 48%-77%] and specificity was 76% [95%CI 62–87%]. Hence, among the 49 patients who failed HFNO (intubation or death) before day 28, 31 (63%) had a ROX index ≤ 6.20 at H0 and were correctly classified positive. Among the 50 patients who remained alive without intubation, 12 (24%) had a ROX index ≤ 6.20 and were hence classified falsely positive (Fig. 2 and see Additional file 1: Figure S2).

Table 2 ROX index accuracy to predict HFNO failure at different times after HFNO initiation
Full size table
Fig. 2
figure 2

ROX index performance to predict HFNO failure at different times after HFNO initiation HFNO: high-flow nasal oxygen therapy; red line gives proportion of patients in the HFNO failure group with a ROX index ≤ a chosen cut-off value; black line gives proportion of patients in the HFNO success group with a ROX index ≤ a chosen cut-off value. For example: at H12, using a Rox index of ≤ 9.75 as cut-off would identify 90% of patients with HFNO failure after H12, whereas this cut-off would identify only 41% of patients with HFNO success after H12, avoiding intubation

Full size image

Also, at H2–H4, H18 and H24, ROX index accuracy was still relatively acceptable (0.70 ≤ AUROC < 0.80) to identify patients who needed intubation (Table 2). Until H12, both the ROX index cut-off value and its sensitivity increased, whereas specificity decreased. At H12, the cut-off based on Youden’s index was 8.73, yielding a sensitivity of 87% [95%CI 70–96%] and a specificity of 45% [95%CI 31–60%] (Table 2). Hence, though identifying 87% of subsequent HFNO failures, a ROX index ≤ 8.73 at H12 was also observed for the majority (55%) of patients who did not need intubation. When applying the previously suggested ROX index cut-off value of 4.88 [17, 18] to our population at H12, sensitivity was 29% [95%CI 14–48%] with a specificity of 90% [95%CI 78-97%].

After H24, when no more than 25 patients failed (see Additional file 1: Figure S3), confidence limits for sensitivity were wide and even < 50% (Table 2) suggesting the absence of a reliable basis for appreciating ROX-index discrimination power.

As regards patients with HFNO failure, conditions of intubation and clinical respiratory parameters are shown in the Additional file 1: Table S4. The main cause of intubation was Covid-19 ARF impairment (91%) and occurred within 29 h after ICU admission for half of them. Intubated patients were severely hypoxemic with a median PaO2/FiO2 of 73 mmHg and a ROX index of 3.83 at the time of intubation. Of note, 45/49 HFNO failures (92%) occurred within 4 days after HFNO initiation (see Additional file 1: Figure S3). We also analyzed intubated patients by underlying hemodynamic status (vasopressors vs non vasopressors) at ICU admission, which is recognized as a risk factor of HFNO failure but not taken into account in the ROX index. Intubated patients with vasopressors were more rapidly intubated after ICU admission (p = 0.0017) and HFNO initiation (p = 0.0010), as well as they had lower PaCO2 levels (p = 0.0262) at the time of intubation as compared to intubated patients without vasopressor at ICU admission (see Additional file 1: Table S4).

By screening covariates at H0 (univariate analysis), potential predictive factors for HFNO failure were (Table 1): SAPSII (p < 0.0001), SOFA score (p = 0.0027), presence of acute renal failure (p = 0.0277), hemodynamic failure (p < 0.0001) and severity of Covid-19 pulmonary lesions, in terms of number of affected quadrants (p = 0.0119) and injury extension (p = 0.0067). As regards respiratory conditions at H0, risk factors associated with HFNO failure were (Table 1): FiO2 (p = 0.0006), SpO2 (p = 0.0011), SpO2 /FiO2 (p = 0.0004), and ROX index (p = 0.0005). By contrast, no evidence was found of RR being related to HFNO failure (p = 0.3555). In patients with available ABG before HFNO initiation (n = 89), we found PaO2 (p = 0.0132), SaO2 (p = 0.0157) and PaO2/FiO2 ratio (p = 0.0087) related to a risk of HFNO failure.

Using backward selection method, the multivariate analysis yielded the following factors known at H0, which were predictive for HFNO failure within 28 days after ICU admission: SAPSII (p = 0.0003), PaO2 (p = 0.0497) and radiographic extension of pulmonary injuries (p = 0.0263) (Table 3).

Table 3 Risk factors at HFNO initiation to predict HFNO failure according to respiratory parameters (multivariate analysis)
Full size table

Regarding HFNO outcomes (Table 1), failure occurred 28 h after H0 on the median, whereas patients in the success group used HFNO for 136 h (p < 0.0001). ICU (p < 0.0001) and hospital (p = 0.0025) length of stay were longer among HFNO failures compared to HFNO successes. Deaths occurred among HFNO failures only, with a 28-day ICU mortality of 27% (p < 0.0001) and no death occurred in hospital after ICU discharge (Table 1).

In particular, there was no evidence of center effects with respect to HFNO outcome, ROX index at H0, and, among those who were intubated, HFNO duration before intubation (see Additional file 1: Table S5).

Discussion

Our results show that, after HFNO initiation, the ROX index discriminates only in a technically acceptable manner at different time-points between patients who were intubated or died within 28 days after ICU admission, and patients who did not need intubation. It also highlights that population- and setting-specific cut-off values rather than defaults like those previously suggested [17,18,19] should be used, in particular in hypoxemic ARF patients with Covid-19. Finally, we also identified SAPS II and the severity of pulmonary lesions at ICU admission as potential independent risk factors for HFNO failure.

Due to its relevant physiological effects and potential benefit on patients’ outcomes [4,5,6,7], and although not recommended at the beginning of the SARS-CoV-2 pandemic [3, 8,9,10], more and more Covid-19 ARF patients were managed with HFNO [12,13,14,15]. In our study, the intubation and ICU mortality rates were 49% and 15%, respectively. Previous studies reported intubation rates varying from 18 to 56%, and ICU or 28-day mortality rates ranging from 7 to 25% in this population [12,13,14,15].

Nevertheless, timely intubation remains a major challenge for ICU clinicians when managing severe hypoxemic ARF patients, including those treated with HFNO [16, 30, 31]. In this way, ROX index was found useful in non-Covid-19 ARF patients [17, 18]. A ROX index ≥ 4.88 at H2, H6 and H12 after HFNO initiation predicted a lower risk of intubation, with accuracy increasing over time [17, 18]. More recently, ROX index was evaluated in Covid-19 ARF populations admitted to ICU [22,23,24,25,26]. Different ROX index cut-off values were proposed to discriminate best between patients on HFNO needing intubation or not, not only in non-Covid-19 [17,18,19] but also in Covid-19 ARF populations [22,23,24,25,26]. Similar to our study, higher ROX cut-offs than 4.88 were generally reported to prevent late intubation in Covid-19 ARF populations [22,23,24,25,26]. For instance, in our study, a cut-off of 4.88 to prevent late intubation would have identified only a minority of those who failed after H12 (29%), whereas a vast majority (87%) would have been identified with the cut-off based on Youden’s index.

Differences regarding cut-off values might be explained, at least partially, by the pathophysiological mechanisms of “happy” or “silent” hypoxemia which can be difficult to recognize early and hence delay ARF management in Covid-19 patients [20, 21]. Indeed, patients can be severely hypoxemic whilst being little or not tachypneic with relatively mild dyspnea, especially during the initial phase of Covid-19 ARF. Although we found no evidence that RR was related to subsequent HFNO failure, a prospective cohort study suggested that RR between 30 min and H6 after HFNO initiation was a simpler and more accurate parameter than ROX index to predict HFNO failure in Covid-19 ARF patients [26]. At the same time, differences could also depend on the FiO2 used. Indeed, we titrated FiO2 lower than reported for non-Covid-19 populations despite similar algorithms [18, 19]. Furthermore, a cohort study on 2040 Covid-19 ARF patients reported a better performance of SpO2/FiO2 ratio than ROX index in terms of the AUROC to predict HFNO failure at admission to emergency department [32]. The ROX index value could also be influenced by the level of gas flow rate set with HFNO [33]. In fact, the risk of intubation may not only depend on respiratory conditions and HFNO settings, but also on non-respiratory functions. A modified ROX index was thus proposed including heart rate to improve its diagnostic accuracy for hypoxemic ARF on HFNO [34]. Finally, we have shown that ROX index could be used to predict Covid-19 patients’ outcomes in the initial phase of ARF. Nevertheless, evidence is lacking regarding the extent to which ROX index cut-off values depend on populations (Covid-19 vs non Covid-19), their characteristics like the underlying cause of ARF, respiratory conditions (RR), or HFNO settings (FiO2, flow rate).

When covariates known at ICU admission were considered one by one, notably, the non-respiratory SOFA score appeared as a potential candidate for the parsimonious model to predict HFNO failure. Unlike others [24], we found no evidence that either this score or ROX index are essential predictors of HFNO failure. Rather, we identified SAPS II and severity of pulmonary lesions at ICU admission as potential risk factors in a parsimonious model.

This study has several limitations. First, the sample size was determined to estimate sensitivity such that its confidence interval was still meaningful but was based on reported results in a non-Covid-19 population [18]. Second, due to its observational design, intubation criteria may vary slightly between centres introducing some heterogeneity in decision-making. Nevertheless, all participating ICUs in the study used similar HFNO practices in hypoxemic ARF [6]. Consequently, the present study also provides some real-life information on HFNO practice delivered to Covid-19 ARF patients in different centres.

In conclusion, ROX index cut-off values based on Youden’s index differed from those previously described [17,18,19] and could be population and setting specific. The ROX index appears to have a technically acceptable but clinically low capability to discriminate between HFNO failures and successes in Covid-19 ARF patients and seems not essential in a parsimonious model to predict HNFO failure in Covid-19 ARF patients. ICU admission parameters (SAPS II and severity of pulmonary injuries at ICU admission) should also be considered because they appeared more useful than ROX index to predict HFNO outcomes in this specific population.