We read the article published recently in Critical Care by  Tan et al. with great interest. We appreciate their effort to evaluate high flow nasal cannula (HFNC) usage in post-extubated chronic obstructive pulmonary disease (COPD) patients with respiratory failure [1]. A non-inferiority study is a reasonable approach given NIV has shown benefit in post-extubation studies [2,3,4]. Unlike superiority trials, non-inferiority studies establish non-inferiority by rejecting a null hypothesis that the tested treatment is worse than the comparator by a pre-established minimum difference (non-inferiority cutoff or delta) based on results from prior studies [5]. However, several issues in this study prevent us from reaching this conclusion.

First, Tan et al. anticipated an NIV failure rate of 22% based on prior trials. The authors determined a delta of 9% for non-inferiority cutoff. They found that the experimental group (HFNC) had failure rates less than the control group (NIV), 22.7% vs. 28.6%, respectively. The absolute risk difference was − 5.8% (CI 95%; − 23.8 to 12.5). Since the CI range extends beyond the predetermined non-inferiority cut-off, inferiority is still a possibility and the study should be considered inconclusive [5]. For further clarification, we created a forest plot to visualize the primary outcomes CIs in relation to the delta point (Fig. 1).

Fig. 1
figure 1

Forest plot of confidence intervals (CI) for primary outcome and the causes of treatment failure

Full size image

Second, the suggested sample size of 44 subjects per group seems insufficient. We don’t have a description of the calculations used by Tan et al., but using their assumptions we calculated that at least 216 patients per arm would be required to prove non-inferiority with a difference of less than 9%, an alpha of 0.05, and a beta of 0.2 (it is worth noting that in the paper it is stated an alpha of 0.5 instead of 0.05, which we think is a typo since such a large error margin is not considered acceptable). A trial with a similar design referenced by Tan et al. [4] calculated a sample size of 300 patients per arm using similar assumptions.

Third, the failure rate in the NIV arm is higher than expected (28.6% vs 22%, OR = 1.3), and is higher than other previous studies [2,3,4]. This may create bias in favor of non-inferiority and should have been discussed further in the paper.

In summary, we conclude that the results of Tan et al.’s study can’t prove non-inferiority of HFNC compared to NIV, although it doesn’t exclude it either. Besides, further clarification regarding the sample size is required.

Authors’ response

  • Jiayan Sun,
  • Dingyu Tan,
  • Joseph Harold Walline &
  • Jun Xu 

Dear Editor,

We would like to thank Drs. Curtis, Kabchi and Alqalyoobi for their detailed comments and helpful feedback. We should first emphasize that the primary endpoint in our study was a composite of re-intubation and switching between non-invasive ventilation (NIV) and high-flow nasal cannula (HFNC), which would better reflect a real-life treatment failure of HFNC or NIV [6]. This composite endpoint had a significant impact on setting the estimated NIV failure rate and sample size. Most previous studies have defined a NIV failure as only tracheal intubation. In our study, the NIV failure rate was 28.6%, but, again, this was a composite of the re-intubation and treatment switch rates, leading to a higher overall NIV failure rate in our study.

In the study with a sample size of 300 mentioned in the letter, the baseline re-intubation rate for the two groups was set at the same value, though the non-inferiority cutoff value was similar to our study [4]. However, recent studies have shown that the treatment failure rate of HFNC is 4–12% lower than that of NIV [4, 7, 8], which significantly reduced the sample size required for our study. With reference to the trial of Kullberg et al. [9], according to different baseline failure rates (maximum 12% difference), we calculated that at least 88 patients would be required to assess a non-inferiority cutoff at 9% using an α = 0.05 (0.50 alpha was indeed a typo) and a β = 0.20.

Combining the statistical trend of high probability, we were careful in our paper to conclude that HFNC after extubation did not result in increased rates of treatment failure compared with NIV, though the CI range extends beyond the predetermined non-inferiority cut-off. Nevertheless, we believe the data we presented is a helpful first step to comparing HFNC with NIV in chronic obstructive pulmonary disease (COPD) patients. As the first randomized controlled trial to compare the failure rate of HFNC to NIV in patients with COPD after invasive ventilation, there may be different interpretations of the ideal sample size and test values. We set the feasible minimum sample size according to our calculations and found that the absolute risk difference between HFNC and NIV was − 5.8% (CI 95%; − 23.8 to 12.5). We agree that future studies with larger sample sizes should be able to narrow the CI range and help determine the best use of HFNC.