Background

Critically ill patients with circulatory shock have increased risks of multi-organ failure, long-term morbidity, and mortality [1]. Advanced hemodynamic monitoring in these patients may inform diagnosis and simultaneously guide management by providing insight into cardiac function, cardiac preload, and afterload [2]. Several methods for measuring cardiac output (CO) exist, varying from invasive (e.g. thermodilution by pulmonary artery catheter (PAC)) to minimally invasive (e.g. pulse contour analysis by FloTrac™ (Edwards Lifesciences, Irvine, USA)) or even non-invasive (e.g. transthoracic Doppler ultrasound by critical care ultrasonography (CCUS)). These methods all have their own merits, disadvantages and requirements [3].

One type of pulse contour analysis is the uncalibrated arterial pressure waveform analysis method to estimate CO (APCO). Reliability of APCO is questioned in patients with hemodynamic instability, and this occurs frequently in patients admitted to the ICU [4]. Therefore, CO measurements obtained by APCO should be interpreted with caution in critically ill patients with circulatory shock [5, 6].

The FloTrac™ system using the APCO method calculates CO based on the principle that aortic pulse pressure is proportional to stroke volume (SV) and inversely related to aortic compliance using a proprietary algorithm. FloTrac™ has been widely studied in more than 70 validation studies as of yet, mostly showing adequate performance in normo- and hypodynamic conditions, but not in patients with large changes in vascular tone which typically occur in patients with circulatory shock [7]. However, these studies vary by the statistical methods and versions of the algorithm used. Recently, the fourth-generation algorithm was developed to improve performance.

Evaluation of the trending ability rather than the agreement of absolute values of CO monitoring devices is increasingly considered in validation studies for assessment of potential clinical usefulness [8]. In addition to one single CO measurement for diagnosing circulatory shock, repeated measurements of CO informing the trending ability could be informative for monitoring and guidance of supportive treatments of patients with circulatory shock.

The aim of our study was to compare both agreements and trending ability for APCO measurements of CO (COAP) with CO routinely measured by CCUS (COCCUS) in critically ill patients with circulatory shock. CCUS was chosen as the reference standard since it is the preferred method for diagnosis, but not for monitoring, of circulatory shock in critically ill patients and is widely available [2, 9]. Importantly, it should be noted that CCUS is not a gold standard reference technique for method comparison studies aiming to evaluate the validity of CO monitors [10].

Methods

This study was a substudy of the Simple Intensive Care Studies-I (SICS-I), which was a single-centre, prospective, observational cohort study in which all consecutive acutely admitted adult patients expected to stay beyond 24 h were included (NCT02912624) [16, 17]. The STROBE guidelines for reporting observational studies were used (Additional file 1) [11]. The checklist for CO monitor method comparison studies was used [10]. The local institutional review board (Medisch Ethische Toetsingscommissie, University Medical Center Groningen) approved the study (M15.168207 and M16.193856). Written informed consent was obtained from all patients.

Selection criteria

In this substudy, all consecutive acutely admitted adult patients with suspected circulatory shock and expected to stay beyond 48 h were included from June 2016 to January 2017. The circulatory shock was defined as the requirement of any dose of vasopressor to maintain a mean arterial pressure (MAP) of 60 mmHg or if the MAP remained below 70 mmHg despite fluid resuscitation (defined by at least 1000 mL of crystalloids). In addition, at least one other sign of organ or tissue hypoperfusion had to be present: altered state of mind (Alert-Voice-Pain-Unresponsive scale) [12], mottled skin (Mottling score ≥ 1 [13]), decreased urine output (≤ 0.3 mL kg−1 h−1) or increased serum lactate level (≥ 2 mmol L−1). Exclusion criteria were inability to obtain sufficient quality CCUS images; no arterial line; atrial fibrillation; and aortic valve or mitral valve diseases known to impair the arterial waveform. We included this group of patients because CO measurements are indicated to identify the type of shock, select necessary therapeutic interventions and evaluate patient’s response to therapy [2].

Objectives

The primary objective was to evaluate COAP measurements in terms of the agreement and trending ability against COCCUS as reference technique in patients with circulatory shock.

Definitions and bias

Patient characteristics including clinical, hemodynamic and laboratory variables as well as Acute Physiology And Chronic Health Evaluation (APACHE) IV and Simplified Acute Physiology Score (SAPS) II values were recorded [14, 15]. Measurements were performed following protocolized definitions and procedures [16, 17].

In short, COCCUS was measured by transthoracic echocardiography using the Vivid-S6 system (General Electric, Horton, Norway) with cardiac probe M3S or M4S, and with default cardiac imaging setting. The parasternal long axis was used to measure the left ventricular outflow tract diameter. In the apical five-chamber view, a pulse wave Doppler signal in the left ventricular outflow tract was used to measure the velocity time integral. COCCUS was calculated using an established formula [18]. CCUS was performed after ICU admission within 6 h and repeated once every 24 h after admission provided there was no interference with clinical care. Researchers were trained in performing CCUS by experienced cardiologist-intensivists.

The FloTrac™ sensor was connected to an indwelling radial artery catheter and an EV1000™ monitor (version 4.00; Edwards Lifesciences, Irvine, USA), which continuously displayed COAP values. The value of COAP displayed on the EV1000™ monitor was registered simultaneously (i.e. ‘beat-to-beat’) with each COCCUS measurement.

All measurements, including CCUS findings, were kept blind for the caregivers. Quality of CCUS images and COCCUS measurements were validated by an independent specialized core laboratory (Groningen Image Core Lab) blinded for the COAP measurements

Statistical analysis

No formal sample size calculation was performed due to lack of data on COAP variation in patients with circulatory shock. Therefore, this study has an exploratory nature.

Data were presented as means with standard deviations or medians with interquartile ranges depending on distributions. Normality of data was checked using the Shapiro-Wilk test. Dichotomous and categorical data were presented in proportions.

Correlations were assessed by scatter plot, and the strength of linear correlation was determined by calculating a Pearson (r) coefficient. Bland-Altman analyses of repeated measurements in each patient were performed and means (bias) and SD of the differences, 95% limits of agreement (LOA) (=mean difference ± 1.96 × SD of the difference) as well as the percentage error of COAP versus COCCUS were calculated [19, 20]. To evaluate the trending ability of COAP versus COCCUS a four-quadrant plot was used and the concordance rate was calculated using an exclusion zone of 0.5 L min−1 [21]. For statistical analysis, we used STATA version 15.0 (StataCorp, College Station, USA).

Results

Participants

During the study period, 184 patients were diagnosed with circulatory shock, but only 24 patients appeared eligible for this study. One hundred patients who had circulatory shock were not included as they were expected to stay for less than 48 h, and 60 patients with circulatory shock were not included because CCUS was not possible or image quality was insufficient to perform measurements. Six patients had to be excluded because study procedures interfered with clinical care, leaving 18 patients to be included. One patient was excluded afterwards for invalid COAP measurements due to improper use of a FloTrac™ sensor. Thus, 17 patients were included in the final analyses (Fig. 1).

Fig. 1
figure 1

Patient flowchart. Abbreviations: CCUS, critical care ultrasonography

Full size image

Bias, precision and correlation

The characteristics of the 17 included patients are shown in Table 1 (and Additional file 2: Table S1). The mean COAP and COCCUS for 89 paired measurements were 5.9 ± 1.9 L min−1 and 5.7 ± 2.0 L min−1, respectively (p = 0.24). A significant correlation was observed for all CO measurements (r2 = 0.60, p < 0.001) (Fig. 2). Bias was 0.2 L min−1 (95% CI − 0.2 to 0.6), with LOA of − 3.6 L min−1 (95% CI − 4.3 to − 2.9) to 4.0 L min−1 (95% CI 3.3 to 4.7) (Fig. 3). Plotting a regression line in the Bland-Altman plot gave no arguments for proportional bias (line not shown). The overall percentage error was 65.6% (95% CI 53.2 to 77.3). Individual cardiac output measurements for each patient are provided in Additional file 3: Table S2.

Table 1 Patient characteristics
Full size table
Fig. 2
figure 2

Scatter plot of cardiac output measured by FloTrac™ and CCUS. Abbreviations: COAP, cardiac output measured using fourth-generation FloTrac™ algorithm; COCCUS, cardiac output measured by critical care ultrasonography

Full size image
Fig. 3
figure 3

Bland-Altman plot for repeated measurements showing the comparison between COAP and COCCUS. The mean bias between COAP and COCCUS and the upper and lower limits of agreement (LOA) are presented. Abbreviations: COAP, cardiac output measured using fourth-generation FloTrac™ algorithm; COCCUS, cardiac output measured by critical care ultrasonography

Full size image

Trending ability

For assessment of trending ability 72 paired measurements were analysed. Trending of measurements was evaluated using a four-quadrant plot (Fig. 4). Forty-five paired measurements showed a clinically relevant change, which was defined as larger than 0.5 L min−1. The concordance rate was 64.4%.

Fig. 4
figure 4

Four-quadrant plot showing the trend of COAP versus COCCUS. Exclusion zone of 0.5 L min−1 (white area). Abbreviations: COAP, cardiac output measured using fourth-generation FloTrac™ algorithm; COCCUS, cardiac output measured by critical care ultrasonography

Full size image

Discussion

In this prospective observational study, agreement and trending ability of COAP was compared with COCCUS in critically ill patients with circulatory shock. COAP showed a low bias of 0.2 L min−1 but a large percentage error of 65.6% when compared with COCCUS, indicating disagreement [20]. Trending ability was poor with a concordance rate of 64.4%. The new FloTrac™ algorithm should not be used for diagnosis or guidance of treatment in critically ill patients with circulatory shock.

Interpretation

There are no data on the reliability of CO measurements with the fourth-generation FloTrac™ software algorithm in critically ill patients with shock as of yet. The main concern with the previous version(s) of the APCO algorithm was the lack of reliability in tracking CO changes after hemodynamic interventions or in patients with sepsis [7, 22].

The low bias and the high percentage error of CO measurements are in accordance with results from another study, which tested the fourth-generation algorithm for tracking CO measurements after administration of phenylephrine to increase vasomotor tone in patients prior to cardiac surgery (bias − 0.7 L min−1; percentage error 55.4%) [23]. Concordance rate for trending ability was 87% which was higher than in our study. In that study, the chosen reference technique for measuring CO was thermodilution.

In a more recent study in patients undergoing cardiac surgery, the new FloTrac™ algorithm also showed lack of agreement and trending ability (bias − 0.4 L min−1; percentage error 37.1%; concordance rate 76%) [24]. The reference technique was thermodilution and in that study, bias was influenced by systemic vascular resistance.

Another study tested the fourth-generation FloTrac™ algorithm in patients undergoing abdominal aortic aneurysm surgery and also found a low bias and high percentage error (bias 0.4 L min−1; percentage error 46.7%) of CO measurements [25]. The concordance rate for trending ability was 26.9% before and after aortic clamping and 47.3% before and after first unclamping of the iliac artery. The reference technique chosen in this study was transoesophageal echocardiography.

Advanced hemodynamic monitoring techniques are currently used to identify the type of shock, to guide choices of interventions and to evaluate the response to therapy. Less invasive hemodynamic monitoring techniques such as APCO are currently not recommended for use in patients with shock, especially when receiving vasopressors [2, 26]. Our findings support this statement.

Implications and generalizability

Even though CO monitoring is considered a cornerstone in diagnosing and managing circulatory shock, the sequential evaluation of the hemodynamic state during shock is only a level 1 recommendation based on low quality of evidence [2].

The abovementioned studies validating the new fourth-generation FloTrac™ algorithm were performed in different target populations and contained different reference techniques, which limit comparability. There is a concern about the interchangeability of COCCUS and CO measurements by thermodilution, and tracking ability of the two methods has only been scarcely assessed and needs evaluation by larger studies [27].

Considerations and limitations

There are several considerations and limitations when interpreting the results of our study. First, since only parallel and no serial CO measurements were performed for each time point, the precision of individual measurements could not be assessed. While only few studies determined the precision of the CCUS and FloTrac™ technologies, it is a given that both methods have some degree of variation which influences precision of agreement [28]. This might influence—and possibly overestimate—the observed bias and precision to an unknown extent, since the precision of the CCUS as reference method was not incorporated.

Second, a stepwise approach and checklist for the complete presentation of CO method comparison research have been published [10]. This checklist includes a design study phase where it is encouraged that criteria for acceptable bias and LOA or percentage error are defined, and a sample size calculation should be performed prior to the conduct of method comparison studies. In our study, we defined clinically acceptable limits based on available literature, but we did not specify a sample size in advance. The current study could serve as a pilot for a further validation study.

Third, during the study period, we included only 17 patients. Patients with circulatory shock were eligible only if they were expected to stay for longer than 48 h and if it was possible to perform CCUS. We chose this definition to ensure that a complete picture of shock treatment could be presented which allowed for the best comparison between the two methods. Last, CCUS was used as a reference technique in our study despite pulmonary or transpulmonary thermodilution being the gold standard for CO method comparison studies [10]. Therefore, we cannot prove direct superiority of either method. In order to do this, a comparison with a thermodilution method will have to be performed. We chose CCUS as reference because it is currently the first-line evaluation modality in patients with circulatory shock and also because it is widely available and used in the ICU for diagnostic purposes [2, 29]. However, images required to make COCCUS measurements are unobtainable in up to 20% of patients [30].

FloTrac™ measurements of CO are still not recommended in critically ill patients [5, 6], and further clinical studies comparing minimally invasive techniques for CO estimation with a reference technique are needed for further validation of these techniques and also for extending applicability to other types of patients, who were initially not the target population.

Conclusions

In critically ill patients with circulatory shock, there was disagreement and clinically unacceptable trending ability between values of cardiac output obtained by uncalibrated arterial pressure waveform analysis and critical care ultrasonography.