Background

When an experimental intervention cannot be compared to placebo, researchers conducting randomized clinical trials have two options: protocolized or usual care control groups [1]. Usual care, also known as routine care, may be defined as the full spectrum of patient care processes and treatment decisions at the discretion of healthcare administrators or individual clinicians that may have an impact on patient outcomes [2]. Protocolized control groups reduce practice variability and may increase the signal-to-noise ratio, thus increasing the likelihood of observing a treatment effect [3]. However, this strategy poses a theoretical risk of comparing two interventions that may be inferior to usual care [4,5,6]. Alternatively, if practice changes over the course of the trial, the control group may become misaligned with usual care by the end of the trial [2, 3, 7]. Given these concerns, experts have questioned the conclusions drawn from randomized clinical trials that have changed practice in transfusion medicine and mechanical ventilation [5, 8, 9] and proposed usual care comparators as an alternative. This approach gives clinicians the freedom to provide what they consider to be optimal care in the control arm and eliminates the risk that an apparent benefit of an experimental intervention results from excess harm introduced in the control arm. However, usual care comparators allow more variability and, thus, may reduce the likelihood of observing a difference in clinical effect if one exists [3, 10].

Accordingly, when interpreting the result of a clinical trial with a usual care control arm, it is relevant to consider the extent to which the outcome of usual care in the trial corresponds to the outcome of usual care before the trial began. In 2018, we launched a clinical trial comparing a permissive hypotension strategy to reduce exposure to vasopressors vs. usual care in critically ill patients (NCT03431181) [11]. Vasopressors are medications given intravenously that increase blood pressure via vasoconstriction and that are commonly used liberally by clinicians despite their potential side effects [12,13,14]. The mean arterial pressure (MAP) in patients receiving vasopressors is most proximally vasopressor titration, although it may also be affected by other aspects of usual care including management of fluids, diuretics and ultrafiltration, and sedation. Concerned that the promotion of the trial and dissemination of background evidence at participating sites would raise awareness regarding the potential risks associated with usual care and modify standard practice during the trial, we compared mean arterial pressure (MAP) values in patients before vs. during the trial. A secondary objective was to compare MAP between three groups: patients treated with vasopressors before the initiation of the trial, patients in the trial’s control group, and patients who were potentially eligible for the trial while it was recruiting but who were not enrolled. Our hypothesis was that MAP would remain unchanged among these 3 groups of patients, reflecting stability in usual care practices.

Methods

Design

Multicenter observational study using data collected both prospectively and retrospectively to compare MAP of patients while receiving vasopressors before and during the Optimal VAsopressor TitratION in patients 65 years and older (OVATION-65) trial [11].

Trial eligibility

More information regarding the design of the OVATION-65 trial is available in the published protocol [11]. Briefly, the OVATION-65 trial enrolled 157 patients of 65 years of age or older who were receiving vasopressors for vasodilatory hypotension across seven Canadian sites (Fig. 1, Supplementary Table 1). Treating physicians had to confirm that patients were expected to receive vasopressors for at least six more hours. Other eligibility criteria appear in Table 1.

Fig. 1
figure 1

Patient’s flow chart – in the nested study. Abbreviations: ECMO, extracorporeal membrane oxygenation; ICU, intensive care unit; OVATION-65, Optimal VAsopressor TitratION in patients 65 years and older. * These patients were fully eligible but not enrolled in OVATION-65 and were not included in the analyses because identifying data, including medical record number, were not available

Full size image
Table 1 Eligibility Criteria
Full size table

Nested observational study eligibility

For this observational study, two investigators independently screened medical records to identify patients who would have met the OVATION-65 eligibility criteria 13 months to 1 month before the trial launched (‘pre-trial period’). They also identified patients who were potentially eligible for inclusion while the trial was ongoing but not enrolled (‘non-enrolled group’). The pre-trial and non-enrolled groups were both identified retrospectively from the list of all patients over 65 years old treated with vasopressors obtained from the medical records departments of participating sites. Of note, not all trial eligibility criteria could be applied retrospectively. For example, as a surrogate for the criterion of an anticipated duration of vasopressor therapy of at least six additional hours, we included patients who received vasopressors for 6 h or more (Table 1). The ‘during-trial period’ consisted of patients randomized in OVATION-65 to usual care (identified prospectively as the trial was ongoing) and the non-enrolled patients (identified retrospectively). We did not include patients identified prospectively as eligible for OVATION-65 but not enrolled (for example, because of lack of consent) because identifying details (including the medical record number) were not recorded for these patients.

Measurements

For patients included in the trial, day 1 refers to the day of randomization whereas for patients in the other 2 groups, day 1 is defined as the day vasopressor therapy began. We recorded dose-rates for all vasopressor agents used (i.e. phenylephrine, epinephrine, vasopressin and dopamine) converting them to norepinephrine equivalents using a previously published formula [12, 15]. Severity of illness was assessed by the Acute Physiology and Chronic Health Evaluation II (APACHE II) score [16] in the 24 h following ICU admission. Patient’s MAP values while receiving vasopressors were collected hourly from day 1 until 24 h following discontinuation of vasopressors or day seven, whichever came first. Since blood pressure values are commonly measured multiple times per hour in ICUs, we used the values recorded closest to each hour mark, following the same instructions as in the OVATION-65 trial. We calculated the mean MAP while receiving vasopressors for each patient using all hourly recorded values while patients were receiving vasopressors (i.e. ignoring MAP values once vasopressors were discontinued).

In addition to actual MAP values, we collected from medical records hourly MAP targets - as prescribed by treating teams in the physician orders - as well as the following clinical outcomes: complications (supraventricular arrhythmia, ventricular arrhythmia, myocardial ischemia, clinically detected stroke, extremity ischemia, mesenteric ischemia and severe acute kidney injury, defined by stage 3 Kidney Disease Improving Global Outcomes criteria [17]), duration of ICU and hospital stay, ICU readmission(s), and ICU and hospital mortality. Patient’s characteristics at baseline (sex, age, comorbidities, admission type, reason of ICU admission, therapy at baseline, participating site and time from trial initiation to hospital admission of the patient) were also collected. We developed a case report form, created a detailed instructions manual, provided adequate training to the research personel involved in data collection and collected data in duplicate for 10% of the charts to ensure data accuracy and consistency. Clinicians involved in the management of vasopressor therapy were informed of the beginning of a trial less than 2 weeks before the site initiation visit and trained during the site initiation visit that corresponds to the start of the trial.

Statistical analysis

Sample size

We included patients enrolled in the usual care control group of the OVATION-65 trial from five of the seven sites (2 sites lacked human resources to participate to this nested observational study; supplementary Table 1). In contrast, one site that was activated had still not enrolled a patient in the trial when enrollment was terminated but did collect retrospective data for this nested study. Data provided by this site were excluded of the adjusted sensitivity analysis. At the five sites that had enrolled a variable number of patients in the trial’s usual care control group, we collected data pertaining to a minimum of 30 patients [18] in each of the pre-trial and non-enrolled groups ensuring equal numbers were treated during the winter (November–April) and summer months (May–October) to account for potential seasonal variations in case-mix. The number of patients in the pre-trial and non-enrolled groups was increased at one site that enrolled more than 30 patients in the trial’s usual care arm. The statistical power of this sample size was over 95% and a difference of 2.5 mmHg in mean MAP could be detected.

Statistical analyses

Interrater agreement was evaluated for the selection of the patients identified retrospectively using a kappa statistic. Categorical data were reported as frequency (percent) and continuous data as mean (standard deviation [SD]) or median (interquartile range [IQR]) as appropriate. For comparisons of categorical variables between groups, Chi-square tests or Fisher’s exact test as dictated by the distribution of the data were used. For the primary analysis, we compared mean MAP of patients while receiving vasopressors in the pre-trial period and the during-trial period (including the non-enrolled and usual care control groups) using a Student T-Test. In a secondary analysis, using a multivariable linear regression model, the effect of trial initiation on mean MAP was measured and adjusted for the following prespecified independent variables chosen in function of their plausible impact on vasopressor management: age, chronic hypertension, APACHE II score, site, and time from trial initiation to hospital admission of the patient at each site. These variables were introduced simultaneously in the model. Sensitivity analyses, both adjusted and unadjusted, compared mean MAP values while receiving vasopressors across the three groups (i.e. pre-trial, non-enrolled, usual care controls).

Two sides p values less than 0.05 were considered statistically significant. We used SAS 9.4 (SAS Institute, Cary, NC, USA) for all analyses.

Results

Patients

Overall, 429 patients fulfilled the eligibility criteria and were included in this nested study: 74 from the trial’s usual care control group, 200 in the pre-trial group, and 155 in the non-enrolled group (Fig. 1, Supplementary Table 1). Agreement between reviewers for selection of patients in the pre-trial and non-enrolled groups was good (weighted kappa = 0.75, 95% confidence interval = 0.55–0.95) [19]. We initially identified a third party in case of disagreements but these were finally all resolved by consensus.

Baseline characteristics and clinical outcomes

Table 2 presents patient characteristics by period and by group. No differences were noted for age (mean, pre-trial period: 74.5 [7.2] vs. during-trial period: 75.2 [6.9] years; p = 0.28) and APACHE II score (median, pre-trial period: 26 [20–31] vs during-trial period: 26 [21–31]; p = 0.47). During the trial period, men outnumbered women (during-trial period: n = 148 [64.6%] vs pre-trial period: n = 110 [55.0%]; p = 0.042), more patients received invasive ventilation (during-trial period: n = 144 [62.9%] vs pre-trial period: n = 102 [51%]; p = 0.013), and fewer received non-invasive ventilation (during-trial period: n = 50 [21.8%] vs pre-trial period: n = 75 [37.5%]; p = 0.0004). The mean MAP at the begining of data collection was similar between the two periods (mean, pre-trial period: 68.2 [13.7] vs during-trial period: 70 [13.1] mmHg; p = 0.16).

Table 2 Baseline Characteristics
Full size table

Clinical outcomes were similar across study periods and groups, including hospital mortality (pre-trial period: n = 75 [37.5%] vs during-trial: n = 78 [34.1%]; p = 0.46) and hospital length of stay (median, pre-trial period: 13.7 [7.3–25.4] vs during-trial period: 12.9 [7.0–25.1] days; p = 0.46) (Table 3).

Table 3 Mean arterial pressure, vasopressor use, and clinical outcomes
Full size table

Mean MAP and vasopressor use among periods and groups

Mean MAP while receiving vasopressors was 72.5 (5.1) in the pre-trial period vs 72.4 (5.0) mmHg in the during-trial period (p = 0.76; Table 3). Figure 2 shows the overall mean MAP while receiving vasopressors by month relative to trial initiation across all sites. The mean prescribed target MAP (mean, pre-trial period: 64.7 [3.1] vs. during-trial period: 65.3 [3.7] mmHg; p = 0.08), duration of vasopressor therapy (mean, pre-trial period: 48.5 [44.3] vs. during-trial period: 43.6 [40.5] hours; p = 0.24), and total vasopressor dose (mean, pre-trial period: 60.1 [122] vs. during-trial period: 56.6 [104.9] mg norepinephrine equivalents; p = 0.75) were similar between periods. Comparing between 3 groups, we found no difference in mean MAP while receiving vasopressors (mean, pre-trial group: 72.5 [5.1] vs non-enrolled group: 72.3 [5.1], vs. usual care control group: 72.5 [4.9] mmHg; p = 0.92).

Fig. 2
figure 2

Mean arterial pressure over time. The x axis indicates the months from trial initiation. The y axis represents mean MAP. The dotted line indicates the trial initiation at each site, which was a different date at each site, and separates the pre-trial period, N = 200 (months − 13 to − 1) of the during-trial period, N = 229 (months 0–23). The number of patients included and the number of participating sites for each month are specified at the bottom of the figure

Full size image

Adjusted analyses did not reveal an association between trial initiation and mean MAP while receiving vasopressors (Table 4) but suggested that higher MAP values were achieved in one of the 5 participating sites.

Table 4 Multivariable linear regression evaluating association between MAP values while receiving vasopressors and trial initiation
Full size table

Discussion

In this nested observational study, the initiation of a trial of permissive hypotension to reduce vasopressor exposure did not impact the MAP achieved at the participating sites, reflecting stability of the net effect of usual care processes before and during the trial. Reasurance that usual care received by patients in the control arm of the trial was similar to usual care before trial initiation and also to usual care as delivered to patients who were not enrolled but treated during the trial will enrich the interpretation of the trial results. The approach consisting of ascertaining potential fluxes in usual care over time and across concurrent control groups would be applicable to other research settings and may help readers of clinical trial reports evaluate the applicability of future trial results.Despite methodologic commentary on the importance of characterizing usual care when compared to an experimental intervention, we were unable to find any study measuring the impact of the initiation of a randomized controlled trial on usual care [1,2,3,4, 7].

Multivariable analysis showed that trial initiation was not associated with change in achieved MAP, but when the post-trial period was divided into patients randomized to usual care in OVATION-65 and non-enrolled patients, the association between usual care control group in OVATION-65 (vs. pre-trial period) and higher achieved MAP was almost statistically significant. This finding may be related to chance, due to a smaller sample size in the usual care control group and greater potential for influential outliers. Alternatively, the finding may reflect residual confounding from differences in severity of illness, or potentially an influence of trial initiation on care practices. However, the direction of effect, if related to the initation of OVATION-65, is opposite to our concern that the trial would have led to lower achieved MAP. Therefore, spurious finding or residual confounding are more likely explanations.

The study exhibits the following strengths. The sample size was sufficiently large to discern differences in the continuous outcome of MAP. We planned a one-month washout period separating the pre-trial and during-trial periods to minimize contamination between these two periods. Data collection that purposefully spanned winter and summer months, to mitigate the potential effects of seasonal case mix variations is also an important strength. In addition, the fact that mean MAP values were consistent with the results of a previously reported observational study published in 2017 reinforces the plausibility of these contemporaneous observations [13].

This study also has limitations. Although patients across study periods presented similar characteristics, small differences in the distribution of certain comorbidities, such as coronary disease, heart failure, and stroke, were observed and may be owed to the relatively small sample size. Mean MAP was numerically lower in the pre-trial and non-enrolled groups, likely because data collection for those patients started from vasopressor initiation. In contrast, for patients in the trial’s usual care control group, data collection started after randomization, at which time patients had already been partially stabilized. In the multivariable linear regression analyses, a higher severity of illness was associated with a lower mean MAP while receiving vasopressors, which may reflect a more severe cardiovascular compromise. The fact that MAP values varied by site reflects fluctuations in local cultures that have been previously described and is common when the certainty of evidence guiding clinical care is low and subject to interpretation [20]. One site contributed 46% of the data for this analysis, which may have masked the impact of trial initiation at other sites. Moreover, two of seven trial sites could not participate to this nested study, and of the five participating sites, one did not enroll patients in the trial. Although this limitation justifies some caution before concluding that the results apply equally across sites, the amount of data collected at each site appropriately reflects the number of patients enrolled from each site in the original trial. Patients in the trial were similar, but not identical, to those identified retrospectively, given the impossibility of retrospectively operationalizing the trial critierion of expected future duration of vasopressor therapy of at least 6 h. Finally, the mean arterial pressure values recorded in this study were similar to those measured in previous observational studies [13, 20], suggesting that foreknowledge of the launch of OVATION-65 was unlikely to have influenced practice in these centres.

Conclusion

The initiation of an unblinded trial comparing a permissive hypotension strategy to usual care in critically ill patients did not impact usual care in the centres where the trial was conducted. The approach consisting of ascertaining potential fluxes in usual care over time and across concurrent control groups would be applicable to other research settings and may alleviate concerns regarding potential biases or permit quantification of said biases.