Background

In clinical practice, adherence is defined as the extent to which a person’s behaviour corresponds with the recommendations made by a healthcare provider [1, 2]. In a research setting, non-adherence hinders the feasibility of clinical trials because patients who do not receive the intended intervention will not be affected by it [3]. In addition, non-adherence may bias trial results because patients who do not adhere to study protocols may have an inherently different prognosis compared to those who comply with the protocol [4]. Accordingly, variations in how protocol adherence is monitored and reported may bear on the interpretation of clinical trials [5,6,7].

The standard definition for protocol adherence is the proportion of all scheduled doses that are received [8]. This approach, however, is not applicable to interventions requiring continuous titration where a total number of scheduled doses is irrelevant. With interventions like continuous infusions of vasopressors and insulin, as well as mechanical ventilation, there exist no finite number of scheduled doses. In this article, we propose an approach to measure and report adherence in trials of continuous interventions titrated by bedside teams. This approach, informed by a recent pilot trial, acknowledges clinically-justified deviations and contextualizes protocol adherence into a pragmatic framework.

Discussion

Case study – The OVATION pilot trial

The OVATION pilot trial was an open-label randomized controlled trial of higher versus lower mean arterial pressure (MAP) targets for vasopressor therapy in shock [9]. The trial interventions were developed with input from clinicians and, with Research Ethics Board approval at each of 11 participating academic hospitals from Canada and the United States, we randomly assigned critically ill patients with vasodilatory shock to a lower (60–65 mmHg) versus higher (75–80 mmHg) MAP target. The primary feasibility objective of the trial was to achieve a 5 mmHg difference in MAP during vasopressor therapy between study groups. Intensive care unit (ICU) research personnel screened and enrolled eligible patients and recorded MAP and vasopressor doses hourly for the duration of vasopressor therapy. This approach enabled the investigators to measure the time within, above, and below assigned target MAP ranges, and to identify whether adjustments to vasopressor infusions complied with the study protocol. From May 2013 to April 2014, 118 patients participated in the OVATION pilot trial and 117 received vasopressor infusions for 695 patient-days.

In this trial, we defined potential protocol deviations a priori as MAP values outside the targeted range for >4 consecutive hours during vasopressor therapy without synchronous and consistent adjustments of vasopressor doses. As such, out of range MAPs did not constitute protocol deviations when the bedside team titrated vasopressors as required (i.e., increased dose if MAP was below the prescribed range or decreased dose if MAP was too high during the 4-h window). This 4-h time window was proposed by the steering committee and approved by all OVATION trial investigators. Moreover, we adjudicated each potential protocol deviation to determine if these events were clinically justified (Additional file 1). This approach was in sharp contrast to the mere reporting of MAP values at selected time points. Herein, we discuss four advantages of this contextual measurement and reporting of protocol adherence in clinical trials of interventions that require continuous titration.

Reason 1: Between-arm separation is a robust measure of adherence to complex protocols.

While protocol deviations may occur with simple interventions, they are more likely to occur in complex interventions. Thus, interventions requiring frequent or continuous titration present the greatest challenge, since protocol deviations may occur at any moment of the trial, in contrast to once or at most several times per day for most other interventions. When designing the OVATION pilot trial, it was not possible to specify an acceptable number of ‘missed doses’ for continuous vasopressor infusions. Instead, the extent of separation in MAP between arms was chosen as a crucial feasibility criterion to proceed to a larger trial. Accordingly, we prespecified what we considered the minimally acceptable separation based on clinical considerations, recognizing that even smaller differences (e.g. difference in MAP <5 mmHg) may be statistically significant but clinically unimportant. We observed a mean difference of 9 mmHg (95% confidence interval: 7 to 11 mmHg; p < 0.01) in average MAP between arms on days with vasopressors administered, thus exceeding the predefined threshold for feasibility of 5 mmHg.

Reason 2: Adherence assessed by protocol deviations varies in function of the definition of deviations and the frequency of measurements.

The ‘80% rule’ refers to the proportion of all scheduled doses that are received and is often cited as the cutoff for acceptable adherence for pharmacological interventions [10]. As mentioned above, with continuous interventions the number of scheduled doses is not finite. As the frequency of monitoring increases, so do the resources required to track protocol adherence. Close monitoring of protocol deviations is unlikely in trials that lack resources, introducing a risk of ascertainment bias. In contrast, with very frequent monitoring of continuous interventions, the likelihood of recording protocol deviations increases but the clinical impact of each deviation diminishes proportionately. Similarly, criteria for protocol deviations that are too sensitive would exaggerate the impact of protocol deviations beyond what is clinically important and unduly undermine the apparent feasibility and internal validity of the trial.

In the OVATION pilot trial, we considered reporting non-adherence as the proportion of hourly MAP values that were out-of-range during vasopressor therapy (Table 1a). This approach was dismissed on the basis that brief fluctuations in MAP values, randomly distributed above and below the targeted range, would not capture meaningful protocol deviations. Instead, we defined potential deviations as MAP values continuously above or below the targeted range for four consecutive hours (Table 1b). Furthermore, acknowledging that clinicians might adhere to the protocol but the patients fail to respond to the intervention, adjustments to vasopressor infusions were incorporated into the definition for protocol deviations. We considered that blood pressure alone was a poor marker of protocol adherence and that compliance with medication delivery was equally important.

Table 1 Protocol adherence according to different definitions
Full size table

Reason 3: Distinguishing clinically-justified vs. not clinically-justified protocol deviations acknowledges clinically sensible bedside decision-making and offers a clear terminology before the trial begins.

The existing literature on protocol adherence focuses mostly on outpatient interventions [2, 11]. In this context, efforts to maximize adherence is directed at study participants. In other settings, such as critical care, protocol deviations do not reflect the behavior of study participants but of the bedside team. During the OVATION pilot trial, a protocol adherence adjudication committee reviewed 164 database-generated protocol adherence alerts, representing 94 unique events. Figure 1 and Table 1c illustrate the classification of protocol adherence alerts. After adjudication, 75 of 94 unique protocol deviation alerts (80%) resulted from situations where the research team could ascertain no justifiable reason for the alert (i.e. these protocol deviations were adjudicated as not clinically-justified). In contrast, 19 protocol adherence alerts corresponded to scenarios where overruling the protocol was “admissible” as per predefined criteria (i.e., clinically-justified protocol deviation). When an unforeseen clinical situation justified the interruption of the trial intervention, we amended the protocol and did not report the event as a protocol deviation, judging these situations to be concordant with usual care [12]. For example, one patient developed acute aortic regurgitation after randomization. This unexpected event required a vasopressor dose reduction and a protocol amendment was prepared for the planned larger trial mandating that clinicians stop the assigned intervention in a similar clinical situation. Such careful assessment of protocol adherence in a pilot study can be used to characterize usual care and provide other useful insights into the acceptability of the intervention. During the OVATION pilot trial, protocol deviations more commonly consisted of MAP values above study targets, even in the higher MAP arm. This was in sharp contrast with stated practices [13]. Thus, by adjudicating protocol deviations, we observed that ICU physicians and nurses appear to be more concerned by the potential risks of hypotension than by the potential risks of vasopressor doses required to achieve higher targets. A contextual analysis of protocol adherence raised the possibility that aggressive vasopressor use has become embedded in standard clinical practice.

Fig. 1
figure 1

Classification of protocol adherence alerts

Full size image

Reason 4: Multiple metrics exist to report protocol deviations.

The traditional approach captures the proportion of patients who receive an arbitrary number of scheduled doses. A thoughtful approach is needed to understand the range of reporting options available and to select a metric for reporting protocol deviations that align with the trial objectives. In addition to the criteria that define protocol deviations, the units of measurement also influence the measure of adherence. Choosing to report the number of protocol deviation events, the number of days with at least one deviation event or the number of patients who ever experienced one protocol deviation provides different information. In the OVATION pilot trial, adherence was similar in both arms using the last definition, but was superior in the high MAP arm when we used the first two definitions (Table 1).

Building on previous work

Various terminologies describe the numerous issues related to protocol adherence. Ellis et al. have categorized non-adherence as ‘erratic adherence’, ‘unwitting non-adherence’ and ‘intelligent non-adherence’ [10]. In some trials, the cessation of a trial intervention prompted by such an event could be considered as a clinically sensible decision with no further comment, a protocol deviation, a protocol violation, or even a serious adverse event without explicit reference to protocol adherence. The distinction between clinically-justified deviations that are not avoidable from the trial perspective and those that are not clinically-justified can be useful to researchers and informed readers of the medical literature [14]. Central and, ideally, blinded adjudication of clinical events and associated protocol decisions can help make these distinctions.

Strengths and limitations

The strengths of this analysis include a priori definitions for protocol deviations, pilot trial feasibility criteria based on between-arm separation in MAP and total vasopressor dose, as well as parallel comparisons between various metrics. One limitation is that our observations are nested in a single, small clinical trial comparing complex critical care interventions and, thus, do not apply to all settings. Our intention was not to address every methodological issue that relates to protocol adherence. Rather, we used the OVATION pilot trial as a case study to illustrate that failure to contextualize the monitoring, reporting and interpretation of protocol adherence may lead to inappropriate conclusions. Another potential limitation is that there is no objective gold standard to determine the suitability of alternative approaches to the monitoring and reporting of protocol adherence. Admittedly, the definitions of protocol deviations in the OVATION pilot trial were arbitrary (e.g. 4-h time window). However, this is in keeping with every other measure of acceptable adherence, such as the 80% rule, which are typically well accepted. Our intention was not to present a superior definition for protocol adherence. Rather, the purpose of this analysis was to exemplify how trialists who are mindful of the context in which a complex research protocols are delivered may plan the monitoring of protocol adherence. While a clearer taxonomy of issues related to protocol adherence would undoubtedly be valuable to trialists and readers of the medical literature, this project was not designed to achieve this goal. Finally, our work focused on current issues related to protocol adherence. In the future, with the rapid development of real-time, data driven algorithms, the nomenclature for protocol adherence may yet become even more complex.

Conclusions

Metrics used for defining protocol deviations in a trial evaluating an intervention requiring continuous titration can significantly modify the interpretation of protocol adherence. Protocol deviations should be defined a priori and adapted to the complexity of the clinical context. With more complex interventions, an adjudication process to determine whether the deviations encountered were clinically justified may be warranted.