INTRODUCTION

Growing concerns about risks associated with prescribed opioids have motivated efforts to improve opioid safety.1,2,3 These efforts have typically focused on all patients with certain characteristics associated with risk,4,5,6,7,8 including mental health diagnoses, substance use disorder, medical complexity,9,10,11,12,13 care coordination challenges,14 treatment disengagement,15 and interacting pharmacotherapies.16,17 While opioid safety initiatives can improve outcomes18,19, their effectiveness is limited by clinician time20,21 and they potentially invite undertreatment of pain 22,23 and/or disengagement from care. The proactive identification of patients at high risk of adverse outcomes from opioid prescriptions and the provision of targeted, interdisciplinary case review could address these challenges and promote improvement in patient outcomes.

In 2018, the Veterans Health Administration (VHA) mandated a case review intervention that targeted opioid analgesic prescribed patients at high risk of adverse outcomes.11 The national VHA policy governing the intervention required providers with expertise in the treatment of common key risk factors to collaborate on a coherent treatment strategy to augment care for specific high-risk patients. Those patients were identified for providers in a clinical decision support system—the Stratification Tool for Opioid Risk Mitigation (STORM) (see Appendix A, B). Providers conducting case reviews were encouraged to use the STORM dashboard to evaluate each patient’s risk factors and determine the need for treatment plan revisions or care augmentation.

The intervention was implemented with a stepped-wedge cluster randomized design. Specifically, the case review mandate applied first to patients with the top 1% of predicted risk of experiencing an overdose or suicide-related event in the next year. Then, at different times at randomly selected medical centers, the threshold was raised to 5%. Our study exploits this randomized policy intervention to address the following three questions: (1) Does a risk-targeted interdisciplinary case review mandate have an impact on the probability of experiencing serious adverse events? (2) Does the mandate have an impact on all-cause mortality? And (3) does the mandate increase the probability that targeted patients receive an interdisciplinary case review?

METHODS

Intervention: STORM Dashboard and Case Review Program

STORM is a population management dashboard that uses predictive analytics to stratify patients according to risk for an overdose- or suicide-related event within the next year and provides decision support based on clinical practice guideline recommendations. The predictive algorithm has been previously validated, and incorporates multiple factors in modeling risk for adverse events, including mental health and substance use disorders, high-dose opioid prescriptions, prior adverse events, previous detoxification treatments, and emergency department encounters, among others11. STORM provides risk estimates, risk factor summaries, tracking of recommended risk mitigation interventions, and information to support care coordination across providers and medical centers.11 Absolute risk model estimates (presented as percentages) are augmented by a categorical rating of risk level as “very high,” “high,” “medium,” or “low.”

On March 8, 2018, VHA issued Notice 2018-08, which mandated development of interdisciplinary teams to conduct case reviews for all patients categorized as “very high” risk on the STORM dashboard (see Appendix A). At the time of Notice issue, “very high” risk was defined as having a predicted risk greater than a cutoff of 16.56%, equivalent to being in the top 1% of predicted risk scores nationally at the time of study design. The mandated case review required a review of a patient’s SAE risk, prescriptions, and risk mitigation strategies, and was documented in the medical record.

Trial Design

The STORM trial was a 23-month, stepped-wedge cluster randomized trial designed to evaluate the impact of the targeted interdisciplinary case review program. Two patient cohorts were created for this study: a treatment cohort who received “very high” risk categorization on the dashboard, and a control cohort who did not. To create the treatment cohort, VHA randomly assigned facilities to expand the “very high” risk designation threshold over time from a cutoff equivalent to within the top 1% nationally to a cutoff equivalent to within the top 5% nationally (i.e., an estimated risk greater than 6.09%).24

Randomization

All 140 VHA medical centers were included in the study and were allocated to one of two clusters (i.e., early versus late expansion of the “very high” risk cohort from top 1% to top 5%) using permuted block randomization.24 All medical centers were blinded to assignment and received the treatment by the end of the study (Figure 1). Each medical center entered the study on 4/18/2018 in the control condition and randomly switched to the treatment condition in two waves: 2/12/2019 (study month 11) or 8/13/2019 (study month 17). The last day of patient enrollment was 9/9/2019, but follow-up data collection continued for an additional 127 days until 3/15/2020, at which point the global SARS-Covid pandemic began altering VHA utilization patterns.25

Figure 1
figure 1

Stepped-wedge allocation of medical centers. Notes: Shaded cells represent treatment periods (5% threshold). White cells represent control periods (1% threshold). The trial start date was April 18, 2018. Step wedge 1 occurred on Feb 12, 2019 (month 11) and step wedge 2 occurred on August 13, 2019 (month 17). The last day of patient enrollment was Nov 9, 2019 (end of month 19). Data collection continued for 127 days until March 15, 2020 (month 23)

Full size image

Participants

Eligible participants were VHA patients with an active prescription for opioids at any point during the study timeframe with a predicted risk of an overdose or suicide-related event between the top 1 and 5% of all patients nationally. Participants with a risk score between the top 0.01% and 1% were excluded from the trial, as they were expected to benefit from treatment, were always designated “very high” risk, and thus never experienced variation in treatment over the course of the study.

The trial protocol was previously published by BMJ Open and registered at isrctn.com with registration number ISRCTN16012111.24 Randomized rollout of the risk-expansion occurred as part of national clinical operations program implementation, and evaluation of the randomized rollout was approved by the VA Boston Healthcare System IRB and R&D Committees (Protocol#3069; approved March 2017).

Outcomes and Data Collection

The primary outcomes of interest were incidence of nine different SAEs that may be impacted by opioid use within 127 days following the first day of “very high” risk designation. SAEs included motor vehicle accident, other accidents, accidental falls, opioid overdose, other drug overdose, sedative overdose, acetaminophen overdose, possible and confirmed suicide-related events, and opioid detoxification (see Appendix C for International Classification of Diseases (ICD)-10 codes, noting that accident and fall codes were filtered to exclude those where the patient’s own behavior was unlikely to contribute to risk). In addition, we included a measure to capture experiencing any of the nine SAEs.

Secondary study outcomes included a dichotomous indicator of case review completion and the number of risk mitigation strategies received following the first day of “very high” risk designation, all documented in a patient’s EHR. Risk mitigation strategies included receipt of a Naloxone kit, opioid informed consent, timely follow-up, timely drug screen, psychosocial assessment, psychosocial treatment, prescription drug monitoring program, suicide safety plan, substance use disorder treatment, and medication-assisted treatment.

The exploratory outcome was all-cause mortality, which was collected from the Corporate Data Warehouse (CDW) administrative mortality files. All outcome measures were censored at 127 days, death, or when a control patient’s facility switched from control to treatment condition (i.e., from the 1% to the 5% threshold—see Figure 1).

Sample Size

The target sample size was 140 medical centers with an average of 352 patients, or 49,280 patients in total. Assuming an intra-cluster correlation coefficient of 0.01 and an alpha of 0.05 provided at least 80% power to detect a relative difference of 15% between treatment and control groups (i.e., rate differences at least as large as 0.009). Sample size calculations were made using the Stata -steppedwedge- command and accounted for changes in the composition of the treatment group over time and clustering of observations by facility24,26.

Statistical Analysis

The analysis of a stepped-wedge randomized trial is conducted by comparing all data in steps before treatment (controls) with the steps after treatment.27 This approach leverages within-cluster and between-cluster information to avoid confounding the treatment effect with changes over time.28 Unlike in a simple randomized controlled trial, controlling for time in a stepped-wedge design is important to avoid confounding the treatment effect with secular trends.

Statistical analysis was based on the principle of intention to treat. A patient-level, binary logistic mixed model regression was used to estimate the impact of treatment on the likelihood of all outcomes. Time fixed effects (month indicators) were included in the model to adjust for national variations in outcomes over time throughout the study. Facility effects were included to control for time-invariant differences between medical centers. Ordinary least squares regression was used to model the number of risk mitigation strategies. All models controlled for individual baseline covariates, including socio-demographic characteristics (i.e., sex, race, age, marital status, homelessness) and co-morbidities (e.g., liver disease, hypothyroidism). A complete list of covariates is available in Appendix D.

A statistically significant estimated treatment effect indicates that the risk of experiencing an SAE or all-cause mortality was affected by inclusion in the “very high” risk cohort subject to mandated review. Standard errors were clustered at the facility level.29

As a preliminary check on randomization, control and treatment baseline patient characteristics were compared using standardized differences.30 All statistical analyses were performed using Stata 14.

Changes to Trial Protocol

Two important changes in methodology were made after trial commencement. First, our original protocol outlined our intention to use time-to-event Cox proportional hazards models.24 However, the proportional hazard assumption was violated by the differential impact of treatment on outcomes over time.31 Nevertheless, results from informal analyses using Cox models were similar to those presented in this manuscript (Appendix E). Second, we included one additional outcome in exploratory analyses: all-cause mortality. We examine mortality because it is universally recorded in our data and is not subject to administrative documentation bias inherent in the SAE records, which are only captured if a patient utilizes VHA facilities. No additional outcomes were examined beyond mortality.

Sensitivity Analyses

The design of this trial involved an open cohort, in which a substantial number of individuals were identified and designated as “very high” risk on the first day of the trial, and other participants became eligible over the course of the study.32 As such, many participants were exposed to both control and treatment conditions if their medical center transitioned from control to treatment during the 127 days of a patient’s follow-up window. In sensitivity analyses, participants who entered into the study during a control condition were allowed to contribute data to both treatment and control conditions if their medical center switched into the treatment condition and their underlying risk scores were still within treatment (1–5%) range. In sensitivity analyses, we expanded the cohort to include patients who had risk scores eligible for “very high” risk prior to the STORM trial start date. In additional analyses, to test for the possibility that effect estimates are confounded by time trends in outcomes, we test the sensitivity of our estimates to the inclusion and exclusion of different time periods over the 23-month study.

Role of the Funding Source

This work is supported by the VHA Office of Research and Development (HSR&D SDR 16-196; QUERI PEC 16-001).

RESULTS

Participant Characteristics

A total of 44,042 unique patients were included in the study (32,197 in the control condition and 11,845 in the treatment condition). The characteristics of patients and their distribution across control and treatment conditions are reported in Table 1. The standardized differences suggest no major imbalances among the control and treatment participants for sex, age, race, marital status, homelessness, number of comorbidities, or underlying predicted risk scores. Of the 28 individual comorbidities examined, only standardized differences in alcohol abuse were greater than 0.10.

Table 1 Baseline Characteristics of Patients in the STORM Stepped Wedge Cluster Randomized Controlled Trial in 140 VHA Medical Centers, 2018–2020
Full size table

Primary Outcome: Serious Adverse Events

The unadjusted prevalence of any SAE (composite) during the study period was 12.2% in both the control and treatment groups (Table 2). Documentation of the nine specific SAEs was relatively rare, ranging from 0.03% for acetaminophen overdose to 7.3% for accidental falls. Although there were small differences in the risk of Other Accidents and Suicide-Related Events in the unadjusted logistic models between control and treatment groups (Table 3), these differences disappeared in the fully adjusted model.

Table 2 Unadjusted rates of 10 outcomes in the STORM Stepped Wedge Cluster Randomized Controlled Trial in 140 VHA Medical Centers, 2018–2020
Full size table
Table 3 Results from regression analyses of serious adverse events and mortality outcomes in the STORM Stepped Wedge Cluster Randomized Controlled Trial in 140 VHA Medical Centers, 2018–2020
Full size table

Secondary Outcomes: Case Review and Number of Risk Mitigation Strategies

The overall rate of case review was 4.2% in the control group and 29.8% in the treatment group (Table 2). The unadjusted odds of receiving a case review were 9.7 in the treatment group relative to the control group (CI: 9.09–10.40), and 5.1 in the fully adjusted model (CI: 3.64–7.23). Adjusted linear regression models reveal that treatment patients received on average 0.5 (CI: 0.385–0.610) risk mitigation strategies more than control patients.

Exploratory Outcome: Mortality

The unadjusted mortality rate during the study period was 6.7% in the control group and 6.9% in the treatment group (Table 2). In the fully adjusted logistic regression model, the odds of all-cause mortality for treatment patients relative to control patients was 0.78 (95% CI: 0.65, 0.94) (Table 3). Unadjusted differences in odds ratios between treatment and control patients are reported in column 1.

Sensitivity Analysis

Findings from the primary specification were robust to cohort definition sensitivity analyses except for the relationship between treatment inclusion and sedative overdose, which was not statistically significant in the primary analysis. These findings are presented in Appendix F. Results remained directionally similar in analyses where the data were restricted to a subset of study month periods, presented in Appendix G.

DISCUSSION

To our knowledge, this is the first study to evaluate the impact of a predictive model-targeted prevention program on reducing adverse outcomes in high-risk patients on opioid analgesics, specifically by providing interdisciplinary case review guided by guideline-based clinical decision support. Using a stepped-wedge randomized design, we estimated the effectiveness of a VHA policy mandate for case reviews of patients identified as “very high” risk by the provider-facing STORM dashboard and its impact on SAEs and mortality among these patients.

We found that being identified as a “very high” risk (treatment) patient was associated with 22% lower odds of all-cause mortality relative to control patients. Mortality impacts of this magnitude are on par with medication interventions for common medications for heart disease prevention.33,34 Comparisons to studies of existing system-wide opioid safety initiatives are challenging, as prior studies typically lack the sample size needed to observe enough variation in mortality outcomes and may also lack access to administrative mortality data that can be linked to patient records during the study period.35,36

We found no statistically significant changes in the risk of having a medical record documented SAE. We also found that treatment patients were more than five times as likely to receive a case review and received an average of 0.5 more risk mitigation strategies relative to control patients. This leads to the hypothesis that a mechanism of reduced mortality risk could be receipt of care arising from case review or risk mitigation strategies. Additional hypothesized mechanisms include changes in utilization patterns or the probability of engaging in care at the VHA or changes in the patterns of opioid prescribing as a result of high-risk designation.

This study’s major contribution to understanding the impact of the targeted case review program depended on its stepped-wedge cluster randomized design. This design is particularly appropriate for studying interventions implemented in large health systems.37 Each facility acted as its own control and provided data in both the control and treatment stages. The stepped-wedges also provided the opportunity to control for the effects of time trends on mortality and SAEs.32

Limitations

Our study has several limitations. First, expansion of the cohort required for case review (i.e., treatment) strained clinician resources and exacerbated logistical challenges in many medical centers, triggering efforts to streamline and improve the case review process. The medical centers that expanded the cohort later in the trial may have learned efficiencies from experiences of colleagues in early treatment sites and expanded case review practices to a broader population before the required expansion. If true, contamination would bias the treatment effect sizes towards zero. A second limitation is that SAEs were only captured if documented in the VHA medical record.38 This recording issue would affect treatment and control patients equally, unless treatment patients are more likely to visit a VHA physician as a result of receiving a case review. This latter possibility may account for the null findings of treatment on SAEs, as the intervention is likely to increase detection and documentation of adverse events by VHA medical providers. By contrast, mortality is universally recorded in patient records available to our study.

A third limitation is the short time-frame for outcome assessment. Case reviews were expected to improve care coordination with a particular focus on behavioral health concerns. Clinical interventions to address substance misuse or maladaptive coping strategies, and facilitate engagement in recovery or rehabilitative activities, may produce delayed benefits that may not be apparent within the first 4 months. A fourth limitation is that predictive algorithms used to generate risk scores may change over time. Although the baseline risk scores between control and treatment groups in this study are nearly identical, it is possible that changes in model performance may influence true underlying risk over time and in different populations. Finally, we are unable to generalize our findings to patients in the highest risk thresholds (0.01–1%) as these groups never experienced variation in treatment assignment and were thus excluded from the trial. Future work should examine the impact of STORM for multiple patient groups across the distribution of risk scores.

CONCLUSIONS

In conclusion, this is the first national study to provide evidence from a randomized trial that a predictive model-targeted prevention program, focused on improving coordination of treatment among a complex patient population, had a substantial impact on mortality risk for patients prescribed opioids. Results suggest that combining risk identification via predictive modeling with clinical decision support-guided interdisciplinary case review can improve patient outcomes in a high-risk patient population. This trial demonstrates the potential for enhancement of care for high-risk patients and interdisciplinary care coordination approaches to help stem the national opioid crisis.