BACKGROUND

The USA is facing an unprecedented opioid epidemic. According to data from the 2015 National Survey of Drug Use and Health, over two million people had a prescription opioid use disorder.1 People who were uninsured, unemployed, or had lower family incomes reported higher rates of opioid use, misuse, or opioid use disorder.2 Opioid prescribing for chronic pain can be challenging to clinicians who have little training in addiction or managing patients who misuse their opioid medications.3 Risk factors for opioid misuse among people on chronic opioid therapy (COT) include a history of substance use disorder, younger age, increased healthcare utilization, and depression or anxiety.4, 5 Predictive tools identify patients at risk of aberrant drug-related behaviors6,7,8 and assist with diagnosing addiction in patients on COT.9 Tools to identify patients at risk of becoming chronic opioid users for both acute and chronic pain are lacking. This is particularly important in the hospital where opioids are commonly prescribed for pain.10 Opioid receipt at hospital discharge has been shown to be associated with an increased risk of chronic opioid use.11 Predictive tools to identify hospitalized patients at risk for future COT may have clinical utility to improve hospital-based pain management with a focus on limiting opioid prescribing when non-opioid analgesics, or other non-pharmaceutical options, may be effective for pain control.

There are several approaches to develop a predictive tool. Traditional models, such as logistic regression, have been used to identify risk factors for COT.5, 12,13,14 Modern methods for prediction include non-parametric and tree-based methods that can handle large amounts of data available in the electronic health record (EHR).15, 16 These methods are comparable to parametric approaches with respect to their prediction performance.16,17,18 This study aimed to identify a parsimonious predictive model of future COT among hospitalized patients not on COT in the 1 year preceding their hospitalization. We used EHR data from an urban, safety net hospital to develop and compare various algorithms with respect to their accuracy, sensitivity, and specificity to predict progression to COT one year following hospital discharge.

METHODS

Study Design and Setting

This was a retrospective cohort study of all hospital discharges from Denver Health Medical Center, an integrated safety net health system in Denver, Colorado, between 2008 and 2014. Patients accessed care at a 477-bed hospital, an emergency department, an urgent care center, community health centers, subspecialty clinics, and a public health department.19 The majority of patients had incomes < 185% of the Federal Poverty Level and 70% were ethnic minorities.20 This study was approved by the Colorado Multiple Institutional Review Board and adhered to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement on reporting predictive models.21

Data Source and Participants

All data were queried from the Denver Health data warehouse which pools demographic, pharmacy, laboratory data, and International Classification of Diseases Diagnosis and Procedural Codes, Ninth Revision (ICD-9 CM) obtained during patient care. Pharmacy data included inpatient opioid prescribing (dosing and type) and outpatient pharmacy data (Table 1). The first hospital discharge for each patient over the study period was categorized as the “index discharge.” From these patients, we excluded patients on COT (defined below) or opioid agonist therapy (methadone or buprenorphine/naloxone) in the 1 year preceding their index discharge. We excluded patients < 15 or > 85 years old, in prison, jail, or police custody, who died following their index discharge, had < 2 healthcare visits to Denver Health three years preceding their index discharge, were undocumented persons receiving emergent hemodialysis, or were obstetric patients. We excluded patients with < 2 healthcare visits to Denver Health, because they were less likely to receive follow-up health care in this system. We excluded incarcerated patients, because ongoing health care and medication dispensing occurs within the correctional system. We excluded obstetric patients, because they tended to be younger and healthier than the medical and surgical patients in the study sample and were not reflective of the overall patient population. Lastly, we excluded subsequent hospital discharges to ensure our dataset only included each patient’s first hospitalization during the study period. We did not exclude patients with malignancy diagnoses, because, increasingly, individuals with cancer are surviving to remission. These patients may develop chronic pain due to cancer burden, exposure to cancer treatment, or other medical comorbidities. While opioid therapy is widely accepted to manage malignancy-related pain, the use of opioids to manage chronic pain in patients who are cancer free following treatment is not grounded in broad consensus.22, 23

Table 1 Patient Demographic Characteristics and Distribution of Potential Predictors of Chronic Opioid Use

Outcome

The study outcome was COT one year following the index discharge. We defined COT as “a 90-day or greater supply of oral opioids with less than a 30-day gap in supply within a 180-day period or receipt of ≥10 opioid prescriptions over one year following the index discharge.”24, 25

Predictor Selection

Predictors were selected based on clinical experience and were informed by the literature.6,7,8,9, 11, 14, 26 We were interested in identifying modifiable factors associated with future COT related to the index discharge that were available in the EHR. Data were identified from previous encounters prior to the index discharge. We identified gender, race/ethnicity, age, and insurance status from registration data collected at the index hospitalization. Insurance status was classified as discount payment plan (Child Health Plan Plus; Colorado Indigent Care Program; Denver Health Financial Assistance Program), Medicare, Medicaid, commercial, or unknown/other/self-pay.

We obtained medical and mental health diagnoses and substance use disorders (alcohol, stimulant, and tobacco) by querying patient encounters in the three years preceding the index discharge using ICD-9 CM codes (Table 1, Online Appendix 1).27 From these diagnoses, we calculated a Charlson Comorbidity Index.28 Discharge diagnoses of acute pain, chronic pain, and neoplasm were reported (Table 1). Surgery during the index hospitalization was determined by ICD-9 CM procedural codes (Online Appendix 1). Cannabis use was not examined, because medical marijuana was legalized in Colorado in 2010, and was inconsistently documented in the EHR.29

We identified opioid analgesics, neuropathic agents, non-steroidal anti-inflammatory drugs (NSAIDs), other analgesics (topical capsaicin, lidocaine), tricyclic antidepressants, and benzodiazepines filled at Denver Health pharmacies in the one year preceding the index discharge (Online Appendix 2). Other data captured included opioid receipt within three days of hospital discharge, milligrams of morphine equivalents (MMEs) administered daily during the hospitalization, length of hospital stay, the number of healthcare encounters one year preceding the index discharge, and the number of healthcare encounters in the one year post index discharge.

Statistical Analysis and Model Development

Patients with and without chronic opioid use were compared with respect to their demographic characteristics using t tests for continuous variables, chi-squared or Fisher’s exact tests for categorical variables, and Cochran-Mantel-Haenszel tests for ordinal variables (Table 1). We examined the relationship between continuous predictors and the probability of COT using locally weighted scatterplot smoother (lowess).30 Variables examined included age, MMEs, Charlson Comorbidity Index, hospital length of stay (LOS), past year opioid and/or non-opioid analgesic receipt (NSAIDs, neuropathic agents, topical capsaicin, lidocaine), the number of healthcare encounters one year preceding the index discharge, and the number of healthcare encounters in the one year post index discharge. Age, MMEs, LOS, and healthcare encounters were non-linearly related to the probability of COT. The relationship between age and COT was quadratic. Hospital length of stay was loglinearly related to COT. Daily MMEs in the hospital and the numbers of past year opioid prescriptions filled preceding the index hospitalization were categorized into clinically significant groups to meaningfully interpret their relationship to COT.31, 32 The Charlson Comorbidity Index was linearly related to COT.

We compared the prediction performance of various binary classification algorithms to determine which method best predicted COT in this population. These algorithms included random forests, least absolute shrinkage and selection operator (lasso), and stepwise logistic regression. We used a temporal split of the data, where the models were trained on years 2008–2011 (65%) and tested on years 2012–2014 (35%). This method aligns with the Tripod prediction model Type 2B.16, 21 Due to the extreme imbalance in the outcome variable, COT (only 5% prevalence), we down-sampled the majority class (no COT) in the training data to create an equal number of COT versus no COT. To down-sample, we took a random sample of the majority class which was equal in size to the minority class. This resulted in 1,061 with COT and 1,061 with no COT. This method has been shown to improve prediction performance in imbalanced data.33 We compared the three algorithms (random forests, lasso, and stepwise logistic regression) with respect to accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and area under the receiver operating curve (AUC) by fitting them to the down-sampled training data and testing them on the hold-out set. All three performed similarly, with slightly higher prediction performance achieved by logistic regression (data not shown). We proceeded with logistic regression, because it performed well, it can be easily implemented in the EHR, and it provides interpretable associations between the explanatory variables and COT.

The multiple logistic regression model was used to estimate associations between the explanatory factors (Table 2) and COT and to predict COT in the test set. The best subset of factors was determined using stepwise selection, with a cutoff of ρ < 0.05. Although p < 0.05 would ordinarily be high for a sample size this large, because we first down-sampled the majority class, we felt that ρ < 0.05 was appropriate. An optimal cutoff for classifying the fitted probabilities from the logistic regression model was estimated using the Youden’s index.34 All analyses were performed in the R programming language.35

Table 2 Logistic Regression Parameter Estimates from the Model Selected Using Stepwise Regression on the Down-Sampled Training Set

RESULTS

From January 1, 2008 to December 31, 2014, there were 159,574 hospital admissions. After applying our exclusion criteria, 27,705 (17.4%) patients remained (Fig. 1). Of these patients, 1457 (5.3%) were on COT one year following their index discharge. Table 1 lists patient demographic and clinical characteristics at the index hospitalization and compares demographic and clinical characteristics of patients with and without chronic opioid use one year following discharge. Patients who progressed to COT were most frequently between the ages 45–54 years old (ρ < 0.0001). Future COT was associated with a history of tobacco use (ρ < 0.0001), a history of acute or chronic pain (ρ < 0.0001), a three-year history of a mental health diagnosis (ρ < 0.0001), and a discharge diagnosis of chronic pain (ρ < 0.0001). Patients who progressed to COT one year post discharge had a higher Charlson Comorbidity Index on hospital admission compared to patients who did not progress to COT (mean 2.4, [standard deviation (SD) 2.5] versus 1.9, [SD 2.2]); ρ < 0.0001). Receipt of opioids, benzodiazepines, NSAIDs, or neuropathic prescriptions one year preceding hospitalization was more common among patients with future COT (p < 0.0001) as was receipt of an opioid at discharge (ρ < 0.0001). Mean MMEs per hospital day was greater among patients who progressed to COT (64.4 mg [SD 76.7 mg)]) versus those who did not (36.2 mg [SD 64.4]) (ρ < 0.0001). Increasing length of hospital stay was also associated with future COT (p = 0.0003).

Fig. 1
figure 1

Flow diagram for study participants. See TIFF file attached.

In the logistic regression model, 13 variables were selected using stepwise selection (Table 2). Patients prescribed opioids at discharge had more than twice the odds of developing COT than those who were not prescribed opioids at discharge (adjusted odds ratio (AOR) 2.33, 95% confidence interval (CI) [1.78, 3.04], Table 2). Increasing MMEs prescribed per day during the index hospitalization was associated with increased odds of developing COT one year post hospital discharge (Table 2). Past year receipt of non-opioid analgesics (AOR 1.92, 95% CI [1.49, 2.48]) and past year receipt of a benzodiazepine (AOR 1.89, 95% CI [1.26, 2.82]) was associated with COT one year post discharge. Prior history of chronic pain (AOR 1.79, 95% CI [1.41, 2.26]) and number of subsequent hospitalizations post index discharge (AOR 1.51, 95% CI [1.39, 1.64) was also associated with future COT.

The multiple logistic regression model correctly predicted 79% of the COT patients and 78% of the no COT patients. The ideal cut point for the fitted probability when optimizing the AUC was 0.45. The accuracy of this model was 78%, and the AUC was 0.86. For comparison, we also fit a stepwise multiple logistic regression model to the full training dataset (before down-sampling the majority class) and used the Youden’s index to determine the ideal cut point, which was estimated to be 0.07. This produced slightly higher specificity (83%), slightly lower sensitivity (74%), and slightly higher accuracy (83%) than the model fit to the down-sampled data; however, the final model contained 20 variables (as compared to 13 in the down-sampled data). Therefore, we chose the model fit to the down-sampled data because it achieved higher sensitivity with fewer variables.

DISCUSSION

In this study we developed a statistical model for prediction of COT among hospitalized patients at an urban, safety net hospital. This model is unique from other opioid risk assessment tools,6, 8, 36 because it accessed data available in the EHR and did not require additional data gathering and documentation from clinicians or other healthcare providers. A predictive model created from EHR data, when incorporated into a clinical workflow, has the potential to rapidly identify high-risk patients and provide real-time alerts to clinicians for decision-making when prescribing opioids during hospitalization and discharge.

In the logistic regression model, noteworthy risk factors for COT included more than 10 mg of morphine equivalents prescribed per day during hospitalization, two or more opioid prescriptions filled in the year preceding the index hospitalization, past year receipt of non-analgesic pain medications, and past year receipt of benzodiazepines in the one year preceding the index hospitalization (Table 2). The association between past year opioid and non-opioid analgesic receipt with future COT has also been shown in patients undergoing bariatric surgery37 and total hip arthroplasty.38 Past year benzodiazepine receipt was also associated with an increased odds of COT. This is concerning, because coadministration of these medications produces a defined increase in rates of adverse events, overdose, and death.39 In our findings, Charlson Comorbidity Index was associated with COT. Patients with functional limitations and greater disease burden often have diagnoses known to be associated with chronic pain, including osteoarthritis,40 fibromyalgia,41 and low back pain.42 Surgery at the index hospitalization was not associated with COT. This is likely because acute surgical pain often resolves and opioids are not needed long term. Finally, opioid receipt at discharge was predictive of COT, an association which has been previously reported.11 This variable is modifiable, and clinicians should consider this relationship when prescribing opioids to high-risk patients at discharge.

This study demonstrates the benefit of using a common generalized linear model technique combined with sampling to access data available in the EHR to create a clinically relevant prediction model. When compared to other prediction models developed using large datasets and machine learning techniques to predict the development of diabetes,43 pancreatitis severity,44 heart failure readmissions,44, 45 and sepsis18 (sensitivities 0.74, 0.87, and 0.92, respectively, and AUCs 0.78, 0.82, and 0.86, respectively), results of our model are encouraging and could benefit clinical practice. While no prediction model has been published to identify hospitalized patients at high-risk of future COT, prediction tools to assess the patient’s risk of opioid misuse have been developed and validated. Such tools include the Screener and Opioid Assessment for Patients with Pain (SOAPP-R; sensitivity 0.81; AUC 0.81),36 the Current Opioid Misuse Measure (COMM; sensitivity 0.77; AUC 0.81),6 and the Opioid Risk Tool (ORT; c = 0.82).8 These tools have not been validated in the hospital setting and administration and scoring of these tools can be time consuming; thus, their feasibility of use in a busy hospital-based practice is limited.

This model addresses an area of medicine which is critically important—the long-term effect of opioid prescribing among hospitalized patients. Accessing electronically available data to develop and integrate prediction models into an EHR offers a promising, time-saving method to address the risk of future chronic opioid use in a fast-paced hospital practice. A predictive tool integrated into an EHR allows for a real-time screening to identify high-risk patients. Other EHR-based tools have reduced morbidity and mortality from thromboembolic disease, sepsis, and infections46,47,48,49,50,51 and illustrate the benefit of implementing predictive tools linked to the EHR to inform clinical practice.

LIMITATIONS

We were limited in creating our model based upon the data available in our EHR. There were inherent limitations to using administrative data which include variability in data collection, i.e., where, when, how, and by whom, as well as any policy changes in the hospital which may affect data collection. These limitations would contribute to under ascertainment bias. However, given the intent of creating a real-time predictive model, we believe using data available in the EHR was appropriate. In our model development, we included variables we felt were most likely to be associated with COT and we may have inadvertently left out other predictive variables. We were unable to capture patients who filled prescriptions at non-affiliated pharmacies or patients who used opioids without a prescription, which may cause selection bias for cohort categorization and under ascertainment bias for COT after one year. Our dataset comes from an urban, safety net healthcare system where the majority of patients are ethnic minorities and are insured by Medicaid; thus, these study results may not be generalizable to the general population. Finally, healthcare systems across the USA use a variety of EHRs which have a range of capabilities with varied practices in generating problems lists, accessing prescription data, and/or accessing health records from other healthcare institutions. This study accessed EHR data from one healthcare institution and may not be generalizable to an institution that utilizes different EHRs.

CONCLUSION

We demonstrated that a generalized linear model that corrects for imbalanced data can be used to predict future COT among hospitalized patients. Our model identified 79% of the future COT patients, which is much better than chance. The model could be easily integrated into the clinical workflow to alert physicians when a patient is at high-risk for COT. Early identification allows for targeted patient education and clinician prompts to modify pain management strategies and opioid prescribing when appropriate.