Naloxone is a life-saving opioid antagonist. Chronic pain guidelines recommend that physicians co-prescribe naloxone to patients at high risk for opioid overdose. However, clinical tools to efficiently identify patients who could benefit from naloxone are lacking.
To develop and validate an overdose predictive model which could be used in primary care settings to assess the need for naloxone.
Derivation site was an integrated health system in Colorado; validation site was a safety-net health system in Colorado.
We developed a predictive model in a cohort of 42,828 patients taking chronic opioid therapy and externally validated the model in 10,708 patients.
Potential predictors and outcomes (nonfatal pharmaceutical and heroin overdoses) were extracted from electronic health records. Fatal overdose outcomes were identified from state vital records. To match the approximate shelf-life of naloxone, we used Cox proportional hazards regression to model the 2-year risk of overdose. Calibration and discrimination were assessed.
A five-variable predictive model showed good calibration and discrimination (bootstrap-corrected c-statistic = 0.73, 95% confidence interval [CI] 0.69–0.78) in the derivation site, with sensitivity of 66.1% and specificity of 66.6%. In the validation site, the model showed good discrimination (c-statistic = 0.75, 95% CI 0.70–0.80) and less than ideal calibration, with sensitivity and specificity of 82.2% and 49.5%, respectively.
Among patients on chronic opioid therapy, the predictive model identified 66–82% of all subsequent opioid overdoses. This model is an efficient screening tool to identify patients who could benefit from naloxone to prevent overdose deaths. Population differences across the two sites limited calibration in the validation site.
Overdoses from pharmaceutical opioids and heroin are a significant public health concern.1,2 Fatalities due to pharmaceutical opioids increased nearly fourfold from 2000 to 2014, and recent data show that heroin overdose rates are rising.1 Since opioids are prescribed in an estimated one in five ambulatory visits for pain, targeted overdose-prevention interventions are needed.3
Naloxone is an effective medication used by medical personnel to reverse opioid-induced respiratory depression. In the 1990s, community-based organizations began distributing naloxone to people using heroin for lay bystander administration.4 The Food and Drug Administration recently approved new delivery devices for bystander naloxone administration, and national efforts have sought to expand access to naloxone.5,6,–7 In 2016, the Centers for Disease Control and Prevention (CDC) issued guidelines encouraging providers to co-prescribe naloxone to patients receiving chronic opioid therapy at high risk for overdose.6 Many states have also passed laws to permit naloxone standing orders, allowing people to acquire naloxone from pharmacies without a prescription.7
Despite these efforts, naloxone uptake appears to be low.8,9,–10 Cost is one significant barrier to expanded access. The Narcan nasal spray is $150 for a two-dose package, while the cost of two doses of the Evzio auto-injector increased from $690 in 2014 to $4500 in 2016.8 In addition to cost barriers, patients receiving chronic opioid therapy may not actively seek naloxone, because they underestimate their risk for overdose, are unaware of the benefits of naloxone, or fear that requesting naloxone will erode trust with their provider.11 Although clinicians can offer and prescribe naloxone to patients, they do not have the tools to efficiently identify patients at highest risk for overdose who would benefit from naloxone.12
Numerous studies have examined risk factors for opioid overdose using complex statistical models designed to establish causation.13,14,15,16,17,18,–19 In contrast, a prognostic model20 to guide clinical decision-making should be based on a limited set of common diagnoses and characteristics that can be easily collected while a provider is seeing the patient. This implies that the model cannot rely on excessively detailed clinical information, nor can it include future changes impacting risk, such as new diagnoses, discontinuation of opioids, or dose changes.
The objective of this study was to build and validate a prognostic statistical model that could be used by primary care providers and health systems to objectively and efficiently identify patients who are appropriate candidates for naloxone prescriptions at the point of care. We developed the model to accurately predict the risk of fatal and nonfatal opioid or heroin overdose within 2 years, the approximate shelf-life of naloxone.21 To maximize generalizability, we aimed to externally validate the model in a health system serving a different patient population.
Study Design, Study Sites, and Sources of Data
We developed and internally validated a prognostic model20 in a cohort of patients taking chronic opioid therapy enrolled in the Kaiser Permanente Colorado (KPCO) health plan. KPCO is a group model integrated health system that provides healthcare to approximately 670,000 members. KPCO operates its own pharmacies, and most opioid prescriptions are dispensed to members in these pharmacies.
The model was externally validated in a separate cohort receiving care at the Denver Health and Hospital Authority (DH). DH is a safety-net system serving more than 139,000 ethnically diverse patients. Every DH primary care facility has its own pharmacy.
Demographic and clinical data on patients in both cohorts were extracted from electronic health records (EHR) and pharmacy databases. These data included diagnoses (International Classification of Diseases [ICD]-9 codes) in outpatient, emergency department, and inpatient settings, and prescriptions dispensed in outpatient pharmacies.
Death data (death date and all recorded causes of deaths using ICD-10 codes) were obtained through linkage with state vital records. Social security number, name (including partial names), and birth date were used to identify and confirm matches.
The study was approved by the KPCO Institutional Review Board and the Colorado Multiple Institutional Review Board. Study data are protected by a federal Certificate of Confidentiality.
In both sites, we identified patients receiving chronic opioid therapy. First, we identified individuals who received one or more opioid prescriptions between January 1, 2006, and December 31, 2014. Among these patients, we identified those with opioid prescriptions dispensed on at least three different dates within 90 days, irrespective of the number of different opioid prescriptions on a given day. These represented cohorts of patients with incident chronic opioid therapy during the 8-year observation period. Among the first three dispensing dates in the 90-day period, the third dispensing date represented the index date for each patient. Opioids were identified using National Drug Codes and prescription dispensing dates from automated pharmacy files. Long-acting and extended-release (LA/ER, including methadone for pain) and short-acting opioids were included, but methadone dispensed from addiction clinics and all buprenorphine-containing products were excluded, as these are largely prescribed for addiction treatment. Patients prescribed chronic opioid therapy who also received methadone or buprenorphine for addiction treatment during follow-up were not excluded. Since many patients are exposed to opioids intermittently prior to starting chronic opioid therapy, patients who had been exposed to opioids prior to the index date were not excluded.
Patients were excluded if they had a cancer diagnosis or a do-not-resuscitate order in the year prior to the index date, or if they were younger than 18 on the index date. In KPCO, patients were also excluded if they were not continuously enrolled 365 days prior to the index date or if they did not have pharmacy coverage. Since DH does not have enrollment, we excluded patients who did not meet the definition of empanelment used in prior studies: at least one primary care visit in the 18 months prior to the index date.22,23
Patients were followed up to 2 years after the index date; follow-up was censored at the time of the opioid overdose, death from any cause, disenrollment from the health plan (KPCO) or more than 18 months since the last healthcare visit (DH), or September 30, 2015.
Opioid Exposure and Other Predictor Variables
We considered several known or suspected correlates of opioid overdose13,14,15,16,17,18,–19 as potential predictors, including demographics; medications dispensed—including opioid medications and their formulations (LA/ER)—tobacco use; and substance abuse/dependence, mental health, and chronic disease diagnoses (Table 1). Demographic variables were extracted from the EHR, and included age on index date, gender, race/ethnicity, and Medicaid coverage. Using automated pharmacy files, medication exposures included opioids, psychotropics, anticonvulsants, stimulants and muscle relaxants. For opioid exposures, we calculated the average daily morphine milligram equivalents (MME) for all opioid prescriptions on the index date or prior prescriptions whose days’ supply covered the index date.24 Dose and formulation variables were based on active prescriptions covering the index date (e.g., a person with a long-acting opioid prescribed 6 days before the index date in a 30-day quantity was classified as being on an LA/ER formulation). These variables were not included in the model as time-varying covariates, because clinicians would not be privy to information about future changes in dose and formulation when employing a prognostic model. History of opioids dispensed prior to meeting the definition of chronic opioid therapy was examined in the model.
Tobacco use was based on social history or a tobacco abuse/dependence diagnosis. Diagnoses such as hepatitis C infection were based on ICD–9 codes collected in routine clinical care based on patient history, laboratory results, and diagnostic procedures. Other than age and opioid dose and formulation, all variables were assessed in the 365 days prior to the index date.
Outcome: Opioid Overdose
Between January 1, 2006, and September 30, 2015, we identified fatal and nonfatal pharmaceutical opioid and heroin overdoses in emergency department and inpatient settings. We included unintentional, undetermined and suicidal overdoses, as all opioid-induced respiratory depression can be reversed with naloxone. Consistent with codes used or validated in prior overdose studies and injury surveillance guidelines, we used ICD-9 codes 965.0–965.02, 965.09, and E850.0–E850.2 for nonfatal overdoses; for fatal overdoses, we used ICD-10 underlying cause of death codes for drug poisoning (X40–X44, X60–X64, X85, Y10–Y14) and a contributing cause of death code indicating opioid involvement (T40.0–T40.4, T40.6).13,17,25,26,27,–28 Overdoses were captured within 2 years of the index date, as the shelf-life of naloxone is approximately 2 years.21
Predictive Model Development and Internal Validity
In the derivation (KPCO) cohort, we used Cox proportional hazards regression to examine associations between baseline predictor variables and incident opioid overdose. Prior to building the model, we examined frequencies and distributions for potential predictor variables to identify missing values, sparse categories, and extreme values. If cause of death was unknown, the patient was excluded from the analysis, since some of these deaths may have been opioid-related.
To avoid overfitting the model, we restricted the number of potential predictors and required at least ten overdoses per degree of freedom. We first grouped medications and diagnoses into broad, clinically meaningful categories so that the model could be used in a typical primary care visit. We then examined the bivariable associations between potential predictors and overdose risk. Guided by statistical and clinical significance, we chose nine predictors for the multivariable model. With backward elimination, we sequentially removed variables with a P value >0.05 from the model. Backward elimination is the preferred choice for variable selection in predictive modeling.20 Continuous variables were tested using simple parametric forms (a single linear term, low-order polynomials, logarithmic) and restricted cubic splines; categorical variables were modeled using indicator variables. We evaluated proportional hazards assumptions using Schoenfeld residual plots and a global goodness-of-fit test.
We evaluated the final model’s internal validity at the derivation site by examining discrimination and calibration.29 Using the Harrell bootstrap resampling,30 we calculated the bootstrap-corrected concordance statistic (c-statistic) to assess discrimination (the extent to which patients who were predicted to be at higher risk exhibit higher overdose rates relative to those predicted to be at lower risk). Calibration (the extent to which the predicted overdose risk agrees with the observed risk) was evaluated with the Greenwood-D’Agostino-Nam test.31 Discrimination and calibration were assessed graphically by dividing the cohort into tertiles based on the predicted risk, and then plotting the mean predicted cumulative risk with the Kaplan-Meier observed cumulative risk for each tertile.32 Widely separated Kaplan-Meier cumulative risk curves indicate better discrimination; agreement between predicted and observed risk over the follow-up indicate better calibration.33 In addition, we calculated the D-statistic to further assess discrimination and the R2 statistic on the basis of the D-statistic to quantify explained variation.33, 34, 35
To improve the model’s clinical utility, the regression coefficients were translated into a point-based risk score, with more points indicating higher overdose risk.36
The external validity of the model was tested in an independent cohort of DH patients (validation cohort) by examining discrimination and calibration.33 We first calculated the 2-year predicted overdose risk for patients in the validation cohort using coefficients for the linear predictor and baseline survival from the derivation model. We then calculated the c-statistic to assess discrimination. The validation cohort was divided into tertiles based on the predicted risk from the derivation model, and discrimination and calibration were assessed graphically. Calibration was further assessed with the Greenwood-D’Agostino-Nam test.31 The D-statistic and R2 statistic were also calculated.33, 34, 35
All analyses were conducted using R statistical software (R Foundation for Statistical Computing, Vienna, Austria) and SAS version 9.4 (SAS Institute Inc., Cary, NC).
Derivation and Internal Validation of Risk Model
At the derivation site, we identified 58,162 adult patients who received chronic opioid therapy and who met enrollment and pharmacy coverage criteria between January 1, 2006, and December 31, 2014. Of these, 15,270 (26.3%) were excluded because they had a diagnosis of cancer or a do-not-resuscitate order. Another 64 (0.1%) patients were excluded because they died of unknown causes, some of which may have been opioid-related. In the cohort of 42,828 patients, there were 110 incident nonfatal opioid overdoses, of which five were from heroin, and 11 fatal opioid overdoses. Among the 42,828 patients, 36 (0.08%) had a prior overdose in the year before the index date. The 2-year Kaplan-Meier overdose risk was 0.33% (95% confidence interval [CI] 0.28–0.40). Baseline characteristics and opioid prescription types covering the index date are displayed in Table 2 and eTable 1 (online), respectively. Overall, the average number of opioid prescriptions over the follow-up was 7.88 (standard deviation [SD] = 9.97). In the year prior to the index date, 40.6% the of cohort had received prior intermittent opioid prescriptions. Among those who received prior prescriptions, the mean number of prescriptions was 2.28 (SD = 1.78).
The following nine variables were selected as potential overdose predictors: age; daily opioid dose; LA/ER opioid prescription; psychotropic prescription; history of opioid prescription; tobacco use; and mental health, substance abuse/dependence, and hepatitis C diagnoses (Table 3). Age showed a U-shaped relationship with overdose risk, with the lowest risk among individuals 55–65 years of age. Thus, a quadratic term for age was included to improve model fit. The final model included five of these variables (age; age-squared; LA/ER formulation; tobacco use; and mental health and substance abuse/dependence diagnoses, Table 3). Schoenfeld residual plots and the global goodness-of-fit test (P = 0.46) indicated that the proportional hazards assumption was met.
The model effectively discriminated between high- and low-risk patients (Table 4), with an uncorrected c-statistic of 0.75 (95% CI 0.70–0.79) and a bootstrap-corrected c-statistic of 0.73 (95% CI 0.69–0.78). The D-statistic was 1.56 (95% CI 1.28–1.85), and the R2 was 36.8% (95% CI 28.04–44.85%). Patients were divided into tertiles according to their predicted overdose risk, and observed and predicted risks were graphed over 2 years (Fig. 1, derivation cohort). The graph demonstrates good discrimination, with the curve for the highest risk tertile well separated from the two lower risk tertiles. The graph also demonstrates effective calibration, since observed and predicted risks show good agreement over time. The Greenwood-D’Agostino-Nam test (P = 0.22) indicated that the model was well-calibrated. When risk was dichotomized into high (third tertile) versus lower (first and second tertiles) risk, the model had sensitivity of 66.1% and specificity of 66.6%. The positive predictive value for the model was 0.56%.
External Validation of Risk Model
At the validation site, we identified a cohort of 12,876 adult patients receiving chronic opioid therapy who met empanelment criteria between January 1, 2006, and December 31, 2014. Of these, 2168 (16.8%) were excluded because they had a diagnosis of cancer or a do-not-resuscitate order. In the cohort of 10,708 patients, there were 65 incident nonfatal overdoses, 15 of which were from heroin, and 53 fatal overdoses, nine of which were from heroin. Among 10,708 patients, 23 (0.2%) had an overdose in the year prior to the index date. The 2-year Kaplan-Meier overdose risk was 1.2% (95% CI 0.97–1.4). The derivation and validation cohorts differed significantly by most baseline characteristics (Table 2). Opioid prescriptions are shown in eTable 1 (online). Utilization rates over the follow-up were similar between sites (eTable 2, online).
When the derivation model was used to predict overdose at the validation site, the c-statistic was 0.75 (95% CI 0.70–0.80), indicating good discrimination. Model calibration, in contrast, was relatively poor. The Greenwood-D’Agostino-Nam test was statistically significant (P < 0.001); Figure 1 (validation cohort) demonstrates that the model under-predicted overdose. When the third tertile was considered high-risk (vs. lower-risk), the model’s sensitivity and specificity were 82.2% and 49.5%, respectively. The positive predictive value was 1.8%. The D-statistic was 1.53 (95% CI 1.24–1.82), and the R2 was 35.8% (95% CI 26.83–44.09%).
The prognostic index was converted to a points-based scale (Table 5) to allow providers to classify patients as low- (≤ 85), medium- (86–103), or high-risk (≥ 104) by summing the points based on patient characteristics. The cut points are based on risk tertiles. As an example, a 50-year-old (84 points for age) with a diagnosis of a substance use disorder (22 points) in the last year would have a score of 106, indicating high 2-year opioid overdose risk.
We developed a prognostic model to identify patients on chronic opioid therapy at high risk for overdose. The model comprised five variables that were readily available from the EHR in two diverse health systems. Although calibration was less than ideal in the validation cohort, the model demonstrated good internal validity and discriminated between patients with and without overdose in both health systems. At the point of care, providers could use the model to risk-stratify patients for naloxone prescribing.
Since naloxone is a safe and effective medication that prevents overdose death, universal prescribing to all patients on chronic opioid therapy could have a significant public health impact. At present, however, evidence supporting the effectiveness of such practice is limited. In addition, due to rising costs,8 universal prescribing may not be feasible in many health systems. Further, clinicians must inform patients of their risk, describe how to recognize an overdose, explain how to use naloxone, and write the prescription. Prior studies suggest these steps may be a barrier to universal naloxone prescribing given brief primary care visits and competing preventive and chronic care needs,12 such as developing alternative pain management strategies and reducing exposure to opioids.
Our results corroborate the findings of prior overdose risk studies among patients prescribed chronic opioid therapy.13,15 The annual overdose incidence rate in our derivation cohort was approximately 167.4/100,000 person-years, and patients taking LA/ER formulations had a significantly higher overdose risk than patients taking short-acting opioids. Moreover, only 3% of patients were initiated on treatment with LA/ER formulations, reflecting CDC prescribing guidelines recommending that physicians start with short-acting opioids.6
Current national policies and guidelines for opioid prescribing practices highlight dose as an important indicator of opioid overdose risk.6,13 In our study, however, opioid dose did not remain in the final model. Dose is likely most predictive of overdose events shortly after a prescription.13 Since our model was designed to predict overdose events for up to 2 years after a prescription, dose may have fluctuated significantly over the follow-up, rendering certain stable variables more predictive of longer-term overdoses. In our model, mental health diagnoses, substance use disorders, LA/ER opioid formulations, and tobacco use were stronger overdose predictors. These results suggest that focusing primarily on dose may not be the best approach for targeting preventive clinical interventions which confer benefit over longer periods.
Prescription or illicit opioid use may lead to overdose for a range of reasons, including prescribing error, suicide, addiction, or risk behavior. While separate models could be developed and tested for each of these pathways to overdose, it is not practical to employ multiple overdose risk models in primary care. Since naloxone is effective regardless of the cause or intent of the opioid poisoning, we employed a broad overdose outcome definition.
In contrast to prior risk models,37,38 we built a parsimonious model based on information readily available in the EHR and externally validated the model. While discrimination was good at the external site, the model’s calibration was less than ideal. This limitation can be attributed to the significant case mix differences between the two cohorts (Table 2). The baseline overdose rate was significantly lower in the derivation cohort, and may in part explain why the model under-predicted overdose risk in the validation cohort. In addition, the ratio of nonfatal to fatal overdose was considerably higher in the derivation cohort than in the validation cohort, suggesting that nonfatal overdoses may have been under-ascertained in the validation cohort. While these factors may have negatively affected the model’s overall performance, the model was also associated with sensitivity of greater than 80% in the validation cohort. This latter result suggests that a large majority of the DH patients at risk for overdose would have been targeted for naloxone prescription, which is appropriate given its favorable risk–benefit profile.39
As described above, the intent of the model was to risk-stratify patients prescribed chronic opioid therapy into groups that would be indicated for naloxone. In addition to guiding providers, the model could be used by health systems to target wide-scale naloxone interventions. However, it may not be advisable to use the model for applications with substantially different risk–benefit profiles. For instance, due to its miscalibration, the model should not be used directly by patients to determine their individual overdose risk. In addition, because of the low positive predictive value of the model, most patients indicated for naloxone will not experience an overdose. Therefore, physicians should not use our model to, for example, decide whom to taper off opioid therapy, which may have unintended adverse effects. For such decisions, physicians should consider other factors that may affect the risk–benefit ratio of tapering, including pain, risk of withdrawal, and risk of transitioning to heroin use.40,41,42,43,44,–45
Amidst the current opioid epidemic, we developed a five-variable model using readily available EHR data to identify high-risk patients who would benefit from naloxone to prevent deaths. While the model needs further evaluation, it was able to predict 66–82% of opioid overdoses in two disparate patient populations.
Rudd RA, Seth P, David F, Scholl L. Increases in Drug and Opioid-Involved Overdose Deaths - United States, 2010-2015. MMWR Morb Mortal Wkly Rep. 2016;65(5051):1445-52. doi:https://doi.org/10.15585/mmwr.mm655051e1
Rudd RA, Paulozzi LJ, Bauer MJ, et al. Increases in heroin overdose deaths - 28 States, 2010 to 2012. MMWR Morb Mortal Wkly Rep 2014;63(39):849-54.
Daubresse M, Chang HY, Yu Y, et al. Ambulatory diagnosis and treatment of nonmalignant pain in the United States, 2000-2010. Med Care 2013;51(10):870-8. doi:https://doi.org/10.1097/MLR.0b013e3182a95d86
Wheeler E, Jones TS, Gilbert MK, Davidson PJ, Centers for Disease Control and Prevention. Opioid Overdose Prevention Programs Providing Naloxone to Laypersons - United States, 2014. MMWR Morb Mortal Wkly Rep 2015;64(23):631-5.
Oliva EM, Nevedal AL, Lewis ET, et al. Patient Perspectives on an Opioid Overdose Education and Naloxone Distribution Program in the US Department of Veterans Affairs. Subst Abus 2016;37(1).
Dowell D, Haegerich TM, Chou R. CDC Guideline for Prescribing Opioids for Chronic Pain - United States, 2016. MMWR Recomm Rep. 2016;65(1):1–49. doi:https://doi.org/10.15585/mmwr.rr6501e1
The Network for Public Health Law. Legal Interventions to Reduce Overdose Mortality: Naloxone Access and Overdose Good Samaritan Laws. PDF. 2017 July. https://www.networkforphl.org/_asset/qz5pvn/network-naloxone-10-4.pdf. Accessed Nov 7, 2017.
Gupta R, Shah ND, Ross JS. The Rising Price of Naloxone — Risks to Efforts to Stem Overdose Deaths. N Engl J Med 2016;375(23):2213-5. doi:https://doi.org/10.1056/NEJMp1609578
Coffin PO, Behar E, Rowe C, et al. Nonrandomized Intervention Study of Naloxone Coprescription for Primary Care Patients Receiving Long-Term Opioid Therapy for Pain. Ann Intern Med 2016;165(4):245-52. doi:https://doi.org/10.7326/m15-2771
Exploring Naloxone Uptake and Use – A Public Meeting. 2015, July 1–2. U.S. Food and Drug Administration, White Oak Campus, Silver Spring, MD. https://www.fda.gov/drugs/newsevents/ucm442236.htm. Accessed Nov 7, 2017
Mueller SR, Koester S, Glanz JM, Gardner EM, Binswanger IA. Attitudes Toward Naloxone Prescribing in Clinical Settings: A Qualitative Study of Patients Prescribed High Dose Opioids for Chronic Non-Cancer Pain. J Gen Intern Med 2017;32(3):277-83. doi:https://doi.org/10.1007/s11606-016-3895-8
Binswanger IA, Koester S, Mueller SR, Gardner EM, Goddard K, Glanz JM. Overdose Education and Naloxone for Patients Prescribed Opioids in Primary Care: A Qualitative Study of Primary Care Staff. J Gen Intern Med 2015;30(12):1837-44. doi:https://doi.org/10.1007/s11606-015-3394-3
Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med 2010;152(2):85-92. doi: https://doi.org/10.1059/0003-4819-152-2-201001190-00006
Liang Y, Goros MW, Turner BJ. Drug Overdose: Differing Risk Models for Women and Men among Opioid Users with Non-Cancer Pain. Pain Med 2016;17(12):2268-79. doi:https://doi.org/10.1093/pm/pnw071
Miller M, Barber CW, Leatherman S, et al. Prescription opioid duration of action and the risk of unintentional overdose among patients receiving opioid therapy. JAMA Intern Med 2015;175(4):608-15. doi:https://doi.org/10.1001/jamainternmed.2014.8071
Bohnert AS, Ilgen MA, Ignacio RV, McCarthy JF, Valenstein M, Blow FC. Risk of death from accidental overdose associated with psychiatric and substance use disorders. Am J Psychiatry 2012;169(1):64-70. doi:https://doi.org/10.1176/appi.ajp.2011.10101476
Bohnert AS, Valenstein M, Bair MJ, et al. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA 2011;305(13):1315-21. doi:https://doi.org/10.1001/jama.2011.370
Bohnert AS, Logan JE, Ganoczy D, Dowell D. A Detailed Exploration Into the Association of Prescribed Opioid Dosage and Overdose Deaths Among Patients With Chronic Pain. Med Care 2016;54(5):435-41. doi:https://doi.org/10.1097/MLR.0000000000000505
Cochran G, Gordon AJ, Lo-Ciganic WH, et al. An Examination of Claims-based Predictors of Overdose from a Large Medicaid Program. Med Care 2017;55(3):291-8. doi:https://doi.org/10.1097/MLR.0000000000000676
Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162(1):W1-73. doi:https://doi.org/10.7326/M14-0698
Lenton SR, Hargreaves KM. Should we conduct a trial of distributing naloxone to heroin users for peer administration to prevent fatal overdose? Med J Aust 2000;173(5):260-3.
Hambidge SJ, Ross C, Shoup JA, et al. Integration of data from a safety net health care system into the Vaccine Safety Datalink. Vaccine 2017;35(9):1329-34. doi:https://doi.org/10.1016/j.vaccine.2017.01.027
Safety Net Medical Home Initiative. Empanelment: Establishing patient-provider relationships. 2013. http://www.safetynetmedicalhome.org/change-concepts/empanelment. Accessed November 6, 2017.
Von Korff M, Saunders K, Thomas Ray G, et al. De facto long-term opioid therapy for noncancer pain. Clin J Pain 2008;24(6):521-7. doi:https://doi.org/10.1097/AJP.0b013e318169d03b
Injury Prevention Workgroup 7. Consensus Recommendations for National and State Poisoning Surveillance. The Safe States Alliance. Atlanta, GA. April 2012.
Binswanger IA, Stern MF, Deyo RA, et al. Release from prison--a high risk of death for former inmates. N Engl J Med 2007;356(2):157-65.
Calcaterra S, Glanz J, Binswanger IA. National trends in pharmaceutical opioid related overdose deaths compared to other substance related overdose deaths: 1999-2009. Drug Alcohol Depend 2013;131(3):263-70. doi:https://doi.org/10.1016/j.drugalcdep.2012.11.018
Green CA, Perrin NA, Janoff SL, Campbell CI, Chilcoat HD, Coplan PM. Assessing the accuracy of opioid overdose and poisoning codes in diagnostic information from electronic health records, claims data, and death records. Pharmacoepidemiol Drug Saf 2017;26(5):509-17. doi:https://doi.org/10.1002/pds.4157
Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691-8.
Harrell Jr FE. Resampling, validating, describing, and simplifying the model. Regression Modeling Strategies. New York: Springer; 2001. p. 87-103.
Demler OV, Paynter NP, Cook NR. Tests of calibration and goodness-of-fit in the survival setting. Stat Med 2015;34(10):1659-80.
Lumley T, Kronmal RA, Cushman M, Manolio TA, Goldstein S. A stroke prediction score in the elderly: validation and Web-based application. J Clin Epidemiol 2002;55(2):129-36.
Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol 2013;13:33. doi:https://doi.org/10.1186/1471-2288-13-33
Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med 2004;23(5):723-48. doi:https://doi.org/10.1002/sim.1621
Royston P. Explained variation for survival models. Stata J 2006;6(1):83-96.
Sullivan LM, Massaro JM, D'Agostino RB. Presentation of multivariate data for clinical use: The Framingham Study risk score functions. Stat Med 2004;23(10):1631-60.
Zedler B, Xie L, Wang L, et al. Development of a Risk Index for Serious Prescription Opioid-Induced Respiratory Depression or Overdose in Veterans' Health Administration Patients. Pain Med 2015;16(8):1566-79. doi:https://doi.org/10.1111/pme.12777
Oliva EM, Bowe T, Tavakoli S, et al. Development and applications of the Veterans Health Administration's Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv 2017;14(1):34-49. doi:https://doi.org/10.1037/ser0000099
Wermeling DP. Review of naloxone safety for opioid overdose: practical considerations for new technology and expanded public access. Ther Adv Drug Saf 2015;6(1):20-31. doi:https://doi.org/10.1177/2042098614564776
Cicero TJ, Ellis MS, Surratt HL, Kurtz SP. The changing face of heroin use in the United States: a retrospective analysis of the past 50 years. JAMA Psychiatry 2014;71(7):821-6. doi:https://doi.org/10.1001/jamapsychiatry.2014.366
Cicero TJ, Ellis MS, Harney J. Shifting Patterns of Prescription Opioid and Heroin Abuse in the United States. N Engl J Med 2015;373(18):1789-90. doi:https://doi.org/10.1056/NEJMc1505541
Jones CM, Logan J, Gladden RM, Bohm MK. Vital Signs: Demographic and Substance Use Trends Among Heroin Users - United States, 2002-2013. MMWR Morb Mortal Wkly Rep 2015;64(26):719-25.
Peavy KM, Banta-Green CJ, Kingston S, Hanrahan M, Merrill JO, Coffin PO. "Hooked on" prescription-type opiates prior to using heroin: results from a survey of syringe exchange clients. J. Psychoactive Drugs 2012;44(3):259-65. doi:https://doi.org/10.1080/02791072.2012.704591
Jones CM. Heroin use and heroin use risk behaviors among nonmedical users of prescription opioid pain relievers – United States, 2002–2004 and 2008–2010. Drug Alcohol Depend 2013;132(1–2):95-100. doi:https://doi.org/10.1016/j.drugalcdep.2013.01.007
Lankenau SE, Teti M, Silva K, Jackson Bloom J, Harocopos A, Treese M. Initiation into prescription opioid misuse amongst young injection drug users. Int J Drug Policy 2012;23(1):37-44. doi: https://doi.org/10.1016/j.drugpo.2011.05.014
We wish to acknowledge the assistance of the following individuals: Ted Palen, MD, Kaiser Permanente Colorado, for medical record review; Joseph Frank, MD, MPH, University of Colorado School of Medicine, for feedback; Kirk Bol, MSPH, Steve Boylls, and Barbara Gabella, MSPH, Colorado Department of Public Health and the Environment, for vital statistics data and feedback. We affirm that we have acknowledged all significant contributions to this work.
Research reported in this publication was supported by the National Institute On Drug Abuse of the National Institutes of Health under award numbers R34DA035952-S1 and R01DA042059. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Preliminary data from earlier portions of this study were presented at the 39th Annual SGIM meeting in Hollywood, Florida, and the 2017 Addiction Health Services Research Meeting, Seattle, Washington.
Conflict of Interest
Dr. Ingrid Binswanger is currently employed by the Colorado Permanente Medical Group and receives royalties from UpToDate but does not have any additional conflict of interest. All remaining authors declare that they do not have a conflict of interest.
About this article
Cite this article
Glanz, J.M., Narwaney, K.J., Mueller, S.R. et al. Prediction Model for Two-Year Risk of Opioid Overdose Among Patients Prescribed Chronic Opioid Therapy. J GEN INTERN MED 33, 1646–1653 (2018). https://doi.org/10.1007/s11606-017-4288-3
- predictive model
- substance use disorder