Skip to main content

Methodological challenges and proposed solutions for evaluating opioid policy effectiveness

Abstract

Opioid-related mortality increased by nearly 400% between 2000 and 2018. In response, federal, state, and local governments have enacted a heterogeneous collection of opioid-related policies in an effort to reverse the opioid crisis, producing a policy landscape that is both complex and dynamic. Correspondingly, there has been a rise in opioid-policy related evaluation studies, as policymakers and other stakeholders seek to understand which policies are most effective. In this paper, we provide an overview of methodological challenges facing opioid policy researchers when conducting opioid policy evaluation studies using observational data, as well as some potential solutions to those challenges. In particular, we discuss the following key challenges: (1) Obtaining high-quality opioid policy data; (2) Appropriately operationalizing and specifying opioid policies; (3) Obtaining high-quality opioid outcome data; (4) Addressing confounding due to systematic differences between policy and non-policy states; (5) Identifying heterogeneous policy effects across states, population subgroups, and time; (6) Disentangling effects of concurrent policies; and (7) Overcoming limited statistical power to detect policy effects afforded by commonly-used methods. We discuss each of these challenges and propose some ways forward to address them. Increasing the methodological rigor of opioid evaluation studies is imperative to identifying and implementing opioid policies that are most effective at reducing opioid-related harms.

This is a preview of subscription content, access via your institution.

References

  • Abadie, A., Diamond, A., Hainmueller, J.: Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J. Am. Stat. Assoc. 105, 493–505 (2010)

    CAS  Google Scholar 

  • Al Achkar, M., Grannis, S., Revere, D., MacKie, P., Howard, M., Gupta, S.: The effects of state rules on opioid prescribing in Indiana. BMC Health Serv. Res. 18, 29 (2018)

    PubMed  PubMed Central  Google Scholar 

  • Alpert, A., Powell, D., Pacula, R.L.: Supply-side drug policy in the presence of substitutes: evidence from the introduction of abuse-deterrent opioids. Am. Econ. J.-Econ. Policy 10, 1–35 (2018)

    Google Scholar 

  • Andraka-Christou, B., Rager, J.B., Brown-Podgorski, B., Silverman, R.D., Watson, D.P.: Pain clinic definitions in the medical literature and US State laws: an integrative systematic review and comparison. Subst. Abuse Treat Prev. Policy 13, 17 (2018)

    PubMed  PubMed Central  Google Scholar 

  • Athey, S., Imbens, G.W.: Design-based analysis in difference-in-differences settings with staggered adoption. National Bureau Of Economic Research. Working Paper 24963 (2018). http://www.nber.org/papers/w24963

  • Ayres, I., Jalal, A.: The impact of prescription drug monitoring programs on U.S. Opioid prescriptions. J. Law. Med. Ethics 46, 387–403 (2018)

    PubMed  Google Scholar 

  • Bachhuber, M.A., Saloner, B., Cunningham, C.O., Barry, C.L.: Medical cannabis laws and opioid analgesic overdose mortality in the United States, 1999–2010. JAMA Intern. Med. 174, 1668–1673 (2014)

    PubMed  PubMed Central  Google Scholar 

  • Barocas, J.A., White, L.F., Wang, J., Walley, A.Y., LaRochelle, M.R., Bernson, D., Land, T., Morgan, J.R., Samet, J.H., Linas, B.P.: Estimated prevalence of opioid use disorder in Massachusetts, 2011–2015: a capture-recapture analysis. Am. J. Public Health 108, 1675–1681 (2018)

    PubMed  PubMed Central  Google Scholar 

  • Ben-Michael, E., Feller, A., Rothstein, J.: The augmented synthetic control method. Working paper. (2019a). https://arxiv.org/abs/1811.04170

  • Ben-Michael, E., Feller, A., Rothstein, J.: Synthetic controls and weighted event studies with staggered adoption. Working paper (2019b). https://arxiv.org/abs/1912.03290.

  • Bertrand, M., Duflo, E., Mullainathan, S.: How much should we trust differences-in-differences estimates? Q. J. Econ. 119, 249–275 (2004)

    Google Scholar 

  • Black, B., Hollingsworth, A., Nunes, L., Simon, K.: The effect of health insurance on mortality: Power analysis and what we can learn from the Affordable Care Act coverage expansions. National Bureau Of Economic Research. Working Paper 25568 (2019). http://www.nber.org/papers/w25568

  • Borusyak, K., Jaravel, X.:. Revisiting event study designs, with an application to the estimation of the marginal propensity to consume. Working paper (2017). https://ssrn.com/abstract=2826228

  • Boslett, A.J., Denham, A., Hill, E.L.: Using contributing causes of death improves prediction of opioid involvement in unclassified drug overdoses in US death records. Addiction 115, 1308–1317 (2020)

    PubMed  Google Scholar 

  • Bradley, E.H., Curry, L.A., Devers, K.J.: Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv. Res. 42, 1758–1772 (2007)

    PubMed  PubMed Central  Google Scholar 

  • Brighthaupt, S.C., Stone, E.M., Rutkow, L., McGinty, E.E.: Effect of pill mill laws on opioid overdose deaths in Ohio & tennessee: a mixed-methods case study. Prev. Med. 126, 105736 (2019)

    CAS  PubMed  Google Scholar 

  • Buchmueller, T.C., Carey, C.: The effect of prescription drug monitoring programs on opioid utilization in Medicare. Am. Econ. J.-Econ. Policy 10, 77–112 (2018)

    Google Scholar 

  • Burris, S., Cloud, L.K., Penn, M.: The growing field of legal epidemiology. J. Public Health Manag. Pract. 26(2), S4–S9 (2020)

    PubMed  Google Scholar 

  • Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., Munafo, M.R.: Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)

    CAS  PubMed  Google Scholar 

  • Callaway, B., Sant’Anna, P.H.C.: Difference-in-differences with multiple time periods. Working paper (2018). https://arxiv.org/pdf/1803.09015.pdf

  • Castillo-Carniglia, A., Ponicki, W.R., Gaidus, A., Gruenewald, P.J., Marshall, B.D.L., Fink, D.S., Martins, S.S., Rivera-Aguirre, A., Wintemute, G.J., Cerda, M.: Prescription drug monitoring programs and opioid overdoses: exploring sources of heterogeneity. Epidemiology 30, 212–220 (2019)

    PubMed  PubMed Central  Google Scholar 

  • Cerda, M.: A public health of consequence: let’s ask about who benefits. Am. J. Public Health 107, 396–397 (2017)

    PubMed  PubMed Central  Google Scholar 

  • Chaisemartin, C.D., D’Haultfoeuille, X.: Two-way fixed effects estimators with heterogeneous treatment effects. Working paper (2019). https://arxiv.org/abs/1803.08807

  • Chapman, S.A., Spetz, J., Lin, J., Chan, K., Schmidt, L.A.: Capturing heterogeneity in medical marijuana policies: a taxonomy of regulatory regimes across the United States. Subst. Use Misuse 51, 1174–1184 (2016)

    PubMed  PubMed Central  Google Scholar 

  • Cicero, T.J., Ellis, M.S.: Abuse-deterrent formulations and the prescription opioid abuse epidemic in the United States: lessons learned from OxyContin. JAMA Psychiatry 72, 424–430 (2015)

    PubMed  Google Scholar 

  • Cole, S.R., Hernan, M.A.: Constructing inverse probability weights for marginal structural models. Am. J. Epidemiol. 168, 656–664 (2008)

    PubMed  PubMed Central  Google Scholar 

  • Darwin Correspondence Project, Letter no. 2130. https://www.darwinproject.ac.uk/letter/DCP-LETT-2130.xml. Accessed April 30 2020

  • Daw, J.R., Hatfield, L.A.: Matching and regression to the mean in difference-in-differences analysis. Health Serv. Res. 53, 4138–4156 (2018a)

    PubMed  PubMed Central  Google Scholar 

  • Daw, J.R., Hatfield, L.A.: Matching in difference-in-differences: between a rock and a hard place. Health Serv. Res. 53, 4111–4117 (2018b)

    PubMed  PubMed Central  Google Scholar 

  • Ellis, A.R., Brookhart, M.A.: Approaches to inverse-probability-of-treatment–weighted estimation with concurrent treatments. J. Clin. Epidemiol. 66, S51–56 (2013)

    PubMed  Google Scholar 

  • Evans, W.N., Lieber, E., Power, P.:. How the reformulation of OxyContin ignited the heroin epidemic. National Bureau of Economic Research. Working Paper 24475 (2018). http://doi.org/10.3386/w24475

  • Gelman, A., Carlin, J.: Beyond power calculations: assessing type s (sign) and type m (magnitude) errors. Perspect. Psychol. Sci. 9, 641–651 (2014)

    PubMed  Google Scholar 

  • Gelman, A., Tuerlinckx, F.A.: Type s error rates for classical and bayesian single and multiple comparison procedures. Comput. Stat. 15, 373–390 (2000)

    Google Scholar 

  • Gertner, A.K., Domino, M.E., Davis, C.S.: Do naloxone access laws increase outpatient naloxone prescriptions? Evidence from Medicaid. Drug Alcohol Depend. 190, 37–41 (2018)

    PubMed  Google Scholar 

  • Gibbons, C.E., Suárez Serrato, J.C., Urbancic, M.B.: Broken or fixed effects? J. Econom. Methods 8, 1 (2019)

    Google Scholar 

  • Goodman-Bacon, A.: Difference-in-differences with variation in treatment timing. National Bureau of Economic Research. Working Paper 25018 (2018). http://www.nber.org/papers/w25018

  • Grant, S., Smart, R., Stein, B.D.: We need a taxonomy of state-level opioid policies, JAMA Health Forum Insights (2020)

  • Griffin, B.A., Schuler, M.S., Stuart, E.A., Patrick, S.W., McNeer, E., Smart, R., Powell, D., Stein, B.D., Schell, T.L., Pacula, R.L.: (under revision.) Moving beyond the classic difference-in-difference model: A simulation study comparing statistical methods for estimating effectiveness of state-level policies. https://arxiv.org/abs/2003.12008

  • Haegerich, T.M., Jones, C.M., Cote, P.O., Robinson, A., Ross, L.: Evidence for state, community and systems-level prevention strategies to address the opioid crisis. Drug Alcohol Depend. 204, 107563 (2019)

    CAS  PubMed  Google Scholar 

  • Haffajee, R.L., Lin, L.A., Bohnert, A.S.B., Goldstick, J.E.: Characteristics of US counties with high opioid overdose mortality and low capacity to deliver medications for opioid use disorder. JAMA Netw. Open 2, 1 (2019)

    Google Scholar 

  • Haffajee, R.L., Mello, M.M., Zhang, F., Zaslavsky, A.M., Larochelle, M.R., Wharam, J.F.: Four states with robust prescription drug monitoring programs reduced opioid dosages. Health Aff. (Millwood) 37, 964–974 (2018)

    Google Scholar 

  • Hedegaard, H., Miniño, A.M., Warner, M.: Drug Overdose Deaths in the United States, 1999–2018. National Center for Health Statistics, Hyattsville, MD (2020)

    Google Scholar 

  • Hernan, M.A., Robins, J.M.: Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183, 758–764 (2016)

    PubMed  PubMed Central  Google Scholar 

  • Horwitz, J., Davis, C.S., McClelland, L.S., Fordon, R.S., Meara, E.: The problem of data quality in analyses of opioid regulation: The case of prescription drug monitoring programs. National Bureau of Economic Research. Working Paper 24947 (2018). https://www.nber.org/papers/w24947.pdf

  • Imai, K., Kim, I.S.: When should we use unit fixed effects regression models for causal inference with longitudinal data? Am. J. Polit. Sci. 63, 467–490 (2019)

    Google Scholar 

  • Ioannidis, J.P.A., Stanley, T.D., Doucouliagos, H.: The power of bias in economics research. Econ. J. 127, F236–F265 (2017)

    Google Scholar 

  • Jackson, J.W.: Diagnostics for confounding of time-varying and other joint exposures. Epidemiology 27, 859–869 (2016)

    PubMed  PubMed Central  Google Scholar 

  • Jones, C.M., Warner, M., Hedegaard, H.W.C.: Data quality considerations when using county-level opioid overdose death rates to inform policy and practice. Drug Alcohol Depend. 204, 107549 (2019)

    PubMed  Google Scholar 

  • Kline, D., Hepler, S., Bonny, A., McKnight, E.: A joint spatial model of opioid-associated deaths and treatment admissions in Ohio. Ann. Epidemiol. 33, 19–23 (2019)

    PubMed  PubMed Central  Google Scholar 

  • Kreif, N., Grieve, R., Hangartner, D., Turner, A.J., Nikolova, S., Sutton, M.: Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ. 25, 1514–1528 (2016)

    PubMed  Google Scholar 

  • Lagisetty, P.A., Ross, R., Bohnert, A., Clay, M., Maust, D.T.: Buprenorphine treatment divide by race/ethnicity and payment. JAMA Psychiatry 76, 979–981 (2019)

    PubMed  PubMed Central  Google Scholar 

  • Lambdin, B.H., Davis, C.S., Wheeler, E., Tueller, S., Kral, A.H.: Naloxone laws facilitate the establishment of overdose education and naloxone distribution programs in the United States. Drug Alcohol Depend. 188, 370–376 (2018)

    PubMed  Google Scholar 

  • Lin, L., Knudsen, H.K.: Comparing buprenorphine-prescribing physicians across nonmetropolitan and metropolitan areas in the United States. Ann. Fam. Med. 17, 212–220 (2019)

    PubMed  Google Scholar 

  • Linden, A., Adams, J.L.: Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation. J. Eval. Clin. Pract. 17, 1231–1238 (2011)

    PubMed  Google Scholar 

  • Lusivika-Nzinga, C., Selinger-Leneman, H., Grabar, S., Costagliola, D., Carrat, F.: Performance of the marginal structural cox model for estimating individual and joined effects of treatments given in combination. BMC Med. Res. Methodol. 17, 160 (2017)

    PubMed  PubMed Central  Google Scholar 

  • Mauri, A.I., Townsend, T.N., Haffajee, R.L.: The association of state opioid misuse prevention policies with patient- and provider-related outcomes: a scoping review. Milbank Q. 98, 57–105 (2020)

    PubMed  Google Scholar 

  • McGinty, E.E., Stone, E.M., Kennedy-Hendricks, A., Bandara, S., Murphy, K.A., Stuart, E.A., Rosenblum, M.A., Daumit, G.L.: Effects of Maryland’s Affordable Care Act Medicaid Health Home Waiver on Quality of Cardiovascular Care Among People with Serious Mental Illness. J. Gen. Intern, Med (2020)

    Google Scholar 

  • McGinty, E.E., Stuart, E.A., Caleb Alexander, G., Barry, C.L., Bicket, M.C., Rutkow, L.: Protocol: mixed-methods study to evaluate implementation, enforcement, and outcomes of US State laws intended to curb high-risk opioid prescribing. Implement Sci. 13, 37 (2018)

    PubMed  PubMed Central  Google Scholar 

  • National Conference of State Legislatures.: Injury prevention legislation database | opioid abuse prevention (2019). http://www.ncsl.org/research/health/injury-prevention-legislation-database.aspx. Accessed April 10, 2020

  • Pacula, R.L., Powell, D.: A supply-side perspective on the opioid crisis. J. Policy Anal. Manag. 37, 438–446 (2018)

    Google Scholar 

  • Pardo, B.: Do more robust prescription drug monitoring programs reduce prescription opioid overdose? Addiction 112, 1773–1783 (2017)

    PubMed  Google Scholar 

  • Pardo, B., Taylor, J., Caulkins, J.P., Kilmer, B., Reuter, P., Stein, B.D.: The Future of Fentanyl and Other Synthetic Opioids. RAND Corporation, Santa Monica, CA (2019)

    Google Scholar 

  • PDAPS.: (2019). http://pdaps.org/datasets/pdmp-implementation-dates

  • Popovici, I., Maclean, J.C., Hijazi, B., Radakrishnan, S.: The effect of state laws designed to prevent nonmedical prescription opioid use on overdose deaths and treatment. Health Econ. 27, 294–305 (2018)

    PubMed  Google Scholar 

  • Powell, D., Pacula, R.L., Jacobson, M.: Do medical marijuana laws reduce addictions and deaths related to pain killers? J. Health Econ. 58, 29–42 (2018)

    PubMed  PubMed Central  Google Scholar 

  • Rigg, K.K., Monnat, S.M., Chavez, M.N.: Opioid-related mortality in rural America: geographic heterogeneity and intervention strategies. Int. J. Drug Policy 57, 119–129 (2018)

    PubMed  Google Scholar 

  • Robins, J.M., Hernan, M.A., Brumback, B.: Marginal structural models and causal inference in epidemiology. Epidemiology 11, 550–560 (2000)

    CAS  PubMed  Google Scholar 

  • Roehler, D.R., Guy Jr., G.P., Jones, C.M.: Buprenorphine prescription dispensing rates and characteristics following federal changes in prescribing policy, 2017-2018: a cross-sectional study. Drug Alcohol Depend. 213, 108083 (2020)

    CAS  PubMed  Google Scholar 

  • Ruhm, C.J.: Corrected US opioid-involved drug poisoning deaths and mortality rates, 1999-2015. Addiction 113, 1339–1344 (2018)

    PubMed  Google Scholar 

  • Ruhm, C.J., et al.: Commentary on Boslett et al. (2020): towards better measurement of drug involvement in fatal overdoses. Addiction 115, 1318–1319 (2020)

    PubMed  Google Scholar 

  • SAMHSA: Key Substance Use and Mental Health Indicators in the United States: Results from the 2018 National Survey on Drug Use and Health. Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration, Rockville, MD (2019)

    Google Scholar 

  • SAMHSA: The Opioid Crisis and the Black/African American population: An Urgent Issue. Office of Behavioral Health Equity, Substance Abuse and Mental Health Services Administration (2020)

    Google Scholar 

  • Sant’Anna, P.H.C., Zhao, J.: Doubly robust difference-in-differences estimators. J. Econom. (2020)

  • Santacatterina, M., Garcia-Pareja, C., Bellocco, R., Sonnerborg, A., Ekstrom, A.M., Bottai, M.: Optimal probability weights for estimating causal effects of time-varying treatments with marginal structural cox models. Stat. Med. 38, 1891–1902 (2019)

    PubMed  Google Scholar 

  • Schell, T., Griffin, B., Morral, A.: Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study. RAND Corporation, Santa Monica, CA (2018)

    Google Scholar 

  • Schell, T.L., Cefalu, M., Griffin, B.A., Smart, R., Morral, A.R.: Changes in firearm mortality following the implementation of state laws regulating firearm access and use. Proc. Natl. Acad. Sci. USA 117, 14906–14910 (2020)

    CAS  PubMed  Google Scholar 

  • Schuler, M.S., Heins, S.E., Smart, R., Griffin, B.A., Powell, D., Stuart, E.A., Pardo, B., Smucker, S., Patrick, S.W., Pacula, R.L., Stein, B.D.: The state of the science in opioid policy research. Drug Alcohol Depend. 214, 108137 (2020)

    CAS  PubMed  Google Scholar 

  • Sharp, A., Jones, A., Sherwood, J., Kutsa, O., Honermann, B., Millett, G.: Impact of Medicaid expansion on access to opioid analgesic medications and medication-assisted treatment. Am. J. Public Health 108, 642–648 (2018)

    PubMed  PubMed Central  Google Scholar 

  • Shrout, P.E., Bolger, N.: Mediation in experimental and nonexperimental studies: new procedures and recommendations. Psychol. Methods 7, 422–445 (2002)

    PubMed  Google Scholar 

  • Smart, R., Kase, C.A., Taylor, E.A., Lumsden, S., Smith, S.R., Stein, B.D.: Strengths and weaknesses of existing data sources to support research to address the opioids crisis. Prev Med Rep 17, 101015 (2020)

    PubMed  Google Scholar 

  • Smith, N., Martins, S.S., Kim, J., Rivera-Aguirre, A., Fink, D.S., Castillo-Carniglia, A., Henry, S.G., Mooney, S.J., Marshall, B.D.L., Davis, C., Cerda, M.: A typology of prescription drug monitoring programs: a latent transition analysis of the evolution of programs from 1999 to 2016. Addiction 114, 248–258 (2019)

    PubMed  Google Scholar 

  • Steenland, K.: Marginal structural models to control for time-varying confounding in occupational and environmental epidemiology. Occup. Environ. Med. 70, 601–602 (2013)

    PubMed  Google Scholar 

  • Stuart, E.A.: Matching methods for causal inference: a review and a look forward. Stat Sci 25, 1–21 (2010)

    PubMed  PubMed Central  Google Scholar 

  • Stuart, E.A., Huskamp, H.A., Duckworth, K., Simmons, J., Song, Z., Chernew, M., Barry, C.L.: Using propensity scores in difference-in-differences models to estimate the effects of a policy change. Health Serv Outcomes Res Methodol 14, 166–182 (2014)

    PubMed  PubMed Central  Google Scholar 

  • The Policy Surveillance Program (2019). Topics. http://lawatlas.org. Accessed June 27, 2019

  • Walley, A.Y., Xuan, Z., Hackman, H.H., Quinn, E., Doe-Simkins, M., Sorensen-Alawad, A., Ruiz, S., Ozonoff, A.: Opioid overdose rates and implementation of overdose education and nasal naloxone distribution in Massachusetts: interrupted time series analysis. BMJ 346, f174 (2013)

    PubMed  PubMed Central  Google Scholar 

  • Wu, L.T., Zhu, H., Swartz, M.S.: Treatment utilization among persons with opioid use disorder in the United States. Drug Alcohol Depend. 169, 117–127 (2016)

    PubMed  PubMed Central  Google Scholar 

  • Xu, Y.Q.: Generalized synthetic control method: causal inference with interactive fixed effects models. Political Analysis 25, 57–76 (2017)

    Google Scholar 

Download references

Funding

MS Schuler, BA Griffin, and EA Stuart were supported by award number P50DA046351 from the National Institute on Drug Abuse. EE McGinty and EA Stuart were supported by award R01DA044987 from the National Institute on Drug Abuse. M Cerdá was supported by award R01DA045872 from the National Institute on Drug Abuse. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Drug Abuse or the National Institutes of Health.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Megan S. Schuler.

Ethics declarations

Conflict of interest

All the authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schuler, M.S., Griffin, B.A., Cerdá, M. et al. Methodological challenges and proposed solutions for evaluating opioid policy effectiveness. Health Serv Outcomes Res Method 21, 21–41 (2021). https://doi.org/10.1007/s10742-020-00228-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-020-00228-2

Keywords

  • Opioid policy
  • Statistical methodology
  • Data quality
  • Treatment heterogeneity
  • Selection bias