Defining a Reference Set to Support Methodological Research in Drug Safety
- 691 Downloads
Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards.
To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues.
Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding.
Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes.
A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.
KeywordsAcute Myocardial Infarction Acute Kidney Injury Product Label Acute Liver Injury Proportional Reporting Ratio
The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health (FNIH) through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc., Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Drs. Ryan and Schuemie are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration. Drs. Duke, Schuemie and Hartzema have previously received funding from FNIH. Emily Welebob and Sarah Valentine have no conflicts of interest to declare.
This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project.from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
- 1.FDA. Guidance for industry: good pharmacovigilance practices and pharmacoepidemiologic assessment. US FDA Center for Drug Evaluation and Research and Center for Biologics Evaluation and Research; 2005.Google Scholar
- 3.FDA. The Sentinel Initiative: A National Strategy for Monitoring Medical Product Safety. May 2008 [cited 2012 September 15]. http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm089474.htm.
- 9.Woodward M. Epidemiology study design and data analysis. London: Chapman & Hall/CRC; 1999.Google Scholar
- 10.Whitaker H. The self controlled case series method. BMJ 2008;337:a1069. http://dx.doi.org/10.1136/bmj.a1069.
- 13.Ryan PB, Powell GE, Pattishall EN, Beach KJ. Performance of screening multiple observational databases for active drug safety surveillance. Poster presented at the 25 annual meeting of the International Society of Pharmacoepidemiology, Providence, Rhode Island, 16–19 August 2009.Google Scholar
- 14.Schuemie MJ, Coloma PM, Straatman H, Herings RM, Trifirò G, Matthews JN, et al. Using electronic health care records for drug safety signal detection: a comparative evaluation of statistical methods. Med Care. 2012;50(10):890–7.Google Scholar
- 23.Friedlin J, Duke J. Applying natural language processing to extract codify adverse drug reaction in medication labels. 2010 [cited 2013 January 3]. http://omop.org/sites/default/files/omop_white_paper_friedlin_08_26_10.pdf.
- 24.Friedlin J, Duke J. Exploration of four outcomes: outcomes and labeling information, in conjunction with other evidence May 2011 [cited 2013 January 3]. http://omop.org/sites/default/files/OMOP%20Report_Duke_Friedlin_05_16_11%20Exploration%20of%20Four%20Outcomes.pdf.
- 25.Tisdale J, Miller D. Drug-induced diseases: prevention, detection, and management. 2nd ed. USA: American Society of Health-System Pharmacists; 2010.Google Scholar
- 26.Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for evidence based epidemiology. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0102-2.
- 27.Ryan PB, Madigan D. Selecting comparators in active surveillance analyses. 2010 [cited 2013 January 3]. http://omop.org/sites/default/files/OMOP%20-%20Selecting%20comparators%20in%20active%20surveillance%20analyses.pdf.
- 30.Hansen RA, Gray MD, Fox BI, Hollingsworth JC, Gao J, Zeng P. How well do various health outcome definitions used in observational studies identify cases that are consistent with expert opinion? Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0104-0.
- 34.Roumie CL, Choma NN, Kaltenbach L, Mitchel EF Jr, Arbogast PG, Griffin MR. Non-aspirin NSAIDs, cyclooxygenase-2 inhibitors and risk for cardiovascular events-stroke, acute myocardial infarction, and death from coronary heart disease. Pharmacoepidemiol Drug Saf. 2009;18(11):1053–63.PubMedCrossRefGoogle Scholar
- 36.Helin-Salmivaara A, Virtanen A, Vesalainen R, Gronroos JM, Klaukka T, Idanpaan-Heikkila JE, et al. NSAID use and the risk of hospitalization for first myocardial infarction in the general population: a nationwide case-control study from Finland. Eur Heart J. 2006;27(14):1657–63.PubMedCrossRefGoogle Scholar
- 44.Reich CG, Ryan PB, Suchard MA. The impact of drug and outcome prevalence on the feasibility and performance of analytical methods for a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0112-0.
- 45.Schuemie MJ, Madigan D, Ryan PB. Empirical performance of longitudinal gamma poisson shrinker (LGPS) and longitudinal evaluation of observational profiles of adverse events related to drugs (LEOPARD): lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0107-x.
- 46.Norén GN, Bergvall T, Ryan PB, Juhlin K, Schuemie MJ, Madigna D. Empirical performance of the calibrated self-controlled cohort analysis within temporal pattern discovery: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0095-x.
- 47.Madigan D, Schuemie MJ, Ryan PB. Empirical performance of the case-control method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0105-z.
- 48.Ryan PB, Schuemie MJ, Gruber S, Zorych I, Madigan D. Empirical performance of a new user cohort method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0099-6.
- 49.Suchard MA, Zorych I, Simpson SE, Schuemie MJ, Ryan PB, Madigan D. Empirical performance of the self-controlled case series design: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0100-4.
- 50.Ryan PB, Schuemie MJ, Madigan D. Empirical performance of a self-controlled cohort method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0101-3.
- 51.DuMouchel B, Ryan PB, Schuemie MJ, Madigan D. Evaluation of disproportionality safety signaling applied to health care databases. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0106-y.
- 52.Schuemie MJ, Gini R, Coloma PM, Straatman H, Herings RMC, Pedersen L, et al. Replication of the OMOP experiment in Europe: evaluating methods for risk identification in electronic health record databases. Drug Saf. 2013 (in this supplement issue). doi: 10.1007/s40264-013-0109-8.
- 53.Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel data-mining methodologies for adverse drug event discovery and analysis. Clin Pharmacol Ther. 2012;91(6):1010–21.Google Scholar
- 64.Warner JJ, Weideman RA, Kelly KC, Brilakis ES, Banerjee S, Cunningham F, et al. The risk of acute myocardial infarction with etodolac is not increased compared to naproxen: a historical cohort analysis of a generic COX-2 selective inhibitor. J Cardiovasc Pharmacol Ther. 2008;13(4):252–60.PubMedCrossRefGoogle Scholar
- 66.de Abajo FJ, García-Rodríguez LA. Risk of upper gastrointestinal tract bleeding associated with selective serotonin reuptake inhibitors and venlafaxine therapy: interaction with nonsteroidal anti-inflammatory drugs and effect of acid-suppressing agents. Arch Gen Psychiatry. 2008;65(7):795–803.PubMedCrossRefGoogle Scholar
- 71.Vidal X, Ibanez L, Vendrell L, Conforti A, Laporte LR, Spanish-Italian Collaborative Group for the Epidemiology of Gastrointestinal B. Risk of upper gastrointestinal bleeding and the degree of serotonin reuptake inhibition by antidepressants: a case-control study. Drug Safety. 2008;31(2):159–68.PubMedCrossRefGoogle Scholar
- 78.Lanas A, García-Rodríguez LA, Arroyo MT, Gomollon F, Feu F, Gonzalez-Perez A, et al. Risk of upper gastrointestinal ulcer bleeding associated with selective cyclo-oxygenase-2 inhibitors, traditional non-aspirin non-steroidal anti-inflammatory drugs, aspirin and combinations. Gut. 2006;55(12):1731–8.PubMedCrossRefGoogle Scholar