Desideratum for Evidence Based Epidemiology
- 359 Downloads
There is great variation in choices of method and specific analytical details in epidemiological studies, resulting in widely varying results even when studying the same drug and outcome in the same database. Not only does this variation undermine the credibility of the research but it limits our ability to improve the methods.
In order to evaluate the performance of methods and analysis choices we used standard references and a literature review to identify 164 positive controls (drug–outcome pairs believed to represent true adverse drug reactions), and 234 negative controls (drug–outcome pairs for which we have confidence there is no direct causal relationship). We tested 3,748 unique analyses (methods in combination with specific analysis choices) that represent the full range of approaches to adjusting for confounding in five large observational datasets on these controls. We also evaluated the impact of increasingly specific outcome definitions, and performed a replication study in six additional datasets. We characterized the performance of each method using the area under the receiver operator curve (AUC), bias, and coverage probability. In addition, we developed simulated datasets that closely matched the characteristics of the observational datasets into which we inserted data consistent with known drug–outcome relationships in order to measure the accuracy of estimates generated by the analyses.
We expect the results of this systematic, empirical evaluation of the performance of these analyses across a moderate range of outcomes and databases to provide important insights into the methods used in epidemiological studies and to increase the consistency with which methods are applied, thereby increasing the confidence in results and our ability to systematically improve our approaches.
KeywordsCoverage Probability Observational Dataset Common Data Model Observational Database Observational Medical Outcome Partnership
The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health (FNIH) through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Dr. Overhage is an employee of Siemens. Drs. Ryan and Stang are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration, and has become an employee of Janssen Research and Development since completing the work described here. Dr. Schuemie has previously received a grant from FNIH.
This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
- 1.Popper K. Science: conjectures and refutations. In: Mace CA, editor. A lecture given at Peterhouse, Cambridge, in Summer 1953, as part of a course on developments and trends in contemporary British philosophy, organized by the British Council; originally published under the title ‘Philosophy of Science: a Personal Report’ in British Philosophy in Mid-Century, 1957.Google Scholar
- 2.Young SS, Karr A. Deming, data and observational studies. Significance. 2011;8(3):116–20.Google Scholar
- 10.Tuma RS. Statisticians set sights on observational studies. J Natl Cancer Inst. 2007;99(9):664–5, 8.Google Scholar
- 23.McDonald CJ. The evolution of Intel’s copy exactly! Technology transfer method. Intel Technol J. 1998;Q4:1–6.Google Scholar
- 24.Terwiesch C, Xu Y. The copy exactly ramp-up strategy: trading-off learning with process change. August 4, 2003, cited 2012 December 24. http://qbox.wharton.upenn.edu/documents/opim/research/P6.pdf.
- 25.Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82–93.Google Scholar
- 27.Stang PE, Ryan PB, Overhage JM, Schuemie MJ, Hartzema AG, Welebob E. Variation in choice of study design: findings from the epidemiology design decision inventory and evaluation (EDDIE) survey. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0103-1.
- 28.The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on methodological standards in pharmacoepidemiology (revision 1). EMA/95098/2010. Cited 2013 January 23. http://www.encepp.eu/standards_and_guidances/documents/ENCePPGuideofMethStandardsinPE.pdf.
- 30.Gagne JJ, Nelson JC, Fireman B, Seeger JD, Toh D, Gerhard T, et al. Taxonomy for monitoring methods within a medical product safety surveillance system: year two report of the mini-sentinel taxonomy project workgroup (workgroup) 2012, cited 2012 October 29. http://www.mini-sentinel.org/work_products/Statistical_Methods/Mini-Sentinel_Methods_Taxonomy-Year-2-Report.pdf.
- 31.Taleb NN. The Black Swan: the impact of the highly improbable. New York: Random House; 2010.Google Scholar
- 35.Suissa S. Time-related biases in pharmacoepidemiology. In: International Society of Pharmacoepidemiology mid-year meeting, Miami Beach, Florida, 2012.Google Scholar
- 41.Ryan PB, Schuemie MJ, Gruber S, Zorych I, Madigan D. Empirical performance of a new user cohort method: Lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0099-6.
- 42.Madigan D, Schuemie MJ, Ryan PB. Empirical performance of the case–control method: lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0105-z.
- 43.Suchard MA, Zorych I, Simpson SE, Schuemie MJ, Ryan PB, Madigan D. Empirical performance of the self-controlled case series design: lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0100-4.
- 44.Ryan PB, Schuemie MJ, Madigan D. Empirical performance of a self-controlled cohort method: lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0101-3.
- 45.Schuemie MJ, Madigan D, Ryan PB. Empirical performance of LGPS and LEOPARD: lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0107-x.
- 46.Norén GN, Bergvall T, Ryan PB, Juhlin K, Schuemie MJ, Madigna D. Empirical performance of the calibrated self-controlled cohort analysis within temporal pattern discovery: lessons for developing a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0095-x.
- 47.DuMouchel B, Ryan PB, Schuemie MJ, Madigan D. Evaluation of disproportionality safety signaling applied to healthcare databases. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0106-y.
- 49.Hartzema AG, Reich C, Ryan PB, Stang PE, Madigna D, Welebob E, et al. Managing data quality for a drug safety surveillance system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0098-7.
- 50.Hansen RA, Gray MD, Fox BI, Hollingsworth JC, Gao J, Zeng P. How well do various health outcome definitions used in observational studies identify cases that are consistent with expert opinion? Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0104-0.
- 51.Reich C, Ryan PB, Schuemie MJ. Alternative outcome definitions and their effect on the performance of methods for observational outcome studies. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0111-1.
- 52.Reich CG, Ryan PB, Suchard MA. The impact of drug and outcome prevalence on the feasibility and performance of analytical methods for a risk identification and analysis system. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0112-0.
- 54.Ryan PB, Schuemie MJ, Welebob E, Duke J, Valentine S, Hartzema AG. Defining a reference set to support methodological research in drug safety. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0097-8.
- 55.Schuemie MJ, Gini R, Coloma PM, Straatman H, Herings RMC, Pedersen L, et al. Replication of the OMOP experiment in Europe: evaluating methods for risk identification in electronic health record databases. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0109-8.
- 56.Ryan PB, Schuemie MJ. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf (in this supplement issue). doi: 10.1007/s40264-013-0110-2.
- 58.Tisdale J, Miller D. Drug-induced diseases: prevention, detection, and management. 2nd ed. USA: American Society of Health-System Pharmacists; 2010.Google Scholar