Skip to main content

Replication of the OMOP Experiment in Europe: Evaluating Methods for Risk Identification in Electronic Health Record Databases

Drug Safety volume 36, pages 159–169 (2013)Cite this article

Abstract

Background

The Observational Medical Outcomes Partnership (OMOP) has just completed a large scale empirical evaluation of statistical methods and analysis choices for risks identification in longitudinal observational healthcare data. This experiment drew data from four large US health insurance claims databases and one US electronic health record (EHR) database, but it is unclear to what extend the findings of this study apply to other data sources.

Objective

To replicate the OMOP experiment in six European EHR databases.

Research Design

Six databases of the EU-ADR (Exploring and Understanding Adverse Drug Reactions) database network participated in this study: Aarhus (Denmark), ARS (Italy), HealthSearch (Italy), IPCI (the Netherlands), Pedianet (Italy), and Pharmo (the Netherlands). All methods in the OMOP experiment were applied to a collection of 165 positive and 234 negative control drug–outcome pairs across four outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding. Area under the receiver operator characteristics curve (AUC) was computed per database and for a combination of all six databases using meta-analysis for random effects. We provide expected values of estimation error as well, based on negative controls.

Results

Similarly to the US experiment, high predictive accuracy was found (AUC >0.8) for some analyses. Self-controlled designs, such as self-controlled case series, IC temporal pattern discovery and self-controlled cohort achieved higher performance than other methods, both in terms of predictive accuracy and observed bias.

Conclusions

The major findings of the recent OMOP experiment were also observed in the European databases.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 49.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

References

  1. Ryan PB, Madigan D, Stang PE, Marc Overhage J, Racoosin JA, Hartzema AG. Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership. Stat Med. 2012;31(30):4401–15.

    Google Scholar 

  2. Schuemie MJ, Coloma PM, Straatman H, Herings RM, Trifirò G, Matthews JN, et al. Using electronic health care records for drug safety signal detection: a comparative evaluation of statistical methods. Med Care. 2012;50(10):890–7.

    PubMed  Article  Google Scholar 

  3. Ryan PB, Stang PE, Overhage JM, Suchard MA, Hartzema AG, DuMouchel W, et al. A comparison empirical performance of methods for a risk identification system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0108-9.

  4. Reich C, Ryan PB, Stang PE, Rocca M. Evaluation of alternative standardized terminologies for medical conditions within a network of observational healthcare databases. J Biomed Inform. 2012;45(4):689–96.

    PubMed  Article  Google Scholar 

  5. Avillach P, Coloma PM, Gini R, Schuemie M, Mougin F, Dufour JC, et al. Harmonization process for the identification of medical events in eight European healthcare databases: the experience from the EU-ADR project. J Am Med Inform Assoc (JAMIA). 2013;20(1):184–92.

    Google Scholar 

  6. Hsia DC, Krushat WM, Fagan AB, Tebbutt JA, Kusserow RP. Accuracy of diagnostic coding for Medicare patients under the prospective-payment system. N Engl J Med. 1988;318(6):352–5.

    PubMed  Article  CAS  Google Scholar 

  7. Madigan D, Schuemie MJ, Ryan PB. Empirical performance of the case-control method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0105-z.

  8. Ryan PB, Schuemie MJ, Gruber S, Zorych I, Zorych D. Empirical performance of a new user cohort method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0099-6.

  9. DuMouchel B, Ryan PB, Schuemie MJ, Madigan D. Evaluation of disproportionality safety signaling applid to health care databases. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0106-y.

  10. Norén GN, Bergvall T, Ryan PB, Juhlin K, Schuemie MJ, Madigan D. Empirical performance of the calibrated self-controlled cohort analysis within temporal pattern discovery: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0095-x.

  11. Schuemie MJ, Madigan D, Ryan PB. Empirical performance of LGPS and LEOPARD: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0107-x.

  12. Ryan PB, Schuemie MJ, Madigan D. Empirical performance of a self-controlled cohort method: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0101-3.

  13. Suchard MA, Zorych I, Simpson SE, Schuemie MJ, Ryan PB, Madigan D. Empirical performance of the self-controlled case series design: lessons for developing a risk identification and analysis system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0100-4.

  14. Hartzema AG, Reich CG, Ryan PB, Stang PE, Madigan D, Welebob E, et al. Managing data quality for a drug safety surveillance system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0098-7.

  15. Armstrong B. A simple estimator of minimum detectable relative risk, sample size, or power in cohort studies. Am J Epidemiol. 1987;126(2):356–8.

    PubMed  Article  CAS  Google Scholar 

  16. Cantor SB, Kattan MW. Determining the area under the ROC curve for a binary diagnostic test. Med Decis Making. 2000;20(4):468–70.

    PubMed  Article  CAS  Google Scholar 

  17. Ryan PB, Schuemie MJ. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0110-2.

  18. Reich CG, Ryan PB, Schuemie MJ. Alternative outcome definitions and their effect on the performance of methods for observational outcome studies. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0111-1.

Download references

Acknowledgments

The authors wish to acknowledge the Observational Medical Outcomes Partnership (OMOP) investigators for their help in running the OMOP methods in the EU-ADR environment. The study was funded by The Foundation for the National Institutes of Health, grant STUR11OMOP. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration, has become an employee of Janssen Research & Development since completing this research, and is an OMOP investigator. OMOP is funded by the Foundation for the National Institutes of Health through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Mr. Straatman and Dr. Herings are employees of the PHARMO Institute for Drug Outcomes Research, which performs financially supported studies for several pharmaceutical companies. All other authors declare to have no conflicts of interests to declare. Dr. Sturkenboom is an employee of the Erasmus University Medical Center, and coordinates studies that are financially supported by several pharmaceutical companies (Novartis, Pfizer, EliLilly, Boehringer), none related to this study.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project.from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.

Author information

Authors and Affiliations

  1. Department of Medical Informatics, Erasmus University Medical Center, P.O. Box 2040, 3000 CA, Rotterdam, The Netherlands

    Martijn J. Schuemie, Rosa Gini, Preciosa M. Coloma, Ron M. C. Herings, Johan van der Lei & Miriam C. J. M. Sturkenboom

  2. Agenzia Regionale di Sanità della Toscana, Florence, Italy

    Rosa Gini & Francesco Innocenti

  3. PHARMO Institute, Utrecht, The Netherlands

    Huub Straatman & Ron M. C. Herings

  4. Department of Clinical Epidemiology, Aarhus University Hospital, Århus, Denmark

    Lars Pedersen

  5. Health Search, Italian College of General Practitioners, Florence, Italy

    Francesco Innocenti & Giampiero Mazzaglia

  6. Pedianet, Società Servizi Telematici SRL, Padova, Italy

    Gino Picelli

  7. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, USA

    Martijn J. Schuemie

Authors
  1. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rosa Gini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Preciosa M. Coloma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huub Straatman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ron M. C. Herings
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lars Pedersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Francesco Innocenti
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Giampiero Mazzaglia
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Gino Picelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Johan van der Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Miriam C. J. M. Sturkenboom
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martijn J. Schuemie.

Additional information

The OMOP research used data from Truven Health Analytics (formerly the Health Business of Thomson Reuters), and includes MarketScan® Research Databases, represented with MarketScan Lab Supplemental (MSLR, 1.2 m persons), MarketScan Medicare Supplemental Beneficiaries (MDCR, 4.6 m persons), MarketScan Multi-State Medicaid (MDCD, 10.8 m persons), MarketScan Commercial Claims and Encounters (CCAE, 46.5 m persons). Data also provided by Quintiles® Practice Research Database (formerly General Electric’s Electronic Health Record, 11.2 m persons) database. GE is an electronic health record database while the other four databases contain administrative claims data.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Mapping from ATC to RxNorm identifiers (xlsx 178 kb)

Below is the link to the electronic supplementary material.

Age-stratified analysis of incidence across OMOP and EU-ADR databases (xlsx 13 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Schuemie, M.J., Gini, R., Coloma, P.M. et al. Replication of the OMOP Experiment in Europe: Evaluating Methods for Risk Identification in Electronic Health Record Databases. Drug Saf 36 (Suppl 1), 159–169 (2013). https://doi.org/10.1007/s40264-013-0109-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-013-0109-8

Keywords

  • Propensity Score
  • Acute Liver Injury
  • Relative Risk Estimate
  • Analysis Choice
  • True Effect Size