Drug Safety

, Volume 40, Issue 5, pp 399–408 | Cite as

Validation of New Signal Detection Methods for Web Query Log Data Compared to Signal Detection Algorithms Used With FAERS

  • Susan Colilla
  • Elad Yom Tov
  • Ling Zhang
  • Marie-Laure Kurzinger
  • Stephanie Tcherny-Lessenot
  • Catherine Penfornis
  • Shang Jen
  • Danny S. Gonzalez
  • Patrick Caubel
  • Susan Welsh
  • Juhaeri Juhaeri
Original Research Article



Post-marketing drug surveillance is largely based on signals found in spontaneous reports from patients and healthcare providers. Rare adverse drug reactions and adverse events (AEs) that may develop after long-term exposure to a drug or from drug interactions may be missed. The US FDA and others have proposed that web-based data could be mined as a resource to detect latent signals associated with adverse drug reactions.


Recently, a web-based search query method called a query log reaction score (QLRS) was developed to detect whether AEs associated with certain drugs could be found from search engine query data. In this study, we compare the performance of two other algorithms, the proportional query ratio (PQR) and the proportional query rate ratio (Q-PRR) against that of two reference signal-detection algorithms (SDAs) commonly used with the FDA AE Reporting System (FAERS) database.


In summary, the web query methods have moderate sensitivity (80%) in detecting signals in web query data compared with reference SDAs in FAERS when the web query data are filtered, but the query metrics generate many false-positives and have low specificity compared with reference SDAs in FAERS.


Future research is needed to find better refinements of query data and/or the metrics to improve the specificity of these web query log algorithms.

Supplementary material

40264_2017_507_MOESM1_ESM.docx (103 kb)
Supplementary material 1 (DOCX 102 kb)


  1. 1.
    Sarker A, Ginn R, Nikfarjam A, O’Connor K, Smith K, Jayaraman S, et al. Utilizing social media data for pharmacovigilance: A review. J Biomed Inform. 2015;54:202–12.CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Fletcher AP. An appraisal of spontaneous adverse event monitoring. Adverse Drug React Toxicol Rev. 1992;11(4):213–27.PubMedGoogle Scholar
  3. 3.
    Harpaz R, Callahan A, Tamang S, Low Y, Odgers D, Finlayson S, et al. Text mining for adverse drug events: the promise, challenges, and state of the art. Drug Saf. 2014;37(10):777–90.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Morel P, Vandel B. Adverse drug reaction monitoring and the Internet: evaluation of the use of the Internet by French Pharmacovigilance Centres and a non-exhaustive survey of websites of interest for collecting information about adverse drug reaction. Therapie. 1999;54(5):525–32.PubMedGoogle Scholar
  5. 5.
    White RW, Harpaz R, Shah NH, DuMouchel W, Horvitz E. Toward enhanced pharmacovigilance using patient-generated data on the internet. Clin Pharmacol Ther. 2014;96(2):239–46.CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Sloane R, Osanlou O, Lewis D, Bollegala D, Maskell S, Pirmohamed M. Social media and pharmacovigilance: a review of the opportunities and challenges. Br J Clin Pharmacol. 2015;80(4):910–20.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Yom-Tov E, Gabrilovich E. Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large-scale analysis of web search queries. J Med Internet Res. 2013;15(6):e124.CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    Yang M, Kiang M, Shang W. Filtering big data from social media–Building an early warning system for adverse drug reactions. J Biomed Inform. 2015;54:230–40.CrossRefPubMedGoogle Scholar
  9. 9.
    Dal Pan G, editor. Adverse event reporting and medication safety considerations: a view from CDER’s Office of Surveillance and Epidemiology. CHPA Regulatory, Scientific, and Quality Conference: Washington, DC; 2013.Google Scholar
  10. 10.
    US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research (CDER). Center for Biologics Evaluation and Research (CBER). Guidance for industry. Postmarketing Safety Reporting for Human Drug and Biological Products Including Vaccines. Rockville: US Department of Health and Human Services; 2001.Google Scholar
  11. 11.
    Odgers DJ, Harpaz R, Callahan A, Stiglic G, Shah NH. Analyzing search behavior of healthcare professionals for drug safety surveillance. Pac Symp Biocomput. 2015:306–17.Google Scholar
  12. 12.
    Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.CrossRefPubMedGoogle Scholar
  13. 13.
    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.CrossRefPubMedGoogle Scholar
  14. 14.
    Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013;93(6):539–46.CrossRefPubMedGoogle Scholar
  15. 15.
    Evans SJ, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.CrossRefPubMedGoogle Scholar
  16. 16.
    Powell GE, Seifert HA, Reblin T, Burstein PJ, Blowers J, Menius JA, et al. Social media listening for routine post-marketing safety surveillance. Drug Saf. 2016;39(5):443–54.CrossRefPubMedGoogle Scholar
  17. 17.
    Austin PC, Steyerberg EW. Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol. 2012;12:82.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Sarntivijai S, Abernethy DR. Use of internet search logs to evaluate potential drug adverse events. Clin Pharmacol Ther. 2014;96(2):149–50.CrossRefPubMedGoogle Scholar
  19. 19.
    DuMouchel W, Ryan PB, Schuemie MJ, Madigan D. Evaluation of disproportionality safety signaling applied to healthcare databases. Drug Saf. 2013;36(Suppl 1):S123–32.CrossRefPubMedGoogle Scholar
  20. 20.
    Harpaz R, Vilar S, Dumouchel W, Salmasian H, Haerian K, Shah NH, et al. Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Am Med Inform Assoc. 2013;20(3):413–9.CrossRefPubMedGoogle Scholar
  21. 21.
    Candore G, Juhlin K, Manlik K, Thakrar B, Quarcoo N, Seabroke S, et al. Comparison of statistical signal detection methods within and across spontaneous reporting databases. Drug Saf. 2015;38(6):577–87.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Susan Colilla
    • 1
  • Elad Yom Tov
    • 2
  • Ling Zhang
    • 1
  • Marie-Laure Kurzinger
    • 3
  • Stephanie Tcherny-Lessenot
    • 3
  • Catherine Penfornis
    • 3
  • Shang Jen
    • 4
  • Danny S. Gonzalez
    • 5
  • Patrick Caubel
    • 6
  • Susan Welsh
    • 1
  • Juhaeri Juhaeri
    • 1
  1. 1.PharmacoepidemiologyGlobal Safety Sciences, SanofiBridgewaterUSA
  2. 2.Microsoft ResearchHerzeliyaIsrael
  3. 3.Pharmacoepidemiology & Signal Detection, Global Safety SciencesSanofiChilly-MazarinFrance
  4. 4.Baxalta US, Inc., Global Drug SafetyCambridgeUSA
  5. 5.US Food and Drug AdministrationSilver SpringUSA
  6. 6.Pfizer, Worldwide SafetyNew YorkUSA

Personalised recommendations