Skip to main content
SpringerLink
Log in

Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Background and Significance

Adverse drug events (ADEs) occur in approximately 2–5% of hospitalized patients, often resulting in poor outcomes or even death. Extraction of ADEs from clinical narratives can accelerate and automate pharmacovigilance. Using state-of-the-art deep-learning neural networks to jointly model concept and relation extraction, we achieved the highest integrated task score in the 2018 Medication and Adverse Drug Event (MADE) 1.0 challenge.

Methods

We used a combined bidirectional long short-term memory (BiLSTM) and conditional random fields (CRF) neural network to detect medical entities relevant to ADEs and a combined BiLSTM and attention network to determine relations, including the adverse drug reaction relation between medication and sign or symptom entities. Using these models, we conducted three experiments: (1) separate and sequential modeling of entities and relations; (2) joint modeling where relations between medications and sign or symptoms determined ADE and indication entities; (3) use of information from external resources such as the US FDA’s adverse event database as additional input to the second method.

Results

Joint modeling improved the overall task accuracy from 0.62 to 0.65 F measure, and the additional use of external resources improved the accuracy to 0.66 F measure. Given the gold-standard medical entity labels, the joint model plus external resources method achieved F measures of 0.83 for ADE-relevant medical entity detection and 0.87 for relation detection.

Conclusion

It is important to use joint modeling techniques and external resources for effectively detecting ADEs from clinical narratives in electronic health record (EHR) systems. While the extraction of entities and relations individually achieved high accuracy, the integrated task still has room for further improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. 2015. arXiv:1508.01991.

  2. Chalapathy R, Borzeshi EZ, Piccardi M. Bidirectional LSTM-CRF for clinical concept extraction. In: Procedings of the clinical natural language processing workshop. 2016. pp. 7–12.

  3. Habibi M, Weber L, Neves M, Wiegandt DL, Leser U. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics. 2017;33(14):i37–48.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Li F, Zhang M, Tian B, Chen B, Fu G, Ji D. Recognizing irregular entities in biomedical text via deep neural networks. Pattern Recognit Lett. 2017;105:105–13.

    Article  Google Scholar 

  5. Dandala B, Mahajan D, Devarakonda M. IBM research system at TAC 2017: adverse drug reactions extraction from drug labels. In: Text analysis conference (TAC) 2017 workshop at NIST. 2017.

  6. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. 2016. arXiv:1409.0473.

  7. Hermann KM, et al. Teaching machines to read and comprehend. In: NIPS’15 proceedings of the 28th international conference on neural information processing systems, vol. 1. 2015. pp. 1693–1701.

  8. Zhou P et al. Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers); 2016. pp. 207–212.

  9. UMass BioNLP. NLP challenges for detecting medication and adverse drug events from electronic health records (MADE 1.0). https://bio-nlp.org/index.php/projects/39-nlp-challenges. Accessed 5 Feb 2018.

  10. US Food and Drug Administration, “FAERS”. https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/default.htm. Accessed 7 Feb 2018.

  11. Ma X, Hovy E. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In: Proceedings of the 54th annual meeting of the association for computational linguistics. 2016. pp. 1064–1074.

  12. Zhang D, Wang D. Relation classification via recurrent neural network. 2015. arXiv:1508.01006.

  13. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.

    Article  CAS  PubMed  Google Scholar 

  14. Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw. 1994;5(2):157–66.

    Article  CAS  PubMed  Google Scholar 

  15. Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 2005;18(5):602–10.

    Article  PubMed  Google Scholar 

  16. Graves A. Generating sequences with recurrent neural networks. 2014. arXiv:1308.0850.

  17. Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Neural architectures for named entity recognition. In: Proceedings of NAACL-HLT. 2016. pp. 260–270.

  18. Collobert R, et al. Natural language processing (almost) from scratch. J Mach Learn Res. 2011;12:2493–537.

    Google Scholar 

  19. Sutton C, McCallum A, et al. An introduction to conditional random fields. Found Trends Mach Learn. 2012;4(4):267–373.

    Article  Google Scholar 

  20. Zhang X, Zhao J, LeCun Y. Character-level convolutional networks for text classification. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in neural information processing systems, vol. 28. New York: Curran Associates Inc.; 2015. pp. 649–57.

    Google Scholar 

  21. Swampillai K, Stevenson M. Extracting relations within and across sentences. Proc Int Conf Recent Adv Nat Lang Process. 2011;2011:25–32.

    Google Scholar 

  22. Quirk C, Poon H. Distant supervision for relation extraction beyond the sentence boundary. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics. 2016. pp. 1171–1182.

  23. Peng N, Poon H, Quirk C, Toutanova K, Yih W. Cross-sentence n-ary relation extraction with graph lstms. Trans Assoc Comput Linguis. 2017;5:101–15.

    Article  Google Scholar 

  24. Griffis D, Shivade C, Fosler-Lussier E, Lai AM. A quantitative and qualitative evaluation of sentence boundary detection for the clinical domain. AMIA Summits Transl Sci Proc. 2016;2016:88.

    PubMed  PubMed Central  Google Scholar 

  25. McCord MC, Bernth A. Using slot grammar. IBM TJ Watson Research Center, Yorktown Heights, NY, IBM Research Reports RC23978; 2010.

  26. Minsky M. Memoir on inventing the confocal scanning microscope. Scanning. 1988;10(4):128–38.

    Article  Google Scholar 

  27. Fillmore CJ. Frame semantics and the nature of language. Ann N Y Acad Sci. 1976;280(1):20–32.

    Article  Google Scholar 

  28. Dandala B, Devarakonda M, Bornea M, Nielson C. Scoring disease-medication associations using advanced NLP, machine learning, and multiple content sources. In: Proceedings of the fifth workshop on building and evaluating resources for biomedical text mining (BioTxtM 2016). 2016. pp. 125–133.

  29. Banda JM, Evans L, Vanguri RS, Tatonetti NP, Ryan PB, Shah NH. A curated and standardized adverse drug event resource to accelerate drug safety research. Sci Data. 2016;3:1–11.

    Article  CAS  Google Scholar 

  30. Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl 1):D267–70.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Rajani NF, Bornea M, Barker K. Stacking with auxiliary features for entity linking in the medical domain. BioNLP. 2017;2017:39–47.

    Google Scholar 

  32. Kingma D, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR). 2015. pp. 1–15.

  33. Jagannatha AN, Yu H. Structured prediction models for RNN based sequence labeling in clinical text. In: Proceedings of the conference on empirical methods in natural language processing. conference on empirical methods in natural language processing, vol. 2016. 2016. p. 856.

  34. Munkhdalai T, Liu F, Yu H. Clinical relation extraction toward drug safety surveillance using electronic health record narratives: classical learning versus deep learning. JMIR Public Health Surveill. 2018;4(2):e29.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Sahu SK, Anand A, Oruganty K, Gattu M. Relation extraction from clinical texts using domain invariant convolutional neural network. In: Proceedings of the 15th workshop on biomedical natural language processing. 2016. pp. 206–215.

  36. Li F, Zhang M, Fu G, Ji D. A neural joint model for entity and relation extraction from biomedical text. BMC Bioinform. 2017;18(1):1–11.

    Article  CAS  Google Scholar 

  37. Chapman AB, Peterson KS, Alba PR, DuVall SL, Patterson OV. Hybrid system for adverse drug event detection. In: Proceedings of machine learning research, vol. 90. 2018. pp. 16–24.

  38. Xu D, Yadav V, Bethard S. UArizona at the MADE 1.0 NLP challenge. In: Proceedings of first international workshop on medication and adverse drug event detection. 2018. vol. 90, pp. 57–65.

Download references

Author information

Authors and Affiliations

  1. IBM Research, Yorktown Heights, NY, USA

    Bharath Dandala, Venkata Joopudi & Murthy Devarakonda

  2. Biomedical Informatics, Arizona State University, Tempe, USA

    Murthy Devarakonda

Authors
  1. Bharath Dandala
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Venkata Joopudi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Murthy Devarakonda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Murthy Devarakonda.

Ethics declarations

Funding

No sources of funding were used to conduct this study or prepare this manuscript.

Approval and consent

This study was conducted on de-identified clinical notes as part of a shared challenge, so no ethical approval or patient consent was required.

Conflict of interest

Bharath Dandala, Venkata Joopudi, and Murthy Devarakonda have no conflicts of interest that are directly relevant to the content of this article. Dr. Devarakonda is now on the faculty in Biomedical Informatics at Arizona State University, USA.

Additional information

Part of a theme issue on "NLP Challenge for Detecting Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0)" guest edited by Feifan Liu, Abhyuday Jagannatha and Hong Yu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dandala, B., Joopudi, V. & Devarakonda, M. Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks. Drug Saf 42, 135–146 (2019). https://doi.org/10.1007/s40264-018-0764-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-018-0764-x

Search

Navigation