Assessment of Word Embedding Techniques for Identification of Personal Experience Tweets Pertaining to Medication Uses

  • Keyuan JiangEmail author
  • Shichao Feng
  • Ricardo A. Calix
  • Gordon R. Bernard
Part of the Studies in Computational Intelligence book series (SCI, volume 843)


Twitter, a general purpose social media service, has seen growing interests as an active data source for possible use of post-market surveillance of medicinal products. Being able to identify Twitter posts of personal experience related to medication use is as important as being able to identify expressions of adverse medical events/reactions for the surveillance purpose. Identifying personal experience tweets is a challenging task, especially in the aspect of engineering features for classification. Word embedding has become a superior alternative to engineered features in many text classification applications. To investigate if word embedding-based methods can perform constantly better than conventional classification methods with engineered features, we assessed the classification performance of 4 word embedding techniques: GloVe, word2vec, fastText, and wordRank. Using a corpus of 22 million unlabeled tweets for learning of word embedding and a corpus of 12,331 annotated tweets for classification, we discovered that word embedding-based classification methods consistently outperform the engineered feature-based classification methods with statistical significance of p < 0.01, but there exist no significantly statistical differences among the 4 study word embedding methods (p < 0.05).


Pharmacovigilance Twitter Word embedding Natural language processing Classification Personal experience 



Authors wish to thank anonymous reviewers in critiquing our work and providing constructive comments that improved the manuscript. Authors wish to acknowledge these individuals for their contribution to this project: Dustin Franz, Ravish Gupta for collecting the Twitter data, Alexandra Vest, Cecelia Lai, Bridget Swindell, Mary Stroud, and Matrika Gupta for annotating the tweets. This work was supported by the National Institutes of Health Grant 1R15LM011999–01.


  1. 1.
    Alvaro, N., Conway, M., Doan, S., Lofi, C., Overington, J., Collier, N.: Crowdsourcing Twitter annotations to identify first-hand experiences of prescription drug use. J. Biomed. Inform. 58, 280–287 (2015)CrossRefGoogle Scholar
  2. 2.
    Baroni, M., Dinu, G., Kruszewski, G.: Don’t count, predict! A systematic comparison of context-counting versus context-predicting semantic vectors. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 238–247 (2014) (Volume 1: Long Papers)Google Scholar
  3. 3.
    Bian, J., Topaloglu, U., Yu, F.: Towards large-scale twitter mining for drug-related adverse events. In: Proceedings of the 2012 International Workshop on Smart Health and Wellbeing, pp. 25–32. ACM (2012)Google Scholar
  4. 4.
    Bojanowski, P., Grave, E., Joulin, A., Mikolov, T. Enriching word vectors with subword information (2016). arXiv:1607.04606
  5. 5.
    Calix, R.A., Gupta, R., Gupta, M., Jiang, K.: Deep gramulator: Improving precision in the classification of personal health-experience tweets with deep learning. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1154–1159. IEEE (2017)Google Scholar
  6. 6.
    Cocos, A., Fiks, A.G., Masino, A.J.: Deep learning for pharmacovigilance: recurrent neural network architectures for labeling adverse drug reactions in Twitter posts. J. Am. Med. Inform. Assoc. 24(4), 813–821 (2017)CrossRefGoogle Scholar
  7. 7.
    Eshleman, R., Singh, R.: Leveraging graph topology and semantic context for pharmacovigilance through twitter-streams. BMC Bioinform. 17(13), 335 (2016)CrossRefGoogle Scholar
  8. 8.
    Freifeld, C.C., Brownstein, J.S., Menone, C.M., Bao, W., Filice, R., Kass-Hout, T., Dasgupta, N.: Digital drug safety surveillance: monitoring pharmaceutical products in twitter. Drug Saf. 37(5), 343–350 (2014)CrossRefGoogle Scholar
  9. 9.
    Hazell, L., Shakir, S.A.: Under-reporting of adverse drug reactions. Drug Saf. 29(5), 385–396 (2006)CrossRefGoogle Scholar
  10. 10.
    Ji, S., Yun, H., Yanardag, P., Matsushima, S., Vishwanathan, S.V.N.: WordRank: Learning Word Embeddings via Ro-bust Ranking. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 658–668 (2016)Google Scholar
  11. 11.
    Jiang, K., Calix, R., Gupta, M.: Construction of a personal experience tweet Corpus for health surveillance. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing, pp. 128–135 (2016)Google Scholar
  12. 12.
    Jiang, K., Zheng, Y.: Mining twitter data for potential drug effects. In: International Conference on Advanced Data Mining and Applications, pp. 434–443. Springer, Berlin (2013)CrossRefGoogle Scholar
  13. 13.
    Jiang, K., Chen, T., Calix, R.A., Bernard, G.R.: Identifying consumer health terms of side effects in twitter posts. Stud. Health Technol. Inform. 251, 273 (2018)Google Scholar
  14. 14.
    Jiang, K., Feng, S., Song, Q., Calix, R.A., Gupta, M., Bernard, G.R.: Identifying tweets of personal health experience through word embedding and LSTM neural network. BMC Bioinform. 19(8), 210 (2018)CrossRefGoogle Scholar
  15. 15.
    Koutkias, V.G., Lillo-Le Louët, A., Jaulent, M.C.: Exploiting heterogeneous publicly available data sources for drug safety surveillance: computational framework and case studies. Expert. Opin. Drug Saf. 16(2), 113–124 (2017)CrossRefGoogle Scholar
  16. 16.
    Lardon, J., Bellet, F., Aboukhamis, R., Asfari, H., Souvignet, J., Jaulent, M.C., Beyens, M., Lillo-LeLouët, A., Bousquet, C.: Evaluating Twitter as a complementary data source for pharmacovigilance. Expert. Opin. Drug Saf. 17(8), 763–774 (2018)CrossRefGoogle Scholar
  17. 17.
    Leaman, R., Wojtulewicz, L., Sullivan, R., Skariah, A., Yang, J., Gonzalez, G.: Towards internet-age pharmacovigilance: extracting adverse drug reactions from user posts to health-related social networks. In: Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, pp. 117–125. Association for Computational Linguistics (2010)Google Scholar
  18. 18.
    Medicines and Healthcare products Regulatory Agency: UK regulator leads innovative EU project on the use of smartphones and social media for drug safety information (2014)Google Scholar
  19. 19.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In Proceedings of Workshop at ICLR (2013)Google Scholar
  20. 20.
    O’Connor, K., Pimpalkhute, P., Nikfarjam, A., Ginn, R., Smith, K. L., & Gonzalez, G.: Pharmacovigilance on twitter? Mining tweets for adverse drug reactions. In: AMIA Annual Symposium Proceedings, p. 924. American Medical Informatics Association (2014).Google Scholar
  21. 21.
    Pennington, J., Socher, R., & Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)Google Scholar
  22. 22.
    Pierce, C.E., Bouri, K., Pamer, C., Proestel, S., Rodriguez, H.W., Van Le, H., Freifeld, C.C., Brownstein, J.S., Walderhaug, M., Edwards, I.R., Dasgupta, N.: Evaluation of facebook and twitter monitoring to detect safety signals for medical products: an analysis of recent fda safety alerts. Drug Saf. 40(4), 317–331 (2017)CrossRefGoogle Scholar
  23. 23.
    Powell, G.E., Seifert, H.A., Reblin, T., Burstein, P.J., Blowers, J., Menius, J.A., Painter, J.L., Thomas, M., Pierce, C.E., Rodriguez, H.W., Brownstein, J.S., Freifeld, C.C., Bell, H.G., Dasgupta, N.: Social media listening for routine post-marketing safety surveillance. Drug Saf. 39(5), 443–454 (2016)CrossRefGoogle Scholar
  24. 24.
    Wijeratne, S., Sheth, A., Bhatt, S., Balasuriya, L., Al-Olimat, H.S., Gaur, M., Yazdavar, A.H., Thirunarayan, K.: Feature Engineering for Twitter-based Applications. Feature Engineering for Machine Learning and Data Analytics, vol. 35 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Keyuan Jiang
    • 1
    Email author
  • Shichao Feng
    • 2
  • Ricardo A. Calix
    • 1
  • Gordon R. Bernard
    • 3
  1. 1.Purdue University NorthwestHammondUSA
  2. 2.University of North TexasDentonUSA
  3. 3.Vanderbilt UniversityNashvilleUSA

Personalised recommendations