Advertisement

An Operational Deep Learning Pipeline for Classifying Life Events from Individual Tweets

  • Xinsong Du
  • Jiang Bian
  • Mattia ProsperiEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 898)

Abstract

We here present an operational deep learning pipeline for classifying life events from individual tweets, using job loss as a use case and Twitter data collected between 2010 and 2013 (historic sample from the public stream). The pipeline includes identification of keywords through snowball sampling, multiple rater manual annotation, supervised deep learning, text processing (word embedding, bag of words) and architecture selection (convolutional, shallow-and-wide convolutional, and long-short-term memory) with parameter optimization, external validation and feedback learning. After model optimization, a shallow-and-wide network with a pre-trained 200-dimensional word2vec achieved a precision of 78% (over an average single keyword precision of 50%) and an area under receiver operating characteristic of 86%. Precision and recall also increased by 5% using bag of words. When tested on tweets with ambiguous annotations (i.e. tweets that were hard for human annotators to classify), the network achieved 65% precision. Finally, on a random set of tweets that did not contain any of the snowballed keywords, 30% were classified as job loss events; this putatively false positive set can be used to reinforce the learner’s training. In conclusion, the pipeline streamlines both the manual and automated process, providing feedback reinforcement (snowballing and external tweets), and shows good performance on classifying individual tweets on the use case, potentially saving human resources needed to collate such data for research studies.

Keywords

Deep learning Job loss Twitter Classification 

Notes

Acknowledgements

MP, JB, and XD are in part supported by US NSF grant SES 1734134.

References

  1. 1.
    Reece, A.G., Reagan, A.J., Lix, K.L.M., Dodds, P.S., Danforth, C.M., Langer, E.J.: Forecasting the onset and course of mental illness with Twitter data. Sci. Rep. 7(1), 13006 (2017).  https://doi.org/10.1038/s41598-017-12961-9CrossRefGoogle Scholar
  2. 2.
    Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010).  https://doi.org/10.1177/0261927X09351676CrossRefGoogle Scholar
  3. 3.
    Alsaedi, N., Burnap, P., Rana, O.: Can we predict a riot? Disruptive event detection using Twitter. ACM Trans. Internet Technol. 17(2), 18:1–18:26 (2017).  https://doi.org/10.1145/2996183CrossRefGoogle Scholar
  4. 4.
    Sumner, C., Byers, A., Boochever, R., Park, G.J.: Predicting dark triad personality traits from Twitter usage and a linguistic analysis of tweets. In: 2012 11th International Conference on Machine Learning and Applications, vol. 2, pp. 386–393 (2012).  https://doi.org/10.1109/ICMLA.2012.218
  5. 5.
    Makazhanov, A., Rafiei, D.: Predicting political preference of Twitter users. In: 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013), pp. 298–305 (2013).  https://doi.org/10.1145/2492517.2492527
  6. 6.
    Conover, M., Gonçalves, B., Ratkiewicz, J., Flammini, A., Menczer, F.: Predicting the political alignment of Twitter users. In: 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pp. 192–199 (2011).  https://doi.org/10.1109/PASSAT/SocialCom.2011.34
  7. 7.
    Ramos, J.P.: Using TF-IDF to determine word relevance in document queries. Presented at the First International Conference on Machine Learning, New Brunswick, NJ, USA (2003)Google Scholar
  8. 8.
    Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42(1–2), 177–196 (2001).  https://doi.org/10.1023/A:1007617005950CrossRefzbMATHGoogle Scholar
  9. 9.
    Won, D., Steinert-Threlkeld, Z.C., Joo, J.: Protest activity detection and perceived violence estimation from social media images. ArXiv:1709.06204 [Cs] (2017). http://arxiv.org/abs/1709.06204
  10. 10.
    Zhang, Z., He, Q., Gao, J., Ni, M.: A deep learning approach for detecting traffic accidents from social media data. Transp. Res. Part C: Emerg. Technol. 86, 580–596 (2018).  https://doi.org/10.1016/j.trc.2017.11CrossRefGoogle Scholar
  11. 11.
    Founta, A.-M., Chatzakou, D., Kourtellis, N., Blackburn, J., Vakali, A., Leontiadis, I.: A unified deep learning architecture for abuse detection. ArXiv:1802.00385 [Cs] (2018). http://arxiv.org/abs/1802.00385
  12. 12.
    Badjatiya, P., Gupta, S., Gupta, M., Varma, V.: Deep learning for hate speech detection in tweets. In: Proceedings of the 26th International Conference on World Wide Web Companion, pp. 759–760. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2017).  https://doi.org/10.1145/3041021.3054223
  13. 13.
    Pitsilis, G.K., Ramampiaro, H., Langseth, H.: Detecting offensive language in tweets using deep learning. ArXiv:1801.04433 [Cs] (2018). http://arxiv.org/abs/1801.04433
  14. 14.
    Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation, pp. 1532–1543. Association for Computational Linguistics (2014).  https://doi.org/10.3115/v1/D14-1162
  15. 15.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality, p. 9 (n.d.)Google Scholar
  16. 16.
    Bian, J., et al.: Mining Twitter to assess the public perception of the “internet of things”. PLoS One 11(7), e0158450 (2016).  https://doi.org/10.1371/journal.pone.0158450CrossRefGoogle Scholar
  17. 17.
    McHugh, M.L.: Interrater reliability: the kappa statistic. Biochem. Med. 22(3), 276–282 (2012)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Goodman, L.A.: Snowball sampling. Ann. Math. Stat. 32(1), 148–170 (1961).  https://doi.org/10.1214/aoms/1177705148MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Nadeau, C., Bengio, Y.: Inference for the generalization error. In: Proceedings of the 12th International Conference on Neural Information Processing Systems, pp. 307–313. MIT Press, Cambridge (1999). http://dl.acm.org/citation.cfm?id=3009657.3009701
  20. 20.
    LeCun, Y., Bengio, Y.: Convolutional networks for images, speech, and time-series, p. 15 (n.d.)Google Scholar
  21. 21.
    Le, H.T., Cerisara, C., Denis, A.: Do convolutional networks need to be deep for text classification? ArXiv:1707.04108 [Cs] (2017). http://arxiv.org/abs/1707.04108
  22. 22.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997).  https://doi.org/10.1162/neco.1997.9.8.1735CrossRefGoogle Scholar
  23. 23.
    Das, S., Wong, W.K., Dietterich, T., Fern, A., Emmott, A.: Incorporating expert feedback into active anomaly discovery. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 853–858 (2016).  https://doi.org/10.1109/ICDM.2016.0102
  24. 24.
    Yin, W., Kann, K., Yu, M., Schütze, H.: Comparative study of CNN and RNN for natural language processing (2017). https://arxiv.org/abs/1702.01923
  25. 25.
    Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. ArXiv:1509.01626 [Cs] (2015). http://arxiv.org/abs/1509.01626
  26. 26.
    The Xapian Project (n.d.). https://xapian.org/. Assessed 28 June 2018
  27. 27.
    Welcome to Python.org (n.d.). https://www.python.org/. Assessed 28 June 2018
  28. 28.
    TensorFlow (n.d.). https://www.tensorflow.org/. Assessed 28 June 2018
  29. 29.
    Palinkas, L.A., Horwitz, S.M., Green, C.A., Wisdom, J.P., Duan, N., Hoagwood, K.: Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm. Policy Ment. Health 42(5), 533–544 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Health Outcomes and Biomedical InformaticsUniversity of FloridaGainesvilleUSA
  2. 2.Department of EpidemiologyUniversity of FloridaGainesvilleUSA

Personalised recommendations