TwitterBERT: Framework for Twitter Sentiment Analysis Based on Pre-trained Language Model Representations

  • Noureddine AzzouzaEmail author
  • Karima Akli-Astouati
  • Roliana Ibrahim
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1073)


Sentiment analysis has been a topic of discussion in the exploration domain of language understanding. Yet, the neural networks deployed in it are deficient to some extent. Currently, the majority of the studies proceeds on identifying the sentiments by focusing on vocabulary and syntax. Moreover, the task is recognised in Natural Language Processing (NLP) and, for calculating the noteworthy and exceptional results, Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) have been employed. In this study, we propose a four-phase framework for Twitter Sentiment Analysis. This setup is based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model as an encoder for generating sentence depictions. For more effective utilisation of this model, we deploy various classification models. Additionally, we concatenate pre-trained representations of word embeddings with BERT representation method to enhance sentiment classification. Experimental results show better implementation when it is evaluated against the baseline framework on all datasets. For example, our best model attains an F1-score of 71.82% on the SemEval 2017 dataset. A comparative analysis on experimental results offers some recommendations on choosing pre-training steps to obtain improved results. The outcomes of the experiment confirm the effectiveness of our system.


Twitter Sentiment Analysis Word embedding CNN LSTM BERT 


  1. 1.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations ICLR 2015 (2015)Google Scholar
  2. 2.
    Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: 31st International Conference on Machine Learning, vol. 32, pp. II–1188–II–1196 (2014)Google Scholar
  3. 3.
    Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.H.: Hierarchical attention networks for document classification. In: HLT-NAACL 2016, pp. 1480–1489 (2016)Google Scholar
  4. 4.
    Jianqiang, Z., Xiaolin, G., Xuejun, Z.: Deep convolution neural networks for Twitter sentiment analysis. IEEE Access 6, 23253–23260 (2018)CrossRefGoogle Scholar
  5. 5.
    Zhang, L., Zhou, Y., Duan, X., Chen, R.: A hierarchical multi-input and output Bi-GRU model for sentiment analysis on customer reviews. In: IOP Conference Series: Materials Science and Engineering, vol. 322, no. 6 (2018)Google Scholar
  6. 6.
    Abid, F., Alam, M., Yasir, M., Li, C.: Sentiment analysis through recurrent variants latterly on convolutional neural network of Twitter. Future Gener. Comput. Syst. 95, 292–308 (2019)CrossRefGoogle Scholar
  7. 7.
    Peters, M., et al.: Deep contextualized word representations. In: NAACL-HLT 2018, pp. 2227–2237 (2018)Google Scholar
  8. 8.
    Ramachandran, P., Liu, P., Le, Q.: Unsupervised pretraining for sequence to sequence learning. In: EMNLP 2017, pp. 383–391 (2017)Google Scholar
  9. 9.
    Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: ACL, no. 1, pp. 328–339 (2018)Google Scholar
  10. 10.
    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bi-directional transformers for language understanding. In: NAACL-HLT 2019, pp. 4171–4186 (2019)Google Scholar
  11. 11.
    Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision.pdf. CS224 N Project Report (2009)Google Scholar
  12. 12.
    Kiritchenko, S., Zhu, X., Mohammad, S.M.: Sentiment analysis of short informal texts. J. Artif. Intell. Res. 50, 723–762 (2014)CrossRefGoogle Scholar
  13. 13.
    Baccianella, S., Esuli, A., Sebastiani, F.: SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of Lr, vol. 10, pp. 2200–2204 (2010)Google Scholar
  14. 14.
    Strapparava, C., Valitutti, A.: WordNet-affect: an affective extension of WordNet. In: Proceedings of 4th International Conference on Language Resources and Evaluation, vol. 4, pp. 1083–1086 (2004)Google Scholar
  15. 15.
    Giahanou, A., Crestani, F.: Like it or not: a survey of Twitter sentiment analysis methods. ACM Comput. Surv. 49, 28:1–28:41 (2016)Google Scholar
  16. 16.
    Arslan, Y., Küçük, D., Birturk, A.: Twitter sentiment analysis experiments using word embeddings on datasets of various scales. In: NLDB 2018, pp. 40–47 (2018)Google Scholar
  17. 17.
    Vo, D.T., Zhang, Y.: Target-dependent Twitter sentiment classification with rich automatic features. In: IJCAI 2015, pp. 1347–1353 (2015)Google Scholar
  18. 18.
    Zhang, P., He, Z.: Using data-driven feature enrichment of text representation and ensemble technique for sentence-level polarity classification. J. Inf. Sci. 41(4), 531–549 (2015)CrossRefGoogle Scholar
  19. 19.
    Liao, S., Wang, J., Yu, R., Sato, K., Cheng, Z.: CNN for situations understanding based on sentiment analysis of Twitter data. Procedia Comput. Sci. 111(2015), 376–381 (2017)CrossRefGoogle Scholar
  20. 20.
    Balikas, G., Moura, S., Amini, M.-R.: Multitask learning for fine-grained twitter sentiment analysis. In: SIGIR 2017, pp. 1005–1008 (2017)Google Scholar
  21. 21.
    Chen, T., Xu, R., He, Y., Wang, X.: Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Syst. Appl. 72, 221–230 (2017)CrossRefGoogle Scholar
  22. 22.
    Kamkarhaghighi, M., Makrehchi, M.: Content Tree Word Embedding for document representation. Expert Syst. Appl. 90, 241–249 (2017)CrossRefGoogle Scholar
  23. 23.
    McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: contextualized word vectors. In: NIPS 2017, pp. 6297–6308 (2017)Google Scholar
  24. 24.
    Radford, A., Salimans, T.: Improving language understanding by generative pre-training. Technical report, OpenAI, pp. 1–12 (2018)Google Scholar
  25. 25.
    Vadicamo, L., et al.: Cross-media learning for image sentiment analysis in the wild. In: ICCV Workshops 2017, pp. 308–317 (2017)Google Scholar
  26. 26.
    Godin, F., Vandersmissen, B., De Neve, W., Van de Walle, R.: Named entity recognition for Twitter microposts using distributed word representations. In: NUT@IJCNLP 2015, pp. 146–153 (2015)Google Scholar
  27. 27.
    Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: EMNLP 2014, pp. 1532–1543 (2014)Google Scholar
  28. 28.
    Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: LREC 2018 (2018)Google Scholar
  29. 29.
    Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: an open multilingual graph of general knowledge. In: AAAI 2017, pp. 4444–4451 (2017)Google Scholar
  30. 30.
    Nakov, P., Rosenthal, S., Kozareva, Z., Stoyanov, V., Ritter, A., Wilson, T.: SemEval-2013 Task 2: sentiment analysis in Twitter. In: SemEval@NAACL-HLT 2013, pp. 312–320 (2013)Google Scholar
  31. 31.
    Rosenthal, S., Ritter, A., Nakov, P., Stoyanov, V.: SemEval-2014 Task 9: sentiment analysis in Twitter. In: SemEval@COLING, pp. 73–80 (2014)Google Scholar
  32. 32.
    Rosenthal, S., Nakov, P., Kiritchenko, S., Mohammad, S., Ritter, A., Stoyanov, V.: SemEval-2015 Task 10: sentiment analysis in Twitter. In: SemEval@NAACL-HLT 2015, pp. 451–463 (2015)Google Scholar
  33. 33.
    Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., Stoyanov, V.: SemEval-2016 Task 4: sentiment analysis in Twitter. In: SemEval@NAACL-HLT 2016, pp. 1–18 (2016)Google Scholar
  34. 34.
    Rosenthal, S., Farra, N., Nakov, P.: SemEval-2017 Task 4: sentiment analysis in Twitter. In: SemEval@ACL 2017, pp. 502–518 (2017)Google Scholar
  35. 35.
    Kenyon-Dean, K., et al.: Sentiment Analysis: It’s Complicated! In: NAACL-HLT 2018, pp. 1886–1895 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Noureddine Azzouza
    • 1
    Email author
  • Karima Akli-Astouati
    • 1
  • Roliana Ibrahim
    • 2
  1. 1.FEI - Department of Computer Science, RIIMA LaboratoryUniversity of Science and Technology Houari BoumedieneBab ezzouarAlgeria
  2. 2.School of Computing, Faculty of EngineeringUniversiti Teknologi Malaysia (UTM)Johor BahruMalaysia

Personalised recommendations