Skip to main content

Empirical study of sentiment analysis tools and techniques on societal topics

Journal of Intelligent Information Systems volume 56, pages 379–407 (2021)Cite this article

Abstract

A surge in public opinions mining against various societal topics using publicly available off-the-shelf sentiment analysis tools is evident in recent times. Since sentiment analysis is a domain-dependent problem, and the majority of the tools are built for customer reviews, the suitability of using such existing off-the-the-shelf tools for a societal topic is subject to investigation. None of the existing studies has thoroughly investigated on societal issues. This paper systematically evaluates the performance of 10 popularly used off-the-shelf tools and 17 state-of-the-art machine learning techniques and investigates their strengths and weaknesses using various societal and non-societal topics.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. http://sentistrength.wlv.ac.uk

  2. http://www.sentiment140.com/

  3. https://cran.r-project.org/web/packages/RSentiment/index.html

  4. http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

  5. https://www.nltk.org/_modules/nltk/sentiment/vader.html

  6. https://www.meaningcloud.com

  7. One of the state of USA

  8. https://en.wikipedia.org/wiki/Hurricane_Harvey

  9. https://en.wikipedia.org/wiki/Hurricane_Sandy

  10. https://cran.r-project.org/package=RSentiment

  11. https://en.wikipedia.org/wiki/Earth_Hour

  12. Obama-McCain Debate

  13. http://services.gate.ac.uk/decarbonet/sentiment/api.html

  14. https://en.wikipedia.org/wiki/Earth_Hour

  15. https://en.wikipedia.org/wiki/Stance_(linguistics)

  16. https://www.cs.york.ac.uk/semeval-2013/task2/

  17. http://saifmohammad.com/WebPages/StanceDataset.htm

  18. http://help.sentiment140.com/for-students/

  19. http://www.yelp.com/dataset_challenge

  20. https://en.wikipedia.org/wiki/Syrian_Civil_War

  21. https://en.wikipedia.org/wiki/Paris_Agreement

  22. Opinionated text in Twitter

  23. http://docs.tweepy.org

  24. https://www.meaningcloud.com/developer/sentiment-analysis

  25. Application Programming Interface

  26. http://sentistrength.wlv.ac.uk

  27. https://indico.io

  28. http://help.sentiment140.com/for-students

  29. https://www.rdocumentation.org/packages/RSentiment/versions/2.2.2

  30. http://corpustext.com/reference/sentiment_afinn.html

  31. https://www.clips.uantwerpen.be/pages/pattern-en

  32. https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

  33. http://sentiwordnet.isti.cnr.it/

  34. https://github.com/cjhutto/vaderSentiment

  35. http://scikit-learn.org/stable/index.html

  36. https://keras.io/

  37. https://en.wikipedia.org/wiki/List_of_emoticons

  38. http://kt.ijs.si/data/Emoji_sentiment_ranking/

References

  • Abbasi, A., Hassan, A., & Dhar, M. (2014). Benchmarking twitter sentiment analysis tools. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC), 14, pp 26–31.

  • Akhtar, M.S., Kumar, A., Ekbal, A., & Bhattacharyya, P. (2016). A hybrid deep learning architecture for sentiment analysis. In Proceedings of the 26th International Conference on Computational Linguistics (COLING): Technical Papers, pp 482–493.

  • Al-Smadi, M., Qawasmeh, O., Al-Ayyoub, M., Jararweh, Y., & Gupta, B. (2017). Deep recurrent neural network vs. support vector machine for aspect-based sentiment analysis of arabic hotels’ reviews. Journal of Computational Science.

  • Baly, R., Hobeica, R., Hajj, H., El-Hajj, W., Shaban, K.B., & Al-Sallab, A. (2016). A meta-framework for modeling the human reading process in sentiment analysis. ACM Transactions on Information Systems (TOIS), 35(1), 1–21.

    Article  Google Scholar 

  • Burnap, P., Williams, M.L., Sloan, L., Rana, O., Housley, W., Edwards, A., Knight, V., Procter, R., & Voss, A. (2014). Tweeting the terror: modelling the social media reaction to the woolwich terrorist attack. Social Network Analysis and Mining, 4(1), 206.

    Article  Google Scholar 

  • Cao, X., MacNaughton, P., Deng, Z., Yin, J., Zhang, X., & Allen, J.G. (2018). Using twitter to better understand the spatiotemporal patterns of public sentiment: A case study in massachusetts, usa. International journal of environmental research and public health, 15(2), 250.

    Article  Google Scholar 

  • Catal, C., & Nangir, M. (2017). A sentiment classification model based on multiple classifiers. Applied Soft Computing, 50, 135–141.

    Article  Google Scholar 

  • Chen, S., Mao, J., Li, G., Ma, C., & Cao, Y. (2020). Uncovering sentiment and retweet patterns of disaster-related tweets from a spatiotemporal perspective–a case study of hurricane harvey. Telematics and Informatics, 47, 101326.

    Article  Google Scholar 

  • Church, K.W., & Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational linguistics, 16(1), 22–29.

    Google Scholar 

  • Cui, H., Mittal, V., & Datar, M. (2006). Comparative experiments on sentiment classification for online product reviews. In Proceedings of the 21st National Conference on Artificial Intelligence, AAAI Press, 2, pp 1265–1270.

  • dos Santos, C., & Gatti, M. (2014). Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of the 25th International Conference on Computational Linguistics (COLING): Technical Papers, pp 69–78.

  • Garay, J., Yap, R., & Sabellano, M.J. (2019). An analysis on the insights of the anti-vaccine movement from social media posts using k-means clustering algorithm and vader sentiment analyzer. IOP Conference Series: Materials Science and Engineering, 482(1), 012043.

    Article  Google Scholar 

  • Giachanou, A., & Crestani, F. (2016). Like it or not: A survey of twitter sentiment analysis methods. ACM Computing Surveys (CSUR), 49(2), 28.

    Article  Google Scholar 

  • Gonçalves, P., Araújo, M., Benevenuto, F., & Cha, M. (2013). Comparing and combining sentiment analysis methods. In Proceedings of the first ACM conference on Online social networks, pp 27–38.

  • Huang, M., Qian, Q., & Zhu, X. (2017). Encoding syntactic knowledge in neural networks for sentiment classification. ACM Transactions on Information Systems (TOIS), 35(3), 1–27.

    Article  Google Scholar 

  • Jianqiang, Z., Xiaolin, G., & Xuejun, Z. (2018). Deep convolution neural networks for twitter sentiment analysis. IEEE Access, 6, 23253–23260.

    Article  Google Scholar 

  • Karamibekr, M., & Ghorbani, A.A. (2012). Sentiment analysis of social issues. In Proceedings of the International Conference on Social Informatics (SocialInformatics), pp 215–221.

  • Kouloumpis, E., Wilson, T., & Moore, J.D. (2011). Twitter sentiment analysis: The good the bad and the omg!. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), p 164.

  • Kušen, E., & Strembeck, M. (2018). Politics, sentiments, and misinformation: An analysis of the twitter discussion on the 2016 austrian presidential elections. Online Social Networks and Media, 5, 37–50.

    Article  Google Scholar 

  • Lerman, K., Arora, M., Gallegos, L., Kumaraguru, P., & Garcia, D. (2016). Emotions, demographics and sociability in twitter interactions. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), pp 201–210.

  • Lu, Y., Sakamoto, K., Shibuki, H., & Mori, T. (2017). Are deep learning methods better for twitter sentiment analysis?. In Proceedings of the 23rd Annual Meeting of Natural Language Processing (Japan), pp 787–790.

  • Maynard, D., & Bontcheva, K. (2016). Challenges of evaluating sentiment analysis tools on social media. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pp 1142–1148.

  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in neural information processing systems, pp 3111–3119.

  • Mohammad, S.M., Sobhani, P., & Kiritchenko, S. (2017). Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3), 26.

    Article  Google Scholar 

  • Mohammad, S.M., Zhu, X., Kiritchenko, S., & Martin, J. (2015). Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4), 480–499.

    Article  Google Scholar 

  • Mostafa, A.M. (2017). An evaluation of sentiment analysis and classification algorithms for arabic textual data. International Journal of Computer Applications, 158(3), 1–8.

    Article  Google Scholar 

  • Neppalli, V.K., Caragea, C., Squicciarini, A., Tapia, A., & Stehle, S. (2017). Sentiment analysis during hurricane sandy in emergency response. International Journal of Disaster Risk Reduction, 21, 213–222.

    Article  Google Scholar 

  • Nguyen, H., & Nguyen, M.-L. (2018). A deep neural architecture for sentence-level sentiment classification in twitter social networking. In Proceedings of the Computational Linguistics, pp 15–27.

  • On, J., Park, H.-A., & Song, T.-M. (2019). Sentiment analysis of social media on childhood vaccination: Development of an ontology. Journal of medical Internet research, 21(6), e13456.

    Article  Google Scholar 

  • Ouyang, X., Zhou, P., Li, CH, & Liu, L. (2015). Sentiment analysis using convolutional neural network. In Proceedings of the IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), pp 2359–2364.

  • Öztürk, N., & Ayvaz, S. (2018). Sentiment analysis on twitter: A text mining approach to the syrian refugee crisis. Telematics and Informatics, 35(1), 136–147. https://doi.org/10.1016/j.tele.2017.10.006.

    Article  Google Scholar 

  • Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In LREc, 10, pp 1320–1326.

  • Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing (EMNLP), 10, pp 79–86.

  • Ribeiro, F.N., Araújo, M., Gonçalves, P., Gonçalves, M.A., & Benevenuto, F. (2016). Sentibench-a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1), 23.

    Article  Google Scholar 

  • Saif, H., He, Y., Fernandez, M., & Alani, H. (2016). Contextual semantics for sentiment analysis of twitter. Information Processing & Management, 52 (1), 5–19.

    Article  Google Scholar 

  • Severyn, A., & Moschitti, A. (2015). Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 959–962.

  • Shannon, C.E. (2001). A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1), 3–55.

    MathSciNet  Article  Google Scholar 

  • Silva, N.F.F.D., Coletta, L.F.S., & Hruschka, E.R. (2016). A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Computing Surveys (CSUR), 49(1), 1–26.

    Google Scholar 

  • Singh, P., Sawhney, R.S., & Kahlon, K.S. (2018). Sentiment analysis of demonetization of 500 & 1000 rupee banknotes by indian government. ICT Express, 4(3), 124–129. https://doi.org/10.1016/j.icte.2017.03.001.

    Article  Google Scholar 

  • Sobhani, P., Mohammad, S., & Kiritchenko, S. (2016). Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pp 159–169.

  • Tumasjan, A., Sprenger, T.O., Sandner, P.G., & Welpe, I.M. (2010). Predicting elections with twitter: What 140 characters reveal about political sentiment. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), 10(1), 178–185.

    Google Scholar 

  • Vargas, S., McCreadie, R., Macdonald, C., & Ounis, I. (2016). Comparing overall and targeted sentiments in social media during crises. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), pp 695–698.

  • Vilares, D., Alonso, M.A., & Gómez-Rodríguez, C. (2017). Supervised sentiment analysis in multilingual environments. Information Processing & Management, 53(3), 595–607.

    Article  Google Scholar 

  • Xia, R., Zong, C., & Li, S. (2011). Ensemble of feature sets and classification algorithms for sentiment classification. Information Sciences, 181(6), 1138–1152.

    Article  Google Scholar 

  • Zhou, G.-Y., & Huang, J.X. (2017). Modeling and mining domain shared knowledge for sentiment analysis. ACM Transactions on Information Systems (TOIS), 36(2), 1–36.

    Article  Google Scholar 

Download references

Acknowledgments

This work is partially funded by the Ministry of Electronics & Information Technology, Government of India.

Author information

Authors and Affiliations

  1. Indian Institute of Technology Guwahati, Assam, India

    Loitongbam Gyanendro Singh & Sanasam Ranbir Singh

Authors
  1. Loitongbam Gyanendro Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanasam Ranbir Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Loitongbam Gyanendro Singh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Singh, L.G., Singh, S.R. Empirical study of sentiment analysis tools and techniques on societal topics. J Intell Inf Syst 56, 379–407 (2021). https://doi.org/10.1007/s10844-020-00616-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-020-00616-7

Keywords

  • Sentiment analysis
  • Societal topics
  • Publicly available sentiment analysis tools
  • Machine learning techniques