Skip to main content

Constructing Sentiment Lexicon with Game for Annotation Collection

  • 111 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 13062)


While research of sentiment analysis became very popular on the global scope, in Slovak language as an under-resourced language there are still many issues to be tackled, especially the lack of resources. In this paper, we introduce a sentiment analysis game designed to collect sentiment annotations. The game is intended for a single player who, motivated by game score, chooses the sentiment category of each word of a sentence. We describe the annotation collection process during which over 12 500 annotations of individual words and over 1 000 annotations of entire sentences were obtained within a week. The collected annotations were used to construct a sentiment lexicon. Using artificial bee colony algorithm, optimal lexicon construction parameters were discovered. To evaluate the final lexicon’s usefulness, we applied simple sentence-level lexicon-based sentiment analysis methods on a manually annotated dataset of mobile phone reviews. The same was done with other existing lexicons for comparison. The results of our experiments show that collecting annotations using our game can be useful as a method for constructing a sentiment lexicon.


  • Lexicons and dictionaries
  • Sentiment analysis
  • Crowdsourcing
  • Games with a purpose
  • Natural language processing

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-89579-2_4
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-89579-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. 1.

  2. 2.

    The name could be loosely translated to Sentiment Devourer.

  3. 3.

  4. 4.

    Over 100 words with intensity 3 were removed.

  5. 5.

  6. 6.


  1. von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04), pp. 319–326. Association for Computing Machinery, New York (2004)

    Google Scholar 

  2. Alqaryouti, O., Siyam, N., Shaalan, K.: A sentiment analysis lexical resource and dataset for government smart apps domain. In: Hassanien, A.E., Tolba, M.F., Shaalan, K., Azar, A.T. (eds.) AISI 2018. AISC, vol. 845, pp. 230–240. Springer, Cham (2019).

    CrossRef  Google Scholar 

  3. Angwin, J., Larson, J., Mattu, S., Kirchner, L.: Machine bias, May 2016. Accessed 23 May 2016

  4. Bouazizi, M., Ohtsuki, T.: Sentiment analysis: from binary to multi-class classification: a pattern-based approach for multi-class sentiment analysis in twitter. In: 2016 IEEE International Conference on Communications (ICC), pp. 1–6 (2016)

    Google Scholar 

  5. Bučar, J., Žnidaršič, M., Povh, J.: Annotated news corpora and a lexicon for sentiment analysis in Slovene. Lang. Resour. Eval. 52, 895–919 (2018)

    CrossRef  Google Scholar 

  6. Chen, Y., Skiena, S.: Building sentiment lexicons for all major languages. In: 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014 - Proceedings of the Conference, vol. 2, pp. 383–389, June 2014

    Google Scholar 

  7. Duwairi, R.M.: Arabic sentiment analysis using supervised classification. In: Proceedings - 2014 International Conference on Future Internet of Things and Cloud, FiCloud 2014, August 2014

    Google Scholar 

  8. Hatzivassiloglou, V., McKeown, K.R.: Predicting the semantic orientation of adjectives. In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, ACL 1998/EACL 1998, pp. 174–181. Association for Computational Linguistics, USA (1997)

    Google Scholar 

  9. Kiritchenko, S., Mohammad, S.M.: Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 811–817. Association for Computational Linguistics, San Diego, California, June 2016

    Google Scholar 

  10. Krchnavy, R., Simko, M.: Sentiment analysis of social network posts in Slovak language. In: 2017 12th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), pp. 20–25 (2017)

    Google Scholar 

  11. Liu, B.: Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press, Cambridge (2015)

    CrossRef  Google Scholar 

  12. Mikula, M., Machova, K.: Combined approach for sentiment analysis in Slovak using a dictionary annotated by particle swarm optimization. Acta Electrotech. Inf. 18, 27–34 (2018)

    CrossRef  Google Scholar 

  13. Musat, C.C., Ghasemi, A., Faltings, B.: Sentiment analysis using a novel human computation game. In: Proceedings of the 3rd Workshop on the People’s Web Meets NLP: Collaboratively Constructed Semantic Resources and their Applications to NLP, pp. 1–9. Association for Computational Linguistics, Jeju, July 2012.

  14. Okruhlica, A.: Slovak sentiment lexicon induction in absence of labeled data. Master’s thesis, Comenius University Bratislava (2013)

    Google Scholar 

  15. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Found. Trends Inf. Retr. 2(1–2), 1–135 (2008)

    CrossRef  Google Scholar 

  16. Pecar, S., Simko, M., Bielikova, M.: Improving sentiment classification in Slovak language. In: Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pp. 114–119. Association for Computational Linguistics, Florence, August 2019

    Google Scholar 

  17. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019)

    CrossRef  Google Scholar 

  18. Simko, J., Bieliková, M.: Semantic Acquisition Games - Harnessing Manpower for Creating Semantics. Springer, Heidelberg (2014).

    CrossRef  Google Scholar 

  19. Speer, R.: ConceptNet numberbatch 17.04: better, less-stereotyped word vectors, April 2017. Accessed 24 Apr 2017

  20. Xu, L., Chamberlain, J.: Cipher: a prototype game-with-a-purpose for detecting errors in text. In: Workshop on Games and Natural Language Processing, pp. 17–25. European Language Resources Association, Marseille, May 2020

    Google Scholar 

  21. You, Q.: Sentiment and emotion analysis for social multimedia: methodologies and applications. In: Proceedings of the 24th ACM International Conference on Multimedia, MM 2016, pp. 1445–1449. Association for Computing Machinery, New York (2016)

    Google Scholar 

  22. Zichermann, G., Linder, J.: Game-Based Marketing: Inspire Customer Loyalty Through Rewards, Challenges, and Contests. Wiley, Hoboken (2010)

    Google Scholar 

  23. Šimko, J., Tvarožek, M., Bieliková, M.: Human computation: image metadata acquisition based on a single-player annotation game. Int. J. Hum. Comput. Stud. 71(10), 933–945 (2013)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Lukáš Radoský .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Radoský, L., Blšták, M. (2021). Constructing Sentiment Lexicon with Game for Annotation Collection. In: Espinosa-Anke, L., Martín-Vide, C., Spasić, I. (eds) Statistical Language and Speech Processing. SLSP 2021. Lecture Notes in Computer Science(), vol 13062. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89578-5

  • Online ISBN: 978-3-030-89579-2

  • eBook Packages: Computer ScienceComputer Science (R0)