Language Resources and Evaluation

, Volume 50, Issue 1, pp 35–65

Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts

  • Preslav Nakov
  • Sara Rosenthal
  • Svetlana Kiritchenko
  • Saif M. Mohammad
  • Zornitsa Kozareva
  • Alan Ritter
  • Veselin Stoyanov
  • Xiaodan Zhu
Original Paper

DOI: 10.1007/s10579-015-9328-1

Cite this article as:
Nakov, P., Rosenthal, S., Kiritchenko, S. et al. Lang Resources & Evaluation (2016) 50: 35. doi:10.1007/s10579-015-9328-1

Abstract

We present the development and evaluation of a semantic analysis task that lies at the intersection of two very trendy lines of research in contemporary computational linguistics: (1) sentiment analysis, and (2) natural language processing of social media text. The task was part of SemEval, the International Workshop on Semantic Evaluation, a semantic evaluation forum previously known as SensEval. The task ran in 2013 and 2014, attracting the highest number of participating teams at SemEval in both years, and there is an ongoing edition in 2015. The task included the creation of a large contextual and message-level polarity corpus consisting of tweets, SMS messages, LiveJournal messages, and a special test set of sarcastic tweets. The evaluation attracted 44 teams in 2013 and 46 in 2014, who used a variety of approaches. The best teams were able to outperform several baselines by sizable margins with improvement across the 2 years the task has been run. We hope that the long-lasting role of this task and the accompanying datasets will be to serve as a test bed for comparing different approaches, thus facilitating research.

Keywords

Sentiment analysis Twitter SemEval 

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Preslav Nakov
    • 1
  • Sara Rosenthal
    • 2
  • Svetlana Kiritchenko
    • 3
  • Saif M. Mohammad
    • 3
  • Zornitsa Kozareva
    • 4
  • Alan Ritter
    • 5
  • Veselin Stoyanov
    • 6
  • Xiaodan Zhu
    • 3
  1. 1.Qatar Computing Research Institute, HBKUDohaQatar
  2. 2.Columbia UniversityNew YorkUSA
  3. 3.National Research Council CanadaOttawaCanada
  4. 4.USC Information Sciences InstituteMarina del ReyUSA
  5. 5.The Ohio State UniversityColumbusUSA
  6. 6.Johns Hopkins UniversityBaltimoreUSA