Influence of Weak Labels for Emotion Recognition of Tweets

  • Olivier Janssens
  • Steven Verstockt
  • Erik Mannens
  • Sofie Van Hoecke
  • Rik Van de Walle
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8891)


Research on emotion recognition of tweets focuses on feature engineering or algorithm design, while dataset labels are barely questioned. Datasets of tweets are often labelled manually or via crowdsourcing, which results in strong labels. These methods are time intensive and can be expensive. Alternatively, tweet hashtags can be used as free, inexpensive weak labels. This paper investigates the impact of using weak labels compared to strong labels. The study uses two label sets for a corpus of tweets. The weakly annotated label set is created employing the hashtags of the tweets, while the strong label set is created by the use of crowdsourcing. Both label sets are used separately as input for five classification algorithms to determine the classification performance of the weak labels. The results indicate only a 9.25% decrease in f1-score when using weak labels. This performance decrease does not outweigh the benefits of having free labels.


Emotion recognition Twitter Annotation Crowdsourcing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lovins, J.B.: Development of a stemming algorithm. Mechanical Translation and Computational Linguistics 11, 22–31 (1968)Google Scholar
  2. 2.
    Bergamo, A., Torresani, L.: Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. Advances in Neural Information Processing Systems 23, 181–189 (2010)Google Scholar
  3. 3.
    Bollen, J., Mao, H., Pepe, A.: Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In: ICWSM, pp. 450–453 (2011)Google Scholar
  4. 4.
    Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock market. Journal of Computational Science 2(1), 1–8 (2011)CrossRefGoogle Scholar
  5. 5.
    Calix, R.A., Mallepudi, S.A., Chen, B.C.B., Knapp, G.M.: Emotion Recognition in Text for 3-D Facial Expression Rendering. IEEE Transactions on Multimedia 12(6), 544–551 (2010)CrossRefGoogle Scholar
  6. 6.
    Chaffar, S., Inkpen, D.: Using a heterogeneous dataset for emotion analysis in text. In: Butz, C., Lingras, P. (eds.) Canadian AI 2011. LNCS, vol. 6657, pp. 62–67. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  7. 7.
    Ekman, P.: Basic emotions. Handbook of Cognition and Emotion, vol. 98. John Wiley & Sons (1999)Google Scholar
  8. 8.
    Janssens, O., Slembrouck, M., Verstockt, S., Hoecke, S.V., Walle, R.V.D.: Real-time Emotion Classification of Tweets. In: IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp. 1430–1431 (2013)Google Scholar
  9. 9.
    Kaufman, S., Rosset, S.: Leakage in data mining: Formulation, detection, and avoidance. In: 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 556–563 (2012)Google Scholar
  10. 10.
    Mohammad, S.: #Emotional Tweets. In: First Joint Conference on Lexical and Computational Semantics, pp. 246–255. Association for Computational Linguistics, Montréal (2012)Google Scholar
  11. 11.
    Paice, C., Husk, G.: Another Stemmer. ACM SIGIR Forum 24, 56–61 (1990)CrossRefGoogle Scholar
  12. 12.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)zbMATHGoogle Scholar
  13. 13.
    Roberts, K., Roach, M.A., Johnson, J.: EmpaTweet: Annotating and Detecting Emotions on Twitter. In: LREC, pp. 3806–3813 (2012)Google Scholar
  14. 14.
    Rui, H., Whinston, A.: Designing a social-broadcasting-based business intelligence system. ACM Transactions on Management Information Systems 2(4) (2011)Google Scholar
  15. 15.
    Suttles, J., Ide, N.: Distant supervision for emotion classification with discrete binary values. In: Gelbukh, A. (ed.) CICLing 2013, Part II. LNCS, vol. 7817, pp. 121–136. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Wang, A., Hoang, C., Kan, M.: Perspectives on crowdsourcing annotations for natural language processing. Language Resources and Evaluation 47(1), 9–31 (2013)CrossRefGoogle Scholar
  17. 17.
    Wang, W., Chen, L., Thirunarayan, K., Sheth, A.P.: Harnessing Twitter “Big Data” for Automatic Emotion Identification. In: 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pp. 587–592 (2012)Google Scholar
  18. 18.
    Willett, P.: The Porter stemming algorithm: then and now (2006),

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Olivier Janssens
    • 1
  • Steven Verstockt
    • 1
  • Erik Mannens
    • 1
  • Sofie Van Hoecke
    • 1
  • Rik Van de Walle
    • 1
  1. 1.Multimedia LabGhent University – iMindsLedeberg-GhentBelgium

Personalised recommendations