A Proposal for New Evaluation Metrics and Result Visualization Technique for Sentiment Analysis Tasks

  • Francisco José Valverde-Albacete
  • Jorge Carrillo-de-Albornoz
  • Carmen Peláez-Moreno
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)


In this paper we propound the use of a number of entropy-based metrics and a visualization tool for the intrinsic evaluation of Sentiment and Reputation Analysis tasks. We provide a theoretical justification for their use and discuss how they complement other accuracy-based metrics. We apply the proposed techniques to the analysis of TASS-SEPLN and RepLab 2012 results and show how the metric is effective for system comparison purposes, for system development and postmortem evaluation.


Mutual Information Evaluation Metrics Sentiment Analysis Joint Entropy Random Decision 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zhu, X., Davidson, I.: Knowledge discovery and data mining: challenges and realities. Premier reference source. Information Science Reference (2007)Google Scholar
  2. 2.
    Thomas, C., Balakrishnan, N.: Improvement in minority attack detection with skewness in network traffic. In: Proc. of SPIE, vol. 6973, pp. 69730N–69730N–12 (2008)Google Scholar
  3. 3.
    Fernandes, J.A., Irigoien, X., Goikoetxea, N., Lozano, J.A., Inza, I., Pérez, A., Bode, A.: Fish recruitment prediction, using robust supervised classification methods. Ecological Modelling 221, 338–352 (2010)CrossRefGoogle Scholar
  4. 4.
    Valverde-Albacete, F.J., Peláez-Moreno, C.: Two information-theoretic tools to assess the performance of multi-class classifiers. Pattern Recognition Letters 31, 1665–1671 (2010)CrossRefGoogle Scholar
  5. 5.
    Meila, M.: Comparing clusterings—an information based distance. Journal of Multivariate Analysis 28, 875–893 (2007)MathSciNetGoogle Scholar
  6. 6.
    Valverde-Albacete, F.J., Peláez-Moreno, C.: 100% classification accuracy considered harmful: The Normalized Information Transfer explains the accuracy paradox (submitted, 2013)Google Scholar
  7. 7.
    Mejía-Navarrete, D., Gallardo-Antolín, A., Peláez-Moreno, C., Valverde-Albacete, F.J.: Feature extraction assessment for an acoustic-event classification task using the entropy triangle. In: Interspeech 2010 (2011)Google Scholar
  8. 8.
    Villena-Román, J., García-Morera, J., Moreno-García, C., Ferrer-Ureña, L., Lana-Serrano, S.: TASS - Workshop on sentiment analysis at SEPLN (2012)Google Scholar
  9. 9.
    Amigó, E., Corujo, A., Gonzalo, J., Meij, E., Rijke, M.: Overview of RepLab 2012: Evaluating online management systems. In: CLEF (2012)Google Scholar
  10. 10.
    Greenwood, M.A., Aswani, N., Bontcheva, K.: Reputation profiling with gate. In: CLEF (2012)Google Scholar
  11. 11.
    Carrillo-de-Albornoz, J., Chugur, I., Amigó, E.: Using an emotion-based model and sentiment analysis techniques to classify polarity for reputation. In: CLEF (2012)Google Scholar
  12. 12.
    Martín-Wanton, T., Carrillo-de-Albornoz, J.: UNED at TASS 2012: Polarity classification and trending topic system. In: Workshop on Sentiment Analysis at SEPLN (2012)Google Scholar
  13. 13.
    Carrillo-de-Albornoz, J., Plaza, L., Gervás, P.: A hybrid approach to emotional sentence polarity and intensity classification. In: Conference on Computational Natural Language Learning, CoNLL 2010, pp. 153–161 (2010)Google Scholar
  14. 14.
    Reyes, A., Rosso, P., Veale, T.: A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation 47, 239–268 (2013)CrossRefGoogle Scholar
  15. 15.
    Reyes, A., Rosso, P.: On the difficulty of automatically detecting irony: beyond a simple case of negation. In: Knowledge and Information Systems, 1–20 (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Francisco José Valverde-Albacete
    • 1
  • Jorge Carrillo-de-Albornoz
    • 1
  • Carmen Peláez-Moreno
    • 2
  1. 1.Departamento de Lenguajes y Sistemas InformáticosUniv. Nacional de Educación a DistanciaMadridSpain
  2. 2.Departamento de Teoría de la Señal y de las ComunicacionesUniversidad Carlos III de MadridLeganésSpain

Personalised recommendations