Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems

  • Enrique Amigó
  • Jorge Carrillo de Albornoz
  • Irina Chugur
  • Adolfo Corujo
  • Julio Gonzalo
  • Tamara Martín
  • Edgar Meij
  • Maarten de Rijke
  • Damiano Spina
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)

Abstract

This paper summarizes the goals, organization, and results of the second RepLab competitive evaluation campaign for Online Reputation Management Systems (RepLab 2013). RepLab focused on the process of monitoring the reputation of companies and individuals, and asked participant systems to annotate different types of information on tweets containing the names of several companies: first tweets had to be classified as related or unrelated to the entity; relevant tweets had to be classified according to their polarity for reputation (Does the content of the tweet have positive or negative implications for the reputation of the entity?), clustered in coherent topics, and clusters had to be ranked according to their priority (potential reputation problems had to come first). The gold standard consists of more than 140,000 tweets annotated by a group of trained annotators supervised and monitored by reputation experts.

Keywords

RepLab Reputation Management Evaluation Methodologies and Metrics Test Collections Text Clustering Sentiment Analysis 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amigó, E., Gonzalo, J., Artiles, J., Verdejo, F.: A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval 12(4), 461–486 (2009)CrossRefGoogle Scholar
  2. 2.
    Amigó, E., Corujo, A., Gonzalo, J., Meij, E., de Rijke, M.: Overview of RepLab 2012: Evaluating Online Reputation Management Systems. In: CLEF 2012 Labs and Workshop Notebook Papers (2012)Google Scholar
  3. 3.
    Amigó, E., Gonzalo, J., Artiles, J., Verdejo, F.: Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. Journal of Artificial Intelligence Research 42(1), 689–718 (2011)MathSciNetMATHGoogle Scholar
  4. 4.
    Amigó, E., Gonzalo, J., Verdejo, F.: A General Evaluation Measure for Document Organization Tasks. In: Proceedings of SIGIR 2013 (July 2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Enrique Amigó
    • 1
  • Jorge Carrillo de Albornoz
    • 1
  • Irina Chugur
    • 1
  • Adolfo Corujo
    • 2
  • Julio Gonzalo
    • 1
  • Tamara Martín
    • 1
  • Edgar Meij
    • 3
  • Maarten de Rijke
    • 4
  • Damiano Spina
    • 1
  1. 1.UNED NLP and IR GroupMadridSpain
  2. 2.Llorente and CuencaMadridSpain
  3. 3.Yahoo! ResearchBarcelonaSpain
  4. 4.ISLAUniversity of AmsterdamAmsterdamThe Netherlands

Personalised recommendations