Advertisement

RepLab: An Evaluation Campaign for Online Monitoring Systems

  • Jorge Carrillo-de-Albornoz
  • Julio GonzaloEmail author
  • Enrique Amigó
Chapter
Part of the The Information Retrieval Series book series (INRE, volume 41)

Abstract

Over a period of 3 years, RepLab was a CLEF initiative where computer scientists and online reputation experts worked together to identify and formalize the computational challenges in the area of online reputation monitoring. Two main results emerged from RepLab: a community of researchers engaged in the problem, and an extensive Twitter test collection comprising more than half a million expert annotations, which cover many relevant tasks in the field of online reputation: named entity resolution, topic detection and tracking, reputational alerts identification, reputational polarity, author profiling, opinion makers identification and reputational dimension classification. It has probably been one of the CLEF labs with a larger set of expert annotations provided to participants in a single year, and one of the labs where the target user community has been more actively engaged in the evaluation campaign. Here we summarize the design and results of the Replab campaigns, and also report on research that has built on RepLab datasets after completion of the 3-year competition cycle.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgement

This research was partially supported by the Spanish Ministry of Science and Innovation (Vemodalen Project, TIN2015-71785-R).

References

  1. Alsop RJ (2006) The 18 immutable laws of corporate reputation: creating, protecting and repairing your most valuable asset. Kogan Page, LondonGoogle Scholar
  2. Amigó E, Gonzalo J, Artiles J, Verdejo F (2009) A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf Retr 12(4):461–486CrossRefGoogle Scholar
  3. Amigó E, Gonzalo J, Artiles J, Verdejo F (2011) Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. J Artif Intell Res 42(1):689–718MathSciNetzbMATHGoogle Scholar
  4. Amigó E, Gonzalo J, Verdejo F (2013) A general evaluation measure for document organization tasks. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’13, pp 643–652. https://doi.org/10.1145/2484028.2484081
  5. Amigó E, Carrillo-de Albornoz J, Almagro-Cádiz M, Gonzalo J, Rodríguez-Vidal J, Verdejo F (2017) Evall: open access evaluation for information access systems. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’17, pp 1301–1304. https://doi.org/10.1145/3077136.3084145
  6. Balahur A, Tanev H (2012) Detecting entity-related events and sentiments from tweets using multilingual resources. In: CLEF (Online Working Notes/Labs/Workshop)Google Scholar
  7. Berrocal A, Luis J, Figuerola CG, Zazo Rodríguez ÁF (2013) Reina at replab2013 topic detection task: community detection. In: CLEF (Working Notes)Google Scholar
  8. Carrillo-de-Albornoz J, Chugur I, Amigó E (2012) Using an emotion-based model and sentiment analysis techniques to classify polarity for reputation. In: CLEF (Online Working Notes/Labs/Workshop)Google Scholar
  9. Carrillo-de Albornoz J, Amigó E, Plaza L, Gonzalo J (2016) Tweet stream summarization for online reputation management. In: Ferro N, Crestani F, Moens MF, Mothe J, Silvestri F, Di Nunzio GM, Hauff C, Silvello G (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 378–389CrossRefGoogle Scholar
  10. Chong WN, Tan G (2010) Obtaining intangible and tangible benefits from corporate social responsibility. Int Rev Bus Res Pap 6(4):360Google Scholar
  11. Cossu JV, Bigot B, Bonnefoy L, Morchid M, Bost X, Senay G, Dufour R, Bouvier V, Torres-Moreno JM, El-Bèze M (2013) Lia at replab 2013. In: CLEF (Working Notes)Google Scholar
  12. Cossu JV, Bigot B, Bonnefoy L, Senay G (2014a) Towards the improvement of topic priority assignment using various topic detection methods for e-reputation monitoring on twitter. In: Métais E, Roche M, Teisseire M (eds) Natural language processing and information systems. Springer International Publishing, Cham, pp 154–159CrossRefGoogle Scholar
  13. Cossu JV, Janod K, Ferreira E, Gaillard J, El-Bèze M (2014b) Lia at replab 2014: 10 methods for 3 tasks. In: 4th international conference of the CLEF initiativeGoogle Scholar
  14. Doorley J, Garcia HF (2011) Reputation management: the key to successful public relations and corporate communication. Routledge, New YorkGoogle Scholar
  15. Filgueiras J, Amir S (2013) Popstar at replab 2013: polarity for reputation classification. In: CLEF (Online Working Notes/Labs/Workshop), vol 60Google Scholar
  16. Gârbacea C, Tsagkias M, de Rijke M (2014) Detecting the reputation polarity of microblog posts. In: Proceedings of the twenty-first european conference on artificial intelligence. IOS Press, Amsterdam, ECAI’14, pp 339–344. https://doi.org/10.3233/978-1-61499-419-0-339
  17. Giachanou A, Gonzalo J, Mele I, Crestani F (2017) Sentiment propagation for predicting reputation polarity. In: Jose JM, Hauff C, Altıngovde IS, Song D, Albakour D, Watt S, Tait J (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 226–238CrossRefGoogle Scholar
  18. Glance N, Hurst M, Nigam K, Siegler M, Stockton R, Tomokiyo T (2005) Deriving marketing intelligence from online discussion. In: Proceedings of the eleventh ACM SIGKDD international conference on knowledge discovery in data mining. ACM, New York, NY, KDD ’05, pp 419–428. https://doi.org/10.1145/1081870.1081919
  19. Hangya V, Farkas R (2013) Filtering and polarity detection for reputation management on tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60Google Scholar
  20. Hoffman T (2008) Online reputation management is hot – but is it ethical. Computerworld (44). https://www.computerworld.com/article/2537007/networking/online-reputation-management-is-hot----but-is-it-ethical-.html
  21. Jansen B, Zhang M, Sobel K, Chowdury A (2009a) Twitter power: tweets as electronic word of mouth. J Am Soc Inf Sci Technol 60(11):2169–2188CrossRefGoogle Scholar
  22. Jansen BJ, Zhang M, Sobel K, Chowdury A (2009b) Twitter power: tweets as electronic word of mouth. J Assoc Inf Sci Technol 60(11):2169–2188CrossRefGoogle Scholar
  23. Kaptein R (2012) Learning to analyze relevancy and polarity of tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60Google Scholar
  24. Kreps DM, Wilson R (1982) Reputation and imperfect information. J Econ Theory 27(2):253–279MathSciNetCrossRefGoogle Scholar
  25. Krishnamurthy B, Gill P, Arlitt M (2008) A few chirps about Twitter. In: Proceedings of the first workshop on online social networks (WOSP’08), pp 19–24Google Scholar
  26. Li YM, Li TY (2013) Deriving market intelligence from microblogs. Decis Support Syst 55(1):206–217. https://doi.org/10.1016/j.dss.2013.01.023. http://www.sciencedirect.com/science/article/pii/S0167923613000511 CrossRefGoogle Scholar
  27. Mabrouk O, Hlaoua L, Nazih Omri M (2018) Profile categorization system based on features reduction. In: International symposium on artificial intelligence and mathematics, ISAIM 2018, Fort Lauderdale, Florida. http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Mabrouk_etal.pdf
  28. Magdy W, Elsayed T (2016) Unsupervised adaptive microblog filtering for broad dynamic topics. Inf Process Manag 52(4):513–528CrossRefGoogle Scholar
  29. Mahalakshmi GS, Koquilamballe K, Sendhilkumar S (2017) Influential detection in twitter using tweet quality analysis. In: 2017 second international conference on recent trends and challenges in computational models (ICRTCCM), pp 315–319.  https://doi.org/10.1109/ICRTCCM.2017.62
  30. Martın T, Spina D, Amigó E, Gonzalo J (2012) Uned at replab 2012: monitoring task. In: CLEF 2012 Working Notes, CLEFGoogle Scholar
  31. Martín-Wanton T, Gonzalo J, Amigó E (2013) An unsupervised transfer learning approach to discover topics for online reputation management. In: Proceedings of the 22nd ACM international conference on conference on information & knowledge management, ACM, New York, NY, CIKM ’13, pp 1565–1568. https://doi.org/10.1145/2505515.2507845
  32. McDonald G, Deveaud R, McCreadie R, Gollins T, Macdonald C, Ounis I (2014) University of glasgow terrier team/project abacá at replab 2014: reputation dimensions task. In: CLEF (Working Notes), pp 1500–1504Google Scholar
  33. McDonald G, Deveaud R, Mccreadie R, Macdonald C, Ounis I (2015) Tweet enrichment for effective dimensions classification in online reputation management. In: 9th international AAAI conference on web and social media, pp 654–657Google Scholar
  34. Nebot V, Rangel F, Berlanga R, Rosso P (2018) Identifying and classifying influencers in twitter only with textual information. In: Proceedings of the NLDB 2018Google Scholar
  35. Panem S, Bansal R, Gupta M, Varma V (2014) Entity tracking in real-time using sub-topic detection on twitter. In: de Rijke M, Kenter T, de Vries AP, Zhai C, de Jong F, Radinsky K, Hofmann K (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 528–533CrossRefGoogle Scholar
  36. Peetz MH, de Rijke M, Kaptein R (2016) Estimating reputation polarity on microblog posts. Inf Process Manag 52(2):193–216. https://doi.org/10.1016/j.ipm.2015.07.003. http://www.sciencedirect.com/science/article/pii/S0306457315000874 CrossRefGoogle Scholar
  37. Qureshi M, Younus A, O’Riordan C, Pasi G (2015) Company name disambiguation in tweets: a two-step filtering approach. In: Information retrieval technology, vol 9460CrossRefGoogle Scholar
  38. Qureshi MA, O’Riordan C, Pasi G (2012) Concept term expansion approach for monitoring reputation of companies on twitter. In: CLEF (Online Working Notes/Labs/Workshop)Google Scholar
  39. Qureshi MA, Younus A, O’Riordan C, Pasi G (2017) A wikipedia-based semantic relatedness framework for effective dimensions classification in online reputation management. J Ambient Intell Humaniz Comput 9:1403CrossRefGoogle Scholar
  40. Rodriguez J, Carrillo-de Albornoz J, Plaza L, Amigó E, Gonzalo J (2019a) Automatic generation of entity-oriented summaries for reputation management. J Ambient Intell Humaniz Comput:1–15. https://link.springer.com/article/10.1007/s12652-019-01255-9
  41. Rodriguez J, Gonzalo J, Plaza L, Anaya H (2019b) Automatic detection of influencers in social networks: authority versus domain signals. J Assoc Inf Sci Technol 70:7Google Scholar
  42. Spina D, Carrillo-de-Albornoz J, Martín-Wanton T, Amigó E, Gonzalo J, Giner F (2013) Uned online reputation monitoring team at replab 2013. In: CLEF (Working Notes)Google Scholar
  43. Spina D, Gonzalo J, Amigó E (2014) Learning similarity functions for topic detection in online reputation monitoring. In: Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval. ACM, New York, NY, SIGIR ’14, pp 527–536. https://doi.org/10.1145/2600428.2609621
  44. Spina D, Peetz MH, de Rijke M (2015) Active learning for entity filtering in microblog streams. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’15, pp 975–978. https://doi.org/10.1145/2766462.2767839
  45. Vilares D, Hermo M, Alonso MA, Gómez-Rodríguez C, Vilares J (2014) Lys at clef replab 2014: creating the state of the art in author influence ranking and reputation classification on twitter. In: CLEF (Working Notes), pp 1468–1478Google Scholar
  46. Villena-Román J, Lana-Serrano S, Moreno C, García-Morera J, Cristóbal JCG (2012) Daedalus at replab 2012: polarity classification and filtering on twitter data. In: CLEF (Online Working Notes/Labs/Workshop), vol 60Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Jorge Carrillo-de-Albornoz
    • 1
  • Julio Gonzalo
    • 1
    Email author
  • Enrique Amigó
    • 1
  1. 1.NLP & IR Group at UNEDMadridSpain

Personalised recommendations