Advertisement

CheckThat! at CLEF 2019: Automatic Identification and Verification of Claims

  • Tamer Elsayed
  • Preslav Nakov
  • Alberto Barrón-Cedeño
  • Maram HasanainEmail author
  • Reem Suwaileh
  • Giovanni Da San Martino
  • Pepa Atanasova
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11438)

Abstract

We introduce the second edition of the CheckThat! Lab, part of the 2019 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes two complementary tasks. Task 1: predict which claims in a political debate should be prioritized for fact-checking. Task 2: rank Web-retrieved pages against a check-worthy claim based on their usefulness for fact-checking, extract useful passages from those pages, and then use them all to decide whether the claim is factually true or false. Checkthat! provides a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) for ranking and F\(_1\) for classification tasks.

References

  1. 1.
    Atanasova, P., et al.: Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims, task 1: check-worthiness. In: CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Avignon, France. CEUR-WS.org (2018)Google Scholar
  2. 2.
    Ba, M.L., Berti-Equille, L., Shah, K., Hammady, H.M.: VERA: a platform for veracity estimation over web data. In: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 159–162. International World Wide Web Conferences Steering Committee (2016)Google Scholar
  3. 3.
    Baly, R., Mohtarami, M., Glass, J., Màrquez, L., Moschitti, A., Nakov, P.: Integrating stance detection and fact checking in a unified corpus. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), vol. 2, pp. 21–27 (2018)Google Scholar
  4. 4.
    Castillo, C., Mendoza, M., Poblete, B.: Information credibility on Twitter. In: Proceedings of the 20th International Conference on World Wide Web, pp. 675–684. ACM (2011)Google Scholar
  5. 5.
    Gencheva, P., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: A context-aware approach for detecting worth-checking claims in political debates. In: Proceedings of the 2017 International Conference on Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria (2017)Google Scholar
  6. 6.
    Hardalov, M., Koychev, I., Nakov, P.: In search of credible news. In: Dichev, C., Agre, G. (eds.) AIMSA 2016. LNCS (LNAI), vol. 9883, pp. 172–180. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-44748-3_17CrossRefGoogle Scholar
  7. 7.
    Hassan, N., Li, C., Tremayne, M.: Detecting check-worthy factual claims in presidential debates. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 1835–1838. ACM (2015)Google Scholar
  8. 8.
    Karadzhov, G., Gencheva, P., Nakov, P., Koychev, I.: We built a fake news & click-bait filter: what happened next will blow your mind! In: Proceedings of the 2017 International Conference on Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria (2017)Google Scholar
  9. 9.
    Karadzhov, G., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: Fully automated fact checking using external sources. In: Proceedings of the 2017 International Conference on Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria (2017)Google Scholar
  10. 10.
    Ma, J., et al.: Detecting rumors from microblogs with recurrent neural networks. In: Proceedings of IJCAI (2016)Google Scholar
  11. 11.
    Mukherjee, S., Weikum, G.: Leveraging joint interactions for credibility analysis in news communities. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 353–362. ACM (2015)Google Scholar
  12. 12.
    Nakov, P., et al.: Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. In: Bellot, I., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 372–387. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-98932-7_32CrossRefGoogle Scholar
  13. 13.
    Popat, K., Mukherjee, S., Strötgen, J., Weikum, G.: Credibility assessment of textual claims on the web. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 2173–2178. ACM (2016)Google Scholar
  14. 14.
    Yasser, K., Kutlu, M., Elsayed, T.: Re-ranking web search results for better fact-checking: a preliminary study. In: Proceedings of 27th ACM International Conference on Information and Knowledge Management (CIKM), Turin, Italy, pp. 1783–1786. ACM (2018)Google Scholar
  15. 15.
    Zubiaga, A., Liakata, M., Procter, R., Hoi, G.W.S., Tolmie, P.: Analysing how people orient to and spread rumours in social media by looking at conversational threads. PloS ONE 11(3), e0150989 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tamer Elsayed
    • 1
  • Preslav Nakov
    • 2
  • Alberto Barrón-Cedeño
    • 2
  • Maram Hasanain
    • 1
    Email author
  • Reem Suwaileh
    • 1
  • Giovanni Da San Martino
    • 2
  • Pepa Atanasova
    • 3
  1. 1.Qatar UniversityDohaQatar
  2. 2.Qatar Computing Research InstituteHBKUDohaQatar
  3. 3.Sofia UniversitySofiaBulgaria

Personalised recommendations