Skip to main content
SpringerLink
Log in

Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their Source

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14163))

Abstract

We describe the sixth edition of the CheckThat! lab, part of the 2023 Conference and Labs of the Evaluation Forum (CLEF). The five previous editions of CheckThat! focused on the main tasks of the information verification pipeline: check-worthiness, verifying whether a claim was fact-checked before, supporting evidence retrieval, and claim verification. In this sixth edition, we zoom into some new problems and for the first time we offer five tasks in seven languages: Arabic, Dutch, English, German, Italian, Spanish, and Turkish. Task 1 asks to determine whether an item —text or text plus image— is check-worthy. Task 2 aims to predict whether a sentence from a news article is subjective or not. Task 3 asks to assess the political bias of the news at the article and at the media outlet level. Task 4 focuses on the factuality of reporting of news media. Finally, Task 5 looks at identifying authorities in Twitter that could help verify a given target claim. For a second year, CheckThat! was the most popular lab at CLEF-2023 in terms of team registrations: 127 teams. About one-third of them (a total of 37) actually participated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.allsides.com.

  2. 2.

    https://www.mediabiasfactcheck.com.

  3. 3.

    https://misbar.com/.

  4. 4.

    https://fatabyyano.net/.

  5. 5.

    https://github.com/JaidedAI/EasyOCR.

  6. 6.

    https://platform.openai.com/docs/models/gpt-3.

  7. 7.

    https://aws.amazon.com/translate/.

  8. 8.

    No enough details were available about the source of these embeddings.

  9. 9.

    https://platform.openai.com/docs/models/gpt-3-5.

References

  1. Abdul-Mageed, M., Elmadany, A., et al.: Arbert & marbert: deep bidirectional transformers for Arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, vol. 1: Long Papers, pp. 7088–7105 (2021)

    Google Scholar 

  2. Alam, F., et al.: Overview of the CLEF-2023 CheckThat! lab task 1 on check-worthiness in multimodal and multigenre content. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  3. Alam, F., et al.: Fighting the COVID-19 infodemic in social media: a holistic perspective and a call to arms. In: Proceedings of the Conference on Web and Social Media, pp. 913–922. ICWSM 2021 (2021)

    Google Scholar 

  4. Alam, F., et al.: Fighting the COVID-19 infodemic: modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In: Findings of EMNLP 2021, pp. 611–649 (2021)

    Google Scholar 

  5. Ali, Z.S., Mansour, W., Elsayed, T., Al-Ali, A.: AraFacts: the first large Arabic dataset of naturally occurring claims. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 231–236 (2021)

    Google Scholar 

  6. Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.): Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  7. Antici, F., et al.: A corpus for sentence-level subjectivity detection on English news articles (2023)

    Google Scholar 

  8. Antoun, W., Baly, F., Hajj, H.: AraBERT: transformer-based model for Arabic language understanding. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pp. 9–15. OSAC 2020, Marseille, France (2020)

    Google Scholar 

  9. Arslan, F., Hassan, N., Li, C., Tremayne, M.: A benchmark dataset of check-worthy factual claims. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 14, pp. 821–829 (2020)

    Google Scholar 

  10. Atanasova, P., et al.: Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. Task 1: check-worthiness. In: Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.) Working Notes of CLEF 2018-Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings (2018)

    Google Scholar 

  11. Atanasova, P., Nakov, P., Karadzhov, G., Mohtarami, M., Da San Martino, G.: Overview of the CLEF-2019 CheckThat! lab on automatic identification and verification of claims. Task 1: check-worthiness. In: Cappellato, L., Ferro, N., Losada, D., Müller, H. (eds.) Working Notes of CLEF 2019 Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings (2019)

    Google Scholar 

  12. Aziz, A., Hossain, M., Chy, A.: CSECU-DSG at CheckThat! 2023: transformer-based fusion approach for multimodal and multigenre check-worthiness. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  13. Ba, M.L., Berti-Equille, L., Shah, K., Hammady, H.M.: VERA: a platform for veracity estimation over web data. In: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 159–162 (2016)

    Google Scholar 

  14. Baly, R., Da San Martino, G., Glass, J., Nakov, P.: We can detect your bias: Predicting the political ideology of news articles. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 4982–4991 (2020)

    Google Scholar 

  15. Baly, R., et al.: What was written vs. who read it: news media profiling using text analysis and social media context. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3364–3374 (2020)

    Google Scholar 

  16. Baly, R., Karadzhov, G., Saleh, A., Glass, J., Nakov, P.: Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. In: Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2109–2116. NAACL-HLT 2019, Minneapolis, MN, USA (2019)

    Google Scholar 

  17. Barbieri, F., Espinosa Anke, L., Camacho-Collados, J.: XLM-T: a multilingual language model toolkit for twitter. arXiv e-prints, pp. arXiv-2104 (2021)

    Google Scholar 

  18. Barrón-Cedeño, A., et al.: The CLEF-2023 CheckThat! Lab: checkworthiness, subjectivity, political bias, factuality, and authority. In: Kamps, J., et al. (eds.) ECIR 2023. LNCS, vol. 13982, pp. 506–517. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_59

    Chapter  Google Scholar 

  19. Barrón-Cedeño, A.: CheckThat! at CLEF 2020: enabling the automatic identification and verification of claims in social media. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 499–507. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_65

    Chapter  Google Scholar 

  20. Barrón-Cedeño, A.: Overview of CheckThat! 2020: automatic identification and verification of claims in social media. In: Arampatzis, A., et al. (eds.) CLEF 2020. LNCS, vol. 12260, pp. 215–236. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58219-7_17

    Chapter  Google Scholar 

  21. Barrón-Cedeño, A., et al.: Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. Task 2: factuality. In: Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.) Working Notes of CLEF 2018-Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings (2018)

    Google Scholar 

  22. Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.): CLEF 2020 Working Notes. CEUR Workshop Proceedings (2020)

    Google Scholar 

  23. Cappellato, L., Ferro, N., Losada, D., Müller, H. (eds.): Working notes of CLEF 2019 conference and labs of the evaluation forum. CEUR Workshop Proceedings (2019)

    Google Scholar 

  24. Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): Working Notes of CLEF 2018-Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings (2018)

    Google Scholar 

  25. Cheema, G.S., Hakimov, S., Sittar, A., Müller-Budack, E., Otto, C., Ewerth, R.: MM-claims: a dataset for multimodal claim detection in social media. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 962–979. Seattle, Washington (2022)

    Google Scholar 

  26. Clark, K., Luong, M., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators. In: Proceedings of the 8th International Conference on Learning Representations. ICLR 2020 (2020)

    Google Scholar 

  27. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)

    Article  Google Scholar 

  28. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. CoRR abs/1911.02116 (2019)

    Google Scholar 

  29. Da San Martino, G., Alam, F., Hasanain, M., Nandi, R.N., Azizov, D., Nakov, P.: Overview of the CLEF-2023 CheckThat! lab task 3 on political bias of news articles and news media. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  30. Da San Martino, G., Barrón-Cedeño, A., Wachsmuth, H., Petrov, R., Nakov, P.: SemEval-2020 task 11: detection of propaganda techniques in news articles. In: Proceedings of the Workshop on Semantic Evaluation, pp. 1377–1414 (2020)

    Google Scholar 

  31. Deniz Türkmen, M., Coşgun, G., Kutlu, M.: TOBB ETU at CheckThat! 2023: utilizing chatGPT to detect subjective statements and political bias. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  32. Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Hoi, G.W.S., Zubiaga, A.: SemEval-2017 task 8: RumourEval: determining rumour veracity and support for rumours. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 69–76 (2017)

    Google Scholar 

  33. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. NAACL-HLT 2019, Minneapolis, MN, USA (2019)

    Google Scholar 

  34. Dey, K., Tarannum, P., Hasan, M.A., Noori, S.R.H.: Nn at CheckThat! 2023: Subjectivity in news articles classification with transformer based models. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  35. Dimitrov, D., et al.: SemEval-2021 Task 6: detection of persuasion techniques in texts and images. In: Proceedings of the International Workshop on Semantic Evaluation, pp. 70–98. SemEval 2021 (2021)

    Google Scholar 

  36. Elsayed, T., et al.: CheckThat! at CLEF 2019: automatic identification and verification of claims. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019. LNCS, vol. 11438, pp. 309–315. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15719-7_41

    Chapter  Google Scholar 

  37. Elsayed, T., et al.: Overview of the CLEF-2019 CheckThat! lab: automatic identification and verification of claims. In: Crestani, F., et al. (eds.) CLEF 2019. LNCS, vol. 11696, pp. 301–321. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28577-7_25

    Chapter  Google Scholar 

  38. Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.): CLEF 2021 working notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum (2021)

    Google Scholar 

  39. Frick, R.A.: Fraunhofer sit at CheckThat! 2023: can LLMs be used for data augmentation & few-shot classification? detecting subjectivity in text using chatGPT. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  40. Frick, R.A., Vogel, I., Choi, J.E.: Fraunhofer SIT at CheckThat! 2023: enhancing the detection of multimodal and multigenre check-worthiness using optical character recognition and model souping. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  41. Galassi, A., et al.: Overview of the CLEF-2023 CheckThat! lab task 2 on subjectivity in news articles. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  42. Hanselowski, A., et al.: A retrospective analysis of the fake news challenge stance-detection task. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1859–1874 (2018)

    Google Scholar 

  43. Haouari, F., Elsayed, T.: Detecting stance of authorities towards rumors in Arabic tweets: a preliminary study. In: Kamps, J., et al. (eds.) ECIR 2023. LNCS, vol. 13981, pp. 430–438. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28238-6_33

    Chapter  Google Scholar 

  44. Haouari, F., Elsayed, T., Mansour, W.: Who can verify this? finding authorities for rumor verification in Twitter. Inf. Process. Manage. 60(4), 103366 (2023)

    Article  Google Scholar 

  45. Haouari, F., Hasanain, M., Suwaileh, R., Elsayed, T.: ArCOV19-rumors: Arabic COVID-19 twitter dataset for misinformation detection. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 72–81 (2021)

    Google Scholar 

  46. Haouari, F., Sheikh Ali, Z., Elsayed, T.: Overview of the CLEF-2023 CheckThat! lab task 5 on authority finding in twitter. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  47. Hasanain, M., et al.: Overview of CheckThat! 2020 Arabic: automatic identification and verification of claims in social media. In: Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.) CLEF 2020 Working Notes. CEUR Workshop Proceedings (2020)

    Google Scholar 

  48. Hasanain, M., Suwaileh, R., Elsayed, T., Barrón-Cedeño, A., Nakov, P.: Overview of the CLEF-2019 CheckThat! lab on automatic identification and verification of claims. Task 2: evidence and factuality. In: Cappellato, L., Ferro, N., Losada, D., Müller, H. (eds.) Working notes of CLEF 2019 conference and labs of the evaluation forum. CEUR Workshop Proceedings (2019)

    Google Scholar 

  49. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. CVPR 2016, Las Vegas, Nevada, USA (2016)

    Google Scholar 

  50. He, P., Gao, J., Chen, W.: DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In: The Eleventh International Conference on Learning Representations (2023)

    Google Scholar 

  51. Inoue, H.: Multi-sample dropout for accelerated training and better generalization. CoRR abs/1905.09788 (2019)

    Google Scholar 

  52. Kartal, Y.S., Kutlu, M.: Re-think before you share: a comprehensive study on prioritizing check-worthy claims. In: IEEE Transactions on Computational Social Systems (2022)

    Google Scholar 

  53. Khalil, A., Jarrah, M., Aldwairi, M., Jararweh, Y.: Detecting Arabic fake news using machine learning. In: Proceedings of the International Conference on Intelligent Data Science Technologies and Applications, pp. 171–177 (2021)

    Google Scholar 

  54. Köhler, J., et al.: Overview of the CLEF-2022 CheckThat! lab task 3 on fake news detection. In: Working Notes of CLEF 2022–Conference and Labs of the Evaluation Forum. CLEF 2022, Bologna, Italy (2022)

    Google Scholar 

  55. Lan, W., Chen, Y., Xu, W., Ritter, A.: An empirical study of pre-trained transformers for Arabic information extraction. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4727–4734 (2020)

    Google Scholar 

  56. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)

    Article  MATH  Google Scholar 

  57. Leistra, F., Caselli, T.: Thesis titan at CheckThat! 2023: language-specific fine-tuning of mdebertav3 for subjectivity detection. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  58. Li, C., Xue, R., Lin, C., Fan, W.: CUCPLUS at CheckThat! 2023: text combination and regularized adversarial training for news media factuality evaluation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  59. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)

    Google Scholar 

  60. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  61. Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp. 11966–11976. CVPR ’22, New Orleans, LA, USA (2022)

    Google Scholar 

  62. Mansour, W., Elsayed, T., Al-Ali, A.: This is not new! spotting previously-verified claims over Twitter. Inf. Process. Manage. 60(4), 103414 (2023)

    Article  Google Scholar 

  63. Mihaylova, T., Karadzhov, G., Atanasova, P., Baly, R., Mohtarami, M., Nakov, P.: SemEval-2019 task 8: fact checking in community question answering forums. In: Proceedings of the Workshop on Semantic Evaluation, pp. 860–869 (2019)

    Google Scholar 

  64. Modzelewski, A., Sosnowski, W., Wierzbicki, A.: DSHacker at CheckThat! 2023: Check-Worthiness in Multigenre and Multilingual Content With GPT-3.5 Data Augmentation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  65. Nakov, P., et al.: Overview of the CLEF-2023 CheckThat! lab task 4 on factuality of reporting of news media. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  66. Nakov, P., et al.: Overview of the CLEF-2022 CheckThat! lab task 1 on identifying relevant claims in tweets. In: Working Notes of CLEF 2022–Conference and Labs of the Evaluation Forum. CLEF 2022, Bologna, Italy (2022)

    Google Scholar 

  67. Nakov, P., et al.: Overview of the CLEF-2022 CheckThat! lab on fighting the COVID-19 infodemic and fake news detection. In: Proceedings of the Conference of the CLEF Association: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. CLEF 2022, Bologna, Italy (2022)

    Google Scholar 

  68. Nakov, P., et al.: The CLEF-2022 CheckThat! Lab on Fighting the COVID-19 Infodemic and Fake News Detection. In: Hagen, M., et al. (eds.) ECIR 2022. LNCS, vol. 13186, pp. 416–428. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99739-7_52

    Chapter  Google Scholar 

  69. Nakov, P., et al.: Overview of the CLEF-2018 lab on automatic identification and verification of claims in political debates. In: Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum. CLEF 2018 (2018)

    Google Scholar 

  70. Nakov, P., et al.: Automated fact-checking for assisting human fact-checkers. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence, pp. 4551–4558 (2021)

    Google Scholar 

  71. Nakov, P., Da San Martino, G., Alam, F., Shaar, S., Mubarak, H., Babulkov, N.: Overview of the CLEF-2022 CheckThat! lab task 2 on detecting previously fact-checked claims. In: Working Notes of CLEF 2022–Conference and Labs of the Evaluation Forum. CLEF 2022, Bologna, Italy (2022)

    Google Scholar 

  72. Nakov, P., et al.: The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In: Hiemstra, D., Moens, M.-F., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds.) ECIR 2021. LNCS, vol. 12657, pp. 639–649. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72240-1_75

    Chapter  Google Scholar 

  73. Nakov, P., et al.: Overview of the CLEF–2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In: Candan, K.S., et al. (eds.) CLEF 2021. LNCS, vol. 12880, pp. 264–291. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85251-1_19

    Chapter  Google Scholar 

  74. Nguyen, D.Q., Vu, T., Nguyen, A.T.: BERTweet: a pre-trained language model for English tweets. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 9–14 (2020)

    Google Scholar 

  75. Nguyen, V.H., Sugiyama, K., Nakov, P., Kan, M.Y.: FANG: leveraging social context for fake news detection using graph representation. In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management, pp. 1165–1174. CIKM 2020 (2020)

    Google Scholar 

  76. Pachov, G., Dimitrov, D., Koychev, I., Nakov, P.: Gpachov at CheckThat! 2023: a diverse multi-approach ensemble for subjectivity detection in news articles. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  77. Panayotov, P., Shukla, U., Sencar, H.T., Nabeel, M., Nakov, P.: GREENER: graph neural networks for news media profiling. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7470–7480. EMNLP 2022, Abu Dhabi, UAE (2022)

    Google Scholar 

  78. Piskorski, J., Stefanovitch, N., Da San Martino, G., Nakov, P.: SemEval-2023 task 3: detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In: Proceedings of the 17th International Workshop on Semantic Evaluation. SemEval 2023, Toronto, Canada (2023)

    Google Scholar 

  79. Pogorelov, K., et al.: FakeNews: Corona virus and 5G conspiracy task at MediaEval 2020. In: MediaEval (2020)

    Google Scholar 

  80. Reddy, R.G., et al.: NewsClaims: a new benchmark for claim detection from news with attribute knowledge. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6002–6018. EMNLP 2022, Abu Dhabi, UAE (2022)

    Google Scholar 

  81. Sadouk, H.T., Sebbak, F., Zekiri, H.E.: Es-vrai at CheckThat! 2023: lcheck worthiness in multimodal and multigenre contents through fusion and sampling approaches. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  82. Sadouk, H.T., Sebbak, F., Zekiri, H.E.: Es-vrai at CheckThat! 2023: Enhancing model performance for subjectivity detection through multilingual data augmentation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  83. Sadouk, H.T., Sebbak, F., Zekiri, H.E.: Es-vrai at CheckThat! 2023: Leveraging bio and lists information for enhanced rumor verification in twitter. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  84. Safaya, A., Abdullatif, M., Yuret, D.: KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 2054–2059. International Committee for Computational Linguistics, Barcelona (online) (2020)

    Google Scholar 

  85. Sawiński, M., et al.: Openfact at CheckThat! 2023: Head-to-head GPT vs. BERT - a comparative study of transformers language models for the detection of check-worthy claims. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  86. Schlicht, I.B., Khellaf, L., Altiok, D.: Dwreco at CheckThat! 2023: enhancing subjectivity detection through style-based data sampling. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  87. Shaar, S., Alam, F., Da San Martino, G., Nakov, P.: The role of context in detecting previously fact-checked claims. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1619–1631 (2022)

    Google Scholar 

  88. Shaar, S., Babulkov, N., Da San Martino, G., Nakov, P.: That is a known lie: detecting previously fact-checked claims. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3607–3618 (2020)

    Google Scholar 

  89. Shaar, S., et al.: Overview of the CLEF-2021 CheckThat! lab task 2 on detecting previously fact-checked claims in tweets and political debates. In: Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.) CLEF 2021 working notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum (2021)

    Google Scholar 

  90. Shaar, S., et al.: Overview of the CLEF-2021 CheckThat! lab task 1 on check-worthiness estimation in tweets and political debates. In: Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.) CLEF 2021 working notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum (2021)

    Google Scholar 

  91. Shaar, S.,et al.: Overview of CheckThat! 2020 English: Automatic identification and verification of claims in social media. In: Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.) CLEF 2020 Working Notes. CEUR Workshop Proceedings (2020)

    Google Scholar 

  92. Shahi, G.K., Struß, J.M., Mandl, T.: Overview of the CLEF-2021 CheckThat! lab: Task 3 on fake news detection. In: Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.) CLEF 2021 working notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum (2021)

    Google Scholar 

  93. Sharma, S., et al.: Findings of the CONSTRAINT 2022 shared task on detecting the hero, the villain, and the victim in memes. In: Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pp. 1–11. Dublin, Ireland (2022)

    Google Scholar 

  94. Sheikh Ali, Z., Mansour, W., Haouari, F., Hasanain, M., Elsayed, T., Al-Ali, A.: Tahaqqaq: a real-time system for assisting twitter users in Arabic claim verification. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (2023)

    Google Scholar 

  95. Shushkevich, E., Cardiff, J.: Tudublin at CheckThat! 2023: Chatgpt for data augmentation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  96. Stefanov, P., Darwish, K., Atanasov, A., Nakov, P.: Predicting the topical stance and political leaning of media using tweets. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 527–537. ACL 2020 (2020)

    Google Scholar 

  97. Tarannum, P., Hasan, M.A., Alam, F., Noori, S.R.H.: Z-Index at CheckThat! 2023: Unimodal and multimodal checkworthiness classification. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  98. Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a large-scale dataset for fact extraction and VERification. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 809–819. NAACL-HLT 2018, New Orleans, Louisiana, USA (2018)

    Google Scholar 

  99. Tran, S., Rodrigues, P., Strauss, B., Williams, E.: Accenture at CheckThat! 2023: Identifying claims with societal impact using NLP data augmentation. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  100. Tran, S., Rodrigues, P., Strauss, B., Williams, E.: Accenture at CheckThat! 2023: Impacts of back-translation on subjectivity detection. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  101. Tran, S., Rodrigues, P., Strauss, B., Williams, E.: Accenture at CheckThat! 2023: Learning to detect factuality levels of news sources. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  102. Tran, S., Rodrigues, P., Strauss, B., Williams, E.: Accenture at CheckThat! 2023: Learning to detect political bias of news articles and sources. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  103. Truică C.O., Apostol, E.S., Paschke, A.: Awakened at CheckThat! 2022: fake news detection using BiLSTM and sentence transformer. In: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum. CLEF 2022, Bologna, Italy (2022)

    Google Scholar 

  104. Vo, N., Lee, K.: The rise of guardians: fact-checking url recommendation to combat fake news. In: Proceedings of the International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 275–284. SIGIR 2018 (2018)

    Google Scholar 

  105. von Däniken, P., Deriu, J., Cieliebak, M.: Zhaw-cai at CheckThat! 2023: Ensembling using kernel averaging. In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, Michalis (eds.) Working Notes of CLEF 2023. Conference and Labs of the Evaluation Forum CLEF 2023, Thessaloniki, Greece (2023)

    Google Scholar 

  106. Wang, N.X., Mahajan, D., Danilevsky, M., Rosenthal, S.: SemEval-2021 task 9: fact verification and evidence finding for tabular data in scientific documents (SEM-TAB-FACTS). In: Proceedings of the 15th International Workshop on Semantic Evaluation, pp. 317–326. SemEval 2021 (2021)

    Google Scholar 

  107. Yasser, K., Kutlu, M., Elsayed, T.: Re-ranking web search results for better fact-checking: a preliminary study. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 1783–1786 (2018)

    Google Scholar 

  108. Zubiaga, A., Liakata, M., Procter, R., Wong Sak Hoi, G., Tolmie, P.: Analysing how people orient to and spread rumours in social media by looking at conversational threads. PloS one 11(3), e0150989 (2016)

    Google Scholar 

Download references

Acknowledgments

The work of F. Alam, M. Hasanain and W. Zaghouani is partially supported by NPRP 13S-0206-200281 and NPRP 14C-0916-210015 from the Qatar National Research Fund (a member of Qatar Foundation). The work of A. Galassi is supported by the European Commission NextGeneration EU programme, PNRR-M4C2-Investimento 1.3, PE00000013-“FAIR” Spoke 8. The work of Fatima Haouari was supported by GSRA grant #GSRA6-1-0611-19074 from the Qatar National Research Fund. The work of Tamer Elsayed was made possible by NPRP grant #NPRP-11S-1204-170060 from the Qatar National Research Fund.

The findings achieved herein are solely the responsibility of the authors.

Author information

Authors and Affiliations

  1. Università di Bologna, Bologna, Italy

    Alberto Barrón-Cedeño, Andrea Galassi & Federico Ruggeri

  2. Qatar Computing Research Institute, Ar-Rayyan, Qatar

    Firoj Alam & Maram Hasanain

  3. University of Padua, Padua, Italy

    Giovanni Da San Martino

  4. Mohamed bin Zayed University of Artificial Intelligence, Masdar, UAE

    Preslav Nakov

  5. Qatar University, Doha, Qatar

    Tamer Elsayed, Dilshod Azizov & Fatima Haouari

  6. University of Groningen, Groningen, The Netherlands

    Tommaso Caselli

  7. L3S Research Center, Leibniz University of Hannover, Hanover, Germany

    Gullal S. Cheema

  8. TOBB University of Economics and Technology, Ankara, Turkey

    Mucahid Kutlu

  9. University of Texas at Arlington, Arlington, USA

    Chengkai Li

  10. University of Applied Sciences Potsdam, Potsdam, Germany

    Julia Maria Struß

  11. Hamad bin Khalifa University, Ar-Rayyan, Qatar

    Wajdi Zaghouani

Authors
  1. Alberto Barrón-Cedeño
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Firoj Alam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Galassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giovanni Da San Martino
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Preslav Nakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tamer Elsayed
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dilshod Azizov
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Tommaso Caselli
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Gullal S. Cheema
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Fatima Haouari
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Maram Hasanain
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Mucahid Kutlu
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Chengkai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Federico Ruggeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Julia Maria Struß
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Wajdi Zaghouani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto Barrón-Cedeño .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. CERTH-ITI, Thessaloniki, Greece

    Theodora Tsikrika

  4. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Utrecht University, Utrecht, The Netherlands

    Anastasia Giachanou

  6. Elsevier, Amsterdam, The Netherlands

    Dan Li

  7. University of Amsterdam, Amsterdam, The Netherlands

    Mohammad Aliannejadi

  8. University of Lausanne, Lausanne, Switzerland

    Michalis Vlachos

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Barrón-Cedeño, A. et al. (2023). Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their Source. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42448-9_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42447-2

  • Online ISBN: 978-3-031-42448-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation