Skip to main content
SpringerLink
Log in

Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12260))

Abstract

In this paper, we present an overview of the eighth edition of the BioASQ challenge, which ran as a lab in the Conference and Labs of the Evaluation Forum (CLEF) 2020. BioASQ is a series of challenges aiming at the promotion of systems and methodologies for large-scale biomedical semantic indexing and question answering. To this end, shared tasks are organized yearly since 2012, where different teams develop systems that compete on the same demanding benchmark datasets that represent the real information needs of experts in the biomedical domain. This year, the challenge has been extended with the introduction of a new task on medical semantic indexing in Spanish. In total, 34 teams with more than 100 systems participated in the three tasks of the challenge. As in previous years, the results of the evaluation reveal that the top-performing systems managed to outperform the strong baselines, which suggests that state-of-the-art systems keep pushing the frontier of research through continuous improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://pubmed.ncbi.nlm.nih.gov/.

  2. 2.

    IBECS includes bibliographic references from scientific articles in health sciences published in Spanish journals. http://ibecs.isciii.es.

  3. 3.

    LILACS is the most important and comprehensive index of scientific and technical literature of Latin America and the Caribbean. It includes 26 countries, 882 journals and 878,285 records, 464,451 of which are full texts https://lilacs.bvsalud.org.

  4. 4.

    Registro Español de Estudios Clínicos, a database containing summaries of clinical trials https://reec.aemps.es/reec/public/web.html.

  5. 5.

    Public healthcare project proposal summaries (Proyectos de Investigación en Salud, diseñado por el Instituto de Salud Carlos III, ISCIII) https://portalfis.isciii.es/es/Paginas/inicio.aspx.

  6. 6.

    29,716 come directly from MeSH and 4,402 are exclusive to DeCS.

  7. 7.

    https://radimrehurek.com/gensim/.

  8. 8.

    https://scikit-learn.org/.

  9. 9.

    https://project.carrot2.org/.

  10. 10.

    https://ai.googleblog.com/2020/05/an-nlu-powered-tool-to-explore-covid-19.html.

  11. 11.

    https://pypi.org/project/language-check/.

  12. 12.

    https://huggingface.co/gsarti/biobert-nli.

  13. 13.

    http://participants-area.bioasq.org/results/8a/.

  14. 14.

    http://participants-area.bioasq.org/Tasks/b/eval_meas_2020/.

  15. 15.

    http://participants-area.bioasq.org/results/8b/phaseA/.

  16. 16.

    http://participants-area.bioasq.org/results/8b/phaseB/.

References

  1. Almagro, M., Unanue, R.M., Fresno, V., Montalvo, S.: ICD-10 coding of Spanish electronic discharge summaries: an extreme classification problem. IEEE Access 8, 100073–100083 (2020)

    Article  Google Scholar 

  2. Almeida, T., Matos, S.: Calling attention to passages for biomedical question answering. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 69–77. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_9

    Chapter  Google Scholar 

  3. Baldwin, B., Carpenter, B.: Lingpipe. Available from World Wide Web (2033). http://alias-i.com/lingpipe

  4. Balikas, G., et al.: Evaluation framework specifications. Project deliverable D4.1, UPMC (05/2013 2013)

    Google Scholar 

  5. Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(suppl\_1), D267–D270 (2004)

    Google Scholar 

  6. Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 (2015)

  7. Chang, W.C., Yu, H.F., Zhong, K., Yang, Y., Dhillon, I.: X-BERT: eXtreme multi-label text classification with using bidirectional encoder representations from transformers. arXiv preprint arXiv:1905.02331 (2019)

  8. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: Electra: pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020)

  9. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 (2017)

  10. Couto, F.M., Lamurias, A.: MER: a shell script and annotation server for minimal named entity recognition and linking. J. Cheminform. 10(1), 1–10 (2018). https://doi.org/10.1186/s13321-018-0312-9

    Article  Google Scholar 

  11. Demsar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)

    MathSciNet  MATH  Google Scholar 

  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT 2019–2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference 1(Mlm), pp. 4171–4186, October 2018. http://arxiv.org/abs/1810.04805

  13. Erkan, G., Radev, D.R.: Lexrank: graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res. 22, 457–479 (2004)

    Article  Google Scholar 

  14. Gormley, C., Tong, Z.: Elasticsearch: The Definitive Guide: A Distributed Real-time Search and Analytics Engine. O’Reilly Media Inc., Sebastopol (2015)

    Google Scholar 

  15. Gururangan, S., et al.: Don’t stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 (2020)

  16. Jain, H., Prabhu, Y., Varma, M.: Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2016, pp. 935–944. ACM Press, New York (2016). https://doi.org/10.1145/2939672.2939756

  17. Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: PubMedQA: a dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019)

  18. Joshi, M., Chen, D., Liu, Y., Weld, D.S., Zettlemoyer, L., Levy, O.: Spanbert: improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguist. 8, 64–77 (2020)

    Article  Google Scholar 

  19. Kim, D., et al.: A neural named entity recognition and multi-type normalization tool for biomedical text mining. IEEE Access 7, 73729–73740 (2019)

    Article  Google Scholar 

  20. Kosmopoulos, A., Partalas, I., Gaussier, E., Paliouras, G., Androutsopoulos, I.: Evaluation measures for hierarchical classification: a unified view and novel approaches. Data Min. Knowl. Disc. 29(3), 820–865 (2014). https://doi.org/10.1007/s10618-014-0382-x

    Article  MathSciNet  MATH  Google Scholar 

  21. Krallinger, M., Krithara, A., Nentidis, A., Paliouras, G., Villegas, M.: BioASQ at CLEF2020: large-scale biomedical semantic indexing and question answering. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 550–556. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_71

    Chapter  Google Scholar 

  22. Kudo, T., Richardson, J.: SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71. Association for Computational Linguistics, Stroudsburg (2018). https://doi.org/10.18653/v1/D18-2012

  23. Lee, J., et al.: BIOBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746 (2019)

  24. Lewis, M., et al.: Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)

  25. Loper, E., Bird, S.: NLTK: the natural language toolkit. arXiv preprint arXiv:cs/0205028 (2002)

  26. Ma, J., Korotkov, I., Yang, Y., Hall, K., McDonald, R.: Zero-shot neural retrieval via domain-targeted synthetic query generation. arXiv preprint arXiv:2004.14503 (2020)

  27. Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404–411 (2004)

    Google Scholar 

  28. Mollá, D., Jones, C.: Classification betters regression in query-based multi-document summarisation techniques for question answering. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1168, pp. 624–635. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43887-6_56

    Chapter  Google Scholar 

  29. Mork, J.G., Demner-Fushman, D., Schmidt, S.C., Aronson, A.R.: Recent enhancements to the NLM medical text indexer. In: Proceedings of Question Answering Lab at CLEF (2014)

    Google Scholar 

  30. Nentidis, A., Bougiatiotis, K., Krithara, A., Paliouras, G.: Results of the seventh edition of the BioASQ challenge. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1168, pp. 553–568. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43887-6_51

    Chapter  Google Scholar 

  31. Neumann, M., King, D., Beltagy, I., Ammar, W.: ScispaCy: fast and robust models for biomedical natural language processing. arXiv preprint arXiv:1902.07669 (2019)

  32. Ozyurt, I.B., Bandrowski, A., Grethe, J.S.: Bio-AnswerFinder: a system to find answers to questions from biomedical texts. Database 2020, 1–12 (2020)

    Article  Google Scholar 

  33. Pang, L., Lan, Y., Guo, J., Xu, J., Xu, J., Cheng, X.: DeepRank: a new deep architecture for relevance ranking in information retrieval. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 257–266 (2017)

    Google Scholar 

  34. Pappas, D., McDonald, R., Brokos, G.I., Androutsopoulos, I.: AUEB at BioASQ 7: document and snippet retrieval. In: Seventh BioASQ Workshop: A Challenge on Large-scale Biomedical Semantic Indexing and Question Answering (2019)

    Google Scholar 

  35. Peng, S., You, R., Wang, H., Zhai, C., Mamitsuka, H., Zhu, S.: DeepMeSH: deep semantic representation for improving large-scale mesh indexing. Bioinformatics 32(12), i70–i79 (2016)

    Article  Google Scholar 

  36. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 31–40, February 2018. http://arxiv.org/abs/1802.05365

  37. Rae, A., Mork, J., Demner-Fushman, D.: Convolutional neural network for automatic MeSH indexing. In: Seventh BioASQ Workshop: A Challenge on Large-scale Biomedical Semantic Indexing and Question Answering (2019)

    Google Scholar 

  38. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)

  39. Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084 (2019)

  40. Ribadas, F.J., De Campos, L.M., Darriba, V.M., Romero, A.E.: CoLe and UTAIat BioASQ 2015: experiments with similarity based descriptor assignment. In: CEUR Workshop Proceedings, vol. 1391 (2015)

    Google Scholar 

  41. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  42. Smith, L., et al.: Overview of BioCreative II gene mention recognition. Genome Biol. 9(S2), S2 (2008). https://doi.org/10.1186/gb-2008-9-s2-s2

    Article  Google Scholar 

  43. Tsatsaronis, G., et al.: An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinform. 16, 138 (2015). https://doi.org/10.1186/s12859-015-0564-6

    Article  Google Scholar 

  44. Tsoumakas, G., Laliotis, M., Markontanatos, N., Vlahavas, I.: Large-scale semantic indexing of biomedical publications. In: 1st BioASQ Workshop: A Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering (2013)

    Google Scholar 

  45. Wei, C.H., Leaman, R., Lu, Z.: Beyond accuracy: creating interoperable and scalable text-mining web services. Bioinformatics (Oxford, England) 32(12), 1907–10 (2016). https://doi.org/10.1093/bioinformatics/btv760

    Article  Google Scholar 

  46. Williams, A., Nangia, N., Bowman, S.R.: A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 (2017)

  47. Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. CoRR abs/1906.08237 (2019). http://arxiv.org/abs/1906.08237

  48. Yang, Z., Zhou, Y., Eric, N.: Learning to answer biomedical questions: OAQA at BioASQ 4B. In: ACL 2016, p. 23 (2016)

    Google Scholar 

  49. Yoon, W., Lee, J., Kim, D., Jeong, M., Kang, J.: Pre-trained language model for biomedical question answering. In: Seventh BioASQ Workshop: A Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering (2019)

    Google Scholar 

  50. You, R., Zhang, Z., Wang, Z., Dai, S., Mamitsuka, H., Zhu, S.: AttentionXML: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. arXiv preprint arXiv:1811.01727 (2018)

  51. Zavorin, I., Mork, J.G., Demner-Fushman, D.: Using learning-to-rank to enhance NLM medical text indexer results. In: ACL 2016, p. 8 (2016)

    Google Scholar 

Download references

Acknowledgments

Google was a proud sponsor of the BioASQ Challenge in 2019. The eighth edition of BioASQ is also sponsored by the Atypon Systems inc. BioASQ is grateful to NLM for providing the baselines for task 8a and to the CMU team for providing the baselines for task 8b. The MESINESP task is sponsored by the Spanish Plan for advancement of Language Technologies (Plan TL) and the Secretaría de Estado para el Avance Digital (SEAD). BioASQ is also grateful to LILACS, SCIELO and Biblioteca virtual en salud and Instituto de salud Carlos III for providing data for the BioASQ MESINESP task.

Author information

Authors and Affiliations

  1. National Center for Scientific Research “Demokritos”, Athens, Greece

    Anastasios Nentidis, Anastasia Krithara, Konstantinos Bougiatiotis & Georgios Paliouras

  2. Aristotle University of Thessaloniki, Thessaloniki, Greece

    Anastasios Nentidis

  3. National and Kapodistrian University of Athens, Athens, Greece

    Konstantinos Bougiatiotis

  4. Barcelona Supercomputing Center, Barcelona, Spain

    Martin Krallinger, Carlos Rodriguez-Penagos & Marta Villegas

Authors
  1. Anastasios Nentidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastasia Krithara
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Konstantinos Bougiatiotis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin Krallinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carlos Rodriguez-Penagos
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marta Villegas
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Georgios Paliouras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anastasios Nentidis .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Theodora Tsikrika

  4. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Faculty of Library, Information and Media Science, University of Tsukuba, Ibaraki, Japan

    Hideo Joho

  6. Department of Computer Science, University of Copenhagen, Copenhagen, Denmark

    Christina Lioma

  7. Brown University, Providence, RI, USA

    Carsten Eickhoff

  8. LIMSI-CNRS, Orsay, France

    Aurélie Névéol

  9. Department of Information Engineering, University of Padova, Padua, Italy

    Linda Cappellato

  10. Department of Information Engineering, University of Padova, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nentidis, A. et al. (2020). Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2020. Lecture Notes in Computer Science(), vol 12260. Springer, Cham. https://doi.org/10.1007/978-3-030-58219-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58219-7_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58218-0

  • Online ISBN: 978-3-030-58219-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation