Abstract
The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural language inference, named entity recognition) on a number of novel text sets. Given the sensitive nature of the data in healthcare, such a benchmark partially closes the problem of Russian medical dataset absence. We prepare the unified format labeling, data split, and evaluation metrics for new tasks. The remaining tasks are from existing datasets with a few modifications. A single-number metric expresses a model’s ability to cope with the benchmark. Moreover, we implement several baseline models, from simple ones to neural networks with transformer architecture, and release the code. Expectedly, the more advanced models yield better performance, but even a simple model is enough for a decent result in some tasks. Furthermore, for all tasks, we provide a human evaluation. Interestingly the models outperform humans in the large-scale classification tasks. However, the advantage of natural intelligence remains in the tasks requiring more knowledge and reasoning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alsentzer, E., et al.: Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72–78. Association for Computational Linguistics, Minneapolis, Minnesota, USA (2019)
Blinov, P., Avetisian, M., Kokh, V., Umerenkov, D., Tuzhilin, A.: Predicting clinical diagnosis from patients electronic health records using BERT-based neural networks. In: Michalowski, M., Moskovitch, R. (eds.) AIME 2020. LNCS (LNAI), vol. 12299, pp. 111–121. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59137-3_11
Blinov, P., Nesterov, A., Zubkova, G., Reshetnikova, A., Kokh, V., Shivade, C.: RuMedNLI: a russian natural language inference dataset for the clinical domain. PhysioNet (2022). https://doi.org/10.13026/gxzd-cf80
Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(suppl-1), D267–D270 (2004)
Clark, C., Lee, K., Chang, M.W., Kwiatkowski, T., Collins, M., Toutanova, K.: Boolq: exploring the surprising difficulty of natural yes/no questions. arXiv preprint. arXiv:1905.10044 (2019)
Crammer, K., Dredze, M., Ganchev, K., Talukdar, P., Carroll, S.: Automatic code assignment to medical text. In: Biological, translational, and clinical language processing, pp. 129–136 (2007)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE (2009)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint. arXiv:1810.04805 (2018)
Gers, F., Schmidhuber, E.: LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Netw. 12(6), 1333–1340 (2001)
Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning: data mining, inference and prediction, 2 edn. Springer (2009). https://doi.org/10.1007/978-0-387-21606-5
Johnson, A.E., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3(1), 1–9 (2016)
Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint. arXiv:1905.07213 (2019)
Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Brodley, C.E., Danyluk, A.P. (eds.) Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, 28 June - 1 July 2001, pp. 282–289. Morgan Kaufmann (2001)
Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
Lewis, P., Ott, M., Du, J., Stoyanov, V.: Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 146–157 (2020)
Peng, Y., Yan, S., Lu, Z.: Transfer learning in biomedical natural language processing: an evaluation of bert and ELMO on ten benchmarking datasets. arXiv preprint. arXiv:1906.05474 (2019)
Purushotham, S., Meng, C., Che, Z., Liu, Y.: Benchmark of deep learning models on large healthcare mimic datasets. arXiv preprint. arXiv:1710.08531 (2017)
Romanov, A., Shivade, C.: Lessons from natural language inference in the clinical domain. arXiv preprint. arXiv:1808.06752 (2018)
Schütze, H., Manning, C.D., Raghavan, P.: Introduction to Information Retrieval, vol. 39. Cambridge University Press, Cambridge (2008)
Shavrina, T., et al.: RussianSuperGLUE: a Russian language understanding evaluation benchmark. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4717–4726. Association for Computational Linguistics (2020)
Shelmanov, A., Smirnov, I., Vishneva, E.: Information extraction from clinical texts in Russian. In: Computational Linguistics and Intellectual Technologies, pp. 560–572 (2015)
Starovoytova, E.A., et al.: Rumedprimedata (2021). https://doi.org/10.5281/zenodo.5765873
Tutubalina, E., et al.: The Russian drug reaction corpus and neural models for drug reactions and effectiveness detection in user reviews. Bioinformatics 37(2), 243–249 (2020)
Wang, A., et al.: Superglue: a stickier benchmark for general-purpose language understanding systems. arXiv preprint. arXiv:1905.00537 (2019)
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Glue: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint. arXiv:1804.07461 (2018)
WHO: International statistical classification of diseases and related health problems. World Health Organization, 10th revision, fifth edition, 2016 edn. (2015)
Zhang, N., et al.: Cblue: a chinese biomedical language understanding evaluation benchmark. arXiv preprint. arXiv:2106.08087 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Blinov, P., Reshetnikova, A., Nesterov, A., Zubkova, G., Kokh, V. (2022). RuMedBench: A Russian Medical Language Understanding Benchmark. In: Michalowski, M., Abidi, S.S.R., Abidi, S. (eds) Artificial Intelligence in Medicine. AIME 2022. Lecture Notes in Computer Science(), vol 13263. Springer, Cham. https://doi.org/10.1007/978-3-031-09342-5_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-09342-5_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09341-8
Online ISBN: 978-3-031-09342-5
eBook Packages: Computer ScienceComputer Science (R0)