Paying Per-Label Attention for Multi-label Extraction from Radiology Reports

  • Patrick SchrempfEmail author
  • Hannah Watson
  • Shadia Mikhael
  • Maciej Pajak
  • Matúš Falis
  • Aneta Lisowska
  • Keith W. Muir
  • David Harris-Birtill
  • Alison Q. O’Neil
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12446)


Training medical image analysis models requires large amounts of expertly annotated data which is time-consuming and expensive to obtain. Images are often accompanied by free-text radiology reports which are a rich source of information. In this paper, we tackle the automated extraction of structured labels from head CT reports for imaging of suspected stroke patients, using deep learning. Firstly, we propose a set of 31 labels which correspond to radiographic findings (e.g. hyperdensity) and clinical impressions (e.g. haemorrhage) related to neurological abnormalities. Secondly, inspired by previous work, we extend existing state-of-the-art neural network models with a label-dependent attention mechanism. Using this mechanism and simple synthetic data augmentation, we are able to robustly extract many labels with a single model, classified according to the radiologist’s reporting (positive, uncertain, negative). This approach can be used in further research to effectively extract many labels from medical text.


NLP Radiology report labelling BERT 



This work is part of the Industrial Centre for AI Research in digital Diagnostics (iCAIRD) which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) [project number: 104690]. We would like to thank the Glasgow Safe Haven for assistance in creating and providing this dataset. Thanks also to The Data Lab for support and funding.

Supplementary material

506088_1_En_29_MOESM1_ESM.pdf (150 kb)
Supplementary material 1 (pdf 149 KB)


  1. 1.
    Alsentzer, E., et al.: Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72–78. Association for Computational Linguistics, Minneapolis, Minnesota, USA, Jun 2019.
  2. 2.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)Google Scholar
  3. 3.
    Banerjee, S., Akkaya, C., Perez-Sorrosal, F., Tsioutsiouliklis, K.: Hierarchical transfer learning for multi-label text classification. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6295–6300 (2019)Google Scholar
  4. 4.
    Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(90001), 267D–270 (2004). Scholar
  5. 5.
    Cho, K., van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. In: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111. Association for Computational Linguistics, Doha, Qatar, October 2014.
  6. 6.
    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019.
  7. 7.
    Drozdov, I., et al.: Supervised and unsupervised language modelling in chest x-ray radiological reports. Plos One 15(3), e0229963 (2020)CrossRefGoogle Scholar
  8. 8.
    Gorinski, P.J., et al.: Named entity recognition for electronic health records: a comparison of rule-based and machine learning approaches. arXiv preprint arXiv:1903.03985 (2019)
  9. 9.
    Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 590–597 (2019)Google Scholar
  10. 10.
    IST-3 collaborative group: Association between brain imaging signs, early and late outcomes, and response to intravenous alteplase after acute ischaemic stroke in the third International Stroke Trial (IST-3): secondary analysis of a randomised controlled trial. Lancet Neurol. 14, pp. 485–496 (2015).
  11. 11.
    Johnson, A.E., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016)CrossRefGoogle Scholar
  12. 12.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)Google Scholar
  13. 13.
    Loper, E., Bird, S.: NLTK: the natural language toolkit. In: Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Association for Computational Linguistics, Philadelphia (2002)Google Scholar
  14. 14.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  15. 15.
    Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., Eisenstein, J.: Explainable prediction of medical codes from clinical text. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1101–1111. Association for Computational Linguistics, New Orleans, Louisiana, Jun 2018.
  16. 16.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Radiological Society of North America: RSNA Intracranial Hemorrhage Detection (Kaggle challenge).
  18. 18.
    Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, ELRA, Valletta, Malta, pp. 45–50, May 2010Google Scholar
  19. 19.
    Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., Lungren, M.P.: CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv preprint arXiv:2004.09167 (2020)
  20. 20.
    Wolf, T., et al.: HuggingFace’s transformers: state-of-the-art natural language processing. ArXiv abs/1910.03771 (2019)Google Scholar
  21. 21.
    Wood, D., et al.: Automated labelling using an attention model for radiology reports of MRI scans (ALARM). In: Medical Imaging with Deep Learning (2020).
  22. 22.
    Yadav, K., Sarioglu, E., Choi, H., Cartwright IV, W.B., Hinds, P.S., Chamberlain, J.M.: Automated outcome classification of computed tomography imaging reports for pediatric traumatic brain injury. Acad. Emerg. Med. 23(2), 171–178 (2016). Scholar
  23. 23.
    Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489 (2016)Google Scholar
  24. 24.
    Yetisgen-Yildiz, M., Gunn, M.L., Xia, F., Payne, T.H.: A text processing pipeline to extract recommendations from radiology reports. J. Biomed. Inf. 46(2), 354–362 (2013)CrossRefGoogle Scholar
  25. 25.
    Zech, J., et al.: Natural language-based machine learning models for the annotation of clinical radiology reports. Radiology 287(2), 570–580 (2018)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Patrick Schrempf
    • 1
    • 2
    Email author
  • Hannah Watson
    • 1
  • Shadia Mikhael
    • 1
  • Maciej Pajak
    • 1
  • Matúš Falis
    • 1
  • Aneta Lisowska
    • 1
  • Keith W. Muir
    • 3
  • David Harris-Birtill
    • 2
  • Alison Q. O’Neil
    • 1
    • 4
  1. 1.Canon Medical Research EuropeEdinburghUK
  2. 2.University of St AndrewsSt AndrewsUK
  3. 3.Institute of Neuroscience & PsychologyUniversity of GlasgowGlasgowUK
  4. 4.University of EdinburghEdinburghUK

Personalised recommendations