Abstract
A comprehensive X-ray imaging report greatly assists the medical professional to investigate an indispensable condition and medication. The preparation of an extensive and diversified medical report by analysing the chest X-ray image is a time-consuming task and requires highly experienced professionals. This work targets the fundamental problem of generating a long and multifarious medical report for the chest X-ray image. It introduces a novel Multilevel Multi-Attention based encoder-decoder approach by combining Context Level Visual Attention and Textual Attention to generate a plausible medical report for different views of chest X-ray images. It exploited the proven ability of the Convolutional Neural Network to acquire course information of visual-spatial regions as an encoder. It leverages the strength of the Long Short-Term Memory network to learn long sequential dependencies and the ability of attention to focus on the prominent section as a decoder. The proposed method emphasizes on contextual coherence in intra and inter-sentence dependency within a report to improve the overall medical report generation quality. The effectiveness of the proposed model is evaluated on the publicly available IU chest X-ray dataset consisting of chest images along with multifarious radiology reports. The final performance of the proposed model is reported using the COCO-caption evaluation API. It shows a significant improvement in a medical report generation task compared to state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, X., et al.: Microsoft COCO captions: data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)
Coche, E.E., Ghaye, B., de Mey, J., Duyck, P.: Comparative Interpretation of CT and Standard Radiography of the Chest. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-540-79942-9
Demner-Fushman, D., et al.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. 23(2), 304–310 (2015)
Dozat, T.: Incorporating Nesterov momentum into Adam (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint arXiv:1711.08195 (2017)
Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4565–4574 (2016)
Krause, J., Johnson, J., Krishna, R., Fei-Fei, L.: A hierarchical approach for generating descriptive image paragraphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 317–325 (2017)
Li, Y., Liang, X., Hu, Z., Xing, E.P.: Hybrid retrieval-generation reinforced agent for medical image report generation. In: Advances in Neural Information Processing Systems, pp. 1537–1547 (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)
Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M.: TieNet: text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9049–9058 (2018)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2015)
Xue, Y., et al.: Multimodal recurrent model with attention for automated radiology report generation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 457–466. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_52
Zhou, P., et al.: Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), vol. 2, pp. 207–212 (2016)
Aknowledgment
This research work was supported by Center for Intelligent Signal and Imaging Research (CISIR), University Teknologi PETRONAS (UTP), Malaysia and Shri Guru Gobind Singhji Institute of Engineering and Technology (SGGSIE&T), Nanded, India with the international grant (015ME0-018).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gajbhiye, G.O., Nandedkar, A.V., Faye, I. (2020). Automatic Report Generation for Chest X-Ray Images: A Multilevel Multi-attention Approach. In: Nain, N., Vipparthi, S., Raman, B. (eds) Computer Vision and Image Processing. CVIP 2019. Communications in Computer and Information Science, vol 1147. Springer, Singapore. https://doi.org/10.1007/978-981-15-4015-8_15
Download citation
DOI: https://doi.org/10.1007/978-981-15-4015-8_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-4014-1
Online ISBN: 978-981-15-4015-8
eBook Packages: Computer ScienceComputer Science (R0)