Abstract
Writing medical image reports is an inefficient and time-consuming task for doctors. Automatically generating medical reports is an essential task of medical data mining, which can alleviate the workload of doctors and improve the standardization of reports. However, the existing methods mainly adopt the CNN-RNN structure to align image features with text features. This structure has difficulty dealing with the dependencies between distant text locations, leading to inconsistent context and semantics in the generated report. In this paper, we propose a knowledge-driven transformer (KdTNet) model for generating coherent medical reports. First, the visual grid and graph convolutional modules are devised to extract fine-grained visual features. Second, we adopt the transformer decoder to generate the hidden semantic states. Subsequently, a BERT-based auxiliary language module is employed to obtain the context language features of reports from the pre-defined medical term knowledge. We design a multimodal information fusion module to adaptively calculate the contribution of visual and linguistic features to the report for report generation. Extensive experiments on two real datasets explicate that our KdTNet model has achieved superior performance in captioning metrics and human evaluation compared with the state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, Z., Song, Y., Chang, T., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 1439–1449 (2020)
Delrue, L., et al.: Difficulties in the interpretation of chest radiography. In: Comparative Interpretation of CT and Standard Radiography of the Chest, pp. 27–49 (2011)
Demner-Fushman, et al.: Preparing a collection of radiology examinations for distribution and retrieval. Med. Inform. Assoc. 23(2), 304–310 (2016)
Devlin, J., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT 2019, pp. 4171–4186 (2019)
Du, G., Zhou, L., Yang, Y., Lü, K., Wang, L.: Deep multiple auto-encoder-based multi-view clustering. Data Sci. Eng. 6(3), 323–338 (2021)
Han, Z., et al.: Unifying neural learning and symbolic reasoning for spinal medical report generation. Med. Image Anal. 67, 101872 (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE CVPR, pp. 770–778 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE CVPR, pp. 2261–2269 (2017)
Jieba: “Jieba” Chinese text segmentation: built to be the best python Chinese word segmentation module (2018). https://github.com/fxsjy/jieba
Jing, B., Xie, P., Xing, E.P.: On the automatic generation of medical imaging reports. In: ACL, pp. 2577–2586 (2018)
Li, C.Y., Liang, X., Hu, Z., Xing, E.P.: Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In: AAAI, pp. 6666–6673 (2019)
Li, M., Wang, F., Chang, X., Liang, X.: Auxiliary signal-guided knowledge encoder-decoder for medical report generation (2020)
Li, Y., Liang, X., Hu, Z., Xing, E.P.: Hybrid retrieval-generation reinforced agent for medical image report generation. In: NeurIPS, pp. 1537–1547 (2018)
Liang, X., Hu, Z., Zhang, H., Lin, L., Xing, E.P.: Symbolic graph reasoning meets convolutions. In: NeurIPS, pp. 1858–1868 (2018)
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pp. 13753–13762. IEEE (2021)
Liu, Y., et al.: Pre-training graph transformer with multimodal side information for recommendation. In: MM 2021, pp. 2853–2861. ACM (2021)
Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via a visual sentinel for image captioning. In: IEEE CVPR, pp. 3242–3250 (2017)
Norcliffe-Brown, W., Vafeias, S., Parisot, S.: Learning conditioned graph structures for interpretable visual question answering. In: NeurIPS 2018, pp. 8344–8353 (2018)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: ACL, pp. 311–318 (2002)
Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: IEEE CVPR, pp. 1179–1195 (2017)
Shin, H., et al.: Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In: IEEE CVPR, pp. 2497–2506 (2016)
Song, Y., Shi, S.: Complementary learning of word embeddings. In: IJCAI 2018, pp. 4368–4374 (2018)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)
Tian, Y., et al.: Joint Chinese word segmentation and part-of-speech tagging via two-way attentions of auto-analyzed knowledge. In: ACL 2020 (2020)
Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: IEEE CVPR, pp. 4566–4575 (2015)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: IEEE CVPR, pp. 3156–3164 (2015)
Xue, Y., et al.: Multimodal recurrent model with attention for automated radiology report generation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 457–466. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_52
Yao, T., Pan, Y., Li, Y., Mei, T.: Exploring visual relationship for image captioning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 711–727. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_42
You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: IEEE CVPR, pp. 4651–4659 (2016)
Zhang, H., et al.: Multiplex word embeddings for selectional preference acquisition. In: Proceedings of the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pp. 5246–5255 (2019)
Zhang, W., Ying, Y., Lu, P., Zha, H.: Learning long- and short-term user literal-preference with multimodal hierarchical transformer network for personalized image caption. In: AAAI 2020, pp. 9571–9578. AAAI Press (2020)
Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A.L., Xu, D.: When radiology report generation meets knowledge graph. In: AAAI, pp. 12910–12917 (2020)
Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network. In: IEEE CVPR 2017, pp. 3549–3557 (2017)
Acknowledgements
The research work was supported by the National Key R&D Program of China (No. 2021YFF0900800), the major Science and Technology Innovation of Shandong Province grant (No. 2021CXGC010108), the Shandong Provincial Key Research and Development Program (Major Scientific and Technological Innovation Project) (No. 2019JZZY011007), and partially supported by the NSFC (No. 91846205).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cao, Y. et al. (2022). KdTNet: Medical Image Report Generation via Knowledge-Driven Transformer. In: Bhattacharya, A., et al. Database Systems for Advanced Applications. DASFAA 2022. Lecture Notes in Computer Science, vol 13247. Springer, Cham. https://doi.org/10.1007/978-3-031-00129-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-00129-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-00128-4
Online ISBN: 978-3-031-00129-1
eBook Packages: Computer ScienceComputer Science (R0)