Abstract
The ambiguity of the decision-making process has been pointed out as the main obstacle to practically applying the deep learning-based method in spite of its outstanding performance. Interpretability can guarantee the confidence of the deep learning system, therefore it is particularly important in the medical field. In this study, a novel deep network is proposed to explain the diagnostic decision with visual pointing map and diagnostic sentence justifying result simultaneously. To increase the accuracy of sentence generation, a visual word constraint model is devised in training justification generator. To verify the proposed method, comparative experiments were conducted on the problem of the diagnosis of breast masses. Experimental results demonstrated that the proposed deep network can explain diagnosis more accurately with various textual justifications.
This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2017-0-01779, A machine learning and statistical inference framework for explainable artificial intelligence).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Berment, H., Becette, V., Mohallem, M., Ferreira, F., Chérel, P.: Masses in mammography: what are the underlying anatomopathological lesions? Diagn. Intervent. Imag. 95(2), 124–133 (2014)
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255. IEEE (2009)
Heath, M., Bowyer, K., Kopans, D., Moore, R., Kegelmeyer, W.: The digital database for screening mammography. In: International Workshop on Digital Mammography, pp. 212–218. Medical Physics Publishing (2000)
Huk Park, D., et al.: Multimodal explanations: justifying decisions and pointing to the evidence. In: CVPR, pp. 8779–8788. IEEE (2018)
Kim, S., Lee, J., Lee, H., Ro, Y.: Visually interpretable deep network for diagnosis of breast masses on mammograms. Phys. Med. & Biol. 63(23), 235025 (2018)
Kim, S., Lee, J., Ro, Y.: Visual evidence for interpreting diagnostic decision of deep neural network in computer-aided diagnosis. In: Medical Imaging 2019: Computer-Aided Diagnosis, vol. 10950, p. 109500K. International Society for Optics and Photonics (2019)
Kim, Y.: Convolutional neural networks for sentence classification. In: Conference on Empirical Methods in Natural Language Processing, pp. 1746–1751 (2014)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. ICLR (2015)
Kooi, T., Litjens, G., Van Ginneken, B., Gubern-Mérida, A., Sánchez, C.I., Mann, R., den Heeten, A., Karssemeijer, N.: Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35, 303–312 (2017)
Lee, K., Talati, N., Oudsema, R., Steinberger, S., Margolies, L.: Bi-rads 3: current and future use of probably benign. Current Radiol. Rep. 6(2), 5 (2018)
Lin, C.: Rouge: a package for automatic evaluation of summaries. Text Summarization Branches Out (2004)
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)
Liu, X., Li, H., Shao, J., Chen, D., Wang, X.: Show, tell and discriminate: image captioning by self-retrieval with partially labeled data. In: ECCV (2018)
Moon, W., Lo, C., Chang, J., Huang, C., Chen, J., Chang, R.: Quantitative ultrasound analysis for classification of bi-rads category 3 breast masses. J. Digital Imag. 26(6), 1091–1098 (2013)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: Annual Meeting of the Association for Computational Linguistics. pp. 311–318 (2002)
of Radiology, A.C.: Breast Imaging Reporting and Data System® (BI-RADS®). American College of Radiology, Reston, Va, 4 edn. (2003)
Selvi, R.: Breast Diseases Imaging and Clinical Management. Springer India, New Delhi (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
Surendiran, B., Vadivel, A.: Mammogram mass classification using various geometric shape and margin features for early detection of breast cancer. Int. J. Med. Eng. Inform. 4(1), 36–54 (2012)
Thomassin-Naggara, I., Tardivon, A., Chopier, J.: Standardized diagnosis and reporting of breast cancer. Diagn. Interv. Imag. 95(7–8), 759–766 (2014)
Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: CVPR, pp. 4566–4575. IEEE (2015)
Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.: Tienet: text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: CVPR, pp. 9049–9058. IEEE (2018)
Wang, Y., Lin, Z., Shen, X., Cohen, S., Cottrell, G.: Skeleton key: image captioning by skeleton-attribute decomposition. In: CVPR, IEEE (2017)
Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: Mdnet: a semantically and visually interpretable medical image diagnosis network. In: CVPR, pp. 6428–6436. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Lee, H., Kim, S.T., Ro, Y.M. (2019). Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis. In: Suzuki, K., et al. Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. ML-CDS IMIMIC 2019 2019. Lecture Notes in Computer Science(), vol 11797. Springer, Cham. https://doi.org/10.1007/978-3-030-33850-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-33850-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33849-7
Online ISBN: 978-3-030-33850-3
eBook Packages: Computer ScienceComputer Science (R0)
