Iterative Attention Mining for Weakly Supervised Thoracic Disease Pattern Localization in Chest X-Rays

  • Jinzheng CaiEmail author
  • Le Lu
  • Adam P. Harrison
  • Xiaoshuang Shi
  • Pingjun Chen
  • Lin Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11071)


Given image labels as the only supervisory signal, we focus on harvesting/mining, thoracic disease localizations from chest X-ray images. Harvesting such localizations from existing datasets allows for the creation of improved data sources for computer-aided diagnosis and retrospective analyses. We train a convolutional neural network (CNN) for image classification and propose an attention mining (AM) strategy to improve the model’s sensitivity or saliency to disease patterns. The intuition of AM is that once the most salient disease area is blocked or hidden from the CNN model, it will pay attention to alternative image regions, while still attempting to make correct predictions. However, the model requires to be properly constrained during AM, otherwise, it may overfit to uncorrelated image parts and forget the valuable knowledge that it has learned from the original image classification task. To alleviate such side effects, we then design a knowledge preservation (KP) loss, which minimizes the discrepancy between responses for X-ray images from the original and the updated networks. Furthermore, we modify the CNN model to include multi-scale aggregation (MSA), improving its localization ability on small-scale disease findings, e.g., lung nodules. We validate our method on the publicly-available ChestX-ray14 dataset, outperforming a class activation map (CAM)-based approach, and demonstrating the value of our novel framework for mining disease locations.

Supplementary material

473975_1_En_66_MOESM1_ESM.pdf (22 mb)
Supplementary material 1 (pdf 22566 KB)


  1. 1.
    Abadi, M., Agarwal, A., Barham, P., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015).
  2. 2.
    Geoffrey, H., et al.: Distilling the knowledge in a neural network. In: NIPS (2014)Google Scholar
  3. 3.
    He, K., et al.: Deep residual learning for image recognition. In: IEEE CVPR (2016)Google Scholar
  4. 4.
    Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv:1711.08195 (2017)
  5. 5.
    Li, Z., Wang, C., Han, M., Xue, Y., et al.: Thoracic disease identification and localization with limited supervision. In: IEEE CVPR (2018)Google Scholar
  6. 6.
    Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: IEEE ICCV, pp. 1520–1528 (2015)Google Scholar
  7. 7.
    Pesce, E., Ypsilantis, P.P., et al.: Learning to detect chest radiographs containing lung nodules using visual attention networks. arXiv:1712.00996 (2017)
  8. 8.
    Rajpurkar, P., Irvin, J., Zhu, K., et al.: Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv:1711.05225 (2017)
  9. 9.
    Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: IEEE ICCV, pp. 3420–3429 (2017)Google Scholar
  10. 10.
    Wang, X., Peng, Y., et al.: Tienet: text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: IEEE CVPR (2018)Google Scholar
  11. 11.
    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR (2017)Google Scholar
  12. 12.
    Wei, Y., et al.: Object region mining with adversarial erasing: a simple classification to semantic segmentation approach. In: IEEE CVPR, pp. 6488–6496 (2017)Google Scholar
  13. 13.
    Wu, Y., et al.: Tensorpack (2016).
  14. 14.
    Yao, L., Poblenz, E., Dagunts, D., et al.: Learning to diagnose from scratch by exploiting dependencies among labels. arXiv:1710.10501 (2017)
  15. 15.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)Google Scholar
  16. 16.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: IEEE CVPR, pp. 2921–2929 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jinzheng Cai
    • 1
    Email author
  • Le Lu
    • 2
  • Adam P. Harrison
    • 2
  • Xiaoshuang Shi
    • 1
  • Pingjun Chen
    • 1
  • Lin Yang
    • 1
  1. 1.University of FloridaGainesvilleUSA
  2. 2.AI-InfraNVIDIA CorpBethesdaUSA

Personalised recommendations