Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs

  • Yuxing TangEmail author
  • Xiaosong Wang
  • Adam P. Harrison
  • Le Lu
  • Jing Xiao
  • Ronald M. Summers
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11046)


In this work, we exploit the task of joint classification and weakly supervised localization of thoracic diseases from chest radiographs, with only image-level disease labels coupled with disease severity-level (DSL) information of a subset. A convolutional neural network (CNN) based attention-guided curriculum learning (AGCL) framework is presented, which leverages the severity-level attributes mined from radiology reports. Images in order of difficulty (grouped by different severity-levels) are fed to CNN to boost the learning gradually. In addition, highly confident samples (measured by classification probabilities) and their corresponding class-conditional heatmaps (generated by the CNN) are extracted and further fed into the AGCL framework to guide the learning of more distinctive convolutional features in the next iteration. A two-path network architecture is designed to regress the heatmaps from selected seed samples in addition to the original classification task. The joint learning scheme can improve the classification and localization performance along with more seed samples for the next iteration. We demonstrate the effectiveness of this iterative refinement framework via extensive experimental evaluations on the publicly available ChestXray14 dataset. AGCL achieves over 5.7% (averaged over 14 diseases) increase in classification AUC and 7%/11% increases in Recall/Precision for the localization task compared to the state of the art.



This research was supported by the Intramural Research Program of the National Institutes of Health Clinical Center and by the Ping An Technology Co., Ltd. through a Cooperative Research and Development Agreement. The authors thank NVIDIA for GPU donation.


  1. 1.
    Bengio, Y., Louradour, J., et al.: Curriculum learning. In: ICML (2009)Google Scholar
  2. 2.
    He, K., et al.: Deep residual learning for image recognition. In: IEEE CVPR (2016)Google Scholar
  3. 3.
    Jin, D., Xu, Z., et al.: CT-realistic lung nodule simulation from 3D conditional generative adversarial networks for robust lung segmentation. In: MICCAI (2018)Google Scholar
  4. 4.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  5. 5.
    Lakhani, P., Sundaram, B.: Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284(2), 574–582 (2017)CrossRefGoogle Scholar
  6. 6.
    Li, Z., Wang, C., Han, M., Xue, Y., Wei, W., Li, L.J., Fei-Fei, L.: Thoracic disease identification and localization with limited supervision. In: IEEE CVPR (2018)Google Scholar
  7. 7.
    Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., et al.: CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv:1711.05225 (2017)
  8. 8.
    Shi, M., Ferrari, V.: Weakly supervised object localization using size estimates. In: ECCV (2016)Google Scholar
  9. 9.
    Tang, Y., Wang, J., Gao, B., et al.: Large scale semi-supervised object detection using visual and semantic knowledge transfer. In: IEEE CVPR (2016)Google Scholar
  10. 10.
    Tang, Y., et al.: Semi-automatic RECIST labeling on CT scans with cascaded convolutional neural networks. In: MICCAI (2018)Google Scholar
  11. 11.
    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR (2017)Google Scholar
  12. 12.
    Wang, X., Peng, Y., et al.: TieNet: text-image embedding network for common thorax disease classification and reporting in chest X-rays. In: IEEE CVPR (2018)Google Scholar
  13. 13.
    Yan, K., et al.: Deeplesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. J. Med. Imaging 5(3), 036501 (2018)CrossRefGoogle Scholar
  14. 14.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: IEEE CVPR (2016)Google Scholar

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2018

Authors and Affiliations

  • Yuxing Tang
    • 1
    Email author
  • Xiaosong Wang
    • 1
  • Adam P. Harrison
    • 1
  • Le Lu
    • 1
  • Jing Xiao
    • 2
  • Ronald M. Summers
    • 1
  1. 1.National Institutes of Health, Clinical CenterBethesdaUSA
  2. 2.Ping An Technology Co., Ltd.ShenzhenChina

Personalised recommendations