InfoMask: Masked Variational Latent Representation to Localize Chest Disease

  • Saeid Asgari TaghanakiEmail author
  • Mohammad Havaei
  • Tess Berthier
  • Francis Dutil
  • Lisa Di Jorio
  • Ghassan Hamarneh
  • Yoshua Bengio
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11769)


The scarcity of richly annotated medical images is limiting supervised deep learning based solutions to medical image analysis tasks, such as localizing discriminatory radiomic disease signatures. Therefore, it is desirable to leverage unsupervised and weakly supervised models. Most recent weakly supervised localization methods apply attention maps or region proposals in a multiple instance learning formulation. While attention maps can be noisy, leading to erroneously highlighted regions, it is not simple to decide on an optimal window/bag size for multiple instance learning approaches. In this paper, we propose a learned spatial masking mechanism to filter out irrelevant background signals from attention maps. The proposed method minimizes mutual information between a masked variational representation and the input while maximizing the information between the masked representation and class labels. This results in more accurate localization of discriminatory regions. We tested the proposed model on the ChestX-ray8 dataset to localize pneumonia from chest X-ray images without using any pixel-level or bounding-box annotations.


Disease localization Variational representation Mutual information 



We thank Joseph Paul Cohen for his insightful discussions and comments.


  1. 1.
    Centers for disease control and prevention. Accessed 25 Mar 2019
  2. 2.
    Alemi, A.A., Fischer, I., Dillon, J.V., Murphy, K.: Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 (2016)
  3. 3.
    Ancona, M., Ceolini, E., Öztireli, C., Gross, M.: Towards better understanding of gradient-based attribution methods for deep neural networks. In: ICLR 2018 (2018)Google Scholar
  4. 4.
    Bency, A.J., Kwon, H., Lee, H., Karthikeyan, S., Manjunath, B.S.: Weakly supervised localization using deep feature maps. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 714–731. Springer, Cham (2016). Scholar
  5. 5.
    Bilen, H., Vedaldi, A.: Weakly supervised deep detection networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2846–2854 (2016)Google Scholar
  6. 6.
    Fan, L.: Adversarial localization network. In: NIPS 2017 Workshop on Learning with Limited Labeled Data (2017)Google Scholar
  7. 7.
    Kumar, M.P., Packer, B., Koller, D.: Self-paced learning for latent variable models. In: Advances in Neural Information Processing Systems, pp. 1189–1197 (2010)Google Scholar
  8. 8.
    Rajpurkar, P., et al.: CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
  9. 9.
    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. CoRR abs/1610.02391 (2016)Google Scholar
  10. 10.
    Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 3145–3153. JMLR. org (2017)Google Scholar
  11. 11.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
  12. 12.
    Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M.: SmoothGrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017)
  13. 13.
    Song, H.O., Girshick, R., Jegelka, S., Mairal, J., Harchaoui, Z., Darrell, T.: On learning to localize objects with minimal supervision. arXiv preprint arXiv:1403.1024 (2014)
  14. 14.
    Song, H.O., Lee, Y.J., Jegelka, S., Darrell, T.: Weakly-supervised discovery of visual pattern configurations. In: Advances in Neural Information Processing Systems, pp. 1637–1645 (2014)Google Scholar
  15. 15.
    Teh, E.W., Rochan, M., Wang, Y.: Attention networks for weakly supervised object localization. In: BMVC, pp. 1–11 (2016)Google Scholar
  16. 16.
    Tishby, N., Pereira, F.C., Bialek, W.: The information bottleneck method. In: The 37th Annual Allerton Conference on Communications, Control, and Computing, pp. 368–377 (1999)Google Scholar
  17. 17.
    Wang, C., Ren, W., Huang, K., Tan, T.: Weakly supervised object localization with latent category learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 431–445. Springer, Cham (2014). Scholar
  18. 18.
    Wang, X., et al.: Weakly supervised learning for whole slide lung cancer image classification. Med. Imaging Deep Learn. (2018)Google Scholar
  19. 19.
    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Vision and Pattern Recognition, pp. 2097–2106 (2017)Google Scholar
  20. 20.
    Wei, Y., et al.: TS2C: tight box mining with surrounding segmentation context for weakly supervised object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 434–450 (2018)Google Scholar
  21. 21.
    WHO: Standardization of interpretation of chest radiographs for the diagnosis of pneumonia in childrenGoogle Scholar
  22. 22.
    Yan, C., Yao, J., Li, R., Xu, Z., Huang, J.: Weakly supervised deep learning for thoracic disease classification and localization on chest X-rays. In: Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 103–110. ACM (2018)Google Scholar
  23. 23.
    Zolna, K., Geras, K.J., Cho, K.: Classifier-agnostic saliency map extraction. arXiv preprint arXiv:1805.08249 (2018)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Saeid Asgari Taghanaki
    • 1
    • 2
    • 3
    Email author
  • Mohammad Havaei
    • 2
  • Tess Berthier
    • 2
  • Francis Dutil
    • 2
  • Lisa Di Jorio
    • 2
  • Ghassan Hamarneh
    • 3
  • Yoshua Bengio
    • 1
  1. 1.MILA, Université de MontréalMontrealCanada
  2. 2.Imagia Inc.MontrealCanada
  3. 3.School of Computing ScienceSimon Fraser UniversityBurnabyCanada

Personalised recommendations