Skip to main content
SpringerLink
Log in

Thoracic Disease Identification and Localization with Limited Supervision

  • Chapter
  • First Online:
Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning. Building a highly accurate prediction model for these tasks usually requires a large number of images manually annotated with labels and finding sites of abnormalities. In reality, however, such annotated data are expensive to acquire, especially the ones with location annotations. We need methods that can work well with only a small amount of location annotations. To address this challenge, we present a unified approach that simultaneously performs disease identification and localization through the same underlying model for all images. We demonstrate that our approach can effectively leverage both class information as well as limited location annotation, and significantly outperforms the comparative reference baseline in both classification and localization tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    While abnormalities, findings, clinical conditions, and diseases have distinct meanings in the medical domain, here, we simply refer to them as diseases and disease labels for the focused discussion in computer vision.

  2. 2.

    The method proposed in [30] did not use the bounding box information for localization training.

  3. 3.

    Later on, we notice a similar definition [19] for this multi-instance problem. We argue that our formulation is in a different context of solving classification and localization in a unified way for images with limited bounding box annotation. Yet, this related work can be viewed as a successful validation of our multi-instance learning based formulation.

  4. 4.

    Here ROC is the Receiver Operating Characteristic, which measures the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings (200 thresholds in this chapter).

  5. 5.

    Using ResNet-v2 [14] shows marginal performance difference for our network compared to ResNet-v1 [13] used in the reference baseline.

  6. 6.

    Note that we treat discrete detected regions as one prediction region, thus IoR is analogous to intersection over the detected bounding box area ratio (IoBB).

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. Software available from https://www.tensorflow.org/

  2. Akselrod-Ballin A, Karlinsky L, Alpert S, Hasoul S, Ben-Ari R, Barkan E (2016) A region based convolutional network for tumor detection and classification in breast mammography. In: International workshop on large-scale annotation of biomedical data and expert label synthesis. Springer, Berlin, pp 197–205

    Google Scholar 

  3. Babenko B. Multiple instance learning: algorithms and applications

    Google Scholar 

  4. Chen X, Xu Y, Wong DWK, Wong TY, Liu J (2015) Glaucoma detection based on deep convolutional neural network. In: 2015 37th Annual International Conference of the IEEE Engineering in medicine and biology society (EMBC). IEEE, pp 715–718

    Google Scholar 

  5. IEEE Standards Committee et al. 754-2008 ieee standard for floating-point arithmetic. IEEE Computer Society Std, 2008, 2008

    Google Scholar 

  6. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. CVPR 2009. IEEE, pp 248–255

    Google Scholar 

  7. Fawcett T (2006) An introduction to roc analysis. Pattern Recognit Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  8. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448

    Google Scholar 

  9. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

    Google Scholar 

  10. Glorot X, Bordes A, Bengio Y (20111) Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp 315–323

    Google Scholar 

  11. Gylys BA, Wedding ME (2017) Medical terminology systems: a body systems approach. FA Davis

    Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. In: European conference on computer vision. Springer, Berlin, pp 346–361

    Chapter  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

    Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision. Springer, Berlin, pp 630–645

    Chapter  Google Scholar 

  15. Hou L, Samaras D, Kurc TM, Gao Y, Davis JE, Saltz JH (2016) Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2424–2433

    Google Scholar 

  16. Hwang S, Kim H-E (2016) Self-transfer learning for fully weakly supervised object localization. arXiv:1602.01625

  17. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, p 448–456

    Google Scholar 

  18. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  19. Liao F, Liang M, Li Z, Hu X, Song S (2017) Evaluate the malignancy of pulmonary nodules using the 3d deep leaky noisy-or network. arXiv:1711.08324

  20. Liu C, Mao J, Sha F, Yuille AL (2017) Attention correctness in neural image captioning. In: AAAI, pp 4176–4182

    Google Scholar 

  21. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Ssd: single shot multibox detector. In: European conference on computer vision. Springer, Berlin, pp 21–37

    Chapter  Google Scholar 

  22. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

    Google Scholar 

  23. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

    Google Scholar 

  24. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

    Google Scholar 

  25. Russakovsky O, Deng J, Hao S, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Via 115(3):211–252

    Article  MathSciNet  Google Scholar 

  26. Shi J, Zheng X, Li Y, Zhang Q, Ying S (2017) Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of alzheimer’s disease. IEEE J Biomed Health Inform

    Google Scholar 

  27. Shin H-C, Roberts K, Lu L, Demner-Fushman D, Yao J, Summers RM (2016) Learning to read chest x-rays: recurrent neural cascade model for automated image annotation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2497–2506

    Google Scholar 

  28. Szegedy C, Liu W, Jia, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

    Google Scholar 

  29. Wang J, Ding H, Azamian F, Zhou B, Iribarren C, Molloi S, Baldi P (2017) Detecting cardiovascular disease from mammograms with deep learning. IEEE Trans Med Imaging

    Google Scholar 

  30. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM (2017) Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 3462–3471

    Google Scholar 

  31. Wu J, Yu Y, Huang C, Yu K (2015) Deep multiple instance learning for image classification and auto-annotation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3460–3469

    Google Scholar 

  32. Yan Z, Zhan Y, Peng Z, Liao S, Shinagawa Y, Zhang S, Metaxas DN, Zhou XS (2016) Multi-instance deep learning: discover discriminative local anatomies for bodypart recognition. IEEE Trans Med Imaging 35(5):1332–1343

    Article  Google Scholar 

  33. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, Berlin, pp 818–833

    Chapter  Google Scholar 

  34. Zhang Z, Chen P, Sapkota M, Yang L (2017) Tandemnet: distilling knowledge from medical images using diagnostic reports as optional semantic references. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 320–328

    Chapter  Google Scholar 

  35. Zhang Z, Xie Y, Xing F, McGough M, Yang L (2017) Mdnet: a semantically and visually interpretable medical image diagnosis network. arXiv:1707.02485

  36. Zhao L, Jia K (2016) Multiscale cnns for brain tumor segmentation and diagnosis. Comput Math Methods Med 2016

    Google Scholar 

  37. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

    Google Scholar 

  38. Zhu W, Lou Q, Vang YS, Xie X (2017) Deep multi-instance networks with sparse label assignment for whole mammogram classification. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 603–611

    Chapter  Google Scholar 

  39. Zilly J, Buhmann JM, Mahapatra D (2017) Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation. Comput Med Imaging Graph 55:28–41

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, Syracuse University, 900 South Crouse Ave., Syracuse, NY, 13210, USA

    Zhe Li

  2. PAII Inc., Palo Alto Research Lab, 3000 El Camino Real, 5 Palo Alto Square, Ste 150, Palo Alto, CA, 94306, USA

    Mei Han

  3. Google, 1600 Amphitheatre Parkway, Mountain View, CA, 94043, USA

    Chong Wang, Yuan Xue, Wei Wei, Li-Jia Li & Li Fei-Fei

Authors
  1. Zhe Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuan Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Li-Jia Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Li Fei-Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhe Li .

Editor information

Editors and Affiliations

  1. Bethesda Research Lab, PAII Inc., Bethesda, MD, USA

    Le Lu

  2. Nvidia Corporation, Bethesda, MD, USA

    Xiaosong Wang

  3. School of Computer Science, University of Adelaide, Adelaide, SA, Australia

    Gustavo Carneiro

  4. Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA

    Lin Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Li, Z. et al. (2019). Thoracic Disease Identification and Localization with Limited Supervision. In: Lu, L., Wang, X., Carneiro, G., Yang, L. (eds) Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-13969-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-13969-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-13968-1

  • Online ISBN: 978-3-030-13969-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation