Skip to main content
SpringerLink
Log in

Differentiable Zooming for Multiple Instance Learning on Whole-Slide Images

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13681))

Included in the following conference series:

Abstract

Multiple Instance Learning (MIL) methods have become increasingly popular for classifying gigapixel-sized Whole-Slide Images (WSIs) in digital pathology. Most MIL methods operate at a single WSI magnification, by processing all the tissue patches. Such a formulation induces high computational requirements and constrains the contextualization of the WSI-level representation to a single scale. Certain MIL methods extend to multiple scales, but they are computationally more demanding. In this paper, inspired by the pathological diagnostic process, we propose ZoomMIL, a method that learns to perform multi-level zooming in an end-to-end manner. ZoomMIL builds WSI representations by aggregating tissue-context information from multiple magnifications. The proposed method outperforms the state-of-the-art MIL methods in WSI classification on two large datasets, while significantly reducing computational demands with regard to Floating-Point Operations (FLOPs) and processing time by 40–50\(\times \). Our code is available at: https://github.com/histocartography/zoommil.

K. Thandiackal and B. Chen—Contributed equally.

G. Jaume—Work done while at IBM Research Europe.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    www.research.ibm.com/haifa/Workshops/BRIGHT.

References

  1. Adnan, M., Kalra, S., Tizhoosh, H.: Representation learning of histopathology images using graph neural networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 988–989 (2020)

    Google Scholar 

  2. Anklin, V., et al.: Learning whole-slide segmentation from inexact and incomplete labels using tissue graphs. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 636–646. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_59

    Chapter  Google Scholar 

  3. Aygüneş, B., Aksoy, S., Cinbiş, R., Kösemehmetoğlu, K., Önder, S., Üner, A.: Graph convolutional networks for region of interest classification in breast histopathology. In: SPIE Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 134–141. SPIE (2020)

    Google Scholar 

  4. Bejnordi, B., Litjens, G., Hermsen, M., Karssemeijer, N., van der Laak, J.: A multi-scale superpixel classification approach to the detection of regions of interest in whole slide histopathology images. In: SPIE Medical Imaging 2015: Digital Pathology, vol. 9420, pp. 99–104. SPIE (2015)

    Google Scholar 

  5. Bejnordi, B., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017)

    Article  Google Scholar 

  6. Bejnordi, B., et al.: Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J. Med. Imaging 4(4), 044504 (2017)

    Article  Google Scholar 

  7. BenTaieb, A., Hamarneh, G.: Predicting cancer with a recurrent visual attention model for histopathology images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 129–137. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_15

    Chapter  Google Scholar 

  8. Berthet, Q., Blondel, M., Teboul, O., Cuturi, M., Vert, J., Bach, F.: Learning with differentiable pertubed optimizers. Adv. Neural. Inf. Process. Syst. 34, 9508–9519 (2020)

    Google Scholar 

  9. Brancati, N., et al.: BRACS: a dataset for BReAst carcinoma subtyping in H &E histology images. arXiv:2111.04740 (2021)

  10. Campanella, G., et al.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019)

    Article  Google Scholar 

  11. Cordonnier, J., Mahendran, A., Dosovitskiy, A.: Differentiable patch selection for image recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2351–2360 (2021)

    Google Scholar 

  12. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)

    Google Scholar 

  13. Dong, N., Kampffmeyer, M., Liang, X., Wang, Z., Dai, W., Xing, E.: Reinforced auto-zoom net: towards accurate and fast breast cancer segmentation in whole-slide images. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) Workshop, pp. 317–325 (2018)

    Google Scholar 

  14. Elmore, J., et al.: Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313, 1122–1132 (2015)

    Article  Google Scholar 

  15. Gao, Y., et al.: Multi-scale learning based segmentation of glands in digital colorectal pathology images. In: SPIE Medical Imaging 2016: Digital Pathology, vol. 9791, pp. 175–180. SPIE (2016)

    Google Scholar 

  16. Gomes, D., Porto, S., Balabram, D., Gobbi, H.: Inter-observer variability between general pathologists and a specialist in breast pathology in the diagnosis of lobular neoplasia, columnar cell lesions, atypical ductal hyperplasia and ductal carcinoma in situ of the breast. Diagn. Pathol. 9, 1–9 (2014)

    Article  Google Scholar 

  17. Hashimoto, N., et al.: Multi-scale domain-adversarial multiple-instance CNN for cancer subtype classification with unannotated histopathological images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3852–3861 (2020)

    Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  19. Ho, D., et al.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Comput. Med. Imaging Graph. 88, 101866 (2021)

    Article  Google Scholar 

  20. Isle, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning (ICML), vol. 35, pp. 2127–2136 (2018)

    Google Scholar 

  21. Jaume, G., Pati, P., Anklin, V., Foncubierta-Rodríguez, A., Gabrani, M.: Histocartography: a toolkit for graph analytics in digital pathology. In: MICCAI Workshop on Computational Pathology, pp. 117–128. PMLR (2021)

    Google Scholar 

  22. Jia, Z., Huang, X., Eric, I., Chang, C., Xu, Y.: Constrained deep weak supervision for histopathology image segmentation. IEEE Trans. Med. Imaging 36, 2376–2388 (2017)

    Article  Google Scholar 

  23. Katharopoulos, A., Fleuret, F.: Processing megapixel images with deep attention-sampling models. In: International Conference on Machine Learning (ICML), vol. 36, pp. 3282–3291. PMLR (2019)

    Google Scholar 

  24. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) arXiv preprint arXiv:1412.6980 (2014)

  25. Kong, S., Henao, R.: Efficient classification of very large images with tiny objects. arXiv:2106.02694 (2021)

  26. Lerousseau, M., Vakalopoulou, M., Deutsch, E., Paragios, N.: SparseConvMIL: sparse convolutional context-aware multiple instance learning for whole slide image classification. In: MICCAI Workshop on Computational Pathology, pp. 129–139. PMLR (2021)

    Google Scholar 

  27. Li, B., Li, Y., Eliceiri, K.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14318–14328 (2021)

    Google Scholar 

  28. Li, J., et al.: A multi-resolution model for histopathology image classification and localization with multiple instance learning. Comput. Biol. Med. 131, 104253 (2021)

    Article  Google Scholar 

  29. Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20

    Chapter  Google Scholar 

  30. Liang, Q., et al.: Weakly supervised biomedical image segmentation by reiterative learning. IEEE J. Biomed. Health Inform. 23, 1205–1214 (2018)

    Article  Google Scholar 

  31. Lu, M., et al.: AI-based pathology predicts origins for cancers of unknown primary. Nature 594, 106–110 (2021)

    Article  Google Scholar 

  32. Lu, M., Williamson, D., Chen, T., Chen, R., Barbieri, M., Mahmood, F.: Data efficient and weakly supervised computational pathology on whole slide images. Nat. Biomed. Eng. 5, 555–570 (2021)

    Article  Google Scholar 

  33. Myronenko, A., Xu, Z., Yang, D., Roth, H.R., Xu, D.: Accounting for dependencies in deep learning based multiple instance learning for whole slide imaging. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 329–338. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_32

    Chapter  Google Scholar 

  34. Oliveira, S., et al.: CAD systems for colorectal cancer from WSI are still not ready for clinical acceptance. Sci. Rep. 11(1), 1–15 (2021)

    Article  Google Scholar 

  35. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 8024–8035 (2019)

    Google Scholar 

  36. Pati, P., et al.: Hierarchical graph representations in digital pathology. Med. Image Anal. 75, 102264 (2021)

    Article  Google Scholar 

  37. Qaiser, T., Rajpoot, N.: Learning where to see: a novel attention model for automated immunohistochemical scoring. IEEE Trans. Med. Imaging 38, 2620–2631 (2019)

    Article  Google Scholar 

  38. Raju, A., Yao, J., Haq, M.M.H., Jonnagaddala, J., Huang, J.: Graph attention multi-instance learning for accurate colorectal cancer staging. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 529–539. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_51

    Chapter  Google Scholar 

  39. Shaban, M., et al.: Context-aware convolutional neural network for grading of colorectal cancer histology images. IEEE Trans. Med. Imaging 39, 2395–2405 (2020)

    Article  Google Scholar 

  40. Shao, Z., et al.: TransMIL: transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural. Inf. Process. Syst. 35, 2136–2147 (2021)

    Google Scholar 

  41. Sirinukunwattana, K., Alham, N.K., Verrill, C., Rittscher, J.: Improving whole slide segmentation through visual context - a systematic study. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 192–200. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_22

    Chapter  Google Scholar 

  42. Tellez, D., Litjens, G., van der Laak, J., Ciompi, F.: Neural image compression for gigapixel histopathology image analysis. IEEE Trans. Pattern Anal. Mach. Intell. 43, 567–578 (2019)

    Article  Google Scholar 

  43. Tokunaga, H., Teramoto, Y., Yoshizawa, A., Bise, R.: Adaptive weighting multi-field-of-view CNN for semantic segmentation in pathology. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12597–12606 (2019)

    Google Scholar 

  44. Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)

    Article  Google Scholar 

  45. Zhao, Y., et al.: Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4837–4846 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Research Europe, Zurich, Switzerland

    Kevin Thandiackal, Boqi Chen, Pushpak Pati & Maria Gabrani

  2. ETH Zurich, Zurich, Switzerland

    Kevin Thandiackal, Boqi Chen & Orcun Goksel

  3. Brigham and Women’s Hospital, Harvard Medical School, Boston, USA

    Guillaume Jaume & Drew F. K. Williamson

  4. Uppsala University, Uppsala, Sweden

    Orcun Goksel

Authors
  1. Kevin Thandiackal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Boqi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pushpak Pati
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guillaume Jaume
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Drew F. K. Williamson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Maria Gabrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Orcun Goksel
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K. Thandiackal and B. Chen—Contributed equally.

Corresponding author

Correspondence to Kevin Thandiackal .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 163198 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Thandiackal, K. et al. (2022). Differentiable Zooming for Multiple Instance Learning on Whole-Slide Images. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13681. Springer, Cham. https://doi.org/10.1007/978-3-031-19803-8_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19803-8_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19802-1

  • Online ISBN: 978-3-031-19803-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation