Skip to main content
View expanded cover

International Conference on Medical Image Computing and Computer-Assisted Intervention

MICCAI 2021: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 pp 592–601Cite as

Learning Visual Features by Colorization for Slide-Consistent Survival Prediction from Whole Slide Images

  • 4854 Accesses

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12908)

Abstract

Recent deep learning techniques have shown promising performance on survival prediction from Whole Slide Images (WSIs). These methods are often based on multiple-step frameworks including patch sampling, feature extraction, and feature aggregation. However, feature extraction typically relies on handcrafted features or Convolutional Neural Networks (CNNs) pretrained on ImageNet without fine-tuning, thus leading to suboptimal performance. Besides, to aggregate features, previous studies focus on WSI-level survival prediction but ignore the heterogeneous information that is present in multiple WSIs acquired for the same patient. To address the above challenges, we propose a survival prediction model that exploits heterogeneous features at the patient-level. Specifically, we introduce colorization as the pretext task to train the CNNs which are tailored for extracting features from patches of WSIs. In addition, we develop a patient-level framework integrating multiple WSIs for survival prediction with consistency and ranking losses. Extensive experiments show that our model achieves state-of-the-art performance on two large-scale public datasets.

Keywords

  • Survival prediction
  • Histopathology
  • Colorization
  • Whole Slide Images
  • Deep learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-87237-3_57
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-87237-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    https://portal.gdc.cancer.gov/.

  2. 2.

    We gratefully acknowledge the help and suggestions from the authors of RankSurv.

References

  1. Abbet, C., Zlobec, I., Bozorgtabar, B., Thiran, J.-P.: Divide-and-rule: self-supervised learning for survival analysis in colorectal cancer. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 480–489. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_46

    CrossRef  Google Scholar 

  2. Davidson-Pilon, C.: lifelines: survival analysis in python. J. Open Sour. Softw. 4(40), 1317 (2019)

    CrossRef  Google Scholar 

  3. Di, D., Li, S., Zhang, J., Gao, Y.: Ranking-based survival prediction on histopathological whole-slide images. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 428–438. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_41

    CrossRef  Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

    Google Scholar 

  5. Heagerty, P.J., Zheng, Y.: Survival model predictive accuracy and ROC curves. Biometrics 61(1), 92–105 (2005). https://doi.org/10.1111/j.0006-341X.2005.030814.x

    MathSciNet  CrossRef  MATH  Google Scholar 

  6. Kandoth, C., et al.: Mutational landscape and significance across 12 major cancer types. Nature 502(7471), 333–339 (2013)

    CrossRef  Google Scholar 

  7. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  8. Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representation learning. In: CVPR, pp. 1920–1929 (2019)

    Google Scholar 

  9. Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 577–593. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_35

    CrossRef  Google Scholar 

  10. Larsson, G., Maire, M., Shakhnarovich, G.: Colorization as a proxy task for visual understanding. In: CVPR, pp. 6874–6883 (2017)

    Google Scholar 

  11. Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20

    CrossRef  Google Scholar 

  12. Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. PNAS 115(13), E2970–E2979 (2018). https://doi.org/10.1073/pnas.1717139115

    CrossRef  Google Scholar 

  13. Muhammad, H., et al.: Unsupervised subtyping of cholangiocarcinoma using a deep clustering convolutional autoencoder. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 604–612. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_67

    CrossRef  Google Scholar 

  14. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    MathSciNet  CrossRef  Google Scholar 

  15. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) NeurIPS, pp. 8024–8035. Curran Associates, Inc. (2019)

    Google Scholar 

  16. Pech-Pacheco, J.L., Cristobal, G., Chamorro-Martinez, J., Fernandez-Valdivia, J.: Diatom autofocusing in brightfield microscopy: a comparative study. In: ICPR 2000, vol. 3, pp. 314–317 (2000). https://doi.org/10.1109/ICPR.2000.903548

  17. Shao, W., et al.: Diagnosis-guided multi-modal feature selection for prognosis prediction of lung squamous cell carcinoma. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 113–121. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_13

    CrossRef  Google Scholar 

  18. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2015)

    Google Scholar 

  19. Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One 15(6), e0233678 (2020)

    CrossRef  Google Scholar 

  20. Xiao, L., et al.: Censoring-aware deep ordinal regression for survival prediction from pathological images. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 449–458. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_43

    CrossRef  Google Scholar 

  21. Yang, H., Tianyi Zhou, J., Cai, J., Soon Ong, Y.: MIML-FCN+: multi-instance multi-label learning via fully convolutional networks with privileged information. In: CVPR (2017)

    Google Scholar 

  22. Yao, J., Zhu, X., Huang, J.: Deep multi-instance learning for survival prediction from whole slide images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 496–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_55

    CrossRef  Google Scholar 

  23. Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)

    CrossRef  Google Scholar 

  24. Yao, J., Zhu, X., Zhu, F., Huang, J.: Deep correlational learning for survival prediction from multi-modality data. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 406–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_46

    CrossRef  Google Scholar 

  25. Yuan, Y., et al.: Assessing the clinical utility of cancer genomic and proteomic data across tumor types. Nat. Biotechnol. 32(7), 644–652 (2014)

    CrossRef  Google Scholar 

  26. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40

    CrossRef  Google Scholar 

  27. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR (2016)

    Google Scholar 

  28. Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: BIBM, pp. 544–547. IEEE, December 2016. https://doi.org/10.1109/BIBM.2016.7822579

  29. Zhu, X., Yao, J., Zhu, F., Huang, J.: WSISA: making survival prediction from whole slide histopathological images. In: CVPR (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, University of New South Wales, Sydney, NSW, 2052, Australia

    Lei Fan, Arcot Sowmya, Erik Meijering & Yang Song

Authors
  1. Lei Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arcot Sowmya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erik Meijering
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Song .

Editor information

Editors and Affiliations

  1. Erasmus MC - University Medical Center Rotterdam, Rotterdam, The Netherlands

    Prof. Dr. Marleen de Bruijne

  2. University of Basel, Allschwil, Switzerland

    Prof. Dr. Philippe C. Cattin

  3. Inria Nancy Grand Est, Villers-lès-Nancy, France

    Stéphane Cotin

  4. ICube, Université de Strasbourg, CNRS, Strasbourg, France

    Nicolas Padoy

  5. National Center for Tumor Diseases (NCT/UCC), Dresden, Germany

    Prof. Stefanie Speidel

  6. Tencent Jarvis Lab, Shenzhen, China

    Yefeng Zheng

  7. ICube, Université de Strasbourg, CNRS, Strasbourg, France

    Caroline Essert

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Fan, L., Sowmya, A., Meijering, E., Song, Y. (2021). Learning Visual Features by Colorization for Slide-Consistent Survival Prediction from Whole Slide Images. In: , et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12908. Springer, Cham. https://doi.org/10.1007/978-3-030-87237-3_57

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87237-3_57

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87236-6

  • Online ISBN: 978-3-030-87237-3

  • eBook Packages: Computer ScienceComputer Science (R0)