Skip to main content
SpringerLink
Log in

Co-heterogeneous and Adaptive Segmentation from Multi-source and Multi-phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12368))

Included in the following conference series:

Abstract

Within medical imaging, organ/pathology segmentation models trained on current publicly available and fully-annotated datasets usually do not well-represent the heterogeneous modalities, phases, pathologies, and clinical scenarios encountered in real environments. On the other hand, there are tremendous amounts of unlabelled patient imaging scans stored by many modern clinical centers. In this work, we present a novel segmentation strategy, co-heterogenous and adaptive segmentation (CHASe), which only requires a small labeled cohort of single phase data to adapt to any unlabeled cohort of heterogenous multi-phase data with possibly new clinical scenarios and pathologies. To do this, we develop a versatile framework that fuses appearance-based semi-supervision, mask-based adversarial domain adaptation, and pseudo-labeling. We also introduce co-heterogeneous training, which is a novel integration of co-training and hetero-modality learning. We evaluate CHASe using a clinically comprehensive and challenging dataset of multi-phase computed tomography (CT) imaging studies (1147 patients and 4577 3D volumes), with a test set of 100 patients. Compared to previous state-of-the-art baselines, CHASe can further improve pathological liver mask Dice-Sørensen coefficients by ranges of \(4.2\%\) to \(9.4\%\), depending on the phase combinations, e.g., from \(84.6\%\) to \(94.0\%\) on non-contrast CTs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A caveat is that the public H-DenseUNet model was only trained on the LiTS subset of \(\mathcal {D}_{\ell }\).

References

  1. Bai, W., et al.: Semi-supervised learning for network-based cardiac MR image segmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 253–260. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_29

    Chapter  Google Scholar 

  2. Ben-Cohen, A., Diamant, I., Klang, E., Amitai, M., Greenspan, H.: Fully convolutional network for liver segmentation and lesions detection. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 77–85. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_9

    Chapter  Google Scholar 

  3. Bilic, P., et al.: The liver tumor segmentation benchmark (LiTS). arXiv:1901.04056 (2019). http://arxiv.org/abs/1901.04056, arXiv: 1901.04056

  4. Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the eleventh annual conference on Computational learning theory, pp. 92–100. Citeseer (1998)

    Google Scholar 

  5. Cai, J., et al.: Accurate weakly-supervised deep lesion segmentation using large-scale clinical annotations: slice-propagated 3D mask generation from 2D RECIST. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 396–404. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_46

    Chapter  Google Scholar 

  6. Chang, W.L., Wang, H.P., Peng, W.H., Chiu, W.C.: All about structure: adapting structural information across domains for boosting semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1900–1909 (2019)

    Google Scholar 

  7. Chaos: Chaos - combined (CT-MR) healthy abdominal organ segmentation (2019). https://chaos.grand-challenge.org/Combined_Healthy_Abdominal_Organ_Segmentation

  8. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv:1606.00915 (2016)

  9. Chen, Y.C., Lin, Y.Y., Yang, M.H., Huang, J.B.: CrDoCo: pixel-level domain transfer with cross-domain consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1791–1800 (2019)

    Google Scholar 

  10. Conze, P.H., et al.: Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced CT scans. Int. J. Comput. Assist. Radiol. Surg. 12(2), 223–233 (2017). https://doi.org/10.1007/s11548-016-1493-1

    Article  Google Scholar 

  11. Gibson, E., et al.: Multi-organ abdominal CT reference standard segmentations (2018). https://doi.org/10.5281/zenodo.1169361. This data set was developed as part of independent research supported by Cancer Research UK (Multidisciplinary C28070/A19985) and the National Institute for Health Research UCL/UCL Hospitals Biomedical Research Centre

  12. Gotra, A., et al.: Liver segmentation: indications, techniques and future directions. Insights Imaging 8(4), 377–392 (2017). https://doi.org/10.1007/s13244-017-0558-1

  13. Han, X.: Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv preprint arXiv:1704.07239 (2017)

  14. Harrison, A.P., Xu, Z., George, K., Lu, L., Summers, R.M., Mollura, D.J.: Progressive and multi-path holistically nested neural networks for pathological lung segmentation from CT images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 621–629. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_71

    Chapter  Google Scholar 

  15. Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: HeMIS: hetero-modal image segmentation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 469–477. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_54

    Chapter  Google Scholar 

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2015)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  18. Heimann, T., et al.: Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans. Med. Imaging 28(8), 1251–1265 (2009). https://doi.org/10.1109/TMI.2009.2013851

    Article  Google Scholar 

  19. Heinrich, M.P., Jenkinson, M., Brady, M., Schnabel, J.A.: MRF-based deformable registration and ventilation estimation of lung CT. IEEE Trans. Med. Imaging 32, 1239–1248 (2013)

    Article  Google Scholar 

  20. Isensee, F., et al.: nnU-Net: self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv:1809.10486 (2018)

  21. Jin, D., et al.: Accurate esophageal gross tumor volume segmentation in PET/CT using two-stream chained 3D deep network fusion. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 182–191. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_21

    Chapter  Google Scholar 

  22. Jin, D., et al.: Deep esophageal clinical target volume delineation using encoded 3D spatial context of tumors, lymph nodes, and organs at risk. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 603–612. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_67

    Chapter  Google Scholar 

  23. Kuo, C., Cheng, S., Lin, C., Hsiao, K., Lee, S.: Texture-based treatment prediction by automatic liver tumor segmentation on computed tomography. In: 2017 International Conference on Computer, Information and Telecommunication Systems (CITS), pp. 128–132 (2017). https://doi.org/10.1109/CITS.2017.8035318

  24. Lee, D.H.: Pseudo-label : the simple and efficient semi-supervised learning method for deep neural networks. In: ICML 2013 Workshop : Challenges in Representation Learning (WREPL) (2013)

    Google Scholar 

  25. Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.W., Heng, P.A.: H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Trans. Med. Imaging 37(12), 2663–2674 (2018)

    Article  Google Scholar 

  26. Li, Y., Liu, L., Tan, R.T.: Decoupled certainty-driven consistency loss for semi-supervised learning (2019)

    Google Scholar 

  27. Lin, J.: Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 37(1), 145–151 (1991)

    Article  MathSciNet  Google Scholar 

  28. Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. IEEE (2016)

    Google Scholar 

  29. Min, S., Chen, X., Zha, Z.J., Wu, F., Zhang, Y.: A two-stream mutual attention network for semi-supervised biomedical segmentation with noisy labels (2019)

    Google Scholar 

  30. Oliva, M., Saini, S.: Liver cancer imaging: role of CT, MRI, US and PET. Cancer Imaging Official Publ. Int. Cancer Imaging Soc. 4(Spec No A), S42–6 (2004). https://doi.org/10.1102/1470-7330.2004.0011

  31. Qiao, S., Shen, W., Zhang, Z., Wang, B., Yuille, A.: Deep co-training for semi-supervised image recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 135–152 (2018)

    Google Scholar 

  32. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  33. Roth, H.R., et al.: DeepOrgan: multi-level deep convolutional networks for automated pancreas segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 556–564. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24553-9_68

    Chapter  Google Scholar 

  34. Roth, H.R., et al.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Med. Image Anal. 45, 94–107 (2018)

    Article  Google Scholar 

  35. Roth, H.R., et al.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Med. Image Anal. 45, 94–107 (2018). https://doi.org/10.1016/j.media.2018.01.006, http://www.sciencedirect.com/science/article/pii/S1361841518300215

  36. Roth, K., Konopczyński, T., Hesser, J.: Liver lesion segmentation with slice-wise 2D Tiramisu and Tversky loss function. arXiv preprint arXiv:1905.03639 (2019)

  37. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  38. Soler, L., et al.: 3D image reconstruction for comparison of algorithm database: a patient specific anatomical and medical image database. Technical report, IRCAD, Strasbourg, France (2010)

    Google Scholar 

  39. Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J., Wu, Z., Ding, X.: Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med. Image Anal. 63, 101693 (2019)

    Article  Google Scholar 

  40. Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7472–7481 (2018)

    Google Scholar 

  41. Vorontsov, E., Abi-Jaoudeh, N., Kadoury, S.: Metastatic liver tumor segmentation using texture-based omni-directional deformable surface models. In: Yoshida, H., Nappi, J., Saini, S. (eds.) Abdominal Imaging. Computational and Clinical Applications. ABD-MICCAI 2014. Lecture Notes in Computer Science, vol. 8676, pp. 74–83. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-13692-9_7

  42. Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Advent: adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2517–2526 (2019)

    Google Scholar 

  43. Wang, R., Cao, S., Ma, K., Meng, D., Zheng, Y.: Pairwise semantic segmentation via conjugate fully convolutional network. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 157–165. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_18

    Chapter  Google Scholar 

  44. Xia, Y., et al.: 3D semi-supervised learning with uncertainty-aware multi-view co-training. arXiv preprint arXiv:1811.12506 (2018)

  45. Yang, D., et al.: Automatic liver segmentation using an adversarial image-to-image network. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 507–515. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_58

    Chapter  Google Scholar 

  46. Zhang, J., Xie, Y., Zhang, P., Chen, H., Xia, Y., Shen, C.: Light-weight hybrid convolutional network for liver tumor segmentation. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 4271–4277. International Joint Conferences on Artificial Intelligence Organization (2019). https://doi.org/10.24963/ijcai.2019/593

  47. Zhang, L., Gopalakrishnan, V., Lu, L., Summers, R.M., Moss, J., Yao, J.: Self-learning to detect and segment cysts in lung CT images without manual annotation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1100–1103. IEEE (2018)

    Google Scholar 

  48. Zhang, Q., Fan, Y., Wan, J., Liu, Y.: An efficient and clinical-oriented 3D liver segmentation method. IEEE Access 5, 18737–18744 (2017). https://doi.org/10.1109/ACCESS.2017.2754298

    Article  Google Scholar 

  49. Zhou, Y., et al.: Semi-supervised 3D abdominal multi-organ segmentation via deep multi-planar co-training. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 121–140. IEEE (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. PAII Inc., Bethesda, MD, USA

    Ashwin Raju, Yuankai Huo, Jinzheng Cai, Le Lu & Adam P. Harrison

  2. The University of Texas at Arlington, Arlington, TX, USA

    Ashwin Raju & Junzhou Huang

  3. Chang Gung Memorial Hospital, Linkou, Taiwan, ROC

    Chi-Tung Cheng & ChienHung Liao

  4. PingAn Technology, Shenzhen, China

    Jing Xiao

Authors
  1. Ashwin Raju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chi-Tung Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuankai Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinzheng Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junzhou Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Le Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. ChienHung Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Adam P. Harrison
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam P. Harrison .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4766 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Raju, A. et al. (2020). Co-heterogeneous and Adaptive Segmentation from Multi-source and Multi-phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12368. Springer, Cham. https://doi.org/10.1007/978-3-030-58592-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58592-1_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58591-4

  • Online ISBN: 978-3-030-58592-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation