Skip to main content

Cooperative Training and Latent Space Data Augmentation for Robust Medical Image Segmentation

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (MICCAI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12903))


Deep learning-based segmentation methods are vulnerable to unforeseen data distribution shifts during deployment, e.g. change of image appearances or contrasts caused by different scanners, unexpected imaging artifacts etc. In this paper, we present a cooperative framework for training image segmentation models and a latent space augmentation method for generating hard examples. Both contributions improve model generalization and robustness with limited data. The cooperative training framework consists of a fast-thinking network (FTN) and a slow-thinking network (STN). The FTN learns decoupled image features and shape features for image reconstruction and segmentation tasks. The STN learns shape priors for segmentation correction and refinement. The two networks are trained in a cooperative manner. The latent space augmentation generates challenging examples for training by masking the decoupled latent space in both channel-wise and spatial-wise manners. We performed extensive experiments on public cardiac imaging datasets. Using only 10 subjects from a single site for training, we demonstrated improved cross-site segmentation performance, and increased robustness against various unforeseen imaging artifacts compared to strong baseline methods. Particularly, cooperative training with latent space data augmentation yields 15% improvement in terms of average Dice score when compared to a standard training method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.

  2. 2.

  3. 3.

  4. 4.


  1. Shen, D., et al.: Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017)

    Article  Google Scholar 

  2. Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)

    Article  Google Scholar 

  3. Dou, Q., et al.: Domain generalization via model-agnostic learning of semantic features. In: Wallach, H.M., et al. (eds.) NeurIPS 2019, pp. 6447–6458 (2019)

    Google Scholar 

  4. Albuquerque, I., et al.: Improving out-of-distribution generalization via multi-task self-supervised pretraining. arXiv preprint arXiv:2003.13525 (2020)

  5. Chattopadhyay, P., Balaji, Y., Hoffman, J.: Learning to balance specificity and invariance for in and out of domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 301–318. Springer, Cham (2020).

    Chapter  Google Scholar 

  6. Wang, S., Yu, L., Li, C., Fu, C.-W., Heng, P.-A.: Learning from extrinsic and intrinsic supervisions for domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 159–176. Springer, Cham (2020).

    Chapter  Google Scholar 

  7. Shankar, S., et al.: Generalizing across domains via cross-gradient training. In: ICLR (2018).

  8. Daniel, K.: Thinking, fast and slow (2017)

    Google Scholar 

  9. Shorten, C., et al.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 60 (2019)

    Article  Google Scholar 

  10. Zhang, L., et al.: Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans. Med. Imaging 39(7), 2531–2540 (2020)

    Article  Google Scholar 

  11. Chen, C., et al.: Improving the generalizability of convolutional neural network-based segmentation on CMR images. Front. Cardiovasc. Med. 7, 105 (2020)

    Article  Google Scholar 

  12. Devries, T., et al.: Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.04552 (2017)

    Google Scholar 

  13. Lopes, R.G., et al.: Improving robustness without sacrificing accuracy with patch gaussian augmentation. CoRR, abs/1906.02611 (2019)

    Google Scholar 

  14. Zhou, Z., et al.: Models genesis: generic autodidactic models for 3D medical image analysis. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 384–393. Springer, Cham (2019).

    Chapter  Google Scholar 

  15. Zhou, Z., et al.: Models genesis. Med. Image Anal. 67, 101840 (2021)

    Article  Google Scholar 

  16. Miyato, T., Maeda, S.-I., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. TPAMI 41, 1979–1993 (2018)

    Article  Google Scholar 

  17. Chen, C., et al.: Realistic adversarial data augmentation for MR image segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 667–677. Springer, Cham (2020).

    Chapter  Google Scholar 

  18. Zhang, X., et al.: Deep adversarial data augmentation for extremely low data regimes. IEEE Trans. Circuits Syst. Video Technol. 31(1), 15–28 (2021)

    Article  Google Scholar 

  19. Zhao, L., et al.: Maximum-entropy adversarial data augmentation for improved generalization and robustness. In: NeurIPS (2020)

    Google Scholar 

  20. Zheng, W., Chen, Z., Lu, J., Zhou, J.: Hardness-aware deep metric learning. In: CVPR, pp. 72–81 (2019)

    Google Scholar 

  21. Zhang, Y., et al.: A survey on multi-task learning. arXiv preprint arXiv:1707.08114 (2017)

  22. Larrazabal, A.J., Martinez, C., Ferrante, E.: Anatomical priors for image segmentation via post-processing with denoising autoencoders. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 585–593. Springer, Cham (2019).

    Chapter  Google Scholar 

  23. Painchaud, N., Skandarani, Y., Judge, T., Bernard, O., Lalande, A., Jodoin, P.-M.: Cardiac MRI segmentation with strong anatomical guarantees. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 632–640. Springer, Cham (2019).

    Chapter  Google Scholar 

  24. Tishby, N., et al.: The information bottleneck method. arXiv preprint physics/0004057 (2000)

    Google Scholar 

  25. Huang, Z., Wang, H., Xing, E.P., Huang, D.: Self-challenging improves cross-domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 124–140. Springer, Cham (2020).

    Chapter  Google Scholar 

  26. Tompson, J., et al.: Efficient object localization using convolutional networks. In: CVPR, pp. 648–656. IEEE Computer Society (2015)

    Google Scholar 

  27. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR, June 2017

    Google Scholar 

  28. Goodfellow, I.J., et al.: Explaining and harnessing adversarial examples. In: ICLR (2015)

    Google Scholar 

  29. Bernard, O., et al.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? TMI 0062(11), 2514–2525 (2018)

    Google Scholar 

  30. Campello, V.M., et al.: Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M&Ms challenge. IEEE Trans. Med. Imaging (under review)

    Google Scholar 

  31. Pérez-García, F., et al.: TorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. arXiv:2003.04696 [cs, eess, stat], March 2020

  32. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).

    Chapter  Google Scholar 

  33. Xu, Z., et al.: Robust and generalizable visual representation learning via random convolutions. In: ICLR (2021)

    Google Scholar 

  34. Cubuk, E.D., et al.: Autoaugment: learning augmentation strategies from data. In: CVPR, pp. 113–123 (2019)

    Google Scholar 

  35. Shaw, R., et al.: MRI k-space motion artefact augmentation: model robustness and task-specific uncertainty. In: Jorge Cardoso, M., et al. (eds.) Proceedings of Machine Learning Research, MIDL, London, UK, 08–10 July 2019, vol. 102, pp. 427–436. PMLR (2019)

    Google Scholar 

  36. Christ, P.F., et al.: Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 415–423. Springer, Cham (2016).

    Chapter  Google Scholar 

Download references


This work was supported by the SmartHeart EPSRC Programme Grant (EP/P001009/1).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Chen Chen .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1677 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, C., Hammernik, K., Ouyang, C., Qin, C., Bai, W., Rueckert, D. (2021). Cooperative Training and Latent Space Data Augmentation for Robust Medical Image Segmentation. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12903. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87198-7

  • Online ISBN: 978-3-030-87199-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics