Skip to main content
SpringerLink
Log in

Joint image and feature adaptative attention-aware networks for cross-modality semantic segmentation

  • S.I. : Deep Geospatial Data Understanding
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Deep learning-based methods are widely used for the task of semantic segmentation in recent years. However, due to the difficulty and labor cost of collecting pixel-level annotations, it is hard to acquire sufficient training images for a certain imaging modality, which greatly hinders the performance of these methods. The intuitive solution to this issue is to train a pre-trained model on label-rich imaging modality (source domain) and then apply the pre-trained model to the label-poor imaging modality (target domain). Unsurprisingly, since the severe domain shift between different modalities, the pre-trained model would perform poorly on the target imaging modality. To this end, we propose a novel unsupervised domain adaptation framework, called Joint Image and Feature Adaptive Attention-aware Networks (JIFAAN), to alleviate the domain shift for cross-modality semantic segmentation. The proposed framework mainly consists of two procedures. The first procedure is image adaptation, which transforms the source domain images into target-like images using the adversarial learning with cycle-consistency constraint. For further bridging the gap between transformed images and target domain images, the second procedure employs feature adaptation to extract the domain-invariant features and thus aligns the distribution in feature space. In particular, we introduce an attention module in the feature adaptation to focus on noteworthy regions and generate attention-aware results. Lastly, we combine two procedures in an end-to-end manner. Experiments on two cross-modality semantic segmentation datasets demonstrate the effectiveness of our proposed framework. Specifically, JIFAAN surpasses the cutting-edge domain adaptation methods and achieves the state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Zhang, Qiming, et al (2019) "Category anchor-guided unsupervised domain adaptation for semantic segmentation." Advances in Neural Information Processing Systems. 433--443.

  2. Gros, Charley, et al (2019) "Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks." Neuroimage 184; 901–915.

  3. Lee, Yongbum, et al (2001) "Automated detection of pulmonary nodules in helical CT images based on an improved template-matching technique." IEEE Transactions on medical imaging 20.7; 595–604.

  4. Coupé, Pierrick, et al (2011) "Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation." NeuroImage 54.2; 940–954.

  5. Litjens, Geert, et al (2017) "A survey on deep learning in medical image analysis." Medical image analysis 42; 60–88.

  6. Long, Jonathan (2015) Evan Shelhamer and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition.

  7. Çiçek, Özgün, et al (2016) "3D U-Net: learning dense volumetric segmentation from sparse annotation." International conference on medical image computing and computer-assisted intervention. Springer, Cham.

  8. Ronneberger, Olaf, Philipp Fischer and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham. (2015).

  9. Pham DL, Chenyang Xu, Prince JL (2000) Current methods in medical image segmentation. Annu Rev Biomed Eng 2(1):315–337

    Article  Google Scholar 

  10. Kaus, Michael R., et al (2001) "Automated segmentation of MR images of brain tumors." Radiology 218.2; 586–591.

  11. Chen, Xu, et al (2020) "Anatomy-Regularized Representation Learning for Cross-Modality Medical Image Segmentation." IEEE Transactions on Medical Imaging 40.1; 274–285.

  12. Zhang, Zizhao, Lin Yang and Yefeng Zheng (2018) "Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition.

  13. Huo, Yuankai, et al (2018) "Synseg-net: Synthetic segmentation without target modality ground truth." IEEE transactions on medical imaging 38.4; 1016–1025.

  14. Chen, Cheng, et al (2019) "Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation." Proceedings of the AAAI Conference on Artificial Intelligence. 33.

  15. Xu, Yonghao, et al (2019) "Self-ensembling attention networks: Addressing domain shift for semantic segmentation." Proceedings of the AAAI Conference on Artificial Intelligence. 33.

  16. Zhu, Jun-Yan, et al (2017) "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision.

  17. Chen, Cheng, et al (2018) "Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation." International workshop on machine learning in medical imaging. Springer, Cham.

  18. Bousmalis, Konstantinos, et al (2017) "Unsupervised pixel-level domain adaptation with generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition.

  19. Hoffman, Judy, et al (2018) "Cycada: Cycle-consistent adversarial domain adaptation." International conference on machine learning. PMLR.

  20. Tsai, Yi-Hsuan, et al (2018) "Learning to adapt structured output space for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  21. Dou, Qi, et al (2016) "Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss." IJCAI. 2018. Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." The Journal of Machine Learning Research 17.1; 2096–2030.

  22. Menze, Bjoern H., et al (2014) "The multimodal brain tumor image segmentation benchmark (BRATS)." IEEE transactions on medical imaging 34.10; 1993–2024.

  23. Zhuang X, Shen J (2016) Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal 31:77–87

    Article  Google Scholar 

  24. Zhang, Wenlu, et al (2015) "Deep convolutional neural networks for multi-modality isointense infant brain image segmentation." NeuroImage 108; 214–224.

  25. Bar, Yaniv, et al (2015) "Deep learning with non-medical training used for chest pathology identification." Medical Imaging 2015: Computer-Aided Diagnosis. Vol. 9414. International Society for Optics and Photonics.

  26. Bergamo A, Torresani L, Fitzgibbon A (2011) Picodes: Learning a compact code for novel-category recognition. Adv Neural Inf Process Syst 24:2088–2096

    Google Scholar 

  27. Prasoon, Adhish, et al (2013) "Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network." International conference on medical image computing and computer-assisted intervention. Springer, Berlin, Heidelberg.

  28. Roth, Holger R., et al (2014) "A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations." International conference on medical image computing and computer-assisted intervention. Springer, Cham.

  29. Dou, Qi, et al (2017) "3D deeply supervised network for automated segmentation of volumetric medical images." Medical image analysis 41; 40–54.

  30. Kamnitsas, Konstantinos, et al (2015) "Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI." Ischemic stroke lesion segmentation 13; 46.

  31. Perone, Christian S., et al (2019) "Unsupervised domain adaptation for medical imaging segmentation with self-ensembling." NeuroImage 194; 1–11.

  32. Gordienko, Yu, et al (2018) "Deep learning with lung segmentation and bone shadow exclusion techniques for chest x-ray analysis of lung cancer." International Conference on Computer Science, Engineering and Education Applications. Springer, Cham.

  33. Zeng, Guodong, et al (2017) "3D U-net with multi-level deep supervision: fully automatic segmentation of proximal femur in 3D MR images." International workshop on machine learning in medical imaging. Springer, Cham.

  34. Richter, Stephan R., et al (2016) "Playing for data: Ground truth from computer games." European conference on computer vision. Springer, Cham.

  35. Ros, German, et al (2016) "The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes." Proceedings of the IEEE conference on computer vision and pattern recognition.

  36. Cordts, Marius, et al (2016) "The cityscapes dataset for semantic urban scene understanding." Proceedings of the IEEE conference on computer vision and pattern recognition.

  37. Zhao, Can, et al (2017) "Whole brain segmentation and labeling from CT using synthetic MR images." International Workshop on Machine Learning in Medical Imaging. Springer, Cham.

  38. Ghifary, Muhammad, et al (2016) "Deep reconstruction-classification networks for unsupervised domain adaptation." European Conference on Computer Vision. Springer, Cham.

  39. Sankaranarayanan, Swami, et al (2018) "Learning from synthetic data: Addressing domain shift for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  40. Zhang, Yiheng, et al (2018) "Fully convolutional adaptation networks for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  41. Chen, et al (2016) "Attention to scale: Scale-aware semantic image segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition.

  42. Wang, et al (2017) "Residual attention network for image classification." Proceedings of the IEEE conference on computer vision and pattern recognition.

  43. Ma, Benteng, et al (2020) "Auto Learning Attention." Advances in Neural Information Processing Systems 33.

  44. Chen, Cheng, et al (2020) "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." IEEE transactions on medical imaging 39.7; 2494–2505.

  45. Isola, Phillip, et al (2017) "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition.

  46. Yu, Fisher (2017) Vladlen Koltun and Thomas Funkhouser. "Dilated residual networks." Proceedings of the IEEE conference on computer vision and pattern recognition.

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under Grant 61822113 and Grant 62076186, in part by Science and Technology Major Project of Hubei Province (Next-Generation AI Technologies) under Grant 2019AEA170 and in part by the Wuhan Chang'e Information Technology Co., Ltd. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.

Author information

Authors and Affiliations

  1. Department of Gastroenterology, Renmin Hospital, Wuhan University, Wuhan, China

    Qihuang Zhong, Fanzhou Zeng & Fei Liao

  2. School of Printing and Packaging, Wuhan University, Wuhan, China

    Qihuang Zhong & Juhua Liu

  3. National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Wuhan University, Wuhan, China

    Qihuang Zhong, Juhua Liu & Bo Du

  4. School of Computer Science and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan, China

    Bo Du

  5. Thinvent Technology Co. LTD, Nanchang, China

    Jedi S. Shang

Authors
  1. Qihuang Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fanzhou Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juhua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jedi S. Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Fei Liao or Juhua Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, Q., Zeng, F., Liao, F. et al. Joint image and feature adaptative attention-aware networks for cross-modality semantic segmentation. Neural Comput & Applic 35, 3665–3676 (2023). https://doi.org/10.1007/s00521-021-06064-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06064-w

Keywords

Search

Navigation