Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences

  • Chen QinEmail author
  • Wenjia Bai
  • Jo Schlemper
  • Steffen E. Petersen
  • Stefan K. Piechnik
  • Stefan Neubauer
  • Daniel Rueckert
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11071)


Cardiac motion estimation and segmentation play important roles in quantitatively assessing cardiac function and diagnosing cardiovascular diseases. In this paper, we propose a novel deep learning method for joint estimation of motion and segmentation from cardiac MR image sequences. The proposed network consists of two branches: a cardiac motion estimation branch which is built on a novel unsupervised Siamese style recurrent spatial transformer network, and a cardiac segmentation branch that is based on a fully convolutional network. In particular, a joint multi-scale feature encoder is learned by optimizing the segmentation branch and the motion estimation branch simultaneously. This enables the weakly-supervised segmentation by taking advantage of features that are unsupervisedly learned in the motion estimation branch from a large amount of unannotated data. Experimental results using cardiac MlRI images from 220 subjects show that the joint learning of both tasks is complementary and the proposed models outperform the competing methods significantly in terms of accuracy and speed.

Supplementary material

473975_1_En_53_MOESM1_ESM.pdf (62 kb)
Supplementary material 1 (pdf 62 KB)


  1. 1.
    Agrawal, P., Carreira, J., Malik, J.: Learning to see by moving. In: ICCV, pp. 37–45 (2015)Google Scholar
  2. 2.
    Avendi, M., Kheradvar, A., Jafarkhani, H.: A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Med. Image Anal. 30, 108–119 (2016)CrossRefGoogle Scholar
  3. 3.
    Bai, W., et al.: Semi-supervised learning for network-based cardiac MR image segmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 253–260. Springer, Cham (2017). Scholar
  4. 4.
    Bai, W., Sinclair, M., Tarroni, G., et al.: Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J. Cardiovasc. Magn. Reson. (2018)Google Scholar
  5. 5.
    Caballero, J., Ledig, C., Aitken, A., et al.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: CVPR (2017)Google Scholar
  6. 6.
    Cheng, J., Tsai, Y.H., Wang, S., Yang, M.H.: SegFlow: joint learning for video object segmentation and optical flow. In: ICCV, pp. 686–695 (2017)Google Scholar
  7. 7.
    De Craene, M., Piella, G., Camara, O., et al.: Temporal diffeomorphic free-form deformation: application to motion and strain estimation from 3D echocardiography. Med. Image Anal. 16(2), 427–450 (2012)CrossRefGoogle Scholar
  8. 8.
    Doersch, C., Zisserman, A.: Multi-task self-supervised visual learning. In: ICCV (2017)Google Scholar
  9. 9.
    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: NIPS, pp. 2017–2025 (2015)Google Scholar
  10. 10.
    Ngo, T.A., Lu, Z., Carneiro, G.: Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance. Med. Image Anal. 35, 159–171 (2017)CrossRefGoogle Scholar
  11. 11.
    Patraucean, V., Handa, A., Cipolla, R.: Spatio-temporal video autoencoder with differentiable memory. In: ICLR Workshop (2016)Google Scholar
  12. 12.
    Rueckert, D., Sonoda, L.I., Hayes, C., et al.: Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans. Med. Imaging 18(8), 712–721 (1999)CrossRefGoogle Scholar
  13. 13.
    Shen, D., Sundar, H., Xue, Z., Fan, Y., Litt, H.: Consistent estimation of cardiac motions by 4D image registration. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3750, pp. 902–910. Springer, Heidelberg (2005). Scholar
  14. 14.
    Shi, W., Zhuang, X., Wang, H., et al.: A comprehensive cardiac motion estimation framework using both untagged and 3-D tagged MR images based on nonrigid registration. IEEE Trans. Med. Imaging 31(6), 1263–1275 (2012)CrossRefGoogle Scholar
  15. 15.
    Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., et al.: A deep metric for multimodal registration. In: MICCAI, pp. 10–18 (2016)Google Scholar
  16. 16.
    Tobon-Gomez, C., De Craene, M., Mcleod, K., et al.: Benchmarking framework for myocardial tracking and deformation algorithms: an open access database. Med. Image Anal. 17(6), 632–648 (2013)CrossRefGoogle Scholar
  17. 17.
    Tsai, Y.H., Yang, M.H., Black, M.J.: Video segmentation via object flow. In: CVPR, pp. 3899–3908 (2016)Google Scholar
  18. 18.
    Uzunova, H., Wilms, M., Handels, H., Ehrhardt, J.: Training CNNs for image registration from few samples with model-based data augmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 223–231. Springer, Cham (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Chen Qin
    • 1
    Email author
  • Wenjia Bai
    • 1
  • Jo Schlemper
    • 1
  • Steffen E. Petersen
    • 2
  • Stefan K. Piechnik
    • 3
  • Stefan Neubauer
    • 3
  • Daniel Rueckert
    • 1
  1. 1.Department of ComputingImperial College LondonLondonUK
  2. 2.NIHR Biomedical Research Centre at BartsQueen Mary University of LondonLondonUK
  3. 3.Division of Cardiovascular Medicine, Radcliffe Department of MedicineUniversity of OxfordOxfordUK

Personalised recommendations