LSTM Spatial Co-transformer Networks for Registration of 3D Fetal US and MR Brain Images

  • Robert WrightEmail author
  • Bishesh Khanal
  • Alberto Gomez
  • Emily Skelton
  • Jacqueline Matthew
  • Jo V. Hajnal
  • Daniel Rueckert
  • Julia A. Schnabel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11076)


In this work, we propose a deep learning-based method for iterative registration of fetal brain images acquired by ultrasound and magnetic resonance, inspired by “Spatial Transformer Networks”. Images are co-aligned to a dual modality spatio-temporal atlas, where computational image analysis may be performed in the future. Our results show better alignment accuracy compared to “Self-Similarity Context descriptors”, a state-of-the-art method developed for multi-modal image registration. Furthermore, our method is robust and able to register highly misaligned images, with any initial orientation, where similarity-based methods typically fail.



This work was supported by the Wellcome/EPSRC Centre for Medical Engineering [WT 203148/Z/16/Z], Wellcome Trust IEH Award [102431] and NVIDIA with the donation of a Titan Xp GPU.


  1. 1.
    Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: CVPR (2018)Google Scholar
  2. 2.
    Cheng, X., Zhang, L., Zheng, Y.: Deep similarity learning for multimodal medical images. Comput. Meth. Biomech. Biomed. Eng. Imaging Vis. 6(3), 248–252 (2018)CrossRefGoogle Scholar
  3. 3.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP, pp. 1724–1734 (2014)Google Scholar
  4. 4.
    Haber, E., Modersitzki, J.: Intensity gradient based registration and fusion of multi-modal images. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) MICCAI 2006. LNCS, vol. 4191, pp. 726–733. Springer, Heidelberg (2006). Scholar
  5. 5.
    Heinrich, M.P., Jenkinson, M., Papież, B.W., Brady, S.M., Schnabel, J.A.: Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 187–194. Springer, Heidelberg (2013). Scholar
  6. 6.
    Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: NIPS, pp. 2017–2025 (2015)Google Scholar
  7. 7.
    Kuklisova-Murgasova, M., et al.: Registration of 3D fetal brain US and MRI. In: MICCAI, pp. 667–674 (2012)CrossRefGoogle Scholar
  8. 8.
    Lin, C.H., Lucey, S.: Inverse compositional spatial transformer networks. In: CVPR, pp. 2252–2260 (2017)Google Scholar
  9. 9.
    Mellor, M., Brady, M.: Phase mutual information as a similarity measure for registration. Med. Image Anal. 9(4), 330–343 (2005). Functional Imaging and Modeling of the Heart - FIMH 2003Google Scholar
  10. 10.
    Ourselin, S., Roche, A., Subsol, G., Pennec, X., Ayache, N.: Reconstructing a 3D structure from serial histological sections. Image Vis. Comput. 19(1), 25–31 (2001)CrossRefGoogle Scholar
  11. 11.
    Pech-Pacheco, J.L., Cristobal, G., Chamorro-Martinez, J., Fernandez-Valdivia, J.: Diatom autofocusing in brightfield microscopy: a comparative study. ICPR 3, 314–317 (2000)Google Scholar
  12. 12.
    Rivaz, H., Karimaghaloo, Z., Collins, D.L.: Self-similarity weighted mutual information: a new nonrigid image registration metric. Med. Image Anal. 18(2), 343–358 (2014)CrossRefGoogle Scholar
  13. 13.
    Rohé, M.-M., Datar, M., Heimann, T., Sermesant, M., Pennec, X.: SVF-Net: learning deformable image registration using shape matching. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 266–274. Springer, Cham (2017). Scholar
  14. 14.
    Serag, A., Aljabar, P., Ball, G., Counsell, S.J., Boardman, J.P., Rutherford, M.A., Edwards, A.D., Hajnal, J.V., Rueckert, D.: Construction of a consistent high-definition spatio-temporal atlas of the developing brain using adaptive Kernel regression. NeuroImage 59(3), 2255–2265 (2012)CrossRefGoogle Scholar
  15. 15.
    Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., Navab, N., Komodakis, N.: A deep metric for multimodal registration. In: MICCAI, pp. 10–18 (2016)Google Scholar
  16. 16.
    Wachinger, C., Navab, N.: Entropy and Laplacian images: structural representations for multi-modal registration. Med. Image Anal. 16(1), 1–17 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Robert Wright
    • 1
    Email author
  • Bishesh Khanal
    • 1
  • Alberto Gomez
    • 1
  • Emily Skelton
    • 1
  • Jacqueline Matthew
    • 1
  • Jo V. Hajnal
    • 1
  • Daniel Rueckert
    • 2
  • Julia A. Schnabel
    • 1
  1. 1.School of Biomedical Engineering and Imaging SciencesKing’s College LondonLondonUK
  2. 2.Department of ComputingImperial College LondonLondonUK

Personalised recommendations