Skip to main content
SpringerLink
Log in
Book cover

International Conference on Medical Image Computing and Computer-Assisted Intervention

MICCAI 2020: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 pp 222–232Cite as

Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12263)

Abstract

Deformable image registration between Computed Tomography (CT) images and Magnetic Resonance (MR) imaging is essential for many image-guided therapies. In this paper, we propose a novel translation-based unsupervised deformable image registration method. Distinct from other translation-based methods that attempt to convert the multimodal problem (e.g., CT-to-MR) into a unimodal problem (e.g., MR-to-MR) via image-to-image translation, our method leverages the deformation fields estimated from both: (i) the translated MR image and (ii) the original CT image in a dual-stream fashion, and automatically learns how to fuse them to achieve better registration performance. The multimodal registration network can be effectively trained by computationally efficient similarity metrics without any ground-truth deformation. Our method has been evaluated on two clinical datasets and demonstrates promising results compared to state-of-the-art traditional and learning-based methods.

Keywords

  • Multimodal registration
  • Generative adversarial network
  • Unsupervised learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S., Yang, B.: Unsupervised medical image translation using cycle-MedGAN. In: 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1–5. IEEE (2019)

    Google Scholar 

  2. Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal. 12(1), 26–41 (2008)

    CrossRef  Google Scholar 

  3. Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C.: A reproducible evaluation of ants similarity metric performance in brain image registration. NeuroImage 54, 2033–2044 (2011)

    CrossRef  Google Scholar 

  4. Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9252–9260 (2018)

    Google Scholar 

  5. Cao, X., et al.: Deformable image registration based on similarity-steered CNN regression. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 300–308. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_35

    CrossRef  Google Scholar 

  6. Cao, X., Yang, J., Wang, L., Xue, Z., Wang, Q., Shen, D.: Deep learning based inter-modality image registration supervised by intra-modality similarity. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 55–63. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00919-9_7

    CrossRef  Google Scholar 

  7. Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)

    CrossRef  Google Scholar 

  8. Heinrich, M.P., et al.: Mind: modality independent neighbourhood descriptor for multi-modal deformable registration. Med. Image Anal. 16(7), 1423–1435 (2012)

    CrossRef  Google Scholar 

  9. Hiasa, Y., et al.: Cross-modality image synthesis from unpaired data using CycleGAN. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N. (eds.) SASHIMI 2018. LNCS, vol. 11037, pp. 31–41. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00536-8_4

    CrossRef  Google Scholar 

  10. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)

    Google Scholar 

  11. Hu, X., Kang, M., Huang, W., Scott, M.R., Wiest, R., Reyes, M.: Dual-stream pyramid registration network. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 382–390. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_43

    CrossRef  Google Scholar 

  12. Hu, Y., et al.: Label-driven weakly-supervised learning for multimodal deformable image registration. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1070–1074. IEEE (2018)

    Google Scholar 

  13. Hu, Y., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 49, 1–13 (2018)

    CrossRef  Google Scholar 

  14. Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)

    Google Scholar 

  15. Kuang, D., Schmah, T.: FAIM – a ConvNet method for unsupervised 3D medical image registration. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 646–654. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32692-0_74

    CrossRef  Google Scholar 

  16. Liu, C., Ma, L., Lu, Z., Jin, X., Xu, J.: Multimodal medical image registration via common representations learning and differentiable geometric constraints. Electron. Lett. 55(6), 316–318 (2019)

    CrossRef  Google Scholar 

  17. Mahapatra, D., Ge, Z., Sedai, S., Chakravorty, R.: Joint registration and segmentation of Xray images using generative adversarial networks. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 73–80. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00919-9_9

    CrossRef  Google Scholar 

  18. Qin, C., Shi, B., Liao, R., Mansi, T., Rueckert, D., Kamen, A.: Unsupervised deformable registration for multi-modal images via disentangled representations. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 249–261. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20351-1_19

    CrossRef  Google Scholar 

  19. Sedghi, A., et al.: Semi-supervised image registration using deep learning. In: Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 10951, p. 109511G. International Society for Optics and Photonics (2019)

    Google Scholar 

  20. Tanner, C., Ozdemir, F., Profanter, R., Vishnevsky, V., Konukoglu, E., Goksel, O.: Generative adversarial networks for mr-ct deformable image registration. arXiv preprint arXiv:1807.07349 (2018)

  21. de Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Išgum, I.: A deep learning framework for unsupervised affine and deformable image registration. Med. Image Anal. 52, 128–143 (2019)

    CrossRef  Google Scholar 

  22. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    CrossRef  Google Scholar 

  23. Wei, D., et al.: Synthesis and inpainting-based MR-CT registration for image-guided thermal ablation of liver tumors. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 512–520. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_57

    CrossRef  Google Scholar 

  24. West, J.B., et al.: Comparison and evaluation of retrospective intermodality image registration techniques. In: Medical Imaging 1996: Image Processing, vol. 2710, pp. 332–347. International Society for Optics and Photonics (1996)

    Google Scholar 

  25. Yang, H., et al.: Unpaired brain MR-to-CT synthesis using a structure-constrained CycleGAN. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00889-5_20

    CrossRef  Google Scholar 

  26. Zhao, S., Lau, T., Luo, J., Eric, I., Chang, C., Xu, Y.: Unsupervised 3D end-to-end medical image registration with volume tweening network. IEEE J. Biomed. Health Inform. (2019)

    Google Scholar 

  27. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Cyclegan (2017). https://github.com/xhujoy/CycleGAN-tensorflow

  28. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgement

This project was supported by the National Institutes of Health (Grant No. R01EB025964, R01DK119269, and P41EB015898) and the Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (Grant No. HW2018008).

Author information

Authors and Affiliations

  1. Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

    Zhe Xu, Jiangpeng Yan & Xiu Li

  2. Brigham and Women’s Hospital, Harvard Medical School, Boston, USA

    Zhe Xu, Jie Luo, Ritvik Pulya, William Wells III & Jayender Jagadeesan

  3. Graduate School of Frontier Sciences, The University of Tokyo, Tokyo, Japan

    Jie Luo

Authors
  1. Zhe Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiangpeng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ritvik Pulya
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. William Wells III
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jayender Jagadeesan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jayender Jagadeesan .

Editor information

Editors and Affiliations

  1. University of Toronto, Toronto, ON, Canada

    Prof. Anne L. Martel

  2. The University of British Columbia, Vancouver, BC, Canada

    Purang Abolmaesumi

  3. University College London, London, UK

    Danail Stoyanov

  4. École Centrale de Nantes, Nantes, France

    Diana Mateus

  5. EURECOM, Biot, France

    Maria A. Zuluaga

  6. Chinese Academy of Sciences, Beijing, China

    S. Kevin Zhou

  7. Sorbonne University, Paris, France

    Daniel Racoceanu

  8. The Hebrew University of Jerusalem, Jerusalem, Israel

    Prof. Leo Joskowicz

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Z. et al. (2020). Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12263. Springer, Cham. https://doi.org/10.1007/978-3-030-59716-0_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59716-0_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59715-3

  • Online ISBN: 978-3-030-59716-0

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation