Skip to main content

Hierarchical Multi-modal Image Registration by Learning Common Feature Representations

Part of the Lecture Notes in Computer Science book series (LNIP,volume 9352)


Mutual information (MI) has been widely used for registering images with different modalities. Since most inter-modality registration methods simply estimate deformations in a local scale, but optimizing MI from the entire image, the estimated deformations for certain structures could be dominated by the surrounding unrelated structures. Also, since there often exist multiple structures in each image, the intensity correlation between two images could be complex and highly nonlinear, which makes global MI unable to precisely guide local image deformation. To solve these issues, we propose a hierarchical inter-modality registration method by robust feature matching. Specifically, we first select a small set of key points at salient image locations to drive the entire image registration. Since the original image features computed from different modalities are often difficult for direct comparison, we propose to learn their common feature representations by projecting them from their native feature spaces to a common space, where the correlations between corresponding features are maximized. Due to the large heterogeneity between two high-dimension feature distributions, we employ Kernel CCA (Canonical Correlation Analysis) to reveal such non-linear feature mappings. Then, our registration method can take advantage of the learned common features to reliably establish correspondences for key points from different modality images by robust feature matching. As more and more key points take part in the registration, our hierarchical feature-based image registration method can efficiently estimate the deformation pathway between two inter-modality images in a global to local manner. We have applied our proposed registration method to prostate CT and MR images, as well as the infant MR brain images in the first year of life. Experimental results show that our method can achieve more accurate registration results, compared to other state-of-the-art image registration methods.

This is a preview of subscription content, access via your institution.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Maintz, J., Viergever, M.: A survey of medical image registration. Medical Image Analysis 2, 1–36 (1998)

    CrossRef  Google Scholar 

  2. Pluim, J.P.W., Maintz, J.B.A., Viergever, M.A.: Mutual-information-based registration of medical images: a survey. IEEE Transactions on Medical Imaging 22, 986–1004 (2003)

    CrossRef  Google Scholar 

  3. Wells, I., William, M., Viola, P., Atsumi, H., Nakajima, S., Kikinis, R.: Multi-modal volume registration by maximization of mutual information. Medical Image Analysis 1, 35–51 (1996)

    CrossRef  Google Scholar 

  4. Knops, Z.F., Maintz, J.B.A., Viergever, M.A., Pluim, J.P.W.: Normalized mutual information based registration using k-means clustering and shading correction. Medical Image Analysis 10, 432–439 (2006)

    CrossRef  Google Scholar 

  5. Wu, G., Kim, M., Wang, Q., Shen, D.: S-HAMMER: Hierarchical Attribute-Guided, Symmetric Diffeomorphic Registration for MR Brain Images. Human Brain Mapping 35 (2014)

    CrossRef  Google Scholar 

  6. Hardoon, D.R., Szedmak, S., Shawe-Taylor, J.: Canonical Correlation Analysis: An Overview with Application to Learning Methods. Neural Computation 16, 2639–2664 (2004)

    CrossRef  Google Scholar 

  7. Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.W.: elastix: a toolbox for intensity based medical image registration. IEEE Transaction on Medical Imaging 29, 196–205 (2010)

    CrossRef  Google Scholar 

  8. Li, G., Wang, L., Shi, F., Lyall, A.E., Lin, W., Gilmore, J.H., Shen, D.: Mapping longitudinal development of local cortical gyrification in infants from birth to 2 years of age. The Journal of Neuroscience 34, 4228–4238 (2014)

    CrossRef  Google Scholar 

  9. Wang, L., Shi, F., Gao, Y., Li, G., Gilmore, J.H., Lin, W., Shen, D.: Integration of sparse multi-modality representation and anatomical constraint for isointense infant brain MR image segmentation. NeuroImage 89, 152–164 (2014)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Dinggang Shen .

Editor information

Editors and Affiliations

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Ge, H., Wu, G., Wang, L., Gao, Y., Shen, D. (2015). Hierarchical Multi-modal Image Registration by Learning Common Feature Representations. In: Zhou, L., Wang, L., Wang, Q., Shi, Y. (eds) Machine Learning in Medical Imaging. MLMI 2015. Lecture Notes in Computer Science(), vol 9352. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24887-5

  • Online ISBN: 978-3-319-24888-2

  • eBook Packages: Computer ScienceComputer Science (R0)