Skip to main content
SpringerLink
Log in

Graph-Based Slice-to-Volume Deformable Registration

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Deformable image registration is a fundamental problem in computer vision and medical image computing. In this paper we investigate the use of graphical models in the context of a particular type of image registration problem, known as slice-to-volume registration. We introduce a scalable, modular and flexible formulation that can accommodate low-rank and high order terms, that simultaneously selects the plane and estimates the in-plane deformation through a single shot optimization approach. The proposed framework is instantiated into different variants seeking either a compromise between computational efficiency (soft plane selection constraints and approximate definition of the data similarity terms through pair-wise components) or exact definition of the data terms and the constraints on the plane selection. Simulated and real-data in the context of ultrasound and magnetic resonance registration (where both framework instantiations as well as different optimization strategies are considered) demonstrate the potentials of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2012). The lazy flipper: Efficient depth-limited exhaustive search in discrete graphical models. In A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato & C. Schmid (Eds.), Computer vision (ECCV 2012) (pp. 154–166). Berlin: Springer.

  • Baudin, P. Y., Goodman, D., Kumar, P., Azzabou, N., Carlier, P. G., Paragios, N., et al. (2013). Discriminative parameter estimation for random walks segmentation. In K. Mori, I. Sakuma, Y. Sato, C. Barillot & N. Navab (Eds.), Medical image computing and computer-assisted intervention (MICCAI 2013) (pp. 219–226). Berlin: Springer.

  • Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society Series B (Methodological), 48, 259–302.

  • Birkfellner, W., Figl, M., Kettenbach, J., Hummel, J., Homolka, P., Schernthaner, R., et al. (2007). Rigid 2D/3D slice-to-volume registration and its application on fluoroscopic CT images. Medical Physics, 34(1), 246. doi:10.1118/1.2401661.

    Article  Google Scholar 

  • Chandler, A. G., Pinder, R. J., Netsch, T., Schnabel, J. A., Hawkes, D. J., Hill, D. L., et al. (2008). Correction of misaligned slices in multi-slice MR cardiac examinations by using slice-to-volume registration. Journal of Cardiovascular Magnetic Resonance, 2008(10), 13.

    Article  Google Scholar 

  • Eresen, A., Li, P., & Ji, J. X. (2014). Correlating 2D histological slice with 3D MRI image volume using smart phone as an interactive tool for muscle study. In 2014 36th Annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 6418–6421). IEEE.

  • Fei, B., Duerk, J. L., & Wilson, D. L. (2002). Automatic 3D registration for interventional MRI-guided treatment of prostate cancer. Computer Aided Surgery, 7(5), 257–267.

    Article  Google Scholar 

  • Ferrante, E., Fecamp, V., & Paragios, N. (2015a). Implicit planar and in-plane deformable mapping in medical images through high order graphs. In IEEE international symposium on biomedical imaging: From nano to macro (ISBI).

  • Ferrante, E., Fecamp, V., & Paragios, N. (2015b). Slice-to-volume deformable registration: Efficient one shot consensus between plane selection and in-plane deformation. International Journal of Computer Assisted Radiology and Surgery (IJCARS), 10(6), 16.

  • Ferrante, E., & Paragios, N. (2013). Non-rigid 2D–3D medical image registration using Markov random fields. In Medical image computing and computer-assisted intervention (MICCAI 2013) (pp. 163–170). Berlin: Springer.

  • Ferrante, E., & Paragios, N. (2017). Slice-to-volume medical image registration: A survey. Medical Image Analysis, 39, 101–123.

  • Gill, S., Abolmaesumi, P., Vikal, S., Mousavi, P., & Fichtinger, G. (2008). Intraoperative prostate tracking with slice-to-volume registration in MRI. In Proceedings of the 20th international conference of the society for medical innovation and technology 2008 (pp. 154–158). Society for Medical Innovation and Technology SMIT.

  • Glocker, B. (2010). Random fields for image registration. PhD Thesis.

  • Glocker, B., Komodakis, N., Paragios, N., & Navab, N. (2009). Approximated curvature penalty in non-rigid registration using pairwise MRFs. In G. Bebis, R. Boyle, B. Parvin, D. Koracin, Y. Kuno, J. Wang, et al. (Eds.), Advances in visual computing (pp. 1101–1109). Berlin: Springer.

  • Glocker, B., Komodakis, N., Tziritas, G., Navab, N., & Paragios, N. (2008). Dense image registration through MRFs and efficient linear programming. Medical Image Analysis, 12(6), 731–741. doi:10.1016/j.media.2008.03.006. Special issue on information processing in medical imaging 2007.

    Article  Google Scholar 

  • Glocker, B., Sotiras, A., Komodakis, N., & Paragios, N. (2011). Deformable medical image registration: Setting the state of the art with discrete methods. Annual Review of Biomedical Engineering, 13, 219–244. doi:10.1146/annurev-bioeng-071910-124649.

    Article  Google Scholar 

  • Huang, X., Moore, J., Guiraudon, G., Jones, D. L., Bainbridge, D., Ren, J., et al. (2009). Dynamic 2D ultrasound and 3D ct image registration of the beating heart. IEEE Transactions on Medical Imaging, 28(8), 1179–1189.

    Article  Google Scholar 

  • Jiang, S., Xue, H., Counsell, S., Anjari, M., Allsop, J., Rutherford, M., et al. (2009). Diffusion tensor imaging (DTI) of the brain in moving subjects: Application to in-utero fetal and ex-utero studies. Magnetic Resonance in Medicine, 62(3), 645–655.

    Article  Google Scholar 

  • Kappes, J. H., Andres, B., Hamprecht, F. A., Schnörr, C., Nowozin, S., Batra, D., et al. (2013). A comparative study of modern inference techniques for discrete energy minimization problems. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2013 (pp. 1328–1335).

  • Kim, B., Boes, J. L., Bland, P. H., Chenevert, T. L., & Meyer, C. R. (1999). Motion correction in fMRI via registration of individual slices into an anatomical volume. Magnetic Resonance in Medicine, 41(5), 964–972.

  • Kohli, P., & Rother, C. (2012). Higher-order models in computer vision. Chapter 1, Image processing and analysis with graphs. Boca Raton: CRC Press.

    MATH  Google Scholar 

  • Komodakis, N., Paragios, N., & Tziritas, G. (2011). MRF energy minimization and beyond via dual decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3), 531–552.

  • Komodakis, N., Tziritas, G., & Paragios, N. (2007). Fast, approximately optimal solutions for single and dynamic MRFs. In IEEE conference on computer vision and pattern recognition, 2007 (CVPR’07) (pp. 1–8). IEEE. http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=4269955.

  • Komodakis, N., Xiang, B., & Paragios, N. (2015). A framework for efficient structured max-margin learning of high-order MRF models. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 37(7), 1425–1441.

  • Kschischang, F. R., Frey, B. J., & Loeliger, H. A. (2001). Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47(2), 498–519.

    Article  MathSciNet  MATH  Google Scholar 

  • Kwon, D., Lee, K. J., Yun, I. D., & Lee, S. U. (2008). Nonrigid image registration using dynamic higher-order MRF model. In D. Forsyth, P. Torr & A. Zisserman (Eds.), Computer Vision (ECCV 2008) (pp. 373–386). Berlin: Springer.

  • Leung, K. Y. E., van Stralen, M., Nemes, A., Voormolen, M. M., van Burken, G., Geleijnse, M. L., et al. (2008). Sparse registration for three-dimensional stress echocardiography. IEEE Transactions on Medical Imaging, 27(11), 1568–1579.

  • Liao, R., Zhang, L., Sun, Y., Miao, S., & Chefd’Hotel, C. (2013). A review of recent advances in registration techniques applied to minimally invasive therapy. IEEE Transactions on Multimedia, 15(5), 983–1000.

    Article  Google Scholar 

  • Markelj, P., Tomaževič, D., Likar, B., & Pernuš, F. (2012). A review of 3D/2D registration methods for image-guided interventions. Medical Image Analysis, 16(3), 642–661.

  • Mercier, L., Del Maestro, R. F., Petrecca, K., Araujo, D., Haegelen, C., & Collins, D. L. (2012). Online database of clinical MR and ultrasound images of brain tumors. Medical Physics, 39, 3253.

    Article  Google Scholar 

  • Murphy, K. P., Weiss, Y., & Jordan, M. I. (1999). Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (UAI’99).

  • Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. The Computer Journal, 7(4), 308–313. doi:10.1093/comjnl/7.4.308.

    Article  MathSciNet  MATH  Google Scholar 

  • Olesch, J., Beuthien, B., Heldmann, S., Papenberg, N., & Fischer, B. (2011). Fast intra-operative non-linear registration of 3D-ct to tracked, selected 2D-ultrasound slices. In SPIE medical imaging (pp. 79642R–79642R). International Society for Optics and Photonics.

  • Osechinskiy, S., & Kruggel, F. (2011). Slice-to-volume nonrigid registration of histological sections to MR images of the human brain. Anatomy Research International, 2011, 1–17. doi:10.1155/2011/287860.

    Article  Google Scholar 

  • Paragios, N., Ferrante, E., Glocker, B., Komodakis, N., Parisot, S., & Zacharaki, E. I. (2016). (Hyper)-graphical models in biomedical image analysis. Medical Image Analysis, 33, 102–106.

  • Paragios, N., Komodakis, N. (2014). Discrete visual perception. In 2014 22nd International conference on pattern recognition (ICPR) (pp. 18–25). IEEE.

  • Porchetto, R., Stramana, F., Paragios, N., & Ferrante, E. (2016). Rigid slice-to-volume medical image registration through Markov random fields. BAMBI Workshop, MICCAI 2016.

  • Ramalingam, S., Kohli, P., Alahari, K., & Torr, P. H. (2008). Exact inference in multi-label crfs with higher order cliques. In IEEE conference on computer vision and pattern recognition, 2008 (CVPR 2008) (pp. 1–8). IEEE.

  • Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L., Leach, M. O., & Hawkes, D. J. (1999). Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Transactions on Medical Imaging, 18(8), 712–721.

    Article  Google Scholar 

  • San José Estépar, R., Westin, C., & Vosburgh, K. (2009). Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures. International Journal of Computer Assisted Radiology and Surgery, 4(6), 549–560.

    Article  Google Scholar 

  • Seshamani, S., Fogtmann, M., Cheng, X., Thomason, M., Gatenby, C., & Studholme, C. (2013). Cascaded slice to volume registration for moving fetal fMRI. In 2013 IEEE 10th international symposium on biomedical imaging (ISBI) (pp. 796–799). IEEE.

  • Shekhovtsov, A., Kovtun, I., & Hlaváč, V. (2008). Efficient MRF deformation model for non-rigid image matching. Computer Vision and Image Understanding, 112(1), 91–99.

    Article  Google Scholar 

  • Sotiras, A., Davatzikos, C., & Paragios, N. (2013). Deformable medical image registration: A survey. IEEE Transactions on Medical Imaging, 32(7), 1153–1190.

    Article  Google Scholar 

  • Sotiras, A., Ou, Y., Glocker, B., Davatzikos, C., & Paragios, N. (2010). Simultaneous geometric–iconic registration. In T. Jiang, N. Navab, J. P. W. Pluim & M. A. Viergever (Eds.), Medical image computing and computer-assisted intervention (MICCAI 2010) (pp. 676–683). Berlin: Springer.

  • Wang, C., Komodakis, N., & Paragios, N. (2013). Markov random field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding, 117(11), 1610–1627.

    Article  Google Scholar 

  • Xu, H., Lasso, A., Fedorov, A., Tuncali, K., Tempany, C., & Fichtinger, G. (2014). Multi-slice-to-volume registration for MRI-guided transperineal prostate biopsy. International Journal of Computer Assisted Radiology and Surgery, 10(5), 1–10.

  • Zikic, D., Glocker, B., Kutter, O., Groher, M., Komodakis, N., Kamen, A., et al. (2010). Linear intensity-based image registration by Markov random fields and discrete optimization. Medical Image Analysis, 14(4), 550–562.

    Article  Google Scholar 

Download references

Acknowledgements

This research was partially supported by European Research Council Starting Grant Diocles (ERC-STG-259112). We thank Mihir Sahasrabudhe for proof-reading the paper, and Puneet Kumar Dokania, Vivien Fecamp and Jorg Kappes for helpful discussions.

Author information

Authors and Affiliations

  1. Center for Visual Computing, CentraleSupelec, INRIA, Universite Paris-Saclay, Paris, France

    Enzo Ferrante & Nikos Paragios

  2. Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, London, UK

    Enzo Ferrante

  3. TheraPanacea, Paris, France

    Nikos Paragios

Authors
  1. Enzo Ferrante
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikos Paragios
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enzo Ferrante.

Additional information

Communicated by Ron Kimmel.

Images in this work are better viewed in color.

Appendix 1: Continuous Slice-to-Volume Registration

Appendix 1: Continuous Slice-to-Volume Registration

In this appendix, we include a brief comparative study among alternative continuous models for slice-to-volume registration. Comparison is performed using the monomodal heart dataset (see Sect. 3.2.1 for a complete description). The aim of this experiment was to choose the most accurate method for deformable registration which (together with the standard rigid model) was then used as baseline for comparison with the discrete approaches proposed in this work (see Sect. 3.2).

Following the literature on slice-to-volume registration (for a complete survey on slice-to-volume registration see Ferrante and Paragios (2017)), we adopted a decoupled model where the transformation consists in a global 6-DOF rigid transformation for plane selection, and a 2D FFD to represent the deformation field. To account for smooth deformations, the FFD is regularized using the Jacobian of the deformation field, a common regularizer used in the deformable image registration community (Sotiras et al. 2013). Optimization was performed through the continuous Nelder-Mead simplex algorithm described in Sect. 3.2, adopted in many slice-to-volume registration studies (see Section 4.1.2 of Ferrante and Paragios 2017). The grid resolution for the 2D FFDs was set to be equivalent to the resolutions used in the discrete experiments (see Sects. 3.2.1, 3.2.2). We run simplex optimization until convergence or until a maximum of 10,000 simplex iterations were achieved. For the rigid model, convergence was always reached in a few seconds. In case of the deformable models, the algorithm did not converge in all the cases, achieving maximum running times of around 40 s for 10,000 iterations. We experimented with more iterations (100,000) but we did not reach significant improvements in the results.

Fig. 16
figure 16

Comparison among four different slice-to-volume registration models optimized using continuous optimization (Nelder-mean simplex algorithm). We compare a simple 6-DOF rigid transformation (Cont Rigid) and three variants of a decoupled model with a global 6-DOF rigid transformation and a 2D FFD. In the first variant (Cont Def-Joint Rig+Def), rigid and deformable parameters are optimized jointly. In the second case (Cont Def-Two Steps) a two steps strategy is adopted: first, only rigid parameters are optimized until convergence; then, both rigid and deformable parameters are optimized jointly. In the last case (Cont Def-Two Steps Indep), a two steps strategy is also adopted, the difference being that when optimizing the deformable parameters, the rigid ones are not modified. The Cont Def-Two Steps model outperforms the others according to all the metrics (distance between rigid transformations, Dice coefficient, specificity, sensitivity, Hausdorff and CMD)

Full size image

Figure 16 summarizes the results for the comparative study including a simple 6-DOF rigid transformation and three variants of a decoupled model with a global 6-DOF rigid transformation and a 2D FFD. As it can be observed, the Cont Def-Two Steps outperforms the other models. That is why it was chosen as baseline for comparison with the discrete approaches proposed in this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ferrante, E., Paragios, N. Graph-Based Slice-to-Volume Deformable Registration. Int J Comput Vis 126, 36–58 (2018). https://doi.org/10.1007/s11263-017-1040-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-017-1040-8

Keywords

Search

Navigation