Deformable multimodal registration for navigation in beating-heart cardiac surgery

  • Jacob J. Peoples
  • Gianluigi Bisleri
  • Randy E. EllisEmail author
Original Article



Minimally invasive beating-heart surgery is currently performed using endoscopes and without navigation. Registration of intraoperative ultrasound to a preoperative cardiac CT scan is a valuable step toward image-guided navigation.


The registration was achieved by first extracting a representative point set from each ultrasound image in the sequence using a deformable registration. A template shape representing the cardiac chambers was deformed through a hierarchy of affine transformations to match each ultrasound image using a generalized expectation maximization algorithm. These extracted point sets were matched to the CT by exhaustively searching over a large number of precomputed slices of 3D geometry. The result is a similarity transformation mapping the intraoperative ultrasound to preoperative CT.


Complete data sets were acquired for four patients. Transesophageal echocardiography ultrasound sequences were deformably registered to a model of oriented points with a mean error of 2.3 mm. Ultrasound and CT scans were registered to a mean of 3 mm, which is comparable to the error of 2.8 mm expected by merging ultrasound registration with uncertainty of cardiac CT.


The proposed algorithm registered 3D CT with dynamic 2D intraoperative imaging. The algorithm aligned the images in both space and time, needing neither dynamic CT imaging nor intraoperative electrocardiograms. The accuracy was sufficient for navigation in thoracoscopically guided beating-heart surgery.


Cardiac surgery Surgical navigation Image registration Transesophageal echocardiography Cardiac CT 



This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN-2018-04430.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

Supplementary material 1 (mp4 4486 KB)


  1. 1.
    Bisleri G, Rosati F, Bontempi L, Curnis A, Muneretto C (2013) Hybrid approach for the treatment of long-standing persistent atrial fibrillation: electrophysiological findings and clinical results. Eur J Cardiothorac Surg 44(5):919–923CrossRefPubMedGoogle Scholar
  2. 2.
    Muneretto C, Bisleri G, Bontempi L, Curnis A (2012) Durable staged hybrid ablation with thoracoscopic and percutaneous approach for treatment of long-standing atrial fibrillation: a 30-month assessment with continuous monitoring. J Thorac Cardiovasc Surg 144(6):1460–1465CrossRefPubMedGoogle Scholar
  3. 3.
    Huang X, Moore J, Guiraudon G, Jones DL, Bainbridge D, Ren J, Peters TM (2009) Dynamic 2D ultrasound and 3D CT image registration of the beating heart. IEEE Trans Med Imaging 28(8):1179–1189CrossRefPubMedGoogle Scholar
  4. 4.
    Luo Z, Cai J, Peters TM, Gu L (2013) Intra-operative 2-D ultrasound and dynamic 3-D aortic model registration for magnetic navigation of transcatheter aortic valve implantation. IEEE Trans Med Imaging 32(11):2152–2165CrossRefPubMedGoogle Scholar
  5. 5.
    Tavard F, Simon A, Leclercq C, Donal E, Hernndez AI, Garreau M (2014) Multimodal registration and data fusion for cardiac resynchronization therapy optimization. IEEE Trans Med Imaging 33(6):1363–1372CrossRefPubMedGoogle Scholar
  6. 6.
    Li FP, Rajchl M, White JA, Goela A, Peters TM (2015) Ultrasound guidance for beating heart mitral valve repair augmented by synthetic dynamic CT. IEEE Trans Med Imaging 34(10):2025–2035CrossRefPubMedGoogle Scholar
  7. 7.
    Khalil A, Faisal A, Lai KW, Ng SC, Liew YM (2017) 2D to 3D fusion of echocardiography and cardiac CT for TAVR and TAVI image guidance. Med Biol Eng Comput 55(8):1317–1326CrossRefPubMedGoogle Scholar
  8. 8.
    Sandoval Z, Castro M, Alirezaie J, Bessire F, Lafon C, Dillenseger JL (2018) Transesophageal 2D ultrasound to 3D computed tomography registration for the guidance of a cardiac arrhythmia therapy. Phys Med Biol 63(15):155007CrossRefPubMedGoogle Scholar
  9. 9.
    Tavakoli V, Amini AA (2013) A survey of shaped-based registration and segmentation techniques for cardiac images. Comput Vis Image Underst 117(9):966–989CrossRefGoogle Scholar
  10. 10.
    Ravikumar N, Gooya A, Frangi AF, Taylor ZA (2017) Generalised coherent point drift for group-wise registration of multi-dimensional point sets. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) Medical image computing and computer-assisted intervention, vol 10433. LNCS. Springer, Cham, pp 309–316Google Scholar
  11. 11.
    Min Z, Wang J, Meng MQH (2018) Robust generalized point cloud registration using hybrid mixture model. In: IEEE Int Conf Robot Autom, pp 4812–4818Google Scholar
  12. 12.
    Besl PJ, McKay ND (1992) A method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256CrossRefGoogle Scholar
  13. 13.
    Ferrante E, Paragios N (2017) Slice-to-volume medical image registration: a survey. Med Image Anal 39:101–123CrossRefPubMedGoogle Scholar
  14. 14.
    Ferrante E, Paragios N (2013) Non-rigid 2D–3D medical image registration using Markov random fields. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N (eds) Medical image computing and computer-assisted intervention, vol 8151. LNCS. Springer, Heidelberg, pp 163–170Google Scholar
  15. 15.
    Ferrante E, Fecamp V, Paragios N (2015) Slice-to-volume deformable registration: efficient one-shot consensus between plane selection and in-plane deformation. Int J Comput Assist Radiol Surg 10(6):791–800CrossRefPubMedGoogle Scholar
  16. 16.
    Ferrante E, Fecamp V, Paragios N (2015) Implicit planar and in-plane deformable mapping in medical images through high order graphs. In: Proc IEEE Int Symp Biomed Imaging, pp 721–724Google Scholar
  17. 17.
    Ferrante E, Paragios N (2018) Graph-based slice-to-volume deformable registration. Int J Comput Vis 126(1):36–58CrossRefGoogle Scholar
  18. 18.
    Zikic D, Glocker B, Kutter O, Groher M, Komodakis N, Kamen A, Paragios N, Navab N (2010) Linear intensity-based image registration by Markov random fields and discrete optimization. Med Image Anal 14(4):550–562CrossRefPubMedGoogle Scholar
  19. 19.
    Porchetto R, Stramana F, Paragios N, Ferrante E (2017) Rigid slice-to-volume medical image registration through Markov random fields. Med Comput Vis Bayesian Graph Models Biomed Imaging 2016:172–185CrossRefGoogle Scholar
  20. 20.
    Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Series B Methodol 39(1):1–38Google Scholar
  21. 21.
    Maiseli B, Gu Y, Gao H (2017) Recent developments and trends in point set registration methods. J Vis Commun Image Represent 46(C):95–106CrossRefGoogle Scholar
  22. 22.
    Meng XL, Rubin DB (1993) Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80(2):267–278CrossRefGoogle Scholar
  23. 23.
    Sra S (2012) A short note on parameter approximation for von Mises–Fisher distributions: and a fast implementation of \(I_s(x)\). Comput Stat 27(1):177–190CrossRefGoogle Scholar
  24. 24.
    Horaud R, Forbes F, Yguel M, Dewaele G, Zhang J (2011) Rigid and articulated point registration with expectation conditional maximization. IEEE Trans Pattern Anal Mach Intell 33(3):587–602CrossRefPubMedGoogle Scholar
  25. 25.
    Buechel RR, Husmann L, Herzog BA, Pazhenkottil AP, Nkoulou R, Ghadri JR, Treyer V, von Schulthess P, Kaufmann PA (2011) Low-dose computed tomography coronary angiography with prospective electrocardiogram triggering: feasibility in a large population. J Am Coll Cardiol 57(3):332–336CrossRefPubMedGoogle Scholar
  26. 26.
    Boersma LV, Castella M, van Boven W, Berruezo A, Yilmaz A, Nadal M, Sandoval E, Calvo N, Brugada J, Kelder J, Wijffels M, Mont L (2012) Atrial fibrillation catheter ablation versus surgical ablation treatment (fast): a 2-center randomized clinical trial. Circulation 125(1):23–30CrossRefPubMedGoogle Scholar
  27. 27.
    Khoynezhad A, Ellenbogen KA, Al-Atassi T, Wang PJ, Kasirajan V, Wang X, Edgerton JR (2017) Hybrid atrial fibrillation ablation: current status and a look ahead. Circ Arrhythm Electrophysiol 10(10):e005263CrossRefPubMedGoogle Scholar
  28. 28.
    Jacob G, Noble JA, Behrenbruch C, Kelion AD, Banning AP (2002) A shape-space-based approach to tracking myocardial borders and quantifying regional left-ventricular function applied in echocardiography. IEEE Trans Med Imaging 21(3):226–238CrossRefPubMedGoogle Scholar
  29. 29.
    Billings S, Taylor R (2015) Generalized iterative most likely oriented-point (G-IMLOP) registration. Int J Comput Assist Radiol Surg 10:1213–1226CrossRefPubMedGoogle Scholar
  30. 30.
    Serafin J, Grisetti G (2015) NICP: dense normal based point cloud registration. Rep U S, pp 742–749Google Scholar
  31. 31.
    Baka N, Metz CT, Schultz CJ, van Geuns R, Niessen WJ, van Walsum T (2014) Oriented Gaussian mixture models for nonrigid 2D/3D coronary artery registration. IEEE Trans Med Imaging 33(5):1023–1034CrossRefPubMedGoogle Scholar
  32. 32.
    Neal RM, Hinton GE (1998) A view of the EM algorithm that justifies incremental, sparse, and other variants. In: Jordan MI (ed) Learning in graphical models. Springer, Dordrecht, pp 355–368CrossRefGoogle Scholar
  33. 33.
    Granger S, Pennec X (2002) Multi-scale EM-ICP: a fast and robust approach for surface registration. Comput Vis ECCV, pp 418–432Google Scholar
  34. 34.
    Bernard F, Salamanca L, Thunberg J, Tack A, Jentsch D, Lamecker H, Zachow S, Hertel F, Goncalves J, Gemmar P (2017) Shape-aware surface reconstruction from sparse 3D point-clouds. Med Image Anal 38:77–89CrossRefPubMedGoogle Scholar
  35. 35.
    Zhou Z, Zheng J, Dai Y, Zhou Z, Chen S (2014) Robust non-rigid point set registration using Student’s-t mixture model. PLoS One 9(3):1–11Google Scholar
  36. 36.
    Ravikumar N, Gooya A, Çimen S, Frangi AF, Taylor ZA (2018) Group-wise similarity registration of point sets using Student’s t-mixture model for statistical shape models. Med Image Anal 44:156–176Google Scholar
  37. 37.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: Proc IEEE Int Conf Comput Vis, vol 2, pp 1150–1157Google Scholar
  38. 38.
    Bay H, Ess A, Tuytelaars T, Gool LV (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346–359CrossRefGoogle Scholar
  39. 39.
    Toews M, Wells WM III (2013) Efficient and robust model-to-image alignment using 3D scale-invariant features. Med Image Anal 17(3):271–282CrossRefPubMedGoogle Scholar
  40. 40.
    Toews M, Wells WM III (2018) Phantomless auto-calibration and online calibration assessment for a tracked freehand 2-D ultrasound probe. IEEE Trans Med Imaging 37(1):262–272CrossRefPubMedGoogle Scholar
  41. 41.
    Machado I, Toews M, Luo J, Unadkat P, Essayed W, George E, Teodoro P, Carvalho H, Martins J, Golland P, Pieper S, Frisken S, Golby A, Wells W III (2018) Non-rigid registration of 3D ultrasound for neurosurgery using automatic feature detection and matching. Int J Comput Assist Radiol Surg 13(10):1525–1538CrossRefPubMedGoogle Scholar
  42. 42.
    Bersvendsen J, Toews M, Danudibroto A, Wells III WM, Urheim S, Estépar RSJ, Samset E (2016) Robust spatio-temporal registration of 4D cardiac ultrasound sequences. Proc SPIE Int Soc Opt EngGoogle Scholar
  43. 43.
    Lasowski R, Benhimane S, Vogel J, Jakobs TF, Zech CJ, Trumm C, Clason C, Navab N (2008) Adaptive visualization for needle guidance in RF liver ablation: taking organ deformation into account. Proc SPIE Int Soc Opt Eng 6918:69180AGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  • Jacob J. Peoples
    • 1
  • Gianluigi Bisleri
    • 1
  • Randy E. Ellis
    • 1
    Email author
  1. 1.Queen’s UniversityKingstonCanada

Personalised recommendations