Skip to main content

Advertisement

SpringerLink
Log in

Registration of preoperative temporal bone CT-scan to otoendoscopic video for augmented-reality based on convolutional neural networks

  • Otology
  • Published:
European Archives of Oto-Rhino-Laryngology Aims and scope Submit manuscript

Abstract

Purpose

Patient-to-image registration is a preliminary step required in surgical navigation based on preoperative images. Human intervention and fiducial markers hamper this task as they are time-consuming and introduce potential errors. We aimed to develop a fully automatic 2D registration system for augmented reality in ear surgery.

Methods

CT-scans and corresponding oto-endoscopic videos were collected from 41 patients (58 ears) undergoing ear examination (vestibular schwannoma before surgery, profound hearing loss requiring cochlear implant, suspicion of perilymphatic fistula, contralateral ears in cases of unilateral chronic otitis media). Two to four images were selected from each case. For the training phase, data from patients (75% of the dataset) and 11 cadaveric specimens were used. Tympanic membranes and malleus handles were contoured on both video images and CT-scans by expert surgeons. The algorithm used a U-Net network for detecting the contours of the tympanic membrane and the malleus on both preoperative CT-scans and endoscopic video frames. Then, contours were processed and registered through an iterative closest point algorithm. Validation was performed on 4 cases and testing on 6 cases. Registration error was measured by overlaying both images and measuring the average and Hausdorff distances.

Results

The proposed registration method yielded a precision compatible with ear surgery with a 2D mean overlay error of \(0.65 \pm 0.60\) mm for the incus and \(0.48 \pm 0.32\) mm for the round window. The average Hausdorff distance for these 2 targets was \(0.98 \pm 0.60\) mm and \(0.78 \pm 0.34\) mm respectively. An outlier case with higher errors (2.3 mm and 1.5 mm average Hausdorff distance for incus and round window respectively) was observed in relation to a high discrepancy between the projection angle of the reconstructed CT-scan and the video image. The maximum duration for the overall process was 18 s.

Conclusions

A fully automatic 2D registration method based on a convolutional neural network and applied to ear surgery was developed. The method did not rely on any external fiducial markers nor human intervention for landmark recognition. The method was fast and its precision was compatible with ear surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The data used in this study are not publicly available but could be requested from the authors upon reasonable request and with permission from the University Hospital of Dijon.

References

  1. Labadie RF, Shah RJ, Harris SS, Cetinkaya E, Haynes DS, Fenlon MR, Juscyzk AS, Galloway RL, Fitzpatrick JM (2004) Submillimetric target-registration error using a novel, non-invasive fiducial system for image-guided otologic surgery. Comput Aided Surg 9:145–153. https://doi.org/10.3109/10929080500066922

    Article  PubMed  Google Scholar 

  2. Reda FA, Noble JH, Labadie RF, Dawant BM (2012) Automatic pre- to intra-operative ct registration for image-guided cochlear implant surgery. IEEE Trans Biomed Eng 59:3070–3077. https://doi.org/10.1109/TBME.2012.2214775

    Article  PubMed  PubMed Central  Google Scholar 

  3. Hong J, Matsumoto N, Ouchida R, Komune S, Hashizume M (2008) Medical navigation system for otologic surgery based on hybrid registration and virtual intraoperative computed tomography. IEEE Trans Biomed Eng 56(2):426–432. https://doi.org/10.1109/TBME.2008.2008168

    Article  PubMed  Google Scholar 

  4. Alzoubi FQ, Tarifi AA, Khader Y, De Carpentier J (2010) Comparison between transtympanic and elevation of tympanomeatal flap approaches in tympanoplasty. Otol Neurotol 31:773–775. https://doi.org/10.1097/MAO.0b013e3181e40a41

    Article  PubMed  Google Scholar 

  5. Srinivasan V, Toynton SC, Mangat KS (1997) Transtympanic myringoplasty in children. Int J Pediatr Otorhinolaryngol 39:199–204. https://doi.org/10.1016/s0165-5876(96)01477-2

    Article  CAS  PubMed  Google Scholar 

  6. Maurer CR, Fitzpatrick JM, Wang MY, Galloway RL, Maciunas RJ, Allen GS (1997) A minimally invasive registration method using surface template-assisted marker positioning (stamp) for image-guided otologic surgery. Otolaryngol Head Neck Surg 16:447–462. https://doi.org/10.1109/42.611354

    Article  Google Scholar 

  7. Hussein R, Lalande A, Guigou C, Grayeli AB (2020) Contribution of augmented reality to minimally invasive computer-assisted cranial base surgery. IEEE J Biomed Health Inform 24:2093–2106. https://doi.org/10.1109/JBHI.2019.2954003

    Article  Google Scholar 

  8. Luers JC, Hüttenbrink K-B (2016) Surgical anatomy and pathology of the middle ear. J Anat 228:338–353. https://doi.org/10.1111/joa.12389

    Article  PubMed  Google Scholar 

  9. Guigou C, Bardin F, Afifi WS, Dillenseger J-P, Ricolfi F, Grayeli AB (2015) Virtual endoscopy to plan transtympanic approach to labyrinthine windows. Otol Neurotol 36:1338–1342. https://doi.org/10.1097/MAO.0000000000000808

    Article  PubMed  Google Scholar 

  10. Kakehata S (2013) Transtympanic endoscopy for diagnosis of middle ear pathology. Otolaryngol Clin N Am 46:227–232. https://doi.org/10.1016/j.otc.2012.10.006

    Article  Google Scholar 

  11. Marroquin R, Lalande A, Hussain R, Guigou C, Grayeli AB (2018) Augmented reality of the middle ear combining otoendoscopy and temporal bone computed tomography. Otol Neurotol 39:931–939. https://doi.org/10.1097/MAO.0000000000001922

    Article  PubMed  Google Scholar 

  12. Gerber N, Gavaghan KA, Bell BJ, Williamson TM, Weisstanner C, Caversaccio M-D, Weber S (2013) High-accuracy patient-to-image registration for the facilitation of image-guided robotic microsurgery on the head. IEEE Trans Biomed Eng 60(4):960–968. https://doi.org/10.1109/TBME.2013.2241063

    Article  PubMed  Google Scholar 

  13. Schneider D, Hermann J, Gerber KA, Ansó J, Caversaccio MD, Weber S, Anschuetz L (2018) Noninvasive registration strategies and advanced image guidance technology for submillimeter surgical navigation accuracy in the lateral skull base. Otol Neurotol 39(10):1326–1335. https://doi.org/10.1097/MAO.0000000000001993

    Article  PubMed  Google Scholar 

  14. Georg E, Mühling J, Marmulla R (2006) Image-to-patient registration techniques in head surgery. Int J Oral Maxillofac Surg 35:1081–1095. https://doi.org/10.1016/j.ijom.2006.09.015

    Article  Google Scholar 

  15. Omara AJ, Manning W, YiFeng F, Zhijian S (2014) Anatomical landmarks for point-matching registration in image-guided neurosurgery. Int J Med Robot Comput Assist Surg 10:55–64. https://doi.org/10.1002/rcs.1509

    Article  Google Scholar 

  16. Grayeli AB (2009) Use of anatomic or invasive markers in association with skin surface registration in image-guided surgery of the temporal bone. Acta Otolaryngol 129:405–410. https://doi.org/10.1080/00016480802579025

    Article  PubMed  Google Scholar 

  17. Hussain R, Lalande A, Marroquin R, Guigou C, Grayeli AB (2020) Video-based augmented reality combining ct-scan and instrument position data to microscope view in middle ear surgery. Sci Rep 10:6767. https://doi.org/10.1038/s41598-020-63839-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Hussain R, Guigou C, Lalande A, Bozorg Grayeli A (2022) Vision-based augmented reality system for middle ear surgery: evaluation in operating room environment. Otol Neurotol 43(3):385–394. https://doi.org/10.1097/MAO.0000000000003441

    Article  PubMed  Google Scholar 

  19. Rosset A, Spadola L, Ratib O (2004) Osirix: an open-source software for navigating in multidimensional dicom images. J Digit Imaging 17:205–216. https://doi.org/10.1007/s10278-004-1014-6

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597

  21. Lam L, Lee SW, Suen CY (1992) Thinning methodologies-a comprehensive survey. IEEE Trans Pattern Anal Mach Intell 14:869–885. https://doi.org/10.1109/34.161346

    Article  Google Scholar 

  22. Chen Y, Medioni G (1992) Object modelling by registration of multiple range images. Image Vis Comput 10:145–155. https://doi.org/10.1016/0262-8856(92)90066-C

    Article  Google Scholar 

  23. Torr PHS, Zisserman A (2000) Mlesac: a new robust estimator with application to estimating image geometry. Comput Vis Image Underst 78:138–156. https://doi.org/10.1006/cviu.1999.0832

    Article  Google Scholar 

  24. Matsumoto N, Yamashita M, Cho B, Komune N, Hashizume M (2020) Asymmetrical surface scanning registration for image-guided otologic surgery: a phantom study. Auris Nasus Larynx 47:574–579. https://doi.org/10.1016/j.anl.2020.01.007

    Article  PubMed  Google Scholar 

  25. Ledderose GJ, Stelter K, Leunig A, Hagedorn H et al (2007) Surface laser registration in ent-surgery: accuracy in the paranasal sinuses—a cadaveric study. Rhinology 45(4):281

    PubMed  Google Scholar 

  26. Kim Y, Li R, Na YH, Lee R, Xing L (2014) Accuracy of surface registration compared to conventional volumetric registration in patient positioning for head-and-neck radiotherapy: a simulation study using patient data. Med Phys 41(12):121701. https://doi.org/10.1118/1.4898103

    Article  PubMed  PubMed Central  Google Scholar 

  27. Hardy SM, Melroy C, White DR, Dubin M, Senior B (2006) A comparison of computer-aided surgery registration methods for endoscopic sinus surgery. Am J Rhinol 20(1):48–52

    Article  PubMed  Google Scholar 

  28. Taleb A, Guigou C, Leclerc S, Lalande A, Bozorg Grayeli A (2023) Image-to-patient registration in computer-assisted surgery of head and neck: state-of-the-art, perspectives, and challenges. J Clin Med 12(16):5398. https://doi.org/10.3390/jcm12165398

    Article  PubMed  PubMed Central  Google Scholar 

  29. Maurer CR, Fitzpatrick JM, Wang MY, Galloway RL, Maciunas RJ, Allen GS (1997) Registration of head volume images using implantable fiducial markers. IEEE Trans Med Imaging 16:447–462. https://doi.org/10.1109/42.611354

    Article  PubMed  Google Scholar 

Download references

Author information

Author notes

  1. Sarah Leclerc and Raabid Hussein contributed equally.

Authors and Affiliations

  1. ICMUB Laboratory UMR CNRS 6302, University of Burgundy Franche Comte, 21000, Dijon, France

    Ali Taleb, Sarah Leclerc, Alain Lalande & Alexis Bozorg-Grayeli

  2. Oticon Medical, 06220, Vallauris, France

    Raabid Hussein

  3. Medical Imaging Department, Dijon University Hospital, 21000, Dijon, France

    Alain Lalande

  4. ENT Department, Dijon University Hospital, 21000, Dijon, France

    Alexis Bozorg-Grayeli

Authors
  1. Ali Taleb
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Leclerc
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raabid Hussein
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alain Lalande
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexis Bozorg-Grayeli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Taleb.

Ethics declarations

Conflict of interest

The authors: Ali Taleb, Drs. Alain Lalande, Sarah Leclerc and Raabid Hussain, Prof. Alexis Bozorg-Grayeli, have no conflicts of interest or financial ties to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taleb, A., Leclerc, S., Hussein, R. et al. Registration of preoperative temporal bone CT-scan to otoendoscopic video for augmented-reality based on convolutional neural networks. Eur Arch Otorhinolaryngol 281, 2921–2930 (2024). https://doi.org/10.1007/s00405-023-08403-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00405-023-08403-0

Keywords

Search

Navigation