Markerless Image-to-Face Registration for Untethered Augmented Reality in Head and Neck Surgery

  • Christina GsaxnerEmail author
  • Antonio Pepe
  • Jürgen Wallner
  • Dieter Schmalstieg
  • Jan Egger
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


In the treatment of head and neck cancer, physicians can benefit from augmented reality in preparing and executing treatment. We present a system allowing a physician wearing an untethered augmented reality headset to see medical visualizations precisely overlaid onto the patient. Our main contribution is a strategy for markerless registration of 3D imaging to the patient’s face. We use a neural network to detect the face using the headset’s depth sensor and register it to computed tomography data. The face registration is seamlessly combined with the headset’s continuous self-localization. We report on registration error and compare our approach to an external, high-precision tracking system.


Augmented reality 3D registration Head and neck cancer 


  1. 1.
    Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical displays: a literature review of augmented reality. J. Disp. Technol. 4(4), 451–467 (2008)CrossRefGoogle Scholar
  2. 2.
    de Oliveira, M.E., Debarba, H.G., Lädermann, A., Chagué, S., Charbonnier, C.: A hand-eye calibration method for augmented reality applied to computer-assisted orthopedic surgery. Int. J. Med. Robot. 15(2), e1969 (2019)CrossRefGoogle Scholar
  3. 3.
    Pratt, P., et al.: Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. Eur. Radiol. Exp. 2(1), 2 (2018)CrossRefGoogle Scholar
  4. 4.
    Incekara, F., Smits, M., Dirven, C., Vincent, A.: Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Neurosurg. 118, e422–e427 (2018)CrossRefGoogle Scholar
  5. 5.
    Ahn, J., Choi, H., Hong, J., Hong, J.: Tracking accuracy of a stereo-camera-based augmented reality navigation system for orthognathic surgery. J. Oral Maxillofac. Surg. 77(5), 1070-e1 (2019)CrossRefGoogle Scholar
  6. 6.
    Chen, X., et al.: Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J. Biomed. Inform. 55, 124–131 (2015)CrossRefGoogle Scholar
  7. 7.
    Hsieh, C.H., Lee, J.D., Wu, C.T.: A Kinect-based medical augmented reality system for craniofacial applications using image-to-patient registration. Neuropsychiatry 07(06), 927–939 (2017)Google Scholar
  8. 8.
    Hajek, J., et al.: Closing the calibration loop: an inside-out-tracking paradigm for augmented reality in orthopedic surgery. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 299–306. Springer, Cham (2018). Scholar
  9. 9.
    Mahmoud, N., et al.: On-patient see-through augmented reality based on visual SLAM. Int. J. Comput. Assist. Radiol. Surg. 12(1), 1–11 (2017)CrossRefGoogle Scholar
  10. 10.
    Klein, G.: Registration on HoloLens. Keynote talk at ISMAR (2017)Google Scholar
  11. 11.
    Durrant-Whyte, H.F.: Sensor models and multisensor integration. Int. J. Robot. Res. 7(6), 97–113 (1988)CrossRefGoogle Scholar
  12. 12.
    Castaldi, P., Leccisotti, L., Bussu, F., Miccichè, F., Rufini, V.: Role of (18)F-FDG PET-CT in head and neck squamous cell carcinoma. Acta Otorhinolaryngol. Ital. 33(1), 1–8 (2013)Google Scholar
  13. 13.
    Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: ISMAR, pp. 1–10 (2007)Google Scholar
  14. 14.
    Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). Scholar
  15. 15.
    Holz, D., Ichim, A.E., Tombari, F., Rusu, R.B., Behnke, S.: Registration with the point cloud library: a modular framework for aligning in 3-D. IEEE Robot. Autom. Mag. 22(4), 110–124 (2015)CrossRefGoogle Scholar
  16. 16.
    Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: ICRA, pp. 3212–3217 (2009)Google Scholar
  17. 17.
    Zhou, Q.-Y., Park, J., Koltun, V.: Fast global registration. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 766–782. Springer, Cham (2016). Scholar
  18. 18.
    Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992)CrossRefGoogle Scholar
  19. 19.
    Tsai, R.Y., Lenz, R.K.: A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans. Robot. Autom. 5(3), 345–358 (1989)CrossRefGoogle Scholar
  20. 20.
    Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marín-Jiménez, M.J.: Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recogn. 47(6), 2280–2292 (2014)CrossRefGoogle Scholar
  21. 21.
    Vassallo, R., Rankin, A., Chen, E.C., Peters, T.M.: Hologram stability evaluation for microsoft hololens. In: SPIE Medical Imaging, vol. 10136, p. 1013614 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Institute of Computer Graphics and VisionGraz University of TechnologyGrazAustria
  2. 2.Division of Oral-, Maxillofacial SurgeryMedical University of GrazGrazAustria
  3. 3.Computer Algorithms for Medicine LaboratoryGrazAustria

Personalised recommendations