AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment



Augmented reality (AR) has emerged as a promising approach to support surgeries; however, its application in real world scenarios is still very limited. Besides sophisticated registration tasks that need to be solved, surgical AR visualizations have not been studied in a standardized and comparative manner. To foster the development of future AR applications, a steerable framework is urgently needed to rapidly evaluate new visualization techniques, explore their individual parameter spaces and define relevant application scenarios.


Inspired by its beneficial usage in the automotive industry, the underlying concept of virtual reality (VR) is capable of transforming complex real environments into controllable virtual ones. We present an interactive VR framework, called Augmented Visualization Box (AVB), in which visualizations for AR can be systematically investigated without explicitly performing an error-prone registration. As use case, a virtual laparoscopic scenario with anatomical surface models was created in a computer game engine. In a study with eleven surgeons, we analyzed this VR setting under different environmental factors and its applicability for a quantitative assessment of different AR overlay concepts.


According to the surgeons, the visual impression of the VR scene is mostly influenced by 2D surface details and lighting conditions. The AR evaluation shows that, depending on the visualization used and its capability to encode depth, 37% to 91% of the experts made wrong decisions, but were convinced of their correctness. These results show that surgeons have more confidence in their decisions, although they are wrong, when supported by AR visualizations.


With AVB, intraoperative situations are realistically simulated to quantitatively benchmark current AR overlay methods. Successful surgical task execution in an AR system can only be facilitated if visualizations are customized toward the surgical task.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4


  1. 1.

  2. 2.

  3. 3.


  1. 1.

    Amir-Khalili A, Nosrati MS, Peyrat JM, Hamarneh G, Abugharbieh R (2013) Uncertainty-encoded augmented reality for robot-assisted partial nephrectomy: a phantom study. In: Augmented reality environments for medical imaging and computer-assisted interventions, Springer, pp 182–191

  2. 2.

    Bernhardt S, Nicolau SA, Soler L, Doignon C (2017) The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 37:66–90

  3. 3.

    Bichlmeier C, Sielhorst T, Heining SM, Navab N (2007) Improving depth perception in medical ar. In: Bildverarbeitung für die Medizin 2007, Springer, pp 217–221

  4. 4.

    Dixon BJ, Daly MJ, Chan H, Vescan AD, Witterick IJ, Irish JC (2013) Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg Endosc 27(2):454–461

  5. 5.

    Elhelw M, Nicolaou M, Chung A, Yang GZ, Atkins MS (2008) A gaze-based study for investigating the perception of visual realism in simulated scenes. ACM Trans Appl Percept 5(1):3:1–3:20

  6. 6.

    Hansen C, Wieferich J, Ritter F, Rieder C, Peitgen HO (2010) Illustrative visualization of 3d planning models for augmented reality in liver surgery. Int J Comput Assist Radiol Surg 5(2):133–141

  7. 7.

    Katić D, Wekerle AL, Görtler J, Spengler P, Bodenstedt S, Röhl S, Suwelack S, Kenngott HG, Wagner M, Müller-Stich BP, Dillmann R, Speidel S (2013) Context-aware augmented reality in laparoscopic surgery. Comput Med Imaging Graph 37(2):174–182

  8. 8.

    Kersten-Oertel M, Jannin P, Collins DL (2013) The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph 37(2):98–112 (special Issue on Mixed Reality Guidance of Therapy - Towards Clinical Implementation)

  9. 9.

    Lerotic M, Chung AJ, Mylonas G, Yang GZ (2007) pq-space based non-photorealistic rendering for augmented reality. In: Ayache N, Ourselin S, Maeder A (eds) Medical image computing and computer-assisted intervention—MICCAI 2007, pp 102–109

  10. 10.

    Medenica Z, Kun AL, Paek T, Palinko O (2011) Augmented reality versus street views: a driving simulator study comparing two emerging navigation aids. In: Proceedings of the 13th international conference on human computer interaction with mobile devices and services, ACM, New York, NY, USA, MobileHCI ’11, pp 265–274

  11. 11.

    Nicolaou M, James A, Lo BPL, Darzi A, Yang GZ (2005) Invisible shadow for navigation and planning in minimal invasive surgery. In: Duncan JS, Gerig G (eds) Medical image computing and computer-assisted intervention—MICCAI 2005, pp 25–32

  12. 12.

    Pfeiffer M, Kenngott H, Preukschas A, Huber M, Bettscheider L, Müller-Stich B, Speidel S (2018) Imhotep: virtual reality framework for surgical applications. Int J Comput Assist Radiol Surg 13:741–748

  13. 13.

    Tiefenbacher P, Lehment NH, Rigoll G (2014) Augmented reality evaluation: a concept utilizing virtual reality. Springer International Publishing, Cham, pp 226–236

  14. 14.

    Wild E, Teber D, Schmid D, Simpfendörfer T, Müller M, Baranski AC, Kenngott H, Kopka K, Maier-Hein L (2016) Robust augmented reality guidance with fluorescent markers in laparoscopic surgery. Int J Comput Assist Radiol Surg 11(6):899–907

  15. 15.

    Yiannakopoulou E, Nikiteas N, Perrea D, Tsigris C (2015) Virtual reality simulators and training in laparoscopic surgery. Int J Surg 13:60–64

Download references


We thank Fraunhofer MEVIS, Bremen, Germany for providing segmentation masks of the liver, tumors and vascular structures with which we have created our 3D models of the liver.

Author information

Correspondence to Sandy Engelhardt.

Ethics declarations


This work is partially funded by the Federal Ministry of Education and Research (BMBF) within then STIMULATE research campus (Grant Number 13GW0095A), by the European Regional Development Fund under the operation number ‘ZS/2016/04/78123’ as part of the initiative “Sachsen-Anhalt WISSENSCHAFT Schwerpunkte” and by the German Research Foundation (DFG) project EN 1197/2-1.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

For this type of study, formal consent is not required.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 69594 KB)

Supplementary material 1 (mp4 69594 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hettig, J., Engelhardt, S., Hansen, C. et al. AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. Int J CARS 13, 1717–1725 (2018).

Download citation


  • Surgical augmented reality
  • Virtual reality
  • Visualization