Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Virtual annotations of the surgical field through an augmented reality transparent display


Existing telestrator-based surgical telementoring systems require a trainee surgeon to shift focus frequently between the operating field and a nearby monitor to acquire and apply instructions from a remote mentor. We present a novel approach to surgical telementoring where annotations are superimposed directly onto the surgical field using an augmented reality (AR) simulated transparent display. We present our first steps towards realizing this vision, using two networked conventional tablets to allow a mentor to remotely annotate the operating field as seen by a trainee. Annotations are anchored to the surgical field as the trainee tablet moves and as the surgical field deforms or becomes occluded. The system is built exclusively from compact commodity-level components—all imaging and processing are performed on the two tablets.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18


  1. 1.

    Agarwal, R., Levinson, A.W., Allaf, M., Makarov, D.V., Nason, A., Su, L.M.: The roboconsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70(5), 970–974 (2007)

  2. 2.

    Amazon.com, I.: Amazon Fire Phone (2014). http://www.amazon.com/firephone

  3. 3.

    Ballantyne, G.H.: Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surg. Endosc. Interv. Tech. 16(10), 1389–1402 (2002)

  4. 4.

    Baričević, D., Höllerer, T., Sen, P., Turk, M.: User-perspective augmented reality magic lens from gradients. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 87–96. ACM (2014)

  5. 5.

    Bashshur, R.L.: On the definition and evaluation of telemedicine. Telemed. J. 1(1), 19–30 (1995). doi:10.1089/tmj.1.1995.1.19

  6. 6.

    Bogen, E.M., Augestad, K.M., Patel, H.R., Lindsetmo, R.O.: Telementoring in education of laparoscopic surgeons: An emerging technology. World J. Gastrointest. Endosc. 6(5), 148–155 (2014). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4024487/

  7. 7.

    Chou, W., Wang, T., Zhang, Y.: Augmented reality based preoperative planning for robot assisted tele-neurosurgery. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 3, pp. 2901–2906 vol. 3 (2004)

  8. 8.

    Ereso, A.Q., Garcia, P., Tseng, E., Gauger, G., Kim, H., Dua, M.M., Victorino, G.P., Guy, T.S.: Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J. Am. College Surg. 211(3), 400–411 (2010)

  9. 9.

    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

  10. 10.

    Google: ATAP Project Tango (2014). https://www.google.com/atap/projecttango/

  11. 11.

    Guo, Y., Henao, O., Jackson, T., Quereshy, F., Okrainec, A.: Commercial videoconferencing for use in telementoring laparoscopic surgery. Med. Meets Virtual Real. 21: NextMed/MMVR21 196, 147 (2014)

  12. 12.

    Itseez: OpenCV (2014). http://opencv.org/

  13. 13.

    Khronos: OpenGL ES—the standard for embedded accelerated 3D graphics (2014). https://www.khronos.org/opengles/

  14. 14.

    Loescher, T., Lee, S.Y., Wachs, J.P.: An augmented reality approach to surgical telementoring. In: Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, pp. 2341–2346. IEEE (2014)

  15. 15.

    Marescaux, J., Diana, M.: Robotics and remote surgery: Next step. In: K.C. Kim (ed.) Robotics in General Surgery, pp. 479–484-. Springer New York (2014)

  16. 16.

    Marescaux, J., Rubino, F.: Telesurgery, telementoring, virtual surgery, and telerobotics. Curr. Urol. Rep. 4(2), 109–113 (2003)

  17. 17.

    Occipital, I.: The Structure Sensor is the first 3D sensor for mobile devices (2014). http://structure.io/

  18. 18.

    Ponce, B.A., Jennings, J.K., Clay, T.B., May, M.B., Huisingh, C., Sheppard, E.D.: Telementoring: Use of augmented reality in orthopaedic education. J. Bone Jt. Surg. 96(10), e84 (2014). http://jbjs.org/content/96/10/e84.abstract

  19. 19.

    Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Computer VisionECCV 2006, pp. 430–443. Springer (2006)

  20. 20.

    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564–2571. IEEE (2011)

  21. 21.

    Satava, R.: Virtual endoscopy. Surg. Endosc. 10(2), 173–174 (1996)

  22. 22.

    Schulam, P., Docimo, S., Saleh, W., Breitenbach, C., Moore, R., Kavoussi, L.: Telesurgical mentoring. Surg. Endosc. 11(10), 1001–1005 (1997)

  23. 23.

    Shenai, M.B., Dillavou, M., Shum, C., Ross, D., Tubbs, R.S., Shih, A., Guthrie, B.L.: Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, ons200-ons207 (2011)

  24. 24.

    Smurro, J.P., Reina, G.A., L’esperance, J.O.: System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom). US Patent App. 14/138,045, Google Patents (2014). https://www.google.com/patents/US20140176661

  25. 25.

    Tomioka, M., Ikeda, S., Sato, K.: Approximated user-perspective rendering in tablet-based augmented reality. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 21–28. IEEE (2013)

  26. 26.

    Treter, S., Perrier, N., Sosa, J.A., Roman, S.: Telementoring: a multi-institutional experience with the introduction of a novel surgical approach for adrenalectomy. Ann. Surg. Oncol. 20(8), 2754–2758 (2013)

  27. 27.

    Unuma, Y., Niikura, T., Komuro, T.: See-through mobile ar system for natural 3d interaction. In: Proceedings of the companion publication of the 19th international conference on Intelligent User Interfaces, pp. 17–20. ACM (2014)

  28. 28.

    Vera, A.M., Russo, M., Mohsin, A., Tsuda, S.: Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg. Endosc. 28(12), 3467–3472 (2014)

Download references


We thank Sthitapragyan Parida for his help with the implementation and demonstration of our telementoring system. We thank Chun-hao Hsu and Aviran Malik for their help with the tablet mount system used in our experiments. We thank Meng-Lin Wu, Xiaoxian Dong, Chengyuan Lin, and the entire computer graphics group at the computer science department of Purdue University for their feedback on this work.

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the Department of Defense.

Author information

Correspondence to Daniel Andersen.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Andersen, D., Popescu, V., Cabrera, M.E. et al. Virtual annotations of the surgical field through an augmented reality transparent display. Vis Comput 32, 1481–1498 (2016). https://doi.org/10.1007/s00371-015-1135-6

Download citation


  • Augmented reality
  • Telementoring
  • Telemedicine
  • Annotation anchoring