The Visual Computer

, Volume 32, Issue 11, pp 1481–1498 | Cite as

Virtual annotations of the surgical field through an augmented reality transparent display

  • Daniel AndersenEmail author
  • Voicu Popescu
  • Maria Eugenia Cabrera
  • Aditya Shanghavi
  • Gerardo Gomez
  • Sherri Marley
  • Brian Mullis
  • Juan Wachs
Original Article


Existing telestrator-based surgical telementoring systems require a trainee surgeon to shift focus frequently between the operating field and a nearby monitor to acquire and apply instructions from a remote mentor. We present a novel approach to surgical telementoring where annotations are superimposed directly onto the surgical field using an augmented reality (AR) simulated transparent display. We present our first steps towards realizing this vision, using two networked conventional tablets to allow a mentor to remotely annotate the operating field as seen by a trainee. Annotations are anchored to the surgical field as the trainee tablet moves and as the surgical field deforms or becomes occluded. The system is built exclusively from compact commodity-level components—all imaging and processing are performed on the two tablets.


Augmented reality Telementoring  Telemedicine  Annotation anchoring 



We thank Sthitapragyan Parida for his help with the implementation and demonstration of our telementoring system. We thank Chun-hao Hsu and Aviran Malik for their help with the tablet mount system used in our experiments. We thank Meng-Lin Wu, Xiaoxian Dong, Chengyuan Lin, and the entire computer graphics group at the computer science department of Purdue University for their feedback on this work.

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the Department of Defense.


  1. 1.
    Agarwal, R., Levinson, A.W., Allaf, M., Makarov, D.V., Nason, A., Su, L.M.: The roboconsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70(5), 970–974 (2007)CrossRefGoogle Scholar
  2. 2., I.: Amazon Fire Phone (2014).
  3. 3.
    Ballantyne, G.H.: Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surg. Endosc. Interv. Tech. 16(10), 1389–1402 (2002)CrossRefGoogle Scholar
  4. 4.
    Baričević, D., Höllerer, T., Sen, P., Turk, M.: User-perspective augmented reality magic lens from gradients. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 87–96. ACM (2014)Google Scholar
  5. 5.
    Bashshur, R.L.: On the definition and evaluation of telemedicine. Telemed. J. 1(1), 19–30 (1995). doi: 10.1089/tmj.1.1995.1.19 CrossRefGoogle Scholar
  6. 6.
    Bogen, E.M., Augestad, K.M., Patel, H.R., Lindsetmo, R.O.: Telementoring in education of laparoscopic surgeons: An emerging technology. World J. Gastrointest. Endosc. 6(5), 148–155 (2014).
  7. 7.
    Chou, W., Wang, T., Zhang, Y.: Augmented reality based preoperative planning for robot assisted tele-neurosurgery. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 3, pp. 2901–2906 vol. 3 (2004)Google Scholar
  8. 8.
    Ereso, A.Q., Garcia, P., Tseng, E., Gauger, G., Kim, H., Dua, M.M., Victorino, G.P., Guy, T.S.: Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J. Am. College Surg. 211(3), 400–411 (2010)CrossRefGoogle Scholar
  9. 9.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Google: ATAP Project Tango (2014).
  11. 11.
    Guo, Y., Henao, O., Jackson, T., Quereshy, F., Okrainec, A.: Commercial videoconferencing for use in telementoring laparoscopic surgery. Med. Meets Virtual Real. 21: NextMed/MMVR21 196, 147 (2014)Google Scholar
  12. 12.
    Itseez: OpenCV (2014).
  13. 13.
    Khronos: OpenGL ES—the standard for embedded accelerated 3D graphics (2014).
  14. 14.
    Loescher, T., Lee, S.Y., Wachs, J.P.: An augmented reality approach to surgical telementoring. In: Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, pp. 2341–2346. IEEE (2014)Google Scholar
  15. 15.
    Marescaux, J., Diana, M.: Robotics and remote surgery: Next step. In: K.C. Kim (ed.) Robotics in General Surgery, pp. 479–484-. Springer New York (2014)Google Scholar
  16. 16.
    Marescaux, J., Rubino, F.: Telesurgery, telementoring, virtual surgery, and telerobotics. Curr. Urol. Rep. 4(2), 109–113 (2003)CrossRefGoogle Scholar
  17. 17.
    Occipital, I.: The Structure Sensor is the first 3D sensor for mobile devices (2014).
  18. 18.
    Ponce, B.A., Jennings, J.K., Clay, T.B., May, M.B., Huisingh, C., Sheppard, E.D.: Telementoring: Use of augmented reality in orthopaedic education. J. Bone Jt. Surg. 96(10), e84 (2014).
  19. 19.
    Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Computer VisionECCV 2006, pp. 430–443. Springer (2006)Google Scholar
  20. 20.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564–2571. IEEE (2011)Google Scholar
  21. 21.
    Satava, R.: Virtual endoscopy. Surg. Endosc. 10(2), 173–174 (1996)CrossRefGoogle Scholar
  22. 22.
    Schulam, P., Docimo, S., Saleh, W., Breitenbach, C., Moore, R., Kavoussi, L.: Telesurgical mentoring. Surg. Endosc. 11(10), 1001–1005 (1997)CrossRefGoogle Scholar
  23. 23.
    Shenai, M.B., Dillavou, M., Shum, C., Ross, D., Tubbs, R.S., Shih, A., Guthrie, B.L.: Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, ons200-ons207 (2011)Google Scholar
  24. 24.
    Smurro, J.P., Reina, G.A., L’esperance, J.O.: System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom). US Patent App. 14/138,045, Google Patents (2014).
  25. 25.
    Tomioka, M., Ikeda, S., Sato, K.: Approximated user-perspective rendering in tablet-based augmented reality. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 21–28. IEEE (2013)Google Scholar
  26. 26.
    Treter, S., Perrier, N., Sosa, J.A., Roman, S.: Telementoring: a multi-institutional experience with the introduction of a novel surgical approach for adrenalectomy. Ann. Surg. Oncol. 20(8), 2754–2758 (2013)CrossRefGoogle Scholar
  27. 27.
    Unuma, Y., Niikura, T., Komuro, T.: See-through mobile ar system for natural 3d interaction. In: Proceedings of the companion publication of the 19th international conference on Intelligent User Interfaces, pp. 17–20. ACM (2014)Google Scholar
  28. 28.
    Vera, A.M., Russo, M., Mohsin, A., Tsuda, S.: Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg. Endosc. 28(12), 3467–3472 (2014)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Department of Computer SciencePurdue UniversityWest LafayetteUSA
  2. 2.School of MedicineIndiana UniversityIndianapolisUSA

Personalised recommendations