Skip to main content
SpringerLink
Log in

Virtual annotations of the surgical field through an augmented reality transparent display

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Existing telestrator-based surgical telementoring systems require a trainee surgeon to shift focus frequently between the operating field and a nearby monitor to acquire and apply instructions from a remote mentor. We present a novel approach to surgical telementoring where annotations are superimposed directly onto the surgical field using an augmented reality (AR) simulated transparent display. We present our first steps towards realizing this vision, using two networked conventional tablets to allow a mentor to remotely annotate the operating field as seen by a trainee. Annotations are anchored to the surgical field as the trainee tablet moves and as the surgical field deforms or becomes occluded. The system is built exclusively from compact commodity-level components—all imaging and processing are performed on the two tablets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Agarwal, R., Levinson, A.W., Allaf, M., Makarov, D.V., Nason, A., Su, L.M.: The roboconsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70(5), 970–974 (2007)

    Article  Google Scholar 

  2. Amazon.com, I.: Amazon Fire Phone (2014). http://www.amazon.com/firephone

  3. Ballantyne, G.H.: Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surg. Endosc. Interv. Tech. 16(10), 1389–1402 (2002)

    Article  Google Scholar 

  4. Baričević, D., Höllerer, T., Sen, P., Turk, M.: User-perspective augmented reality magic lens from gradients. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 87–96. ACM (2014)

  5. Bashshur, R.L.: On the definition and evaluation of telemedicine. Telemed. J. 1(1), 19–30 (1995). doi:10.1089/tmj.1.1995.1.19

    Article  Google Scholar 

  6. Bogen, E.M., Augestad, K.M., Patel, H.R., Lindsetmo, R.O.: Telementoring in education of laparoscopic surgeons: An emerging technology. World J. Gastrointest. Endosc. 6(5), 148–155 (2014). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4024487/

  7. Chou, W., Wang, T., Zhang, Y.: Augmented reality based preoperative planning for robot assisted tele-neurosurgery. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 3, pp. 2901–2906 vol. 3 (2004)

  8. Ereso, A.Q., Garcia, P., Tseng, E., Gauger, G., Kim, H., Dua, M.M., Victorino, G.P., Guy, T.S.: Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J. Am. College Surg. 211(3), 400–411 (2010)

    Article  Google Scholar 

  9. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  10. Google: ATAP Project Tango (2014). https://www.google.com/atap/projecttango/

  11. Guo, Y., Henao, O., Jackson, T., Quereshy, F., Okrainec, A.: Commercial videoconferencing for use in telementoring laparoscopic surgery. Med. Meets Virtual Real. 21: NextMed/MMVR21 196, 147 (2014)

    Google Scholar 

  12. Itseez: OpenCV (2014). http://opencv.org/

  13. Khronos: OpenGL ES—the standard for embedded accelerated 3D graphics (2014). https://www.khronos.org/opengles/

  14. Loescher, T., Lee, S.Y., Wachs, J.P.: An augmented reality approach to surgical telementoring. In: Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, pp. 2341–2346. IEEE (2014)

  15. Marescaux, J., Diana, M.: Robotics and remote surgery: Next step. In: K.C. Kim (ed.) Robotics in General Surgery, pp. 479–484-. Springer New York (2014)

  16. Marescaux, J., Rubino, F.: Telesurgery, telementoring, virtual surgery, and telerobotics. Curr. Urol. Rep. 4(2), 109–113 (2003)

    Article  Google Scholar 

  17. Occipital, I.: The Structure Sensor is the first 3D sensor for mobile devices (2014). http://structure.io/

  18. Ponce, B.A., Jennings, J.K., Clay, T.B., May, M.B., Huisingh, C., Sheppard, E.D.: Telementoring: Use of augmented reality in orthopaedic education. J. Bone Jt. Surg. 96(10), e84 (2014). http://jbjs.org/content/96/10/e84.abstract

  19. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Computer VisionECCV 2006, pp. 430–443. Springer (2006)

  20. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564–2571. IEEE (2011)

  21. Satava, R.: Virtual endoscopy. Surg. Endosc. 10(2), 173–174 (1996)

    Article  Google Scholar 

  22. Schulam, P., Docimo, S., Saleh, W., Breitenbach, C., Moore, R., Kavoussi, L.: Telesurgical mentoring. Surg. Endosc. 11(10), 1001–1005 (1997)

    Article  Google Scholar 

  23. Shenai, M.B., Dillavou, M., Shum, C., Ross, D., Tubbs, R.S., Shih, A., Guthrie, B.L.: Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, ons200-ons207 (2011)

  24. Smurro, J.P., Reina, G.A., L’esperance, J.O.: System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom). US Patent App. 14/138,045, Google Patents (2014). https://www.google.com/patents/US20140176661

  25. Tomioka, M., Ikeda, S., Sato, K.: Approximated user-perspective rendering in tablet-based augmented reality. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 21–28. IEEE (2013)

  26. Treter, S., Perrier, N., Sosa, J.A., Roman, S.: Telementoring: a multi-institutional experience with the introduction of a novel surgical approach for adrenalectomy. Ann. Surg. Oncol. 20(8), 2754–2758 (2013)

    Article  Google Scholar 

  27. Unuma, Y., Niikura, T., Komuro, T.: See-through mobile ar system for natural 3d interaction. In: Proceedings of the companion publication of the 19th international conference on Intelligent User Interfaces, pp. 17–20. ACM (2014)

  28. Vera, A.M., Russo, M., Mohsin, A., Tsuda, S.: Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg. Endosc. 28(12), 3467–3472 (2014)

    Article  Google Scholar 

Download references

Acknowledgments

We thank Sthitapragyan Parida for his help with the implementation and demonstration of our telementoring system. We thank Chun-hao Hsu and Aviran Malik for their help with the tablet mount system used in our experiments. We thank Meng-Lin Wu, Xiaoxian Dong, Chengyuan Lin, and the entire computer graphics group at the computer science department of Purdue University for their feedback on this work.

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the Department of Defense.

Author information

Authors and Affiliations

  1. Department of Computer Science, Purdue University, West Lafayette, IN, USA

    Daniel Andersen, Voicu Popescu, Maria Eugenia Cabrera, Aditya Shanghavi & Juan Wachs

  2. School of Medicine, Indiana University, Indianapolis, IN, USA

    Gerardo Gomez, Sherri Marley & Brian Mullis

Authors
  1. Daniel Andersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Voicu Popescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Eugenia Cabrera
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aditya Shanghavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gerardo Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sherri Marley
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Brian Mullis
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Juan Wachs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Andersen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Andersen, D., Popescu, V., Cabrera, M.E. et al. Virtual annotations of the surgical field through an augmented reality transparent display. Vis Comput 32, 1481–1498 (2016). https://doi.org/10.1007/s00371-015-1135-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-015-1135-6

Keywords

Search

Navigation