Virtual annotations of the surgical field through an augmented reality transparent display
- 696 Downloads
Existing telestrator-based surgical telementoring systems require a trainee surgeon to shift focus frequently between the operating field and a nearby monitor to acquire and apply instructions from a remote mentor. We present a novel approach to surgical telementoring where annotations are superimposed directly onto the surgical field using an augmented reality (AR) simulated transparent display. We present our first steps towards realizing this vision, using two networked conventional tablets to allow a mentor to remotely annotate the operating field as seen by a trainee. Annotations are anchored to the surgical field as the trainee tablet moves and as the surgical field deforms or becomes occluded. The system is built exclusively from compact commodity-level components—all imaging and processing are performed on the two tablets.
KeywordsAugmented reality Telementoring Telemedicine Annotation anchoring
We thank Sthitapragyan Parida for his help with the implementation and demonstration of our telementoring system. We thank Chun-hao Hsu and Aviran Malik for their help with the tablet mount system used in our experiments. We thank Meng-Lin Wu, Xiaoxian Dong, Chengyuan Lin, and the entire computer graphics group at the computer science department of Purdue University for their feedback on this work.
This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the Department of Defense.
- 2.Amazon.com, I.: Amazon Fire Phone (2014). http://www.amazon.com/firephone
- 4.Baričević, D., Höllerer, T., Sen, P., Turk, M.: User-perspective augmented reality magic lens from gradients. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 87–96. ACM (2014)Google Scholar
- 6.Bogen, E.M., Augestad, K.M., Patel, H.R., Lindsetmo, R.O.: Telementoring in education of laparoscopic surgeons: An emerging technology. World J. Gastrointest. Endosc. 6(5), 148–155 (2014). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4024487/
- 7.Chou, W., Wang, T., Zhang, Y.: Augmented reality based preoperative planning for robot assisted tele-neurosurgery. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 3, pp. 2901–2906 vol. 3 (2004)Google Scholar
- 10.Google: ATAP Project Tango (2014). https://www.google.com/atap/projecttango/
- 11.Guo, Y., Henao, O., Jackson, T., Quereshy, F., Okrainec, A.: Commercial videoconferencing for use in telementoring laparoscopic surgery. Med. Meets Virtual Real. 21: NextMed/MMVR21 196, 147 (2014)Google Scholar
- 12.Itseez: OpenCV (2014). http://opencv.org/
- 13.Khronos: OpenGL ES—the standard for embedded accelerated 3D graphics (2014). https://www.khronos.org/opengles/
- 14.Loescher, T., Lee, S.Y., Wachs, J.P.: An augmented reality approach to surgical telementoring. In: Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, pp. 2341–2346. IEEE (2014)Google Scholar
- 15.Marescaux, J., Diana, M.: Robotics and remote surgery: Next step. In: K.C. Kim (ed.) Robotics in General Surgery, pp. 479–484-. Springer New York (2014)Google Scholar
- 17.Occipital, I.: The Structure Sensor is the first 3D sensor for mobile devices (2014). http://structure.io/
- 18.Ponce, B.A., Jennings, J.K., Clay, T.B., May, M.B., Huisingh, C., Sheppard, E.D.: Telementoring: Use of augmented reality in orthopaedic education. J. Bone Jt. Surg. 96(10), e84 (2014). http://jbjs.org/content/96/10/e84.abstract
- 19.Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Computer VisionECCV 2006, pp. 430–443. Springer (2006)Google Scholar
- 20.Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564–2571. IEEE (2011)Google Scholar
- 23.Shenai, M.B., Dillavou, M., Shum, C., Ross, D., Tubbs, R.S., Shih, A., Guthrie, B.L.: Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, ons200-ons207 (2011)Google Scholar
- 24.Smurro, J.P., Reina, G.A., L’esperance, J.O.: System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom). US Patent App. 14/138,045, Google Patents (2014). https://www.google.com/patents/US20140176661
- 25.Tomioka, M., Ikeda, S., Sato, K.: Approximated user-perspective rendering in tablet-based augmented reality. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 21–28. IEEE (2013)Google Scholar
- 27.Unuma, Y., Niikura, T., Komuro, T.: See-through mobile ar system for natural 3d interaction. In: Proceedings of the companion publication of the 19th international conference on Intelligent User Interfaces, pp. 17–20. ACM (2014)Google Scholar