Abstract
This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its coordinates wirelessly to a HoloLens device. In the scenario of surgical training, a virtual surgical scene with pre-recorded surgical annotations is superimposed with the actual surgical scene so that the surgical trainee is able to operate following virtual instructions. In the scenario of telementored surgery, the virtual surgical scene is co-registered with the actual surgical scene so that the virtual scalpel remotely mentored by an experienced surgeon provides the AR guidance for the inexperienced on-site operator. The performance characteristics of the proposed AR telementoring system are verified by benchtop experiments. The clinical applicability of the proposed system in telementored skin grafting surgery and fasciotomy is validated in a New Zealand rabbit model. Our benchtop and in vivo experiments demonstrate the potential to improve surgical performance and reduce healthcare disparities in remote areas with limited resources.
Similar content being viewed by others
References
Andersen, D., V. Popescu, M. E. Cabrera, A. Shanghavi, G. Gomez, et al. Medical telementoring using an augmented reality transparent display. Surgery 159:1646–1653, 2016.
Augestad, K. M., J. G. Bellika, A. Budrionis, T. Chomutare, R. O. Lindsetmo, et al. Surgical telementoring in knowledge translation—clinical outcomes and educational benefits: a comprehensive review. Surg. Innov. 20:273–281, 2013.
Augestad, K. M., and R. O. Lindsetmo. Overcoming distance: video-conferencing as a clinical and educational tool among surgeons. World J. Surg. 33:1356–1365, 2009.
Bashshur, R., and G. W. Shannon. History of Telemedicine: Evolution, Context, and Transformation. New Rochelle, NY: Mary Ann Liebe, 2009.
Bogen, E. M., K. M. Augestad, H. R. Patel, and R. O. Lindsetmo. Telementoring in education of laparoscopic surgeons: an emerging technology. World J. Gastrointest. Endosc. 6:148–155, 2014.
Budrionis, A., K. M. Augestad, H. R. Patel, and J. G. Bellika. An evaluation framework for defining the contributions of telestration in surgical telementoring. Interact. J. Med. Res. 2(2):14, 2013.
Chen, H., A. S. Lee, M. Swift, J. C. Tang, in Proceedings of the 3rd International Workshop on Immersive Media Experiences - ImmersiveME ‘15, 2015.
DeBakey, M. E. Telemedicine has now come of age. Telemed. J. 1(1):3–4, 1995.
Demartines, N., U. Otto, D. Mutter, L. Labler, A. von Weymarn, et al. An evaluation of telemedicine in surgery: telediagnosis compared with direct diagnosis. Arch. Surg. 135:849–853, 2000.
Di Cerbo, A., J. C. Morales-Medina, B. Palmieri, and T. Iannitti. Narrative review of telemedicine consultation in medical practice. Patient Prefer. Adher. 9:65–75, 2015.
Doarn, C. R. Telemedicine in tomorrow’s operating room: a natural fit. Semin. Laparosc. Surg. 10:121–126, 2003.
Francescutti, L. H., M. Sauve, and A. S. Prasad. Natural disasters and healthcare: lessons to be learned. Healthc. Manag. Forum 30:53–55, 2017.
Fullman, N., J. Yearwood, S. M. Abay, C. Abbafati, F. Abd-Allah, et al. Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: a systematic analysis from the Global Burden of Disease Study 2016. Lancet 391:2236–2271, 2018.
Li, C., and B. Tang. J. Phys. 1168:032045, 2019.
Miner, R. C. Image-guided neurosurgery. J. Med. Imaging Radiat. Sci. 48:328–335, 2017.
Moretti-Marques, R., M. P. Salcedo, D. Callegaro Filho, A. Lopes, M. Vieira, et al. Telementoring in gynecologic oncology training: changing lives in Mozambique. Int. J. Gynecol. Cancer 2019. https://doi.org/10.1136/ijgc-2019-000653.
Rickard, J., F. Ntirenganya, G. Ntakiyiruta, and K. Chu. Global health in the 21st century: equity in surgical training partnerships. J. Surg. Educ. 76:9–13, 2019.
Sebajang, H., P. Trudeau, A. Dougall, S. Hegge, C. McKinley, et al. Telementoring: an important enabling tool for the community surgeon. Surg. Innov. 12:327–331, 2005.
Sereno, S., D. Mutter, B. Dallemagne, C. D. Smith, and J. Marescaux. Telementoring for minimally invasive surgical training by wireless robot. Surg. Innov. 14:184–191, 2007.
Shao, P., H. Ding, J. Wang, P. Liu, Q. Ling, et al. Designing a wearable navigation system for image-guided cancer resection surgery. Ann. Biomed. Eng. 42:2228–2237, 2014.
Shenai, M. B., M. Dillavou, C. Shum, D. Ross, R. S. Tubbs, et al. Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68:200–207, 2011.
Siegrist, J., and M. Marmot. Health inequalities and the psychosocial environment—two scientific challenges. Soc. Sci. Med. 58:1463–1473, 2004.
Strode, S. W., S. Gustke, and A. Allen. Technical and clinical progress in telemedicine. JAMA 281:1066–1068, 1999.
Tepper, O. M., H. L. Rudy, A. Lefkowitz, K. A. Weimer, S. M. Marks, et al. Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast. Reconstr. Surg. 140:1066–1070, 2017.
Wiborg, A., and B. Widder. Teleneurology to improve stroke care in rural areas. Stroke 34:2951–2956, 2003.
Zhang, Z., J. Pei, D. Wang, Q. Gan, J. Ye, et al. A wearable Goggle navigation system for dual-mode optical and ultrasound localization of suspicious lesions: validation studies using tissue-simulating phantoms and an ex vivo human breast tissue Model. PLoS ONE 11(7):e0157854, 2016.
Acknowledgments
Funding for this project was partially from an Anhui Province Science and Technology Major Project (No. 17030801004) and an OSUCCC Intramural Research Program Pelotonia Award.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Associate Editor Andreas Anayiotos oversaw the review of this article.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (MP4 10644 kb)
Supplementary material 2 (MP4 12398 kb)
Supplementary material 3 (MP4 6354 kb)
Rights and permissions
About this article
Cite this article
Liu, P., Li, C., Xiao, C. et al. A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens. Ann Biomed Eng 49, 287–298 (2021). https://doi.org/10.1007/s10439-020-02538-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10439-020-02538-5