Skip to main content
SpringerLink
Log in

A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens

  • Original Article
  • Published:
Annals of Biomedical Engineering Aims and scope Submit manuscript

Abstract

This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its coordinates wirelessly to a HoloLens device. In the scenario of surgical training, a virtual surgical scene with pre-recorded surgical annotations is superimposed with the actual surgical scene so that the surgical trainee is able to operate following virtual instructions. In the scenario of telementored surgery, the virtual surgical scene is co-registered with the actual surgical scene so that the virtual scalpel remotely mentored by an experienced surgeon provides the AR guidance for the inexperienced on-site operator. The performance characteristics of the proposed AR telementoring system are verified by benchtop experiments. The clinical applicability of the proposed system in telementored skin grafting surgery and fasciotomy is validated in a New Zealand rabbit model. Our benchtop and in vivo experiments demonstrate the potential to improve surgical performance and reduce healthcare disparities in remote areas with limited resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8

Similar content being viewed by others

References

  1. Andersen, D., V. Popescu, M. E. Cabrera, A. Shanghavi, G. Gomez, et al. Medical telementoring using an augmented reality transparent display. Surgery 159:1646–1653, 2016.

    Article  Google Scholar 

  2. Augestad, K. M., J. G. Bellika, A. Budrionis, T. Chomutare, R. O. Lindsetmo, et al. Surgical telementoring in knowledge translation—clinical outcomes and educational benefits: a comprehensive review. Surg. Innov. 20:273–281, 2013.

    Article  Google Scholar 

  3. Augestad, K. M., and R. O. Lindsetmo. Overcoming distance: video-conferencing as a clinical and educational tool among surgeons. World J. Surg. 33:1356–1365, 2009.

    Article  Google Scholar 

  4. Bashshur, R., and G. W. Shannon. History of Telemedicine: Evolution, Context, and Transformation. New Rochelle, NY: Mary Ann Liebe, 2009.

    Google Scholar 

  5. Bogen, E. M., K. M. Augestad, H. R. Patel, and R. O. Lindsetmo. Telementoring in education of laparoscopic surgeons: an emerging technology. World J. Gastrointest. Endosc. 6:148–155, 2014.

    Article  Google Scholar 

  6. Budrionis, A., K. M. Augestad, H. R. Patel, and J. G. Bellika. An evaluation framework for defining the contributions of telestration in surgical telementoring. Interact. J. Med. Res. 2(2):14, 2013.

    Article  Google Scholar 

  7. Chen, H., A. S. Lee, M. Swift, J. C. Tang, in Proceedings of the 3rd International Workshop on Immersive Media Experiences - ImmersiveME ‘15, 2015.

  8. DeBakey, M. E. Telemedicine has now come of age. Telemed. J. 1(1):3–4, 1995.

    Article  Google Scholar 

  9. Demartines, N., U. Otto, D. Mutter, L. Labler, A. von Weymarn, et al. An evaluation of telemedicine in surgery: telediagnosis compared with direct diagnosis. Arch. Surg. 135:849–853, 2000.

    Article  CAS  Google Scholar 

  10. Di Cerbo, A., J. C. Morales-Medina, B. Palmieri, and T. Iannitti. Narrative review of telemedicine consultation in medical practice. Patient Prefer. Adher. 9:65–75, 2015.

    Google Scholar 

  11. Doarn, C. R. Telemedicine in tomorrow’s operating room: a natural fit. Semin. Laparosc. Surg. 10:121–126, 2003.

    PubMed  Google Scholar 

  12. Francescutti, L. H., M. Sauve, and A. S. Prasad. Natural disasters and healthcare: lessons to be learned. Healthc. Manag. Forum 30:53–55, 2017.

    Article  Google Scholar 

  13. Fullman, N., J. Yearwood, S. M. Abay, C. Abbafati, F. Abd-Allah, et al. Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: a systematic analysis from the Global Burden of Disease Study 2016. Lancet 391:2236–2271, 2018.

    Article  Google Scholar 

  14. Li, C., and B. Tang. J. Phys. 1168:032045, 2019.

    Google Scholar 

  15. Miner, R. C. Image-guided neurosurgery. J. Med. Imaging Radiat. Sci. 48:328–335, 2017.

    Article  Google Scholar 

  16. Moretti-Marques, R., M. P. Salcedo, D. Callegaro Filho, A. Lopes, M. Vieira, et al. Telementoring in gynecologic oncology training: changing lives in Mozambique. Int. J. Gynecol. Cancer 2019. https://doi.org/10.1136/ijgc-2019-000653.

    Article  PubMed  Google Scholar 

  17. Rickard, J., F. Ntirenganya, G. Ntakiyiruta, and K. Chu. Global health in the 21st century: equity in surgical training partnerships. J. Surg. Educ. 76:9–13, 2019.

    Article  Google Scholar 

  18. Sebajang, H., P. Trudeau, A. Dougall, S. Hegge, C. McKinley, et al. Telementoring: an important enabling tool for the community surgeon. Surg. Innov. 12:327–331, 2005.

    Article  Google Scholar 

  19. Sereno, S., D. Mutter, B. Dallemagne, C. D. Smith, and J. Marescaux. Telementoring for minimally invasive surgical training by wireless robot. Surg. Innov. 14:184–191, 2007.

    Article  CAS  Google Scholar 

  20. Shao, P., H. Ding, J. Wang, P. Liu, Q. Ling, et al. Designing a wearable navigation system for image-guided cancer resection surgery. Ann. Biomed. Eng. 42:2228–2237, 2014.

    Article  Google Scholar 

  21. Shenai, M. B., M. Dillavou, C. Shum, D. Ross, R. S. Tubbs, et al. Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68:200–207, 2011.

    PubMed  Google Scholar 

  22. Siegrist, J., and M. Marmot. Health inequalities and the psychosocial environment—two scientific challenges. Soc. Sci. Med. 58:1463–1473, 2004.

    Article  Google Scholar 

  23. Strode, S. W., S. Gustke, and A. Allen. Technical and clinical progress in telemedicine. JAMA 281:1066–1068, 1999.

    Article  CAS  Google Scholar 

  24. Tepper, O. M., H. L. Rudy, A. Lefkowitz, K. A. Weimer, S. M. Marks, et al. Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast. Reconstr. Surg. 140:1066–1070, 2017.

    Article  CAS  Google Scholar 

  25. Wiborg, A., and B. Widder. Teleneurology to improve stroke care in rural areas. Stroke 34:2951–2956, 2003.

    Article  Google Scholar 

  26. Zhang, Z., J. Pei, D. Wang, Q. Gan, J. Ye, et al. A wearable Goggle navigation system for dual-mode optical and ultrasound localization of suspicious lesions: validation studies using tissue-simulating phantoms and an ex vivo human breast tissue Model. PLoS ONE 11(7):e0157854, 2016.

    Article  Google Scholar 

Download references

Acknowledgments

Funding for this project was partially from an Anhui Province Science and Technology Major Project (No. 17030801004) and an OSUCCC Intramural Research Program Pelotonia Award.

Author information

Author notes

  1. Peng Liu and Chenmeng Li have contributed equally to this work.

Authors and Affiliations

  1. Department of Precision Machinery and Instrumentation, University of Science and Technology of China, Hefei, Anhui, China

    Peng Liu, Chenmeng Li, Zeshu Zhang, Junqi Ma, Jian Gao, Pengfei Shao & Ronald Xu

  2. Department of Biomedical Engineering, The Ohio State University, Columbus, USA

    Chenmeng Li, Zeshu Zhang & Ronald Xu

  3. Photogrammetric Computer Vision Laboratory, The Ohio State University, Columbus, USA

    Changlin Xiao & Alper Yilmaz

  4. Department of Surgery, The Ohio State University, Columbus, USA

    Ian Valerio & Timothy M. Pawlik

  5. Department of Rehabilitation Medicine, The Second Hospital of Anhui Medical University, Hefei, Anhui, China

    Chengbiao Ding

Authors
  1. Peng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenmeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Changlin Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zeshu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junqi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jian Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Pengfei Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ian Valerio
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Timothy M. Pawlik
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chengbiao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Alper Yilmaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Ronald Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Alper Yilmaz or Ronald Xu.

Additional information

Associate Editor Andreas Anayiotos oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (MP4 10644 kb)

Supplementary material 2 (MP4 12398 kb)

Supplementary material 3 (MP4 6354 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, P., Li, C., Xiao, C. et al. A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens. Ann Biomed Eng 49, 287–298 (2021). https://doi.org/10.1007/s10439-020-02538-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10439-020-02538-5

Keywords

Search

Navigation