Skip to main content

Advertisement

SpringerLink
Log in

Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Optical see-through head-mounted displays (OST-HMD) feature an unhindered and instantaneous view of the surgery site and can enable a mixed reality experience for surgeons during procedures. In this paper, we present a systematic approach to identify the criteria for evaluation of OST-HMD technologies for specific clinical scenarios, which benefit from using an object-anchored 2D-display visualizing medical information.

Methods

Criteria for evaluating the performance of OST-HMDs for visualization of medical information and its usage are identified and proposed. These include text readability, contrast perception, task load, frame rate, and system lag. We choose to compare three commercially available OST-HMDs, which are representatives of currently available head-mounted display technologies. A multi-user study and an offline experiment are conducted to evaluate their performance.

Results

Statistical analysis demonstrates that Microsoft HoloLens performs best among the three tested OST-HMDs, in terms of contrast perception, task load, and frame rate, while ODG R-7 offers similar text readability. The integration of indoor localization and fiducial tracking on the HoloLens provides significantly less system lag in a relatively motionless scenario.

Conclusions

With ever more OST-HMDs appearing on the market, the proposed criteria could be used in the evaluation of their suitability for mixed reality surgical intervention. Currently, Microsoft HoloLens may be more suitable than ODG R-7 and Epson Moverio BT-200 for clinical usability in terms of the evaluated criteria. To the best of our knowledge, this is the first paper that presents a methodology and conducts experiments to evaluate and compare OST-HMDs for their use as object-anchored 2D-display during interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. ODG R-7: https://www.osterhoutgroup.com/products-r7-glasses.

  2. Testing Smart-glasses to Superimpose Images for Surgery: http://www.hopkinsmedicine.org/news/articles/testing-smartglasses-to-superimpose-images-for-surgery.

  3. Epson Moverio BT-200: https://epson.com/moverio-augmented-reality.

  4. Microsoft HoloLens: https://www.microsoft.com/microsoft-hololens.

  5. Vuforia SDK: https://www.vuforia.com/.

  6. Unity3D Game Engine: https://unity3d.com/.

  7. GoPro Hero4: https://gopro.com/.

References

  1. Abe Y, Sato S, Kato K, Hyakumachi T, Yanagibashi Y, Ito M, Abumi K (2013) A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. J Neurosurg Spine 19(4):492–501

    Article  PubMed  Google Scholar 

  2. Armstrong DG, Rankin TM, Giovinco NA, Mills JL, Matsuoka Y (2014) A heads-up display for diabetic limb salvage surgery a view through the google looking glass. J Diabetes Sci Technol 8:951–956

    Article  PubMed  PubMed Central  Google Scholar 

  3. Azimi E, Doswell J, Kazanzides P (2012) Augmented reality goggles with an integrated tracking system for navigation in neurosurgery. In: 2012 IEEE Virtual Reality Workshops (VRW), pp 123–124. doi:10.1109/VR.2012.6180913

  4. Badiali G, Ferrari V, Cutolo F, Freschi C, Caramella D, Bianchi A, Marchetti C (2014) Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J Craniomaxillofacial Surg 42(8):1970–1976

    Article  Google Scholar 

  5. Bajura M, Fuchs H, Ohbuchi R (1992) Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. SIGGRAPH 26(2):203–210. doi:10.1145/142920.134061

  6. Bichlmeier C, Ockert B, Heining SM, Ahmadi A, Navab N (2008) Stepping into the operating theater: ARAV—augmented reality aided vertebroplasty. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. ISMAR ’08, pp 165–166. IEEE Computer Society, Washington, DC. doi:10.1109/ISMAR.2008.4637348

  7. Chen X, Xu L, Wang Y, Wang H, Wang F, Zeng X, Wang Q, Egger J (2015) Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J Biomed Inform 55:124–131

    Article  PubMed  Google Scholar 

  8. Cutolo F, Parchi PD, Ferrari V (2014) Video see through AR head-mounted display for medical procedures. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), p 393–396. doi:10.1109/ISMAR.2014.6948504

  9. Hart SG (2006) NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50. Sage Publications, pp 904–908. doi:10.1177/154193120605000909

  10. Herron D, Lantis Ii J, Maykel J, Basu C, Schwaitzberg S (1999) The 3-D monitor and head-mounted display. Surg Endosc 13(8):751–755

    Article  CAS  PubMed  Google Scholar 

  11. Hoffman DM, Girshick AR, Akeley K, Banks MS (2008) Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. J Vision 8(3):33–33

    Article  Google Scholar 

  12. Itoh Y, Amano T, Iwai D, Klinker G (2016) Gaussian light field: estimation of viewpoint-dependent blur for optical see-through head-mounted displays. IEEE Trans Vis Comput Graph 22(11):2368–2376

    Article  PubMed  Google Scholar 

  13. Janin AL, Mizell DW, Caudell TP (1993) Calibration of head-mounted displays for augmented reality applications. In: Proceedings of VR annual international symposium, p 246–255. doi:10.1109/VRAIS.1993.380772

  14. Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Graph 18(2):332–352

    Article  PubMed  Google Scholar 

  15. Kersten-Oertel M, Jannin P, Collins DL (2013) The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph 37(2):98–112

    Article  PubMed  Google Scholar 

  16. Koesveld J, Tetteroo G, Graaf E (2003) Use of head-mounted display in transanal endoscopic microsurgery. Surg Endosc 17(6):943–946

    Article  PubMed  Google Scholar 

  17. Kramida G (2016) Resolving the vergence-accommodation conflict in head-mounted displays. IEEE Trans Vis Comput Graph 22(7):1912–1931

    Article  PubMed  Google Scholar 

  18. Kress B, Starner T (2013) A review of head-mounted displays (HMD) technologies and applications for consumer electronics. Proc SPIE 8720:87200A–87200A-13. doi:10.1117/12.2015654

  19. Liao H, Inomata T, Sakuma I, Dohi T (2010) Three-dimensional augmented reality for mriguided surgery using integral videography auto stereoscopic-image overlay. IEEE Trans Biomed Eng 57(6):1476–1486

    Article  PubMed  Google Scholar 

  20. Maithel S, Villegas L, Stylopoulos N, Dawson S, Jones D (2005) Simulated laparoscopy using a head-mounted display vs traditional video monitor: an assessment of performance and muscle fatigue. Surg Endosc 19(3):406–411

    Article  CAS  PubMed  Google Scholar 

  21. Martin-Gonzalez A, Heining SM, Navab N (2009) Head-mounted virtual loupe with sight-based activation for surgical applications. In: 2009 8th IEEE international symposium on mixed and augmented reality, pp 207–208

  22. Mentler T, Wolters C, Herczeg M (2015) Use cases and usability challenges for head-mounted displays in healthcare. Curr Dir Biomed Eng 1(1):534–537

    Google Scholar 

  23. Milgram P, Colquhoun H (1999) A taxonomy of real and virtual world display integration. Mixed Real Merg Real Virtual World 1:1–26

    Google Scholar 

  24. Morisawa T, Kida H, Kusumi F, Okinaga S, Ohana M (2015) Endoscopic submucosal dissection using head-mounted display. Gastroenterology 149(2):290–291

    Article  PubMed  Google Scholar 

  25. Ortega G, Wolff A, Baumgaertner M, Kendoff D (2008) Usefulness of a head mounted monitor device for viewing intraoperative fluoroscopy during orthopaedic procedures. Arch Orthop Trauma Surg 128(10):1123–1126

    Article  CAS  PubMed  Google Scholar 

  26. Qian L, Winkler A, Fuerst B, Kazanzides P, Navab N (2016) Reduction of interaction space in single point active alignment method for optical see-through head-mounted display calibration. In: 2016 IEEE international symposium on mixed and augmented reality (ISMAR-Adjunct), pp 156–157

  27. Queisner M (2016) Medical screen operations: how head-mounted displays transform action and perception in surgical practice. Mediat 6(1):30–51

    Google Scholar 

  28. Rolland JP, Fuchs H (2000) Optical versus video see-through head-mounted displays in medical visualization. Presence Teleoperator Virtual Environ 9(3):287–309

    Article  Google Scholar 

  29. Sadda P, Azimi E, Jallo G, Doswell J, Kazanzides P (2012) Surgical navigation with a head-mounted tracking system and display. Stud Health Technol Inform 184:363–369

    Google Scholar 

  30. Shuhaiber JH (2004) Augmented reality in surgery. Arch Surg 139(2):170–174

    Article  PubMed  Google Scholar 

  31. Suthau T, Vetter M, Hassenpflug P, Meinzer HP, Hellwich O (2002) A concept work for augmented reality visualisation based on a medical application in liver surgery. ISPRS Arch 34(5):274–280

    Google Scholar 

  32. Sutherland IE (1968) A head-mounted three dimensional display. In: Proceedings of the fall joint computer conference, part I, ACM, p 757–764, 9–11 December 1968

  33. Thomsen MN, Lang RD (2004) An experimental comparison of 3-dimensional and 2-dimensional endoscopic systems in a model. Arthroscopy 20(4):419–423

    Article  PubMed  Google Scholar 

  34. Thrun S, Leonard JJ (2008) Simultaneous localization and mapping. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Berlin, pp 871–889

  35. Tuceryan M, Navab N (2000) Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. In: Proceedings IEEE and ACM international symposium on augmented reality (ISAR 2000), pp 149–158

  36. Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose Tracking from natural features on mobile phones. In: Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality, Washington, DC, USA, pp 125–134

  37. Wang H, Wang F, Leong APY, Xu L, Chen X, Wang Q (2015) Precision insertion of percutaneous sacroiliac screws using a novel augmented reality-based navigation system: a pilot study. Int Orthop 40:1–7

    Google Scholar 

  38. Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, Liao H (2014) Augmented reality navigation with automatic marker-free image registration using 3-d image overlay for dental surgery. IEEE Trans Biomed Eng 61(4):1295–1304

    Article  PubMed  Google Scholar 

  39. Wilcoxon F, Katti S, Wilcox RA (1970) Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test. Sel Table Math Stat 1:171–259

    Google Scholar 

  40. Yoon JW, Chen RE, Han PK, Si P, Freeman WD, Pirris SM (2016) Technical feasibility and safety of an intraoperative head-up display device during spine instrumentation. Int J Med Robot Comput Assist Surg. doi:10.1002/rcs.1770

  41. Yoshida S, Kihara K, Takeshita H, Nakanishi Y, Kijima T, Ishioka J, Matsuoka Y, Numao N, Saito K, Fujii Y (2014) Head-mounted display for a personal integrated image monitoring system: ureteral stent placement. Urol Int 94(1):117–120

  42. Zhang X, Chen G, Liao H (2016) High quality see-through surgical guidance system using enhanced 3d autostereoscopic augmented reality. IEEE Trans Biomed Eng. doi:10.1109/TBME.2016.2624632

Download references

Author information

Author notes

    Authors and Affiliations

    1. Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA

      Long Qian, Alexander Barthel, Nassir Navab & Bernhard Fuerst

    2. Computer Aided Medical Procedures, Technische Universität München, Munich, Germany

      Alexander Barthel & Nassir Navab

    3. Department of Orthopaedic Surgery, Johns Hopkins Hospital, Baltimore, MD, USA

      Alex Johnson & Greg Osgood

    4. Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA

      Long Qian & Peter Kazanzides

    Authors
    1. Long Qian
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Alexander Barthel
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Alex Johnson
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Greg Osgood
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Peter Kazanzides
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Nassir Navab
      View author publications

      You can also search for this author in PubMed Google Scholar

    7. Bernhard Fuerst
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Long Qian.

    Ethics declarations

    Funding

    This study was funded by NIAMS of the National Institutes of Health under award number T32AR067708.

    Conflict of interest

    The authors declare that they have no conflict of interest.

    Ethical approval

    All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article contains a study with human participants, which was approved by the JHU Homewood Institutional Review Board under the numbers HIRB00003228 and HIRB00004665.

    Informed consent

    Informed consent was obtained from all individual participants included in the study.

    Additional information

    L. Qian and A. Barthel: contributed equally and should be considered joint first authors

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Qian, L., Barthel, A., Johnson, A. et al. Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display. Int J CARS 12, 901–910 (2017). https://doi.org/10.1007/s11548-017-1564-y

    Download citation

    • Received:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11548-017-1564-y

    Keywords

    Search

    Navigation