Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

  • Long Qian
  • Alexander Barthel
  • Alex Johnson
  • Greg Osgood
  • Peter Kazanzides
  • Nassir Navab
  • Bernhard Fuerst
Original Article

Abstract

Purpose

Optical see-through head-mounted displays (OST-HMD) feature an unhindered and instantaneous view of the surgery site and can enable a mixed reality experience for surgeons during procedures. In this paper, we present a systematic approach to identify the criteria for evaluation of OST-HMD technologies for specific clinical scenarios, which benefit from using an object-anchored 2D-display visualizing medical information.

Methods

Criteria for evaluating the performance of OST-HMDs for visualization of medical information and its usage are identified and proposed. These include text readability, contrast perception, task load, frame rate, and system lag. We choose to compare three commercially available OST-HMDs, which are representatives of currently available head-mounted display technologies. A multi-user study and an offline experiment are conducted to evaluate their performance.

Results

Statistical analysis demonstrates that Microsoft HoloLens performs best among the three tested OST-HMDs, in terms of contrast perception, task load, and frame rate, while ODG R-7 offers similar text readability. The integration of indoor localization and fiducial tracking on the HoloLens provides significantly less system lag in a relatively motionless scenario.

Conclusions

With ever more OST-HMDs appearing on the market, the proposed criteria could be used in the evaluation of their suitability for mixed reality surgical intervention. Currently, Microsoft HoloLens may be more suitable than ODG R-7 and Epson Moverio BT-200 for clinical usability in terms of the evaluated criteria. To the best of our knowledge, this is the first paper that presents a methodology and conducts experiments to evaluate and compare OST-HMDs for their use as object-anchored 2D-display during interventions.

Keywords

Mixed reality Intervention Optical see-through head-mounted display User study 

Notes

Compliance with ethical standards

Funding

This study was funded by NIAMS of the National Institutes of Health under award number T32AR067708.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article contains a study with human participants, which was approved by the JHU Homewood Institutional Review Board under the numbers HIRB00003228 and HIRB00004665.

Informed consent

Informed consent was obtained from all individual participants included in the study.

References

  1. 1.
    Abe Y, Sato S, Kato K, Hyakumachi T, Yanagibashi Y, Ito M, Abumi K (2013) A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. J Neurosurg Spine 19(4):492–501CrossRefPubMedGoogle Scholar
  2. 2.
    Armstrong DG, Rankin TM, Giovinco NA, Mills JL, Matsuoka Y (2014) A heads-up display for diabetic limb salvage surgery a view through the google looking glass. J Diabetes Sci Technol 8:951–956CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Azimi E, Doswell J, Kazanzides P (2012) Augmented reality goggles with an integrated tracking system for navigation in neurosurgery. In: 2012 IEEE Virtual Reality Workshops (VRW), pp 123–124. doi: 10.1109/VR.2012.6180913
  4. 4.
    Badiali G, Ferrari V, Cutolo F, Freschi C, Caramella D, Bianchi A, Marchetti C (2014) Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J Craniomaxillofacial Surg 42(8):1970–1976CrossRefGoogle Scholar
  5. 5.
    Bajura M, Fuchs H, Ohbuchi R (1992) Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. SIGGRAPH 26(2):203–210. doi: 10.1145/142920.134061
  6. 6.
    Bichlmeier C, Ockert B, Heining SM, Ahmadi A, Navab N (2008) Stepping into the operating theater: ARAV—augmented reality aided vertebroplasty. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. ISMAR ’08, pp 165–166. IEEE Computer Society, Washington, DC. doi: 10.1109/ISMAR.2008.4637348
  7. 7.
    Chen X, Xu L, Wang Y, Wang H, Wang F, Zeng X, Wang Q, Egger J (2015) Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J Biomed Inform 55:124–131CrossRefPubMedGoogle Scholar
  8. 8.
    Cutolo F, Parchi PD, Ferrari V (2014) Video see through AR head-mounted display for medical procedures. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), p 393–396. doi: 10.1109/ISMAR.2014.6948504
  9. 9.
    Hart SG (2006) NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50. Sage Publications, pp 904–908. doi: 10.1177/154193120605000909
  10. 10.
    Herron D, Lantis Ii J, Maykel J, Basu C, Schwaitzberg S (1999) The 3-D monitor and head-mounted display. Surg Endosc 13(8):751–755CrossRefPubMedGoogle Scholar
  11. 11.
    Hoffman DM, Girshick AR, Akeley K, Banks MS (2008) Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. J Vision 8(3):33–33CrossRefGoogle Scholar
  12. 12.
    Itoh Y, Amano T, Iwai D, Klinker G (2016) Gaussian light field: estimation of viewpoint-dependent blur for optical see-through head-mounted displays. IEEE Trans Vis Comput Graph 22(11):2368–2376CrossRefPubMedGoogle Scholar
  13. 13.
    Janin AL, Mizell DW, Caudell TP (1993) Calibration of head-mounted displays for augmented reality applications. In: Proceedings of VR annual international symposium, p 246–255. doi: 10.1109/VRAIS.1993.380772
  14. 14.
    Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Graph 18(2):332–352CrossRefPubMedGoogle Scholar
  15. 15.
    Kersten-Oertel M, Jannin P, Collins DL (2013) The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph 37(2):98–112CrossRefPubMedGoogle Scholar
  16. 16.
    Koesveld J, Tetteroo G, Graaf E (2003) Use of head-mounted display in transanal endoscopic microsurgery. Surg Endosc 17(6):943–946CrossRefPubMedGoogle Scholar
  17. 17.
    Kramida G (2016) Resolving the vergence-accommodation conflict in head-mounted displays. IEEE Trans Vis Comput Graph 22(7):1912–1931CrossRefPubMedGoogle Scholar
  18. 18.
    Kress B, Starner T (2013) A review of head-mounted displays (HMD) technologies and applications for consumer electronics. Proc SPIE 8720:87200A–87200A-13. doi: 10.1117/12.2015654
  19. 19.
    Liao H, Inomata T, Sakuma I, Dohi T (2010) Three-dimensional augmented reality for mriguided surgery using integral videography auto stereoscopic-image overlay. IEEE Trans Biomed Eng 57(6):1476–1486CrossRefPubMedGoogle Scholar
  20. 20.
    Maithel S, Villegas L, Stylopoulos N, Dawson S, Jones D (2005) Simulated laparoscopy using a head-mounted display vs traditional video monitor: an assessment of performance and muscle fatigue. Surg Endosc 19(3):406–411CrossRefPubMedGoogle Scholar
  21. 21.
    Martin-Gonzalez A, Heining SM, Navab N (2009) Head-mounted virtual loupe with sight-based activation for surgical applications. In: 2009 8th IEEE international symposium on mixed and augmented reality, pp 207–208Google Scholar
  22. 22.
    Mentler T, Wolters C, Herczeg M (2015) Use cases and usability challenges for head-mounted displays in healthcare. Curr Dir Biomed Eng 1(1):534–537Google Scholar
  23. 23.
    Milgram P, Colquhoun H (1999) A taxonomy of real and virtual world display integration. Mixed Real Merg Real Virtual World 1:1–26Google Scholar
  24. 24.
    Morisawa T, Kida H, Kusumi F, Okinaga S, Ohana M (2015) Endoscopic submucosal dissection using head-mounted display. Gastroenterology 149(2):290–291CrossRefPubMedGoogle Scholar
  25. 25.
    Ortega G, Wolff A, Baumgaertner M, Kendoff D (2008) Usefulness of a head mounted monitor device for viewing intraoperative fluoroscopy during orthopaedic procedures. Arch Orthop Trauma Surg 128(10):1123–1126CrossRefPubMedGoogle Scholar
  26. 26.
    Qian L, Winkler A, Fuerst B, Kazanzides P, Navab N (2016) Reduction of interaction space in single point active alignment method for optical see-through head-mounted display calibration. In: 2016 IEEE international symposium on mixed and augmented reality (ISMAR-Adjunct), pp 156–157Google Scholar
  27. 27.
    Queisner M (2016) Medical screen operations: how head-mounted displays transform action and perception in surgical practice. Mediat 6(1):30–51Google Scholar
  28. 28.
    Rolland JP, Fuchs H (2000) Optical versus video see-through head-mounted displays in medical visualization. Presence Teleoperator Virtual Environ 9(3):287–309CrossRefGoogle Scholar
  29. 29.
    Sadda P, Azimi E, Jallo G, Doswell J, Kazanzides P (2012) Surgical navigation with a head-mounted tracking system and display. Stud Health Technol Inform 184:363–369Google Scholar
  30. 30.
    Shuhaiber JH (2004) Augmented reality in surgery. Arch Surg 139(2):170–174CrossRefPubMedGoogle Scholar
  31. 31.
    Suthau T, Vetter M, Hassenpflug P, Meinzer HP, Hellwich O (2002) A concept work for augmented reality visualisation based on a medical application in liver surgery. ISPRS Arch 34(5):274–280Google Scholar
  32. 32.
    Sutherland IE (1968) A head-mounted three dimensional display. In: Proceedings of the fall joint computer conference, part I, ACM, p 757–764, 9–11 December 1968Google Scholar
  33. 33.
    Thomsen MN, Lang RD (2004) An experimental comparison of 3-dimensional and 2-dimensional endoscopic systems in a model. Arthroscopy 20(4):419–423CrossRefPubMedGoogle Scholar
  34. 34.
    Thrun S, Leonard JJ (2008) Simultaneous localization and mapping. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Berlin, pp 871–889Google Scholar
  35. 35.
    Tuceryan M, Navab N (2000) Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. In: Proceedings IEEE and ACM international symposium on augmented reality (ISAR 2000), pp 149–158Google Scholar
  36. 36.
    Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose Tracking from natural features on mobile phones. In: Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality, Washington, DC, USA, pp 125–134Google Scholar
  37. 37.
    Wang H, Wang F, Leong APY, Xu L, Chen X, Wang Q (2015) Precision insertion of percutaneous sacroiliac screws using a novel augmented reality-based navigation system: a pilot study. Int Orthop 40:1–7Google Scholar
  38. 38.
    Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, Liao H (2014) Augmented reality navigation with automatic marker-free image registration using 3-d image overlay for dental surgery. IEEE Trans Biomed Eng 61(4):1295–1304CrossRefPubMedGoogle Scholar
  39. 39.
    Wilcoxon F, Katti S, Wilcox RA (1970) Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test. Sel Table Math Stat 1:171–259Google Scholar
  40. 40.
    Yoon JW, Chen RE, Han PK, Si P, Freeman WD, Pirris SM (2016) Technical feasibility and safety of an intraoperative head-up display device during spine instrumentation. Int J Med Robot Comput Assist Surg. doi: 10.1002/rcs.1770
  41. 41.
    Yoshida S, Kihara K, Takeshita H, Nakanishi Y, Kijima T, Ishioka J, Matsuoka Y, Numao N, Saito K, Fujii Y (2014) Head-mounted display for a personal integrated image monitoring system: ureteral stent placement. Urol Int 94(1):117–120Google Scholar
  42. 42.
    Zhang X, Chen G, Liao H (2016) High quality see-through surgical guidance system using enhanced 3d autostereoscopic augmented reality. IEEE Trans Biomed Eng. doi: 10.1109/TBME.2016.2624632

Copyright information

© CARS 2017

Authors and Affiliations

  • Long Qian
    • 1
    • 4
  • Alexander Barthel
    • 1
    • 2
  • Alex Johnson
    • 3
  • Greg Osgood
    • 3
  • Peter Kazanzides
    • 4
  • Nassir Navab
    • 1
    • 2
  • Bernhard Fuerst
    • 1
  1. 1.Computer Aided Medical ProceduresJohns Hopkins UniversityBaltimoreUSA
  2. 2.Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
  3. 3.Department of Orthopaedic SurgeryJohns Hopkins HospitalBaltimoreUSA
  4. 4.Laboratory for Computational Sensing and RoboticsJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations