Journal of Digital Imaging

, Volume 31, Issue 1, pp 26–31 | Cite as

Workflow for Visualization of Neuroimaging Data with an Augmented Reality Device

  • Christof KarmonikEmail author
  • Timothy B. Boone
  • Rose Khavari


Commercial availability of three-dimensional (3D) augmented reality (AR) devices has increased interest in using this novel technology for visualizing neuroimaging data. Here, a technical workflow and algorithm for importing 3D surface-based segmentations derived from magnetic resonance imaging data into a head-mounted AR device is presented and illustrated on selected examples: the pial cortical surface of the human brain, fMRI BOLD maps, reconstructed white matter tracts, and a brain network of functional connectivity.


Augemented reality Functional magnetic resonance imaging 3D segmentation 

Supplementary material


(MP4 81.0 mb)


(MP4 87.9 mb)


  1. 1.
    Azuma RT: A survey of augmented reality. Presence Teleop Virt 6(4):355–385, 1997CrossRefGoogle Scholar
  2. 2.
    Milgram P, Kishino F: A taxonomy of mixed reality visual displays. IEIC Transactions on Information and Systems, E77D:12, 1321–29,1994Google Scholar
  3. 3.
    Abe Y, Sato S, Kato K et al.: A novel 3D guidance system using augmented reality for percutaneous vertebroplasty. J Neurosurg Spine 19(4):492–501, 2013CrossRefPubMedGoogle Scholar
  4. 4.
    Blackwell M, Morgan F, DiGioia, 3rd AM: Augmented reality and its future in orthopaedics. Clin Orthop Relat Res 354:111–122, 1998CrossRefGoogle Scholar
  5. 5.
    Kerner KF, et al.: Augmented reality for teaching endotracheal intubation: MR imaging to create anatomically correct models. AMIA Annu Symp Proc p. 888,2003Google Scholar
  6. 6.
    Nicolau S et al.: Augmented reality in laparoscopic surgical oncology. Surg Oncol 20(3):189–201, 2011CrossRefPubMedGoogle Scholar
  7. 7.
    Fritz J et al.: Augmented reality visualization using image overlay technology for MR-guided interventions: cadaveric bone biopsy at 1.5 T. Invest Radiol 48(6):464–470, 2013CrossRefPubMedGoogle Scholar
  8. 8.
    Volonte F et al.: Augmented reality to the rescue of the minimally invasive surgeon. The usefulness of the interposition of stereoscopic images in the Da Vinci robotic console. Int J Med Robot 9(3):e34–e38, 2013CrossRefPubMedGoogle Scholar
  9. 9.
    Markman A et al.: Augmented reality three-dimensional object visualization and recognition with axially distributed sensing. Opt Lett 41(2):297–300, 2016CrossRefPubMedGoogle Scholar
  10. 10.
    Chinnock C: Virtual reality in surgery and medicine. Hosp Technol Ser 13(18):1–48, 1994PubMedGoogle Scholar
  11. 11.
    Ota D et al.: Virtual reality in surgical education. Comput Biol Med 25(2):127–137, 1995CrossRefPubMedGoogle Scholar
  12. 12.
    Olofsson J et al.: Advanced 3D-visualization, including virtual reality, distributed by PCs, in brain research, clinical radiology and education. Stud Health Technol Inform 50:357–358, 1998PubMedGoogle Scholar
  13. 13.
    Webb G et al.: Virtual reality and interactive 3D as effective tools for medical training. Stud Health Technol Inform 94:392–394, 2003PubMedGoogle Scholar
  14. 14.
    Farber M et al.: Virtual reality simulator for the training of lumbar punctures. Methods Inf Med 48(5):493–501, 2009CrossRefPubMedGoogle Scholar
  15. 15.
    Clarke DB et al.: Virtual reality simulator: demonstrated use in neurosurgical oncology. Surg Innov 20(2):190–197, 2013CrossRefPubMedGoogle Scholar
  16. 16.
    Mi SH et al.: A 3D virtual reality simulator for training of minimally invasive surgery. Conf Proc IEEE Eng Med Biol Soc 2014:349–352, 2014PubMedGoogle Scholar
  17. 17.
    Khavari R et al.: Functional magnetic resonance imaging with concurrent urodynamic testing identifies brain structures involved in micturition cycle in patients with multiple sclerosis. J Urol 197:438–444, 2016CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Shy M et al.: Functional magnetic resonance imaging during urodynamic testing identifies brain structures initiating micturition. J Urol 192(4):1149–1154, 2014CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Bidgood, Jr WD, Horii SC: Introduction to the ACR-NEMA DICOM standard. Radiographics 12(2):345–355, 1992CrossRefPubMedGoogle Scholar
  20. 20.
    John NW et al.: MedX3D: standards enabled desktop medical 3D. Stud Health Technol Inform 132:189–194, 2008PubMedGoogle Scholar
  21. 21.
    Cox RW: AFNI: what a long strange trip it’s been. Neuroimage 62(2):743–747, 2012CrossRefPubMedGoogle Scholar
  22. 22.
    Larobina M, Murino L: Medical image file formats. J Digit Imaging 27(2):200–206, 2014CrossRefPubMedGoogle Scholar
  23. 23.
    Schneider CA, Rasband WS, Eliceiri KW: NIH image to ImageJ: 25 years of image analysis. Nat Methods 9(7):671–675, 2012CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Xie S et al.: DiffusionKit: a light one-stop solution for diffusion MRI data analysis. J Neurosci Methods 273:107–119, 2016CrossRefPubMedGoogle Scholar
  25. 25.
    Karmonik C et al.: Music listening modulates functional connectivity and information flow in the human brain. Brain Connect, 2016. doi:  10.1089/brain.2016.0428
  26. 26.
    Berlage T: Augmented-reality communication for diagnostic tasks in cardiology. IEEE Trans Inf Technol Biomed 2(3):169–173, 1998CrossRefPubMedGoogle Scholar
  27. 27.
    Sato Y et al.: Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization. IEEE Trans Med Imaging 17(5):681–693, 1998CrossRefPubMedGoogle Scholar
  28. 28.
    Kawamata T et al.: Endoscopic augmented reality navigation system for endonasal transsphenoidal surgery to treat pituitary tumors: technical note. Neurosurgery 50(6):1393–1397, 2002PubMedGoogle Scholar
  29. 29.
    Paul P, Fleig O, Jannin P: Augmented virtuality based on stereoscopic reconstruction in multimodal image-guided neurosurgery: methods and performance evaluation. IEEE Trans Med Imaging 24(11):1500–1511, 2005CrossRefPubMedGoogle Scholar
  30. 30.
    Lukosch S, Billinghurst M, Alem L et al.: The effect of view independence in a collaborative AR system. Computer supported cooperative work. J Collab Comput 24(6):563–589, 2015Google Scholar
  31. 31.
    Lukosch S, Billinghurst M, Alem L et al.: Collaboration in augmented reality. Computer supported cooperative work. J Collab Comput 24(6):515–525, 2015Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2017

Authors and Affiliations

  • Christof Karmonik
    • 1
    Email author
  • Timothy B. Boone
    • 2
  • Rose Khavari
    • 2
  1. 1.MRI coreHouston Methodist Hospital Research InstituteHoustonUSA
  2. 2.Department of UrologyHouston Methodist HospitalHoustonUSA

Personalised recommendations