Journal of Digital Imaging

, Volume 25, Issue 6, pp 792–801 | Cite as

Interactive Visualization and Analysis of Multimodal Datasets for Surgical Applications

  • Can Kirmizibayrak
  • Yeny Yim
  • Mike Wakid
  • James Hahn


Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.


Volume visualization Human–computer interaction Volume rendering Image-guided surgery 


  1. 1.
    Correa CD: Visualizing what lies inside. SIGGRAPH Comput Graph 43(2):1–6, 2009CrossRefGoogle Scholar
  2. 2.
    Luo H, Mittal R, Zheng X, Bielamowicz S, Walsh R, Hahn J: An immersed-boundary method for flow–structure interaction in biological systems with application to phonation. J Comput Phys 227(22):9303–9332, 2008PubMedCrossRefGoogle Scholar
  3. 3.
    Perrin DP, Vasilyev NV, Novotny P, Stoll J, Howe RD, Dupont PE, et al: Image guided surgical interventions. Curr Probl Surg 46(9):730–766, 2009PubMedCrossRefGoogle Scholar
  4. 4.
    Peters TM, Cleary K: Image-guided interventions: technology and applications. Springer, New York, 2008CrossRefGoogle Scholar
  5. 5.
    Peters TM: Image-guidance for surgical procedures. Phys Med Biol 14:R505, 2006CrossRefGoogle Scholar
  6. 6.
    Kruger J, Schneider J, Westermann R: ClearView: an interactive context preserving hotspot visualization technique. IEEE Trans Visual Comput Graph 12(5):941–948, 2006CrossRefGoogle Scholar
  7. 7.
    Svakhine N, Ebert DS, Stredney D: Illustration motifs for effective medical volume illustration. IEEE Comput Graph Appl 25(3):31–39, 2005PubMedCrossRefGoogle Scholar
  8. 8.
    Bier EA, Stone MC, Pier K, Buxton W and DeRose TD: Toolglass and magic lenses: The see-through interface. In: Proceedings of the 20th annual conference on Computer graphics and interactive techniques, Anaheim, CA, 1993, pp 73–80Google Scholar
  9. 9.
    Wang L, Zhao Y, Mueller K and Kaufman A: The magic volume lens: an interactive focus+context technique for volume rendering. pp 367–374Google Scholar
  10. 10.
    Viega J, Conway MJ, Williams G and Pausch R: 3D magic lenses. In: Proceedings of the 9th annual ACM symposium on User interface software and technology, Seattle, Washington, United States, 1996Google Scholar
  11. 11.
    Bruckner S and Gröller ME: VolumeShop: an interactive system for direct volume illustration. In: IEEE Visualization, 2005, pp 671–678Google Scholar
  12. 12.
    Burger K, Kruger J, Westermann R: Direct Volume Editing. IEEE Trans Visual Comput Graph 14(6):1388–1395, 2008CrossRefGoogle Scholar
  13. 13.
    Kirmizibayrak C: Interactive volume visualization and editing methods for surgical applications. Ph.D. Dissertation, Department of Computer Science, The George Washington University, Washington, DC, 2011Google Scholar
  14. 14.
    Zhang Z: Iterative point matching for registration of free-form curves and surfaces. Int J Comput Vision 13(2):119–152, 1994CrossRefGoogle Scholar
  15. 15.
    Gerasimov P, Fernando R and Green S: Shader Model 3.0: using vertex textures, white paper DA-01373-001_v00, NVIDIA, 2004Google Scholar
  16. 16.
    Scheuermann T and Hensley: Efficient histogram generation using scattering on GPUs. In: Proceedings of the 2007 symposium on Interactive 3D graphics and games, Seattle, Washington, 2007, pp 33–37Google Scholar
  17. 17.
    Li W, Fan Z, Wei X and Kaufman A: GPU-Based flow simulation with complex boundaries. GPU Gems II, 2003Google Scholar
  18. 18.
    Yim Y, Wakid M, Kirmizibayrak C, Bielamowicz S and Hahn JK: Registration of 3D CT data to 2D endoscopic image using a gradient mutual information based viewpoint matching for image-guided Medialization Laryngoplasty, JCSE 4(4):368–387, 2010Google Scholar
  19. 19.
    Studholme C, Hill DLG, Hawkes DJ: An overlap invariant entropy measure of 3D medical image alignment. Pattern Recogn 32(1):71–86, 1999CrossRefGoogle Scholar
  20. 20.
    Johnson R, O’Hara K, Sellen A, Cousins C and Criminisi A: Exploring the potential for touchless interaction in image-guided interventional radiology. In: Proceedings of the 2011 annual conference on Human factors in computing systems, Vancouver, BC, Canada, 2011, pp 3323–3332Google Scholar
  21. 21.
    Kirmizibayrak C, Radeva N, Wakid M, Philbeck J, Sibert J and Hahn J: Evaluation of gesture based interfaces for medical volume visualization tasks. In: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, Hong Kong, China, 2011, pp 69–74Google Scholar
  22. 22.
    Helferty JP, Sherbondy AJ, Kiraly AP, Higgins WE: Computer-based system for the virtual-endoscopic guidance of bronchoscopy. Comp Vis Image Understand 108(1–2):171–187, 2007CrossRefGoogle Scholar
  23. 23.
    OpenNI Organization. “OpenNI User Guide,” 05/01, 2011;
  24. 24.
    Jin G, Baek N, Hahn JK, Bielamowicz S, Mittal R, Walsh R: Image guided medialization laryngoplasty. Computer Animation and Virtual Worlds 20(1):67–77, 2009PubMedCrossRefGoogle Scholar
  25. 25.
    Bielamowicz S: Perspectives on medialization laryngoplasty. Otolaryngolog Clin North America 37(1):139, 2004CrossRefGoogle Scholar
  26. 26.
    Luo H, Mittal R and Bielamowicz SA: Analysis of flow-structure interaction in the larynx during phonation using an immersed-boundary method. J Acoust Soc Am vol. 126:no. 2, 2009Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2012

Authors and Affiliations

  • Can Kirmizibayrak
    • 1
    • 2
    • 3
  • Yeny Yim
    • 1
    • 2
  • Mike Wakid
    • 1
    • 2
  • James Hahn
    • 1
    • 2
  1. 1.Department of Computer ScienceThe George Washington UniversityWashingtonUSA
  2. 2.Institute for Biomedical EngineeringThe George Washington UniversityWashingtonUSA
  3. 3.Department of Radiation OncologyStanford School of MedicineStanfordUSA

Personalised recommendations