Skip to main content

Interactive Visualization and Analysis of Multimodal Datasets for Surgical Applications

Journal of Digital Imaging volume 25, pages 792–801 (2012)Cite this article

Abstract

Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Correa CD: Visualizing what lies inside. SIGGRAPH Comput Graph 43(2):1–6, 2009

    Article  Google Scholar 

  2. Luo H, Mittal R, Zheng X, Bielamowicz S, Walsh R, Hahn J: An immersed-boundary method for flow–structure interaction in biological systems with application to phonation. J Comput Phys 227(22):9303–9332, 2008

    PubMed  Article  Google Scholar 

  3. Perrin DP, Vasilyev NV, Novotny P, Stoll J, Howe RD, Dupont PE, et al: Image guided surgical interventions. Curr Probl Surg 46(9):730–766, 2009

    PubMed  Article  Google Scholar 

  4. Peters TM, Cleary K: Image-guided interventions: technology and applications. Springer, New York, 2008

    Book  Google Scholar 

  5. Peters TM: Image-guidance for surgical procedures. Phys Med Biol 14:R505, 2006

    Article  Google Scholar 

  6. Kruger J, Schneider J, Westermann R: ClearView: an interactive context preserving hotspot visualization technique. IEEE Trans Visual Comput Graph 12(5):941–948, 2006

    Article  Google Scholar 

  7. Svakhine N, Ebert DS, Stredney D: Illustration motifs for effective medical volume illustration. IEEE Comput Graph Appl 25(3):31–39, 2005

    PubMed  Article  Google Scholar 

  8. Bier EA, Stone MC, Pier K, Buxton W and DeRose TD: Toolglass and magic lenses: The see-through interface. In: Proceedings of the 20th annual conference on Computer graphics and interactive techniques, Anaheim, CA, 1993, pp 73–80

  9. Wang L, Zhao Y, Mueller K and Kaufman A: The magic volume lens: an interactive focus+context technique for volume rendering. pp 367–374

  10. Viega J, Conway MJ, Williams G and Pausch R: 3D magic lenses. In: Proceedings of the 9th annual ACM symposium on User interface software and technology, Seattle, Washington, United States, 1996

  11. Bruckner S and Gröller ME: VolumeShop: an interactive system for direct volume illustration. In: IEEE Visualization, 2005, pp 671–678

  12. Burger K, Kruger J, Westermann R: Direct Volume Editing. IEEE Trans Visual Comput Graph 14(6):1388–1395, 2008

    Article  Google Scholar 

  13. Kirmizibayrak C: Interactive volume visualization and editing methods for surgical applications. Ph.D. Dissertation, Department of Computer Science, The George Washington University, Washington, DC, 2011

  14. Zhang Z: Iterative point matching for registration of free-form curves and surfaces. Int J Comput Vision 13(2):119–152, 1994

    Article  Google Scholar 

  15. Gerasimov P, Fernando R and Green S: Shader Model 3.0: using vertex textures, white paper DA-01373-001_v00, NVIDIA, 2004

  16. Scheuermann T and Hensley: Efficient histogram generation using scattering on GPUs. In: Proceedings of the 2007 symposium on Interactive 3D graphics and games, Seattle, Washington, 2007, pp 33–37

  17. Li W, Fan Z, Wei X and Kaufman A: GPU-Based flow simulation with complex boundaries. GPU Gems II, 2003

  18. Yim Y, Wakid M, Kirmizibayrak C, Bielamowicz S and Hahn JK: Registration of 3D CT data to 2D endoscopic image using a gradient mutual information based viewpoint matching for image-guided Medialization Laryngoplasty, JCSE 4(4):368–387, 2010

  19. Studholme C, Hill DLG, Hawkes DJ: An overlap invariant entropy measure of 3D medical image alignment. Pattern Recogn 32(1):71–86, 1999

    Article  Google Scholar 

  20. Johnson R, O’Hara K, Sellen A, Cousins C and Criminisi A: Exploring the potential for touchless interaction in image-guided interventional radiology. In: Proceedings of the 2011 annual conference on Human factors in computing systems, Vancouver, BC, Canada, 2011, pp 3323–3332

  21. Kirmizibayrak C, Radeva N, Wakid M, Philbeck J, Sibert J and Hahn J: Evaluation of gesture based interfaces for medical volume visualization tasks. In: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, Hong Kong, China, 2011, pp 69–74

  22. Helferty JP, Sherbondy AJ, Kiraly AP, Higgins WE: Computer-based system for the virtual-endoscopic guidance of bronchoscopy. Comp Vis Image Understand 108(1–2):171–187, 2007

    Article  Google Scholar 

  23. OpenNI Organization. “OpenNI User Guide,” 05/01, 2011; http://www.openni.org/documentation

  24. Jin G, Baek N, Hahn JK, Bielamowicz S, Mittal R, Walsh R: Image guided medialization laryngoplasty. Computer Animation and Virtual Worlds 20(1):67–77, 2009

    PubMed  Article  Google Scholar 

  25. Bielamowicz S: Perspectives on medialization laryngoplasty. Otolaryngolog Clin North America 37(1):139, 2004

    Article  Google Scholar 

  26. Luo H, Mittal R and Bielamowicz SA: Analysis of flow-structure interaction in the larynx during phonation using an immersed-boundary method. J Acoust Soc Am vol. 126:no. 2, 2009

Download references

Acknowledgment

This work was supported by NIH grant R01 DC007125 to develop computer-based tools for medialization laryngoplasty. The authors would like to thank Dr. Steven Bielamowicz for his help and informative discussions for application of our methods to otolaryngology, and Dr. Rajat Mittal and Qian Xue for providing the CFD simulation datasets.

Author information

Authors and Affiliations

  1. Department of Computer Science, The George Washington University, Washington, DC, USA

    Can Kirmizibayrak, Yeny Yim, Mike Wakid & James Hahn

  2. Institute for Biomedical Engineering, The George Washington University, Washington, DC, USA

    Can Kirmizibayrak, Yeny Yim, Mike Wakid & James Hahn

  3. Department of Radiation Oncology, Stanford School of Medicine, 875 Blake Wilbur Drive, Stanford, CA, 94305, USA

    Can Kirmizibayrak

Authors
  1. Can Kirmizibayrak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yeny Yim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mike Wakid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James Hahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Can Kirmizibayrak.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kirmizibayrak, C., Yim, Y., Wakid, M. et al. Interactive Visualization and Analysis of Multimodal Datasets for Surgical Applications. J Digit Imaging 25, 792–801 (2012). https://doi.org/10.1007/s10278-012-9461-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-012-9461-y

Keywords

  • Volume visualization
  • Human–computer interaction
  • Volume rendering
  • Image-guided surgery