The Ultrasound Visualization Pipeline

  • Åsmund Birkeland
  • Veronika Šoltészová
  • Dieter Hönigmann
  • Odd Helge Gilja
  • Svein Brekke
  • Timo Ropinski
  • Ivan Viola
Chapter
Part of the Mathematics and Visualization book series (MATHVISUAL)

Abstract

Radiology is one of the main tools in modern medicine. A numerous set of deceases, ailments and treatments utilize accurate images of the patient. Ultrasound is one of the most frequently used imaging modality in medicine. The high spatial resolution, its interactive nature and non-invasiveness makes it the first choice in many examinations. Image interpretation is one of ultrasound’s main challenges. Much training is required to obtain a confident skill level in ultrasound-based diagnostics. State-of-the-art graphics techniques is needed to provide meaningful visualizations of ultrasound in real-time. In this paper we present the process-pipeline for ultrasound visualization, including an overview of the tasks performed in the specific steps. To provide an insight into the trends of ultrasound visualization research, we have selected a set of significant publications and divided them into a technique-based taxonomy covering the topics pre-processing, segmentation, registration, rendering and augmented reality. For the different technique types we discuss the difference between ultrasound-based techniques and techniques for other modalities.

Keywords

Ultrasound Bio-medical visualization 

Notes

Acknowledgments

This work has been carried out within the IllustraSound research project (#193170), which is funded by the VERDIKT program of the Norwegian Research Council with support of the MedViz network in Bergen, Norway (PK1760-5897-Project 11). We would also like to thank Helwig Hauser for invaluable help and fruitful discussions.

References

  1. 1.
    Ag, K.: 3d ultrasound: a dedicated system. Eur. Radiol. 9, S331–S333 (1999). doi: 10.1007/PL00014067 CrossRefGoogle Scholar
  2. 2.
    Allen, R.C., Rubin, M.L.: Chromostereopsis. Surv. Ophthalmology 26(1), 22–27 (1981)CrossRefGoogle Scholar
  3. 3.
    Angelelli, P., Viola, I., Nylund, K., Gilja, O.H., Hauser, H.: Guided visualization of ultrasound image sequences. In: Proceedings of Visual Computing for Biomedicine, pp. 125–132 (2010)Google Scholar
  4. 4.
    Bajura, M., Fuchs, H., Ohbuchi, R.: Merging virtual objects with the real world: seeing ultrasound imagery within the patient. Proc. ACM SIGGRAPH 26(2), 203–210 (1992)CrossRefGoogle Scholar
  5. 5.
    Belohlavek, M., Dutt, V., Greenleaf, J., Foley, D.A., Gerber, T.C., Seward, J.B.: Multidimensional ultrasonic visualization in cardiology. Proc. IEEE Ultrason. Symp. 2, 1137–1145 (1992)Google Scholar
  6. 6.
    Berg, S., Torp, H., Martens, D., Steen, E., Samstad, S., Hik, I., Olstad, B.: Dynamic three-dimensional freehand echocardiography using raw digital ultrasound data. Ultrasound Med. Biol. 25(5), 745–753 (1999)CrossRefGoogle Scholar
  7. 7.
    Burns, M., Haidacher, M., Wein, W., Viola, I., Gröller, E.: Feature emphasis and contextual cutaways for multimodal medical visualization. In: Proceedings of Eurographics/IEEE VGTC Symposium on Visualization (EuroVis 2007), pp. 275–282 (2007)Google Scholar
  8. 8.
    Carneiro, G., Amat, F., Georgescu, B., Good, S., Comaniciu, D.: Semantic-based indexing of fetal anatomies from 3-D ultrasound data using global/semi-local context and sequential sampling. In: Proceedings of Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  9. 9.
    Chen, Y., Yin, R., Flynn, P., Broschat, S.: Aggressive region growing for speckle reduction in ultrasound images. Pattern Recogn. Lett. 24(4–5), 677–691 (2003)CrossRefGoogle Scholar
  10. 10.
    Crimmins, T.R.: The geometric filter for speckle reduction. Appl. Optics 24(10), 1438–1443 (1985)CrossRefGoogle Scholar
  11. 11.
    Curiel, L., Chopra, R., Hynynen, K.: Progress in multimodality imaging: truly simultaneous ultrasound and magnetic resonance imaging. IEEE Trans. Med. Imaging 26(12), 1740–1746 (2007)CrossRefGoogle Scholar
  12. 12.
    Deprez, J., Brusseau, E., Schmitt, C., Cloutier, G., Basset, O.: 3d estimation of soft biological tissue deformation from radio-frequency ultrasound volume acquisitions. Med. Image Anal. 13(1), 116–127 (2009)CrossRefGoogle Scholar
  13. 13.
    Deutch, T.D., Joergner, I., Matson, D.O., Oehninger, S., Bocca, S., Hönigmann, D., Abuhamad, A.: Automated assessment of ovarian follicles using a novel three-dimensional ultrasound software. Fertility Steril. 92(5), 1562–1568 (2009)CrossRefGoogle Scholar
  14. 14.
    Einthoven, W.: Stereoscopie durch Farbendifferenz. Von Graefes. Archives 19, 211–238 (1885)Google Scholar
  15. 15.
    Fattal, R., Lischinski, D.: Variational classification for visualization of 3D ultrasound data. In: Proceedings of IEEE Visualization, pp. 403–410 (2001)Google Scholar
  16. 16.
    Forsberg, F., Healey, A.J., Leeman, S., Jensen, J.A.: Assessment of hybrid speckle reduction algorithms. Phys. Med. Biol. 36(11), 1539–1549 (1991)CrossRefGoogle Scholar
  17. 17.
    Fuller, D.B., Jin, H., Koziol, J.A., Feng, A.C.: CT-ultrasound fusion prostate brachytherapy: a dynamic dosimetry feedback and improvement method. A report of 54 consecutive cases. Brachytherapy 4(3), 207–216 (2005)CrossRefGoogle Scholar
  18. 18.
    Garrett, W.F., Fuchs, H., Whitton, M.C., State, A.: Real-time incremental visualization of dynamic ultrasound volumes using parallel BSP trees. In: Proceedings of IEEE Visualization, pp. 235–240 (1996)Google Scholar
  19. 19.
    GE Healthcare: A closer look at GE’s pocket-sized vscan ultrasound. http://www.gereports.com/a-closer-look-at-ges-pocket-sized-vscan-ultrasound/. Accessed 13 Sept 2010
  20. 20.
    Gee, A., Prager, R., Treece, G., Berman, L.: Narrow-band volume rendering for freehand 3D ultrasound. Comput. Graph. 26(3), 463–476 (2002)CrossRefGoogle Scholar
  21. 21.
    Gee, A., Prager, R., Treece, G., Cash, C., Berman, L.: Processing and visualizing three-dimensional ultrasound data. Br. J. Radiol. 77, 186–193 (2004)CrossRefGoogle Scholar
  22. 22.
    Gilja, O.H., Hatlebakk, J.G., Ødegaard, S., Berstad, A., Viola, I., Giertsen, C., Hausken, T., Gregersen, H.: Advanced imaging and visualization in gastrointestinal disorders. World J. Gastroenterol. 13, 1408–1421 (2007)Google Scholar
  23. 23.
    Gilja, O.H., Hausken, T., Ødegaard, S., Wendelbo, Ø., Thierley, M.: Mobile ultrasonography in a medical department. Tidskr. Nor. Lægeforen. 19, 270–285 (2003)Google Scholar
  24. 24.
    Gilja, O.H., Heimdal, A., Hausken, T., Gregersen, H., Matre, K., Berstad, A., Ødegaard, S.: Strain during gastric contractions can be measured using doppler ultrasonography. Ultrasound Med. Biol. 28(11–12), 1457–1465 (2002)CrossRefGoogle Scholar
  25. 25.
    Gobbi, D., Peters, T.: Interactive intra-operative 3D ultrasound reconstruction and visualization. In: Proceedings of Medical Image Computing and Computer-Assisted Intervention, pp. 156–163 (2002)Google Scholar
  26. 26.
    Gupta, N., Swamy, M.N.S., Plotkin, E.: Despeckling of medical ultrasound images using data and rate adaptive lossy compression. IEEE Trans. Med. Imaging 24(6), 743–754 (2005)CrossRefGoogle Scholar
  27. 27.
    Heimdal, A., Sten, A., Torp, H., Skje, T.: Real-time strain rate imaging of the left ventricle by ultrasound. J. Am. Soc. Echocardiogr. 11(11), 1013–1019 (1998)CrossRefGoogle Scholar
  28. 28.
    Hönigmann, D., Ruisz, J., Haider, C.: Adaptive design of a global opacity transfer function for direct volume rendering of ultrasound data. In: IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, pp. 489–496 (2003)Google Scholar
  29. 29.
    Huang, T., Yang, G., Tang, G.: A fast two-dimensional median filtering algorithm. IEEE Trans. Acoust. Speech Signal Process. 27(1), 13–18 (1979)CrossRefGoogle Scholar
  30. 30.
    Jan, U., Sandker, D., Rauberger, M., Matthies, H.K., Overhoff, H.M.: Enhanced visualization of ultrasound volumes for diagnostic and therapeutic purposes. In: Proceedings of 4th European Conference of the International Federation for Medical and Biological Engineering, IFMBE Proceedings, pp. 689–692, IFMBE (2009)Google Scholar
  31. 31.
    Jones, M.G., Shipley, J.A., Robinson, T.M.: Visualisation of 4-D colour and power Doppler data. Ultrasound Med. Biol. 29(12), 1735–1747 (2003)CrossRefGoogle Scholar
  32. 32.
    Karadayi, K., Managuli, R., Kim, Y.: Three-dimensional ultrasound: from acquisition to visualization and from algorithms to systems. IEEE Rev. Biomed. Eng. 2, 23–39 (2009)CrossRefGoogle Scholar
  33. 33.
    Karamalis, A., Wein, W., Kutter, O., Navab, N.: Fast hybrid freehand ultrasound volume reconstruction. In: Proceedings of SPIE Medical Imaging. vol. 7261, pp. 726114–726118 (2009)Google Scholar
  34. 34.
    Kindlmann, G., Durkin, J.W.: Semi-automatic generation of transfer functions for direct volume rendering. In: Proceedings of IEEE Symposium on Volume Visualization, pp. 79–86 (1998)Google Scholar
  35. 35.
    King, A.P., Ma, Y., Yao, C., Jansen, C., Razavi, R., Rhode, K.S., Penney, G.P.: Image-to-physical registration for image-guided interventions using 3-d ultrasound and an ultrasound imaging model. In: Proceedings of International Conference on Information Processing in Medical Imaging, pp. 188–201. Springer, Berlin (2009)Google Scholar
  36. 36.
    Kniss, J., Kindlmann, G., Hansen, C.: Multidimensional transfer functions for interactive volume rendering. Trans. Visual. Comput. Graph. 8(3), 270–285 (2002)CrossRefGoogle Scholar
  37. 37.
    Kniss, J., Premoze, S., Hansen, C., Ebert, D.: Interactive translucent volume rendering and procedural modeling. In: Proceedings of IEEE Visualization, pp. 109–116 (2002)Google Scholar
  38. 38.
    Krissian, K., Westin, C.F., Kikinis, R., Vosburgh, K.G.: Oriented speckle reducing anisotropic diffusion. IEEE Trans. Image Process. 16(5), 1412–1424 (2007)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Kuo, J., Bredthauer, G.R., Castellucci, J.B., von Ramm, O.T.: Interactive volume rendering of real-time three-dimensional ultrasound images. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 54(2), 313–318 (2007)CrossRefGoogle Scholar
  40. 40.
    Lange, T., Papenberg, N., Heldmann, S., Modersitzki, J., Fischer, B., Lamecker, H., Schlag, P.: 3D ultrasound-CT registration of the liver using combined landmark-intensity information. Int. J. Comput. Assist. Radiol. Surg. 4, 79–88 (2008)CrossRefGoogle Scholar
  41. 41.
    Lee, T., Chaudhuri, A., Porikli, F., Shen, H.: Cyclestack: Inferring periodic behavior via temporal sequence visualization in ultrasound video. In: Proceedings of IEEE Pacific Visualization, pp. 89–96, IEEE (2010)Google Scholar
  42. 42.
    Levoy, M.: Display of surfaces from volume data. IEEE Comput. Graph. Appl. 8(3), 29–37 (1988)CrossRefGoogle Scholar
  43. 43.
    Lim, S., Kwon, K., Shin, B.S.: GPU-based interactive visualization framework for ultrasound datasets. Computer Anim. Virtual Worlds 20(1), 11–23 (2008)CrossRefGoogle Scholar
  44. 44.
    Lindemann, F., Ropinski, T.: Advanced light material interaction for direct volume rendering. In: Proceedings of IEEE/EG International Symposium on Volume Graphics, pp. 101–108 (2010)Google Scholar
  45. 45.
    Lizzi, F.L., Feleppa, E.J.: Image processing and pre-processing for medical ultrasound. In: Proceedings of Imagery Pattern Recognition Workshop, p. 187 (2000)Google Scholar
  46. 46.
    Nelson, T.R., Elvins, T.T.: Visualization of 3D ultrasound data. IEEE Comput. Graph. Appl. 13(6), 50–57 (1993)CrossRefGoogle Scholar
  47. 47.
    Nikas, D.C., Hartov, A., Lunn, K., Rick, K., Paulsen, K., Roberts, D.W.: Coregistered intraoperative ultrasonography in resection of malignant glioma. Neurosurg. Focus 14(2), e6 (2003)CrossRefGoogle Scholar
  48. 48.
    Noble, J.A., Boukerroui, D.: Ultrasound image segmentation: a survey. IEEE Trans. Med. Imaging 25(8), 987–1010 (2006)CrossRefGoogle Scholar
  49. 49.
    Ødegaard, S., Gilja, O.H., Gregersen, H.: Basic and New Aspects of Gastrointestinal Ultrasonography. Advanced Series in Biomechanics. World Scientific, Singapore (2005)Google Scholar
  50. 50.
    Ødegaard, S., Nesje, L.B., Gilja, O.H.: Atlas of Endoscopic Ultrasonography. Fagbokforlaget, Bergen (2007)Google Scholar
  51. 51.
    Papenberg, N., Lange, T., Modersitzki, J., Schlag, P.M., Fischer, B.: Image registration for CT and intra-operative ultrasound data of the liver. In: SPIE Medical Imaging, vol. 6918, p. 691808. SPIE (2008)Google Scholar
  52. 52.
    Penney, G., Blackall, J., Hamady, M., Sabharwal, Y., Adam, A., Hawkes, D.: Registration of freehand 3D ultrasound and magnetic resonance liver images. Med. Image Anal. 8(1), 81–91 (2004)CrossRefGoogle Scholar
  53. 53.
    Penney, G.P., Barratt, D.C., Chan, C.S.K., Slomczykowski, M., Carter, T.J., Edwards, P.J., Hawkes, D.J.: Cadaver validation of intensity-based ultrasound to CT registration. Med. Image Anal. 10(3), 385–395 (2006)CrossRefGoogle Scholar
  54. 54.
    Petersch, B., Hadwiger, M., Hauser, H., Hönigmann, D.: Real time computation and temporal coherence of opacity transfer functions for direct volume rendering of ultrasound data. Comput. Med. Imaging Graph. 29(1), 53–63 (2005)CrossRefGoogle Scholar
  55. 55.
    Petersch, B., Hönigmann, D.: Blood flow in its context: combining 3D B-mode and color Doppler ultrasonic data. Trans. Visual. Comput. Graph. 13(4), 748–757 (2007)CrossRefGoogle Scholar
  56. 56.
    Petersch, B., Serrano-Serrano, O., Hönigmann, D.: 3D soft segmentation and visualization of medical data based on nonlinear diffusion and distance functions. In: Proceesings of Eurographics/IEEE VGTC Symposium on Visualization (Eurovis 2006), pp. 331–338 (2006)Google Scholar
  57. 57.
    Postema, M., Gilja, O.H.: Ultrasound-directed drug delivery. Curr. Pharm. Biotechnol. 8, 355–361(7) (2007)Google Scholar
  58. 58.
    Prager, R., Gee, A., Treece, G., Berman, L.: Freehand 3D ultrasound without voxels: volume measurement and visualisation using the Stradx system. Ultrasonics 40(1–8), 109–115 (2002)CrossRefGoogle Scholar
  59. 59.
    Prager, R.W., Gee, A., Berman, L.: Stradx: real-time acquisition and visualization of freehand three-dimensional ultrasound. Med. Image Anal. 3(2), 129–140 (1999)CrossRefGoogle Scholar
  60. 60.
    Rohling, R., Gee, A., Berman, L.: A comparison of freehand three-dimensional ultrasound reconstruction techniques. Med. Image Anal. 3(4), 339–359 (1999)CrossRefGoogle Scholar
  61. 61.
    Rohling, R., Gee, A., Berman, L., Treece, G.: Radial basis function interpolation for 3D ultrasound. Lect. Notes Comput. Sci. 1613, 478–483 (1999)CrossRefGoogle Scholar
  62. 62.
    Ropinski, T., Döring, C., Rezk-Salama, C.: Interactive volumetric lighting simulating scattering and shadowing. In: Proceedings of IEEE Pacific Visualization, pp. 169–176 (2010)Google Scholar
  63. 63.
    Roxborough, T., Nielson, G.M.: Tetrahedron based, least squares, progressive volume models with application to freehand ultrasound data. In: Proceedings of IEEE Visualization, pp. 93–100 (2000)Google Scholar
  64. 64.
    Sakas, G., Schreyer, L., Grimm, M.: Preprocessing and volume rendering of 3D ultrasonic data. IEEE Comput. Graph. Appl. 15(4), 47–54 (1995)CrossRefGoogle Scholar
  65. 65.
    Sato, Y., Nakamoto, M., Tamaki, Y., Sasama, T., Sakita, I., Nakajima, Y., Monden, M., Tamura, S.: Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization. IEEE Trans. Med. Imaging 17(5), 681–693 (1998)CrossRefGoogle Scholar
  66. 66.
    Schott, M., Pegoraro, V., Hansen, C., Boulanger, K., Bouatouch, K.: A directional occlusion shading model for interactive direct volume rendering. In: Proceedings of Eurographics, pp. 855–862 (2009)Google Scholar
  67. 67.
    Shankar, P.M.: Speckle reduction in ultrasonic images through a maximum likelihood based adaptive filter. Phys. Med. Biol. 51(21), 5591–5602 (2006)CrossRefGoogle Scholar
  68. 68.
    Shelton, D., Stetten, G.D., Chang, W.: Ultrasound visualization with the sonic flashlight. In: Proceedings of ACM SIGGRAPH, p. 82 (2002)Google Scholar
  69. 69.
    Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical displays: a literature review of augmented reality. J. Disp. Technol. 4(4), 451–467 (2008)CrossRefGoogle Scholar
  70. 70.
    Solberg, O.V., Lindseth, F., Torp, H., Blake, R.E., Hernes, T.A.N.: Freehand 3d ultrasound reconstruction algorithms: a review. Ultrasound Med. Biol. 33(7), 991–1009 (2007)CrossRefGoogle Scholar
  71. 71.
    SonoWand: Sonowand—product description. http://www.sonowand.com/ (2011)
  72. 72.
    Steen, E., Olstad, B.: Volume rendering of 3D medical ultrasound data using direct feature mapping. IEEE Trans. Med. Imaging 13(3), 517–525 (1994)CrossRefGoogle Scholar
  73. 73.
    Stetten, G.D., Chib, V.S., Tamburo, R.J.: Tomographic reflection to merge ultrasound images with direct vision. In: Proceedings of 29th Applied Imagery Pattern Recognition Workshop, pp. 200–205 (2000)Google Scholar
  74. 74.
    Thune, N., Gilja, O.H., Hausken, T., Matre, K.: A practical method for estimating enclosed volumes using 3D ultrasound. Eur. J. Ultrasound 3(1), 83–92 (1996)CrossRefGoogle Scholar
  75. 75.
    Viola, I., Nylund, K., Øye, O.K., Ulvang, D.M., Gilja, O.H.: Illustrated ultrasound for multimodal data interpretation of liver examinations. In: Proceedings of Eurographics Workshop on Visual Computing in Biomedicine, pp. 125–133 (2008)Google Scholar
  76. 76.
    Šoltészová, V., Patel, D., Bruckner, S.: A multidirectional occlusion shading model for direct volume rendering. Comput. Graph. Forum 29(3), 883–891 (2010)CrossRefGoogle Scholar
  77. 77.
    Wang, S., Sun, Y., Chang, F.: Artifact removal and texture-based rendering for visualization of 3D fetal ultrasound images. Med. Biol. Eng. Comput. 46(6), 575–588 (2008)CrossRefGoogle Scholar
  78. 78.
    Wein, W., Brunke, S., Khamene, A., Callstrom, M.R., Navab, N.: Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med. Image Anal. 12(5), 577–585 (2008)CrossRefGoogle Scholar
  79. 79.
    Wein, W., Khamene, A.: Image-based method for in-vivo freehand ultrasound calibration. In: SPIE Medical Imaging, vol. 6920, p. 69200K (2008)Google Scholar
  80. 80.
    Wein, W., Pache, F., Roeper, B., Navab, N.: Backward-warping ultrasound reconstruction for improving diagnostic value and registration. In: Proceedings of Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science, pp. 750–757. Springer (2006)Google Scholar
  81. 81.
    Wu, B., Klatzky, R.L., Shelton, D., Stetten, G.D.: Psychophysical evaluation of in-situ ultrasound visualization. Trans. Visual. Comput. Graph. 11(6), 684–693 (2005)CrossRefGoogle Scholar
  82. 82.
    Yoo, Y.M., Managuli, R., Kim, Y.: New multi-volume rendering technique for three-dimensional power Doppler imaging. Ultrasonics 46(4), 313–322 (2007)CrossRefGoogle Scholar
  83. 83.
    Zhang, W., Noble, J.A., Brady, J.M.: Real time 3-D ultrasound to MR cardiovascular image registration using a phase-based approach. In: Proceedings of IEEE International Symposium on Biomedical Imaging: Macro to Nano, pp. 666–669 (2006)Google Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Åsmund Birkeland
    • 1
  • Veronika Šoltészová
    • 1
    • 6
  • Dieter Hönigmann
    • 2
  • Odd Helge Gilja
    • 3
  • Svein Brekke
    • 1
    • 4
  • Timo Ropinski
    • 5
  • Ivan Viola
    • 1
    • 7
  1. 1.University of BergenBergenNorway
  2. 2.N22 Research and Technology TransferWiener NeustadtAustria
  3. 3.Haukeland University HospitalBergenNorway
  4. 4.Archer—The Well CompanyBergenNorway
  5. 5.Linköping UniversityLinköpingSweden
  6. 6.Christian Michelsen ResearchBergenNorway
  7. 7.Vienna University of TechnologyViennaAustria

Personalised recommendations