Skip to main content
SpringerLink
Log in

A Transfer Function Design for Medical Volume Data Using a Knowledge Database Based on Deep Image and Primitive Intensity Profile Features Retrieval

  • Regular Paper
  • Special Section of CGI 2023
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Direct volume rendering (DVR) is a technique that emphasizes structures of interest (SOIs) within a volume visually, while simultaneously depicting adjacent regional information, e.g., the spatial location of a structure concerning its neighbors. In DVR, transfer function (TF) plays a key role by enabling accurate identification of SOIs interactively as well as ensuring appropriate visibility of them. TF generation typically involves non-intuitive trial-and-error optimization of rendering parameters, which is time-consuming and inefficient. Attempts at mitigating this manual process have led to approaches that make use of a knowledge database consisting of pre-designed TFs by domain experts. In these approaches, a user navigates the knowledge database to find the most suitable pre-designed TF for their input volume to visualize the SOIs. Although these approaches potentially reduce the workload to generate the TFs, they, however, require manual TF navigation of the knowledge database, as well as the likely fine tuning of the selected TF to suit the input. In this work, we propose a TF design approach, CBR-TF, where we introduce a new content-based retrieval (CBR) method to automatically navigate the knowledge database. Instead of pre-designed TFs, our knowledge database contains volumes with SOI labels. Given an input volume, our CBR-TF approach retrieves relevant volumes (with SOI labels) from the knowledge database; the retrieved labels are then used to generate and optimize TFs of the input. This approach largely reduces manual TF navigation and fine tuning. For our CBR-TF approach, we introduce a novel volumetric image feature which includes both a local primitive intensity profile along the SOIs and regional spatial semantics available from the co-planar images to the profile. For the regional spatial semantics, we adopt a convolutional neural network to obtain high-level image feature representations. For the intensity profile, we extend the dynamic time warping technique to address subtle alignment differences between similar profiles (SOIs). Finally, we propose a two-stage CBR scheme to enable the use of these two different feature representations in a complementary manner, thereby improving SOI retrieval performance. We demonstrate the capabilities of our CBR-TF approach with comparison with a conventional approach in visualization, where an intensity profile matching algorithm is used, and also with potential use-cases in medical volume visualization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin J C, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M, Buatti J, Aylward S, Miller J V, Pieper S, Kikinis R. 3D Slicer as an image computing platform for the quantitative imaging network. Magnetic Resonance Imaging, 2012, 30(9): 1323–1341. DOI: https://doi.org/10.1016/j.mri.2012.05.001.

    Article  Google Scholar 

  2. Ljung P, Krüger J, Groller E, Hadwiger M, Hansen C D, Ynnerman A. State of the art in transfer functions for direct volume rendering. Computer Graphics Forum, 2016, 35(3): 669–691. DOI: https://doi.org/10.1111/cgf.12934.

    Article  Google Scholar 

  3. Kniss J, Kindlmann G, Hansen C. Interactive volume rendering using multi-dimensional transfer functions and direct manipulation widgets. In Proc. the 2001 IEEE Visualization, Oct. 2001, pp.255–562. DOI: https://doi.org/10.1109/VISUAL.2001.964519.

  4. Correa C, Ma K L. Size-based transfer functions: A new volume exploration technique. IEEE Trans. Visualization and Computer Graphics, 2008, 14(6): 1380–1387. DOI: https://doi.org/10.1109/TVCG.2008.162.

    Article  Google Scholar 

  5. Caban J J, Rheingans P. Texture-based transfer functions for direct volume rendering. IEEE Trans. Visualization and Computer Graphics, 2008, 14(6): 3364–3371. DOI: https://doi.org/10.1109/TVCG.2008.169.

    Article  Google Scholar 

  6. Jung Y, Kim J, Kumar A, Feng D D, Fulham M. Feature of interest-based direct volume rendering using contextual saliency-driven ray profile analysis. Computer Graphics Forum, 2018, 37(6): 5–19. DOI: https://doi.org/10.1111/cgf.13308.

    Article  Google Scholar 

  7. Ropinski T, Praßni J, Steinicke F, Hinrichs K. Stroke-based transfer function design. In Proc. the 5th Eurographics/IEEE VGTC Conference on Point-Based Graphics, Aug. 2008, pp.41–48.

  8. Guo H Q, Mao N Y, Yuan X R. WYSIWYG (what you see is what you get) volume visualization. IEEE Trans. Visualization and Computer Graphics, 2011, 17(12): 2106–2114. DOI: https://doi.org/10.1109/TVCG.2011.261.

    Article  Google Scholar 

  9. Correa C D, Ma K L. Visibility histograms and visibility-driven transfer functions. IEEE Trans. Visualization and Computer Graphics, 2011, 17(2): 192–204. DOI: https://doi.org/10.1109/TVCG.2010.35.

    Article  Google Scholar 

  10. Jung Y, Kim J, Eberl S, Fulham M, Feng D D. Visibility-driven PET-CT visualisation with region of interest (ROI) segmentation. The Visual Computer, 2013, 29(6): 805–815. DOI: https://doi.org/10.1007/s00371-013-0833-1.

    Article  Google Scholar 

  11. Jung Y, Kim J, Kumar A, Feng D D, Fulham M. Efficient visibility-driven medical image visualisation via adaptive binned visibility histogram. Computerized Medical Imaging and Graphics, 2016, 51: 40–49. DOI: https://doi.org/10.1016/j.compmedimag.2016.04.003.

    Article  Google Scholar 

  12. Jung Y, Kim J, Bi L, Kumar A, Feng D D, Fulham M. A direct volume rendering visualization approach for serial PET-CT scans that preserves anatomical consistency. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(5): 733–744. DOI: https://doi.org/10.1007/s11548-019-01916-2.

    Article  Google Scholar 

  13. Marks J, Andalman B, Beardsley P A, Freeman W, Gibson S, Hodgins J, Kang T, Mirtich B, Pfister H, Ruml W, Ryall K, Seims J, Shieber S. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proc. the 24th Annual Conference on Computer Graphics and Interactive Techniques, Aug. 1997, pp.389–400. DOI: https://doi.org/10.1145/258734.258887.

  14. Guo H Q, Li W, Yuan X R. Transfer function map. In Proc. the 2014 IEEE Pacific Visualization Symposium, Mar. 2014, pp.262–266. DOI: https://doi.org/10.1109/PacificVis.2014.24.

  15. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436–444. DOI: https://doi.org/10.1038/nature14539.

    Article  Google Scholar 

  16. Kumar A, Kim J, Cai W D, Fulham M, Feng D G. Content-based medical image retrieval: A survey of applications to multidimensional and multimodality data. Journal of Digital Imaging, 2013, 26(6): 1025–1039. DOI: https://doi.org/10.1007/s10278-013-9619-2.

    Article  Google Scholar 

  17. Kohlmann P, Bruckner S, Kanitsar A, Groller M E. Contextual picking of volumetric structures. In Proc. the 2009 IEEE Pacific Visualization Symposium, Apr. 2009, pp.185–192. DOI: https://doi.org/10.1109/PACIFICVIS.2009.4906855.

  18. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84–90. DOI: https://doi.org/10.1145/3065386.

    Article  Google Scholar 

  19. Keogh E, Ratanamahatana C A. Exact indexing of dynamic time warping. Knowledge and Information Systems, 2005, 7(3): 358–386. DOI: https://doi.org/10.1007/s10115-004-0154-9.

    Article  Google Scholar 

  20. Soler L, Hostettler A, Agnus V, Charnoz A, Fasquel J B, Moreau J, Osswald A B, Bouhadjar M, Marescaux J. 3D image reconstruction for comparison of algorithm database: A patient specific anatomical and medical image database. Technical Report, IRCAD, 2010. https://www.ircad.fr/research/data-sets/liver-segmentation-3d-ircadb-01/, Mar. 2024.

  21. Castro S, Königy A, Löffelmanny H, Gröllery E. Transfer function specification for the visualization of medical data. Technical Report, Vienna University of Technology, 1998. https://citeseerx.ist.psu.edu/doc_view/pid/08d0bdf2ffe297661e78568baa8f612c91d8e1c1, Mar. 2024.

  22. Harrower M, Brewer C A. ColorBrewer.org: An online tool for selecting colour schemes for maps. The Cartographic Journal, 2003, 40(1): 27–37.

    Article  Google Scholar 

  23. Nelder J A, Mead R. A simplex method for function minimization. The Computer Journal, 1965, 7(4): 308–313. DOI: https://doi.org/10.1093/comjnl/7.4.308.

    Article  MathSciNet  Google Scholar 

  24. Lagarias J C, Reeds J A, Wright M H, Wright P E. Convergence properties of the Nelder-Mead simplex method in low dimensions. SIAM Journal on Optimization, 1998, 9(1): 112–147. DOI: https://doi.org/10.1137/S1052623496303470.

    Article  MathSciNet  Google Scholar 

  25. Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In Proc. the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.1–9. DOI: https://doi.org/10.1109/CVPR.2015.7298594.

  26. He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.770–778. DOI: https://doi.org/10.1109/CVPR.2016.90.

  27. Shin H C, Roth H R, Gao M C, Lu L, Xu Z Y, Nogues I, Yao J H, Mollura D, Summers R M. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Medical Imaging, 2016, 35(5): 1285–1298. DOI: https://doi.org/10.1109/TMI.2016.2528162.

    Article  Google Scholar 

  28. Meyer-Spradow J, Ropinski T, Mensmann J, Hinrichs K. Voreen: A rapid-prototyping environment for ray-casting-based volume visualizations. IEEE Computer Graphics and Applications, 2009, 29(6): 6–13. DOI: https://doi.org/10.1109/MCG.2009.130.

    Article  Google Scholar 

  29. Ahmed K T, Ummesafi S, Iqbal A. Content based image retrieval using image features information fusion. Information Fusion, 2019, 51: 76–99. DOI: https://doi.org/10.1016/j.inffus.2018.11.004.

    Article  Google Scholar 

  30. Vishraj R, Gupta S, Singh S. A comprehensive review of content-based image retrieval systems using deep learning and hand-crafted features in medical imaging: Research challenges and future directions. Computers and Electrical Engineering, 2022, 104: 108450. DOI: https://doi.org/10.1016/j.compeleceng.2022.108450.

    Article  Google Scholar 

  31. Wasserthal J, Breit H C, Meyer M T, Pradella M, Hinck D, Sauter A W, Heye T, Boll D T, Cyriac J, Yang S, Bach M, Segeroth M. TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images. Radiology: Artificial Intelligence, 2023, 5(5): e230024. DOI: https://doi.org/10.1148/ryai.230024.

    Google Scholar 

  32. Zhang C Y, Zheng H, Gu Y. Dive into the details of self-supervised learning for medical image analysis. Medical Image Analysis, 2023, 89: 102879. DOI: https://doi.org/10.1016/j.media.2023.102879.

    Article  Google Scholar 

  33. Chen X X, Wang X M, Zhang K, Fung K M, Thai T C, Moore K, Mannel R S, Liu H, Zheng B, Qiu Y C. Recent advances and clinical applications of deep learning in medical image analysis. Medical Image Analysis, 2022, 79: 102444. DOI: https://doi.org/10.1016/j.media.2022.102444.

    Article  Google Scholar 

  34. Wang L, Qian X M, Zhang Y T, Shen J L, Cao X C. Enhancing sketch-based image retrieval by CNN semantic re-ranking. IEEE Trans. Cybernetics, 2020, 50(7): 3330–3342. DOI: https://doi.org/10.1109/TCYB.2019.2894498.

    Article  Google Scholar 

  35. Huang R Z, Ma K L. RGVis: Region growing based techniques for volume visualization. In Proc. the 11th Pacific Conference on Computer Graphics and Applications, Oct. 2003, pp.355–363. DOI: https://doi.org/10.1109/PCCGA.2003.1238277.

Download references

Author information

Authors and Affiliations

  1. School of Computing, Gachon University, Seongnam, 13120, Korea

    Younhyun Jung

  2. School of Computer Science, The University of Sydney, Sydney, 2006, Australia

    Younhyun Jung, Jim Kong & Jinman Kim

  3. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Bin Sheng  (盛 斌)

Authors
  1. Younhyun Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jim Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Sheng  (盛 斌)
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinman Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Sheng  (盛 斌).

Ethics declarations

Conflict of Interest The authors declare that they have no conflict of interest.

Additional information

This work was supported by the Korea Health Technology Research and Development Project through the Korea Health Industry Development Institute under Grant No. HI22C1651, the National Research Foundation of Korea (NRF) under Grant No. 2021R1F1A1059554, and the Culture, Sports and Tourism Research and Development Program through the Korea Creative Content Agency Grant funded by the Ministry of Culture, Sports and Tourism of Korea under Grant No. RS-2023-00227648.

Younhyun Jung received his B.S. degree in computer science from Inha University, Incheon, in 2008, and his Ph.D. degree in computer science from The University of Sydney, Sydney, in 2016. He is currently an assistant professor in computer science with School of Computing, Gachon University, Seongnam. He was a software engineer with Samsung Electronics from 2007 to 2010. His current research interests include volume rendering and multimodal medical image visualization.

Jim Kong received his B.S. (Hons.) degree in computer science from The University of Sydney, Sydney, in 2016. He is currently a software development engineer in Amazon, Sydney.

Bin Sheng received his B.A. degree in English and his B.Eng. degree in computer science from Huazhong University of Science and Technology, Wuhan, in 2004, and his M.Sc. degree in software engineering from the University of Macau, Macau, in 2007, and his Ph.D. degree in computer science and engineering from The Chinese University of Hong Kong, Hong Kong, in 2011. He is currently a professor with the Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai. He is an associate editor of the IEEE Transactions on Circuits and Systems for Video Technology, and The Visual Computer Journal. His current research interests include virtual reality and computer graphics.

Jinman Kim received his B.S. (Hons.) and his Ph.D. degrees in computer science both from the University of Sydney, Sydney, in 2001 and 2006, respectively. From 2008 to 2012, he was an ARC Post-Doctoral Research Fellow, one year leave from 2009 to 2010 to join the MIRALab Research Group, Geneva, as a Marie Curie senior research fellow. Since 2013, he has been with the School of Information Technologies, The University of Sydney, Sydney, where he was a senior lecturer, and became a professor in 2022. His current research interests include medical image analysis and visualization, computeraided diagnosis, and telehealth technologies. He is an associate editor of The Visual Computer Journal.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jung, Y., Kong, J., Sheng, B. et al. A Transfer Function Design for Medical Volume Data Using a Knowledge Database Based on Deep Image and Primitive Intensity Profile Features Retrieval. J. Comput. Sci. Technol. (2024). https://doi.org/10.1007/s11390-024-3419-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11390-024-3419-7

Keywords

Search

Navigation