Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images

  • Ashnil KumarEmail author
  • Jinman Kim
  • Lei Bi
  • Michael Fulham
  • Dagan Feng
Original Article


Purpose Content-based image retrieval (CBIR) in medicine has been demonstrated to improve evidence-based diagnosis, education, and teaching. However, the low clinical adoption of CBIR is partially because the focus of most studies has been the development of feature extraction and similarity measurement algorithms with limited work on facilitating better understanding of the similarity between complex volumetric and multi-modality medical images. In this paper, we present a method for defining user interfaces (UIs) that enable effective human user interpretation of retrieved images.

Methods We derived a set of visualisation and interaction requirements based on the characteristics of modern volumetric medical images. We implemented a UI that visualised multiple views of a single image, displayed abstractions of image data, and provided access to supplementary non-image data. We also defined interactions for refining the search and visually indicating the similarities between images. We applied the UI for the retrieval of multi-modality positron emission tomography and computed tomography (PET-CT) images. We conducted a user survey to evaluate the capabilities of our UI.

Results Our proposed method obtained a high rating (\(\ge \)4 out of 5) in the majority of survey questions. In particular, the survey responses indicated the UI presented all the information necessary to understand the retrieved images, and did so in an intuitive manner.

Conclusion Our proposed UI design improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The implementation of CBIR UIs designed to assist human interpretation could facilitate wider adoption of medical CBIR systems.


Content-based image retrieval Retrieval interpretation User interface PET-CT 



Content-based image retrieval


Computed tomography


Picture archiving and communications systems


Positron emission tomography


Combined positron emission tomography and computed tomography


Conflict of Interest



  1. 1.
    Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications–clinical benefits and future directions. Int J Med Inform 73(1):1–23Google Scholar
  2. 2.
    Müller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: IEEE ICME, pp 683– 686Google Scholar
  3. 3.
    Müller H, Kalpathy-Cramer J, Caputo B, Syeda-Mahmood T, Wang F (2010) Overview of the first workshop on medical content—based retrieval for clinical decision support at MICCAI 2009. In: LNCS, vol 5853, pp 1–17Google Scholar
  4. 4.
    Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256(1): 243–252Google Scholar
  5. 5.
    Müller H, Rosset A, Garcia A, Vallée J-P, Geissbuhler A (2005) Benefits of content-based visual data access in radiology. Radiographics 25(3):849–858PubMedCrossRefGoogle Scholar
  6. 6.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal 22(12):1349–1380CrossRefGoogle Scholar
  7. 7.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):5: 1–5:60Google Scholar
  8. 8.
    Long LR, Antani S, Deserno TM, Thoma GR (2009) Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions. Int J Health Info Syst Inform 4(1):1–16CrossRefGoogle Scholar
  9. 9.
    Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimed Comput 2(1):1–19CrossRefGoogle Scholar
  10. 10.
    Deserno T, Antani S, Long R (2009) Ontology of gaps in content-based image retrieval. J Digit Imaging 22:202–215PubMedCrossRefGoogle Scholar
  11. 11.
    Shyu C-R, Brodley CE, Kak AC, Kosaka A, Aisen AM, Broderick LS (1999) ASSERT: a physician-in-the-loop content-based retrieval system for HRCT image databases. Comput Vis Image Underst 75(1–2):111–132CrossRefGoogle Scholar
  12. 12.
    Quellec G, Lamard M, Cazuguel G, Cochener B, Roux C (2010) Wavelet optimization for content-based image retrieval in medical databases. Med Image Anal 14(2):227–241PubMedCrossRefGoogle Scholar
  13. 13.
    Quellec G, Lamard M, Bekri L, Cazuguel G, Roux C, Cochener B (2010) Medical case retrieval from a committee of decision trees. IEEE Trans Inf Technol B 14(5):1227–1235CrossRefGoogle Scholar
  14. 14.
    Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1):108–118PubMedCrossRefGoogle Scholar
  15. 15.
    Townsend DW, Beyer T, Blodgett TM (2003) PET/CT scanners: a hardware approach to image fusion. Semin Nucl Med 33(3):193–204PubMedCrossRefGoogle Scholar
  16. 16.
    Judenhofer MS, Catana C, Swann BK, Siegel SB, Jung W-I, Nutt RE, Cherry SR, Claussen CD, Pichler BJ (2007) PET/MR images acquired with a compact MR-compatible PET detector in a 7-T magnet. Radiology 244(3):807–814PubMedCrossRefGoogle Scholar
  17. 17.
    Blodgett TM, Meltzer CC, Townsend DW (2007) PET/CT: form and function. Radiology 242(2):360–385PubMedCrossRefGoogle Scholar
  18. 18.
    Deserno T, Güld M, Plodowski B, Spitzer K, Wein B, Schubert H, Ney H, Seidl T (2008) Extended query refinement for medical image retrieval. J Digit Imaging 21:280–289 Google Scholar
  19. 19.
    Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) SPIRS: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24PubMedCrossRefGoogle Scholar
  20. 20.
    Kumar A, Haraguchi D, Kim J, Wen L, Eberl S, Fulham M, Feng DD (2011) A query and visualisation interface for a PET-CT image retrieval system. Int J Comput Assist Radiol Surg 6(Supplement 1):69–69Google Scholar
  21. 21.
    Kumar A, Kim J, Feng D, Fulham M (2012) Graph-based retrieval of multi-modality medical images: a comparison of representations using simulated images. In: IEEE Symp CBMS, pp 1–6Google Scholar
  22. 22.
    Kumar A, Kim J, Wen L, Feng D (2012) A graph-based approach to the retrieval of volumetric PET-CT lung images. In: IEEE EMBC, pp 5408–5411Google Scholar
  23. 23.
    Hu S, Hoffman EA, Reinhardt JM (2001) Automatic lung segmentation for accurate quantitation of volumetric X-ray CT images. IEEE Trans Med Imaging 20(6):490–498PubMedCrossRefGoogle Scholar
  24. 24.
    Bradley J, Thorstad WL, Mutic S, Miller TR, Dehdashti F, Siegel BA, Bosch W, Bertrand RJ (2004) Impact of FDG-PET on radiation therapy volume delineation in non-small-cell lung cancer. Int J Radiat Oncol Biol Phys 59(1):78–86PubMedCrossRefGoogle Scholar
  25. 25.
    Petrakis EGM (2002) Design and evaluation of spatial similarity approaches for image retrieval. Image Vis Comput 20(1):59–76CrossRefGoogle Scholar
  26. 26.
    Neuhaus M, Riesen K, Bunke H (2006) Fast suboptimal algorithms for the computation of graph edit distance. In: LNCS, vol 4109, pp 163–172Google Scholar
  27. 27.
    Tory M, Moller T (2004) Human factors in visualization research. IEEE Trans Vis Comput Graph 10(1):72–84PubMedCrossRefGoogle Scholar
  28. 28.
    Wilson ML (2011) Search user interface design. Synth Lect Info Concepts Retr Serv 3(3):1–143Google Scholar
  29. 29.
    Bunke H (1999) Error correcting graph matching: on the influence of the underlying cost function. IEEE Trans Pattern Anal 21(9):917–922CrossRefGoogle Scholar
  30. 30.
    Abramoff MD, Magelhaes PJ, Ram SJ (2004) Image processing with ImageJ. Biophot Int 11(7):36–42Google Scholar
  31. 31.
    O’Madadhain J, Fisher D, White S, Boey Y (2003) The JUNG (java universal network/graph) framework. Last updated: 24/01/2010, last accessed: 12/03/2013
  32. 32.
    Nielsen J, Landauer TK (1993) A mathematical model of the finding of usability problems. In: Proc. INTERACT ’93 CHI ’93 Human Fact Comput Sys, pp 206–213Google Scholar
  33. 33.
    Law EL-C, Roto V, Hassenzahl M, Vermeeren APOS, Kort J. (2009) Understanding, scoping and defining user experience: a survey approach. In: Proc. SIGCHI Conf. Human Factors, Comput. Syst., pp 719–728Google Scholar
  34. 34.
    Aisen AM, Broderick LS, Winer-Muram H, Brodley CE, Kak AC, Pavlopoulou C, Dy J, Shyu C-R, Marchiori A (2003) Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology 228(1):265–270PubMedCrossRefGoogle Scholar

Copyright information

© CARS 2013

Authors and Affiliations

  • Ashnil Kumar
    • 1
    Email author
  • Jinman Kim
    • 1
  • Lei Bi
    • 1
  • Michael Fulham
    • 1
    • 2
    • 3
  • Dagan Feng
    • 1
    • 4
  1. 1.Biomedical and Multimedia Information Technology (BMIT) Research GroupSchool of Information Technologies, University of SydneySydneyAustralia
  2. 2.Department of Molecular ImagingRoyal Prince Alfred HospitalSydneyAustralia
  3. 3.Sydney Medical SchoolUniversity of SydneySydneyAustralia
  4. 4.Med-X Research InstituteShanghai Jiao Tong University ShanghaiChina

Personalised recommendations