Skip to main content

Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images


Purpose Content-based image retrieval (CBIR) in medicine has been demonstrated to improve evidence-based diagnosis, education, and teaching. However, the low clinical adoption of CBIR is partially because the focus of most studies has been the development of feature extraction and similarity measurement algorithms with limited work on facilitating better understanding of the similarity between complex volumetric and multi-modality medical images. In this paper, we present a method for defining user interfaces (UIs) that enable effective human user interpretation of retrieved images.

Methods We derived a set of visualisation and interaction requirements based on the characteristics of modern volumetric medical images. We implemented a UI that visualised multiple views of a single image, displayed abstractions of image data, and provided access to supplementary non-image data. We also defined interactions for refining the search and visually indicating the similarities between images. We applied the UI for the retrieval of multi-modality positron emission tomography and computed tomography (PET-CT) images. We conducted a user survey to evaluate the capabilities of our UI.

Results Our proposed method obtained a high rating (\(\ge \)4 out of 5) in the majority of survey questions. In particular, the survey responses indicated the UI presented all the information necessary to understand the retrieved images, and did so in an intuitive manner.

Conclusion Our proposed UI design improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The implementation of CBIR UIs designed to assist human interpretation could facilitate wider adoption of medical CBIR systems.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8



Content-based image retrieval


Computed tomography


Picture archiving and communications systems


Positron emission tomography


Combined positron emission tomography and computed tomography


  1. Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications–clinical benefits and future directions. Int J Med Inform 73(1):1–23

    Google Scholar 

  2. Müller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: IEEE ICME, pp 683– 686

  3. Müller H, Kalpathy-Cramer J, Caputo B, Syeda-Mahmood T, Wang F (2010) Overview of the first workshop on medical content—based retrieval for clinical decision support at MICCAI 2009. In: LNCS, vol 5853, pp 1–17

  4. Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256(1): 243–252

    Google Scholar 

  5. Müller H, Rosset A, Garcia A, Vallée J-P, Geissbuhler A (2005) Benefits of content-based visual data access in radiology. Radiographics 25(3):849–858

    PubMed  Article  Google Scholar 

  6. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal 22(12):1349–1380

    Article  Google Scholar 

  7. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):5: 1–5:60

    Google Scholar 

  8. Long LR, Antani S, Deserno TM, Thoma GR (2009) Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions. Int J Health Info Syst Inform 4(1):1–16

    Article  Google Scholar 

  9. Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimed Comput 2(1):1–19

    Article  Google Scholar 

  10. Deserno T, Antani S, Long R (2009) Ontology of gaps in content-based image retrieval. J Digit Imaging 22:202–215

    PubMed  Article  Google Scholar 

  11. Shyu C-R, Brodley CE, Kak AC, Kosaka A, Aisen AM, Broderick LS (1999) ASSERT: a physician-in-the-loop content-based retrieval system for HRCT image databases. Comput Vis Image Underst 75(1–2):111–132

    Article  Google Scholar 

  12. Quellec G, Lamard M, Cazuguel G, Cochener B, Roux C (2010) Wavelet optimization for content-based image retrieval in medical databases. Med Image Anal 14(2):227–241

    PubMed  Article  CAS  Google Scholar 

  13. Quellec G, Lamard M, Bekri L, Cazuguel G, Roux C, Cochener B (2010) Medical case retrieval from a committee of decision trees. IEEE Trans Inf Technol B 14(5):1227–1235

    Article  Google Scholar 

  14. Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1):108–118

    PubMed  Article  Google Scholar 

  15. Townsend DW, Beyer T, Blodgett TM (2003) PET/CT scanners: a hardware approach to image fusion. Semin Nucl Med 33(3):193–204

    PubMed  Article  Google Scholar 

  16. Judenhofer MS, Catana C, Swann BK, Siegel SB, Jung W-I, Nutt RE, Cherry SR, Claussen CD, Pichler BJ (2007) PET/MR images acquired with a compact MR-compatible PET detector in a 7-T magnet. Radiology 244(3):807–814

    PubMed  Article  Google Scholar 

  17. Blodgett TM, Meltzer CC, Townsend DW (2007) PET/CT: form and function. Radiology 242(2):360–385

    PubMed  Article  Google Scholar 

  18. Deserno T, Güld M, Plodowski B, Spitzer K, Wein B, Schubert H, Ney H, Seidl T (2008) Extended query refinement for medical image retrieval. J Digit Imaging 21:280–289

    Google Scholar 

  19. Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) SPIRS: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24

    PubMed  Article  Google Scholar 

  20. Kumar A, Haraguchi D, Kim J, Wen L, Eberl S, Fulham M, Feng DD (2011) A query and visualisation interface for a PET-CT image retrieval system. Int J Comput Assist Radiol Surg 6(Supplement 1):69–69

    Google Scholar 

  21. Kumar A, Kim J, Feng D, Fulham M (2012) Graph-based retrieval of multi-modality medical images: a comparison of representations using simulated images. In: IEEE Symp CBMS, pp 1–6

  22. Kumar A, Kim J, Wen L, Feng D (2012) A graph-based approach to the retrieval of volumetric PET-CT lung images. In: IEEE EMBC, pp 5408–5411

  23. Hu S, Hoffman EA, Reinhardt JM (2001) Automatic lung segmentation for accurate quantitation of volumetric X-ray CT images. IEEE Trans Med Imaging 20(6):490–498

    PubMed  Article  CAS  Google Scholar 

  24. Bradley J, Thorstad WL, Mutic S, Miller TR, Dehdashti F, Siegel BA, Bosch W, Bertrand RJ (2004) Impact of FDG-PET on radiation therapy volume delineation in non-small-cell lung cancer. Int J Radiat Oncol Biol Phys 59(1):78–86

    PubMed  Article  Google Scholar 

  25. Petrakis EGM (2002) Design and evaluation of spatial similarity approaches for image retrieval. Image Vis Comput 20(1):59–76

    Article  Google Scholar 

  26. Neuhaus M, Riesen K, Bunke H (2006) Fast suboptimal algorithms for the computation of graph edit distance. In: LNCS, vol 4109, pp 163–172

  27. Tory M, Moller T (2004) Human factors in visualization research. IEEE Trans Vis Comput Graph 10(1):72–84

    PubMed  Article  Google Scholar 

  28. Wilson ML (2011) Search user interface design. Synth Lect Info Concepts Retr Serv 3(3):1–143

    CAS  Google Scholar 

  29. Bunke H (1999) Error correcting graph matching: on the influence of the underlying cost function. IEEE Trans Pattern Anal 21(9):917–922

    Article  Google Scholar 

  30. Abramoff MD, Magelhaes PJ, Ram SJ (2004) Image processing with ImageJ. Biophot Int 11(7):36–42

    Google Scholar 

  31. O’Madadhain J, Fisher D, White S, Boey Y (2003) The JUNG (java universal network/graph) framework. Last updated: 24/01/2010, last accessed: 12/03/2013

  32. Nielsen J, Landauer TK (1993) A mathematical model of the finding of usability problems. In: Proc. INTERACT ’93 CHI ’93 Human Fact Comput Sys, pp 206–213

  33. Law EL-C, Roto V, Hassenzahl M, Vermeeren APOS, Kort J. (2009) Understanding, scoping and defining user experience: a survey approach. In: Proc. SIGCHI Conf. Human Factors, Comput. Syst., pp 719–728

  34. Aisen AM, Broderick LS, Winer-Muram H, Brodley CE, Kak AC, Pavlopoulou C, Dy J, Shyu C-R, Marchiori A (2003) Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology 228(1):265–270

    PubMed  Article  Google Scholar 

Download references

Conflict of Interest


Author information

Authors and Affiliations


Corresponding author

Correspondence to Ashnil Kumar.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kumar, A., Kim, J., Bi, L. et al. Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images. Int J CARS 8, 1003–1014 (2013).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Content-based image retrieval
  • Retrieval interpretation
  • User interface
  • PET-CT