Purpose Content-based image retrieval (CBIR) in medicine has been demonstrated to improve evidence-based diagnosis, education, and teaching. However, the low clinical adoption of CBIR is partially because the focus of most studies has been the development of feature extraction and similarity measurement algorithms with limited work on facilitating better understanding of the similarity between complex volumetric and multi-modality medical images. In this paper, we present a method for defining user interfaces (UIs) that enable effective human user interpretation of retrieved images.
Methods We derived a set of visualisation and interaction requirements based on the characteristics of modern volumetric medical images. We implemented a UI that visualised multiple views of a single image, displayed abstractions of image data, and provided access to supplementary non-image data. We also defined interactions for refining the search and visually indicating the similarities between images. We applied the UI for the retrieval of multi-modality positron emission tomography and computed tomography (PET-CT) images. We conducted a user survey to evaluate the capabilities of our UI.
Results Our proposed method obtained a high rating (\(\ge \)4 out of 5) in the majority of survey questions. In particular, the survey responses indicated the UI presented all the information necessary to understand the retrieved images, and did so in an intuitive manner.
Conclusion Our proposed UI design improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The implementation of CBIR UIs designed to assist human interpretation could facilitate wider adoption of medical CBIR systems.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Content-based image retrieval
Picture archiving and communications systems
Positron emission tomography
Combined positron emission tomography and computed tomography
Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications–clinical benefits and future directions. Int J Med Inform 73(1):1–23
Müller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: IEEE ICME, pp 683– 686
Müller H, Kalpathy-Cramer J, Caputo B, Syeda-Mahmood T, Wang F (2010) Overview of the first workshop on medical content—based retrieval for clinical decision support at MICCAI 2009. In: LNCS, vol 5853, pp 1–17
Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256(1): 243–252
Müller H, Rosset A, Garcia A, Vallée J-P, Geissbuhler A (2005) Benefits of content-based visual data access in radiology. Radiographics 25(3):849–858
Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal 22(12):1349–1380
Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):5: 1–5:60
Long LR, Antani S, Deserno TM, Thoma GR (2009) Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions. Int J Health Info Syst Inform 4(1):1–16
Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimed Comput 2(1):1–19
Deserno T, Antani S, Long R (2009) Ontology of gaps in content-based image retrieval. J Digit Imaging 22:202–215
Shyu C-R, Brodley CE, Kak AC, Kosaka A, Aisen AM, Broderick LS (1999) ASSERT: a physician-in-the-loop content-based retrieval system for HRCT image databases. Comput Vis Image Underst 75(1–2):111–132
Quellec G, Lamard M, Cazuguel G, Cochener B, Roux C (2010) Wavelet optimization for content-based image retrieval in medical databases. Med Image Anal 14(2):227–241
Quellec G, Lamard M, Bekri L, Cazuguel G, Roux C, Cochener B (2010) Medical case retrieval from a committee of decision trees. IEEE Trans Inf Technol B 14(5):1227–1235
Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1):108–118
Townsend DW, Beyer T, Blodgett TM (2003) PET/CT scanners: a hardware approach to image fusion. Semin Nucl Med 33(3):193–204
Judenhofer MS, Catana C, Swann BK, Siegel SB, Jung W-I, Nutt RE, Cherry SR, Claussen CD, Pichler BJ (2007) PET/MR images acquired with a compact MR-compatible PET detector in a 7-T magnet. Radiology 244(3):807–814
Blodgett TM, Meltzer CC, Townsend DW (2007) PET/CT: form and function. Radiology 242(2):360–385
Deserno T, Güld M, Plodowski B, Spitzer K, Wein B, Schubert H, Ney H, Seidl T (2008) Extended query refinement for medical image retrieval. J Digit Imaging 21:280–289
Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) SPIRS: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24
Kumar A, Haraguchi D, Kim J, Wen L, Eberl S, Fulham M, Feng DD (2011) A query and visualisation interface for a PET-CT image retrieval system. Int J Comput Assist Radiol Surg 6(Supplement 1):69–69
Kumar A, Kim J, Feng D, Fulham M (2012) Graph-based retrieval of multi-modality medical images: a comparison of representations using simulated images. In: IEEE Symp CBMS, pp 1–6
Kumar A, Kim J, Wen L, Feng D (2012) A graph-based approach to the retrieval of volumetric PET-CT lung images. In: IEEE EMBC, pp 5408–5411
Hu S, Hoffman EA, Reinhardt JM (2001) Automatic lung segmentation for accurate quantitation of volumetric X-ray CT images. IEEE Trans Med Imaging 20(6):490–498
Bradley J, Thorstad WL, Mutic S, Miller TR, Dehdashti F, Siegel BA, Bosch W, Bertrand RJ (2004) Impact of FDG-PET on radiation therapy volume delineation in non-small-cell lung cancer. Int J Radiat Oncol Biol Phys 59(1):78–86
Petrakis EGM (2002) Design and evaluation of spatial similarity approaches for image retrieval. Image Vis Comput 20(1):59–76
Neuhaus M, Riesen K, Bunke H (2006) Fast suboptimal algorithms for the computation of graph edit distance. In: LNCS, vol 4109, pp 163–172
Tory M, Moller T (2004) Human factors in visualization research. IEEE Trans Vis Comput Graph 10(1):72–84
Wilson ML (2011) Search user interface design. Synth Lect Info Concepts Retr Serv 3(3):1–143
Bunke H (1999) Error correcting graph matching: on the influence of the underlying cost function. IEEE Trans Pattern Anal 21(9):917–922
Abramoff MD, Magelhaes PJ, Ram SJ (2004) Image processing with ImageJ. Biophot Int 11(7):36–42
O’Madadhain J, Fisher D, White S, Boey Y (2003) The JUNG (java universal network/graph) framework. http://jung.sourceforge.net. Last updated: 24/01/2010, last accessed: 12/03/2013
Nielsen J, Landauer TK (1993) A mathematical model of the finding of usability problems. In: Proc. INTERACT ’93 CHI ’93 Human Fact Comput Sys, pp 206–213
Law EL-C, Roto V, Hassenzahl M, Vermeeren APOS, Kort J. (2009) Understanding, scoping and defining user experience: a survey approach. In: Proc. SIGCHI Conf. Human Factors, Comput. Syst., pp 719–728
Aisen AM, Broderick LS, Winer-Muram H, Brodley CE, Kak AC, Pavlopoulou C, Dy J, Shyu C-R, Marchiori A (2003) Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology 228(1):265–270
Conflict of Interest
About this article
Cite this article
Kumar, A., Kim, J., Bi, L. et al. Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images. Int J CARS 8, 1003–1014 (2013). https://doi.org/10.1007/s11548-013-0896-5
- Content-based image retrieval
- Retrieval interpretation
- User interface