Journal of Digital Imaging

, Volume 24, Issue 2, pp 208–222 | Cite as

Content-Based Image Retrieval in Radiology: Current Status and Future Directions

  • Ceyhun Burak Akgül
  • Daniel L. Rubin
  • Sandy Napel
  • Christopher F. Beaulieu
  • Hayit Greenspan
  • Burak AcarEmail author


Diagnostic radiology requires accurate interpretation of complex signals in medical images. Content-based image retrieval (CBIR) techniques could be valuable to radiologists in assessing medical images by identifying similar images in large archives that could assist with decision support. Many advances have occurred in CBIR, and a variety of systems have appeared in nonmedical domains; however, permeation of these methods into radiology has been limited. Our goal in this review is to survey CBIR methods and systems from the perspective of application to radiology and to identify approaches developed in nonmedical applications that could be translated to radiology. Radiology images pose specific challenges compared with images in the consumer domain; they contain varied, rich, and often subtle features that need to be recognized in assessing image similarity. Radiology images also provide rich opportunities for CBIR: rich metadata about image semantics are provided by radiologists, and this information is not yet being used to its fullest advantage in CBIR systems. By integrating pixel-based and metadata-based image feature analysis, substantial advances of CBIR in medicine could ensue, with CBIR systems becoming an important tool in radiology practice.

Key words

Content-based image retrieval imaging informatics information storage and retrieval digital image management decision support 



This work has partly been supported by NIH CA72023 and TÜBİTAK KARİYER-DRESS (104E035).


  1. 1.
    Rubin GD: Data explosion: the challenge of multidetector-row CT. Eur J Radiol 36(2):74–80, 2000PubMedCrossRefGoogle Scholar
  2. 2.
    Siegle RL, et al: Rates of disagreement in imaging interpretation in a group of community hospitals. Acad Radiol 5(3):148–154, 1998PubMedCrossRefGoogle Scholar
  3. 3.
    Barlow WE, et al: Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst 96(24):1840–1850, 2004PubMedCrossRefGoogle Scholar
  4. 4.
    McDonald CJ: Medical heuristics: the silent adjudicators of clinical practice. Ann Intern Med 124:56–62, 1996PubMedGoogle Scholar
  5. 5.
    Doi K: Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 31(4–5):198–211, 2007PubMedCrossRefGoogle Scholar
  6. 6.
    Burnside ES, et al: Bayesian network to predict breast cancer risk of mammographic microcalcifications and reduce number of benign biopsy results: initial experience. Radiology 240(3):666–673, 2006PubMedCrossRefGoogle Scholar
  7. 7.
    Rubin DL, Burnside ES, Shachter R: A Bayesian Network to assist mammography interpretation. In: M.L. Brandeau, F.S. F, and W.P. Pierskalla, Eds. Operations Research and Health Care. Kluwer Academic Publishers: Boston, 2004, pp 695–720Google Scholar
  8. 8.
    Kahn CE: Artificial intelligence in radiology: decision support systems. Radiographics 14:849–861, 1994PubMedGoogle Scholar
  9. 9.
    Datta R, et al: Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2), 2008Google Scholar
  10. 10.
    Muller H, et al: A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. Int J Med Informatics 73(1):1–23, 2004CrossRefGoogle Scholar
  11. 11.
    Smeulders AWM, et al: Content-based image retrieval at the end of the early years. IEEE Trans Patt Anal Mach Intell 22(12):1349–1380, 2000CrossRefGoogle Scholar
  12. 12.
    Swain MJ, Ballard DH: Color indexing. Int J Comput Vis 7(1):11–32, 1991CrossRefGoogle Scholar
  13. 13.
    Comaniciu D, Meer P, Foran DJ: Image-guided decision support system for pathology. Mach Vis Appl 11(4):213–224, 1999CrossRefGoogle Scholar
  14. 14.
    Kwak DM, et al: Content-based ultrasound image retrieval using a coarse to fine approach. Ann N Y Acad Sci 980:212–224, 2002PubMedCrossRefGoogle Scholar
  15. 15.
    Lim J, Chevallet J-P: Vismed: A visual vocabulary approach for medical image indexing and retrieval. in Second Asia Information Retrieval Symposium. 2005. Jeju Island, KoreaGoogle Scholar
  16. 16.
    Shyu CR, et al: ASSERT: a physician-in-the-loop content-based image retrieval system for HRCT image databases. Comput Vis Image Underst 75(1/2):111–132, 1999CrossRefGoogle Scholar
  17. 17.
    Cauvin JM, et al: Computer-assisted diagnosis system in digestive endoscopy. IEEE Trans Inf Technol Biomed 7(4):256–262, 2003PubMedCrossRefGoogle Scholar
  18. 18.
    Güld MO, et al: Content-Based Retrieval of Medical Images by Combining Global Features. Accessing Multilingual Information Repositories. in Accessing Multilingual Information Repositories. 2005: Springer LNCS 4022Google Scholar
  19. 19.
    Lubbers K, et al: A Probabilistic Approach to Medical Image Retrieval, in Multilingual Information Access for Text, Speech and Images. Springer Berlin, 2005, pp 761–772Google Scholar
  20. 20.
    Doyle S, et al: Using manifold learning for content-based image retrieval of prostate histopathology. in MICCAI 2007 Workshop on Content-Based Image Retrieval for Biomedical Image Archives. Brisbane, Australia, 2007Google Scholar
  21. 21.
    Gletsos M, et al: A computer-aided diagnostic system to characterize CT focal liver lesions: design and optimization of a neural network classifier. IEEE Trans Inf Technol Biomed 7(3):153–162, 2003PubMedCrossRefGoogle Scholar
  22. 22.
    Zhu H, et al: A new local multiscale Fourier analysis for medical imaging. Med Phys 30:1134–1141, 2003PubMedCrossRefGoogle Scholar
  23. 23.
    Oliveira MC, Cirne W, Marques PDA: Towards applying content-based image retrieval in the clinical routine. Future Gener Comput Syst 23(3):466–474, 2007CrossRefGoogle Scholar
  24. 24.
    Rahman M, Bhattacharya P, Desai BC: A framework for medical image retrieval using machine learning and statistical similarity matching techniques with relevance feedback. IEEE Trans Inf Technol Biomed 11(1):58–69, 2007PubMedCrossRefGoogle Scholar
  25. 25.
    Mao J, Jain AK: Texture classification and segmentation using multiresolution simultaneous autoregressive models. Pattern Recogn 25(2):173–188, 1992CrossRefGoogle Scholar
  26. 26.
    Iqbal Q, Aggarwal JK: Combining structure, color and texture for image retrieval: a performance evaluation. in International Conference on Pattern Recognition (ICPR). Quebec City, Canada, 2002Google Scholar
  27. 27.
    Belongie S, Malik J, Puzicha J: Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Machine Intelligence 24(4):509–522, 2002CrossRefGoogle Scholar
  28. 28.
    Akgul CB, et al: 3D model retrieval using probability density-based shape descriptors. IEEE Trans Pattern Anal Machine Intelligence 31(6):1117–1133, 2009CrossRefGoogle Scholar
  29. 29.
    Gokturk SB, et al: A statistical 3-D pattern processing method for computer-aided detection of polyps in CT colonography. IEEE Trans Med Imag 20(12):1251–1260, 2001CrossRefGoogle Scholar
  30. 30.
    Yoshida H, Nappi J: Three-dimensional computer-aided diagnosis scheme for detection of colonic polyps. IEEE Trans Med Imag 20(12):1261–1274, 2001CrossRefGoogle Scholar
  31. 31.
    Rubin GD, et al: Pulmonary nodules on multi-detector row CT scans: performance comparison of radiologists and computer-aided detection. Radiology 234(1):274–283, 2005PubMedCrossRefGoogle Scholar
  32. 32.
    Alto H, Rangayyan RM, Desautels JEL: Content-based retrieval and analysis of mammographic masses. Journal of Electronic Imaging 14(2):1–17, 2005CrossRefGoogle Scholar
  33. 33.
    Qian XN, Tagare HD: Optimal embedding for shape indexing in medical image databases. in Medical Image Computing and Computer-Assisted Intervention (MICCAI). 2005Google Scholar
  34. 34.
    Antani S, et al: Evaluation of shape similarity measurement methods for spine X-ray images. J Vis Commun Image Represent 15(3):285–302, 2004CrossRefGoogle Scholar
  35. 35.
    Balmachnova E, et al: Content-based image retrieval by means of scale-space top-points and differential invariants. in MICCAI 2007 Workshop on Content-based Image Retrieval for Biomedical Image Archives. Brisbane, Australia, 2007Google Scholar
  36. 36.
    Golland P, et al: Detection and analysis of statistical differences in anatomical shape. Med Image Anal 9(1):69–86, 2005PubMedCrossRefGoogle Scholar
  37. 37.
    Bansal R, et al: Statistical analyses of brain surfaces using Gaussian random fields on 2-D manifolds. IEEE Trans Med Imaging 26(1):46–57, 2007PubMedCrossRefGoogle Scholar
  38. 38.
    Toews M, Arbel T: A statistical parts-based model of anatomical variability. IEEE Trans Med Imag 26(4):497–508, 2007CrossRefGoogle Scholar
  39. 39.
    Wang JZ: Pathfinder: multiresolution region-based searching of pathology images using IRM. in AMIA Symp. 2000Google Scholar
  40. 40.
    Pokrajac D, et al: Applying spatial distribution analysis techniques to classification of 3D medical images. Artif Intell Med 33(3):261–280, 2005PubMedCrossRefGoogle Scholar
  41. 41.
    Nielsen J, Nelson M, Liu L: Image-matching as a medical diagnostic support tool (DST) for brain diseases in children. Comput Med Imaging Graph 29(2/3):195–202, 2005PubMedCrossRefGoogle Scholar
  42. 42.
    Sasso G, et al: A visual query-by-example image database for chest CT images: potential role as a decision and educational support tool for radiologists. J Digit Imaging 18(1):78–84, 2005PubMedCrossRefGoogle Scholar
  43. 43.
    Petrakis EGM, Faloutsos C, Lin KI: ImageMap: an image indexing method based on spatial similarity. IEEE Trans Knowl Data Eng 14(5):979–987, 2002CrossRefGoogle Scholar
  44. 44.
    Shantanu HJ, Washington M: Statistical shape analysis: clustering, learning, and testing. IEEE Trans Pattern Anal Mach Intell 27(4):590–602, 2005CrossRefGoogle Scholar
  45. 45.
    Tong L, Hongbin Z: Riemannian manifold learning. IEEE Trans Pattern Anal Mach Intell 30(5):796–809, 2008CrossRefGoogle Scholar
  46. 46.
    Zhang J, et al: Object representation and recognition in shape spaces. Pattern Recogn 36(5):1143–1154, 2003CrossRefGoogle Scholar
  47. 47.
    Rubner Y, Tomasi C, Guibas LJ: The earth mover's distance as a metric for image retrieval. Int J Comput Vis 40(2):99–121, 2000CrossRefGoogle Scholar
  48. 48.
    Bunke H, Irniger C, Neuhaus M: Graph Matching—Challenges and Potential Solutions, in Image Analysis and Processing–ICIAP 2005, 2005, pp 1–10Google Scholar
  49. 49.
    Glaunès J, et al: Large deformation diffeomorphic metric curve mapping. Int J Comput Vis 80(3):317–336, 2008PubMedCrossRefGoogle Scholar
  50. 50.
    Veltkamp RC: Shape matching: similarity measures and algorithms. in Shape Modeling and Applications, SMI 2001 International Conference on, 2001Google Scholar
  51. 51.
    Akgül CB, et al: Similarity learning for 3D object retrieval using relevance feedback and risk minimization. To appear in Int. Journal of Computer Vision, 2010Google Scholar
  52. 52.
    Rahmani R, et al: Localized content-based image retrieval. IEEE Trans Pattern Anal Mach Intell 30(11):1902–1912, 2008PubMedCrossRefGoogle Scholar
  53. 53.
    Rui Y, et al: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans Circuits Syst Video Technol 8(5):644–655, 1998CrossRefGoogle Scholar
  54. 54.
    Tao D, et al: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans Pattern Anal Mach Intell 28(7):1088–1099, 2006PubMedCrossRefGoogle Scholar
  55. 55.
    Keysers D, et al: Statistical framework for model-based image retrieval in medical applications. J Electron Imaging 12(1):59–68, 2003CrossRefGoogle Scholar
  56. 56.
    Müller H, et al: The Use of MedGIFT and EasyIR for ImageCLEF 2005. in Accessing Multilingual Information Repositories. 2005: Springer LNCS 4022Google Scholar
  57. 57.
    Mohammad-Reza S, et al: Content-based image database system for epilepsy. Comput Methods Programs Biomed 79(3):209–226, 2005CrossRefGoogle Scholar
  58. 58.
    El-Naqa I, et al: A similarity learning approach to content-based image retrieval: application to digital mammography. IEEETransactions On Medical Imaging 23(10):1233–1244, 2004CrossRefGoogle Scholar
  59. 59.
    Aschkenasy VS, et al: Unsupervised image classification of medical ultrasound data by multiresolution elastic registration. Ultrasound Med Biol 32(7):1047–1054, 2006PubMedCrossRefGoogle Scholar
  60. 60.
    Kim J, et al: A new way for multidimensional medical data management: Volume of interest (VOI)-based retrieval of medical images with visual and functional features. IEEE Trans Inf Technol Biomed 10(3):598–607, 2006PubMedCrossRefGoogle Scholar
  61. 61.
    Amores J, Radeva P: Registration and retrieval of highly elastic bodies using contextual information. Pattern Recognit Lett 26(11):1720–1731, 2005CrossRefGoogle Scholar
  62. 62.
    Chin Y, et al: An automatic liver segmentation initialization information retrieval strategy for a CBIR followed by a new liver volume segmentation method for CT and MRI image datasets. in MICCAI 2007 Workshop on Content-based Image Retrieval for Biomedical Image Archives. 2007. Brisbane, AustraliaGoogle Scholar
  63. 63.
    Miller MI, Younes L: Group actions, homeomorphisms, and matching: a general framework. Int J Comput Vis 41(1):61–84, 2001CrossRefGoogle Scholar
  64. 64.
    Carson C, et al: Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans Pattern Anal Mach Intell 24(8):1026–1038, 2002CrossRefGoogle Scholar
  65. 65.
    Korn P, et al: Fast and effective retrieval of medical tumor shapes. IEEE Trans Knowl Data Eng 10(6):889–904, 1998CrossRefGoogle Scholar
  66. 66.
    Greenspan H, Pinhas AT: Medical image categorization and retrieval for PACS using the GMM-KL framework. IEEE Trans Inf Technol Biomed 11(2):190–202, 2007PubMedCrossRefGoogle Scholar
  67. 67.
    Avni U, et al., X_ray image categorization and retrieval using patch-based visual words representation, in IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI). 2009Google Scholar
  68. 68.
    Brown R, et al: The use of magnetic resonance imaging to noninvasively detect genetic signatures in oligodendroglioma. Clin Cancer Res 14(8):2357–2362, 2008PubMedCrossRefGoogle Scholar
  69. 69.
    Badr Y, Chbeir R: Automatic Image Description Based on Textual Data, in Journal on Data Semantics VII. 2006. pp. 196–218Google Scholar
  70. 70.
    Besançon R, et al: Cross-Media Feedback Strategies: Merging Text and Image Information to Improve Image Retrieval, in Multilingual Information Access for Text, Speech and Images. 2005. pp. 709–717Google Scholar
  71. 71.
    Barb AS, Chi-Ren S, Sethi YP: Knowledge representation and sharing using visual semantic modeling for diagnostic medical image databases. IEEE Trans Inf Technol Biomed 9(4):538–553, 2005PubMedCrossRefGoogle Scholar
  72. 72.
    Langlotz CP: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6):1595–1597, 2006PubMedCrossRefGoogle Scholar
  73. 73.
    Rubin DL, et al: Medical Imaging on the Semantic Web: Annotation and Image Markup. in AAAI Spring Symposium Series, Semantic Scientific Knowledge Integration. 2008. Stanford UniversityGoogle Scholar
  74. 74.
    Syeda-Mahmood T, et al: AALIM: Multimodal Mining for Cardiac Decision Support. Comput Cardiol 1 and 2:209–212, 2007CrossRefGoogle Scholar
  75. 75.
    Syeda-Mahmood T, Beymer D, Wang F: Shape-based matching of ECG recordings. 2007 Annual International Conference of the Ieee Engineering in Medicine and Biology Society, Vols 1–16:2012–2018, 2007Google Scholar
  76. 76.
    Syeda-Mahmood T, et al: Characterizing spatio-temporal patterns for disease discrimination in cardiac echo videos. Medical Image Computing and Computer-Assisted Intervention–MICCAI, Pt 1. Proceedings 4791:261–269, 2007Google Scholar
  77. 77.
    Syeda-Mahmood T, Beymer D, Amir A: Disease-specific extraction of text from cardiac echo videos for decision support, in Intl. Conf on Document Analysis and Recognition (ICDAR). 2009Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2010

Authors and Affiliations

  • Ceyhun Burak Akgül
    • 1
  • Daniel L. Rubin
    • 2
  • Sandy Napel
    • 2
  • Christopher F. Beaulieu
    • 2
  • Hayit Greenspan
    • 3
  • Burak Acar
    • 1
    Email author
  1. 1.Electrical and Electronics Engineering Department, Volumetric Analysis and Visualization (VAVlab) Lab.Boğaziçi UniversityIstanbulTurkey
  2. 2.Diagnostic RadiologyStanford UniversityStanfordUSA
  3. 3.Department of Biomedical Engineering, The Iby and Aladar Fleischman Faculty of EngineeringTel Aviv UniversityRamat AvivIsrael

Personalised recommendations