Skip to main content

Content-Based Image Retrieval in Radiology: Current Status and Future Directions

Journal of Digital Imaging volume 24, pages 208–222 (2011)Cite this article

Abstract

Diagnostic radiology requires accurate interpretation of complex signals in medical images. Content-based image retrieval (CBIR) techniques could be valuable to radiologists in assessing medical images by identifying similar images in large archives that could assist with decision support. Many advances have occurred in CBIR, and a variety of systems have appeared in nonmedical domains; however, permeation of these methods into radiology has been limited. Our goal in this review is to survey CBIR methods and systems from the perspective of application to radiology and to identify approaches developed in nonmedical applications that could be translated to radiology. Radiology images pose specific challenges compared with images in the consumer domain; they contain varied, rich, and often subtle features that need to be recognized in assessing image similarity. Radiology images also provide rich opportunities for CBIR: rich metadata about image semantics are provided by radiologists, and this information is not yet being used to its fullest advantage in CBIR systems. By integrating pixel-based and metadata-based image feature analysis, substantial advances of CBIR in medicine could ensue, with CBIR systems becoming an important tool in radiology practice.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

References

  1. Rubin GD: Data explosion: the challenge of multidetector-row CT. Eur J Radiol 36(2):74–80, 2000

    PubMed  Article  CAS  Google Scholar 

  2. Siegle RL, et al: Rates of disagreement in imaging interpretation in a group of community hospitals. Acad Radiol 5(3):148–154, 1998

    PubMed  Article  CAS  Google Scholar 

  3. Barlow WE, et al: Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst 96(24):1840–1850, 2004

    PubMed  Article  Google Scholar 

  4. McDonald CJ: Medical heuristics: the silent adjudicators of clinical practice. Ann Intern Med 124:56–62, 1996

    PubMed  CAS  Google Scholar 

  5. Doi K: Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 31(4–5):198–211, 2007

    PubMed  Article  Google Scholar 

  6. Burnside ES, et al: Bayesian network to predict breast cancer risk of mammographic microcalcifications and reduce number of benign biopsy results: initial experience. Radiology 240(3):666–673, 2006

    PubMed  Article  Google Scholar 

  7. Rubin DL, Burnside ES, Shachter R: A Bayesian Network to assist mammography interpretation. In: M.L. Brandeau, F.S. F, and W.P. Pierskalla, Eds. Operations Research and Health Care. Kluwer Academic Publishers: Boston, 2004, pp 695–720

  8. Kahn CE: Artificial intelligence in radiology: decision support systems. Radiographics 14:849–861, 1994

    PubMed  Google Scholar 

  9. Datta R, et al: Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2), 2008

  10. Muller H, et al: A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. Int J Med Informatics 73(1):1–23, 2004

    Article  Google Scholar 

  11. Smeulders AWM, et al: Content-based image retrieval at the end of the early years. IEEE Trans Patt Anal Mach Intell 22(12):1349–1380, 2000

    Article  Google Scholar 

  12. Swain MJ, Ballard DH: Color indexing. Int J Comput Vis 7(1):11–32, 1991

    Article  Google Scholar 

  13. Comaniciu D, Meer P, Foran DJ: Image-guided decision support system for pathology. Mach Vis Appl 11(4):213–224, 1999

    Article  Google Scholar 

  14. Kwak DM, et al: Content-based ultrasound image retrieval using a coarse to fine approach. Ann N Y Acad Sci 980:212–224, 2002

    PubMed  Article  Google Scholar 

  15. Lim J, Chevallet J-P: Vismed: A visual vocabulary approach for medical image indexing and retrieval. in Second Asia Information Retrieval Symposium. 2005. Jeju Island, Korea

  16. Shyu CR, et al: ASSERT: a physician-in-the-loop content-based image retrieval system for HRCT image databases. Comput Vis Image Underst 75(1/2):111–132, 1999

    Article  Google Scholar 

  17. Cauvin JM, et al: Computer-assisted diagnosis system in digestive endoscopy. IEEE Trans Inf Technol Biomed 7(4):256–262, 2003

    PubMed  Article  Google Scholar 

  18. Güld MO, et al: Content-Based Retrieval of Medical Images by Combining Global Features. Accessing Multilingual Information Repositories. in Accessing Multilingual Information Repositories. 2005: Springer LNCS 4022

  19. Lubbers K, et al: A Probabilistic Approach to Medical Image Retrieval, in Multilingual Information Access for Text, Speech and Images. Springer Berlin, 2005, pp 761–772

  20. Doyle S, et al: Using manifold learning for content-based image retrieval of prostate histopathology. in MICCAI 2007 Workshop on Content-Based Image Retrieval for Biomedical Image Archives. Brisbane, Australia, 2007

  21. Gletsos M, et al: A computer-aided diagnostic system to characterize CT focal liver lesions: design and optimization of a neural network classifier. IEEE Trans Inf Technol Biomed 7(3):153–162, 2003

    PubMed  Article  Google Scholar 

  22. Zhu H, et al: A new local multiscale Fourier analysis for medical imaging. Med Phys 30:1134–1141, 2003

    PubMed  Article  CAS  Google Scholar 

  23. Oliveira MC, Cirne W, Marques PDA: Towards applying content-based image retrieval in the clinical routine. Future Gener Comput Syst 23(3):466–474, 2007

    Article  Google Scholar 

  24. Rahman M, Bhattacharya P, Desai BC: A framework for medical image retrieval using machine learning and statistical similarity matching techniques with relevance feedback. IEEE Trans Inf Technol Biomed 11(1):58–69, 2007

    PubMed  Article  Google Scholar 

  25. Mao J, Jain AK: Texture classification and segmentation using multiresolution simultaneous autoregressive models. Pattern Recogn 25(2):173–188, 1992

    Article  Google Scholar 

  26. Iqbal Q, Aggarwal JK: Combining structure, color and texture for image retrieval: a performance evaluation. in International Conference on Pattern Recognition (ICPR). Quebec City, Canada, 2002

  27. Belongie S, Malik J, Puzicha J: Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Machine Intelligence 24(4):509–522, 2002

    Article  Google Scholar 

  28. Akgul CB, et al: 3D model retrieval using probability density-based shape descriptors. IEEE Trans Pattern Anal Machine Intelligence 31(6):1117–1133, 2009

    Article  Google Scholar 

  29. Gokturk SB, et al: A statistical 3-D pattern processing method for computer-aided detection of polyps in CT colonography. IEEE Trans Med Imag 20(12):1251–1260, 2001

    Article  CAS  Google Scholar 

  30. Yoshida H, Nappi J: Three-dimensional computer-aided diagnosis scheme for detection of colonic polyps. IEEE Trans Med Imag 20(12):1261–1274, 2001

    Article  CAS  Google Scholar 

  31. Rubin GD, et al: Pulmonary nodules on multi-detector row CT scans: performance comparison of radiologists and computer-aided detection. Radiology 234(1):274–283, 2005

    PubMed  Article  Google Scholar 

  32. Alto H, Rangayyan RM, Desautels JEL: Content-based retrieval and analysis of mammographic masses. Journal of Electronic Imaging 14(2):1–17, 2005

    Article  Google Scholar 

  33. Qian XN, Tagare HD: Optimal embedding for shape indexing in medical image databases. in Medical Image Computing and Computer-Assisted Intervention (MICCAI). 2005

  34. Antani S, et al: Evaluation of shape similarity measurement methods for spine X-ray images. J Vis Commun Image Represent 15(3):285–302, 2004

    Article  Google Scholar 

  35. Balmachnova E, et al: Content-based image retrieval by means of scale-space top-points and differential invariants. in MICCAI 2007 Workshop on Content-based Image Retrieval for Biomedical Image Archives. Brisbane, Australia, 2007

  36. Golland P, et al: Detection and analysis of statistical differences in anatomical shape. Med Image Anal 9(1):69–86, 2005

    PubMed  Article  Google Scholar 

  37. Bansal R, et al: Statistical analyses of brain surfaces using Gaussian random fields on 2-D manifolds. IEEE Trans Med Imaging 26(1):46–57, 2007

    PubMed  Article  Google Scholar 

  38. Toews M, Arbel T: A statistical parts-based model of anatomical variability. IEEE Trans Med Imag 26(4):497–508, 2007

    Article  Google Scholar 

  39. Wang JZ: Pathfinder: multiresolution region-based searching of pathology images using IRM. in AMIA Symp. 2000

  40. Pokrajac D, et al: Applying spatial distribution analysis techniques to classification of 3D medical images. Artif Intell Med 33(3):261–280, 2005

    PubMed  Article  Google Scholar 

  41. Nielsen J, Nelson M, Liu L: Image-matching as a medical diagnostic support tool (DST) for brain diseases in children. Comput Med Imaging Graph 29(2/3):195–202, 2005

    PubMed  Article  Google Scholar 

  42. Sasso G, et al: A visual query-by-example image database for chest CT images: potential role as a decision and educational support tool for radiologists. J Digit Imaging 18(1):78–84, 2005

    PubMed  Article  Google Scholar 

  43. Petrakis EGM, Faloutsos C, Lin KI: ImageMap: an image indexing method based on spatial similarity. IEEE Trans Knowl Data Eng 14(5):979–987, 2002

    Article  Google Scholar 

  44. Shantanu HJ, Washington M: Statistical shape analysis: clustering, learning, and testing. IEEE Trans Pattern Anal Mach Intell 27(4):590–602, 2005

    Article  Google Scholar 

  45. Tong L, Hongbin Z: Riemannian manifold learning. IEEE Trans Pattern Anal Mach Intell 30(5):796–809, 2008

    Article  Google Scholar 

  46. Zhang J, et al: Object representation and recognition in shape spaces. Pattern Recogn 36(5):1143–1154, 2003

    Article  Google Scholar 

  47. Rubner Y, Tomasi C, Guibas LJ: The earth mover's distance as a metric for image retrieval. Int J Comput Vis 40(2):99–121, 2000

    Article  Google Scholar 

  48. Bunke H, Irniger C, Neuhaus M: Graph Matching—Challenges and Potential Solutions, in Image Analysis and Processing–ICIAP 2005, 2005, pp 1–10

  49. Glaunès J, et al: Large deformation diffeomorphic metric curve mapping. Int J Comput Vis 80(3):317–336, 2008

    PubMed  Article  Google Scholar 

  50. Veltkamp RC: Shape matching: similarity measures and algorithms. in Shape Modeling and Applications, SMI 2001 International Conference on, 2001

  51. Akgül CB, et al: Similarity learning for 3D object retrieval using relevance feedback and risk minimization. To appear in Int. Journal of Computer Vision, 2010

  52. Rahmani R, et al: Localized content-based image retrieval. IEEE Trans Pattern Anal Mach Intell 30(11):1902–1912, 2008

    PubMed  Article  Google Scholar 

  53. Rui Y, et al: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans Circuits Syst Video Technol 8(5):644–655, 1998

    Article  Google Scholar 

  54. Tao D, et al: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans Pattern Anal Mach Intell 28(7):1088–1099, 2006

    PubMed  Article  Google Scholar 

  55. Keysers D, et al: Statistical framework for model-based image retrieval in medical applications. J Electron Imaging 12(1):59–68, 2003

    Article  Google Scholar 

  56. Müller H, et al: The Use of MedGIFT and EasyIR for ImageCLEF 2005. in Accessing Multilingual Information Repositories. 2005: Springer LNCS 4022

  57. Mohammad-Reza S, et al: Content-based image database system for epilepsy. Comput Methods Programs Biomed 79(3):209–226, 2005

    Article  Google Scholar 

  58. El-Naqa I, et al: A similarity learning approach to content-based image retrieval: application to digital mammography. IEEETransactions On Medical Imaging 23(10):1233–1244, 2004

    Article  Google Scholar 

  59. Aschkenasy VS, et al: Unsupervised image classification of medical ultrasound data by multiresolution elastic registration. Ultrasound Med Biol 32(7):1047–1054, 2006

    PubMed  Article  Google Scholar 

  60. Kim J, et al: A new way for multidimensional medical data management: Volume of interest (VOI)-based retrieval of medical images with visual and functional features. IEEE Trans Inf Technol Biomed 10(3):598–607, 2006

    PubMed  Article  Google Scholar 

  61. Amores J, Radeva P: Registration and retrieval of highly elastic bodies using contextual information. Pattern Recognit Lett 26(11):1720–1731, 2005

    Article  Google Scholar 

  62. Chin Y, et al: An automatic liver segmentation initialization information retrieval strategy for a CBIR followed by a new liver volume segmentation method for CT and MRI image datasets. in MICCAI 2007 Workshop on Content-based Image Retrieval for Biomedical Image Archives. 2007. Brisbane, Australia

  63. Miller MI, Younes L: Group actions, homeomorphisms, and matching: a general framework. Int J Comput Vis 41(1):61–84, 2001

    Article  Google Scholar 

  64. Carson C, et al: Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans Pattern Anal Mach Intell 24(8):1026–1038, 2002

    Article  Google Scholar 

  65. Korn P, et al: Fast and effective retrieval of medical tumor shapes. IEEE Trans Knowl Data Eng 10(6):889–904, 1998

    Article  Google Scholar 

  66. Greenspan H, Pinhas AT: Medical image categorization and retrieval for PACS using the GMM-KL framework. IEEE Trans Inf Technol Biomed 11(2):190–202, 2007

    PubMed  Article  Google Scholar 

  67. Avni U, et al., X_ray image categorization and retrieval using patch-based visual words representation, in IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI). 2009

  68. Brown R, et al: The use of magnetic resonance imaging to noninvasively detect genetic signatures in oligodendroglioma. Clin Cancer Res 14(8):2357–2362, 2008

    PubMed  Article  CAS  Google Scholar 

  69. Badr Y, Chbeir R: Automatic Image Description Based on Textual Data, in Journal on Data Semantics VII. 2006. pp. 196–218

  70. Besançon R, et al: Cross-Media Feedback Strategies: Merging Text and Image Information to Improve Image Retrieval, in Multilingual Information Access for Text, Speech and Images. 2005. pp. 709–717

  71. Barb AS, Chi-Ren S, Sethi YP: Knowledge representation and sharing using visual semantic modeling for diagnostic medical image databases. IEEE Trans Inf Technol Biomed 9(4):538–553, 2005

    PubMed  Article  Google Scholar 

  72. Langlotz CP: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6):1595–1597, 2006

    PubMed  Article  Google Scholar 

  73. Rubin DL, et al: Medical Imaging on the Semantic Web: Annotation and Image Markup. in AAAI Spring Symposium Series, Semantic Scientific Knowledge Integration. 2008. Stanford University

  74. Syeda-Mahmood T, et al: AALIM: Multimodal Mining for Cardiac Decision Support. Comput Cardiol 1 and 2:209–212, 2007

    Article  Google Scholar 

  75. Syeda-Mahmood T, Beymer D, Wang F: Shape-based matching of ECG recordings. 2007 Annual International Conference of the Ieee Engineering in Medicine and Biology Society, Vols 1–16:2012–2018, 2007

  76. Syeda-Mahmood T, et al: Characterizing spatio-temporal patterns for disease discrimination in cardiac echo videos. Medical Image Computing and Computer-Assisted Intervention–MICCAI, Pt 1. Proceedings 4791:261–269, 2007

    Google Scholar 

  77. Syeda-Mahmood T, Beymer D, Amir A: Disease-specific extraction of text from cardiac echo videos for decision support, in Intl. Conf on Document Analysis and Recognition (ICDAR). 2009

Download references

Acknowledgments

This work has partly been supported by NIH CA72023 and TÜBİTAK KARİYER-DRESS (104E035).

Author information

Affiliations

  1. Electrical and Electronics Engineering Department, Volumetric Analysis and Visualization (VAVlab) Lab., Boğaziçi University, Istanbul, Turkey

    Ceyhun Burak Akgül & Burak Acar

  2. Diagnostic Radiology, Stanford University, Stanford, CA, 94305, USA

    Daniel L. Rubin, Sandy Napel & Christopher F. Beaulieu

  3. Department of Biomedical Engineering, The Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Ramat Aviv, Israel

    Hayit Greenspan

Authors
  1. Ceyhun Burak Akgül
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel L. Rubin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandy Napel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christopher F. Beaulieu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hayit Greenspan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Burak Acar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Burak Acar.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Akgül, C.B., Rubin, D.L., Napel, S. et al. Content-Based Image Retrieval in Radiology: Current Status and Future Directions. J Digit Imaging 24, 208–222 (2011). https://doi.org/10.1007/s10278-010-9290-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-010-9290-9

Key words

  • Content-based image retrieval
  • imaging informatics
  • information storage and retrieval
  • digital image management
  • decision support