Skip to main content

Ontology of Gaps in Content-Based Image Retrieval

Journal of Digital Imaging volume 22, pages 202–215 (2009)Cite this article

Abstract

Content-based image retrieval (CBIR) is a promising technology to enrich the core functionality of picture archiving and communication systems (PACS). CBIR has a potential for making a strong impact in diagnostics, research, and education. Research as reported in the scientific literature, however, has not made significant inroads as medical CBIR applications incorporated into routine clinical medicine or medical research. The cause is often attributed (without supporting analysis) to the inability of these applications in overcoming the “semantic gap.” The semantic gap divides the high-level scene understanding and interpretation available with human cognitive capabilities from the low-level pixel analysis of computers, based on mathematical processing and artificial intelligence methods. In this paper, we suggest a more systematic and comprehensive view of the concept of “gaps” in medical CBIR research. In particular, we define an ontology of 14 gaps that addresses the image content and features, as well as system performance and usability. In addition to these gaps, we identify seven system characteristics that impact CBIR applicability and performance. The framework we have created can be used a posteriori to compare medical CBIR systems and approaches for specific biomedical image domains and goals and a priori during the design phase of a medical CBIR application, as the systematic analysis of gaps provides detailed insight in system comparison and helps to direct future research.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1

References

  1. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R: Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380, 2000

    Article  Google Scholar 

  2. Niblack W, Barber R, Equitz W, Flickner M, Glasman E, Petkovic D, Yanker P, Faloutsos C, Taubin G: The QBIC project: Querying images by content using color, texture, and shape. Proc SPIE 1908:173–187, 1993

    Article  Google Scholar 

  3. Tagare HD, Jaffe CC, Duncan J: Medical image databases: A content-based retrieval approach. J Am Med Inform Assoc—JAMIA 4(3):184–198, 1997

    CAS  Google Scholar 

  4. Tang LHA, Hanka R, Ip HHS: A review of intelligent content-based indexing and browsing of medical images. Health Inform J 1(5):40–49, 1999

    Article  Google Scholar 

  5. Müller H, Michoux N, Bandon D, Geissbuhler A: A review of content-based image retrieval systems in medical applications. Clinical benefits and future directions. Int J Med Inform 73(1):1–23, 2004

    PubMed  Article  Google Scholar 

  6. Qi H, Snyder WE: Content-based image retrieval in picture archiving and communications systems. J Digit Imaging 12(2 Suppl 1):81–83, 1999

    PubMed  CAS  Article  Google Scholar 

  7. Müller H, Rosset A, Garcia A, Vallée JP, Geissbuhler A: Informatics in radiology (Inforad): Benefits of content-based visual data accessing radiology. Radiographics 25(3):849–858, 2005

    PubMed  Article  Google Scholar 

  8. Lehmann TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB: Content-based image retrieval in medical applications. Methods Inf Med 43(4):354–361, 2004

    PubMed  CAS  Google Scholar 

  9. Müller H, Clough P, Hersh W, Deselaers T, Lehmann T, Geissbuhler A: Evaluation axes for medical image retrieval systems—the ImageCLEF experience. Proc. 13th Annual ACM International Conference on Multimedia 1014–1022, 2005

  10. Eakins J, Graham M: Content-based image retrieval. Report 39, JISC Technology Applications Programme, University of Northumbria at Newcastle, October 1999

  11. Lehmann TM: Digitale Bildverarbeitung für Routineanwendungen. Evaluierung und Integration am Beispiel der Medizin. Deutscher Universitäts-Verlag, GWV Fachverlage, Wiesbaden, 2005 (in German)

  12. Enser P, Sandom C: Towards a comprehensive survey of the semantic gap in visual image retrieval. Lect Notes Comput Sci 2728:291–299, 2003

    Article  Google Scholar 

  13. Bosch A, Muñoz X, Martí R: Which is the best way to organize/classify images by content? Image Vis Comput 25:778–791, 2007

    Article  Google Scholar 

  14. Raicu DS, Varubangkul E, Cisneros JG, Furst JD, Channin DS, Armato III, SG: Semantics and image content integration for pulmonary nodule interpretation in thoracic computer tomography. Proceedings SPIE 6512:OS1–OS12, 2007

    Google Scholar 

  15. Tagare HD, Vos FM, Jaffe CC, Duncan JS: Arrangement: a spatial relation between parts for evaluating similarity of tomographic section. IEEE Trans Pattern Anal Mach Intell 17(9):880–893, 1995

    Article  Google Scholar 

  16. Mortensen EN, Barrett WA: Intelligent scissors for image composition. Proc SIGGRAPH 191–198, 1995

  17. Lee YJ, Bajcsy P: An information gathering system for medical image inspection. Proc SPIE 5748:374–381, 2005

    Article  Google Scholar 

  18. Traina C, Jr., Traina AJM, Araujo MRB, Bueno JM, Chino FJT, Razente H, Azevedo-Marques PM: Using an image-extended relational database to support content-based image retrieval in a PACS. Comput Methods Programs Biomed 80(Suppl. 1):S71–S83, 2005

    PubMed  Article  Google Scholar 

  19. Xue Z, Antani S, Long R, Thoma G: Comparative performance analysis of cervix ROI extraction and specular reflection removal algorithms for uterine cervix image analysis. Proc SPIE 6512:4I1–4I9, 2007

    Google Scholar 

  20. Stanley RJ, Moss RH, Van Stoecker W, Aggarwal C: A fuzzy-based histogram analysis technique for skin lesion discrimination in dermatology clinical images. Comput Med Imaging Graph 27:387–96, 2003

    PubMed  Article  Google Scholar 

  21. Tao Y, Lo S-CB, Freedman MT, Xuan J: A preliminary study of content-based mammographic masses retrieval. Proc. SPIE 6514:1Z–1–12, 2007

    Google Scholar 

  22. Kim J, Cai W, Feng D, Wu H: A new way for multidimensional medical data management: volume of interest (VOI)-based retrieval of medical images with visual and functional features. IEEE Trans Inf Technol Biomed 10(3):598–607, 2006

    PubMed  Article  Google Scholar 

  23. Howarth P, Yavlinsky A, Heesch D, Ruger S: Medical image retrieval using texture, locality and colour. Lect Notes Comput Sci 3491:740–749, 2005

    Article  Google Scholar 

  24. Xue Z, Antani SK, Long LR, Jeronimo J, Thoma GR: Investigating CBIR techniques for cervicographic images. Proc AMIA 2007 (in press)

  25. Squire DM, Muller W, Muller H, Raki J: Content-based query of image databases, inspirations from text retrieval: inverted files, frequency-based weights and relevance feedback. Proc 10th Scandinavian Conference on Image Analysis (SCIA’99)

  26. Huston L, Sukthankar R, Wickremesinghe R, Satyanarayanan M, Ganger GR, Riedel E, Ailamake A. Diamond: a storage architecture for early discard in interactive search. Proc 3rd USENIX Conference on File and Storage Technologies (FAST’04)

  27. Qian X, Tagare HD: Optimal embedding for shape indexing in medical image databases. Lect Notes Comput Sci 3750:377–84, 2005

    Article  Google Scholar 

  28. Korn P, Sidiropoulous N, Faloutsos C, Siegel E, Protopapas Z: Fast and effective retrieval of medical tumor shapes. IEEE Trans Knowl Data Eng 10(6):889–904, 1998

    Article  Google Scholar 

  29. Xu X, Lee DJ, Antani S, Long LR: Pre-indexing for fast partial shape matching of vertebrae images. Proc 19th International Symposium on Computer-Based Medical Systems (CBMS 2006), 105–10

  30. Antani S, Cheng J, Long J, Long LR, Thoma GR: Medical validation and CBIR of spine x-ray images over the Internet. Proc SPIE 6061 0J(1–9), 2006

  31. Xu X, Lee DJ, Antani SK, Long LR. Relevance feedback for spine x-ray retrieval. Proc 18th International Symposium on Computer-Based Medical Systems 197–202, 2005

  32. Deserno TM, Güld MO, Plodowski B, Spitzer K, Wein BB, Schubert H, Ney H, Seidl T: Extended query refinement for medical image retrieval. Journal of Digital Imaging 2007 (in press). DOI 10.1007/s10278-007-90374

Download references

Acknowledgment

This research was supported [in part] by the Intramural Research Program of the U.S. National Institutes of Health (NIH), U.S. National Library of Medicine (NLM), and the U.S. Lister Hill National Center for Biomedical Communications (LHNCBC).

Author information

Affiliations

  1. Department of Medical Informatics, Aachen University of Technology (RWTH), Pauwelsstr. 30, 52057, Aachen, Germany

    Thomas M. Deserno

  2. US National Library of Medicine, US National Institutes of Health, 8600 Rockville Pike, Bethesda, MD, 20894, USA

    Thomas M. Deserno, Sameer Antani & Rodney Long

Authors
  1. Thomas M. Deserno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sameer Antani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rodney Long
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas M. Deserno.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Deserno, T.M., Antani, S. & Long, R. Ontology of Gaps in Content-Based Image Retrieval. J Digit Imaging 22, 202–215 (2009). https://doi.org/10.1007/s10278-007-9092-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-007-9092-x

Key words

  • Content-based image retrieval (CBIR)
  • pattern recognition
  • picture archiving and communication systems (PACS)
  • information system integration
  • data mining
  • information retrieval
  • semantic gap