Reducing the semantic gap in content-based image retrieval in mammography with relevance feedback and inclusion of expert knowledge

  • Paulo Mazzoncini de Azevedo-MarquesEmail author
  • Natália Abdala Rosa
  • Agma Juci Machado Traina
  • Caetano TrainaJr
  • Sérgio Koodi Kinoshita
  • Rangaraj Mandayam Rangayyan
Original Article



We investigate the use of relevance feedback (RFb) and the inclusion of expert knowledge to reduce the semantic gap in content-based image retrieval (CBIR) of mammograms.

Materials and methods

Tests were conducted with radiologists, in which their judgment of the relevance of the retrieved images was used with techniques of query-point movement to incorporate RFb. The measures of similarity of images used for CBIR were based upon textural characteristics and the distribution of density of fibroglandular tissue in the breast. The features used include statistics of the gray-level histogram, texture features based upon the gray-level co-occurrence matrix, moment-based features, measures computed in the Radon domain, and granulometric measures. Queries for CBIR with RFb were executed by three radiologists. The performance of CBIR was measured in terms of precision of retrieval and a measure of relevance-weighted precision (RWP) of retrieval.


The results indicate improvement due to RFb of up to 62% in precision and 39% in RWP.


The gain in performance of CBIR with RFb depended upon the BI-RADS breast density index of the query mammographic image, with greater improvement in cases of mammograms with higher density.


Computer-assisted radiographic image interpretation Computer-assisted image processing Automated pattern recognition Information retrieval 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Müller H, Michoux N, Bandon D and Geissbuhler A (2004). A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int J Med Inform 73: 1–23 PubMedCrossRefGoogle Scholar
  2. 2.
    El-Naqa I, Yang Y, Galatsanos NP, Nishikawa RM and Wernick MN (2004). A similarity learning approach to content-based image retrieval: application to digital mammography. IEEE Trans Med Imaging 23: 1233–1244 PubMedCrossRefGoogle Scholar
  3. 3.
    Li Q, Li F, Shiraishi J, Katsuragawa S, Sone S and Doi K (2003). Investigation of new psychophysical measures for evaluation of similar images on thoracic computed tomography for distinction between benign and malignant nodules. Med Phys 30: 2584–2593 PubMedCrossRefGoogle Scholar
  4. 4.
    Tourassi GD, Eltonsy NH, Graham J, Floyd CE and Elmaghraby AS (2005). Feature and knowledge based analysis for reduction of false positives in the computerized detection of masses in screening mammography. Conf Proc IEEE Eng Med Biol Soc 6: 6524–6527 PubMedGoogle Scholar
  5. 5.
    Kinoshita SK, Azevedo-Marques PM, Rodrigues JAH, Rangayyan RM and Pereira RR (2007). Content-based retrieval of mammograms using visual features related to breast density patterns. J Digit Imaging 20: 172–190 PubMedCrossRefGoogle Scholar
  6. 6.
    Azevedo-Marques PM, Honda MH, Rodrigues JAH, Santos RR, Traina AJM, Traina Júnior C and Bueno JM (2002). Recuperação de Imagem Baseada em Conteúdo: Uso de Atributos de Textura para Caracterização de Microcalcificações Mamográficas. Rev Bras Radiol 35: 93–98 Google Scholar
  7. 7.
    Haralick RM (1979). Statistical and structural approaches to texture. Proc IEEE 1979: 786–804 CrossRefGoogle Scholar
  8. 8.
    Alto H, Rangayyan RM, Desautels JEL (2005) Content-based retrieval and analysis of mammographic masses. J Electron Imaging 14:023016:1–17Google Scholar
  9. 9.
    Muramatsu C, Li Q, Suzuki K, Schmidt RA, Shiraishi J, Newstead GM and Doi K (2005). Investigation of psychophysical measure for evaluation of similar images for mammographic masses: preliminary results. Med Phys 32: 2295–2304 PubMedCrossRefGoogle Scholar
  10. 10.
    Tao EY, Sklansky J (1996) Analysis of mammograms aided by database of images of calcifications and textures. In: Medical imaging: image processing. Proceedings of SPIE, pp 988–995Google Scholar
  11. 11.
    Ornes CJ, Valentino DJ, Yoon H-J, Eisenman JI, Sklansky J (2001) Search engine for remote database-aided interpretation of digitized mammograms. In: Siegel El, Huang HK (eds) Medical imaging: PACS and integrated medical information systems: design and evaluation, pp 132–137Google Scholar
  12. 12.
    El-Naqa I, Yang Y, Galatsanos NP, Wernick MN (2002) Content-based image retrieval for digital mammography. In: Yongyi Y (eds) IEEE proceedings of the international conference on image processing, Rochester, pp 141–144Google Scholar
  13. 13.
    Nakagawa T, Hara T, Fujita H, Iwase T, Endo T (2002) Image retrieval system of mammographic masses by using local pattern matching technique. In: Peitgen H-O (ed) Digital mammography. Proceedings of the IWDM 2002 Bremen, Germany. Springer, New York, pp 562–565Google Scholar
  14. 14.
    Nakagawa T, Hara T, Fujita H, Iwase T, Endo T (2002) Development of a computer-aided sketch system for mammograms. In: Peitgen H-O (ed) Digital mammography. Proceedings of the IWDM 2002 Bremen, Germany. Springer, New York, pp 581–583Google Scholar
  15. 15.
    Baeza-Yates RA and Ribeiro-Neto BA (1999). Modern information retrieval. Addison-Wesley, Wokingham Google Scholar
  16. 16.
    Ortega-Binderberger M and Mehrotra S (2003). Relevance feedback in multimedia databases. In: Furht, B and Marques, O (eds) Handbook of video databases—design and applications, pp 511–536. CRC Press, Boca Raton Google Scholar
  17. 17.
    Zhou XS and Huang TS (2003). Relevance feedback in image retrieval: A comprehensive review. Multimed Syst 8: 536–544 CrossRefGoogle Scholar
  18. 18.
    ACR (2003) BI-RADS® —mammography, 4th edn. American College of Radiology, RestonGoogle Scholar
  19. 19.
    Haralick RM, Shanmugam K and Dinstein I (1973). Textural features for image classification. IEEE Trans Syst Man Cybern 3: 610–621 CrossRefGoogle Scholar
  20. 20.
    Gonzalez RC and Woods RE (2007). Digital image processing. Prentice Hall, Englewood Cliffs Google Scholar
  21. 21.
    Ramm AG and Katsevich AI (1996). The radon transform and local tomography. CRC Press, Boca Raton Google Scholar
  22. 22.
    Hu M-K (1962). Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8: 179–187 Google Scholar
  23. 23.
    Traina C Jr, Traina AJM, Faloutsos C and Seeger B (2002). Fast indexing and visualization of metric datasets using slim-trees. IEEE Trans Knowl Data Eng 14: 244–260CrossRefGoogle Scholar
  24. 24.
    Rocchio JJ (1971). Relevance feedback in information retrieval. In: Cliffs, E (eds) The SMART retrieval system: experiments in automatic document processing, pp 313–323. Prentice Hall, Englewood Cliffs Google Scholar
  25. 25.
    Traina A, Marques J, Traina C (2006) Fighting the semantic gap on CBIR systems through new relevance feedback techniques. In: CBMS ’06: proceedings of the 19th IEEE international symposium on computer-based medical systems. IEEE Computer Society, Salt Lake City, pp 881–886Google Scholar

Copyright information

© CARS 2008

Authors and Affiliations

  • Paulo Mazzoncini de Azevedo-Marques
    • 1
    Email author
  • Natália Abdala Rosa
    • 1
  • Agma Juci Machado Traina
    • 2
  • Caetano TrainaJr
    • 2
  • Sérgio Koodi Kinoshita
    • 1
  • Rangaraj Mandayam Rangayyan
    • 3
  1. 1.Image Science and Medical Physics Center, Internal Medicine Department, Faculty of Medicine of Ribeirão PretoUniversity of São PauloRibeirão PretoBrazil
  2. 2.Institute of Mathematics and Computing SciencesUniversity of São PauloSão CarlosBrazil
  3. 3.Department of Electrical and Computer Engineering, Schulich School of EngineeringUniversity of CalgaryCalgaryCanada

Personalised recommendations