Skip to main content

Semantic Retrieval of Radiological Images with Relevance Feedback

  • Conference paper
  • First Online:
Multimodal Retrieval in the Medical Domain (MRDM 2015)


Content-based image retrieval can assist radiologists by finding similar images in databases as a means to providing decision support. In general, images are indexed using low-level features, and given a new query image, a distance function is used to find the best matches in the feature space. However, using low-level features to capture the appearance of diseases in images is challenging and the semantic gap between these features and the high-level visual concepts in radiology may impair the system performance. In addition, the results of these systems are fixed and cannot be updated based on user’s intention. We present a new framework that enables retrieving similar images based on high-level semantic image annotations and user feedback. In this framework, database images are automatically annotated with semantic terms. Image retrieval is then performed by computing the similarity between image annotations using a new similarity measure, which takes into account both image-based and ontological inter-term similarities. Finally, a relevance feedback mechanism allows the user to iteratively mark the returned answers, informing which images are relevant according to the query. This information is used to infer user-defined inter-term similarities that are then injected in the image similarity measure to produce a new set of retrieved images. We validated this approach for the retrieval of liver lesions from CT images and annotated with terms of the RadLex ontology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 34.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 44.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.

    A dendrogram is built using the Ascendant Hierarchical Clustering (AHC) algorithm [23].


  1. Rubin, G.D.: Data explosion: the challenge of multidetector-row CT. Eur. J. Radiol. 36(2), 74–80 (2000)

    Article  Google Scholar 

  2. Aigrain, P., Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: a state-of-the-art review. Multimedia Tools Appl. 3, 179–202 (1996)

    Article  Google Scholar 

  3. Van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1271–1283 (2010)

    Article  Google Scholar 

  4. Yang, W., Lu, Z., Yu, M., Huang, M., Feng, Q., Chen, W.: Content-based retrieval of focal liver lesions using Bag-of-Visual-Words representations of single- and multiphase contrast-enhanced CT images. J. Digit. Imaging 25, 708–719 (2012)

    Article  Google Scholar 

  5. André, B., Vercauteren, T., Buchner, A.M., Wallace, M.B., Ayache, N.: Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans. Med. Imaging 31(6), 1276–1288 (2012)

    Article  Google Scholar 

  6. Mojsilovic, A., Rogowitz, B.: Capturing image semantics with low-level descriptors. In: IEEE ICIP, pp. 18–21 (2001)

    Google Scholar 

  7. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)

    Google Scholar 

  8. Napel, S.A., Beaulieu, C.F., Rodriguez, C., Cui, J., Xu, J., Gupta, A., Korenblum, D., Greenspan, H., Ma, Y., Rubin, D.L.: Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 256(1), 243–252 (2010)

    Article  Google Scholar 

  9. Ma, H., Zhu, J., Lyu, M.R.T., King, I.: Bridging the semantic gap between images and tags. IEEE Trans. Multimedia 12(5), 462–473 (2010)

    Article  Google Scholar 

  10. Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45(1), 346–362 (2012)

    Article  Google Scholar 

  11. Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C.F., Rubin, D.L.: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med. Image Anal. 18(7), 1082–1100 (2014)

    Article  Google Scholar 

  12. Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: a comprehensive review. Multimedia Syst. 8(6), 536–544 (2003)

    Article  Google Scholar 

  13. Cheng, P.C., Chien, B.C., Ke, H.R., Yang, W.P.: A two-level relevance feedback mechanism for image retrieval. Expert Syst. Appl. 34(3), 2193–2200 (2008)

    Article  Google Scholar 

  14. Liu, Y., Zhang, D., Lu, G., Ma, W.Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 40(1), 262–282 (2007)

    Article  MATH  Google Scholar 

  15. Zhang, H., Chen, Z., Li, M., Su, Z.: Relevance feedback and learning in content-based image search. World Wide Web 6(2), 131–155 (2003)

    Article  Google Scholar 

  16. Doulamis, N., Doulamis, A.: Evaluation of relevance feedback schemes in content-based in retrieval systems. Sig. Process. Image Commun. 21(4), 334–357 (2006)

    Article  MATH  Google Scholar 

  17. Kurtz, C., Gançarski, P., Passat, N., Puissant, A.: A hierarchical semantic-based distance for nominal histogram comparison. Data Knowl. Eng. 87(1), 206–225 (2013)

    Article  Google Scholar 

  18. Depeursinge, A., Kurtz, C., Beaulieu, C.F., Rubin, D.L.: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans. Med. Imaging 33(8), 1669–1676 (2014)

    Article  Google Scholar 

  19. Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: ACL, pp. 133–138 (1994)

    Google Scholar 

  20. Al-Mubaid, H., Nguyen, H.A.: A cluster-based approach for semantic similarity in the biomedical domain. In: IEEE EMBC, pp. 2713–2717 (2006)

    Google Scholar 

  21. Kurtz, C., Beaulieu, C.F., Napel, S., Rubin, D.L.: A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J. Biomed. Inf. 49(1), 227–244 (2014)

    Article  Google Scholar 

  22. Kurtz, C., Passat, N., Gançarski, P., Puissant, A.: A histogram semantic-based distance for multiresolution image classification. In: IEEE ICIP, pp. 1157–1160 (2012)

    Google Scholar 

  23. Ward, J.H.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)

    Article  MathSciNet  Google Scholar 

  24. Lux, M.: Content based image retrieval with LIRE. In: ACM MM, pp. 735–738 (2011)

    Google Scholar 

  25. Langlotz, C.P.: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6), 1595–1597 (2006)

    Article  Google Scholar 

  26. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Camille Kurtz .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Kurtz, C., Idoux, PA., Thangali, A., Cloppet, F., Beaulieu, C.F., Rubin, D.L. (2015). Semantic Retrieval of Radiological Images with Relevance Feedback. In: Müller, H., Jimenez del Toro, O., Hanbury, A., Langs, G., Foncubierta Rodriguez, A. (eds) Multimodal Retrieval in the Medical Domain. MRDM 2015. Lecture Notes in Computer Science(), vol 9059. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24470-9

  • Online ISBN: 978-3-319-24471-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics