Semantic Retrieval of Radiological Images with Relevance Feedback

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9059)


Content-based image retrieval can assist radiologists by finding similar images in databases as a means to providing decision support. In general, images are indexed using low-level features, and given a new query image, a distance function is used to find the best matches in the feature space. However, using low-level features to capture the appearance of diseases in images is challenging and the semantic gap between these features and the high-level visual concepts in radiology may impair the system performance. In addition, the results of these systems are fixed and cannot be updated based on user’s intention. We present a new framework that enables retrieving similar images based on high-level semantic image annotations and user feedback. In this framework, database images are automatically annotated with semantic terms. Image retrieval is then performed by computing the similarity between image annotations using a new similarity measure, which takes into account both image-based and ontological inter-term similarities. Finally, a relevance feedback mechanism allows the user to iteratively mark the returned answers, informing which images are relevant according to the query. This information is used to infer user-defined inter-term similarities that are then injected in the image similarity measure to produce a new set of retrieved images. We validated this approach for the retrieval of liver lesions from CT images and annotated with terms of the RadLex ontology.


Image retrieval Riesz wavelets Image annotation RadLex Semantic gap Relevance feedback Computed tomographic (CT) images 


  1. 1.
    Rubin, G.D.: Data explosion: the challenge of multidetector-row CT. Eur. J. Radiol. 36(2), 74–80 (2000)CrossRefGoogle Scholar
  2. 2.
    Aigrain, P., Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: a state-of-the-art review. Multimedia Tools Appl. 3, 179–202 (1996)CrossRefGoogle Scholar
  3. 3.
    Van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1271–1283 (2010)CrossRefGoogle Scholar
  4. 4.
    Yang, W., Lu, Z., Yu, M., Huang, M., Feng, Q., Chen, W.: Content-based retrieval of focal liver lesions using Bag-of-Visual-Words representations of single- and multiphase contrast-enhanced CT images. J. Digit. Imaging 25, 708–719 (2012)CrossRefGoogle Scholar
  5. 5.
    André, B., Vercauteren, T., Buchner, A.M., Wallace, M.B., Ayache, N.: Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans. Med. Imaging 31(6), 1276–1288 (2012)CrossRefGoogle Scholar
  6. 6.
    Mojsilovic, A., Rogowitz, B.: Capturing image semantics with low-level descriptors. In: IEEE ICIP, pp. 18–21 (2001)Google Scholar
  7. 7.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)Google Scholar
  8. 8.
    Napel, S.A., Beaulieu, C.F., Rodriguez, C., Cui, J., Xu, J., Gupta, A., Korenblum, D., Greenspan, H., Ma, Y., Rubin, D.L.: Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 256(1), 243–252 (2010)CrossRefGoogle Scholar
  9. 9.
    Ma, H., Zhu, J., Lyu, M.R.T., King, I.: Bridging the semantic gap between images and tags. IEEE Trans. Multimedia 12(5), 462–473 (2010)CrossRefGoogle Scholar
  10. 10.
    Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45(1), 346–362 (2012)CrossRefGoogle Scholar
  11. 11.
    Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C.F., Rubin, D.L.: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med. Image Anal. 18(7), 1082–1100 (2014)CrossRefGoogle Scholar
  12. 12.
    Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: a comprehensive review. Multimedia Syst. 8(6), 536–544 (2003)CrossRefGoogle Scholar
  13. 13.
    Cheng, P.C., Chien, B.C., Ke, H.R., Yang, W.P.: A two-level relevance feedback mechanism for image retrieval. Expert Syst. Appl. 34(3), 2193–2200 (2008)CrossRefGoogle Scholar
  14. 14.
    Liu, Y., Zhang, D., Lu, G., Ma, W.Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 40(1), 262–282 (2007)CrossRefzbMATHGoogle Scholar
  15. 15.
    Zhang, H., Chen, Z., Li, M., Su, Z.: Relevance feedback and learning in content-based image search. World Wide Web 6(2), 131–155 (2003)CrossRefGoogle Scholar
  16. 16.
    Doulamis, N., Doulamis, A.: Evaluation of relevance feedback schemes in content-based in retrieval systems. Sig. Process. Image Commun. 21(4), 334–357 (2006)CrossRefzbMATHGoogle Scholar
  17. 17.
    Kurtz, C., Gançarski, P., Passat, N., Puissant, A.: A hierarchical semantic-based distance for nominal histogram comparison. Data Knowl. Eng. 87(1), 206–225 (2013)CrossRefGoogle Scholar
  18. 18.
    Depeursinge, A., Kurtz, C., Beaulieu, C.F., Rubin, D.L.: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans. Med. Imaging 33(8), 1669–1676 (2014)CrossRefGoogle Scholar
  19. 19.
    Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: ACL, pp. 133–138 (1994)Google Scholar
  20. 20.
    Al-Mubaid, H., Nguyen, H.A.: A cluster-based approach for semantic similarity in the biomedical domain. In: IEEE EMBC, pp. 2713–2717 (2006)Google Scholar
  21. 21.
    Kurtz, C., Beaulieu, C.F., Napel, S., Rubin, D.L.: A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J. Biomed. Inf. 49(1), 227–244 (2014)CrossRefGoogle Scholar
  22. 22.
    Kurtz, C., Passat, N., Gançarski, P., Puissant, A.: A histogram semantic-based distance for multiresolution image classification. In: IEEE ICIP, pp. 1157–1160 (2012)Google Scholar
  23. 23.
    Ward, J.H.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Lux, M.: Content based image retrieval with LIRE. In: ACM MM, pp. 735–738 (2011)Google Scholar
  25. 25.
    Langlotz, C.P.: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6), 1595–1597 (2006)CrossRefGoogle Scholar
  26. 26.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.LIPADEUniversity Paris DescartesParisFrance
  2. 2.Department of Radiology, School of MedicineStanford UniversityStanfordUSA

Personalised recommendations