Skip to main content

Semantic Retrieval of Radiological Images with Relevance Feedback

Part of the Lecture Notes in Computer Science book series (LNISA,volume 9059)

Abstract

Content-based image retrieval can assist radiologists by finding similar images in databases as a means to providing decision support. In general, images are indexed using low-level features, and given a new query image, a distance function is used to find the best matches in the feature space. However, using low-level features to capture the appearance of diseases in images is challenging and the semantic gap between these features and the high-level visual concepts in radiology may impair the system performance. In addition, the results of these systems are fixed and cannot be updated based on user’s intention. We present a new framework that enables retrieving similar images based on high-level semantic image annotations and user feedback. In this framework, database images are automatically annotated with semantic terms. Image retrieval is then performed by computing the similarity between image annotations using a new similarity measure, which takes into account both image-based and ontological inter-term similarities. Finally, a relevance feedback mechanism allows the user to iteratively mark the returned answers, informing which images are relevant according to the query. This information is used to infer user-defined inter-term similarities that are then injected in the image similarity measure to produce a new set of retrieved images. We validated this approach for the retrieval of liver lesions from CT images and annotated with terms of the RadLex ontology.

Keywords

  • Image retrieval
  • Riesz wavelets
  • Image annotation
  • RadLex
  • Semantic gap
  • Relevance feedback
  • Computed tomographic (CT) images

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-24471-6_2
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   39.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-24471-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   49.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Notes

  1. 1.

    A dendrogram is built using the Ascendant Hierarchical Clustering (AHC) algorithm [23].

References

  1. Rubin, G.D.: Data explosion: the challenge of multidetector-row CT. Eur. J. Radiol. 36(2), 74–80 (2000)

    CrossRef  Google Scholar 

  2. Aigrain, P., Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: a state-of-the-art review. Multimedia Tools Appl. 3, 179–202 (1996)

    CrossRef  Google Scholar 

  3. Van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1271–1283 (2010)

    CrossRef  Google Scholar 

  4. Yang, W., Lu, Z., Yu, M., Huang, M., Feng, Q., Chen, W.: Content-based retrieval of focal liver lesions using Bag-of-Visual-Words representations of single- and multiphase contrast-enhanced CT images. J. Digit. Imaging 25, 708–719 (2012)

    CrossRef  Google Scholar 

  5. André, B., Vercauteren, T., Buchner, A.M., Wallace, M.B., Ayache, N.: Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans. Med. Imaging 31(6), 1276–1288 (2012)

    CrossRef  Google Scholar 

  6. Mojsilovic, A., Rogowitz, B.: Capturing image semantics with low-level descriptors. In: IEEE ICIP, pp. 18–21 (2001)

    Google Scholar 

  7. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)

    Google Scholar 

  8. Napel, S.A., Beaulieu, C.F., Rodriguez, C., Cui, J., Xu, J., Gupta, A., Korenblum, D., Greenspan, H., Ma, Y., Rubin, D.L.: Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 256(1), 243–252 (2010)

    CrossRef  Google Scholar 

  9. Ma, H., Zhu, J., Lyu, M.R.T., King, I.: Bridging the semantic gap between images and tags. IEEE Trans. Multimedia 12(5), 462–473 (2010)

    CrossRef  Google Scholar 

  10. Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45(1), 346–362 (2012)

    CrossRef  Google Scholar 

  11. Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C.F., Rubin, D.L.: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med. Image Anal. 18(7), 1082–1100 (2014)

    CrossRef  Google Scholar 

  12. Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: a comprehensive review. Multimedia Syst. 8(6), 536–544 (2003)

    CrossRef  Google Scholar 

  13. Cheng, P.C., Chien, B.C., Ke, H.R., Yang, W.P.: A two-level relevance feedback mechanism for image retrieval. Expert Syst. Appl. 34(3), 2193–2200 (2008)

    CrossRef  Google Scholar 

  14. Liu, Y., Zhang, D., Lu, G., Ma, W.Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 40(1), 262–282 (2007)

    CrossRef  MATH  Google Scholar 

  15. Zhang, H., Chen, Z., Li, M., Su, Z.: Relevance feedback and learning in content-based image search. World Wide Web 6(2), 131–155 (2003)

    CrossRef  Google Scholar 

  16. Doulamis, N., Doulamis, A.: Evaluation of relevance feedback schemes in content-based in retrieval systems. Sig. Process. Image Commun. 21(4), 334–357 (2006)

    CrossRef  MATH  Google Scholar 

  17. Kurtz, C., Gançarski, P., Passat, N., Puissant, A.: A hierarchical semantic-based distance for nominal histogram comparison. Data Knowl. Eng. 87(1), 206–225 (2013)

    CrossRef  Google Scholar 

  18. Depeursinge, A., Kurtz, C., Beaulieu, C.F., Rubin, D.L.: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans. Med. Imaging 33(8), 1669–1676 (2014)

    CrossRef  Google Scholar 

  19. Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: ACL, pp. 133–138 (1994)

    Google Scholar 

  20. Al-Mubaid, H., Nguyen, H.A.: A cluster-based approach for semantic similarity in the biomedical domain. In: IEEE EMBC, pp. 2713–2717 (2006)

    Google Scholar 

  21. Kurtz, C., Beaulieu, C.F., Napel, S., Rubin, D.L.: A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J. Biomed. Inf. 49(1), 227–244 (2014)

    CrossRef  Google Scholar 

  22. Kurtz, C., Passat, N., Gançarski, P., Puissant, A.: A histogram semantic-based distance for multiresolution image classification. In: IEEE ICIP, pp. 1157–1160 (2012)

    Google Scholar 

  23. Ward, J.H.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)

    MathSciNet  CrossRef  Google Scholar 

  24. Lux, M.: Content based image retrieval with LIRE. In: ACM MM, pp. 735–738 (2011)

    Google Scholar 

  25. Langlotz, C.P.: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6), 1595–1597 (2006)

    CrossRef  Google Scholar 

  26. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Camille Kurtz .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Kurtz, C., Idoux, PA., Thangali, A., Cloppet, F., Beaulieu, C.F., Rubin, D.L. (2015). Semantic Retrieval of Radiological Images with Relevance Feedback. In: Müller, H., Jimenez del Toro, O., Hanbury, A., Langs, G., Foncubierta Rodriguez, A. (eds) Multimodal Retrieval in the Medical Domain. MRDM 2015. Lecture Notes in Computer Science(), vol 9059. Springer, Cham. https://doi.org/10.1007/978-3-319-24471-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24471-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24470-9

  • Online ISBN: 978-3-319-24471-6

  • eBook Packages: Computer ScienceComputer Science (R0)