Abstract
Content-based image retrieval can assist radiologists by finding similar images in databases as a means to providing decision support. In general, images are indexed using low-level features, and given a new query image, a distance function is used to find the best matches in the feature space. However, using low-level features to capture the appearance of diseases in images is challenging and the semantic gap between these features and the high-level visual concepts in radiology may impair the system performance. In addition, the results of these systems are fixed and cannot be updated based on user’s intention. We present a new framework that enables retrieving similar images based on high-level semantic image annotations and user feedback. In this framework, database images are automatically annotated with semantic terms. Image retrieval is then performed by computing the similarity between image annotations using a new similarity measure, which takes into account both image-based and ontological inter-term similarities. Finally, a relevance feedback mechanism allows the user to iteratively mark the returned answers, informing which images are relevant according to the query. This information is used to infer user-defined inter-term similarities that are then injected in the image similarity measure to produce a new set of retrieved images. We validated this approach for the retrieval of liver lesions from CT images and annotated with terms of the RadLex ontology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A dendrogram is built using the Ascendant Hierarchical Clustering (AHC) algorithm [23].
References
Rubin, G.D.: Data explosion: the challenge of multidetector-row CT. Eur. J. Radiol. 36(2), 74–80 (2000)
Aigrain, P., Zhang, H., Petkovic, D.: Content-based representation and retrieval of visual media: a state-of-the-art review. Multimedia Tools Appl. 3, 179–202 (1996)
Van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1271–1283 (2010)
Yang, W., Lu, Z., Yu, M., Huang, M., Feng, Q., Chen, W.: Content-based retrieval of focal liver lesions using Bag-of-Visual-Words representations of single- and multiphase contrast-enhanced CT images. J. Digit. Imaging 25, 708–719 (2012)
André, B., Vercauteren, T., Buchner, A.M., Wallace, M.B., Ayache, N.: Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans. Med. Imaging 31(6), 1276–1288 (2012)
Mojsilovic, A., Rogowitz, B.: Capturing image semantics with low-level descriptors. In: IEEE ICIP, pp. 18–21 (2001)
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)
Napel, S.A., Beaulieu, C.F., Rodriguez, C., Cui, J., Xu, J., Gupta, A., Korenblum, D., Greenspan, H., Ma, Y., Rubin, D.L.: Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 256(1), 243–252 (2010)
Ma, H., Zhu, J., Lyu, M.R.T., King, I.: Bridging the semantic gap between images and tags. IEEE Trans. Multimedia 12(5), 462–473 (2010)
Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45(1), 346–362 (2012)
Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C.F., Rubin, D.L.: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med. Image Anal. 18(7), 1082–1100 (2014)
Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: a comprehensive review. Multimedia Syst. 8(6), 536–544 (2003)
Cheng, P.C., Chien, B.C., Ke, H.R., Yang, W.P.: A two-level relevance feedback mechanism for image retrieval. Expert Syst. Appl. 34(3), 2193–2200 (2008)
Liu, Y., Zhang, D., Lu, G., Ma, W.Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 40(1), 262–282 (2007)
Zhang, H., Chen, Z., Li, M., Su, Z.: Relevance feedback and learning in content-based image search. World Wide Web 6(2), 131–155 (2003)
Doulamis, N., Doulamis, A.: Evaluation of relevance feedback schemes in content-based in retrieval systems. Sig. Process. Image Commun. 21(4), 334–357 (2006)
Kurtz, C., Gançarski, P., Passat, N., Puissant, A.: A hierarchical semantic-based distance for nominal histogram comparison. Data Knowl. Eng. 87(1), 206–225 (2013)
Depeursinge, A., Kurtz, C., Beaulieu, C.F., Rubin, D.L.: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans. Med. Imaging 33(8), 1669–1676 (2014)
Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: ACL, pp. 133–138 (1994)
Al-Mubaid, H., Nguyen, H.A.: A cluster-based approach for semantic similarity in the biomedical domain. In: IEEE EMBC, pp. 2713–2717 (2006)
Kurtz, C., Beaulieu, C.F., Napel, S., Rubin, D.L.: A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J. Biomed. Inf. 49(1), 227–244 (2014)
Kurtz, C., Passat, N., Gançarski, P., Puissant, A.: A histogram semantic-based distance for multiresolution image classification. In: IEEE ICIP, pp. 1157–1160 (2012)
Ward, J.H.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)
Lux, M.: Content based image retrieval with LIRE. In: ACM MM, pp. 735–738 (2011)
Langlotz, C.P.: RadLex: a new method for indexing online educational materials. RadioGraphics 26(6), 1595–1597 (2006)
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Kurtz, C., Idoux, PA., Thangali, A., Cloppet, F., Beaulieu, C.F., Rubin, D.L. (2015). Semantic Retrieval of Radiological Images with Relevance Feedback. In: Müller, H., Jimenez del Toro, O., Hanbury, A., Langs, G., Foncubierta Rodriguez, A. (eds) Multimodal Retrieval in the Medical Domain. MRDM 2015. Lecture Notes in Computer Science(), vol 9059. Springer, Cham. https://doi.org/10.1007/978-3-319-24471-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-24471-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24470-9
Online ISBN: 978-3-319-24471-6
eBook Packages: Computer ScienceComputer Science (R0)