Multimodal Medical Image Retrieval

  • Ivan KitanovskiEmail author
  • Katarina Trojacanec
  • Ivica Dimitrovski
  • Suzana Loskovska
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 207)


Medical image retrieval is one of the crucial tasks in everyday medical practices. This paper investigates three forms of medical image retrieval: text, visual and multimodal retrieval. We investigate by evaluating different weighting models for text retrieval. In the case of the visual retrieval, we focused on extracting low-level features and examining their performance. For, the multimodal retrieval we used late fusion to combine the best text and visual results. We found that the choice of weighting model for text retrieval dramatically influences the outcome of the multimodal retrieval. The results from the text and visual retrieval are fused using linear combination, which is among the simplest and most frequently used methods. Our results clearly show that the fusion of text and visual retrieval with an appropriate fusion technique improves the retrieval performance.


Information Retrieval Medical Imaging Content-based Image Retrieval Medical Image Retrieval 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kalpathy–Cramer, J., Muller, H., Bedrick, S., Eggel, I., de Herrera, A.G.S., Tsikrika, T.: Overview of the CLEF 2011 medical image classification and retrieval tasks (2011)Google Scholar
  2. 2.
    Sonka, M., Hlavac, V., Boyle, R., et al.: Image processing, analysis, and machine vision, vol. 2. PWS publishing Pacific Grove, CA (1999)Google Scholar
  3. 3.
    Deb, S., Zhang, Y.: An overview of content-based image retrieval techniques. In: 18th International Conference on Advanced Information Networking and Applications, vol. 1, pp. 59–64 (2004)Google Scholar
  4. 4.
    Müller, H., Kalpathy–Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Kahn Jr., C.E., Hersh, W.: Overview of the CLEF 2009 medical image retrieval track. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T., et al. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 72–84. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Alpkocak, A., Ozturkmenoglu, O., Berber, T., Vahid, A.H., Hamed, R.G.: DEMIR at ImageCLEFMed 2011: Evaluation of Fusion Techniques for Multimodal Content-based Medical Image Retrieval. In: 12th Workshop of the Cross-Language Evaluation Forum (CLEF), Amsterdam, Netherlands (2011)Google Scholar
  6. 6.
    Csurka, G., Clinchant, S., Jacquet, G.: XRCE’s Participation at Medical Image Modality Classification and Ad-hoc Retrieval Tasks of ImageCLEF (2011)Google Scholar
  7. 7.
    Gkoufas, Y., Morou, A., Kalamboukis, T.: IPL at ImageCLEF 2011 Medical Retrieval Task. Working Notes of CLEF (2011)Google Scholar
  8. 8.
    Castellanos, A., Benavent, X., Benavent, J., Garcia-Serrano, A.: UNED-UV at Medical Retrieval Task of ImageCLEF (2011)Google Scholar
  9. 9.
    Amati, G., Van Rijsbergen, C.J.: Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS) 20, 357–389 (2002)CrossRefGoogle Scholar
  10. 10.
    Hiemstra, D.: A probabilistic justification for using tf-idf term weighting in information retrieval. International Journal on Digital Libraries 3, 131–139 (2000)CrossRefGoogle Scholar
  11. 11.
    Chatzichristofis, S.A., Boutalis, Y.S.: CEDD: Color and edge directivity descriptor: A compact descriptor for image indexing and retrieval. In: Gasteratos, A., Vincze, M., Tsotsos, J.K. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 312–322. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Chatzichristofis, S.A., Boutalis, Y.S.: Fcth: Fuzzy color and texture histogram-a low level feature for accurate image retrieval. In: Ninth International Workshop on Image Analysis for Multimedia Interactive Services, pp. 191–196 (2008)Google Scholar
  13. 13.
    Chatzichristofis, S.A., Boutalis, Y.S.: Content based radiology image retrieval using a fuzzy rule based scalable composite descriptor. Multimedia Tools and Applications 46, 493–519 (2010)CrossRefGoogle Scholar
  14. 14.
    Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16, 345–379 (2010)CrossRefGoogle Scholar
  15. 15.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Johnson, D.: Terrier information retrieval platform. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 517–519. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  16. 16.
    Porter, M.F.: An algorithm for suffix stripping (1980)Google Scholar
  17. 17.
    Chatzichristofis, S.A., Boutalis, Y.S., Lux, M.: Img (rummager): An interactive content based image retrieval system. In: Second International Workshop on Similarity Search and Applications, pp. 151–153 (2009)Google Scholar
  18. 18.
    Croft, W.B.: Combining approaches to information retrieval. In: Advances in Information Retrieval, pp. 1–36 (2002)Google Scholar
  19. 19.
    Jain, A., Nandakumar, K., Ross, A.: Score normalization in multimodal biometric systems. Pattern Recognition 38, 2270–2285 (2005)CrossRefGoogle Scholar
  20. 20.
    Manning, C.D., Raghavan, P., Schutze, H.: Introduction to information retrieval. Cambridge University Press, Cambridge (2008)zbMATHCrossRefGoogle Scholar
  21. 21.
    He, B., Ounis, I.: Term frequency normalisation tuning for BM25 and DFR models. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 200–214. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ivan Kitanovski
    • 1
    Email author
  • Katarina Trojacanec
    • 1
  • Ivica Dimitrovski
    • 1
  • Suzana Loskovska
    • 1
  1. 1.Faculty of Computer Science and EngineeringSs. Cyril and Methodious UniversitySkopjeMacedonia

Personalised recommendations