Medical Image Retrieval Using Multimodal Data

  • Ivan Kitanovski
  • Ivica Dimitrovski
  • Gjorgji Madjarov
  • Suzana Loskovska
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8777)


In this paper we propose a system for medical image retrieval using multimodal data. The system can be separated in an off-line and on-line phase. The off-line phase deals with modality classification of the images by their visual content. For this part we use state-of-the-art opponentSIFT visual features to describe the image content, as for the classification we use SVMs. The modality classification labels all images in the database with their corresponding modality. The off-line phase, also, implements the text-based retrieval structure of the system. In this part we index the text associated with the images using the open-source search engine Terrier IR. In the on-line phase the retrieval is performed. In this phase the system receives a text query. The system processes the query and performs the text-based retrieval with Terrier IR and the initial results are generated. Afterwards, the images in the initial results are re-ranked based on their modality and the final results are provided. Our system was evaluated against the standardized ImageCLEF 2013 medical dataset. Our system reported results with a mean average precision of 0.32, which is state-of-the-art performance on the dataset.


medical image retrieval retrieval in medical texts image modality classification visual image descriptors 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Choplin, R., Boehme, J., Maynard, C.: Picture archiving and communication systems: an overview. Radiographics 12(1), 127–129 (1992)CrossRefGoogle Scholar
  2. 2.
    de Herrera, A.G.S., Kalpathy-Cramer, J., Fushman, D.D., Antani, S., Müller, H.: Overview of the imageclef 2013 medical tasks. In: Working notes of CLEF 2013 (2013)Google Scholar
  3. 3.
    Lehmann, T.M., Wein, B.B., Dahmen, J., Bredno, J., Vogelsang, F., Kohnen, M.: Content-based image retrieval in medical applications: a novel multistep approach. In: Proceedings of SPIE: Storage and Retrieval for Media Databases, vol. 3972, pp. 312–320 (2000)Google Scholar
  4. 4.
    Shyu, C.-R., Brodley, C.E., Kak, A.C., Kosaka, A., Aisen, A.M., Broderick, L.S.: Assert: A physician-in-the-loop content-based retrieval system for HRCT image databases. Computer Vision and Image Understanding 75(12), 111–132 (1999)CrossRefGoogle Scholar
  5. 5.
    Simonyan, K., Modat, M., Ourselin, S., Cash, D., Criminisi, A., Zisserman, A.: Immediate ROI search for 3-D medical images. In: Greenspan, H., Müller, H., Syeda-Mahmood, T. (eds.) MCBR-CDS 2012. LNCS, vol. 7723, pp. 56–67. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  6. 6.
    El-Naqa, I., Yang, Y., Galatsanos, N.P., Nishikawa, R.M., Wernick, M.N.: A similarity learning approach to content-based image retrieval: application to digital mammography. IEEE Transactions on Medical Imaging 23(10), 1233–1244 (2004)CrossRefGoogle Scholar
  7. 7.
    Müller, H., de Herrera, A.G.S., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S., Eggel, I.: Overview of the imageclef 2012 medical image retrieval and classification tasks. In: CLEF (Online Working Notes/Labs/Workshop) (2012)Google Scholar
  8. 8.
    Medical retrieval task, (accessed: July 03, 2014)
  9. 9.
    Guld, M.O., Kohnen, M., Keysers, D., Schubert, H., Wein, B.B., Bredno, J., Lehmann, T.M.: Quality of DICOM header information for image categorization. In: Medical Imaging 2002: PACS and Integrated Medical Information Systems: Design and Evaluation, SPIE, vol. 4685, pp. 280–287 (2002)Google Scholar
  10. 10.
    van de Sande, K., Gevers, T., Snoek, C.: Evaluating color fescriptors for object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(9), 1582–1596 (2010)CrossRefGoogle Scholar
  11. 11.
    Hearst, M.A., Divoli, A., Guturu, H., Ksikes, A., Nakov, P., Wooldridge, M.A., Ye, J.: Biotext search engine: beyond abstract search. Bioinformatics 23(16), 2196–2197 (2007)CrossRefGoogle Scholar
  12. 12.
    Kyriakopoulou, A., Stathopoulos, S., Lourentzou, I., Kalamboukis, T.: Ipl at clef 2013 medical retrieval task. In: CLEF (Online Working Notes/Labs/Workshop) (2013)Google Scholar
  13. 13.
    Kahn Jr., C.E., Thao, C.: Goldminer: a radiology image search engine. American Journal of Roentgenology 188(6), 1475–1478 (2007)CrossRefGoogle Scholar
  14. 14.
    Xu, S., McCusker, J., Krauthammer, M.: Yale image finder (yif): a new search engine for retrieving biomedical images. Bioinformatics 24(17), 1968–1970 (2008)CrossRefGoogle Scholar
  15. 15.
    Ceylan, N.M., Ozturkmenoglu, O., Alpkocak, A.: Demir at imageclefmed 2013: The effects of modality classification to information retrieval. In: CLEF (Online Working Notes/Labs/Workshop) (2013)Google Scholar
  16. 16.
    Rahman, M.M., You, D., Simpson, M.S., Antani, S.K., Demner-Fushman, D., Thoma, G.R.: Multimodal biomedical image retrieval using hierarchical classification and modality fusion. International Journal of Multimedia Information Retrieval 2(3), 159–173 (2013)CrossRefGoogle Scholar
  17. 17.
    Kitanovski, I., Trojacanec, K., Dimitrovski, I., Loshkovska, S.: Merging words and concepts for medical articles retrieval. In: Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, pp. 25–28. Le Centre De Hautes Etudes Internationales D’Informatique Documentaire (2013)Google Scholar
  18. 18.
    Kitanovski, I., Trojacanec, K., Dimitrovski, I., Loskovska, S.: Multimodal medical image retrieval. In: Markovski, S., Gushev, M. (eds.) ICT Innovations 2012. AISC, vol. 207, pp. 81–89. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Johnson, D.: Terrier information retrieval platform. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 517–519. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  20. 20.
    Macdonald, C., Plachouras, V., He, B., Lioma, C., Ounis, I.: University of Glasgow at webclef 2005: Experiments in per-field normalisation and language specific stemming. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D., et al. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 898–907. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  21. 21.
    Amati, G., Van Rijsbergen, C.J.: Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS) 20(4), 357–389 (2002)CrossRefGoogle Scholar
  22. 22.
    Kitanovski, I., Dimitrovski, I., Loskovska, S.: Fcse at medical tasks of imageclef 2013. In: CLEF (Online Working Notes/Labs/Workshop) (2013)Google Scholar
  23. 23.
    Dimitrovski, I., Kocev, D., Loskovska, S., Dzeroski, S.: Hierarchical annotation of medical images. Pattern Recognition 44(10-11), 2436–2449 (2011)CrossRefGoogle Scholar
  24. 24.
    Tommasi, T., Orabona, F., Caputo, B.: Discriminative cue integration for medical image annotation. Pattern Recognition Letters 29(15), 1996–2002 (2008)CrossRefGoogle Scholar
  25. 25.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  26. 26.
    van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Transactions on Pattern Analysis and Machine Intelligence 99(1)Google Scholar
  27. 27.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), software available at
  28. 28.
    Lin, H.-T., Lin, C.-J., Weng, R.C.: A note on Platt’s probabilistic outputs for support vector machines. Machine Learning 68, 267–276 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ivan Kitanovski
    • 1
  • Ivica Dimitrovski
    • 1
  • Gjorgji Madjarov
    • 1
  • Suzana Loskovska
    • 1
  1. 1.Faculty of Computer Science and EngineeringUniversity “Ss. Cyril and Methodius”SkopjeMacedonia

Personalised recommendations