Biomedical Image Retrieval Using Multimodal Context and Concept Feature Spaces

  • Md. Mahmudur Rahman
  • Sameer K. Antani
  • Dina Demner Fushman
  • George R. Thoma
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7075)


This paper presents a unified medical image retrieval method that integrates visual features and text keywords using multimodal classification and filtering. For content-based image search, concepts derived from visual features are modeled using support vector machine (SVM)-based classification of local patches from local image regions. Text keywords from associated metadata provides the context and are indexed using the vector space model of information retrieval. The concept and context vectors are combined and trained for SVM classification at a global level for image modality (e.g., CT, MR, x-ray, etc.) detection. In this method, the probabilistic outputs from the modality categorization are used to filter images so that the search can be performed only on a candidate subset. An evaluation of the method on ImageCLEFmed 2010 dataset of 77,000 images, XML annotations and topics results in a mean average precision (MAP) score of 0.1125. It demonstrates the effectiveness and efficiency of the proposed multimodal framework compared to using only a single modality or without using any classification information.


Support Vector Machine Image Retrieval Query Image Mean Average Precision Visual Concept 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Winfield, W., Lain, E., Horn, T., Hoskyn, J.: Eosinophilic cellulitislike reaction to subcutaneous etanercept injection. Arch. Dermatol. 142 (2), 218–220 (2006)CrossRefGoogle Scholar
  2. 2.
    Müller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A Review of Content-Based Image Retrieval Systems in Medical Applications Clinical Benefits and Future Directions. Int. J. of Med. Inform. 73 (1), 1–23 (2004)CrossRefGoogle Scholar
  3. 3.
    Wong, T.C.: Medical Image Databases. Springer, New York (1998)CrossRefGoogle Scholar
  4. 4.
    Müller, H., Kalpathy-Cramer, J., Eggel, I., Bedrick, S., Reisetter, J., Kahn. Jr., C.E., Hersh, W.R.: Overview of the CLEF, Medical Image Retrieval Track. In: CLEF (Notebook Papers/LABs/Workshops) (2010)Google Scholar
  5. 5.
    Lehmann, T.M., Güld, M.O., Deselaers, T., Keysers, D., Schubert, H., Spitzer, K., Ney, H., Wein, B.B.: Automatic categorization of medical images for content-based retrieval and data mining. Comput. Med. Imag. and Graph. 29, 143–155 (2005)CrossRefGoogle Scholar
  6. 6.
    Florea, F., Müller, H., Rogozan, A., Geissbuhler, A., Darmoni, S.: Medical image categorization with MedIC and MedGIFT. In: Proc. Med. Inform. Europe (MIE 2006), Maastricht, Netherlands, pp. 3–11 (2006)Google Scholar
  7. 7.
    Yates, R.B., Neto, B.R.: Modern Information Retrieval. Addison-Wesley (1999)Google Scholar
  8. 8.
    Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)zbMATHGoogle Scholar
  9. 9.
    Wu, T.F., Lin, C.J., Weng, R.C.: Probability Estimates for Multi-class Classification by Pairwise Coupling. J. of Mach. Learn. Research 5, 975–1005 (2004)zbMATHGoogle Scholar
  10. 10.
    Rahman, M.M., Antani, S.K., Thoma, G.R.: A Medical Image Retrieval Framework in Correlation Enhanced Visual Concept Feature Space. In: Proc. 22nd IEEE International Symposium on Computer-Based Medical Systems (CBMS), Albuquerque, New, Mexico, USA, August 3-4 (2009)Google Scholar
  11. 11.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. Software (2001),

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Md. Mahmudur Rahman
    • 1
  • Sameer K. Antani
    • 1
  • Dina Demner Fushman
    • 1
  • George R. Thoma
    • 1
  1. 1.U.S. National Library of MedicineNational Institutes of HealthBethesdaUSA

Personalised recommendations