Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identification and classification

  • Md Mahmudur RahmanEmail author
  • Daekeun You
  • Matthew S. Simpson
  • Sameer K. Antani
  • Dina Demner-Fushman
  • George R. Thoma
Regular Paper


In biomedical articles, authors often use annotation markers such as arrows, letters, or symbols overlaid on figures and illustrations to highlight ROIs. These annotations are then referenced and correlated with concepts in the caption text or figure citations in the article text. This association creates a bridge between the visual characteristics of important regions within an image and their semantic interpretation. Identifying these assists in extracting ROIs that are likely to be highly relevant to the discussion in the article text. The aim of this work is to perform semantic search without knowing the concept keyword or the specific name of the visual pattern or appearance. We consider the problem of cross and multimodal retrieval of images from articles which contains components of text and images. Our proposed method localizes and recognizes the annotations by utilizing a combination of rule-based and statistical image processing techniques. The image regions are then annotated for classification using biomedical concepts obtained from a glossary of imaging terms. Similar automatic ROI extraction can be applied to query images, or to interactively mark an ROI. As a result, visual characteristics of the ROIs can be mapped to text concepts and then used to search image captions. In addition, the system can toggle the search process from purely perceptual to a conceptual one (crossmodal) based on utilizing user feedback or integrate both perceptual and conceptual search in a multimodal search process. The hypothesis, that such approaches would improve biomedical image retrieval, was validated through experiments on a biomedical article dataset of thoracic CT scans.


Biomedical image annotation  Multimodal biomedical information retrieval  Cross-modal retrieval Image overlay extraction  Classification Relevance feedback 



This research was supported by the Intramural Research Program of the National Institutes of Health (NIH), the National Library of Medicine (NLM), and the Lister Hill National Center for Biomedical Communications (LHNCBC).


  1. 1.
    Demner-Fushman D, Antani SK, Thoma GR (2007) Automatically finding images for clinical decision support. In: Proceedings of workshop on data mining in medicine, 7th IEEE international conference on data mining, pp 139–144Google Scholar
  2. 2.
    Demner-Fushman D, Antani SK, Simpson M, Thoma GR (2009) Annotation and retrieval of clinically relevant images. Int J Med Inform 78(12):e59–e67CrossRefGoogle Scholar
  3. 3.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  4. 4.
    Simpson M, You D, Rahman MM, Demner-Fushman D, Antani SK, Thoma GR (2012) Towards the creation of a visual ontology of biomedical imaging entities. In: AMIA 2012 Annual Symposium, Nov. 3–7, Chicago, ILGoogle Scholar
  5. 5.
    Pallisa E, Sanz P, Roman A, Maj J, Andreu J, Cceres J (2002) Lymphangioleiomyomatosis: pulmonary and abdominal findings with pathologic correlation. Radiographics 22:S185–S198CrossRefGoogle Scholar
  6. 6.
    Rahman MM, Antani SK, Thoma GR (2009) A medical image retrieval framework in correlation enhanced visual concept feature space. In: 22nd IEEE international symposium on computer-based medical systems (CBMS), Albuquerque, New Mexico, USAGoogle Scholar
  7. 7.
    Zhou XS, Huang TS (2003) Relevance feedback for image retrieval: a comprehensive review. Multimed Syst 8(6):536–544CrossRefGoogle Scholar
  8. 8.
    Rui Y, Huang TS (1999) Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Circuits Syst Video Technol 8(5):644–655Google Scholar
  9. 9.
    Demner-Fushman D, Antani SK, Simpson M, Thoma GR (2012) Design and development of a multimodal biomedical information retrieval system. JCSE 6(2):168–177Google Scholar
  10. 10.
    Mori Y, Takahashi H, Oka R (1999) Image-to-word transformation based on dividing and vector quantizing images with words. In: MISRM’99 first international workshop on multimedia intelligent storage and retrieval managementGoogle Scholar
  11. 11.
    Duygulu P, Barnard K, de Freitas N, Forsyth D (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Seventh European conference on computer vision, pp 97–112Google Scholar
  12. 12.
    Yang Y, Zhuang Y, Wu F, Pan Y (2008) Harmonizing hierarchical manifolds for multimedia document semantics understanding and cross-media retrieval. IEEE Trans Multimed 10(3):437–446CrossRefGoogle Scholar
  13. 13.
    Jeon J, Lavrenko V, Manmatha R (2003) Automatic image annotation and retrieval using cross-media relevance models. In: Proceeding SIGIR ’03 Proceedings of the 26th annual international ACM SIGIR conference on research and development in informaion retrieval, pp 119–126Google Scholar
  14. 14.
    Rasiwasia N, Costa Pereira J, Coviello E, Doyle G, Lanckriet GRG, Levy R, Vasconcelos N (2010) A new approach to cross-modal multimedia retrieval. In: Proceedings of the international conference on multimedia, MM ’10, pp 251–260Google Scholar
  15. 15.
    Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv (CSUR) 40(2):1–60Google Scholar
  16. 16.
    Chen N (2006) A survey of indexing and retrieval of multimodal documents: text and images. Technical Report, 2006–505, Queen’s UniversityGoogle Scholar
  17. 17.
    Kalpathy-Cramer J, Müller H, Bedrick S, Eggel I, Garcia Seco de Herrera A, Tsikrika T (2011) Overview of the CLEF 2011 medical image classification and retrieval tasks. CLEF 2011 Working Notes, Amsterdam, The NetherlandsGoogle Scholar
  18. 18.
    Müller H, Herrera A, Kalpathy-Cramer J, Demner-Fushman D, Antani S, Ivan E (2012) Overview of the ImageCLEF 2012 medical image retrieval and classification tasks. In the Working Notes for the CLEF 2012 Labs and Workshop, 17–20 September. Rome, Italy, AmsterdamGoogle Scholar
  19. 19.
    Müller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Reisetter J Jr., Kahn CE Jr., Hersh WR (2010) Overview of the CLEF 2010 medical image retrieval track. CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Padua, Italy, September 20–23Google Scholar
  20. 20.
    You D, Antani SK, Demner-Fushman D, Rahman MM, Govindaraju V, Thoma GR (2011) Biomedical article retrieval using multimodal features and image annotations in region-based CBIR. In: Document recognition and retrieval XVIII. Proceedings of SPIE, vol 7874, p 7874 0KGoogle Scholar
  21. 21.
    Zhang D, Lu G (2003) A comparative study on shape retrieval using fourier descriptors with different shape signatures. J Vis Commun Image Represent 14(1):41–60CrossRefGoogle Scholar
  22. 22.
    Sluimer IC, Schilham S, Prokop M, van Ginneken B (2006) Computer analysis of computed tomography scans of the lung: a survey. IEEE Trans Med Imaging 25(4):385–405Google Scholar
  23. 23.
    Sluimer IC, van Waes PF, Viergever MA, van Ginneken B (2003) Computeraided diagnosis in high resolution ct of the lungs. Med Phys 30(12):3081–3090CrossRefGoogle Scholar
  24. 24.
    Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3:610–621CrossRefGoogle Scholar
  25. 25.
    Srinivasan GN, Shobha G (2008) Statistical texture analysis. Proc World Acad Sci Eng Technol 36. ISSN 2070–3740Google Scholar
  26. 26.
    Zhang D, Wong A, Indrawan M, Lu G (2000) Content-based image retrieval using gabor texture features. IEEE Trans Pattern Anal Mach Intell 13–15Google Scholar
  27. 27.
    Tamura H, Mori S, Yamawaki T (1978) Textural features corresponding to visual perception. IEEE Trans Syst Man Cybern 8(6):460–472CrossRefGoogle Scholar
  28. 28.
    Chatzichristofis SA, Boutalis YS (2008) Cedd: color and edge directivity descriptor. A compact descriptor for image indexing and retrieval. In: 6th international conference on computer vision systems, ICVS 2008. LNCS, vol 5008, pp 312–322Google Scholar
  29. 29.
    Chatzichristofis SA, Boutalis YS (2008) Fcth: fuzzy color and texture histogram a low level feature for accurate image retrieval. In: Proceedings of the 9th international workshop on image analysis for multimedia interactive services, WIAMIS, pp 191–196. doi: 10.1109/WIAMIS.2008.24
  30. 30.
    Lux M, Chatzichristofis SA (2008) Lire: lucene image retrieval: an extensible java CBIR library. In: Proceedings of the 16th ACM international conference on multimedia, Vancouver, British Columbia, Canada. doi: 10.1145/1459359.1459577
  31. 31.
    Yates RB, Neto BR (1999) Modern information retrieval, 1st edn. Addison Wesley, ReadingGoogle Scholar
  32. 32.
    Krishnam MS, Suh RD, Tomasian A, Goldin JG, Lai C, Brown K et al (2007) Postoperative complications of lung transplantation: radiologic findings along a time continuum. Radiographics 27(4):957–974Google Scholar
  33. 33.
    Hansell DM, Bankier AA, MacMahon H, McLoud TC, Müller NL, Remy J (2008) Fleischner society: glossary of terms for thoracic imaging. Radiology 246(3):697–722CrossRefGoogle Scholar
  34. 34.
    Uzuner Ö, South BR, Shen S, DuVall SL (2011) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 18(5):552–556Google Scholar
  35. 35.
    Vapnik V (1998) Statistical learning theory. Wiley, New YorkzbMATHGoogle Scholar
  36. 36.
    Wu TF, Lin CJ, Weng RC (2004) Probability estimates for multi-class classification by pairwise coupling. J Mach Learn Res 5:975–1005Google Scholar

Copyright information

© © Springer-Verlag London (outside the USA) 2014

Authors and Affiliations

  • Md Mahmudur Rahman
    • 1
    Email author
  • Daekeun You
    • 1
  • Matthew S. Simpson
    • 1
  • Sameer K. Antani
    • 1
  • Dina Demner-Fushman
    • 1
  • George R. Thoma
    • 1
  1. 1.US National Library of MedicineNational Institutes of HealthBethesdaUSA

Personalised recommendations