A Semantic Fusion Approach Between Medical Images and Reports Using UMLS

  • Daniel Racoceanu
  • Caroline Lacoste
  • Roxana Teodorescu
  • Nicolas Vuillemenot
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4182)


One of the main challenges in content-based image retrieval still remains to bridge the gap between low-level features and semantic information. In this paper, we present our first results concerning a medical image retrieval approach using a semantic medical image and report indexing within a fusion framework, based on the Unified Medical Language System (UMLS) metathesaurus. We propose a structured learning framework based on Support Vector Machines to facilitate modular design and extract medical semantics from images. We developed two complementary visual indexing approaches within this framework: a global indexing to access image modality, and a local indexing to access semantic local features. Visual indexes and textual indexes – extracted from medical reports using MetaMap software application – constitute the input of the late fusion module. A weighted vectorial norm fusion algorithm allows the retrieval system to increase its meaningfulness, efficiency and robustness. First results on the CLEF medical database are presented. The important perspectives of this approach in terms of semantic query expansion and data-mining are discussed.


Medical Image Image Retrieval Medical Report Query Expansion Image Indexing 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abe, H., MacMahon, H., Engelmann, R., Li, Q., Shiraishi, J., Katsuragawa, S., Aoyama, M., Ishida, T., Ashizawa, K., Metz, C.E., Doi, K.: Computer-aided diagnosis in chest radiography: Results of large-scale observer tests at the 1996- 2001 rsna scientific assemblies. RadioGraphics 23(1), 255–265 (2003)CrossRefGoogle Scholar
  2. 2.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. Journal of machine learning research 3, 1107–1135 (2003)MATHCrossRefGoogle Scholar
  3. 3.
    Barnard, K., Forsyth, D.: Learning the semantics of words and pictures. In: Proceedings of the International Conference on Computer Vision, vol. 2, pp. 408–415 (2001)Google Scholar
  4. 4.
    Bui, A.A.T., Taira, R.K., Dionision, J.D.N., Aberle, D.R., El-Saden, S., Kangarloo, H.: Evidence-based radiology. Academic Radiology 9(6), 662–669 (2002)CrossRefGoogle Scholar
  5. 5.
    Chang, N.-S., Fu, K.-S.: Query-by-pictorial-example. IEEE Transactions on Software Engineering 6(6), 519–524 (1980)CrossRefGoogle Scholar
  6. 6.
    Chu, W.W., Alfonso, F.C., Ricky, K.T.: Knowledge-based image retrieval with spatial and temporal constructs. IEEE Transactions on Knowledge and Data Engineering 10, 872–888 (1998)CrossRefGoogle Scholar
  7. 7.
    Fleury, C.: Apports reciproques des informations textuelles et visuelles par analyse de la semantique latente pour la recherche d’information. Master’s thesis, Intelligence, Interaction and Information, CLIPS-MRIM Grenoble, France, IPAL UMI CNRS 2955, Singapore (June 2006)Google Scholar
  8. 8.
    Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., Steele, D., Yanker, P.: Query by image and video content: The qbic system. IEEE Computer 28(9), 23–32 (1995)Google Scholar
  9. 9.
    Guld, M.O., Kohnen, M., Keysers, D., Schubert, H., Wein, B.B., Bredno, J., Lehmann, T.M.: Quality of dicom header information for image categorization. In: Proceedings of the International Symposium on Medical Imaging, vol. 4685, pp. 280–287. SPIE, San Jose (2002)Google Scholar
  10. 10.
    Kahn, C.E.: Artificial intelligence in radiology: Decision support systems. RadioGraphics 14, 849–861 (1994)Google Scholar
  11. 11.
    Korn, P., Sidiropoulos, N., Faloutsos, C., Siegel, E., Protopapas, Z.: Fast and effective retrieval of medical tumor shapes. IEEE Transactions on Knowledge and Data Engineering 10, 889–904 (1998)CrossRefGoogle Scholar
  12. 12.
    LeBozec, C., Jaulent, M.-C., Zapletal, E., Degoulet, P.: Unified modeling language and design of a case-based retrieval system in medical imaging. In: Proceedings of the Annual Symposium of the American Society for Medical Informatics, Nashville, TN, USA (1998)Google Scholar
  13. 13.
    Lehmann, T.M., Gld, M.O., Thies, C., Fischer, B., Spitzer, K., Keysers, D., Ney, H., Kohnen, M., Schubert, H., Wein, B.B.: Content-based image retrieval in medical application. Methods of Information in Medicine 43(4), 354–361 (2004)Google Scholar
  14. 14.
    Lim, J., Chevallet, J.P.: Vismed: a visual vocabulary approach for medical image indexing and retrieval. In: Proceedings of the Asia Information Retrieval Symposium, pp. 84–96 (2005)Google Scholar
  15. 15.
    Lim, J., Jin, J.: A structured learning framework for content-based image indexing and visual query. Multimedia Systems 10, 317–331 (2005)CrossRefGoogle Scholar
  16. 16.
    Muller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications - clinical benefits and future directions. International Journal of Medical Informatics 73, 1–23 (2004)CrossRefGoogle Scholar
  17. 17.
    Niblack, W., Barber, R., Equitz, W., Flickner, M.D., Glasman, E.H., Petkovic, D., Yanker, P., Faloutsos, C., Taubin, G.: QBICproject: querying images by content, using color, texture, and shape. In: Niblack, W. (ed.) Storage and Retrieval for Image and Video Databases, vol. 1908, pp. 173–187. SPIE, San Jose (1993)Google Scholar
  18. 18.
    Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: Tools for content-based manipulation of image databases. International Journal of Computer Vision 18, 233–254 (1996)CrossRefGoogle Scholar
  19. 19.
    Sclaroff, S., la Cascia, M., Sethi, S.: Unifyng textual and visual cues for contentbased image retrieval on the world wide web. Computer Vision and Image Understanding 75(1/2), 86–98 (1998)Google Scholar
  20. 20.
    Shyu, C.-R., Brodley, C.E., Kak, A.C., Kosaka, A., Aisen, A.M., Broderick, L.S.: ASSERT: A physician-in-the-loop content-based retrieval system for HRCT image databases. Computer Vision and Image Understanding 75, 111–132 (1999); Special issue on content-based access for image and video librariesGoogle Scholar
  21. 21.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Contentbased image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1349–1380 (2000)CrossRefGoogle Scholar
  22. 22.
    Westerveld, T.: Image retrieval: Content versus context. In: Recherche d’Information Assistee par Ordinateur (2000)Google Scholar
  23. 23.
    Zhao, R., Grosky, W.: Narrowing the semantic gap - improved text-based web document retrieval using visual features. IEEE Transactions on Multimedia 4(2), 189–200 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Daniel Racoceanu
    • 1
    • 2
  • Caroline Lacoste
    • 1
  • Roxana Teodorescu
    • 1
    • 3
  • Nicolas Vuillemenot
    • 1
    • 4
  1. 1.IPAL-Image Perception, Access and Language – UMI-CNRS 2955, Institute for Infocomm ResearchA*STARSingapore
  2. 2.University of Franche-ComteBesanconFrance
  3. 3.“Politehnica” University from TimisoaraRomania
  4. 4.Ecole Nationale Superieure de Mecaniques et Microtechniques de BesanconFrance

Personalised recommendations