RadSem: Semantic Annotation and Retrieval for Medical Images

  • Manuel Möller
  • Sven Regel
  • Michael Sintek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5554)


We present a tool for semantic medical image annotation and retrieval. It leverages the MEDICO ontology which covers formal background information from various biomedical ontologies such as the Foundational Model of Anatomy (FMA), terminologies like ICD-10 and RadLex and covers various aspects of clinical procedures. This ontology is used during several steps of annotation and retrieval: (1) We developed an ontology-driven metadata extractor for the medical image format DICOM. Its output contains, e. g., person name, age, image acquisition parameters, body region, etc. (2) The output from (1) is used to simplify the manual annotation by providing intuitive visualizations and to provide a preselected subset of annotation concepts. Furthermore, the extracted metadata is linked together with anatomical annotations and clinical findings to generate a unified view of a patient’s medical history. (3) On the search side we perform query expansion based on the structure of the medical ontologies. (4) Our ontology for clinical data management allows us to link and combine patients, medical images and annotations together in a comprehensive result list. (5) The medical annotations are further extended by links to external sources like Wikipedia to provide additional information.


Query Expansion Image Annotation Semantic Annotation Biomedical Ontology DICOM Standard 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Agne, S., Reuschling, C., Dengel, A.: DynaQ – dynamic queries for electronic document management. In: 10th IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW 2006) (2006)Google Scholar
  2. 2.
    Bechhofer, S., Volz, R., Lord, P.W.: Cooking the semantic web with the OWL API. In: Fensel, D., Sycara, K.P., Mylopoulos, J. (eds.) ISWC 2003. LNCS, vol. 2870, pp. 659–675. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. 3.
    Bodenreider, O.: Biomedical ontologies in action: role in knowledge management, data integration and decision support. In: International Medical Informatics Association (IMIA) (ed.) IMIA Yearbook 2008, pp. 67–79. Schattauer (2008)Google Scholar
  4. 4.
    World Health Organization Geneva. International statistical classification of diseases and related health problems (2004)Google Scholar
  5. 5.
    Gruber, T.R.: Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies 43, 907–928 (1995)CrossRefGoogle Scholar
  6. 6.
    Güld, M.O., Kohnen, M., Keysers, D., Schubert, H., Wein, B.B., Bredno, J., Lehmann, T.M.: Quality of DICOM header information for image categorization. Proc. SPIE 4685(39), 280–287 (2002)CrossRefGoogle Scholar
  7. 7.
    Hayes, P.: RDF semantics. W3C Recommendation (February 2004)Google Scholar
  8. 8.
    Langlotz, C.P.: Radlex: A new method for indexing online educational materials. RadioGraphics 26, 1595–1597 (2006)CrossRefGoogle Scholar
  9. 9.
    McGuinness, D.L., van Harmelen, F.: OWL Web Ontology Language overview. W3C recommendation, World Wide Web Consortium (February 2004)Google Scholar
  10. 10.
    Mejino, J.L., Rubin, D.L., Brinkley, J.F.: FMA-RadLex: An application ontology of radiological anatomy derived from the foundational model of anatomy reference ontology. In: AMIA Annual Symposium Proceedings, p. 465 (2008)Google Scholar
  11. 11.
    Möller, M., Sintek, M.: A scalable architecture for cross-modal semantic annotation and retrieval. In: Dengel, A.R., Berns, K., Breuel, T.M. (eds.) KI 2008. LNCS, vol. 5243. Springer, Heidelberg (2008)Google Scholar
  12. 12.
    Möller, M., Sintek, M., Buitelaar, P., Mukherjee, S., Zhou, X.S., Freund, J.: Medical image understanding through the integration of cross-modal object recognition with formal domain knowledge. In: Proc. of HEALTHINF 2008, Funchal, Madeira, Portugal, vol. 1, pp. 134–141 (2008)Google Scholar
  13. 13.
    Mojsilovic, A., Gomes, J., Rogowitz, B.E.: Semantic-friendly indexing and querying of images based on the extraction of the objective semantic cues. International Journal of Computer Vision 56, 79–107 (2004)CrossRefGoogle Scholar
  14. 14.
    Mueen, A., Zainuddin, R., Baba, M.S.: Automatic multilevel medical image annotation and retrieval. Journal of Digital Imaging 21(3), 290–295 (2008)CrossRefGoogle Scholar
  15. 15.
    Mykowiecka, A., Marciniak, M., Podsiadly-Marczykowska, T.: Data-driven ontologies for an information extraction system from polish mammography reports. In: Proc. of the 10th International Protégé Conference, Budapest, Hungary (July 2007)Google Scholar
  16. 16.
    Pietriga, E., Bizer, C., Karger, D.R., Lee, R.: Fresnel: A browser-independent presentation vocabulary for RDF. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 158–171. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  17. 17.
    Prud’hommeaux, E., Seaborne, A.: SPARQL query language for RDF. Technical report, W3C (March 2007)Google Scholar
  18. 18.
    Raicu, D.S., Varutbangkul, E., Furst, J.D., Armato III, S.G.: Modeling Semantics from Image Data: Opportunities from LIDC. International Journal of Biomedical Engineering and Technology (2007)Google Scholar
  19. 19.
    Rosse, C., Mejino, R.L.V.: A reference ontology for bioinformatics: The foundational model of anatomy. Journal of Biomedical Informatics 36, 478–500 (2003)CrossRefGoogle Scholar
  20. 20.
    Rubin, D.L., Mongkolwat, P., Kleper, V., Supekar, K., Channin, D.S.: Medical imaging on the semantic web: Annotation and image markup. In: AAAI Spring Symposium Series, Stanford University (2008)Google Scholar
  21. 21.
    Semy, S.K., Pulvermacher, M.K., Obrst, L.J.: Toward the use of an upper ontology for U.S. government and U.S. military domains: An evaluation. Technical report, MITRE Corporation (September 2004)Google Scholar
  22. 22.
    Temal, L., Dojat, M., Kassel, G., Gibaud, B.: Towards an ontology for sharing medical images and regions of interest in neuroimaging. Journal of Biomedical Informatics (2008)Google Scholar
  23. 23.
    Yao, J., Antani, S., Long, R., Thoma, G., Zhang, Z.: Automatic medical image annotation and retrieval using SECC. In: Proc. of 19th International Symposium on Computer-Based Medical Systems (CBMS 2006), Salt Lake City (June 2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Manuel Möller
    • 1
  • Sven Regel
    • 2
  • Michael Sintek
    • 1
  1. 1.German Research Center for Artificial Intelligence (DFKI) GmbHKaiserslauternGermany
  2. 2.Chemnitz University of TechnologyChemnitzGermany

Personalised recommendations