Journal of Digital Imaging

, Volume 30, Issue 1, pp 4–10 | Cite as

Building and Querying RDF/OWL Database of Semantically Annotated Nuclear Medicine Images

  • Kyung Hoon Hwang
  • Haejun Lee
  • Geon Koh
  • Debra Willrett
  • Daniel L. RubinEmail author


As the use of positron emission tomography-computed tomography (PET-CT) has increased rapidly, there is a need to retrieve relevant medical images that can assist image interpretation. However, the images themselves lack the explicit information needed for query. We constructed a semantically structured database of nuclear medicine images using the Annotation and Image Markup (AIM) format and evaluated the ability the AIM annotations to improve image search. We created AIM annotation templates specific to the nuclear medicine domain and used them to annotate 100 nuclear medicine PET-CT studies in AIM format using controlled vocabulary. We evaluated image retrieval from 20 specific clinical queries. As the gold standard, two nuclear medicine physicians manually retrieved the relevant images from the image database using free text search of radiology reports for the same queries. We compared query results with the manually retrieved results obtained by the physicians. The query performance indicated a 98 % recall for simple queries and a 89 % recall for complex queries. In total, the queries provided 95 % (75 of 79 images) recall, 100 % precision, and an F1 score of 0.97 for the 20 clinical queries. Three of the four images missed by the queries required reasoning for successful retrieval. Nuclear medicine images augmented using semantic annotations in AIM enabled high recall and precision for simple queries, helping physicians to retrieve the relevant images. Further study using a larger data set and the implementation of an inference engine may improve query results for more complex queries.


Image retrieval Nuclear medicine PET Controlled vocabulary Protégé AIM ePAD 



This work was supported in part by grants from the National Cancer Institute, National Institutes of Health, U01CA142555 and 1U01CA190214.

Compliance with Ethical Standards

This study was approved by the Institutional Review Board and written consent was waived.


  1. 1.
    Liu Y, Zhang D, Lu G, Ma W-Y: A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40:262–282, 2007CrossRefGoogle Scholar
  2. 2.
    Faloutsos C, Barber R, Flickner M, Hafner J, Niblack W, Petkovic D, Equitz W: Efficient and effective querying by image content. J Intell Inf Syst 3:231–262, 1994CrossRefGoogle Scholar
  3. 3.
    Pentland A, Picard RW, Scaroff S: Photobook: content-based manipulation for image databases. Int J Comput Vis 18:233–254, 1996CrossRefGoogle Scholar
  4. 4.
    Gupta A, Jain R: Visual information retrieval. Commun ACM 40:70–79, 1997CrossRefGoogle Scholar
  5. 5.
    Smith JR, Chang SF: VisualSeek: a fully automatic content-based query system. Proceedings of the Fourth ACM International Conference on Multimedia, ACM Multimedia’96, Boston, MA, Nov 1996Google Scholar
  6. 6.
    Ma WY, Manjunath B: Netra: a toolbox for navigating large image databases. Proceedings of the IEEE International Conference on Image Processing 1:1997, pp 568–571Google Scholar
  7. 7.
    Eakins JP: Towards intelligent image retrieval. Pattern Recogn 35:3–14, 2002CrossRefGoogle Scholar
  8. 8.
    Brodley C, Kak A, Shyu C, Dy J, Broderick L, Aisen AM: Content-based retrieval from medical image database: a synergy of human interaction, machine learning and computer vision. Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), Orlando, FL, Jul 1999Google Scholar
  9. 9.
    Bray T, Paoli J, Sperberg-McQueen CM: Extensible Markup Language (XML) 1.0, W3C recommendation. Available at Accessed 15 September 2016
  10. 10.
    Lassila O, Swick RR: Resource Description Framework (RDF) Model and Syntax Specification, W3C recommendation. Available at Accessed 15 September 2016
  11. 11.
    Bao J, Kendall EF, McGuinness DL, Patel-Schneider PF: OWL 2 Web Ontology Language Quick Reference Guide (Second Edition), W3C recommendation. Available at Accessed 15 September 2016
  12. 12.
    Langlotz CP: RadLex: a new method for indexing online educational materials. Radiographics 26:1595–1597, 2006CrossRefPubMedGoogle Scholar
  13. 13.
    Rubin DL: Creating and curating a terminology for radiology: ontology modeling and analysis. J Digit Imaging 21:355–362, 2008CrossRefPubMedGoogle Scholar
  14. 14.
    Hong Y, Zhang J, Heilbrun ME, Kahn Jr, CE: Analysis of RadLex coverage and term co-occurrence in radiology reporting templates. J Digit Imaging 25:56–62, 2012CrossRefPubMedGoogle Scholar
  15. 15.
    Rosse C, Mejino Jr, JL: A reference ontology for biomedical informatics: the foundational model of anatomy. J Biomed Inform 36:478–500, 2003CrossRefPubMedGoogle Scholar
  16. 16.
    Sherter AL: Building a vocabulary. A new, improved version of SNOMED has the potential to ease the collection and analysis of clinical data. Health Data Manag 6:76–77, 1998Google Scholar
  17. 17.
    Nachimuthu SK, Lau LM: Practical issues in using SNOMED CT as a reference terminology. Stud Health Technol Inform 129:640–644, 2007PubMedGoogle Scholar
  18. 18.
    Mongkolwat P, Kleper V, Talbot S, Rubin DL: The National Cancer Informatics Program (NCIP) Annotation and Image Markup (AIM) foundation model. J Digit Imaging 27:692–701, 2014CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Rubin DL, Mongkolwat P, Kleper V, Supekar K, Channin DS: Medical Imaging on the Semantic Web: Annotation and Image Markup, Association for the Advancement of Artificial Intelligence, 2008. Spring Symposium Series, Stanford, 2008Google Scholar
  20. 20.
    Channin DS, Mongkolwat P, Kleper V, Rubin DL: The annotation and image mark-up project. Radiology 253:590–592, 2009CrossRefPubMedGoogle Scholar
  21. 21.
    Rubin DL, Rodriguez C, Shah P, Beaulieu C: iPad: Semantic annotation and markup of radiological images. AMIA Annu Symp Proc:626–630, 2008Google Scholar
  22. 22.
    Channin DS, Mongkolwat P, Kleper V, Sepukar K, Rubin DL: The caBIG annotation and image Markup project. J Digit Imaging 23:217–225, 2010CrossRefPubMedGoogle Scholar
  23. 23.
    Mongkolwat P, Channin DS, Kleper V, Rubin DL: Informatics in radiology: An open-source and open-access cancer biomedical informatics grid annotation and image markup template builder. Radiographics 32:1223–1232, 2012CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Moreira DA, Hage C, Luque EF, Willrett D, Rubin DL: 3D markup of radiological images in ePAD, a web-based image annotation tool. IEEE 28th International Symposium on Computer-Based Medical Systems, Proc, 2015, pp 97–102Google Scholar
  25. 25.
    Prud'hommeaux E, Seaborne A: SPARQL Query Language for RDF. W3C Recommendation. Available at Accessed 25 September 2015
  26. 26.
    Pathak J, Kiefer RC, Chute CG: Using linked data for mining drug-drug interactions in electronic health records. Stud Health Technol Inform 192:682–686, 2013PubMedPubMedCentralGoogle Scholar
  27. 27.
    Mate S, Kopcke F, Toddenroth D, Martin M, Prokosch H-U, Burkle T, Ganslandt T: Ontology-based data integration between clinical and research systems. PLoS ONE, 2015. doi: 10.1371/journal.pone.0116656
  28. 28.
    MacMahon H, Austin JHM, Gamsu G, Herold CJ, Jett JR, Naidich DP, Patz EF, Swensen S: Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner society. Radiology 237:395–400, 2005CrossRefPubMedGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2016

Authors and Affiliations

  • Kyung Hoon Hwang
    • 1
  • Haejun Lee
    • 1
  • Geon Koh
    • 1
  • Debra Willrett
    • 2
  • Daniel L. Rubin
    • 2
    • 3
    Email author
  1. 1.Department of Nuclear MedicineGachon University Gil Medical CenterIncheonSouth Korea
  2. 2.Department of RadiologyStanford UniversityStanfordUSA
  3. 3.Department of Medicine (Biomedical Informatics Research)Stanford UniversityStanfordUSA

Personalised recommendations