Skip to main content
SpringerLink
Log in

Building and Querying RDF/OWL Database of Semantically Annotated Nuclear Medicine Images

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

As the use of positron emission tomography-computed tomography (PET-CT) has increased rapidly, there is a need to retrieve relevant medical images that can assist image interpretation. However, the images themselves lack the explicit information needed for query. We constructed a semantically structured database of nuclear medicine images using the Annotation and Image Markup (AIM) format and evaluated the ability the AIM annotations to improve image search. We created AIM annotation templates specific to the nuclear medicine domain and used them to annotate 100 nuclear medicine PET-CT studies in AIM format using controlled vocabulary. We evaluated image retrieval from 20 specific clinical queries. As the gold standard, two nuclear medicine physicians manually retrieved the relevant images from the image database using free text search of radiology reports for the same queries. We compared query results with the manually retrieved results obtained by the physicians. The query performance indicated a 98 % recall for simple queries and a 89 % recall for complex queries. In total, the queries provided 95 % (75 of 79 images) recall, 100 % precision, and an F1 score of 0.97 for the 20 clinical queries. Three of the four images missed by the queries required reasoning for successful retrieval. Nuclear medicine images augmented using semantic annotations in AIM enabled high recall and precision for simple queries, helping physicians to retrieve the relevant images. Further study using a larger data set and the implementation of an inference engine may improve query results for more complex queries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Liu Y, Zhang D, Lu G, Ma W-Y: A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40:262–282, 2007

    Article  Google Scholar 

  2. Faloutsos C, Barber R, Flickner M, Hafner J, Niblack W, Petkovic D, Equitz W: Efficient and effective querying by image content. J Intell Inf Syst 3:231–262, 1994

    Article  Google Scholar 

  3. Pentland A, Picard RW, Scaroff S: Photobook: content-based manipulation for image databases. Int J Comput Vis 18:233–254, 1996

    Article  Google Scholar 

  4. Gupta A, Jain R: Visual information retrieval. Commun ACM 40:70–79, 1997

    Article  Google Scholar 

  5. Smith JR, Chang SF: VisualSeek: a fully automatic content-based query system. Proceedings of the Fourth ACM International Conference on Multimedia, ACM Multimedia’96, Boston, MA, Nov 1996

  6. Ma WY, Manjunath B: Netra: a toolbox for navigating large image databases. Proceedings of the IEEE International Conference on Image Processing 1:1997, pp 568–571

  7. Eakins JP: Towards intelligent image retrieval. Pattern Recogn 35:3–14, 2002

    Article  Google Scholar 

  8. Brodley C, Kak A, Shyu C, Dy J, Broderick L, Aisen AM: Content-based retrieval from medical image database: a synergy of human interaction, machine learning and computer vision. Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), Orlando, FL, Jul 1999

  9. Bray T, Paoli J, Sperberg-McQueen CM: Extensible Markup Language (XML) 1.0, W3C recommendation. Available at http://www.w3.org/TR/REC-xml. Accessed 15 September 2016

  10. Lassila O, Swick RR: Resource Description Framework (RDF) Model and Syntax Specification, W3C recommendation. Available at http://www.w3.org/TR/PR-rdf-syntax. Accessed 15 September 2016

  11. Bao J, Kendall EF, McGuinness DL, Patel-Schneider PF: OWL 2 Web Ontology Language Quick Reference Guide (Second Edition), W3C recommendation. Available at http://www.w3.org/TR/2012/REC-owl2-quick-reference-20121211/. Accessed 15 September 2016

  12. Langlotz CP: RadLex: a new method for indexing online educational materials. Radiographics 26:1595–1597, 2006

    Article  PubMed  Google Scholar 

  13. Rubin DL: Creating and curating a terminology for radiology: ontology modeling and analysis. J Digit Imaging 21:355–362, 2008

    Article  PubMed  Google Scholar 

  14. Hong Y, Zhang J, Heilbrun ME, Kahn Jr, CE: Analysis of RadLex coverage and term co-occurrence in radiology reporting templates. J Digit Imaging 25:56–62, 2012

    Article  PubMed  Google Scholar 

  15. Rosse C, Mejino Jr, JL: A reference ontology for biomedical informatics: the foundational model of anatomy. J Biomed Inform 36:478–500, 2003

    Article  PubMed  Google Scholar 

  16. Sherter AL: Building a vocabulary. A new, improved version of SNOMED has the potential to ease the collection and analysis of clinical data. Health Data Manag 6:76–77, 1998

    Google Scholar 

  17. Nachimuthu SK, Lau LM: Practical issues in using SNOMED CT as a reference terminology. Stud Health Technol Inform 129:640–644, 2007

    PubMed  Google Scholar 

  18. Mongkolwat P, Kleper V, Talbot S, Rubin DL: The National Cancer Informatics Program (NCIP) Annotation and Image Markup (AIM) foundation model. J Digit Imaging 27:692–701, 2014

    Article  PubMed  PubMed Central  Google Scholar 

  19. Rubin DL, Mongkolwat P, Kleper V, Supekar K, Channin DS: Medical Imaging on the Semantic Web: Annotation and Image Markup, Association for the Advancement of Artificial Intelligence, 2008. Spring Symposium Series, Stanford, 2008

  20. Channin DS, Mongkolwat P, Kleper V, Rubin DL: The annotation and image mark-up project. Radiology 253:590–592, 2009

    Article  PubMed  Google Scholar 

  21. Rubin DL, Rodriguez C, Shah P, Beaulieu C: iPad: Semantic annotation and markup of radiological images. AMIA Annu Symp Proc:626–630, 2008

  22. Channin DS, Mongkolwat P, Kleper V, Sepukar K, Rubin DL: The caBIG annotation and image Markup project. J Digit Imaging 23:217–225, 2010

    Article  PubMed  Google Scholar 

  23. Mongkolwat P, Channin DS, Kleper V, Rubin DL: Informatics in radiology: An open-source and open-access cancer biomedical informatics grid annotation and image markup template builder. Radiographics 32:1223–1232, 2012

    Article  PubMed  PubMed Central  Google Scholar 

  24. Moreira DA, Hage C, Luque EF, Willrett D, Rubin DL: 3D markup of radiological images in ePAD, a web-based image annotation tool. IEEE 28th International Symposium on Computer-Based Medical Systems, Proc, 2015, pp 97–102

  25. Prud'hommeaux E, Seaborne A: SPARQL Query Language for RDF. W3C Recommendation. Available at http://www.w3.org/TR/rdf-sparql-query. Accessed 25 September 2015

  26. Pathak J, Kiefer RC, Chute CG: Using linked data for mining drug-drug interactions in electronic health records. Stud Health Technol Inform 192:682–686, 2013

    PubMed  PubMed Central  Google Scholar 

  27. Mate S, Kopcke F, Toddenroth D, Martin M, Prokosch H-U, Burkle T, Ganslandt T: Ontology-based data integration between clinical and research systems. PLoS ONE, 2015. doi:10.1371/journal.pone.0116656

  28. MacMahon H, Austin JHM, Gamsu G, Herold CJ, Jett JR, Naidich DP, Patz EF, Swensen S: Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner society. Radiology 237:395–400, 2005

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

This work was supported in part by grants from the National Cancer Institute, National Institutes of Health, U01CA142555 and 1U01CA190214.

Author information

Authors and Affiliations

  1. Department of Nuclear Medicine, Gachon University Gil Medical Center, Incheon, South Korea

    Kyung Hoon Hwang, Haejun Lee & Geon Koh

  2. Department of Radiology, Stanford University, Stanford, CA, USA

    Debra Willrett & Daniel L. Rubin

  3. Department of Medicine (Biomedical Informatics Research), Stanford University, Stanford, CA, USA

    Daniel L. Rubin

Authors
  1. Kyung Hoon Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haejun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Geon Koh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Debra Willrett
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel L. Rubin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel L. Rubin.

Ethics declarations

This study was approved by the Institutional Review Board and written consent was waived.

Appendix

Appendix

SPARQL Query

  • Simple Single Query

    1. (1)

      Retrieve PET-CT studies containing lesions with maximum SUV greater than 10.0.

    2. (2)

      Retrieve PET-CT studies containing lesions in the upper lobe of the right lung.

  • Simple Double Query

    1. (3)

      Retrieve PET-CT studies where lesions in the laryngopharynx have maximum SUV greater than or equal to 3.0.

    2. (4)

      Retrieve PET-CT studies where lesions in the gallbladder have maximum SUV greater than or equal to 3.0.

    3. (5)

      Retrieve PET-CT studies where lesions in the lower lobe of the left lung have maximum SUV greater than or equal to 3.0.

    4. (6)

      Retrieve PET-CT studies where lesions in the cavitated organ have maximum SUV greater than or equal to 3.0.

  • Simple Temporal Query

    1. (7)

      Retrieve PET-CT studies where lesions are in the lymph nodes and maximum SUV decreased by more than 20 % in the next PET-CT studies.

    2. (8)

      Retrieve PET-CT studies containing mild hypermetabolic lesions that disappeared in the next PET-CT studies.

    3. (9)

      Retrieve PET-CT studies containing moderate hypermetabolic lesions that disappeared in the next PET-CT studies.

    4. (10)

      Retrieve PET-CT studies containing severe hypermetabolic lesions that disappeared in the next PET-CT studies.

    5. (11)

      Retrieve PET-CT studies containing lesions with maximum SUV greater than or equal to 3.0 and showing no interval change in the next PET-CT studies.

  • Complex Temporal Query (Including Radiologic Criteria)

    1. (12)

      Retrieve PET-CT studies with lymph node lesions that showed no interval change for more than 2 years.

    2. (13)

      Retrieve PET-CT studies with lung lesions that showed no interval change for more than 3 years.

    3. (14)

      Retrieve PET-CT studies with thyroid lesions that showed no interval change for more than 1 year.

    4. (15)

      Retrieve PET-CT studies with thyroid or lymph node lesions that showed no interval change for more than 2 years.

    5. (16)

      Retrieve PET-CT studies with stomach lesions that showed no interval change for more than 1 year since the recommendation of an endoscopy.

    6. (17)

      Retrieve PET-CT studies containing lung nodules sized 0.4 ∼ 0.6 cm in diameter that showed no interval change for more than 6 months (the Fleischner criteria).

  • Complex Reasoning Query

    1. (18)

      Retrieve PET-CT studies that demonstrated lymphoma in stage 3.

    2. (19)

      Retrieve PET-CT studies that demonstrated gastric lymphoma in stage 2

    3. (20)

      Retrieve PET-CT studies that show a potential of biliary obstruction.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hwang, K.H., Lee, H., Koh, G. et al. Building and Querying RDF/OWL Database of Semantically Annotated Nuclear Medicine Images. J Digit Imaging 30, 4–10 (2017). https://doi.org/10.1007/s10278-016-9916-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-016-9916-7

Keywords

Search

Navigation