A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotations

  • A. B. SpanierEmail author
  • N. Caplan
  • J. Sosna
  • B. Acar
  • L. Joskowicz
Original Article



The goal of medical content-based image retrieval (M-CBIR) is to assist radiologists in the decision-making process by retrieving medical cases similar to a given image. One of the key interests of radiologists is lesions and their annotations, since the patient treatment depends on the lesion diagnosis. Therefore, a key feature of M-CBIR systems is the retrieval of scans with the most similar lesion annotations. To be of value, M-CBIR systems should be fully automatic to handle large case databases.


We present a fully automatic end-to-end method for the retrieval of CT scans with similar liver lesion annotations. The input is a database of abdominal CT scans labeled with liver lesions, a query CT scan, and optionally one radiologist-specified lesion annotation of interest. The output is an ordered list of the database CT scans with the most similar liver lesion annotations. The method starts by automatically segmenting the liver in the scan. It then extracts a histogram-based features vector from the segmented region, learns the features’ relative importance, and ranks the database scans according to the relative importance measure. The main advantages of our method are that it fully automates the end-to-end querying process, that it uses simple and efficient techniques that are scalable to large datasets, and that it produces quality retrieval results using an unannotated CT scan.


Our experimental results on 9 CT queries on a dataset of 41 volumetric CT scans from the 2014 Image CLEF Liver Annotation Task yield an average retrieval accuracy (Normalized Discounted Cumulative Gain index) of 0.77 and 0.84 without/with annotation, respectively.


Fully automatic end-to-end retrieval of similar cases based on image information alone, rather that on disease diagnosis, may help radiologists to better diagnose liver lesions.


Medical content-based image retrieval Image features Liver lesions Annotations 



This research was supported in part by Israel Ministry of Science, Technology and Space, Grant 53681, 2016-19, and by the Oppenheimer Applied Research Grant, The Hebrew University, TUBITAK ARDEB Grant No. 110E264, 2015-16.

Compliance with ethical standards

Conflict of interest

None of the authors has any conflict of interest. The authors have no personal financial or institutional interest in any of the materials, software or devices described in this article.

Human and animal rights

No animals or humans were involved in this research. All scans were anonymized before delivery to the researchers.


  1. 1.
    Horton KM, Bluemke DA, Hruban RH, Soyer P, Fishman EK (1999) CT and MR imaging of benign hepatic and biliary tumors. Radio Gr 19:431–451Google Scholar
  2. 2.
    Pandey P, Lewis H, Pandey A, Schmidt C, Dillhoff M, Kamel IR, Pawlik TM (2017) Updates in hepatic oncology imaging. Surg Oncol 26:195–206CrossRefPubMedGoogle Scholar
  3. 3.
    DeVita VT, Lawrence TSRS (2011) DeVita, Hellman, and Rosenberg’s cancer: principles and practice of oncology. Lippincott Williams & Wilkins, PhiladelphiaGoogle Scholar
  4. 4.
    Assy N, Nasser G, Djibre A, Beniashvili Z, Elias S, Zidan J (2009) Characteristics of common solid liver lesions and recommendations for diagnostic workup. World J Gastroenterol 15:3217–27CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Oliver JH, Baron RL (1996) Helical biphasic contrast-enhanced CT of the liver: technique, indications, interpretation, and pitfalls. Radiology 201:1–14CrossRefPubMedGoogle Scholar
  6. 6.
    MacDonald SL, Cowan IA, Floyd R, Mackintosh S, Graham R, Jenkins E, Hamilton R (2013) Measuring and managing radiologist workload: application of lean and constraint theories and production planning principles to planning radiology services in a major tertiary hospital. J Med Imaging Radiat Oncol 57:544–550CrossRefPubMedGoogle Scholar
  7. 7.
    Zhao B, Tan Y, Bell DJ, Marley SE, Guo P, Mann H, Scott MLJ, Schwartz LH, Ghiorghiu DC (2013) Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol 82:959–968CrossRefPubMedGoogle Scholar
  8. 8.
    Suzuki C, Torkzad MR, Jacobsson H, Åström G, Sundin A, Hatschek T, Fujii H, Blomqvist L (2010) Interobserver and intraobserver variability in the response evaluation of cancer therapy according to RECIST and WHO-criteria. Acta Oncol (Madr) 49:509–514CrossRefGoogle Scholar
  9. 9.
    Dankerl P, Cavallaro A, Tsymbal A, Costa MJ, Suehling M, Janka R, Uder M, Hammon M (2013) A retrieval-based computer-aided diagnosis system for the characterization of liver lesions in CT scans. Acad Radiol 20:1526–34CrossRefPubMedGoogle Scholar
  10. 10.
    Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int J Med Inform 73:1–23CrossRefPubMedGoogle Scholar
  11. 11.
    Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B (2011) Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24:208–22CrossRefPubMedGoogle Scholar
  12. 12.
    Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recognit Lett 40:262–282CrossRefGoogle Scholar
  13. 13.
    Grosky WI (2002) Narrowing the semantic gap—improved text-based web document retrieval using visual features. IEEE Trans Multimed 4:189–200CrossRefGoogle Scholar
  14. 14.
    Deselaers T, Deserno TM, Müller H (2008) Automatic medical image annotation in ImageCLEF 2007: overview, results, and discussion. Pattern Recognit Lett 29:1988–1995CrossRefGoogle Scholar
  15. 15.
    van Ginneken B, Schaefer-Prokop CM, Prokop M (2011) Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 261:719–32CrossRefPubMedGoogle Scholar
  16. 16.
    Ponciano-Silva M, Souza JP, Bugatti PH, Bedo MVN, Kaster DS, Braga RT V, Bellucci \({\hat{A}}\)D, Azevedo-Marques PM, Traina C, Traina AJM (2013) Does a CBIR system really impact decisions of physicians in a clinical environment? In: Proceedings of the 26th IEEE international symposium on computer based medical system CBMS 2013, pp 41–46Google Scholar
  17. 17.
    An C, Rakhmonova G, Choi JY, Kim MJ (2016) Liver imaging reporting and data system (LI-RADS) version 2014: understanding and application of the diagnostic algorithm. Clin Mol Hepatol 22:296–307CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Mitchell DG, Bruix J, Sherman M, Sirlin CB (2015) LI-RADS (Liver Imaging Reporting and Data System): Summary, discussion, and consensus of the LI-RADS Management Working Group and future directions. Hepatology 61:1056–1065CrossRefPubMedGoogle Scholar
  19. 19.
    Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256:243–252CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D (2014) Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans Med Imaging 33:1669–1676CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Kumar A, Dyer S, Kim J, Li C, Leong PHW, Fulham M, Feng D (2016) Adapting content-based image retrieval techniques for the semantic annotation of medical images. Comput Med Imag Gr 49:37–45CrossRefGoogle Scholar
  22. 22.
    Xu J, Napel S, Greenspan H, Beaulieu CF, Agrawal N, Rubin DL (2012) Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval. Med Phys 39:5405CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Spanier AB, Joskowicz L (2014) Towards content-based image retrieval?: From computer generated features to semantic descriptions of liver CT scans. In: CLEF online work, notes. pp 438–447Google Scholar
  24. 24.
    Spanier AB, Joskowicz L (2017) Automatic Atlas-free multi-organ segmentation of contrast-enhanced CT scans. In: Hanbury A, Müller H, Langs G (eds) Cloud-Based Benchmarking Med Image Anal. Springer, Berlin, pp 145–164CrossRefGoogle Scholar
  25. 25.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2011) SLIC superpixels compared to state-of-the-art superpixel methods. Pattern Anal Mach Intell IEEE Trans 34:2274–2282CrossRefGoogle Scholar
  26. 26.
    Cha SH, Srihari SN (2002) On measuring the distance between histograms. Pattern Recognit 35:1355–1370CrossRefGoogle Scholar
  27. 27.
    Marvasti NB, Kökciyan N, Türkay R, Yazici A, Yolum P, Üsküdarl S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF (working notes), pp 329–340Google Scholar
  28. 28.
    Yolum P, Üsküdarl S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF online work, notes, pp 329–340Google Scholar
  29. 29.
    Pedregosa F, Grisel O, Weiss R, Passos A, Brucher M (2011) Scikit-learn?: Machine Learning in Python. J Mach Res 12:2825–2830Google Scholar
  30. 30.
    Kalervo J, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4):422–446Google Scholar
  31. 31.
    Spanier AB, Cohen D, Joskowicz L (2017) A new method for the automatic retrieval of medical cases based on the RadLex ontology. Int J Comput Assist Radiol Surg 12:471–484CrossRefPubMedGoogle Scholar
  32. 32.
    Chapelle O, Haffner P, Vapnik VN (1999) Support vector machines for histogram-based image classification. IEEE Trans Neural Netw 10:1055–1064CrossRefPubMedGoogle Scholar
  33. 33.
    Echegaray S, Gevaert O, Shah R, Kamaya A, Louie J, Kothary N, Napel S (2015) Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma. J Med Imaging 2:41011CrossRefGoogle Scholar
  34. 34.
    Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20:422–446CrossRefGoogle Scholar

Copyright information

© CARS 2017

Authors and Affiliations

  1. 1.The Rachel and Selim Benin School of Computer Science and EngineeringThe Hebrew University of Jerusalem Givat Ram CampusJerusalemIsrael
  2. 2.Alexander Grass Center for BioengineeringThe Hebrew University of JerusalemJerusalemIsrael
  3. 3.Department of RadiologyHadassah Hebrew University Medical CenterJerusalemIsrael
  4. 4.Department of Electrical-Electronics EngineeringBogazici UniversityIstanbulTurkey

Personalised recommendations