Texture-specific bag of visual words model and spatial cone matching-based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images
- 289 Downloads
The bag of visual words (BoVW) model is a powerful tool for feature representation that can integrate various handcrafted features like intensity, texture, and spatial information. In this paper, we propose a novel BoVW-based method that incorporates texture and spatial information for the content-based image retrieval to assist radiologists in clinical diagnosis.
This paper presents a texture-specific BoVW method to represent focal liver lesions (FLLs). Pixels in the region of interest (ROI) are classified into nine texture categories using the rotation-invariant uniform local binary pattern method. The BoVW-based features are calculated for each texture category. In addition, a spatial cone matching (SCM)-based representation strategy is proposed to describe the spatial information of the visual words in the ROI. In a pilot study, eight radiologists with different clinical experience performed diagnoses for 20 cases with and without the top six retrieved results. A total of 132 multiphase computed tomography volumes including five pathological types were collected.
The texture-specific BoVW was compared to other BoVW-based methods using the constructed dataset of FLLs. The results show that our proposed model outperforms the other three BoVW methods in discriminating different lesions. The SCM method, which adds spatial information to the orderless BoVW model, impacted the retrieval performance. In the pilot trial, the average diagnosis accuracy of the radiologists was improved from 66 to 80% using the retrieval system.
The preliminary results indicate that the texture-specific features and the SCM-based BoVW features can effectively characterize various liver lesions. The retrieval system has the potential to improve the diagnostic accuracy and the confidence of the radiologists.
KeywordsContent-based image retrieval Texture-specific Bag of visual words Spatial cone matching
This research was supported in part by the National Key Basic Research Program of China under the Grant No. 2015CB352400, in part by the National Key Research and Development Program of China under the Grant No. 2016YFB1200203-03, in part by the Recruitment Program of Global Experts (HAIOU Program) from Zhejiang Province, China, in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant No. 15H01130, No. 15K00253 and No.16H01436, and in part by the MEXT Support Program for the Strategic Research Foundation at Private Universities (2013–2017).
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. For this type of study, formal consent is not required.
- 2.Diamant I, Goldberger J, Klang E, Amitai M (2015) Multiphase liver lesions classification using relevant visual words based on mutual information. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI). IEEE, pp 407–410Google Scholar
- 6.Yu M, Feng Q, Yang W, Gao Y, Chen W (2012) Extraction of lesion-partitioned features and retrieval of contrast-enhanced liver images. Comput Math Methods Med 2012Google Scholar
- 9.Xu Y, Lin L, Hu H, Yu H, Jin C, Wang J, Han X, Chen YW (2016) Combined density, texture and shape features of multi-phase contrast-enhanced CT images for CBIR of focal liver lesions: a preliminary study. In: Chen YW, Toro C, Tanaka S, Howlett RJ, Jain LC (eds) Innovation in medicine and healthcare 2015. Springer, Berlin, pp 215–224CrossRefGoogle Scholar
- 10.Ojala T, Pietikainen M, Harwood D (1994) Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In: Proceedings of the 12th IAPR international conference on Pattern recognition, 1994. Vol. 1-conference a: computer vision & image processing. IEEE, pp 582–585Google Scholar
- 11.Burner A, Donner R, Mayerhoefer M, Holzer M, Kainberger F, Langs G (2011) Texture bags: anomaly retrieval in medical images based on local 3D-texture similarity. In: MICCAI international workshop on medical content-based retrieval for clinical decision support. Springer, Berlin, pp 116–127Google Scholar
- 12.Banerji S, Sinha A, Liu C (2013) A new bag of words LBP (BoWL) descriptor for scene image classification. In: International conference on computer analysis of images and patterns. Springer, Berlin, pp 490–497Google Scholar
- 13.Asherov M, Diamant I, Greenspan H (2014) Lung texture classification using bag of visual words. In: SPIE medical imaging. International society for optics and photonics, pp 90352K–90352KGoogle Scholar
- 15.Foncubierta-Rodríguez A, García Seco de Herrera A, Müller H (2013) Medical image retrieval using bag of meaningful visual words: unsupervised visual vocabulary pruning with PLSA. ACM international workshop on multimedia indexing and information retrieval for healthcare, pp 75–82Google Scholar
- 16.del Toro OAJ, Foncubiertarodríguez A, Depeursinge A, Müller H (2015) Texture classification of anatomical structures in CT using a context-free machine learning approach. In: SPIE medical imaging, vol 9414, pp 94140W–94140W-14Google Scholar
- 17.Depeursinge A, Foncubierta–Rodriguez A, Van de Ville D, Müller H (2012) Multiscale lung texture signature learning using the Riesz transform. Medical image computing & computer-assisted intervention: Miccai international conference on medical image computing & computer-assisted intervention, vol 15, p 517Google Scholar
- 18.Depeursinge A, Foncubierta-Rodriguez A, Van de Ville D, Müller H (2011) Lung texture classification using locally–oriented Riesz components. In: Medical image computing and computer-assisted intervention (MICCAI 2011). Springer, BerlinGoogle Scholar
- 19.Csurka G, Perronnin F (2010) Fisher vectors: beyond bag-of-visual-words image representations. In: International conference on computer vision, imaging and computer graphics. Springer, Berlin pp 28–42Google Scholar
- 20.Gadermayr M, Kogler H, Uhl A, Vécsei A (2015) Comparing endoscopic imaging configurations in computer-aided celiac disease diagnosis. In: 2015 International conference on image processing theory, tools and applications (IPTA). IEEE, pp 446–451Google Scholar
- 24.Roth HR, Lu L, Seff A, Cherry KM, Hoffman J, Wang S, Liu J, Turkey E, Summers RM (2014) A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 520–527Google Scholar
- 25.Akselrod-Ballin A, Karlinsky L, Alpert S, Hasoul S, Ben-Ari R, Barkan E (2016) A region based convolutional network for tumor detection and classification in breast mammography. In: International workshop on large-scale annotation of biomedical data and expert label synthesis. Springer, Berlin, pp 197–205Google Scholar
- 26.de Brebisson A, Montana G (2015) Deep neural networks for anatomical brain segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20–28Google Scholar
- 28.Anavi Y, Kogan I, Gelbart E, Geva O, Greenspan H (2015) A comparative study for chest radiograph image retrieval using binary texture and deep learning classification. In: 2015 37th Annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2940–2943Google Scholar
- 29.Bar Y, Diamant I, Wolf L, Greenspan H (2015) Deep learning with non-medical training used for chest pathology identification. In: SPIE medical imaging, international society for optics and photonics, pp 94140V–94140VGoogle Scholar
- 30.Bar Y, Diamant I, Wolf L, Lieberman S, Konen E, Greenspan H (2015) Chest pathology detection using deep learning with non-medical training. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI). IEEE, pp 294–297Google Scholar
- 31.Andrearczyk V, Whelan PF (2016) Deep learning for biomedical texture image analysis. In: Proceedings of the 18th Irish machine vision and image processing conference (IMVIP2016)Google Scholar
- 33.Vivanti R (2015) Automatic liver tumor segmentation in follow-up CT studies using convolutional neural networks. In: Proceedings of the MICCAI2015Google Scholar
- 34.Todoroki Y, Han X, Iwamoto Y, Lin L, Hu H, Chen YW (2017) Detection of liver tumor candidates from CT images using deep convolutional neural networks. In: Proceedings of the international conference on innovation in medicine and healthcare, pp 140–145Google Scholar
- 37.Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). IEEE, vol 2, pp 2169–2178Google Scholar
- 39.Marvasti NB, Kökciyan N, Türkay R, Yazici A, Yolum P, Üsküdarli S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF (working notes), pp 329–340Google Scholar
- 40.Dong C, Chen YW, Lin L, Hu H, Jin C, Yu H, Han X, Tateyama T (2016) Simultaneous segmentation of multiple organs using random walks. J Inf Process 24(2):320–329Google Scholar
- 41.Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: European conference on machine learning. Springer, Berlin, pp 137–142Google Scholar
- 43.Barla A, Odone F, Verri A (2003) Histogram intersection kernel for image classification. In: Proceedings of the 2003 international conference on image processing (ICIP 2003), vol 3. IEEE, p III-513Google Scholar
- 44.Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE international conference on image processing 2005, vol 2. IEEE, p II-370Google Scholar