Influence of segmentation margin on machine learning–based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas

  • Burak KocakEmail author
  • Ece Ates
  • Emine Sebnem Durmaz
  • Melis Baykara Ulusan
  • Ozgur Kilickesmez
Imaging Informatics and Artificial Intelligence



To determine the possible influence of segmentation margin on each step (feature reproducibility, selection, and classification) of the machine learning (ML)-based high-dimensional quantitative computed tomography (CT) texture analysis (qCT-TA) of renal clear cell carcinomas (RcCCs).

Materials and methods

For this retrospective study, 47 patients with RcCC were included from a public database. Two segmentations were obtained by two radiologists for each tumour: (i) contour-focused and (ii) margin shrinkage of 2 mm. Texture features were extracted from original, filtered, and transformed CT images. Feature selection was done using a correlation-based algorithm. The ML classifier was k-nearest neighbours. Classifications were performed with and without using synthetic minority over-sampling technique. Reference standard was nuclear grade (low versus high). Intraclass correlation coefficient (ICC), Pearson’s correlation coefficient, Wilcoxon signed-ranks test, and McNemar’s test were used in the analysis.


The segmentation with margin shrinkage of 2 mm (772 of 828; 93.2%) yielded more texture features with excellent reproducibility (ICC ≥ 0.9) than contour-focused segmentation (714 of 828; 86.2%), p < 0.0001. The feature selection algorithms resulted in different feature subsets for two segmentation datasets with only one common feature. All ML-based models based on contour-focused segmentation (area under the curve [AUC] range, 0.865–0.984) performed better than those with margin shrinkage of 2 mm (AUC range, 0.745–0.887), p < 0.05.


Each step of the ML-based high-dimensional qCT-TA was susceptible to a slight change of 2 mm in segmentation margin. Despite yielding fewer features with excellent reproducibility, use of the contour-focused segmentation provided better classification performance for distinguishing nuclear grade.

Key Points

Each step of a machine learning (ML)-based high-dimensional quantitative computed tomography texture analysis (qCT-TA) is sensitive to even a slight change of 2 mm in segmentation margin.

Despite yielding fewer texture features with excellent reproducibility, performing the segmentation focusing on the outermost boundary of the tumours provides better classification performance in ML-based qCT-TA of renal clear cell carcinomas for distinguishing nuclear grade.

Findings of an ML-based high-dimensional qCT-TA may not be reproducible in clinical practice even using the same feature selection algorithm and ML classifier unless the possible influence of the segmentation margin is considered.


Multidetector computed tomography Radiomics Clear cell renal cell carcinoma Machine learning Artificial intelligence 



Area under the curve


Contrast-enhanced computed tomography


Computed tomography


Intraclass correlation coefficient


k-nearest neighbours


Laplacian of Gaussian


Nearest neighbours


Quantitative computed tomography texture analysis


Quantitative texture analysis


Renal cell carcinoma


Renal clear cell carcinoma


The Cancer Genome Atlas-Kidney Renal Clear Cell Carcinoma


Waikato environment for knowledge analysis



The abstract of this study has also been submitted to the European Congress of Radiology (ECR) 2019 (Control Number 19-P-339-ECR). The authors acknowledge that they have previously used this public database (The Cancer Genome Atlas-Kidney Renal Clear Cell Carcinoma [TCGA-KIRC]) in different context.


The authors state that this work has not received any funding.

Compliance with ethical standards


The scientific guarantor of this publication is Burak Kocak, MD.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors (Burak Kocak, MD) has significant statistical expertise.

Informed consent

Written informed consent was not required for this study because all patients’ data included in this study are publicly and freely available for scientific purposes (The Cancer Genome Atlas-Kidney Renal Clear Cell Carcinoma [TCGA-KIRC]).

Ethical approval

Institutional Review Board approval was not required because all patients’ data included in this study are publicly and freely available for scientific purposes (The Cancer Genome Atlas-Kidney Renal Clear Cell Carcinoma [TCGA-KIRC]).

Study subjects or cohorts overlap

The authors acknowledge that they have previously used this public database (The Cancer Genome Atlas-Kidney Renal Clear Cell Carcinoma [TCGA-KIRC]) in different context.


• retrospective

• experimental

• performed using a publicly and freely available database

Supplementary material

330_2019_6003_MOESM1_ESM.docx (49 kb)
ESM 1 (DOCX 48 kb)


  1. 1.
    Lubner MG, Smith AD, Sandrasegaran K, Sahani DV, Pickhardt PJ (2017) CT texture analysis: definitions, applications, biologic correlates, and challenges. RadioGraphics 37:1483–1503.
  2. 2.
    Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577. CrossRefPubMedGoogle Scholar
  3. 3.
    Kocak B, Yardimci AH, Bektas CT et al (2018) Textural differences between renal cell carcinoma subtypes: machine learning-based quantitative computed tomography texture analysis with independent external validation. Eur J Radiol 107:149–157. CrossRefPubMedGoogle Scholar
  4. 4.
    Bektas CT, Kocak B, Yardimci AH et al (2018) Clear cell renal cell carcinoma: machine learning-based quantitative computed tomography texture analysis for prediction of Fuhrman nuclear grade. Eur Radiol.
  5. 5.
    Berenguer R, Pastor-Juan MDR, Canales-Vázquez J et al (2018) Radiomics of CT features may be nonreproducible and redundant: influence of CT acquisition parameters. Radiology 288:407–415.
  6. 6.
    Shafiq-ul-Hassan M, Zhang GG, Latifi K et al (2017) Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels. Med Phys 44:1050–1062. CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Mackin D, Fave X, Zhang L et al (2015) Measuring computed tomography scanner variability of radiomics features. Invest Radiol 50:757–765. CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    Balagurunathan Y, Gu Y, Wang H et al (2014) Reproducibility and prognosis of quantitative features extracted from CT images. Transl Oncol 7:72–87CrossRefGoogle Scholar
  9. 9.
    Balagurunathan Y, Kumar V, Gu Y et al (2014) Test-retest reproducibility analysis of lung CT image features. J Digit Imaging 27:805–823. CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Hunter LA, Krafft S, Stingo F et al (2013) High quality machine-robust image features: identification in nonsmall cell lung cancer computed tomography images. Med Phys 40:121916. CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Leijenaar RT, Carvalho S, Velazquez ER et al (2013) Stability of FDG-PET radiomics features: an integrated analysis of test-retest and inter-observer variability. Acta Oncol 52:1391–1397. CrossRefPubMedGoogle Scholar
  12. 12.
    Echegaray S, Gevaert O, Shah R et al (2015) Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma. J Med Imaging (Bellingham) 2:041011. CrossRefGoogle Scholar
  13. 13.
    Parmar C, Rios Velazquez E, Leijenaar R et al (2014) Robust radiomics feature quantification using semiautomatic volumetric segmentation. PLoS One 9:e102107. CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Fave X, Zhang L, Yang J et al (2016) Impact of image preprocessing on the volume dependence and prognostic potential of radiomics features in non-small cell lung cancer. Transl Cancer Res 5:349–363. CrossRefGoogle Scholar
  15. 15.
    Hu P, Wang J, Zhong H et al (2016) Reproducibility with repeat CT in radiomics study for rectal cancer. Oncotarget 7:71440–71446. CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Kim H, Park CM, Lee M et al (2016) Impact of reconstruction algorithms on CT radiomic features of pulmonary tumors: analysis of intra- and inter-reader variability and inter-reconstruction algorithm variability. PLoS One 11:e0164924. CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Akin O, Elnajjar P, Heller M, et al (2016) Radiology Data from The Cancer Genome Atlas Kidney Renal Clear Cell Carcinoma [TCGA-KIRC] collection. The Cancer Imaging Archive. Accessed 4 May 2018
  18. 18.
    Clark K, Vendt B, Smith K et al (2013) The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26:1045–1057. CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Kocak B, Durmaz ES, Ates E, Ulusan MB (2019) Radiogenomics in clear cell renal cell carcinoma: machine learning–based high-dimensional quantitative CT texture analysis in predicting PBRM1 mutation status. AJR Am J Roentgenol 1–9.
  20. 20.
    Cohan RH, Sherman LS, Korobkin M, Bass JC, Francis IR (1995) Renal masses: assessment of corticomedullary-phase and nephrographic-phase CT scans. Radiology 196:445–451.
  21. 21.
    van Griethuysen JJM, Fedorov A, Parmar C et al (2017) Computational radiomics system to decode the radiographic phenotype. Cancer Res 77:e104–e107. CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Koo TK, Li MY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15:155–163. CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Kohavi R, John GH (1997) Wrappers for feature subset selection. Artif Intell 97:273–324. CrossRefGoogle Scholar
  24. 24.
    Bermejo P, Gamez JA, Puerta JM (2011) Improving incremental wrapper-based subset selection via replacement and early stopping. Intern J Pattern Recognit Artif Intell 25:605–625.
  25. 25.
    Dormann CF, Elith J, Bacher S et al (2013) Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography (Cop) 36:27–46. CrossRefGoogle Scholar
  26. 26.
    Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357. CrossRefGoogle Scholar
  27. 27.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30Google Scholar
  28. 28.
    Yan L, Liu Z, Wang G et al (2015) Angiomyolipoma with minimal fat. Acad Radiol 22:1115–1121. CrossRefPubMedGoogle Scholar
  29. 29.
    Hodgdon T, McInnes MD, Schieda N, Flood TA, Lamb L, Thornhill RE (2015) Can quantitative CT texture analysis be used to differentiate fat-poor renal angiomyolipoma from renal cell carcinoma on unenhanced CT images? Radiology 276:787–796.
  30. 30.
    Klatte T, Patard JJ, de Martino M et al (2008) Tumor size does not predict risk of metastatic disease or prognosis of small renal cell carcinomas. J Urol 179:1719–1726.
  31. 31.
    Frank I, Blute ML, Cheville JC, Lohse CM, Weaver AL, Zincke H (2002) An outcome prediction model for patients with clear cell renal cell carcinoma treated with radical nephrectomy based on tumor stage, size, grade and necrosis: the SSIGN score. J Urol 168:2395–2400.
  32. 32.
    Zisman A, Pantuck AJ, Dorey F et al (2002) Mathematical model to predict individual survival for patients with renal cell carcinoma. J Clin Oncol 20:1368–1374. CrossRefPubMedGoogle Scholar
  33. 33.
    Marconi L, Dabestani S, Lam TB et al (2016) Systematic review and meta-analysis of diagnostic accuracy of percutaneous renal tumour biopsy. Eur Urol 69:660–673. CrossRefPubMedGoogle Scholar
  34. 34.
    Jewett MAS, Mattar K, Basiuk J et al (2011) Active surveillance of small renal masses: progression patterns of early stage kidney cancer. Eur Urol 60:39–44. CrossRefPubMedGoogle Scholar
  35. 35.
    Abou Youssif T, Tanguay S (2009) Natural history and management of small renal masses. Curr Oncol 16(Suppl 1):S2–S7PubMedGoogle Scholar
  36. 36.
    Ng F, Kozarski R, Ganeshan B, Goh V (2013) Assessment of tumor heterogeneity by CT texture analysis: can the largest cross-sectional area be used as an alternative to whole tumor analysis? Eur J Radiol 82:342–348. CrossRefPubMedGoogle Scholar
  37. 37.
    Schieda N, Lim RS, Krishna S, McInnes MDF, Flood TA, Thornhill RE (2018) Diagnostic accuracy of unenhanced CT analysis to differentiate low-grade from high-grade chromophobe renal cell carcinoma. Am J Roentgenol:1–9.
  38. 38.
    Collewet G, Strzelecki M, Mariette F (2004) Influence of MRI acquisition protocols and image intensity normalization methods on texture classification. Magn Reson Imaging 22:81–91. CrossRefPubMedGoogle Scholar

Copyright information

© European Society of Radiology 2019

Authors and Affiliations

  1. 1.Department of RadiologyIstanbul Training and Research HospitalIstanbulTurkey
  2. 2.Department of Radiology, Cerrahpasa Medical FacultyIstanbul University-CerrahpasaIstanbulTurkey

Personalised recommendations