Journal of Digital Imaging

, Volume 27, Issue 6, pp 805–823 | Cite as

Test–Retest Reproducibility Analysis of Lung CT Image Features

  • Yoganand Balagurunathan
  • Virendra Kumar
  • Yuhua Gu
  • Jongphil Kim
  • Hua Wang
  • Ying Liu
  • Dmitry B. Goldgof
  • Lawrence O. Hall
  • Rene Korn
  • Binsheng Zhao
  • Lawrence H. Schwartz
  • Satrajit Basu
  • Steven Eschrich
  • Robert A. Gatenby
  • Robert J. GilliesEmail author


Quantitative size, shape, and texture features derived from computed tomographic (CT) images may be useful as predictive, prognostic, or response biomarkers in non-small cell lung cancer (NSCLC). However, to be useful, such features must be reproducible, non-redundant, and have a large dynamic range. We developed a set of quantitative three-dimensional (3D) features to describe segmented tumors and evaluated their reproducibility to select features with high potential to have prognostic utility. Thirty-two patients with NSCLC were subjected to unenhanced thoracic CT scans acquired within 15 min of each other under an approved protocol. Primary lung cancer lesions were segmented using semi-automatic 3D region growing algorithms. Following segmentation, 219 quantitative 3D features were extracted from each lesion, corresponding to size, shape, and texture, including features in transformed spaces (laws, wavelets). The most informative features were selected using the concordance correlation coefficient across test–retest, the biological range and a feature independence measure. There were 66 (30.14 %) features with concordance correlation coefficient ≥ 0.90 across test–retest and acceptable dynamic range. Of these, 42 features were non-redundant after grouping features with R 2 Bet ≥ 0.95. These reproducible features were found to be predictive of radiological prognosis. The area under the curve (AUC) was 91 % for a size-based feature and 92 % for the texture features (runlength, laws). We tested the ability of image features to predict a radiological prognostic score on an independent NSCLC (39 adenocarcinoma) samples, the AUC for texture features (runlength emphasis, energy) was 0.84 while the conventional size-based features (volume, longest diameter) was 0.80. Test–retest and correlation analyses have identified non-redundant CT image features with both high intra-patient reproducibility and inter-patient biological range. Thus making the case that quantitative image features are informative and prognostic biomarkers for NSCLC.


Test–retest reproducibility Lung cancer CT Quantitative image features 

Supplementary material

10278_2014_9716_MOESM1_ESM.pdf (626 kb)
ESM 1 (PDF 626 kb)


  1. 1.
    Nguyen T, Rangayyan R: Shape analysis of breast masses in mammograms via the fractal dimension. Conf Proc IEEE Eng Med Biol Soc 3:3210–3213, 2005PubMedGoogle Scholar
  2. 2.
    Schuster DP: The opportunities and challenges of developing imaging biomarkers to study lung function and disease. Am J Respir Crit Care Med 176(3):224–230, 2007PubMedCrossRefGoogle Scholar
  3. 3.
    Suzuki C, Jacobsson H, Hatschek T, et al: Radiologic measurements of tumor response to treatment: practical approaches and limitations. Radiographics 28(2):329–344, 2008PubMedCrossRefGoogle Scholar
  4. 4.
    Tuma RS: Sometimes size doesn't matter: reevaluating RECIST and tumor response rate endpoints. J Natl Cancer Inst 98(18):1272–1274, 2006PubMedCrossRefGoogle Scholar
  5. 5.
    Ganeshan B, Abaleke S, Young RC, et al: Texture analysis of non-small cell lung cancer on unenhanced computed tomography: initial evidence for a relationship with tumour glucose metabolism and stage. Cancer Imaging 10:137–143, 2010PubMedCentralPubMedCrossRefGoogle Scholar
  6. 6.
    Way TW, Sahiner B, Chan HP, et al: Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys 36(7):3086–3098, 2009PubMedCentralPubMedCrossRefGoogle Scholar
  7. 7.
    Samala R, Moreno W, You Y, et al: A novel approach to nodule feature optimization on thin section thoracic CT. Acad Radiol 16(4):418–427, 2009PubMedCrossRefGoogle Scholar
  8. 8.
    Lee MC, Boroczky L, Sungur-Stasik K, et al: Computer-aided diagnosis of pulmonary nodules using a two-step approach for feature selection and classifier ensemble construction. Artif Intell Med 50(1):43–53, 2010PubMedCrossRefGoogle Scholar
  9. 9.
    Zhu Y, Tan Y, Hua Y, et al: Feature selection and performance evaluation of support vector machine (SVM)-based classifier for differentiating benign and malignant pulmonary nodules by computed tomography. J Digit Imaging 23(1):51–65, 2010PubMedCentralPubMedCrossRefGoogle Scholar
  10. 10.
    Al-Kadi O, Watson D: Texture analysis of aggressive and nonaggressive lung tumor CE CT images. IEEE Trans Biomed Eng 55(7):1822–1830, 2008PubMedCrossRefGoogle Scholar
  11. 11.
    Kido S, Kuriyama K, Higashiyama M, et al: Fractal analysis of internal and peripheral textures of small peripheral bronchogenic carcinomas in thin-section computed tomography: comparison of bronchioloalveolar cell carcinomas with nonbronchioloalveolar cell carcinomas. J Comput Assist Tomogr 27(1):56–61, 2003PubMedCrossRefGoogle Scholar
  12. 12.
    Segal E, Sirlin CB, Ooi C, et al: Decoding global gene expression programs in liver cancer by noninvasive imaging. Nat Biotechnol 25(6):675–680, 2007PubMedCrossRefGoogle Scholar
  13. 13.
    Buckler AJ, Mozley PD, Schwartz L, et al: Volumetric CT in lung cancer: an example for the qualification of imaging as a biomarker. Acad Radiol 17(1):107–115, 2010PubMedCrossRefGoogle Scholar
  14. 14.
    America RSoN: Quantitative imaging biomarker alliance for volumetric CT image analysis: roadmap for a staged validation plan, 2010Google Scholar
  15. 15.
    Zhao B, James LP, Moskowitz CS, et al: Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology 252(1):263–272, 2009PubMedCentralPubMedCrossRefGoogle Scholar
  16. 16.
    RIDER. The Reference Image Database to Evaluate Therapy Response. Available at:;jsessionid=C78203F71E49C7EA3A43E0D213CE5555. Accessed 24 Jun 2014
  17. 17.
    Gu Y, Kumar V, Hall LO, et al: Automated delineation of lung tumors from CT images using a single click ensemble segmentation approach. Pattern Recogn 46(3):692–702, 2013CrossRefGoogle Scholar
  18. 18.
    NBIA. National Biomedical Imaging Archive. Available at: Accessed 30 June 2014
  19. 19.
    Definiens. Definiens AG, Munchen, Germany. Available at: Accessed 30 June 2014
  20. 20.
    Athelogou M, Schmidt G, Schaepe A, et al: Cognition network technology—a novel multimodal image analysis technique for automatic identification and quantification of biological image contents. In: Shorte SL, Frischknecht F Eds. Book cognition network technology—a novel multimodal image analysis technique for automatic identification and quantification of biological image contents. Springer-Verlag, New York City, 2007, pp 407–422Google Scholar
  21. 21.
    Baatz M, Zimmermann J, Blackmore CG: Automated analysis and detailed quantification of biomedical images using Definiens Congnition Network Technology. Comb Chem High Throughput Screen 12(9):908–916, 2009PubMedCrossRefGoogle Scholar
  22. 22.
    Bendtsen C, Kietzmann M, Korn R, Mozley P, Schmidt G, Binnig G: X-ray computed tomography: semiautomated volumetric analysis of late-stage lung tumors as a basis for response assessments. Int J Biomed Imaging, vol 2011, 2011Google Scholar
  23. 23.
    Basu S, Hall LO, Goldgof DB, et al: Developing a classifier model for lung tumors in ct-scan images. IEEE Intl Conf on Systems, Man and Cybernetics, (SMC 2011), Anchorage, Alaska, 2011Google Scholar
  24. 24.
    Lin LI-K: A concordance correlation coefficient to evaluate reproducibility. Biometrics 45:13, 1989CrossRefGoogle Scholar
  25. 25.
    RGD Steel JT: Principles and procedures of statistics. McGraw-Hill, New York, 1960Google Scholar
  26. 26.
    Colin C, Frank AW, Gramaji H, et al: An R-square measured of goodness of fit for some common nonlinear regression models. J Econ 77(2):1790–1792, 1997Google Scholar
  27. 27.
    Aoki T, Tomoda Y, Watanabe H, et al: Peripheral lung adenocarcinoma: correlation of thin-section CT findings with histologic prognostic factors and survival. Radiology 220(3):803–809, 2001PubMedCrossRefGoogle Scholar
  28. 28.
    Takashima S MY, Hasegawa M, Saito A, Haniuda M, Kadoya M. High-resolution CT features: prognostic significance in peripheral lung adenocarcinoma with bronchioloalveolar carcinoma components. Respiration: Int Rev Thorac Dis 70(1), 2003Google Scholar
  29. 29.
    Subramanian J, Simon R: Gene exression-based signature in lung cancer: ready for clinical use? JNCI 102(7):464–474, 2010PubMedCentralPubMedCrossRefGoogle Scholar
  30. 30.
    Jain AK, Zongker D: Feature selection: evaluation, application, and small sample performance. IEEE Trans Pattern Anal 19(2):153–158, 1997CrossRefGoogle Scholar
  31. 31.
    Pudil P, Novovičová J, Kittler J: Floating search methods in feature selection. Pattern Recogn Lett 15:1119–1125, 1994CrossRefGoogle Scholar
  32. 32.
    Saeys Y, Inza I: A review of feature selection techniques in bioinformatics. Bioinformatics 23:2507–2517, 2007PubMedCrossRefGoogle Scholar
  33. 33.
    Landis JR, Koch G: The measurement of observer agreement for categorical data. Biometrics 33:159–174, 1977PubMedCrossRefGoogle Scholar
  34. 34.
    Ganeshan B, Panayiotou E, Burnand K, et al: Tumour heterogeneity in non-small cell lung carcinoma assessed by CT texture analysis: a potential marker of survival. Eur Radiol 22(4):796–802, 2012PubMedCrossRefGoogle Scholar
  35. 35.
    Yanagawa M, Tanaka Y, Kusumoto M, et al: Automated assessment of malignant degree of small peripheral adenocarcinomas using volumetric CT data: correlation with pathologic prognostic factors. Lung Cancer 70(3):286–294, 2010PubMedCrossRefGoogle Scholar
  36. 36.
    John S: A direct approach to false discovery rate. J R Stat Soc B 64(3):479–498, 2002CrossRefGoogle Scholar
  37. 37.
    Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc 57(1):289–300, 1995Google Scholar
  38. 38.
    Zhao B, Oxnard G, Moskowitz CS, et al: A pilot study of volume measurement as a method of tumor response evaluation to aid biomarker development. Clin Cancer Res 16(18):4647–4653, 2010PubMedCentralPubMedCrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2014

Authors and Affiliations

  • Yoganand Balagurunathan
    • 1
  • Virendra Kumar
    • 1
  • Yuhua Gu
    • 1
  • Jongphil Kim
    • 2
  • Hua Wang
    • 1
    • 7
  • Ying Liu
    • 1
    • 7
  • Dmitry B. Goldgof
    • 3
  • Lawrence O. Hall
    • 3
  • Rene Korn
    • 4
  • Binsheng Zhao
    • 5
  • Lawrence H. Schwartz
    • 5
  • Satrajit Basu
    • 3
  • Steven Eschrich
    • 2
  • Robert A. Gatenby
    • 6
  • Robert J. Gillies
    • 1
    • 6
    • 8
    Email author
  1. 1.Department of Cancer Imaging and MetabolismH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  2. 2.Biostatistics and BioinformaticsH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  3. 3.Department of Computer Science and EngineeringUniversity of South FloridaTampaUSA
  4. 4.Definiens AGMunchenGermany
  5. 5.Department of RadiologyColumbia UniversityNew YorkUSA
  6. 6.RadiologyH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  7. 7.Department of RadiologyTianjin Medical University Cancer Institute and HospitalTianjinChina
  8. 8.Experimental Imaging ProgramH. Lee Moffitt Cancer Center and Research InstituteTampaUSA

Personalised recommendations