Radiomic feature reproducibility in contrast-enhanced CT of the pancreas is affected by variabilities in scan parameters and manual segmentation

  • Rikiya Yamashita
  • Thomas Perrin
  • Jayasree Chakraborty
  • Joanne F. Chou
  • Natally Horvat
  • Maura A. Koszalka
  • Abhishek Midya
  • Mithat Gonen
  • Peter Allen
  • William R. Jarnagin
  • Amber L. Simpson
  • Richard K. G. DoEmail author



This study aims to measure the reproducibility of radiomic features in pancreatic parenchyma and ductal adenocarcinomas (PDAC) in patients who underwent consecutive contrast-enhanced computed tomography (CECT) scans.


In this IRB-approved and HIPAA-compliant retrospective study, 37 pairs of scans from 37 unique patients who underwent CECTs within a 2-week interval were included in the analysis of the reproducibility of features derived from pancreatic parenchyma, and a subset of 18 pairs of scans were further analyzed for the reproducibility of features derived from PDAC. In each patient, pancreatic parenchyma and pancreatic tumor (when present) were manually segmented by two radiologists independently. A total of 266 radiomic features were extracted from the pancreatic parenchyma and tumor region and also the volume and diameter of the tumor. The concordance correlation coefficient (CCC) was calculated to assess feature reproducibility for each patient in three scenarios: (1) different radiologists, same CECT; (2) same radiologist, different CECTs; and (3) different radiologists, different CECTs.


Among pancreatic parenchyma-derived features, using a threshold of CCC > 0.90, 58/266 (21.8%) and 48/266 (18.1%) features met the threshold for scenario 1, 14/266 (5.3%) and 15/266 (5.6%) for scenario 2, and 14/266 (5.3%) and 10/266 (3.8%) for scenario 3. Among pancreatic tumor-derived features, 11/268 (4.1%) and 17/268 (6.3%) features met the threshold for scenario 1, 1/268 (0.4%) and 5/268 (1.9%) features met the threshold for scenario 2, and no features for scenario 3 met the threshold, respectively.


Variations between CECT scans affected radiomic feature reproducibility to a greater extent than variation in segmentation. A smaller number of pancreatic tumor-derived radiomic features were reproducible compared with pancreatic parenchyma-derived radiomic features under the same conditions.

Key Points

• For pancreatic-derived radiomic features from contrast-enhanced CT (CECT), fewer than 25% are reproducible (with a threshold of CCC < 0.9) in a clinical heterogeneous dataset.

• Variations between CECT scans affected the number of reproducible radiomic features to a greater extent than variations in radiologist segmentation.

• A smaller number of pancreatic tumor-derived radiomic features were reproducible compared with pancreatic parenchyma-derived radiomic features under the same conditions.


Reproducibility of results Pancreatic ductal carcinoma X-ray computed tomography Radiomics Texture analysis 

Abbreviations and acronyms


Angle co-occurrence matrix


Concordance correlation coefficient


Contrast-enhanced computed tomography


Digital Imaging and Communications in Medicine


Fractal dimension


Gray-level co-occurrence matrix


Intraclass correlation coefficient


Intensity histogram


Interquartile range


Local binary pattern


Pancreatic ductal adenocarcinoma


Run length matrix



We thank Luz Adriana Escobar Hoyos and Juliana Brooke Schilsky for their support in figure creation and Joanne Chin for editorial assistance.


This study was funded in part through the 2016 Society of Abdominal Radiology Wylie J. Dodds Research Award, the National Institutes of Health/National Cancer Institute Cancer Center Support Grant P30 CA008748, and the Japan Society Promotion of Science Overseas Research Fellowships JSPS/OT/290125.

Compliance with ethical standards


The scientific guarantor of this publication is Richard K. G. Do, MD, PhD.

Conflict of interest

The authors declare that they have no conflict of interest.

Statistics and biometry

Two of the authors (Joanne Chou and Mithat Gonen, PhD) have significant statistical expertise.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.


• retrospective

• cross-sectional study

• performed at one institution

Supplementary material

330_2019_6381_MOESM1_ESM.docx (24 kb)
ESM 1 (DOCX 24 kb)
330_2019_6381_MOESM2_ESM.xlsx (18 kb)
ESM 2 (XLSX 17 kb)
330_2019_6381_MOESM3_ESM.docx (15.8 mb)
ESM 3 (DOCX 16189 kb)


  1. 1.
    Siegel RL, Miller KD, Jemal A (2018) Cancer statistics, 2018. CA Cancer J Clin 68:7–30. CrossRefGoogle Scholar
  2. 2.
    Garrido-Laguna I, Hidalgo M (2015) Pancreatic cancer: from state-of-the-art treatments to promising novel therapies. Nat Rev Clin Oncol 12:319–334. CrossRefGoogle Scholar
  3. 3.
    Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577. CrossRefGoogle Scholar
  4. 4.
    Lambin P, Rios-Velazquez E, Leijenaar R et al (2012) Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 48:441–446. CrossRefGoogle Scholar
  5. 5.
    Aerts HJWL, Velazquez ER, Leijenaar RTH et al (2014) Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun 5:4006. CrossRefGoogle Scholar
  6. 6.
    Sandrasegaran K, Lin Y, Asare-Sawiri M et al (2019) CT texture analysis of pancreatic cancer. Eur Radiol 29(3):1067–1073.
  7. 7.
    Chen X, Oshima K, Schott D et al (2017) Assessment of treatment response during chemoradiation therapy for pancreatic cancer based on quantitative radiomic analysis of daily CTs: an exploratory study. PLoS One 12:e0178961. CrossRefGoogle Scholar
  8. 8.
    Chakraborty J, Langdon-Embry L, Cunanan KM et al (2017) Preliminary study of tumor heterogeneity in imaging predicts two year survival in pancreatic cancer patients. PLoS One 12:e0188022. CrossRefGoogle Scholar
  9. 9.
    Kim BR, Kim JH, Ahn SJ et al (2019) CT prediction of resectability and prognosis in patients with pancreatic ductal adenocarcinoma after neoadjuvant treatment using image findings and texture analysis. Eur Radiol 29(1):362–372.
  10. 10.
    Yun G, Kim YH, Lee YJ et al (2018) Tumor heterogeneity of pancreas head cancer assessed by CT texture analysis: association with survival outcomes after curative resection. Sci Rep 8:7226. CrossRefGoogle Scholar
  11. 11.
    Attiyeh MA, Chakraborty J, Doussot A et al (2018) Survival prediction in pancreatic ductal adenocarcinoma by quantitative computed tomography image analysis. Ann Surg Oncol 25:1034–1042. CrossRefGoogle Scholar
  12. 12.
    Berenguer R, Pastor-Juan MDR, Canales-Vázquez J et al (2018) Radiomics of CT features may be nonreproducible and redundant: influence of CT acquisition parameters. Radiology 288(2):407–415.
  13. 13.
    Perrin T, Midya A, Yamashita R et al (2018) Short-term reproducibility of radiomic features in liver parenchyma and liver malignancies on contrast-enhanced CT imaging. Abdom Radiol (NY) 43(12):3271–3278.
  14. 14.
    van Timmeren JE, Leijenaar RTH, van Elmpt W et al (2016) Test-retest data for radiomics feature stability analysis: generalizable or study-specific? Tomography 2:361–365. CrossRefGoogle Scholar
  15. 15.
    Summers RM (2017) Texture analysis in radiology: does the emperor have no clothes? Abdom Radiol (NY) 42:342–345. CrossRefGoogle Scholar
  16. 16.
    Kumar V, Gu Y, Basu S et al (2012) Radiomics: the process and the challenges. Magn Reson Imaging 30:1234–1248. CrossRefGoogle Scholar
  17. 17.
    Lambin P, Leijenaar RTH, Deist TM et al (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762. CrossRefGoogle Scholar
  18. 18.
    Zhao B, Tan Y, Tsai W-Y et al (2016) Reproducibility of radiomics for deciphering tumor phenotype with imaging. Sci Rep 6:23428. CrossRefGoogle Scholar
  19. 19.
    Pavic M, Bogowicz M, Würms X et al (2018) Influence of inter-observer delineation variability on radiomics stability in different tumor sites. Acta Oncol 57(8):1070–1074.
  20. 20.
    Mingqiang Y, Kidiyo K, Joseph R (2008) A survey of shape feature extraction techniques. Peng-Yeng Yin. Pattern Recognition, IN-TECH, pp 43–90.
  21. 21.
    Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3:610–621. CrossRefGoogle Scholar
  22. 22.
    Yang X, Tridandapani S, Beitler JJ et al (2012) Ultrasound GLCM texture analysis of radiation-induced parotid-gland injury in head-and-neck cancer radiotherapy: an in vivo study of late toxicity. Med Phys 39:5732–5739. CrossRefGoogle Scholar
  23. 23.
    Banik S, Rangayyan RM, Desautels JEL (2013) Measures of angular spread and entropy for the detection of architectural distortion in prior mammograms. Int J Comput Assist Radiol Surg 8:121–134. CrossRefGoogle Scholar
  24. 24.
    Tang X (1998) Texture information in run-length matrices. IEEE Trans Image Process 7:1602–1609. CrossRefGoogle Scholar
  25. 25.
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29:51–59. CrossRefGoogle Scholar
  26. 26.
    Pietikäinen M, Hadid A, Zhao G, Ahonen T (2011) Local binary patterns for still images. In: Computer vision using local binary patterns. Springer, London, pp 13–47CrossRefGoogle Scholar
  27. 27.
    Mehta R, Egiazarian K (2013) Rotated local binary pattern (RLBP): rotation invariant texture descriptor. In 2nd International Conference on Pattern Recognition Applications and Methods, ICPRAM 2013, Barcelona, Spain, 15.-18.2.2013, pp 497-502. (International Conference on Pattern Recognition Applications and Methods). Institute of Electrical and Electronics Engineers IEEEGoogle Scholar
  28. 28.
    Al-Kadi OS, Watson D (2008) Texture analysis of aggressive and nonaggressive lung tumor CE CT images. IEEE Trans Biomed Eng 55:1822–1830. CrossRefGoogle Scholar
  29. 29.
    Costa AF, Humpire-Mamani G, Traina AJM (2012) An efficient algorithm for fractal analysis of textures. 25th SIBGRAPI Conference on Graphics, Patterns and Images, Ouro Preto, pp 39–46.
  30. 30.
    Chakraborty J, Rangayyan RM, Banik S et al (2012) Statistical measures of orientation of texture for the detection of architectural distortion in prior mammograms of interval-cancer. J Electron Imaging 21:033010–033011. CrossRefGoogle Scholar
  31. 31.
    Chakraborty J, Rangayyan RM, Banik S, et al (2012) Detection of architectural distortion in prior mammograms using statistical measures of orientation of texture. Proc. SPIE 8315, Medical Imaging 2012: Computer-Aided Diagnosis, 831521 (23 February 2012).
  32. 32.
    Chakraborty J, Midya A, Mukhopadhyay S, Sadhu A (2013) Automatic characterization of masses in mammograms. 6th International Conference on Biomedical Engineering and Informatics, Hangzhou, pp 111–115.
  33. 33.
    Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26:297. CrossRefGoogle Scholar
  34. 34.
    ISO 5725-2:1994 - Accuracy (trueness and precision) of measurement methods and results—part 2: basic method for the determination of repeatability and reproducibility of a standard measurement method. Accessed 14 Dec 2018
  35. 35.
    Carrasco JL, Jover L (2003) Estimating the generalized concordance correlation coefficient through variance components. Biometrics 59:849–858. CrossRefGoogle Scholar
  36. 36.
    Balagurunathan Y, Gu Y, Wang H et al (2014) Reproducibility and prognosis of quantitative features extracted from CT images. Transl Oncol 7:72–87. CrossRefGoogle Scholar
  37. 37.
    Balagurunathan Y, Kumar V, Gu Y et al (2014) Test-retest reproducibility analysis of lung CT image features. J Digit Imaging 27:805–823. CrossRefGoogle Scholar
  38. 38.
    Armato SG, Meyer CR, Mcnitt-Gray MF et al (2008) The Reference Image Database to Evaluate Response to therapy in lung cancer (RIDER) project: a resource for the development of change-analysis software. Clin Pharmacol Ther 84:448–456. CrossRefGoogle Scholar
  39. 39.
    Zwanenburg A, Leger S, Vallières M, et al (2016) Image biomarker standardisation initiative. arXiv:1612.07003v7 [cs.CV]Google Scholar

Copyright information

© European Society of Radiology 2019

Authors and Affiliations

  • Rikiya Yamashita
    • 1
  • Thomas Perrin
    • 2
  • Jayasree Chakraborty
    • 2
  • Joanne F. Chou
    • 3
  • Natally Horvat
    • 1
  • Maura A. Koszalka
    • 2
  • Abhishek Midya
    • 2
  • Mithat Gonen
    • 3
  • Peter Allen
    • 2
  • William R. Jarnagin
    • 2
  • Amber L. Simpson
    • 2
  • Richard K. G. Do
    • 1
    Email author
  1. 1.Department of RadiologyMemorial Sloan Kettering Cancer CenterNew YorkUSA
  2. 2.Department of SurgeryMemorial Sloan Kettering Cancer CenterNew YorkUSA
  3. 3.Department of Epidemiology and BiostatisticsMemorial Sloan Kettering Cancer CenterNew YorkUSA

Personalised recommendations