Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement
- 412 Downloads
To evaluate radiomics studies according to radiomics quality score (RQS) and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) to provide objective measurement of radiomics research.
Materials and methods
PubMed and Embase were searched for studies published in high clinical imaging journals until December 2018 using the terms “radiomics” and “radiogenomics.” Studies were scored against the items in the RQS and TRIPOD guidelines. Subgroup analyses were performed for journal type (clinical vs. imaging), intended use (diagnostic vs. prognostic), and imaging modality (CT vs. MRI), and articles were compared using Fisher’s exact test and Mann-Whitney analysis.
Seventy-seven articles were included. The mean RQS score was 26.1% of the maximum (9.4 out of 36). The RQS was low in demonstration of clinical utility (19.5%), test-retest analysis (6.5%), prospective study (3.9%), and open science (3.9%). None of the studies conducted a phantom or cost-effectiveness analysis. The adherence rate for TRIPOD was 57.8% (mean) and was particularly low in reporting title (2.6%), stating study objective in abstract and introduction (7.8% and 16.9%), blind assessment of outcome (14.3%), sample size (6.5%), and missing data (11.7%) categories. Studies in clinical journals scored higher and more frequently adopted external validation than imaging journals.
The overall scientific quality and reporting of radiomics studies is insufficient. Scientific improvements need to be made to feature reproducibility, analysis of clinical utility, and open science categories. Reporting of study objectives, blind assessment, sample size, and missing data is deemed to be necessary.
• The overall scientific quality and reporting of radiomics studies is insufficient.
• The RQS was low in demonstration of clinical utility, test-retest analysis, prospective study, and open science.
• Room for improvement was shown in TRIPOD in stating study objective in abstract and introduction, blind assessment of outcome, sample size, and missing data categories.
KeywordsNeoplasm Machine learning Quality improvement Computed tomography Magnetic resonance imaging
Radiomics quality score,
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (grant number: NRF-2017R1A2A2A05001217 and grant number: NRF-2017R1C1B2007258).
Compliance with ethical standards
The scientific guarantor of this publication is Jeong Hoon Kim.
Conflict of interest
The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.
Statistics and biometry
One of the authors has significant statistical expertise (Seo Young Park, 8 years of experience).
Written informed consent was not required because of the nature of our study, which was a study based on research articles.
Institutional Review Board approval was not required because of the nature of our study, which was a study based on research articles.
• cross-sectional study
• performed at one institution
- 2.Lambin P, Leijenaar RTH, Deist TM et al (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762Google Scholar
- 8.Moons KG, Altman DG, Reitsma JB et al (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1–W73Google Scholar
- 10.Whiting PF, Rutjes AW, Westwood ME et al (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155:529–536Google Scholar
- 14.Huang YQ, Liang CH, He L et al (2016) Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. J Clin Oncol 34:2157–2164Google Scholar
- 16.Kickingereder P, Burth S, Wick A et al (2016) Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology 280:880–889Google Scholar
- 25.Lohmann P, Stoffels G, Ceccon G et al (2017) Radiation injury vs. recurrent brain metastasis: combining textural feature radiomics analysis and standard parameters may increase (18)F-FET PET accuracy without dynamic scans. Eur Radiol 27:2916–2927Google Scholar
- 28.Wang J, Wu CJ, Bao ML, Zhang J, Wang XN, Zhang YD (2017) Machine learning-based analysis of MR radiomics can help to improve the diagnostic performance of PI-RADS v2 in clinically relevant prostate cancer. Eur Radiol 27:4082–4090Google Scholar
- 31.Yuan M, Zhang YD, Pu XH et al (2017) Comparison of a radiomic biomarker with volumetric analysis for decoding tumour phenotypes of lung adenocarcinoma with different disease-specific survival. Eur Radiol 27:4857–4865Google Scholar
- 39.Chen Y, Chen TW, Wu CQ et al (2018) Radiomics model of contrast-enhanced computed tomography for predicting the recurrence of acute pancreatitis. Eur Radiol 29:4408–4417Google Scholar
- 41.Dong F, Li Q, Xu D et al (2018) Differentiation between pilocytic astrocytoma and glioblastoma: a decision tree model using contrast-enhanced magnetic resonance imaging-derived quantitative radiomic features. Eur Radiol 29:3968–3975Google Scholar
- 48.Kim JY, Park JE, Jo Y et al (2018) Incorporating diffusion- and perfusion-weighted MRI into a radiomics model improves diagnostic performance for pseudoprogression in glioblastoma patients. Neuro Oncol 21:404–414Google Scholar
- 49.Kniep HC, Madesta F, Schneider T et al (2018) Radiomics of brain MRI: utility in prediction of metastatic tumor type. Radiology 180946Google Scholar
- 50.Multiparametric ultrasomics of significant liver fibrosis: a machine learning-based analysis. Eur Radiol 29:1496–1506Google Scholar
- 53.Li ZC, Bai H, Sun Q et al (2018) Multiregional radiomics features from multiparametric MRI for prediction of MGMT methylation status in glioblastoma multiforme: a multicentre study. Eur Radiol 28:3640–3650Google Scholar
- 54.Liang W, Yang P, Huang R et al (2018) A combined nomogram model to preoperatively predict histologic grade in pancreatic neuroendocrine tumors. Clin Cancer Res 25:584–594Google Scholar
- 56.Lu CF, Hsu FT, Hsieh KL et al (2018) Machine learning-based radiomics for molecular subtyping of gliomas. Clin Cancer Res 24:4429–4436Google Scholar
- 68.Truhn D, Schrading S, Haarburger C, Schneider H, Merhof D, Kuhl C (2018) Radiomic versus convolutional neural networks analysis for classification of contrast-enhancing lesions at multiparametric breast MRI. Radiology 181352Google Scholar
- 73.Zhang Y, Zhang B, Liang F et al (2018) Radiomics features on noncontrast-enhanced CT scan can precisely classify AVM-related hematomas from other spontaneous intraparenchymal hematoma types. Eur Radiol 29:2157–2165Google Scholar
- 74.Zhang Z, Yang J, Ho A et al (2018) A predictive model for distinguishing radiation necrosis from tumour progression after gamma knife radiosurgery based on radiomic features from MR images. Eur Radiol 28:2255–2263Google Scholar
- 76.Zinn PO, Singh SK, Kotrotsou A et al (2018) A coclinical radiogenomic validation study: conserved magnetic resonance radiomic appearance of periostin-expressing glioblastoma in patients and xenograft models. Clin Cancer Res 24:6288–6299Google Scholar
- 77.Choe J, Lee SM, Do KH et al (2019) Prognostic value of radiomic analysis of iodine overlay maps from dual-energy computed tomography in patients with resectable lung cancer. Eur Radiol 29:915–923Google Scholar
- 78.Hu T, Wang S, Huang L et al (2019) A clinical-radiomics nomogram for the preoperative prediction of lung metastasis in colorectal cancer patients with indeterminate pulmonary nodules. Eur Radiol 29:439–449Google Scholar
- 79.Ji GW, Zhang YD, Zhang H et al (2019) Biliary tract cancer at CT: a radiomics-based model to predict lymph node metastasis and survival outcomes. Radiology 290:90–98Google Scholar
- 80.Kontos D, Winham SJ, Oustimov A et al (2019) Radiomic phenotypes of mammographic parenchymal complexity: toward augmenting breast density in breast cancer risk assessment. Radiology 290:41–49Google Scholar
- 81.Qu J, Shen C, Qin J et al (2019) The MR radiomic signature can predict preoperative lymph node metastasis in patients with esophageal cancer. Eur Radiol 29:906–914Google Scholar
- 82.Tan X, Ma Z, Yan L, Ye W, Liu Z, Liang C (2019) Radiomics nomogram outperforms size criteria in discriminating lymph node metastasis in resectable esophageal squamous cell carcinoma. Eur Radiol 29:392–400Google Scholar
- 83.Wei J, Yang G, Hao X et al (2019) A multi-sequence and habitat-based MRI radiomics signature for preoperative prediction of MGMT promoter methylation in astrocytomas with prognostic implication. Eur Radiol 29:877–888Google Scholar
- 84.Bonekamp D, Kohl S, Wiesenfarth M et al (2018) Radiomic machine learning for characterization of prostate lesions with MRI: comparison to ADC values. Radiology 289:128–137Google Scholar
- 85.Park H, Lim Y, Ko ES et al (2018) Radiomics signature on magnetic resonance imaging: association with disease-free survival in patients with invasive breast cancer. Clin Cancer Res 24:4705–4714Google Scholar