Machine Learning Methods for Optimal Radiomics-Based Differentiation Between Recurrence and Inflammation: Application to Nasopharyngeal Carcinoma Post-therapy PET/CT Images



To identify optimal machine learning methods for radiomics-based differentiation of local recurrence versus inflammation from post-treatment nasopharyngeal positron emission tomography/X-ray computed tomography (PET/CT) images.


Seventy-six nasopharyngeal carcinoma (NPC) patients were enrolled (41/35 local recurrence/inflammation as confirmed by pathology). Four hundred eighty-seven radiomics features were extracted from PET images for each patient. The diagnostic performance was investigated for 42 cross-combinations derived from 6 feature selection methods and 7 classifiers. Of the original cohort, 70 % was applied for feature selection and classifier development, and the remaining 30 % used as an independent validation set. The diagnostic performance was evaluated using area under the ROC curve (AUC), test error, sensitivity, and specificity. Furthermore, the performance of the radiomics signatures against routine features was statistically compared using DeLong’s method.


The cross-combination fisher score (FSCR) + k-nearest neighborhood (KNN), FSCR + support vector machines with radial basis function kernel (RBF-SVM), FSCR + random forest (RF), and minimum redundancy maximum relevance (MRMR) + RBF-SVM outperformed others in terms of accuracy (AUC 0.883, 0.867, 0.892, 0.883; sensitivity 0.833, 0.864, 0.831, 0.750; specificity 1, 1, 0.873, 1) and reliability (test error 0.091, 0.136, 0.150, 0.136). Compared with conventional metrics, the radiomics signatures showed higher AUC values (0.867–0.892 vs. 0.817), though the differences were not statistically significant (p = 0.462–0.560).


This study identified the most accurate and reliable machine learning methods, which could enhance the application of radiomics methods in the precision of diagnosis of NPC.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2


  1. 1.

    Chan ATC, Teo PML, Ngan RK, Leung TW, Lau WH, Zee B, Leung SF, Cheung FY, Yeo W, Yiu HH, Yu KH, Chiu KW, Chan DT, Mok T, Yuen KT, Mo F, Lai M, Kwan WH, Choi P, Johnson PJ (2002) Concurrent chemotherapy-radiotherapy compared with radiotherapy alone in locoregionally advanced nasopharyngeal carcinoma: progression-free survival analysis of a phase III randomized trial. J Clin Oncol 20:2038–2044

    CAS  PubMed  Google Scholar 

  2. 2.

    Al-Sarraf M, LeBlanc M, Giri PG et al (1998) Chemoradiotherapy versus radiotherapy in patients with advanced nasopharyngeal cancer: phase III randomized intergroup study 0099. J Clin Oncol 16:1310–1317

    CAS  PubMed  Google Scholar 

  3. 3.

    Han F, Zhao C, Huang S-M, Lu LX, Huang Y, Deng XW, Mai WY, Teh BS, Butler EB, Lu TX (2012) Long-term outcomes and prognostic factors of re-irradiation for locally recurrent nasopharyngeal carcinoma using intensity-modulated radiotherapy. Clin Oncol 24:569–576

    Google Scholar 

  4. 4.

    Sun X, Su S, Chen C, Han F, Zhao C, Xiao W, Deng X, Huang S, Lin C, Lu T (2014) Long-term outcomes of intensity-modulated radiotherapy for 868 patients with nasopharyngeal carcinoma: an analysis of survival and treatment toxicities. Radiother Oncol 110:398–403

    PubMed  Google Scholar 

  5. 5.

    Leung T-W, Tung SY, Sze W-K, Sze WM, Wong VYW, Wong CS, O SK (2000) Salvage radiation therapy for locally recurrent nasopharyngeal carcinoma. Int J Radiat Oncol 48:1331–1338

    CAS  Google Scholar 

  6. 6.

    Comoretto M, Balestreri L, Borsatti E, Cimitan M, Franchin G, Lise M (2008) Detection and restaging of residual and/or recurrent nasopharyngeal carcinoma after chemotherapy and radiation therapy: comparison of MR imaging and FDG PET/CT. Radiology 249:203–211

    PubMed  Google Scholar 

  7. 7.

    Yen R-F, Hung R-L, Pan M-H, Wang YH, Huang KM, Lui LT, Kao CH (2003) 18-Fluoro-2-deoxyglucose positron emission tomography in detecting residual/recurrent nasopharyngeal carcinomas and comparison with magnetic resonance imaging. Cancer 98:283–287

    PubMed  Google Scholar 

  8. 8.

    Zhou H, Shen G, Zhang W, Cai H, Zhou Y, Li L (2016) 18F-FDG PET/CT for the diagnosis of residual or recurrent nasopharyngeal carcinoma after radiotherapy: a metaanalysis. J Nucl Med 57:342–347

    CAS  PubMed  Google Scholar 

  9. 9.

    Shuhang N, Tungchieh JC, Shengchieh C et al (2004) Clinical usefulness of 18F-FDG PET in nasopharyngeal carcinoma patients with questionable MRI findings for recurrence. J Nucl Med 45:1669–1676

    Google Scholar 

  10. 10.

    Wu H-b, Wang Q-s, Wang M-f et al (2011) Preliminary study of 11C-choline PET/CT for T staging of locally advanced nasopharyngeal carcinoma: comparison with 18F-FDG PET/CT. J Nucl Med 52:341–346

    CAS  PubMed  Google Scholar 

  11. 11.

    Lee N, Yoo IR, Park SY, Yoon H, Lee Y, Oh JK (2015) Significance of incidental nasopharyngeal uptake on 18F-FDG PET/CT: patterns of benign/physiologic uptake and differentiation from malignancy. Nucl Med Mol Imaging 49:11–18

    CAS  PubMed  Google Scholar 

  12. 12.

    Wahl RL (2009) Principles and practice of PET and PET/CT. Lippincott Williams & Wilkins, Philadelphia

    Google Scholar 

  13. 13.

    Gerlinger M, Rowan AJ, Horswell S, Math M, Larkin J, Endesfelder D, Gronroos E, Martinez P, Matthews N, Stewart A, Tarpey P, Varela I, Phillimore B, Begum S, McDonald N, Butler A, Jones D, Raine K, Latimer C, Santos CR, Nohadani M, Eklund AC, Spencer-Dene B, Clark G, Pickering L, Stamp G, Gore M, Szallasi Z, Downward J, Futreal PA, Swanton C (2012) Intratumor heterogeneity and branched evolution revealed by multiregion sequencing. N Engl J Med 366:883–892

    CAS  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RGPM, Granton P, Zegers CML, Gillies R, Boellard R, Dekker A, Aerts HJWL (2012) Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 48:441–446

    PubMed  PubMed Central  Google Scholar 

  15. 15.

    Aerts HJWL (2016) The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol 2:1636

    PubMed  Google Scholar 

  16. 16.

    Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong EEC, van Timmeren J, Sanduleanu S, Larue RTHM, Even AJG, Jochems A, van Wijk Y, Woodruff H, van Soest J, Lustberg T, Roelofs E, van Elmpt W, Dekker A, Mottaghy FM, Wildberger JE, Walsh S (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762

    PubMed  Google Scholar 

  17. 17.

    Coroller TP, Grossmann P, Hou Y, Rios Velazquez E, Leijenaar RTH, Hermann G, Lambin P, Haibe-Kains B, Mak RH, Aerts HJWL (2015) CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma. Radiother Oncol 114:345–350

    PubMed  PubMed Central  Google Scholar 

  18. 18.

    Li H, Zhu Y, Burnside ES, Huang E, Drukker K, Hoadley KA, Fan C, Conzen SD, Zuley M, Net JM, Sutton E, Whitman GJ, Morris E, Perou CM, Ji Y, Giger ML (2016) Quantitative MRI radiomics in the prediction of molecular classifications of breast cancer subtypes in the TCGA/TCIA data set. Npj Breast Cancer 2:16012

    PubMed  PubMed Central  Google Scholar 

  19. 19.

    Oakden-Rayner L, Carneiro G, Bessen T, Nascimento JC, Bradley AP, Palmer LJ (2017) Precision radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework. Sci Rep 7:1648

    PubMed  PubMed Central  Google Scholar 

  20. 20.

    Vallières M, Freeman CR, Skamene SR, El Naqa I (2015) A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Phys Med Biol 60:5471–5496

    PubMed  Google Scholar 

  21. 21.

    Cheng N-M, Fang Y-HD, Lee L, Chang JTC, Tsan DL, Ng SH, Wang HM, Liao CT, Yang LY, Hsu CH, Yen TC (2015) Zone-size nonuniformity of 18F-FDG PET regional textural features predicts survival in patients with oropharyngeal cancer. Eur J Nucl Med Mol Imaging 42:419–428

    CAS  PubMed  Google Scholar 

  22. 22.

    Win T, Miles KA, Janes SM, Ganeshan B, Shastry M, Endozo R, Meagher M, Shortman RI, Wan S, Kayani I, Ell PJ, Groves AM (2013) Tumor heterogeneity and permeability as measured on the CT component of PET/CT predict survival in patients with non-small cell lung cancer. Clin Cancer Res 19:3591–3599

    CAS  PubMed  Google Scholar 

  23. 23.

    Lovinfosse P, Janvary ZL, Coucke P, Jodogne S, Bernard C, Hatt M, Visvikis D, Jansen N, Duysinx B, Hustinx R (2016) FDG PET/CT texture analysis for predicting the outcome of lung cancer treated by stereotactic body radiation therapy. Eur J Nucl Med Mol Imaging 43:1453–1460

    CAS  PubMed  Google Scholar 

  24. 24.

    Mu W, Chen Z, Liang Y, Shen W, Yang F, Dai R, Wu N, Tian J (2015) Staging of cervical cancer based on tumor heterogeneity characterized by texture features on 18F-FDG PET images. Phys Med Biol 60:5123–5139

    PubMed  Google Scholar 

  25. 25.

    Galavis PE, Hollensen C, Jallow N, Paliwal B, Jeraj R (2010) Variability of textural features in FDG PET images due to different acquisition modes and reconstruction parameters. Acta Oncol 49:1012–1016

    PubMed  PubMed Central  Google Scholar 

  26. 26.

    Doumou G, Siddique M, Tsoumpas C, Goh V, Cook GJ (2015) The precision of textural analysis in 18 F-FDG-PET scans of oesophageal cancer. Eur Radiol 25:2805–2812

    PubMed  Google Scholar 

  27. 27.

    Leijenaar RTH, Nalbantov G, Carvalho S, van Elmpt WJC, Troost EGC, Boellaard R, Aerts HJWL, Gillies RJ, Lambin P (2015) The effect of SUV discretization in quantitative FDG-PET radiomics: the need for standardized methodology in tumor texture analysis. Sci Rep 5:11075

    CAS  PubMed  PubMed Central  Google Scholar 

  28. 28.

    Shiri I, Rahmim A, Abdollahi H et al (2017) Applying radiomics and machine learning on PET images to predict lung metastases in soft tissue sarcoma patients. Eur J Nucl Med Mol Imaging 44:S47

    Google Scholar 

  29. 29.

    Wu W, Parmar C, Grossmann P et al (2016) Exploratory study to identify radiomics classifiers for lung cancer histology. Front Oncol 6:71

    PubMed  PubMed Central  Google Scholar 

  30. 30.

    Parmar C, Grossmann P, Rietveld D et al (2015) Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer. Front Oncol 5:272

    PubMed  PubMed Central  Google Scholar 

  31. 31.

    Parmar C, Grossmann P, Bussink J, Lambin P, Aerts HJWL (2015) Machine learning methods for quantitative radiomic biomarkers. Sci Rep 5:13087

    CAS  PubMed  PubMed Central  Google Scholar 

  32. 32.

    Hawkins SH, Korecki JN, Balagurunathan Y, Yuhua Gu, Kumar V, Basu S, Hall LO, Goldgof DB, Gatenby RA, Gillies RJ (2014) Predicting outcomes of non-small cell lung cancer using CT image features. IEEE Access 2:1418–1426

    Google Scholar 

  33. 33.

    Boellaard R, Delgado-Bolton R, Oyen WJG, Giammarile F, Tatsch K, Eschner W, Verzijlbergen FJ, Barrington SF, Pike LC, Weber WA, Stroobants S, Delbeke D, Donohoe KJ, Holbrook S, Graham MM, Testanera G, Hoekstra OS, Zijlstra J, Visser E, Hoekstra CJ, Pruim J, Willemsen A, Arends B, Kotzerke J, Bockisch A, Beyer T, Chiti A, Krause BJ, European Association of Nuclear Medicine (EANM) (2015) FDG PET/CT: EANM procedure guidelines for tumour imaging: version 2.0. Eur J Nucl Med Mol Imaging 42:328–354

    CAS  Google Scholar 

  34. 34.

    Ashrafinia S, Dalaie P, Yan R et al (2018) Application of texture and radiomics analysis to clinical myocardial perfusion SPECT imaging [abstract]. J Nucl Med 59:94

    Google Scholar 

  35. 35.

    Ashrafinia S (2019) Quantitative nuclear medicine imaging using advanced image reconstruction and radiomics. Johns Hopkins University, Ph.D. dissertation

    Google Scholar 

  36. 36.

    Zwanenburg A, Leger S, Vallières M, Löck S (2019) Image biomarker standardisation initiative. ArXiv Preprint ArXiv: 1612.07003v7

  37. 37.

    Brown G, Pocock A, Zhao M-J, Luján M (2012) Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J Mach Learn Res 13:27–66

    Google Scholar 

  38. 38.

    Roffo G (2016) Feature selection library (MATLAB Toolbox). ArXiv Preprint ArXiv 1607:01327

    Google Scholar 

  39. 39.

    Foley D (1972) Considerations of sample and feature size. IEEE Trans Inf Theory 18:618–626

    Google Scholar 

  40. 40.

    Chalkidou A, O’Doherty MJ, Marsden PK (2015) False discovery rates in PET and CT studies with texture features: a systematic review. PLoS One 10:e0124165

    PubMed  PubMed Central  Google Scholar 

  41. 41.

    DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

    CAS  PubMed  PubMed Central  Google Scholar 

  42. 42.

    Cook GJR, Siddique M, Taylor BP, Yip C, Chicklore S, Goh V (2014) Radiomics in PET: principles and applications. Clin Transl Imaging 2:269–276

    Google Scholar 

  43. 43.

    Yip SSF, Aerts HJWL (2016) Applications and limitations of radiomics. Phys Med Biol 61:R150–R166

    CAS  PubMed  PubMed Central  Google Scholar 

  44. 44.

    Soussan M, Orlhac F, Boubaya M, Zelek L, Ziol M, Eder V, Buvat I (2014) Relationship between tumor heterogeneity measured on FDG-PET/CT and pathological prognostic factors in invasive breast cancer. PLoS One 9:e94017

    PubMed  PubMed Central  Google Scholar 

  45. 45.

    Yang X, He J, Wang J, Li W, Liu C, Gao D, Guan Y (2018) CT-based radiomics signature for differentiating solitary granulomatous nodules from solid lung adenocarcinoma. Lung Cancer 125:109–114

    Google Scholar 

  46. 46.

    van Timmeren JE, Leijenaar RTH, van Elmpt W, Reymen B, Oberije C, Monshouwer R, Bussink J, Brink C, Hansen O, Lambin P (2017) Survival prediction of non-small cell lung cancer patients using radiomics analyses of cone-beam CT images. Radiother Oncol 123:363–369

    PubMed  Google Scholar 

  47. 47.

    Lv W, Yuan Q, Wang Q, Ma J, Feng Q, Chen W, Rahmim A, Lu L (2019) Radiomics analysis of PET and CT components of PET/CT imaging integrated with clinical parameters: application to prognosis for nasopharyngeal carcinoma. Mol Imaging Biol.

    CAS  PubMed  Google Scholar 

  48. 48.

    Yip SSF, Coroller TP, Sanford NN et al (2016) Relationship between the temporal changes in positron-emission-tomography-imaging-based textural features and pathologic response and survival in esophageal cancer patients. Front Oncol 6:72

    PubMed  PubMed Central  Google Scholar 

  49. 49.

    Coroller TP, Agrawal V, Narayan V, Hou Y, Grossmann P, Lee SW, Mak RH, Aerts HJWL (2016) Radiomic phenotype features predict pathological response in non-small cell lung cancer. Radiother Oncol 119:480–486

    PubMed  PubMed Central  Google Scholar 

  50. 50.

    Deist TM, Dankers FJWM, Valdes G, Wijsman R, Hsu IC, Oberije C, Lustberg T, Soest J, Hoebers F, Jochems A, el Naqa I, Wee L, Morin O, Raleigh DR, Bots W, Kaanders JH, Belderbos J, Kwint M, Solberg T, Monshouwer R, Bussink J, Dekker A, Lambin P (2018) Machine learning algorithms for outcome prediction in (chemo)radiotherapy: an empirical comparison of classifiers. Med Phys 45:3449–3459

    PubMed  PubMed Central  Google Scholar 

  51. 51.

    Lu L, Lv W, Jiang J, Ma J, Feng Q, Rahmim A, Chen W (2016) Robustness of radiomic features in [11C]choline and [18F]FDG PET/CT imaging of nasopharyngeal carcinoma: impact of segmentation and discretization. Mol Imaging Biol 18:935–945

    CAS  PubMed  Google Scholar 

  52. 52.

    Lv W, Yuan Q, Wang Q, Ma J, Jiang J, Yang W, Feng Q, Chen W, Rahmim A, Lu L (2018) Robustness versus disease differentiation when varying parameter settings in radiomics features: application to nasopharyngeal PET/CT. Eur Radiol 28:3245–3254

    PubMed  Google Scholar 

  53. 53.

    Wu Z, Zhao J, Gao P, Song Y, Sun J, Chen X, Ma B, Wang Z (2017) Prognostic value of pretreatment standardized uptake value of F-18-fluorodeoxyglucose PET in patients with gastric cancer: a meta-analysis. BMC Cancer 17:275

    PubMed  PubMed Central  Google Scholar 

  54. 54.

    Hyun SH, Choi JY, Shim YM, Kim K, Lee SJ, Cho YS, Lee JY, Lee KH, Kim BT (2010) Prognostic value of metabolic tumor volume measured by 18F-fluorodeoxyglucose positron emission tomography in patients with esophageal carcinoma. Ann Surg Oncol 17:115–122

    PubMed  Google Scholar 

  55. 55.

    Nakamura K, Hongo A, Kodama J, Hiramatsu Y (2011) The measurement of SUVmax of the primary tumor is predictive of prognosis for patients with endometrial cancer. Gynecol Oncol 123:82–87

    PubMed  Google Scholar 

  56. 56.

    Brooks FJ, Grigsby PW (2014) The effect of small tumor volumes on studies of intratumoral heterogeneity of tracer uptake. J Nucl Med 55:37–42

    CAS  PubMed  Google Scholar 

  57. 57.

    Hatt M, Majdoub M, Vallieres M, Tixier F, le Rest CC, Groheux D, Hindie E, Martineau A, Pradier O, Hustinx R, Perdrisot R, Guillevin R, el Naqa I, Visvikis D (2015) 18F-FDG PET uptake characterization through texture analysis: investigating the complementary nature of heterogeneity and functional tumor volume in a multi-cancer site patient cohort. J Nucl Med 56:38–44

    CAS  PubMed  Google Scholar 

Download references


This study was supported by the National Natural Science Foundation of China under grant 81871437, and the Guangdong Province Universities and Colleges Pearl River Scholar Funded Scheme (Lijun Lu, 2018).

Author information



Corresponding author

Correspondence to Lijun Lu.

Ethics declarations

This retrospective study was approved by the Institutional Review Board and informed consent was waived.

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material


(PDF 569 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Du, D., Feng, H., Lv, W. et al. Machine Learning Methods for Optimal Radiomics-Based Differentiation Between Recurrence and Inflammation: Application to Nasopharyngeal Carcinoma Post-therapy PET/CT Images. Mol Imaging Biol 22, 730–738 (2020).

Download citation

Key words

  • Radiomics
  • Nasopharyngeal carcinoma
  • Machine learning
  • Diagnosis
  • PET/CT