Skip to main content
SpringerLink
Log in

External validation of an MR-based radiomic model predictive of locoregional control in oropharyngeal cancer

  • Head and Neck
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To externally validate a pre-treatment MR-based radiomics model predictive of locoregional control in oropharyngeal squamous cell carcinoma (OPSCC) and to assess the impact of differences between datasets on the predictive performance.

Methods

Radiomic features, as defined in our previously published radiomics model, were extracted from the primary tumor volumes of 157 OPSCC patients in a different institute. The developed radiomics model was validated using this cohort. Additionally, parameters influencing performance, such as patient subgroups, MRI acquisition, and post-processing steps on prediction performance will be investigated. For this analysis, matched subgroups (based on human papillomavirus (HPV) status of the tumor, T-stage, and tumor subsite) and a subgroup with only patients with 4-mm slice thickness were studied. Also the influence of harmonization techniques (ComBat harmonization, quantile normalization) and the impact of feature stability across observers and centers were studied. Model performances were assessed by area under the curve (AUC), sensitivity, and specificity.

Results

Performance of the published model (AUC/sensitivity/specificity: 0.74/0.75/0.60) drops when applied on the validation cohort (AUC/sensitivity/specificity: 0.64/0.68/0.60). The performance of the full validation cohort improves slightly when the model is validated using a patient group with comparable HPV status of the tumor (AUC/sensitivity/specificity: 0.68/0.74/0.60), using patients acquired with a slice thickness of 4 mm (AUC/sensitivity/specificity: 0.67/0.73/0.57), or when quantile harmonization was performed (AUC/sensitivity/specificity: 0.66/0.69/0.60).

Conclusion

The previously published model shows its generalizability and can be applied on data acquired from different vendors and protocols. Harmonization techniques and subgroup definition influence performance of predictive radiomics models.

Key Points

• Radiomics, a noninvasive quantitative image analysis technique, can support the radiologist by enhancing diagnostic accuracy and/or treatment decision-making.

• A previously published model shows its generalizability and could be applied on data acquired from different vendors and protocols.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AUC:

Area under the curve

CRT:

Chemoradiotherapy

DSC:

Dice similarity coefficient

HD:

Hausdorff distance

HPV:

Human papillomavirus

ICC:

Intraobserver correlation coefficient

LRC:

Locoregional control

MRI:

Magnetic resonance imaging

OPSCC:

Oropharyngeal squamous cell carcinoma

References

  1. Fruehwald-Pallamar J, Hesselink JR, Mafee MF, Holzer-Fruehwald L, Czerny C, Mayerhoefer ME (2016) Texture-based analysis of 100 MR examinations of head and neck tumors - is it possible to discriminate between benign and malignant masses in a multicenter trial? NMR Biomed. 188:195–202. https://doi.org/10.1055/s-0041-106066

    Article  CAS  Google Scholar 

  2. Bos P, van den Brekel MWM, Gouw ZAR et al (2021) Clinical variables and magnetic resonance imaging-based radiomics predict human papillomavirus status of oropharyngeal cancer. Head Neck. 43(2):485–495. https://doi.org/10.1002/hed.26505

    Article  PubMed  Google Scholar 

  3. Yu Y, He Z, Ouyang J et al (2021) Magnetic resonance imaging radiomics predicts preoperative axillary lymph node metastasis to support surgical decisions and is associated with tumor microenvironment in invasive breast cancer: a machine learning, multicenter study. EBioMedicine. 69:103460. https://doi.org/10.1016/j.ebiom.2021.103460

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Aerts HJWL, Velazquez ER, Leijenaar RTH et al (2014) Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 5:4006. https://doi.org/10.1038/ncomms5006

    Article  CAS  PubMed  Google Scholar 

  5. Mes SW, van Velden FHP, Peltenburg B et al (2020) Outcome prediction of head and neck squamous cell carcinoma by MRI radiomic signatures. Eur Radiol. 30:6311–6321. https://doi.org/10.1007/s00330-020-06962-y

    Article  PubMed  PubMed Central  Google Scholar 

  6. Bos P, van den Brekel MWM, Gouw ZAR et al (2021) Improved outcome prediction of oropharyngeal cancer by combining clinical and MRI features in machine learning models. Eur J Radiol. 139:109701. https://doi.org/10.1016/j.ejrad.2021.109701

    Article  PubMed  Google Scholar 

  7. Guha A, Connor S, Anjari M et al (2020) Radiomic analysis for response assessment in advanced head and neck cancers, a distant dream or an inevitable reality? A systematic review of the current level of evidence. Br J Radiol. 93:20190496. https://doi.org/10.1259/bjr.20190496

    Article  PubMed  PubMed Central  Google Scholar 

  8. Granzier RWY, Ibrahim A, Primakov SP et al (2021) MRI-based radiomics analysis for the pretreatment prediction of pathologic complete tumor response to neoadjuvant systemic therapy in breast cancer patients: a multicenter study. Cancers. 13:2447. https://doi.org/10.3390/cancers13102447

    Article  PubMed  PubMed Central  Google Scholar 

  9. Song J, Yin Y, Wang H, Chang Z, Liu Z, Cui L (2020) A review of original articles published in the emerging field of radiomics. Eur J Radiol. 127:108991. https://doi.org/10.1016/j.ejrad.2020.108991

    Article  PubMed  Google Scholar 

  10. Park JE, Park SY, Kim HJ, Kim HS (2019) Reproducibility and generalizability in radiomics modeling: possible strategies in radiologic and statistical perspectives. Korean J Radiol. 20:1124–1137. https://doi.org/10.3348/kjr.2018.0070

    Article  PubMed  PubMed Central  Google Scholar 

  11. Jethanandani A, Lin TA, Volpe S et al (2018) Exploring applications of radiomics in magnetic resonance imaging of head and neck cancer: a systematic review. Front Oncol. 8:131. https://doi.org/10.3389/fonc.2018.00131

    Article  PubMed  PubMed Central  Google Scholar 

  12. Rai R, Holloway LC, Brink C et al (2020) Multicenter evaluation of MRI-based radiomic features: a phantom study. Med Phys. 47(7):3053–3063. https://doi.org/10.1002/mp.14173

    Article  CAS  Google Scholar 

  13. Mayerhoefer ME, Szomolanyi P, Jirak D, Materka A, Trattnig S (2009) Effects of MRI acquisition parameter variations and protocol heterogeneity on the results of texture analysis and pattern discrimination: an application-oriented study. Med Phys. 36(4):1236–1243. https://doi.org/10.1118/1.3081408

    Article  PubMed  Google Scholar 

  14. Wahid KA, He R, McDonald BA et al (2021) Intensity standardization methods in magnetic resonance imaging of head and neck cancer. Phys Imag Radiat Oncol. 20:88–93. https://doi.org/10.1016/j.phro.2021.11.001

    Article  Google Scholar 

  15. van Velden FHP, Kramer GM, Frings V et al (2016) Repeatability of radiomic features in non-small-cell lung cancer [18F]FDG-PET/CT studies: impact of reconstruction and delineation. Mol Imaging Biol. 18:788–795. https://doi.org/10.1007/s11307-016-0940-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Ibrahim A, Refaee T, Leijenaar RTH et al (2021) The application of a workflow integrating the variable reproducibility and harmonizability of radiomic features on a phantom dataset. PLoS One. 16(5):e0251147. https://doi.org/10.1371/journal.pone.0251147

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Leijenaar RTH, Carvalho S, Hoebers FJP et al (2015) External validation of a prognostic CT-based radiomic signature in oropharyngeal squamous cell carcinoma. Acta Oncol. 54:1423–1429. https://doi.org/10.3109/0284186X.2015.1061214

    Article  CAS  PubMed  Google Scholar 

  18. Leijenaar RTH, Bogowicz M, Jochems A et al (2018) Development and validation of a radiomic signature to predict HPV (p16) status from standard CT imaging: a multicenter study. Br J Radiol. 91:2017049811075. https://doi.org/10.1259/bjr.20170498

    Article  Google Scholar 

  19. Zhai TT, Wesseling F, Langendijk JA et al (2021) External validation of nodal failure prediction models including radiomics in head and neck cancer. Oral Oncol. 112:105083. https://doi.org/10.1016/j.oraloncology.2020.105083

    Article  CAS  PubMed  Google Scholar 

  20. Romeo V, Cuocolo R, Ricciardi C et al (2020) Prediction of tumor grade and nodal status in oropharyngeal and oral cavity squamous-cell carcinoma using a radiomic approach. Anticancer Res. 40:271–280. https://doi.org/10.21873/anticanres.13949

    Article  PubMed  Google Scholar 

  21. Yuan Y, Ren J, Shi Y, Tao X (2019) MRI-based radiomic signature as predictive marker for patients with head and neck squamous cell carcinoma. Eur J Radiol. 117:193–198. https://doi.org/10.1016/j.ejrad.2019.06.019

    Article  PubMed  Google Scholar 

  22. Martens RM, Noij DP, Koopman T et al (2019) Predictive value of quantitative diffusion-weighted imaging and 18-F-FDG-PET in head and neck squamous cell carcinoma treated by (chemo)radiotherapy. Eur J Radiol. 113:39–50. https://doi.org/10.1016/j.ejrad.2019.01.031

    Article  PubMed  Google Scholar 

  23. Martens RM, Koopman T, Lavini C et al (2021) Multiparametric functional MRI and 18F-FDG-PET for survival prediction in patients with head and neck squamous cell carcinoma treated with (chemo)radiation. Eur Radiol. 31:616–628. https://doi.org/10.1007/s00330-020-07163-3

    Article  PubMed  Google Scholar 

  24. van Griethuysen JJM, Fedorov A, Parmar C et al (2017) Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77:e104–e107. https://doi.org/10.1158/0008-5472.CAN-17-0339

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology. 26(3):297–302. https://doi.org/10.2307/1932409

    Article  Google Scholar 

  26. Huttenlocher DP, Klanderman GA, Rucklidge WJ (1992) Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 15(9):850–863. https://doi.org/10.1109/CVPR.1992.223209

    Article  Google Scholar 

  27. Johnson WE, Li C, Rabinovic A (2007) Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 8(1):118–127. https://doi.org/10.1093/biostatistics/kxj037

    Article  PubMed  Google Scholar 

  28. Fortin JP, Cullen N, Sheline YI et al (2019) Harmonization of cortical thickness measurements across scanners and sites. Neuroimage. 167:104–120. https://doi.org/10.1016/j.neuroimage.2017.11.024

    Article  Google Scholar 

  29. Warnat P, Eils R, Brors B (2005) Cross-platform analysis of cancer microarray data improves gene expression based classification of phenotypes. BMC Bioinform. 6:265. https://doi.org/10.1186/1471-2105-6-265

    Article  CAS  Google Scholar 

  30. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 143:29–36. https://doi.org/10.1148/radiology.143.1.7063747

    Article  CAS  PubMed  Google Scholar 

  31. Baeßler B, Weiss K, Dos Santos DP (2019) Robustness and reproducibility of radiomics in magnetic resonance imaging: a phantom study. Invest Radiol. 54:221–228. https://doi.org/10.1097/RLI.0000000000000530

    Article  PubMed  Google Scholar 

  32. Galizia MS, Töre HG, Chalian H, McCarthy R, Salem R, Yaghmai V (2012) MDCT necrosis quantification in the assessment of hepatocellular carcinoma response to yttrium 90 radioembolization therapy. Comparison of two-dimensional and volumetric techniques. Acad Radiol. 19:48–54. https://doi.org/10.1016/j.acra.2011.09.005

    Article  PubMed  Google Scholar 

  33. Zhou Z, Li S, Qin G, Folkert M, Jiang S, Wang J (2020) Multi-objective based radiomic feature selection for lesion malignancy classification. IEEE J Biomed Health Inform. 24(1):194–204. https://doi.org/10.1109/JBHI.2019.2902298

    Article  PubMed  Google Scholar 

  34. Ang KK, Harris J, Wheeler R et al (2020) Human papillomavirus and survival of patients with oropharyngeal cancer. N Engl J Med. 363:24–35. https://doi.org/10.1056/NEJMoa0912217

    Article  CAS  Google Scholar 

  35. Fakhry C, Zhang Q, Nguyen-Tan PF et al (2014) Human papillomavirus and overall survival after progression of oropharyngeal squamous cell carcinoma. J Clin Oncol. 32:3365–3373. https://doi.org/10.1200/JCO.2014.55.1937

    Article  PubMed  PubMed Central  Google Scholar 

  36. Park SH, Lim H, Bae BK et al (2021) Robustness of magnetic resonance radiomic features to pixel size resampling and interpolation in patients with cervical cancer. Cancer Imaging. 21:19. https://doi.org/10.1186/s40644-021-00388-5

    Article  PubMed  PubMed Central  Google Scholar 

  37. Papadimitroulas P, Brocki L, Chung NC et al (2021) Artificial intelligence: deep learning in oncological radiomics and challenges of interpretability and data harmonization. Phys Med. 83:108–121. https://doi.org/10.1016/j.ejmp.2021.03.009

    Article  PubMed  Google Scholar 

  38. Masson I, Da-ano R, Lucia F et al (2021) Statistical harmonization can improve the development of a multicenter CT based radiomic model predictive of non-response to induction chemotherapy in laryngeal cancers. Med Phys. 48(7):4099–4109. https://doi.org/10.1002/mp.14948

    Article  PubMed  Google Scholar 

  39. Foy JJ, Al-Hallaq HA, Grekoski V et al (2020) Harmonization of radiomic feature variability resulting from differences in CT image acquisition and reconstruction: assessment in a cadaveric liver. Phys Med Biol. 65:205008. https://doi.org/10.1088/1361-6560/abb172

    Article  PubMed  Google Scholar 

  40. Da-Ano R, Visvikis D, Hatt M (2020) Harmonization strategies for multicenter radiomics investigations. Phys Med Biol. 65:24TR02. https://doi.org/10.1088/1361-6560/aba798

    Article  CAS  PubMed  Google Scholar 

  41. Ligero M, Jordi-Ollero O, Bernatowicz K et al (2021) Minimizing acquisition-related radiomics variability by image resampling and batch effect correction to allow for large-scale data analysis. Eur Radiol. 31:1460–1470. https://doi.org/10.1007/s00330-020-07174-0

    Article  PubMed  Google Scholar 

  42. Noortman WA, Vriens D, Mooij CDY et al (2021) The influence of the exclusion of central necrosis on [18F]FDG PET radiomic analysis. Diagnostics. 11:1296. https://doi.org/10.3390/diagnostics11071296

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Papanikolaou N, Matos C, Koh DM (2020) How to develop a meaningful radiomic signature for clinical use in oncologic patients. Cancer Imaging. 20:33. https://doi.org/10.1186/s40644-020-00311-4

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We hereby acknowledge financial support from the Verwelius Foundation and Willem Meindert De Hoop Stichting.

Funding

This study has received funding by Verwelius Foundation and Willem Meindert De Hoop Stichting.

Author information

Authors and Affiliations

  1. Department of Radiology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066, CX, Amsterdam, The Netherlands

    Paula Bos, Joost J. M. van Griethuysen, Regina G. H. Beets-Tan & Jonas A. Castelijns

  2. Department of Head and Neck Oncology and Surgery, The Netherlands Cancer Institute, Amsterdam, The Netherlands

    Paula Bos & Michiel W. M. van den Brekel

  3. GROW School for Oncology and Developmental Biology – University of Maastricht, Maastricht, The Netherlands

    Paula Bos

  4. Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Cancer Center Amsterdam, Amsterdam, The Netherlands

    Roland M. Martens, Pim de Graaf, Bas Jasperse, Ronald Boellaard & Jonas A. Castelijns

  5. Department of Otolaryngology – Head and Neck Surgery, Amsterdam UMC, Vrije Universiteit Amsterdam, Cancer Center Amsterdam, Amsterdam, The Netherlands

    C. René Leemans

  6. Epidemiology & Biostatistics, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands

    Mark A. van de Wiel

  7. MRC Biostatistics Unit, Cambridge University, Cambridge, UK

    Mark A. van de Wiel

  8. Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, Amsterdam, The Netherlands

    Michiel W. M. van den Brekel

Authors
  1. Paula Bos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roland M. Martens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pim de Graaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bas Jasperse
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joost J. M. van Griethuysen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ronald Boellaard
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. C. René Leemans
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Regina G. H. Beets-Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mark A. van de Wiel
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Michiel W. M. van den Brekel
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jonas A. Castelijns
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paula Bos.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Jonas Castelijns.

Conflict of interest

Regina G. H. Beets-Tan is member of the European Radiology Advisory Editorial Board. She has not taken part in the review or selection process of this article. The remaining authors of this manuscript have declared no conflict of interest.

Statistics and biometry

Mark A. van de Wiel kindly provided statistical advice for this manuscript.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

• retrospective

• diagnostic or prognostic study

• multicenter study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bos, P., Martens, R.M., de Graaf, P. et al. External validation of an MR-based radiomic model predictive of locoregional control in oropharyngeal cancer. Eur Radiol 33, 2850–2860 (2023). https://doi.org/10.1007/s00330-022-09255-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-09255-8

Keywords

Search

Navigation