Abstract
We propose a novel method for the prediction of patient prognosis with Head and Neck cancer (H&N) from FDG-PET/CT images. In particular, we aim at automatically predicting Disease-Free Survival (DFS) for patients treated with radiotherapy or both radiotherapy and chemotherapy. We design a multi-task deep UNet to learn both the segmentation of the primary Gross Tumor Volume (GTVt) and the outcome of the patient from PET and CT images. The motivation for this approach lies in the complementarity of the two tasks and the shared visual features relevant to both tasks. A multi-modal (PET and CT) 3D UNet is trained with a combination of survival and Dice losses to jointly learn the two tasks. The model is evaluated on the HECKTOR 2020 dataset consisting of 239 H&N patients with PET, CT, GTVt contours and DFS data (five centers). The results are compared with a standard Cox PET/CT radiomics model. The proposed multi-task CNN reaches a C-index of 0.723, outperforming both the deep radiomics model without segmentation (C-index of 0.650) and the standard radiomics model (C-index of 0.695). Besides the improved performance in outcome prediction, the main advantage of the proposed multi-task approach is that it can predict patient prognosis without a manual delineation of the GTVt, a tedious and time-consuming process that hinders the validation of large-scale radiomics studies. The code will be shared for reproducibility on our GitHub repository.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
The main difference is the reduced number of filters to be able to use a larger batch-size for the survival loss in Sect. 2.2.
- 3.
24 GLCM, 16 GLRLM, and 16 GLDZM features. A Fixed Bin Number (FBN) of 64 and a Fixed Bin Size (FBS) of 50 are used for CT. A FBN of 8 and a FBS of 1 are used for PET.
References
Gillies, R.J., Kinahan, P.E., Hricak, H.: Radiomics: images are more than pictures, they are data. Radiology 278(2), 563–577 (2016)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Andrearczyk, V., et al.: Overview of the HECKTOR challenge at MICCAI 2020: automatic head and neck tumor segmentation in PET/CT. In: Andrearczyk, V., Oreiller, V., Depeursinge, A. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 1–21. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67194-5_1
Faraggi, D., Simon, R.: A neural network model for survival data. Stat. Med. 14(1), 73–82 (1995)
Ranganath, R., Perotte, A., Elhadad, N., Blei, D.: Deep survival analysis. In: Machine Learning for Healthcare Conference, pp. 101–114. PMLR (2016)
Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y.: DeepSurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Med. Res. Methodol. 18(1), 1–12 (2018)
Steingrimsson, J.A., Morrison, S.: Deep learning for survival outcomes. Stat. Med. 39(17), 2339–2349 (2020)
Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T.: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II, vol. 11384. Springer, Heidelberg (2019). https://doi.org/10.1007/978-3-030-11726-9
Baek, S., et al.: Deep segmentation networks predict survival of non-small cell lung cancer. Sci. Rep. 9(1), 1–10 (2019)
Parekh, V.S., Jacobs, M.A.: Deep learning and radiomics in precision medicine. Expert Rev. Precis. Med. Drug Dev. 4(2), 59–72 (2019)
Diamant, A., Chatterjee, A., Vallières, M., Shenouda, G., Seuntjens, J.: Deep learning in head & neck cancer outcome prediction. Sci. Rep. 9(1), 1–10 (2019)
Zhang, Y., Lobo-Mueller, E.M., Karanicolas, P., Gallinger, S., Haider, M.A., Khalvati, F.: CNN-based survival model for pancreatic ductal adenocarcinoma in medical imaging. BMC Med. Imaging 20(1), 1–8 (2020)
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. 115(13), E2970–E2979 (2018)
Li, H., et al.: Deep convolutional neural networks for imaging data based survival analysis of rectal cancer. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 846–849. IEEE (2019)
Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)
Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., Savarese, S.: Which tasks should be learned together in multi-task learning? In: International Conference on Machine Learning, pp. 9120–9132. PMLR (2020)
Mlynarski, P., Delingette, H., Criminisi, A., Ayache, N.: Deep learning with mixed supervision for brain tumor segmentation. J. Med. Imaging 6(3), 034002 (2019)
Weninger, L., Liu, Q., Merhof, D.: Multi-task learning for brain tumor segmentation. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11992, pp. 327–337. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46640-4_31
Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: classification and segmentation
Graziani, M., Otálora, S., Muller, H., Andrearczyk, V.: Guiding CNNs towards relevant concepts by multi-task and adversarial learning. arXiv preprint arXiv:2008.01478 (2020)
Iantsen, A., Visvikis, D., Hatt, M.: Squeeze-and-excitation normalization for automated delineation of head and neck primary tumors in combined PET and CT images. In: Andrearczyk, V., Oreiller, V., Depeursinge, A. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 37–43. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67194-5_4
Van Griethuysen, J.J.M., et al.: Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77(21), e104–e107 (2017)
Suter, Y., et al.: Radiomics for glioblastoma survival analysis in pre-operative MRI: exploring feature robustness, class boundaries, and machine learning techniques. Cancer Imaging 20(1), 1–13 (2020)
Lambin, P., et al.: Radiomics: the bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol. 14(12), 749–762 (2017)
David, C.R., et al.: Regression models and life tables (with discussion). J. Roy. Stat. Soc. 34(2), 187–220 (1972)
Harrell, F.E., Lee, K.L., Mark, D.B.: Tutorial in biostatistics multivariable prognostic models. Stat. Med. 15, 361–387 (1996)
Andrearczyk, V., Oreiller, V., Depeursinge, A.: Oropharynx detection in PET-CT for tumor segmentation. In: Irish Machine Vision and Image Processing (2020)
Vallieres, M., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 7(1), 1–14 (2017)
Chennupati, S., Sistu, G., Yogamani, S., Rawashdeh, S.A.: MultiNet++: multi-stream feature aggregation and geometric loss strategy for multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491 (2018)
Andrearczyk, V., Depeursinge, A., Müller, H.: Neural network training for cross-protocol radiomic feature standardization in computed tomography. J. Med. Imaging 6(3), 024008 (2019)
Acknowledgements
This work was partially supported by the Swiss National Science Foundation (SNSF, grant 205320_179069) and the Swiss Personalized Health Network (SPHN via the IMAGINE and QA4IQI projects).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Andrearczyk, V. et al. (2021). Multi-task Deep Segmentation and Radiomics for Automatic Prognosis in Head and Neck Cancer. In: Rekik, I., Adeli, E., Park, S.H., Schnabel, J. (eds) Predictive Intelligence in Medicine. PRIME 2021. Lecture Notes in Computer Science(), vol 12928. Springer, Cham. https://doi.org/10.1007/978-3-030-87602-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-87602-9_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87601-2
Online ISBN: 978-3-030-87602-9
eBook Packages: Computer ScienceComputer Science (R0)
