Skip to main content

Advertisement

SpringerLink
Log in

Multimodal deep learning model on interim [18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma

  • Imaging Informatics and Artificial Intelligence
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim 18F-fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL.

Methods

Initially, 205 DLBCL patients undergoing interim [18F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance.

Results

The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability.

Conclusions

The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment.

Key Points

• The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients.

• Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance.

• Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on reasonable request from the corresponding author.

Abbreviations

[18F]FDG:

18F-fluoro-2-deoxyglucose

AUC:

Area under the receiver operating characteristic curve

DLBCL:

Diffuse large B-cell lymphoma

ECOG:

Eastern Cooperative Oncology Group

IQR:

Interquartile range

LDH:

Lactate dehydrogenase

LSTM:

Long short-term memory

MDL:

Multimodal deep learning

PET/CT:

Positron emission tomography/computed tomography

PTF:

Primary treatment failure

R-CHOP:

Rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisone

R-IPI:

Revised international prognostic index

TMTV:

Total metabolic tumour volume

References

  1. Sung H, Ferlay J, Siegel RL et al (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 71(3):209–249

    Article  Google Scholar 

  2. Feugier P, Van Hoof A, Sebban C et al (2005) Long-term results of the R-CHOP study in the treatment of elderly patients with diffuse large B-cell lymphoma: a study by the Groupe d’Etude des Lymphomes de l’Adulte. J Clin Oncol 23:4117–4126

    Article  CAS  Google Scholar 

  3. Crump M, Neelapu SS, Farooq U et al (2017) Outcomes in refractory diffuse large B-cell lymphoma: results from the international SCHOLAR-1 study. Blood 130(16):1800–1808

    Article  CAS  Google Scholar 

  4. Gisselbrecht C, Neste EVD (2018) How I manage patients with relapsed/refractory diffuse large B cell lymphoma. Br J Haematol 182:633–643

    Article  Google Scholar 

  5. Locke FL, Ghobadi A, Jacobson CA et al (2019) Long-term safety and activity of axicabtagene ciloleucel in refractory large B-cell lymphoma (ZUMA-1): a single-arm, multicentre, phase 1-2 trial. Lancet Oncol 20(1):31–42

    Article  CAS  Google Scholar 

  6. Kalakonda N, Maerevoet M, Cavallo F et al (2020) Selinexor in patients with relapsed or refractory diffuse large B-cell lymphoma (SADAL): a single-arm, multinational, multicentre, open-label, phase 2 trial. Lancet Haematol 7(7):e511–e522

    Article  Google Scholar 

  7. Hawkes EA, Barraclough A, Sehn LH (2022) Limited-stage diffuse large B-cell lymphoma. Blood 139(6):822–834

    Article  CAS  Google Scholar 

  8. Lv X, Wang Q, Ge X, Xue C, Liu X (2021) Application of high-throughput gene sequencing in lymphoma. Exp Mol Pathol 119:104606

    Article  CAS  Google Scholar 

  9. Sehn LH, Berry B, Chhanabhai M et al (2007) The revised International Prognostic Index (R-IPI) is a better predictor of outcome than the standard IPI for patients with diffuse large B-cell lymphoma treated with R-CHOP. Blood 190(6):1857–1861

    Article  Google Scholar 

  10. Xu-Monette ZY, Wu L, Visco C et al (2012) Mutational profile and prognostic significance of TP53 in diffuse large B-cell lymphoma patients treated with R-CHOP: report from an international DLBCL rituximab-CHOP consortium program study. Blood 120(19):3986–3996

    Article  CAS  Google Scholar 

  11. Juweid ME, Stroobants S, Hoekstra OS et al (2007) Use of positron emission tomography for response assessment of lymphoma: consensus of the Imaging Subcommittee of International Harmonization Project in Lymphoma. J Clin Oncol 25(5):571–578

    Article  Google Scholar 

  12. Cheson BD, Pfistner B, Juweid ME et al (2007) Revised response criteria for malignant lymphoma. J Clin Oncol 25(5):579–586

    Article  Google Scholar 

  13. Cheson BD (2011) Role of functional imaging in the management of lymphoma. J Clin Oncol 29(14):1844–1854

    Article  Google Scholar 

  14. Jerusalem G, Beguin Y, Fassotte MF et al (2000) Persistent tumor [18F]FDG uptake after a few cycles of polychemotherapy is predictive of treatment failure in non-Hodgkin’s lymphoma. Haematologica 85(6):613–618

  15. Spaepen K, Stroobants S, Dupont P et al (2002) Early restaging positron emission tomography with (18)F-fluorodeoxyglucose predicts outcome in patients with aggressive non-Hodgkin's lymphoma. Ann Oncol 13(9):1356–1363

    Article  CAS  Google Scholar 

  16. Kirienko M, Biroli M, Gelardi F, Seregni E, Chiti A, Sollini M (2021) Deep learning in nuclear medicine—focus on CNN-based approaches for PET/CT and PET/MR: where do we stand? Clin Transl Imaging 9:37–55

    Article  Google Scholar 

  17. Roth HR, Lu L, Seff A et al (2014) A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Med Image Comput Comput Assist Interv 17:520–527

    Google Scholar 

  18. Wallis D, Soussan M, Lacroix M, Akl P, Duboucher C, Buvat I (2022) An [18F]FDG-PET/CT deep learning method for fully automated detection of pathological mediastinal lymph nodes in lung cancer patients. Eur J Nucl Med Mol Imaging 49:881–888

    Article  CAS  Google Scholar 

  19. Bi L, Kim J, Kumar A, Wen L, Feng D, Fulham M (2017) Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies. Comput Med Imaging Graph 60:3–10

    Article  Google Scholar 

  20. Hu H, Shen L, Zhou T, Decazes P, Vera P, Ruan S (2020) Lymphoma segmentation in PET images based on multi-view and Conv3D fusion strategy. IEEE 17th International Symposium on Biomedical Imaging (ISBI):1197-1200

  21. Revailler W, Cottereau AS, Rossi C et al (2022) Deep learning approach to automatize TMTV calculations regardless of segmentation methodology for major FDG-avid lymphomas. Diagnostics (Basel) 12(2):417

    Article  CAS  Google Scholar 

  22. Sadik M, Lind E, Polymeri E, Enqvist O, Ulén J, Trägårdh E (2019) Automated quantification of reference levels in liver and mediastinal blood pool for the Deauville therapy response classification using FDG-PET/CT in Hodgkin and non-Hodgkin lymphomas. Clin Physiol Funct Imaging 39(1):78–84

    Article  Google Scholar 

  23. Capobianco N, Meignan M, Cottereau AS et al (2021) Deep-learning [18F]FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma. J Nucl Med 62(1):30–36

  24. Seidler M, Forghani B, Reinhold C et al (2019) Dual-energy CT texture analysis with machine learning for the evaluation and characterization of cervical lymphadenopathy. Comput Struct Biotechnol J 17:1009–1015

    Article  Google Scholar 

  25. Ganeshan B, Miles KA, Babikir S et al (2017) CT-based texture analysis potentially provides prognostic information complementary to interim FDG-PET for patients with Hodgkin’s and aggressive non-Hodgkin’s lymphomas. Eur Radiol 27:1012–1020

    Article  CAS  Google Scholar 

  26. Santiago R, Jimenez JO, Forghani R et al (2021) CT-based radiomics model with machine learning for predicting primary treatment failure in diffuse large B-cell Lymphoma. Transl Oncol 14(10):101188

    Article  CAS  Google Scholar 

  27. Zhou T, Ruan S, Canu S et al (2019) A review: Deep learning for medical image segmentation using multi-modality fusion. Array 3:100004

    Article  Google Scholar 

  28. Li K, Zhang R, Cai W (2021) Deep learning convolutional neural network (DLCNN): unleashing the potential of 18 F-FDG PET/CT in lymphoma. Am J Nucl Med Mol Imaging 11(4):327–331

    Google Scholar 

  29. Jin C, Yu H, Ke J et al (2021) Predicting treatment response from longitudinal images using multi-task deep learning. Nat Commun 12:1851

    Article  CAS  Google Scholar 

  30. Kumar A, Fulham M, Feng D, Kim J (2020) Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans Med Imaging 39(1):204–217

    Article  Google Scholar 

  31. Donahue J, Hendricks LA, Rohrbach M et al (2017) Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans Pattern Anal Mach Intell 39(4):677–691

    Article  Google Scholar 

  32. Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. Preprint arXiv:2002.05709

  33. Zhong Z, Kim Y, Plichta K et al (2019) Simultaneous co-segmentation of tumors in PET-CT images using deep fully convolutional networks. Med Phys 46(2):619–633

    Article  Google Scholar 

  34. Zhao X, Li L, Lu W, Tan S (2018) Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys Med Biol 64(1):015011

    Article  Google Scholar 

  35. Humphries SM, Notary AM, Centeno JP et al (2019) Deep learning enables automatic classification of emphysema pattern at CT. Radiology 294(2):434–444

    Article  Google Scholar 

  36. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  CAS  Google Scholar 

  37. Lu N, Wu Y, Feng L, Song J (2019) Deep learning for fall detection: three-dimensional CNN combined with LSTM on video kinematic data. IEEE J Biomed Health Inform 23(1):314–323

    Article  Google Scholar 

  38. Abadi M, Barham P, Chen J, et al (2016) TensorFlow: a system for large-scale machine learning. Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation:265-283

  39. Chetlur A, Woolley C, Vandermersch P, et al (2014) cuDNN: efficient primitives for deep learning. Preprint arXiv:1410.0759

  40. Zhou B, Khosla A, Lapedriza A, Oliva A, Torrralba A (2016) Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR):2921-2929

  41. Du D, Feng H, Lv W et al (2020) Machine learning methods for optimal radiomics-based differentiation between recurrence and inflammation: application to nasopharyngeal carcinoma post-therapy PET/CT images. Mol Imaging Biol 22:730–738

    Article  CAS  Google Scholar 

  42. Yuan C, Zhang M, Huang X et al (2021) Diffuse large B-cell lymphoma segmentation in PET-CT images via hybrid learning for feature fusion. Med Phys 48(7):3665–3678

    Article  Google Scholar 

  43. Peng Y, Bi L, Guo Y, Feng D, Fulham M, Kim J (2019) Deep multi-modality collaborative learning for distant metastases predication in PET-CT soft-tissue sarcoma studies. Annu Int Conf IEEE Eng Med Biol Soc:3658-3688

  44. Zhang W, Li R, Deng H et al (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. Neuroimage 108:214–224

  45. Zhong Z, Kim Y, Zhou L, et al (2018) 3D fully convolutional networks for co-segmentation of tumors on PET-CT images. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI):228-231

Download references

Code availability

The code of our study is publicly accessible at https://github.com/cyuan-sjtu/MDL-model.

Funding

This study has received funding by the National Natural Science Foundation of China (81974276, 81830007, 81520108003, 81670176, and 82070204), Chang Jiang Scholars Program, Shanghai Municipal Education Commission Gaofeng Clinical Medicine Grant Support (20152206, and 20152208), Clinical Research Plan of Shanghai Hospital Development Center (SHDC2020CR1032B), Multicenter Clinical Research Project by Shanghai Jiao Tong University School of Medicine (DLY201601), Collaborative Innovation Center of Systems Biomedicine, and the Samuel Waxman Cancer Research Foundation.

Author information

Author notes

  1. Cheng Yuan, Qing Shi, and Xinyun Huang contributed equally to this work.

Authors and Affiliations

  1. School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200040, China

    Cheng Yuan & Dahong Qian

  2. Shanghai Institute of Hematology, State Key Laboratory of Medical Genomics, National Research Center for Translational Medicine at Shanghai, Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China

    Qing Shi, Li Wang, Yang He & Weili Zhao

  3. Department of Nuclear Medicine, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China

    Xinyun Huang & Biao Li

Authors
  1. Cheng Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yang He
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Biao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Weili Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dahong Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Biao Li, Weili Zhao or Dahong Qian.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Dr. Dahong Qian.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was not required for this study because of its retrospective nature.

Ethics approval

Institutional Review Board approval was obtained at Shanghai Ruijin Hospital.

Methodology

• Retrospective

• Diagnostic or prognostic study

• Performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 94 kb)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, C., Shi, Q., Huang, X. et al. Multimodal deep learning model on interim [18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma. Eur Radiol 33, 77–88 (2023). https://doi.org/10.1007/s00330-022-09031-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-09031-8

Keywords

Search

Navigation