Abstract
Introduction
In recent years, machine learning algorithms have led to innovative tools for medical imaging analysis. The purpose of the present review was to summarize the literature on the developing field of deep learning (DL), particularly the application of convolutional neural networks (CNNs) in PET/CT and PET/MR.
Methods
We performed the literature search, referring to “convolutional neural networks” and “positron emission tomography” on PubMed/MEDLINE, for potentially relevant articles published up until July 24th, 2020.
Results
After the screening process, 63 articles were finally included; these embraced both the technical (n = 23) and the clinical field (n = 40). Technical studies aimed at investigating the role of CNN-based methods for image quality improvement (n = 11) and on technical issues (n = 12), mainly attenuation correction. Clinical studies explored CNN applications in oncology lung cancer (n = 7), head and neck cancer (n = 4), esophageal cancer (n = 2), lymphoma (n = 3), prostate cancer (N = 4), cervical cancer (n = 1), sarcomas (n = 1), multiple cancer types (n = 4), in neurology (n = 10) and cardiology (n = 1); three additional studies belonged to “other” category. In oncology, the studies aimed at detection, diagnosis, and prognostication of cancer. In neurology, the majority of the studies aimed at diagnosing Alzheimer Disease and stratification of the risk. CNN-based algorithms demonstrated promising results with performances equal or even higher compared to conventional approaches.
Discussion
Overall, CNN applications for PET/CT and PET/MR are exponentially growing, demonstrating encouraging results for both technical and clinical purposes. Novel research strategies emerged to face the challenges of DL algorithms development. Education and confidence with DL-based tools are needed for proper technology implementation.
Similar content being viewed by others
Data availability
Not applicable.
Code availability
Not applicable.
References
Erickson BJ (2019) Deep learning and machine learning in imaging: basic principles. Artificial intelligence in medical imaging. Springer International Publishing, Cham, pp 39–46
Benjamens S, Dhunnoo P, Meskó B (2020) The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. npj Digit Med 3:118
Zhou L, Schaefferkoetter JD, Tham IWK, Huang G, Yan J (2020) Supervised learning with cyclegan for low-dose FDG PET image denoising. Med Image Anal 65:101770
Xiang L, Qiao Y, Nie D, An L, Lin W, Wang Q et al (2017) Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing 267:406–416
Spuhler K, Serrano-Sosa M, Cattell R, DeLorenzo C, Huang C (2020) Full-count PET recovery from low-count image using a dilated convolutional neural network. Med Phys 47:4928–4938
Song TA, Chowdhury SR, Yang F, Dutta J (2020) PET image super-resolution using generative adversarial networks. Neural Netw 125:83–91
Song T-A, Chowdhury SR, Yang F, Dutta J (2020) Super-resolution PET imaging using convolutional neural networks. IEEE Trans Comput Imaging 6:518–528
Whiteley W, Gregor J (2019) CNN-based PET sinogram repair to mitigate defective block detectors. Phys Med Biol 64:235017
Hong X, Zan Y, Weng F, Tao W, Peng Q, Huang Q (2018) Enhancing the image quality via transferred deep residual learning of coarse PET sinograms. IEEE Trans Med Imaging IEEE 37:2322–2332
Zatcepin A, Pizzichemi M, Polesel A, Paganoni M, Auffray E, Ziegler SI et al (2020) Improving depth-of-interaction resolution in pixellated PET detectors using neural networks. Phys Med Biol 65:175017
Liu C-C, Huang H-M (2019) Partial-ring PET image restoration using a deep learning based method. Phys Med Biol 64:225014
Kim K, Wu D, Gong K, Dutta J, Kim JH, Son YD et al (2018) Penalized PET reconstruction using deep learning prior and local linear fitting. IEEE Trans Med Imaging 37:1478–1487
Gong K, Guan J, Kim K, Zhang X, Yang J, Seo Y et al (2019) Iterative PET image reconstruction using convolutional neural network representation. IEEE Trans Med Imaging 38:675–685
Blanc-Durand P, Khalife M, Sgard B, Kaushik S, Soret M, Tiss A et al (2019) Attenuation correction using 3D deep convolutional neural network for brain 18FFDG PET/MR: Comparison with Atlas, ZTE and CT based attenuation correction. PLoS ONE 14:1–12
Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y et al (2018) Zero-echo-time and dixon deep pseudo-CT (ZeDD CT): Direct generation of pseudo-CT images for Pelvic PET/MRI attenuation correction using deep convolutional neural networks with multiparametric MRI. J Nucl Med 59:852–858
Bradshaw TJ, Zhao G, Jang H, Liu F, McMillan AB (2018) Feasibility of deep learning-based PET/MR attenuation correction in the pelvis using only diagnostic MR images. Tomogr (Ann Arbor, Mich) 4:138–147
Hwang D, Kim KY, Kang SK, Seo S, Paeng JC, Lee DS et al (2018) Improving the accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. J Nucl Med 59:1624–1629
Hwang D, Kang SK, Kim KY, Seo S, Paeng JC, Lee DS et al (2019) Generation of PET attenuation map for whole-body time-of-flight 18F-FDG PET/MRI using a deep neural network trained with simultaneously reconstructed activity and attenuation maps. J Nucl Med 60:1183–1189
Arabi H, Bortolin K, Ginovart N, Garibotto V, Zaidi H (2020) Deep learning-guided joint attenuation and scatter correction in multitracer neuroimaging studies. Hum Brain Mapp. https://doi.org/10.1002/hbm.25039
Spuhler KD, Gardus J, Gao Y, DeLorenzo C, Parsey R, Huang C (2019) Synthesis of patient-specific transmission data for PET attenuation correction for PET/MRI neuroimaging using a convolutional neural network. J Nucl Med 60:555–560
Berg E, Cherry SR (2018) Using convolutional neural networks to estimate time-of-flight from PET detector waveforms. Phys Med Biol 63:1–15
Xu J, Liu H (2019) Three-dimensional convolutional neural networks for simultaneous dual-tracer PET imaging. Phys Med Biol 64:185016
Kumar A, Fulham M, Feng D, Kim J (2020) Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans Med Imaging IEEE 39:204–217
Lindgren Belal S, Sadik M, Kaboteh R, Enqvist O, Ulén J, Poulsen MH et al (2019) Deep learning for segmentation of 49 selected bones in CT scans: First step in automated PET/CT-based 3D quantification of skeletal metastases. Eur J Radiol 113:89–95
Lee MS, Hwang D, Kim JH, Lee JS (2019) Deep-dose: a voxel dose estimation method using deep convolutional neural network for personalized internal dosimetry. Sci Rep 9:10308
Blanc-Durand P, Van Der Gucht A, Schaefer N, Itti E, Prior JO (2018) Automatic lesion detection and segmentation of 18F-FET PET in gliomas: a full 3D U-Net convolutional neural network study. PLoS ONE 13:0195798
Huang B, Chen Z, Wu P-M, Ye Y, Feng S-T, Wong C-YO et al (2018) Fully automated delineation of gross tumor volume for head and neck cancer on PET-CT using deep learning: a dual-center study. Contrast Media Mol Imaging 2018:8923028
Olin AB, Hansen AE, Rasmussen JH, Ladefoged CN, Berthelsen AK, Håkansson K et al (2020) Feasibility of multiparametric positron emission tomography/magnetic resonance imaging as a one-stop shop for radiation therapy planning for patients with head and neck cancer. Int J Radiat Oncol 108:1329–1338
Chen L, Zhou Z, Sher D, Zhang Q, Shah J, Pham N-L et al (2019) Combining many-objective radiomics and 3D convolutional neural network through evidential reasoning to predict lymph node metastasis in head and neck cancer. Phys Med Biol 64:075011
Capobianco N, Meignan MA, Cottereau A-S, Vercellino L, Sibille L, Spottiswoode B et al (2020) Deep learning FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma. J Nucl Med 62:30–36
Sadik M, Lind E, Polymeri E, Enqvist O, Ulén J, Trägårdh E (2019) Automated quantification of reference levels in liver and mediastinal blood pool for the Deauville therapy response classification using FDG-PET/CT in Hodgkin and non-Hodgkin lymphomas. Clin Physiol Funct Imaging 39:78–84
Bi L, Kim J, Kumar A, Wen L, Feng D, Fulham M (2017) Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies. Comput Med Imaging Graph 60:3–10
Teramoto A, Fujita H, Yamamuro O, Tamaki T (2016) Automated detection of pulmonary nodules in PET/CT images: Ensemble false-positive reduction using a convolutional neural network technique. Med Phys 43:2821–2827
Zhang R, Cheng C, Zhao X, Li X (2019) Multiscale mask R-CNN–based lung tumor detection using PET imaging. Mol Imaging 18:153601211986353
Zhao X, Li L, Lu W, Tan S (2018) Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys Med Biol 64:015011
Kirienko M, Sollini M, Silvestri G, Mognetti S, Voulaz E, Antunovic L et al (2018) Convolutional neural networks promising in lung cancer T-parameter assessment on baseline FDG-PET/CT. Contrast Media Mol Imaging 2018:1382309
Wang H, Zhou Z, Li Y, Chen Z, Lu P, Wang W et al (2017) comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images. EJNMMI Res 7:11
Tau N, Stundzia A, Yasufuku K, Hussey D, Metser U (2020) Convolutional neural networks in predicting nodal and distant metastatic potential of newly diagnosed non-small cell lung cancer on FDG PET images. Am J Roentgenol 215:192–197
Baek S, He Y, Allen BG, Buatti JM, Smith BJ, Tong L et al (2019) Deep segmentation networks predict survival of non-small cell lung cancer. Sci Rep 9:17286
Ypsilantis P-P, Siddique M, Sohn H-M, Davies A, Cook G, Goh V et al (2015) Predicting response to neoadjuvant chemotherapy with PET imaging using convolutional neural networks. PLoS ONE 10:e0137036
Yang C-K, Yeh JC-Y, Yu W-H, Chien L-I, Lin K-H, Huang W-S et al (2019) Deep convolutional neural network-based positron emission tomography analysis predicts esophageal cancer outcome. J Clin Med 8:844
Polymeri E, Sadik M, Kaboteh R, Borrelli P, Enqvist O, Ulén J et al (2020) Deep learning-based quantification of PET/CT prostate gland uptake: association with overall survival. Clin Physiol Funct Imaging 40:106–113
Mortensen MA, Borrelli P, Poulsen MH, Gerke O, Enqvist O, Ulén J et al (2019) Artificial intelligence-based versus manual assessment of prostate cancer in the prostate gland: a method comparison study. Clin Physiol Funct Imaging 39:399–406
Hartenstein A, Lübbe F, Baur ADJ, Rudolph MM, Furth C, Brenner W et al (2020) Prostate cancer nodal staging: using deep learning to predict 68Ga-PSMA-positivity from CT imaging alone. Sci Rep 10:3398
Lee JJ, Yang H, Franc BL, Iagaru A, Davidzon GA (2020) Deep learning detection of prostate cancer recurrence with 18F-FACBC (fluciclovine, Axumin®) positron emission tomography. Eur J Nucl Med Mol Imaging. https://doi.org/10.1007/s00259-020-04912-w
Nobashi T, Zacharias C, Ellis JK, Ferri V, Koran ME, Franc BL et al (2020) Performance comparison of individual and ensemble CNN models for the classification of brain 18F-FDG-PET scans. J Digit Imaging 33:447–455
Shaish H, Mutasa S, Makkar J, Chang P, Schwartz L, Ahmed F (2019) Prediction of lymph node maximum standardized uptake value in patients with cancer using a 3D convolutional neural network: a proof-of-concept study. Am J Roentgenol 212:238–244
Sibille L, Seifert R, Avramovic N, Vehren T, Spottiswoode B, Zuehlsdorff S et al (2020) 18F-FDG PET/CT uptake classification in lymphoma and lung cancer by using deep convolutional neural networks. Radiology 294:445–452
Kawauchi K, Furuya S, Hirata K, Katoh C, Manabe O, Kobayashi K et al (2020) A convolutional neural network-based system to classify patients using FDG PET/CT examinations. BMC Cancer 20:227
Chen L, Shen C, Zhou Z, Maquilan G, Albuquerque K, Folkert MR et al (2019) Automatic PET cervical tumor segmentation by combining deep learning and anatomic prior. Phys Med Biol 64:085019
Peng Y, Bi L, Guo Y, Feng D, Fulham M, Kim J (2019) Deep multi-modality collaborative learning for distant metastases predication in PET-CT soft-tissue sarcoma studies. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBS. IEEE, pp 3658–3661
Ding Y, Sohn JH, Kawczynski MG, Trivedi H, Harnish R, Jenkins NW et al (2019) A deep learning model to predict a diagnosis of Alzheimer disease by using 18 F-FDG PET of the brain. Radiology 290:456–464
Liu M, Cheng D, Wang K, Wang Y (2018) Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 16:295–308
Liu M, Cheng D, Yan W (2018) Classification of Alzheimer’s disease by combination of convolutional and recurrent neural networks using FDG-PET images. Front Neuroinform 12:1–12
Huang Y, Xu J, Zhou Y, Tong T, Zhuang X (2019) Diagnosis of Alzheimer’s disease via multi-modality 3D convolutional neural network. Front Neurosci 13:1–12
Kim J-Y, Suh HY, Ryoo HG, Oh D, Choi H, Paeng JC et al (2010) Amyloid PET Quantification Via End-to-End Training of a Deep Learning. Nucl Med Mol Imaging 2019(53):340–348
Choi H, Jin KH (2018) Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. Behav Brain Res 344:103–109
Yee E, Popuri K, Beg MF (2020) Quantifying brain metabolism from FDG-PET images into a probability of Alzheimer’s dementia score. Hum Brain Mapp 41:5–16
Choi H, Kim YK, Yoon EJ, Lee JY, Lee DS (2020) Cognitive signature of brain FDG PET based on deep learning: domain transfer from Alzheimer’s disease to Parkinson’s disease. Eur J Nucl Med Mol Imaging 47:403–412
Zhao Y, Wu P, Wang J, Li H, Navab N, Yakushev I, et al (2019) A 3D Deep Residual Convolutional Neural Network for Differential Diagnosis of Parkinsonian Syndromes on 18F-FDG PET Images. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBS. IEEE, pp 3531–3534
Manzanera OM, Meles SK, Leenders KL, Renken RJ, Pagani M, Arnaldi D et al (2019) Scaled subprofile modeling and convolutional neural networks for the identification of Parkinson’s disease in 3D nuclear imaging data. Int J Neural Syst 29:1950010
Togo R, Hirata K, Manabe O, Ohira H, Tsujino I, Magota K et al (2019) Cardiac sarcoidosis classification with deep convolutional neural network-based features using polar maps. Comput Biol Med 104:81–86
Guo J, Gong E, Fan AP, Goubran M, Khalighi MM, Zaharchuk G (2019) Predicting 15 O-Water PET cerebral blood flow maps from multi-contrast MRI using a deep convolutional neural network with evaluation of training cohort bias. J Cereb Blood Flow Metab 40:2240–2253
Xiong X, Linhardt TJ, Liu W, Smith BJ, Sun W, Bauer C et al (2020) A 3D deep convolutional neural network approach for the automated measurement of cerebellum tracer uptake in FDG PET-CT scans. Med Phys 47:1058–1066
Kawauchi K, Hirata K, Katoh C, Ichikawa S, Manabe O, Kobayashi K et al (2019) A convolutional neural network-based system to prevent patient misidentification in FDG-PET examinations. Sci Rep 9:7192
van Dyk DA, Meng X-L (2001) The art of data augmentation. J Comput Graph Stat 10:1–50
Kermany DDS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(1122–1131):e1129
Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P et al (2013) The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging J Digit Imaging 26:1045–1057
Abadi E, Segars WP, Tsui BMW, Kinahan PE, Bottenus N, Frangi AF et al (2020) Virtual clinical trials in medical imaging: a review. J Med Imaging 7:42805
Wasserman P, Freels P, Szames D, Kurra C, Hernandez M (2020) The technophysics year: transformation of diagnostic radiology’s clinical year as a matter of necessity. Acad Radiol. https://doi.org/10.1016/j.acra.2020.04.045
Joshi AV (2020) Machine Learning and Artificial Intelligence. Springer International Publishing, Cham
Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ et al (2017) Deep learning: a primer for radiologists. RadioGraphics 37:2113–2131
Nensa F, Demircioglu A, Rischpler C (2019) Artificial intelligence in nuclear medicine. J Nucl Med 60:29S-37S
Currie G, Rohren E (2020) Intelligent imaging in nuclear medicine: the principles of artificial intelligence, machine learning and deep learning. Semin Nucl Med. https://doi.org/10.1053/j.semnuclmed.2020.08.002
(2020) European Association of Nuclear Medicine October 22 – 30, 2020 Virtual. Eur J Nucl Med Mol Imaging 47, 1–753. https://doi.org/10.1007/s00259-020-04988-4
AI resources and training (2020). https://www.rsna.org/en/education/ai-resources-and-training. Accessed 25 Sep 2020
Funding
The present publication was not supported by any funding from public or private sources.
Author information
Authors and Affiliations
Contributions
MK: Literature search and review, manuscript drafting. MS and MK: Content planning and critical data assessment. MB: Manuscript drafting. FG: Manuscript editing and figures’ preparation. ES: Manuscript critical revision and editing. AC: Manuscript critical revision and editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflict of interest with the present work.
Human and animal rights
This article does not contain any studies with human or animal subjects performed by any authors since the present paper deals with a literature review.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Kirienko, M., Biroli, M., Gelardi, F. et al. Deep learning in Nuclear Medicine—focus on CNN-based approaches for PET/CT and PET/MR: where do we stand?. Clin Transl Imaging 9, 37–55 (2021). https://doi.org/10.1007/s40336-021-00411-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40336-021-00411-6