Skip to main content

Advertisement

Log in

Deep learning in Nuclear Medicine—focus on CNN-based approaches for PET/CT and PET/MR: where do we stand?

  • Expert Review
  • Published:
Clinical and Translational Imaging Aims and scope Submit manuscript

Abstract

Introduction

In recent years, machine learning algorithms have led to innovative tools for medical imaging analysis. The purpose of the present review was to summarize the literature on the developing field of deep learning (DL), particularly the application of convolutional neural networks (CNNs) in PET/CT and PET/MR.

Methods

We performed the literature search, referring to “convolutional neural networks” and “positron emission tomography” on PubMed/MEDLINE, for potentially relevant articles published up until July 24th, 2020.

Results

After the screening process, 63 articles were finally included; these embraced both the technical (n = 23) and the clinical field (n = 40). Technical studies aimed at investigating the role of CNN-based methods for image quality improvement (n = 11) and on technical issues (n = 12), mainly attenuation correction. Clinical studies explored CNN applications in oncology lung cancer (n = 7), head and neck cancer (n = 4), esophageal cancer (n = 2), lymphoma (n = 3), prostate cancer (N = 4), cervical cancer (n = 1), sarcomas (n = 1), multiple cancer types (n = 4), in neurology (n = 10) and cardiology (n = 1); three additional studies belonged to “other” category. In oncology, the studies aimed at detection, diagnosis, and prognostication of cancer. In neurology, the majority of the studies aimed at diagnosing Alzheimer Disease and stratification of the risk. CNN-based algorithms demonstrated promising results with performances equal or even higher compared to conventional approaches.

Discussion

Overall, CNN applications for PET/CT and PET/MR are exponentially growing, demonstrating encouraging results for both technical and clinical purposes. Novel research strategies emerged to face the challenges of DL algorithms development. Education and confidence with DL-based tools are needed for proper technology implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Erickson BJ (2019) Deep learning and machine learning in imaging: basic principles. Artificial intelligence in medical imaging. Springer International Publishing, Cham, pp 39–46

    Chapter  Google Scholar 

  2. Benjamens S, Dhunnoo P, Meskó B (2020) The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. npj Digit Med 3:118

    Article  PubMed  PubMed Central  Google Scholar 

  3. Zhou L, Schaefferkoetter JD, Tham IWK, Huang G, Yan J (2020) Supervised learning with cyclegan for low-dose FDG PET image denoising. Med Image Anal 65:101770

    Article  PubMed  Google Scholar 

  4. Xiang L, Qiao Y, Nie D, An L, Lin W, Wang Q et al (2017) Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing 267:406–416

    Article  PubMed  PubMed Central  Google Scholar 

  5. Spuhler K, Serrano-Sosa M, Cattell R, DeLorenzo C, Huang C (2020) Full-count PET recovery from low-count image using a dilated convolutional neural network. Med Phys 47:4928–4938

    Article  PubMed  Google Scholar 

  6. Song TA, Chowdhury SR, Yang F, Dutta J (2020) PET image super-resolution using generative adversarial networks. Neural Netw 125:83–91

    Article  PubMed  PubMed Central  Google Scholar 

  7. Song T-A, Chowdhury SR, Yang F, Dutta J (2020) Super-resolution PET imaging using convolutional neural networks. IEEE Trans Comput Imaging 6:518–528

    Article  PubMed  PubMed Central  Google Scholar 

  8. Whiteley W, Gregor J (2019) CNN-based PET sinogram repair to mitigate defective block detectors. Phys Med Biol 64:235017

    Article  PubMed  Google Scholar 

  9. Hong X, Zan Y, Weng F, Tao W, Peng Q, Huang Q (2018) Enhancing the image quality via transferred deep residual learning of coarse PET sinograms. IEEE Trans Med Imaging IEEE 37:2322–2332

    Article  Google Scholar 

  10. Zatcepin A, Pizzichemi M, Polesel A, Paganoni M, Auffray E, Ziegler SI et al (2020) Improving depth-of-interaction resolution in pixellated PET detectors using neural networks. Phys Med Biol 65:175017

    Article  CAS  PubMed  Google Scholar 

  11. Liu C-C, Huang H-M (2019) Partial-ring PET image restoration using a deep learning based method. Phys Med Biol 64:225014

    Article  CAS  PubMed  Google Scholar 

  12. Kim K, Wu D, Gong K, Dutta J, Kim JH, Son YD et al (2018) Penalized PET reconstruction using deep learning prior and local linear fitting. IEEE Trans Med Imaging 37:1478–1487

    Article  PubMed  PubMed Central  Google Scholar 

  13. Gong K, Guan J, Kim K, Zhang X, Yang J, Seo Y et al (2019) Iterative PET image reconstruction using convolutional neural network representation. IEEE Trans Med Imaging 38:675–685

    Article  Google Scholar 

  14. Blanc-Durand P, Khalife M, Sgard B, Kaushik S, Soret M, Tiss A et al (2019) Attenuation correction using 3D deep convolutional neural network for brain 18FFDG PET/MR: Comparison with Atlas, ZTE and CT based attenuation correction. PLoS ONE 14:1–12

    Article  CAS  Google Scholar 

  15. Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y et al (2018) Zero-echo-time and dixon deep pseudo-CT (ZeDD CT): Direct generation of pseudo-CT images for Pelvic PET/MRI attenuation correction using deep convolutional neural networks with multiparametric MRI. J Nucl Med 59:852–858

    Article  PubMed  PubMed Central  Google Scholar 

  16. Bradshaw TJ, Zhao G, Jang H, Liu F, McMillan AB (2018) Feasibility of deep learning-based PET/MR attenuation correction in the pelvis using only diagnostic MR images. Tomogr (Ann Arbor, Mich) 4:138–147

    Article  Google Scholar 

  17. Hwang D, Kim KY, Kang SK, Seo S, Paeng JC, Lee DS et al (2018) Improving the accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. J Nucl Med 59:1624–1629

    Article  CAS  PubMed  Google Scholar 

  18. Hwang D, Kang SK, Kim KY, Seo S, Paeng JC, Lee DS et al (2019) Generation of PET attenuation map for whole-body time-of-flight 18F-FDG PET/MRI using a deep neural network trained with simultaneously reconstructed activity and attenuation maps. J Nucl Med 60:1183–1189

    Article  PubMed  PubMed Central  Google Scholar 

  19. Arabi H, Bortolin K, Ginovart N, Garibotto V, Zaidi H (2020) Deep learning-guided joint attenuation and scatter correction in multitracer neuroimaging studies. Hum Brain Mapp. https://doi.org/10.1002/hbm.25039

    Article  PubMed  PubMed Central  Google Scholar 

  20. Spuhler KD, Gardus J, Gao Y, DeLorenzo C, Parsey R, Huang C (2019) Synthesis of patient-specific transmission data for PET attenuation correction for PET/MRI neuroimaging using a convolutional neural network. J Nucl Med 60:555–560

    Article  CAS  PubMed  Google Scholar 

  21. Berg E, Cherry SR (2018) Using convolutional neural networks to estimate time-of-flight from PET detector waveforms. Phys Med Biol 63:1–15

    Article  CAS  Google Scholar 

  22. Xu J, Liu H (2019) Three-dimensional convolutional neural networks for simultaneous dual-tracer PET imaging. Phys Med Biol 64:185016

    Article  CAS  PubMed  Google Scholar 

  23. Kumar A, Fulham M, Feng D, Kim J (2020) Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans Med Imaging IEEE 39:204–217

    Article  Google Scholar 

  24. Lindgren Belal S, Sadik M, Kaboteh R, Enqvist O, Ulén J, Poulsen MH et al (2019) Deep learning for segmentation of 49 selected bones in CT scans: First step in automated PET/CT-based 3D quantification of skeletal metastases. Eur J Radiol 113:89–95

    Article  PubMed  Google Scholar 

  25. Lee MS, Hwang D, Kim JH, Lee JS (2019) Deep-dose: a voxel dose estimation method using deep convolutional neural network for personalized internal dosimetry. Sci Rep 9:10308

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  26. Blanc-Durand P, Van Der Gucht A, Schaefer N, Itti E, Prior JO (2018) Automatic lesion detection and segmentation of 18F-FET PET in gliomas: a full 3D U-Net convolutional neural network study. PLoS ONE 13:0195798

    Article  Google Scholar 

  27. Huang B, Chen Z, Wu P-M, Ye Y, Feng S-T, Wong C-YO et al (2018) Fully automated delineation of gross tumor volume for head and neck cancer on PET-CT using deep learning: a dual-center study. Contrast Media Mol Imaging 2018:8923028

    Article  PubMed  PubMed Central  Google Scholar 

  28. Olin AB, Hansen AE, Rasmussen JH, Ladefoged CN, Berthelsen AK, Håkansson K et al (2020) Feasibility of multiparametric positron emission tomography/magnetic resonance imaging as a one-stop shop for radiation therapy planning for patients with head and neck cancer. Int J Radiat Oncol 108:1329–1338

    Article  Google Scholar 

  29. Chen L, Zhou Z, Sher D, Zhang Q, Shah J, Pham N-L et al (2019) Combining many-objective radiomics and 3D convolutional neural network through evidential reasoning to predict lymph node metastasis in head and neck cancer. Phys Med Biol 64:075011

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Capobianco N, Meignan MA, Cottereau A-S, Vercellino L, Sibille L, Spottiswoode B et al (2020) Deep learning FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma. J Nucl Med 62:30–36

    Article  PubMed  Google Scholar 

  31. Sadik M, Lind E, Polymeri E, Enqvist O, Ulén J, Trägårdh E (2019) Automated quantification of reference levels in liver and mediastinal blood pool for the Deauville therapy response classification using FDG-PET/CT in Hodgkin and non-Hodgkin lymphomas. Clin Physiol Funct Imaging 39:78–84

    Article  PubMed  Google Scholar 

  32. Bi L, Kim J, Kumar A, Wen L, Feng D, Fulham M (2017) Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies. Comput Med Imaging Graph 60:3–10

    Article  PubMed  Google Scholar 

  33. Teramoto A, Fujita H, Yamamuro O, Tamaki T (2016) Automated detection of pulmonary nodules in PET/CT images: Ensemble false-positive reduction using a convolutional neural network technique. Med Phys 43:2821–2827

    Article  PubMed  Google Scholar 

  34. Zhang R, Cheng C, Zhao X, Li X (2019) Multiscale mask R-CNN–based lung tumor detection using PET imaging. Mol Imaging 18:153601211986353

    Article  Google Scholar 

  35. Zhao X, Li L, Lu W, Tan S (2018) Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys Med Biol 64:015011

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kirienko M, Sollini M, Silvestri G, Mognetti S, Voulaz E, Antunovic L et al (2018) Convolutional neural networks promising in lung cancer T-parameter assessment on baseline FDG-PET/CT. Contrast Media Mol Imaging 2018:1382309

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  37. Wang H, Zhou Z, Li Y, Chen Z, Lu P, Wang W et al (2017) comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images. EJNMMI Res 7:11

    Article  PubMed  PubMed Central  Google Scholar 

  38. Tau N, Stundzia A, Yasufuku K, Hussey D, Metser U (2020) Convolutional neural networks in predicting nodal and distant metastatic potential of newly diagnosed non-small cell lung cancer on FDG PET images. Am J Roentgenol 215:192–197

    Article  Google Scholar 

  39. Baek S, He Y, Allen BG, Buatti JM, Smith BJ, Tong L et al (2019) Deep segmentation networks predict survival of non-small cell lung cancer. Sci Rep 9:17286

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  40. Ypsilantis P-P, Siddique M, Sohn H-M, Davies A, Cook G, Goh V et al (2015) Predicting response to neoadjuvant chemotherapy with PET imaging using convolutional neural networks. PLoS ONE 10:e0137036

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  41. Yang C-K, Yeh JC-Y, Yu W-H, Chien L-I, Lin K-H, Huang W-S et al (2019) Deep convolutional neural network-based positron emission tomography analysis predicts esophageal cancer outcome. J Clin Med 8:844

    Article  PubMed Central  Google Scholar 

  42. Polymeri E, Sadik M, Kaboteh R, Borrelli P, Enqvist O, Ulén J et al (2020) Deep learning-based quantification of PET/CT prostate gland uptake: association with overall survival. Clin Physiol Funct Imaging 40:106–113

    Article  CAS  PubMed  Google Scholar 

  43. Mortensen MA, Borrelli P, Poulsen MH, Gerke O, Enqvist O, Ulén J et al (2019) Artificial intelligence-based versus manual assessment of prostate cancer in the prostate gland: a method comparison study. Clin Physiol Funct Imaging 39:399–406

    Article  CAS  PubMed  Google Scholar 

  44. Hartenstein A, Lübbe F, Baur ADJ, Rudolph MM, Furth C, Brenner W et al (2020) Prostate cancer nodal staging: using deep learning to predict 68Ga-PSMA-positivity from CT imaging alone. Sci Rep 10:3398

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Lee JJ, Yang H, Franc BL, Iagaru A, Davidzon GA (2020) Deep learning detection of prostate cancer recurrence with 18F-FACBC (fluciclovine, Axumin®) positron emission tomography. Eur J Nucl Med Mol Imaging. https://doi.org/10.1007/s00259-020-04912-w

    Article  PubMed  PubMed Central  Google Scholar 

  46. Nobashi T, Zacharias C, Ellis JK, Ferri V, Koran ME, Franc BL et al (2020) Performance comparison of individual and ensemble CNN models for the classification of brain 18F-FDG-PET scans. J Digit Imaging 33:447–455

    Article  PubMed  Google Scholar 

  47. Shaish H, Mutasa S, Makkar J, Chang P, Schwartz L, Ahmed F (2019) Prediction of lymph node maximum standardized uptake value in patients with cancer using a 3D convolutional neural network: a proof-of-concept study. Am J Roentgenol 212:238–244

    Article  Google Scholar 

  48. Sibille L, Seifert R, Avramovic N, Vehren T, Spottiswoode B, Zuehlsdorff S et al (2020) 18F-FDG PET/CT uptake classification in lymphoma and lung cancer by using deep convolutional neural networks. Radiology 294:445–452

    Article  PubMed  Google Scholar 

  49. Kawauchi K, Furuya S, Hirata K, Katoh C, Manabe O, Kobayashi K et al (2020) A convolutional neural network-based system to classify patients using FDG PET/CT examinations. BMC Cancer 20:227

    Article  PubMed  PubMed Central  Google Scholar 

  50. Chen L, Shen C, Zhou Z, Maquilan G, Albuquerque K, Folkert MR et al (2019) Automatic PET cervical tumor segmentation by combining deep learning and anatomic prior. Phys Med Biol 64:085019

    Article  PubMed  PubMed Central  Google Scholar 

  51. Peng Y, Bi L, Guo Y, Feng D, Fulham M, Kim J (2019) Deep multi-modality collaborative learning for distant metastases predication in PET-CT soft-tissue sarcoma studies. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBS. IEEE, pp 3658–3661

  52. Ding Y, Sohn JH, Kawczynski MG, Trivedi H, Harnish R, Jenkins NW et al (2019) A deep learning model to predict a diagnosis of Alzheimer disease by using 18 F-FDG PET of the brain. Radiology 290:456–464

    Article  PubMed  Google Scholar 

  53. Liu M, Cheng D, Wang K, Wang Y (2018) Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 16:295–308

    Article  PubMed  Google Scholar 

  54. Liu M, Cheng D, Yan W (2018) Classification of Alzheimer’s disease by combination of convolutional and recurrent neural networks using FDG-PET images. Front Neuroinform 12:1–12

    Article  Google Scholar 

  55. Huang Y, Xu J, Zhou Y, Tong T, Zhuang X (2019) Diagnosis of Alzheimer’s disease via multi-modality 3D convolutional neural network. Front Neurosci 13:1–12

    Article  CAS  Google Scholar 

  56. Kim J-Y, Suh HY, Ryoo HG, Oh D, Choi H, Paeng JC et al (2010) Amyloid PET Quantification Via End-to-End Training of a Deep Learning. Nucl Med Mol Imaging 2019(53):340–348

    Google Scholar 

  57. Choi H, Jin KH (2018) Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. Behav Brain Res 344:103–109

    Article  CAS  PubMed  Google Scholar 

  58. Yee E, Popuri K, Beg MF (2020) Quantifying brain metabolism from FDG-PET images into a probability of Alzheimer’s dementia score. Hum Brain Mapp 41:5–16

    Article  PubMed  Google Scholar 

  59. Choi H, Kim YK, Yoon EJ, Lee JY, Lee DS (2020) Cognitive signature of brain FDG PET based on deep learning: domain transfer from Alzheimer’s disease to Parkinson’s disease. Eur J Nucl Med Mol Imaging 47:403–412

    Article  PubMed  Google Scholar 

  60. Zhao Y, Wu P, Wang J, Li H, Navab N, Yakushev I, et al (2019) A 3D Deep Residual Convolutional Neural Network for Differential Diagnosis of Parkinsonian Syndromes on 18F-FDG PET Images. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBS. IEEE, pp 3531–3534

  61. Manzanera OM, Meles SK, Leenders KL, Renken RJ, Pagani M, Arnaldi D et al (2019) Scaled subprofile modeling and convolutional neural networks for the identification of Parkinson’s disease in 3D nuclear imaging data. Int J Neural Syst 29:1950010

    Article  PubMed  Google Scholar 

  62. Togo R, Hirata K, Manabe O, Ohira H, Tsujino I, Magota K et al (2019) Cardiac sarcoidosis classification with deep convolutional neural network-based features using polar maps. Comput Biol Med 104:81–86

    Article  PubMed  Google Scholar 

  63. Guo J, Gong E, Fan AP, Goubran M, Khalighi MM, Zaharchuk G (2019) Predicting 15 O-Water PET cerebral blood flow maps from multi-contrast MRI using a deep convolutional neural network with evaluation of training cohort bias. J Cereb Blood Flow Metab 40:2240–2253

    Article  PubMed  PubMed Central  Google Scholar 

  64. Xiong X, Linhardt TJ, Liu W, Smith BJ, Sun W, Bauer C et al (2020) A 3D deep convolutional neural network approach for the automated measurement of cerebellum tracer uptake in FDG PET-CT scans. Med Phys 47:1058–1066

    Article  PubMed  Google Scholar 

  65. Kawauchi K, Hirata K, Katoh C, Ichikawa S, Manabe O, Kobayashi K et al (2019) A convolutional neural network-based system to prevent patient misidentification in FDG-PET examinations. Sci Rep 9:7192

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  66. van Dyk DA, Meng X-L (2001) The art of data augmentation. J Comput Graph Stat 10:1–50

    Article  Google Scholar 

  67. Kermany DDS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(1122–1131):e1129

    Google Scholar 

  68. Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P et al (2013) The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging J Digit Imaging 26:1045–1057

    Article  PubMed  Google Scholar 

  69. Abadi E, Segars WP, Tsui BMW, Kinahan PE, Bottenus N, Frangi AF et al (2020) Virtual clinical trials in medical imaging: a review. J Med Imaging 7:42805

    Article  Google Scholar 

  70. Wasserman P, Freels P, Szames D, Kurra C, Hernandez M (2020) The technophysics year: transformation of diagnostic radiology’s clinical year as a matter of necessity. Acad Radiol. https://doi.org/10.1016/j.acra.2020.04.045

    Article  PubMed  Google Scholar 

  71. Joshi AV (2020) Machine Learning and Artificial Intelligence. Springer International Publishing, Cham

    Book  Google Scholar 

  72. Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ et al (2017) Deep learning: a primer for radiologists. RadioGraphics 37:2113–2131

    Article  PubMed  Google Scholar 

  73. Nensa F, Demircioglu A, Rischpler C (2019) Artificial intelligence in nuclear medicine. J Nucl Med 60:29S-37S

    Article  PubMed  Google Scholar 

  74. Currie G, Rohren E (2020) Intelligent imaging in nuclear medicine: the principles of artificial intelligence, machine learning and deep learning. Semin Nucl Med. https://doi.org/10.1053/j.semnuclmed.2020.08.002

    Article  PubMed  Google Scholar 

  75. (2020) European Association of Nuclear Medicine October 22 – 30, 2020 Virtual. Eur J Nucl Med Mol Imaging 47, 1–753. https://doi.org/10.1007/s00259-020-04988-4

  76. AI resources and training (2020). https://www.rsna.org/en/education/ai-resources-and-training. Accessed 25 Sep 2020

Download references

Funding

The present publication was not supported by any funding from public or private sources.

Author information

Authors and Affiliations

Authors

Contributions

MK: Literature search and review, manuscript drafting. MS and MK: Content planning and critical data assessment. MB: Manuscript drafting. FG: Manuscript editing and figures’ preparation. ES: Manuscript critical revision and editing. AC: Manuscript critical revision and editing.

Corresponding author

Correspondence to Margarita Kirienko.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest with the present work.

Human and animal rights

This article does not contain any studies with human or animal subjects performed by any authors since the present paper deals with a literature review.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 347 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kirienko, M., Biroli, M., Gelardi, F. et al. Deep learning in Nuclear Medicine—focus on CNN-based approaches for PET/CT and PET/MR: where do we stand?. Clin Transl Imaging 9, 37–55 (2021). https://doi.org/10.1007/s40336-021-00411-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40336-021-00411-6

Keywords

Navigation