Deep Learning in Nuclear Medicine and Molecular Imaging: Current Perspectives and Future Directions
Recent advances in deep learning have impacted various scientific and industrial fields. Due to the rapid application of deep learning in biomedical data, molecular imaging has also started to adopt this technique. In this regard, it is expected that deep learning will potentially affect the roles of molecular imaging experts as well as clinical decision making. This review firstly offers a basic overview of deep learning particularly for image data analysis to give knowledge to nuclear medicine physicians and researchers. Because of the unique characteristics and distinctive aims of various types of molecular imaging, deep learning applications can be different from other fields. In this context, the review deals with current perspectives of deep learning in molecular imaging particularly in terms of development of biomarkers. Finally, future challenges of deep learning application for molecular imaging and future roles of experts in molecular imaging will be discussed.
KeywordsDeep learning Molecular imaging Machine learning Convolutional neural network Precision medicine
Compliance with Ethical Standards
Conflict of Interest
Hongyoon Choi declares no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
For this study formal consent is not required.
- 6.Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst. 2012;25:1090–8.Google Scholar
- 9.Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M et al. A survey on deep learning in medical image analysis. arXiv:170205747. 2017.Google Scholar
- 10.Xu Y, Mo T, Feng Q, Zhong P, Lai M, Eric I et al., editors. Deep learning of feature representation with multiple instance learning for medical image analysis. Acoustics, Speech and Signal Processing (ICASSP), 2014 I.E. International Conference; 2014.Google Scholar
- 15.Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Cogn Model. 1988;5:1.Google Scholar
- 16.Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. Adv Neural Inf Proces Syst. 2006;19:153–160.Google Scholar
- 18.Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.Google Scholar
- 19.Ioffe S, Szegedy C, editors. Batch normalization: accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning; 2015.Google Scholar
- 20.Nair V, Hinton GE, editors. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10); 2010.Google Scholar
- 21.Oquab M, Bottou L, Laptev I, Sivic J, editors. Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014.Google Scholar
- 23.Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D et al., editors. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition; 2015.Google Scholar
- 24.He K, Zhang X, Ren S, Sun J, editors. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016.Google Scholar
- 25.Vinyals O, Toshev A, Bengio S, Erhan D, editors. Show and tell: a neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015.Google Scholar
- 26.Girshick R, editor. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision; 2015.Google Scholar
- 27.Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. Adv Neural Inf Proces Syst. 2015;28:91–9.Google Scholar
- 28.Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:160600915. 2016.Google Scholar
- 29.Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:151100561. 2015.Google Scholar
- 35.Roth HR, Lu L, Farag A, Shin H-C, Liu J, Turkbey EB et al., editors. Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin: Springer; 2015.Google Scholar
- 37.Shen W, Zhou M, Yang F, Yang C, Tian J, editors. Multi-scale convolutional neural networks for lung nodule classification. International Conference on Information Processing in Medical Imaging. Berlin: Springer; 2015.Google Scholar
- 40.de Brebisson A, Montana G, editors. Deep neural networks for anatomical brain segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2015.Google Scholar
- 41.Chen H, Dou Q, Yu L, Heng P-A. Voxresnet: deep voxelwise residual networks for volumetric brain segmentation. arXiv:160805895. 2016.Google Scholar
- 45.Ghafoorian M, Karssemeijer N, Heskes T, van Uden IW, Sanchez CI, Litjens G, et al. Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Sci Rep. 2017;7. https://doi.org/10.1038/s41598-017-05300-5.
- 46.Trebeschi S, van Griethuysen JJ, Lambregts DM, Lahaye MJ, Parmer C, Bakers FC, et al. Deep learning for fully-automated localization and segmentation of rectal cancer on multiparametric MR. Sci Rep. 2017;7Google Scholar
- 47.Han D, Yu J, Yu Y, Zhang G, Zhong X, Lu J, et al. Comparison of 18 F-fluorothymidine and 18 F-fluorodeoxyglucose PET/CT in delineating gross tumor volume by optimal threshold in patients with squamous cell carcinoma of thoracic esophagus. Int J Radiat Oncol Biol Phys. 2010;76:1235–41.CrossRefPubMedGoogle Scholar
- 50.Kim J, Kwon Lee J, Mu Lee K, editors. Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016.Google Scholar
- 51.Xiang L, Qiao Y, Nie D, An L, Wang Q, Shen D. Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing. 2017. https://doi.org/10.1016/j.neucom.2017.06.048.
- 52.Jiao J, Ourselin S. Fast PET reconstruction using Multi-scale Fully Convolutional Neural Networks. arXiv:170407244. 2017.Google Scholar
- 54.Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. arXiv:161107004. 2016.Google Scholar
- 55.Nie D, Trullo R, Petitjean C, Ruan S, Shen D. Medical Image Synthesis with Context-Aware Generative Adversarial Networks. arXiv:161205362. 2016.Google Scholar
- 57.Bezrukov I, Mantlik F, Schmidt H, Schölkopf B, Pichler BJ, editors. MR-based PET attenuation correction for PET/MR imaging. Seminars in nuclear medicine. Amsterdam: Elsevier; 2013.Google Scholar
- 59.Choi H, Jin KH. Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. arXiv:170406033. 2017.Google Scholar
- 60.Choi H, Ha S, Im HJ, Paek SH, Lee DS. Refining diagnosis of Parkinson's disease with deep learning-based interpretation of dopamine transporter imaging. Neuroimage Clin. 2017. https://doi.org/10.1016/j.nicl.2017.09.010.
- 62.Li Z, Wang Y, Yu J, Guo Y, Cao W. Deep learning based Radiomics (DLR) and its usage in noninvasive IDH1 prediction for low grade glioma. Sci Rep. 2017;7Google Scholar
- 63.Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Cavalho S, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014;5Google Scholar
- 65.Choi H. Functional connectivity patterns of autism spectrum disorder identified by deep feature learning. arXiv:170707932. 2017.Google Scholar
- 67.Chaudhary K, Poirion OB, Lu L, Garmire L. Deep Learning based multi-omics integration robustly predicts survival in liver cancer. Clin Cancer Res. 2017. https://doi.org/10.1158/1078-0432.CCR-17-0853.
- 68.Choi H, Na KJ. A risk stratification model for lung cancer based on gene coexpression network. bioRxiv. 2017. https://doi.org/10.1101/179770.
- 69.Gal Y, Ghahramani Z, editors. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. International Conference on Machine Learning; 2016.Google Scholar
- 71.Carneiro G, Nascimento J, Bradley AP, editors. Unregistered multiview mammogram analysis with pre-trained deep learning models. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berllin: Springer; 2015.Google Scholar
- 73.Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. arXiv:170502315. 2017.Google Scholar