DeepNeuro: an open-source deep learning toolbox for neuroimaging

Abstract

Translating deep learning research from theory into clinical practice has unique challenges, specifically in the field of neuroimaging. In this paper, we present DeepNeuro, a Python-based deep learning framework that puts deep neural networks for neuroimaging into practical usage with a minimum of friction during implementation. We show how this framework can be used to design deep learning pipelines that can load and preprocess data, design and train various neural network architectures, and evaluate and visualize the results of trained networks on evaluation data. We present a way of reproducibly packaging data pre- and postprocessing functions common in the neuroimaging community, which facilitates consistent performance of networks across variable users, institutions, and scanners. We show how deep learning pipelines created with DeepNeuro can be concisely packaged into shareable Docker and Singularity containers with user-friendly command-line interfaces.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., & et al. (2016). Tensorflow: a system for large-scale machine learning. In OSDI, 16, 265–283.

    Google Scholar 

  2. Avants, B.B., Tustison, N.J., Stauffer, M., Song, G., Wu, B., & Gee, J.C. (2014). The insight toolkit image registration framework. Frontiers in neuroinformatics, 8, 44.

    PubMed  PubMed Central  Article  Google Scholar 

  3. Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., & Davatzikos, C. (2017). Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 170 117, 4.

    Google Scholar 

  4. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010). Theano: A cpu and gpu math compiler in python. In Proc 9th Python in Science Conf, Vol. 1.

  5. Bezanson, J., Edelman, A., Karpinski, S., & Shah, V.B. (2017). Julia: a fresh approach to numerical computing. SIAM review, 59(1), 65–98.

    Article  Google Scholar 

  6. Brown, J.M., Campbell, J.P., Beers, A., Chang, K., Ostmo, S., Chan, R.P., Dy, J., Erdogmus, D., Ioannidis, S., Kalpathy-Cramer, J., & et al. (2018). Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA ophthalmology.

  7. Buda, M., Maki, A., & Mazurowski, M.A. (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106, 249–259.

    PubMed  Article  Google Scholar 

  8. Chang, K., Bai, H.X., Zhou, H., Su, C., Bi, W.L., Agbodza, E., Kavouridis, V.K., Senders, J.T., Boaro, A., Beers, A., & et al. (2018a). Residual convolutional neural network for the determination of idh status in low-and high-grade gliomas from mr imaging. Clinical Cancer Research, 24(5), 1073–1081.

  9. Chang, K., Balachandar, N., Lam, C., Yi, D., Brown, J., Beers, A., & Rosen, B. (2018b). Rubin, d. L., and Kalpathy-Cramer, J: Distributed deep learning networks among institutions for medical imaging. Journal of the American Medical Informatics Association.

  10. Chang, K., Beers, A.L., Bai, H.X., Brown, J.M., Ly, K.I., Li, X., Senders, J.T., Kavouridis, V.K., Boaro, A., Su, C., & et al. (2019). Automatic assessment of glioma burden: A deep learning algorithm for fully automated volumetric and bi-dimensional measurement. Neuro-oncology.

  11. Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015). Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv:1512.01274.

  12. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., & Ronneberger, O. (2016). 3d u-net: learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention pages 424–432. Springer.

  13. Clunie, D.A. (2000). DICOM structured reporting. PixelMed publishing.

  14. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning pages 160–167. ACM.

  15. Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115.

    CAS  Article  Google Scholar 

  16. Fedorov, A., Beichel, R., Kalpathy-Cramer, J., Finet, J., Fillion-Robin, J.-C., Pujol, S., Bauer, C., Jennings, D., Fennessy, F., Sonka, M., & et al. (2012). 3d slicer as an image computing platform for the quantitative imaging network. Magnetic resonance imaging, 30(9), 1323–1341.

    PubMed  PubMed Central  Article  Google Scholar 

  17. Gheiratmand, M., Rish, I., Cecchi, G.A., Brown, M.R., Greiner, R., Polosecki, P.I., Bashivan, P., Greenshaw, A.J., Ramasubbu, R., & Dursun, S.M. (2017). Learning stable and predictive network-based patterns of schizophrenia and its clinical symptoms. NPJ schizophrenia, 3(1), 22.

    PubMed  PubMed Central  Article  Google Scholar 

  18. Gibson, E., Li, W., Sudre, C., Fidon, L., Shakir, D.I., Wang, G., Eaton-Rosen, Z., Gray, R., Doel, T., Hu, Y., & et al. (2018). Niftynet: a deep-learning platform for medical imaging. Computer methods and programs in biomedicine, 158, 113–122.

    PubMed  PubMed Central  Article  Google Scholar 

  19. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets.

  20. Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., & et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama, 316(22), 2402–2410.

    PubMed  Article  Google Scholar 

  21. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition.

  22. Herz, C., Fillion-Robin, J. -C., Onken, M., Riesmeier, J., Lasso, A., Pinter, C., Fichtinger, G., Pieper, S., Clunie, D., Kikinis, R., & et al. (2017). Dcmqi: an open source library for standardized communication of quantitative image analysis results using dicom. Cancer research, 77(21), e87–e90.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  23. Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A. -r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., & et al. (2012). Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal processing magazine, 29(6), 82–97.

    Article  Google Scholar 

  24. Hunter, J.D. (2007). Matplotlib: a 2d graphics environment. Computing in science & engineering, 9(3), 90–95.

    Article  Google Scholar 

  25. Hussain, Z., Gimenez, F., Yi, D., & Rubin, D. (2017). Differential data augmentation techniques for medical imaging classification tasks. In AMIA Annual Symposium Proceedings, volume 2017, page 979. American Medical Informatics Association.

  26. Imaging, C. (2018). bioinformatics lab at the harvard medical school, b.. w. h., and institute D.-F. C Modelhub.ai.

  27. Jones, E., Oliphant, T., & Peterson, P. (2014). {SciPy}: open source scientific tools for {Python}.

  28. Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., & et al. (2017a). Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. In International Conference on Information Processing in Medical Imaging, pages 597–609. Springer.

  29. Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., & Glocker, B. (2017b). Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis, 36, 61–78.

  30. Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196.

  31. Kindlmann, G. (2008). Bigler. J., and Van Uitert, D Nrrd file format.

  32. Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In In Advances in neural information processing systems (pp. 1097–1105).

  33. Kurtzer, G.M., Sochat, V., & Bauer, M.W. (2017). Singularity: Scientific containers for mobility of compute. PloS one, 12(5), e0177–459.

    Article  Google Scholar 

  34. Larobina, M., & Murino, L. (2014). Medical image file formats. Journal of digital imaging, 27(2), 200–206.

    PubMed  Article  Google Scholar 

  35. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. nature, 521(7553), 436.

  36. Lee, C.S., Baughman, D.M., & Lee, A.Y. (2017). Deep learning is effective for classifying normal versus age-related macular degeneration oct images. Ophthalmology Retina, 1(4), 322–327.

    PubMed  PubMed Central  Article  Google Scholar 

  37. Liu, S., Liu, S., Cai, W., Pujol, S., Kikinis, R., & Feng, D. (2014). Early diagnosis of alzheimer’s disease with deep learning. In Biomedical Imaging (ISBI), 2014 IEEE 11th International Symposium on, pages 1015–1018. IEEE.

  38. Mehrtash, A., Pesteie, M., Hetherington, J., Behringer, P.A., Kapur, T., Wells, W.M., Rohling, R., Fedorov, A., & Abolmaesumi, P. (2017). Deepinfer: open-source deep learning deployment toolkit for image-guided therapy. In Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling, volume 10135, page 101351K. International Society for Optics and Photonics.

  39. Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., & et al. (2015). The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10), 1993.

    PubMed  Article  Google Scholar 

  40. Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239), 2.

    Google Scholar 

  41. Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J.T. (2017). Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics.

  42. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch.

  43. Pawlowski, N., Ktena, S.I., Lee, M.C., Kainz, B., Rueckert, D., Glocker, B., & Rajchl, M. (2017). arXiv:1711.06853.

  44. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention pages 234–241. Springer.

  45. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.

  46. Sperduto, P.W., Berkey, B., Gaspar, L.E., Mehta, M., & Curran, W. (2008). A new prognostic index and comparison to three other indices for patients with brain metastases: an analysis of 1,960 patients in the rtog database. International Journal of Radiation Oncology* Biology* Physics, 70(2), 510–514.

    Article  Google Scholar 

  47. Stonnington, C.M., Tan, G., Klöppel, S., Chu, C., Draganski, B., Jack, Jr C.R., Chen, K., Ashburner, J., & Frackowiak, R.S. (2008). Interpreting scan data acquired from multiple scanners: a study with alzheimer’s disease. NeuroImage, 39(3), 1180–1185.

  48. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In In Proceedings of the IEEE conference on computer vision and pattern recognition, (pp. 1–9).

  49. van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J.D., Yager, N., Gouillart, E., & Yu, T. (2014). the scikit-image contributors scikit-image: image processing in Python.

  50. Walt, S. v. d., Colbert, S.C., & Varoquaux, G. (2011). The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering, 13(2), 22–30.

    Article  Google Scholar 

  51. Winzeck, S., Hakim, A., McKinley, R., Pinto, J.A.A.D.S., Alves, V., Silva, C., Pisov, M., Krivov, E., Belyaev, M., Monteiro, M., & et al. (2018). Isles 2016 & 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral mri. Frontiers in Neurology, 9, 679.

    PubMed  PubMed Central  Article  Google Scholar 

Download references

Acknowledgements

The Center for Clinical Data Science at Massachusetts General Hospital and the Brigham and Woman’s Hospital provided technical and hardware support for the development of DeepNeuro, including access to high-powered graphical processing units.

Funding

This project was supported by a training grant from the NIH Blueprint for Neuroscience Research (T90DA022759/R90DA023427) and the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under award number 5T32EB1680 to KC. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This study was supported by National Institutes of Health grants U01 CA154601, U24 CA180927, and U24 CA180918 to JKC. This research was carried out in whole or in part at the Athinoula A. Martinos Center for Biomedical Imaging at the Massachusetts General Hospital, using resources provided by the Center for Functional Neuroimaging Technologies, P41EB015896, a P41 Biotechnology Resource Grant supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB), National Institutes of Health.

Author information

Affiliations

Authors

Contributions

AB, with contributions from JB and KC, was the original creator of the DeepNeuro package and all of its component parts. KH and JP contributed to modules within DeepNeuro, as well as additional data processing features. EG conducted clinical trials at Massachusetts General Hospital, contributed to data organization for clinical datasets, and was consulted on matters pertaining to clinical trials during DeepNeuro development. KIL contributed to the evaluation of results of trained network algorithms. ST and PB conducted clinical trials for brain metastases, supporting the development of DeepNeuro’s segmentation modules. BR and JKC guided the conceptual development of the package, especially with regards to clinical end-users. All the authors read, reviewed, and approved the manuscript.

Corresponding author

Correspondence to Jayashree Kalpathy-Cramer.

Ethics declarations

Conflict of interests

JKC is a consultant/advisory board member for Infotech, Soft. ERG is an advisory board member for Blue Earth Diagnostics. BR is on the advisory board for ARIA, Butterfly, Inc., DGMIF (Daegu-Gyeongbuk Medical Innovation Foundation), QMENTA, Subtle Medical, Inc., is a consultant for Broadview Ventures, Janssen Scientific, ECRI Institute, GlaxoSmithKine, Hyperfine Research, Inc., Peking University, Wolf Greenfield, Superconducting Systems, Inc., Robins Kaplin, LLC, Millennium Pharmaceuticals, GE Healthcare, Siemens, Quinn Emanuel Trial Lawyers, Samsung, Shenzhen Maternity and Child Healthcare Hospital, and is a founder of BLINKAI Technologies, Inc. PB is a consultant for Angiochem, Lilly, Tesaro, Genentech-Roche; has received honoraria from Genentech-Roche and Merckl and has received institutional funding from Merck and Pfizer. SMT receives institutional research funding from Novartis, Genentech, Eli Lilly, Pfizer, Merck, Exelixis, Eisai, Bristol Meyers Squibb, AstraZeneca, Cyclacel, Immunomedics, Odenate, and Nektar. SMT has served as an advisor/consultant to Novartis, Eli Lilly, Pfizer, Merck, AstraZeneca, Eisai, Puma, Genentech, Immunomedics, Nektar, Tesaro, and Nanostring. The other authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Information Sharing Statement

DeepNeuro is open-source, free, and available at https://github.com/QTIM-Lab/DeepNeuro (RRID:SCR 016911). DeepNeuro makes use of the following packages via Python wrappers: 3D Slicer (RRID:SCR 005619), ANTS (RRID:SCR 004757), and dcmqi (RRID:SCR 016933). DeepNeuro depends on the following third-party Python libraries: tqdm, scikit-image, scipy (RRID:SCR 008058), numpy, pydot, matplotlib (RRID:SCR 008624), imageio, six, pyyaml, nibabel (RRID:SCR 002498), pynrrd, and pydicom (RRID:SCR 002573) (van der Walt et al. 2014; Jones et al. 2014; Hunter 2007).

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Beers, A., Brown, J., Chang, K. et al. DeepNeuro: an open-source deep learning toolbox for neuroimaging. Neuroinform 19, 127–140 (2021). https://doi.org/10.1007/s12021-020-09477-5

Download citation

Keywords

  • Neuroimaging
  • Deep learning
  • Preprocessing
  • Augmentation
  • Docker