Deep Transfer Learning for Whole-Brain FMRI Analyses

  • Armin W. Thomas
  • Klaus-Robert Müller
  • Wojciech SamekEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11796)


The application of deep learning (DL) models to the decoding of cognitive states from whole-brain functional Magnetic Resonance Imaging (fMRI) data is often hindered by the small sample size and high dimensionality of these datasets. Especially, in clinical settings, where patient data are scarce. In this work, we demonstrate that transfer learning represents a solution to this problem. Particularly, we show that a DL model, which has been previously trained on a large openly available fMRI dataset of the Human Connectome Project, outperforms a model variant with the same architecture, but which is trained from scratch, when both are applied to the data of a new, unrelated fMRI task. The pre-trained DL model variant is able to correctly decode 67.51% of the cognitive states from a test dataset with 100 individuals, when fine-tuned on a dataset of the size of only three subjects.


fMRI Decoding Deep learning Transfer learning 


  1. 1.
    Barch, D.M., Burgess, G.C., Harms, M.P., et al.: Function in the human connectome: task-fmri and individual differences in behavior. Neuroimage 80, 169–189 (2013)CrossRefGoogle Scholar
  2. 2.
    Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., et al.: The minimal preprocessing pipelines for the human connectome project. Neuroimage 80, 105–124 (2013)CrossRefGoogle Scholar
  3. 3.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS, pp. 249–256 (2010)Google Scholar
  4. 4.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  5. 5.
    Huang, H., Hu, X., Zhao, Y., et al.: Modeling task fMRI data via deep convolutional autoencoder. IEEE Trans. Med. Imaging 37(7), 1551–1561 (2017)Google Scholar
  6. 6.
    Jang, H., Plis, S.M., Calhoun, V.D., et al.: Task-specific feature extraction and classification of fmri volumes using a deep neural network initialized with a deep belief network: evaluation using sensorimotor tasks. NeuroImage 145, 314–328 (2017)CrossRefGoogle Scholar
  7. 7.
    Lapuschkin, S., Wäldchen, S., Binder, A., et al.: Unmasking clever hans predictors and assessing what machines really learn. Nature Commun. 10, 1096 (2019)CrossRefGoogle Scholar
  8. 8.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)CrossRefGoogle Scholar
  9. 9.
    LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 3361(10), 1995 (1995)Google Scholar
  10. 10.
    Lemm, S., Blankertz, B., Dickhaus, T., Müller, K.R.: Introduction to machine learning for brain imaging. Neuroimage 56(2), 387–399 (2011)CrossRefGoogle Scholar
  11. 11.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)Google Scholar
  12. 12.
    Plis, S.M., Hjelm, D.R., Salakhutdinov, R., et al.: Deep learning for neuroimaging: a validation study. Front. Neurosci. 8, 229 (2014)CrossRefGoogle Scholar
  13. 13.
    Thomas, A.W., Heekeren, H.R., Müller, K.R., et al.: Analyzing neuroimaging data through recurrent deep learning models. arXiv preprint arXiv:1810.09945 (2018)
  14. 14.
    Uğurbil, K., Xu, J., Auerbach, E.J., et al.: Pushing spatial and temporal resolution for functional and diffusion mri in the human connectome project. Neuroimage 80, 80–104 (2013)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Technische Universität BerlinBerlinGermany
  2. 2.Max Planck School of CognitionLeipzigGermany
  3. 3.Korea UniversitySeoulSouth Korea
  4. 4.Max Planck Institute for InformaticsSaarbrückenGermany
  5. 5.Fraunhofer Heinrich Hertz InstituteBerlinGermany

Personalised recommendations