Skip to main content

fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review

Abstract

One of the most significant challenges in the neuroscience community is to understand how the human brain works. Recent progress in neuroimaging techniques have validated that it is possible to decode a person’s thoughts, memories, and emotions via functional magnetic resonance imaging (i.e., fMRI) since it can measure the neural activation of human brains with satisfied spatiotemporal resolutions. However, the unprecedented scale and complexity of the fMRI data have presented critical computational bottlenecks requiring new scientific analytic tools. Given the increasingly important role of machine learning in neuroscience, a great many machine learning algorithms are presented to analyze brain activities from the fMRI data. In this paper, we mainly provide a comprehensive and up-to-date review of machine learning methods for analyzing neural activities with the following three aspects, i.e., brain image functional alignment, brain activity pattern analysis, and visual stimuli reconstruction. In addition, online resources and open research problems on brain pattern analysis are also provided for the convenience of future research.

References

  1. [1]

    J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, P. J. Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, vol. 72, no. 2, pp. 404–116, 2011. DOI: https://doi.org/10.1016/j.neuron.2011.08.026.

    Google Scholar 

  2. [2]

    M. B. Cai, N. W. Schuck, J. W. Pillow, Y. Niv. A Bayesian method for reducing bias in neural representational similarity analysis. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 4958–4966, 2016.

  3. [3]

    J. H. Tao, J. Huang, Y. Li, Z. Lian, M. Y. Niu. Semi-supervised ladder networks for speech emotion recognition. International Journal of Automation and Computing, vol. 16, no. 4, pp. 437–448, 2019. DOI: https://doi.org/10.1007/s11633-019-1175-x.

    Google Scholar 

  4. [4]

    A. M. Michael, M. Anderson, R. L. Miller, T. Adali, V. D. Calhoun. Preserving subject variability in group fMRI analysis: Performance evaluation of GICA vs. IVA. Frontiers in Systems Neuroscience, vol. 8, Article number 106, 2014. DOI: https://doi.org/10.3389/fnsys.2014.00106.

  5. [5]

    Z. F. Wen, T. Y. Yu, Z. L. Yu, Y. Q. Li. Grouped sparse Bayesian learning for voxel selection in multivoxel pattern analysis of fMRI data. NeuroImage, vol. 184, pp. 417–430, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2018.09.031.

    Google Scholar 

  6. [6]

    D. Haputhanthri, G. Brihadiswaran, S. Gunathilaka, D. Meedeniya, S. Jayarathna, M. Jaime, C. Harshaw. Integration of facial thermography in EEG-based classification of ASD. International Journal of Automation and Computing, to be published. DOI: https://doi.org/10.1007/s11633-020-1231-6.

  7. [7]

    J. V. Haxby. Multivariate pattern analysis of fMRI: The early beginnings. NeuroImage, vol. 62, no. 2, pp. 852–855, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2012.03.016.

    Google Scholar 

  8. [8]

    C. D. Du, C. Y. Du, L. J. Huang, H. G. He. Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 8, pp. 2310–2323, 2019. DOI: https://doi.org/10.1109/TNNLS.2018.2882456.

    MathSciNet  Google Scholar 

  9. [9]

    A. Lorbert, P. J. Ramadge. Kernel hyperalignment. In Proceedings of the 25th International Conference on Neural Information Processing Systems Lake Tahoe, USA, pp. 1790–1798, 2012.

  10. [10]

    Y. Zhan, J. C. Zhang, S. T. Song, L. Yao. Visual image reconstruction from fMRI activation using multi-scale support vector machine decoders. In Proceedings of the 15th International Conference on Human-Computer Interaction, Springer, Las Vegas, USA, pp. 491–497, 2013. DOI: https://doi.org/10.1007/978-3-642-39342-6_54.

    Google Scholar 

  11. [11]

    Y. Kamitani, F. Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, vol. 8, no. 5, pp. 679–685, 2005. DOI: https://doi.org/10.1038/nn1444.

    Google Scholar 

  12. [12]

    K. N. Kay, T. Naselaris, R. J. Prenger, J. L. Gallant. Identifying natural images from human brain activity. Nature, vol. 452, no. 7185, pp. 352–355, 2008. DOI: https://doi.org/10.1038/nature06713.

    Google Scholar 

  13. [13]

    F. De Martino, G. Valente, N. Staeren, J. Ashburner, R. Goebel, E. Formisano. Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns. NeuroImage, vol. 43, no. 1, pp. 44–58, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.06.037.

    Google Scholar 

  14. [14]

    O. Yamashita, M. A. Sato, T. Yoshioka, F. Tong, Y. Kamitani. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage, vol. 42, no. 4, pp. 1414–1429, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.05.050.

    Google Scholar 

  15. [15]

    W. D. Li, M. X. Liu, F. Chen, D. Q. Zhang. Graph-based decoding model for functional alignment of unaligned fMRI data. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, USA, pp. 2653–2660, 2020.

  16. [16]

    M. Yousefnezhad, D. Q. Zhang. Local discriminant hyper-alignment for multi-subject fMRI data alignment. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, California, USA, pp. 59–65, 2017. DOI

  17. [17]

    M. Yousefnezhad, D. Q. Zhang. Deep hyperalignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 1603–1611, 2017.

  18. [18]

    J. S. Guntupalli, M. Hanke, Y. O. Halchenko, A. C. Connolly, P. J. Ramadge, J. V. Haxby. A model of representational spaces in human cortex. Cerebral Cortex, vol. 26, pp. 2919–2934, 2016. DOI: https://doi.org/10.1093/cercor/bhw068.

    Google Scholar 

  19. [19]

    P. H. Chen. Multi-view Representation Learning with Applications to Functional Neuroimaging Data, Princeton University, Ph. D. dissertation, USA, 2017.

  20. [20]

    G. H. Shen, K. Dwivedi, K. Majima, T. Horikawa, Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, vol. 13, Article number 21, 2019. DOI: https://doi.org/10.3389/fncom.2019.00021.

  21. [21]

    T. Naselaris, K. N. Kay, S. Nishimoto, J. L. Gallant. Encoding and decoding in fMRI. NeuroImage, vol. 56, no. 2, pp. 400–410, 2011. DOI: https://doi.org/10.1016/j.neuroimage.2010.07.073.

    Google Scholar 

  22. [22]

    C. D. Du, J. P. Li, L. J. Huang, H. G. He. Brain encoding and decoding in fMRI with bidirectional deep generative models. Engineering, vol. 5, no. 5, pp. 948–953, 2019. DOI: https://doi.org/10.1016/j.eng.2019.03.010.

    Google Scholar 

  23. [23]

    K. Vakamudi, S. Posse, R. Jung, B. Cushnyr, M. O. Chohan. Real-time presurgical resting-state fMRI in patients with brain tumors: Quality control and comparison with task-fMRI and intraoperative mapping. Human Brain Mapping, vol. 41, pp. 797–814, 2020. DOI: https://doi.org/10.1002/hbm.24840.

    Google Scholar 

  24. [24]

    J. Talairach P. Tournoux. 3-dimensional proportional system: An approach to cerebral imaging. Co-Planar Stereotaxic Atlas of the Human Brain. Thieme, 1988.

  25. [25]

    A. C. Evans, D. L. Collins, S. R. Mills, E. D. Brown, R. L. Kelly, T. M. Peters. 3D statistical neuroanatomical models from 305 MRI volumes. In Proceedings of IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference, IEEE, San Francisco, USA, pp. 1813–1817, 1993. DOI: https://doi.org/10.1109/NSSMIC.1993.373602.

    Google Scholar 

  26. [26]

    W. Chau, A. R. McIntosh. The Talairach coordinate of a point in the MNI space: How to interpret it. NeuroImage, vol. 25, no. 2, pp. 408–416, 2005. DOI: https://doi.org/10.1016/j.neuroimage.2004.12.007.

    Google Scholar 

  27. [27]

    B. R. Conroy, B. D. Singer, J. V. Haxby, P. J. Ramadge. fMRI-based inter-subject cortical alignment using functional connectivity. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 378–386, 2009.

  28. [28]

    M. R. Sabuncu, B. D. Singer, B. Conroy, R. E. Bryan, P. J. Ramadge, J. V. Haxby. Function-based intersubject alignment of human cortical anatomy. Cerebral Cortex, vol. 20, no. 1, pp. 130–140, 2009. DOI: https://doi.org/10.1093/cercor/bhp085.

    Google Scholar 

  29. [29]

    J. P. Dmochowski, P. Sajda, J. Dias, L. C. Parra. Correlated components of ongoing EEG point to emotionally laden attention-a possible marker of engagement? Frontiers in Human Neuroscience, vol. 6, Article number 112, 2012. DOI: https://doi.org/10.3389/fnhum.2012.00112.

  30. [30]

    P. H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, vol. 31, no. 1, pp. 1–10, 1966. DOI: https://doi.org/10.1007/BF02289451.

    MathSciNet  MATH  Google Scholar 

  31. [31]

    H. Xu, A. Lorbert, P. J. Ramadge, J. S. Guntupalli, J. V. Haxby. Regularized hyperalignment of multi-set fMRI data. In Proceedings of IEEE Statistical Signal Processing Workshop, IEEE, Ann Arbor, USA, pp. 229–232, 2012. DOI: https://doi.org/10.1109/SSP.2012.6319668.

  32. [32]

    P. H. Chen, J. S. Guntupalli, J. V. Haxby, P. J. Ramadge. Joint SVD-Hyperalignment for multi-subject FMRI data alignment. In Proceedings of IEEE International Workshop on Machine Learning for Signal Processing, IEEE, Reims, France, 2014. DOI: https://doi.org/10.1109/MLSP.2014.6958912.

    Google Scholar 

  33. [33]

    P. H. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. V. Haxby, P. J. Ramadge. A reduced-dimension fMRI shared response model. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 460–468, 2015.

  34. [34]

    J. Sui, G. Pearlson, A. Caprihan, T. Adali, K. A. Kiehl, J. Y. Liu, J. Yamamoto, V. D. Calhoun. Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. NeuroImage, vol. 57, no. 3, pp. 839–855, 0011. DOI: https://doi.org/10.1016/j.neuroimage.2011.05.055.

    Google Scholar 

  35. [35]

    J. Sui, H. He, G. D. Pearlson, T. Adali, K. A. Kiehl, Q. B. Yu, V. P. Clark, E. Castro, T. White, B. A. Mueller, B. C. Ho, N. C. Andreasen, V. D. Calhoun. Three-way (N-way) fusion of brain imaging data based on mCCA+ jICA and its application to discriminating schizophrenia. NeuroImage, vol. 66, pp. 119–132, 2013. DOI: https://doi.org/10.1016/j.neuroimage.2012.10.051.

    Google Scholar 

  36. [36]

    P. H. Chen, X. Zhu, H. J. Zhang, J. S. Turek, J. Chen, T. L. Willke, U. Hasson, P. J. Ramadge. A convolutional autoencoder for multi-subject fMRI data aggregation. [Online], Available: https://arxiv.org/abs/1608.04846, 2016.

  37. [37]

    U. Hasson, O. Landesman, B. Knappmeyer, I. Vallines, N. Rubin, D. J. Heeger. Neurocinematics: The neuroscience of film. Projections, vol. 2, no. 1, pp. 1–26, 2008. DOI: https://doi.org/10.3167/proj.2008.020102.

    Google Scholar 

  38. [38]

    J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, P. Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, vol. 293, no. 5539, pp. 2425–2430, 2001. DOI: https://doi.org/10.1126/science.1063736.

    Google Scholar 

  39. [39]

    J. V. Haxby, A. C. Connolly, J. S. Guntupalli. Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, vol. 37, pp. 435–456, 2014. DOI: https://doi.org/10.1146/annurev-neuro-062012-170325.

  40. [40]

    C. Allfeld, J. D. Haynes. Mutt-voxel pattern analysis. Brain Mapping, vol. 1, pp. 641–646, 2015. DOI: https://doi.org/10.1016/B978-0-12-397025-1.00345-6.

    Google Scholar 

  41. [41]

    M. Yousefnezhad, D. Q. Zhang. Anatomical pattern analysis for decoding visual stimuli in human brains. Cognitive Computation, vol. 10, no. 2, pp. 284–295, 2018. DOI: https://doi.org/10.1007/s12559-017-9518-9.

    Google Scholar 

  42. [42]

    D. D. Wagner, R. S. Chavez, T. W. Broom. Decoding the neural representation of self and person knowledge with multivariate pattern analysis and data-driven approaches. Wiley Interdisciplinary Reviews: Cognitive Science, vol. 10, no. 1, Article number e1482, 2019. DOI: https://doi.org/10.1002/wcs.1482.

    Google Scholar 

  43. [43]

    A. C. Connolly, J. S. Guntupalli, J. Gors, M. Hanke, Y. O. Halchenko, Y. C. Wu, H. Abdi, J. V. Haxby. The representation of biological classes in the human brain. Journal of Neuroscience, vol. 32, no. 8, pp. 2608–2618, 2012. DOI: https://doi.org/10.1523/JNEUROSCI.5547-11.2012.

    Google Scholar 

  44. [44]

    S. Ryali, K. Supekar, D. A. Abrams, V. Menon. Sparse logistic regression for whole-brain classification of fMRI data. NeuroImage, vol. 51, no. 2, pp. 752–764, 2010. DOI: https://doi.org/10.1016/j.neuroimage.2010.02.040.

    Google Scholar 

  45. [45]

    L. Grosenick, B. Klingenberg, S. Greer, J. Taylor, B. Knutson. Whole-brain sparse penalized discriminant analysis for predicting choice. NeuroImage, vol. 47, no. S1, Article number S58, 2009. DOI: https://doi.org/10.1016/S1053-8119(09)70232-0.

    Google Scholar 

  46. [46]

    C. van Meel, A. Baeck, C. R. Gillebert, J. Wagemans, H. P. O. de Beeck. The representation of symmetry in multi-voxel response patterns and functional connectivity throughout the ventral visual stream. NeuroImage, vol. 191, pp. 216–224, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2019.02.030.

    Google Scholar 

  47. [47]

    N. Kriegeskorte, R. Goebel, P. Bandettini. Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 10, pp. 3863–3868, 2006. DOI https://doi.org/10.1073/pnas.0600244103.

    Google Scholar 

  48. [48]

    N. Kriegeskorte, M. Mur, P. Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, vol. 2, Article number 4, 2008. DOI: https://doi.org/10.3389/neuro.06.004.2008.

  49. [49]

    M. Yuan, Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 68, no. 1, pp. 49–67, 2006. DOI: https://doi.org/10.1111/j.1467-9868.2005.00532.x.

    MathSciNet  MATH  Google Scholar 

  50. [50]

    A. E. Hoerl, R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, vol. 42, no. 1, pp. 80–86, 2000. DOI: https://doi.org/10.1080/00401706.2000.10485983.

    MATH  Google Scholar 

  51. [51]

    H. Zou, T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 67, no. 2, pp. 301–320, 2005. DOI: https://doi.org/10.1111/j.1467-9868.2005.00503.x.

    MathSciNet  MATH  Google Scholar 

  52. [52]

    A. G. Huth, S. Nishimoto, A. T. Vu, J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, vol. 76, no. 6, pp. 1210–1224, 2012. DOI: https://doi.org/10.1016/j.neuron.2012.10.014.

    Google Scholar 

  53. [53]

    L. Su, E. Fonteneau, W. Marslen-Wilson, N. Kriegeskorte. Spatiotemporal searchlight representational similarity analysis in EMEG source space. In Proceedings of the 2nd International Workshop on Pattern Recognition in NeuroImaging, IEEE, London, UK, pp. 97–100, 2012. DOI: https://doi.org/10.1109/PRNI.2012.26.

    Google Scholar 

  54. [54]

    E. A. Wasserman, A. Chakroff, R. Saxe, L. Young. Illuminating the conceptual structure of the space of moral violations with searchlight representational similarity analysis. NeuroImage, vol. 159, pp. 371–387, 2017. DOI: https://doi.org/10.1016/j.neuroimage.2017.07.043

    Google Scholar 

  55. [55]

    M. V. Peelen, A. Caramazza Conceptual object representations in human anterior temporal cortex. Journal of Neuroscience, vol. 32, no. 45, pp. 15728–15736, 2012 DOI:https://doi.org/10.1523/JNEUROSCI.1953-12

    Google Scholar 

  56. [56]

    D. J. Kravitz, C. S. Peng, C. I. Baker Real-world scene representations in high-level visual cortex: It’s the spaces more than the places Journal of Neuroscience, vol. 31, no. 20, pp. 7322–7333, 2011 DOI: https://doi.org/10.1523/JNEUROSCI.4588-102011

    Google Scholar 

  57. [57]

    G. Handjaras, E. Ricciardi, A. Leo, A. Lenci, L. Cecchetti, M. Cosottini, G. Marotta, P. Pietrini How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge NeuroImage, vol. 135, pp. 232–242, 2016 DOI: https://doi.org/10.1016/j.neuroimage.2016.04.063

    Google Scholar 

  58. [58]

    D. I. Tamir, M. A. Thornton, J. M. Contreras, J. P. Mitchell Neural evidence that three dimensions organize mental state representation: Rationality, social impact, and valence Proceedings of the National Academy of Sciences of the United States of America, vol. 113, no. 1, pp. 194–199, 2016 DOI: https://doi.org/10.1073/pnas.1511905112.

    Google Scholar 

  59. [59]

    R. S. Chavez, T. F. Heatherton Representational similarity of social and valence information in the medial pFC Journal of Cognitive Neuroscience, vol. 27, no. 1, pp. 73–82, 2015. DOI: https://doi.org/10.1162/jocn_a_00697.

    Google Scholar 

  60. [60]

    B. Thirion, E. Duchesnay, E Hubbard, J. Dubois, J. B. Poline, D. Lebihan, S. Dehaene. Inverse retinotopy: Inferring the visual content of images from brain activation patterns. NeuroImage, vol. 33, no. 4, pp. 1104–1116, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2006.06.062.

    Google Scholar 

  61. [61]

    Y. Miyawaki, H. Uchida, O. Yamashita, M. A. Sato, Y. Morito, H. C. Tanabe, N. Sadato, Y. Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, vol. 60, no. 5, pp. 915–929, 2008. DOI: https://doi.org/10.1016/j.neuron.2008.11.004.

    Google Scholar 

  62. [62]

    T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, J. L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, vol. 63, no. 6, pp. 902–915, 2009. DOI: https://doi.org/10.1016/j.neuron.2009.09.006.

    Google Scholar 

  63. [63]

    S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, J. L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, vol. 21, no. 19, pp. 1641–1646, 2011. DOI: https://doi.org/10.1016/j.cub.2011.08.031.

    Google Scholar 

  64. [64]

    Y. Fujiwara, Y. Miyawaki, Y. Kamitani. Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Computation, vol. 25, no. 4, pp. 979–1005, 2013. DOI: https://doi.org/10.1162/NECO_a_00423.

    MathSciNet  MATH  Google Scholar 

  65. [65]

    A. S. Cowen, M. M. Chun, B. A. Kuhl. Neural portraits of perception: Reconstructing face images from evoked brain activity. NeuroImage, vol. 94, pp. 12–22, 2014. DOI: https://doi.org/10.1016/j.neuroimage.2014.03.018.

    Google Scholar 

  66. [66]

    C. D. Du, C. Y. Du, H. G. He. Sharing deep generative representation for perceived image reconstruction from human brain activity. In Proceedings of International Joint Conference on Neural Networks, IEEE, Anchorage, USA, pp. 1049–1056, 2017. DOI: https://doi.org/10.1109/IJCNN.2017.7965968.

    Google Scholar 

  67. [67]

    D. P. Kingma, M. Welling. Auto-encoding variational Bayes. [Onliee], Aaailbble: https://arxiv.org/abs/1312.6114, 2014.

  68. [68]

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680, 2014.

  69. [69]

    T. Horikawa, Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, vol. 8, Article number 15037, 2017. DOI: https://doi.org/10.1038/ncomms15037.

  70. [70]

    G. St-Yves, T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, IEEE, Miyazaki, Japan, pp. 1054–1061, 2018. DOI: https://doi.org/10.1109/SMC.2018.00187.

    Google Scholar 

  71. [71]

    Y. Güçlütürk, U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, M. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 4249–4260, 2017.

  72. [72]

    K. Seeliger, U. Güçlü, L. Ambrogioni, Y Güçlütürk, M. A. J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, vol. 181, pp. 775–785, 2018. DOI: https://doi.org/10.1016/j.neuroimage.2018.07.043.

    Google Scholar 

  73. [73]

    R. VanRullen, L. Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology, vol. 2, no. 1, Article number 193, 2019. DOI: https://doi.org/10.1038/s42003-019-0438-y.

    Google Scholar 

  74. [74]

    R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, M. Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019.

  75. [75]

    D. Li, C. D. Du, H. G. He. Semi-supervised cross-modal image generation with generative adversarial networks. Pattern Recognition, vol. 100, Article number 107085, 2020. DOI: https://doi.org/10.1016/j.patcog.2019.107085.

  76. [76]

    C. D. Du, C. Y. Du, H. Wang, J. P. Li, W. L. Zheng, B. L. Lu, H. G. He. Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In Proceedings of the 26th ACM international conference on Multimedia, ACM, Seoul, Republic of Korea, pp. 108–116, 2018. DOI: https://doi.org/10.1145/3240508.3240528.

    Google Scholar 

  77. [77]

    X. Cai, F. P. Nie, W. D. Cai, H. Huang. Heterogeneous image features integration via multi-modal semi-supervised learning model. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, 2013. DOI: https://doi.org/10.1109/ICCV.2013.218.

    Google Scholar 

  78. [78]

    Z. X. Zhang, F. Ringeval, B. Dong, E. Coutinho, B. Schuller. Enhanced semi-supervised learning for multimodal emotion recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Shanghai, China, 2016. DOI: https://doi.org/10.1109/ICASSP.2016.7472666.

    Google Scholar 

  79. [79]

    T. Schonberg, C. R. Fox, J. A. Mumford, E. Congdon, C. Trepel, R. A. Poldrack. Decreasing ventromedial prefrontal cortex activity during sequential risk-taking: An fMRI investigation of the balloon analog risk task. Frontiers in Neuroscience, vol. 6, Article number 80, 2012. DOI: https://doi.org/10.3389/fnins.2012.00080.

  80. [80]

    A. R. Aron, M. A. Gluck, R. A. Poldrack. Long-term test-retest reliability of functional MRI in a classification learning task. NeuroImage, vol. 29, no. 3, pp. 1000–1006, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2005.08.010.

    Google Scholar 

  81. [81]

    S. M. Tom, C. R. Fox, C. Trepel, R. A. Poldrack. The neural basis of loss aversion in decision-making under risk. Science, vol. 315, no. 5811, pp. 515–518, 2007. DOI: https://doi.org/10.1126/science.1134239.

    Google Scholar 

  82. [82]

    K. Foerde, B. J. Knowlton, and R. A. Poldrack. Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 31, pp. 11778–11783, 2006. DOI: https://doi.org/10.1073/pnas.0602659103.

    Google Scholar 

  83. [83]

    R. A. Poldrack, J. Clark, E. J. Paré-Blagoev, D. Shohamy, J. C. Moyano, C. Myers, M. A. Gluck. Interactive memory systems in the human brain. Nature, vol. 414, no. 6863, pp. 546–550, 2001. DOI: https://doi.org/10.1038/35107080.

    Google Scholar 

  84. [84]

    A. M. C. Kelly, L. Q. Uddin, B. B. Biswal, F. X. Castellanos, M. P. Milham. Competition between functional brain networks mediates behavioral variability. NeuroImage, vol. 39, no. 1, pp. 527–537, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2007.08.008.

    Google Scholar 

  85. [85]

    K. J. Duncan, C. Pattamadilok, I. Knierim, J. T. Devlin. Consistency and variability in unctional localisers. NeuroImage, vol. 46, no. 4, pp. 1018–1026, 2009. DOI: https://doi.org/10.1016/j.neuroimage.2009.03.014.

    Google Scholar 

  86. [86]

    J. M. Walz, R. I. Goldman, M. Carapezza, J. S. Muraskin, T. R. Brown, P. Sajda. Simultaneous EEG-MRI reveals temporal evolution of coupling between supramodal cortical attention networks and the brainstem. Journal of Neuroscience, vol. 33, no. 49, pp. 19212–19222, 2013. DOI: https://doi.org/10.1523/JNEUROSCI.2649-13.2013.

    Google Scholar 

  87. [87]

    T. D. Verstynen. The organization and dynamics of corticostriatal pathways link the medial orbitofrontal cortex to uture behavioral responses. Journal of Neurophysiology, vol. 112, no. 10, pp. 2457–2469, 2014. DOI: https://doi.org/10.1152/jn.00221.2014.

    Google Scholar 

  88. [88]

    M. G. Veldhuizen, R. K. Babbs, B. Patel, W. Fobbs, N. B. Kroemer, E. Garcia, M. R. Yeomans, D. M. Small. Integration of sweet taste and metabolism determines carbohydrate reward. Current Biology, vol. 27, no. 16, pp. 2476–2485.e6, 2017. DOI: https://doi.org/10.1016/j.cub.2017.07.018.

    Google Scholar 

  89. [89]

    M. Hanke, Y. O. Halchenko, P. B. Sederberg, S. J. Hanson, J. V. Haxby, S. Pollmann. PyMVPA: A python toolbox or multivariate pattern analysis of MRI data. Neuroinformatics, vol. 7, no. 1, pp. 37–53, 2009. DOI: https://doi.org/10.1007/s12021-008-9041-y.

    Google Scholar 

  90. [90]

    M. Hanke, Y. O. Halchenko, P. B. Sederberg, E. Olivetti, I. Fründ, J. W. Rieger, C. S. Herrmann, J. V. Haxby, S. JoséHanson, S. Pollmann. PyMVPA: A unifying approach to the analysis of neuroscientific data. Frontiers in Neuroinformatics, vol. 3, Article number 3, 2009. DOI: https://doi.org/10.3389/neuro.11.003.2009.

  91. [91]

    H. J. Zhang, P. H. Chen, P. Ramadge. Transfer learning on fMRI datasets. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, Playa Blanca, Spain, pp. 595–603, 2018.

  92. [92]

    M. L. Wang, D. Q. Zhang, J. S. Huang, P. T. Yap, D. G. Shen, M. X. Liu. Identifying autism spectrum disorder with multi-site fMRI via low-rank domain adaptation. IEEE Transactions on Medical Imaging, vol. 39, no. 3, pp. 644–655, 2020. DOI: https://doi.org/10.1109/TMI.2019.2933160.

    Google Scholar 

  93. [93]

    X. Zhang, Q. Yang. Transfer hierarchical attention network for generative dialog system. International Journal of Automation and Computing, vol. 16, no. 6, pp. 720–736, 2019. DOI: https://doi.org/10.1007/s11633-019-1200-0.

    Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 61876082, 61861130366, 6173-2006 and 61902183), National Key Research and Development Program of China (Nos. 2018 YFC2001600, 2018YFC 2001602), the Royal Society-Academy of Medical Sciences Newton Advanced Fellowship (No. NAFR1-0371), and China Postdoctoral Science Foundation funded project (No. 2019M661831).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Dao-Qiang Zhang.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Shuo Huang received the B. Sc. degree in software engineering from Northeastern University, China in 2015. He is currently a Ph. D. degree candidate in software engineering in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), China.

His research interests include machine learning and human brain decoding.

Wei Shao received the B. Sc. and M. Sc. degrees in information and computing science from Nanjing University of Technology, China in 2009 and 2012, respectively, and the Ph. D. degree in software engineering from Nanjing University of Aeronautics and Astronautics, China in 2018.

His research interests include machine learning and bioinformatics.

Mei-Ling Wang received the M. Sc. degree in information and communication engineering from Nanjing University of Information Science and Technology, China in 2016. She is currently a Ph. D. degree candidate in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China.

Her research interests include machine learning and brain imaging genetics.

Dao-Qiang Zhang received the B. Sc. and Ph. D. degrees in computer science from Nanjing University of Aeronautics and Astronautics, China in 1999, and 2004, respectively. He joined Department of Computer Science and Engineering of NUAA as a lecturer in 2004, and is a professor at present. He has published over 200 scientific articles in refereed international journals such as IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Medical Imaging, IEEE Transactions on Image Processing, Neuroimage, Human Brain Mapping, Medical Image Analysis; and conference proceedings such as IJCAI, AAAI, NIPS, CVPR, MICCAI, KDD, with 12 000+ citations by Google Scholar. He was nominated for the National Excellent Doctoral Dissertation Award of China in 2006, won the Best Paper Award and the Best Student Award of several international conferences such as PRICAI’06, STMI’12 and BICS’16, etc. He has served as a program committee member for several international conferences such as IJCAI, AAAI, NIPS, MICCAI, SDM, PRICAI, ACML, etc. He is a member of the Machine Learning Society of the Chinese Association of Artificial Intelligence (CAAI), and the Artificial Intelligence & Pattern Recognition Society of the China Computer Federation (CCF).

His research interests include machine learning, pattern recognition, data minining and medical image analysis.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Huang, S., Shao, W., Wang, ML. et al. fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review. Int. J. Autom. Comput. 18, 170–184 (2021). https://doi.org/10.1007/s11633-020-1263-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-020-1263-y

Keywords

  • Functional magnetic resonance imaging (fMRI)
  • functional alignment
  • brain activity
  • brain decoding
  • visual stimuli reconstruction