Skip to main content

Advertisement

SpringerLink
Go to cart
  1. Home
  2. International Journal of Automation and Computing
  3. Article
fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review
Download PDF
Your article has downloaded

Similar articles being viewed by others

Slider with three articles shown per slide. Use the Previous and Next buttons to navigate the slides or the slide controller buttons at the end to navigate through each slide.

The Last Frontier

01 January 2019

Kavita Dorai

fMRIflows: A Consortium of Fully Automatic Univariate and Multivariate fMRI Processing Pipelines

27 December 2022

Michael P. Notter, Peer Herholz, … Micah M. Murray

Establishing the functional relevancy of white matter connections in the visual system and beyond

30 November 2021

Mareike Grotheer, Emily Kubota & Kalanit Grill-Spector

fMRI data processing in MRTOOL: to what extent does anatomical registration affect the reliability of functional results?

22 November 2018

Marco Ganzetti, Gaia Amaranta Taberna & Dante Mantini

Reconstruction of 3D Images from Human Activity by a Compound Reconstruction Model

28 April 2022

Hongna Zheng, Li Yao & Zhiying Long

Rapid event-related, BOLD fMRI, non-human primates (NHP): choose two out of three

04 May 2020

Vassilis Pelekanos, Robert M. Mok, … Nikolaus Kriegeskorte

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

14 July 2022

Yi-Jun Zhang, Zhao-Fei Yu, … Tie-Jun Huang

Review Paper: Reporting Practices for Task fMRI Studies

06 October 2022

Freya Acar, Camille Maumet, … Beatrijs Moerkerke

Multi-Objective Cognitive Model: a Supervised Approach for Multi-subject fMRI Analysis

09 August 2018

Muhammad Yousefnezhad & Daoqiang Zhang

Download PDF
  • Review
  • Open Access
  • Published: 16 January 2021

fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review

  • Shuo Huang  ORCID: orcid.org/0000-0002-3267-18161,2 na1,
  • Wei Shao  ORCID: orcid.org/0000-0003-1476-20681,2 na1,
  • Mei-Ling Wang  ORCID: orcid.org/0000-0001-6569-27981,2 &
  • …
  • Dao-Qiang Zhang  ORCID: orcid.org/0000-0002-5658-76431,2 

International Journal of Automation and Computing volume 18, pages 170–184 (2021)Cite this article

  • 1568 Accesses

  • 3 Citations

  • 2 Altmetric

  • Metrics details

Abstract

One of the most significant challenges in the neuroscience community is to understand how the human brain works. Recent progress in neuroimaging techniques have validated that it is possible to decode a person’s thoughts, memories, and emotions via functional magnetic resonance imaging (i.e., fMRI) since it can measure the neural activation of human brains with satisfied spatiotemporal resolutions. However, the unprecedented scale and complexity of the fMRI data have presented critical computational bottlenecks requiring new scientific analytic tools. Given the increasingly important role of machine learning in neuroscience, a great many machine learning algorithms are presented to analyze brain activities from the fMRI data. In this paper, we mainly provide a comprehensive and up-to-date review of machine learning methods for analyzing neural activities with the following three aspects, i.e., brain image functional alignment, brain activity pattern analysis, and visual stimuli reconstruction. In addition, online resources and open research problems on brain pattern analysis are also provided for the convenience of future research.

Download to read the full article text

Working on a manuscript?

Avoid the common mistakes

References

  1. J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, P. J. Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, vol. 72, no. 2, pp. 404–116, 2011. DOI: https://doi.org/10.1016/j.neuron.2011.08.026.

    Article  Google Scholar 

  2. M. B. Cai, N. W. Schuck, J. W. Pillow, Y. Niv. A Bayesian method for reducing bias in neural representational similarity analysis. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 4958–4966, 2016.

  3. J. H. Tao, J. Huang, Y. Li, Z. Lian, M. Y. Niu. Semi-supervised ladder networks for speech emotion recognition. International Journal of Automation and Computing, vol. 16, no. 4, pp. 437–448, 2019. DOI: https://doi.org/10.1007/s11633-019-1175-x.

    Article  Google Scholar 

  4. A. M. Michael, M. Anderson, R. L. Miller, T. Adali, V. D. Calhoun. Preserving subject variability in group fMRI analysis: Performance evaluation of GICA vs. IVA. Frontiers in Systems Neuroscience, vol. 8, Article number 106, 2014. DOI: https://doi.org/10.3389/fnsys.2014.00106.

  5. Z. F. Wen, T. Y. Yu, Z. L. Yu, Y. Q. Li. Grouped sparse Bayesian learning for voxel selection in multivoxel pattern analysis of fMRI data. NeuroImage, vol. 184, pp. 417–430, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2018.09.031.

    Article  Google Scholar 

  6. D. Haputhanthri, G. Brihadiswaran, S. Gunathilaka, D. Meedeniya, S. Jayarathna, M. Jaime, C. Harshaw. Integration of facial thermography in EEG-based classification of ASD. International Journal of Automation and Computing, to be published. DOI: https://doi.org/10.1007/s11633-020-1231-6.

  7. J. V. Haxby. Multivariate pattern analysis of fMRI: The early beginnings. NeuroImage, vol. 62, no. 2, pp. 852–855, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2012.03.016.

    Article  Google Scholar 

  8. C. D. Du, C. Y. Du, L. J. Huang, H. G. He. Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 8, pp. 2310–2323, 2019. DOI: https://doi.org/10.1109/TNNLS.2018.2882456.

    Article  MathSciNet  Google Scholar 

  9. A. Lorbert, P. J. Ramadge. Kernel hyperalignment. In Proceedings of the 25th International Conference on Neural Information Processing Systems Lake Tahoe, USA, pp. 1790–1798, 2012.

  10. Y. Zhan, J. C. Zhang, S. T. Song, L. Yao. Visual image reconstruction from fMRI activation using multi-scale support vector machine decoders. In Proceedings of the 15th International Conference on Human-Computer Interaction, Springer, Las Vegas, USA, pp. 491–497, 2013. DOI: https://doi.org/10.1007/978-3-642-39342-6_54.

    Google Scholar 

  11. Y. Kamitani, F. Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, vol. 8, no. 5, pp. 679–685, 2005. DOI: https://doi.org/10.1038/nn1444.

    Article  Google Scholar 

  12. K. N. Kay, T. Naselaris, R. J. Prenger, J. L. Gallant. Identifying natural images from human brain activity. Nature, vol. 452, no. 7185, pp. 352–355, 2008. DOI: https://doi.org/10.1038/nature06713.

    Article  Google Scholar 

  13. F. De Martino, G. Valente, N. Staeren, J. Ashburner, R. Goebel, E. Formisano. Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns. NeuroImage, vol. 43, no. 1, pp. 44–58, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.06.037.

    Article  Google Scholar 

  14. O. Yamashita, M. A. Sato, T. Yoshioka, F. Tong, Y. Kamitani. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage, vol. 42, no. 4, pp. 1414–1429, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.05.050.

    Article  Google Scholar 

  15. W. D. Li, M. X. Liu, F. Chen, D. Q. Zhang. Graph-based decoding model for functional alignment of unaligned fMRI data. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, USA, pp. 2653–2660, 2020.

  16. M. Yousefnezhad, D. Q. Zhang. Local discriminant hyper-alignment for multi-subject fMRI data alignment. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, California, USA, pp. 59–65, 2017. DOI

  17. M. Yousefnezhad, D. Q. Zhang. Deep hyperalignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 1603–1611, 2017.

  18. J. S. Guntupalli, M. Hanke, Y. O. Halchenko, A. C. Connolly, P. J. Ramadge, J. V. Haxby. A model of representational spaces in human cortex. Cerebral Cortex, vol. 26, pp. 2919–2934, 2016. DOI: https://doi.org/10.1093/cercor/bhw068.

    Article  Google Scholar 

  19. P. H. Chen. Multi-view Representation Learning with Applications to Functional Neuroimaging Data, Princeton University, Ph. D. dissertation, USA, 2017.

  20. G. H. Shen, K. Dwivedi, K. Majima, T. Horikawa, Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, vol. 13, Article number 21, 2019. DOI: https://doi.org/10.3389/fncom.2019.00021.

  21. T. Naselaris, K. N. Kay, S. Nishimoto, J. L. Gallant. Encoding and decoding in fMRI. NeuroImage, vol. 56, no. 2, pp. 400–410, 2011. DOI: https://doi.org/10.1016/j.neuroimage.2010.07.073.

    Article  Google Scholar 

  22. C. D. Du, J. P. Li, L. J. Huang, H. G. He. Brain encoding and decoding in fMRI with bidirectional deep generative models. Engineering, vol. 5, no. 5, pp. 948–953, 2019. DOI: https://doi.org/10.1016/j.eng.2019.03.010.

    Article  Google Scholar 

  23. K. Vakamudi, S. Posse, R. Jung, B. Cushnyr, M. O. Chohan. Real-time presurgical resting-state fMRI in patients with brain tumors: Quality control and comparison with task-fMRI and intraoperative mapping. Human Brain Mapping, vol. 41, pp. 797–814, 2020. DOI: https://doi.org/10.1002/hbm.24840.

    Article  Google Scholar 

  24. J. Talairach P. Tournoux. 3-dimensional proportional system: An approach to cerebral imaging. Co-Planar Stereotaxic Atlas of the Human Brain. Thieme, 1988.

  25. A. C. Evans, D. L. Collins, S. R. Mills, E. D. Brown, R. L. Kelly, T. M. Peters. 3D statistical neuroanatomical models from 305 MRI volumes. In Proceedings of IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference, IEEE, San Francisco, USA, pp. 1813–1817, 1993. DOI: https://doi.org/10.1109/NSSMIC.1993.373602.

    Google Scholar 

  26. W. Chau, A. R. McIntosh. The Talairach coordinate of a point in the MNI space: How to interpret it. NeuroImage, vol. 25, no. 2, pp. 408–416, 2005. DOI: https://doi.org/10.1016/j.neuroimage.2004.12.007.

    Article  Google Scholar 

  27. B. R. Conroy, B. D. Singer, J. V. Haxby, P. J. Ramadge. fMRI-based inter-subject cortical alignment using functional connectivity. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 378–386, 2009.

  28. M. R. Sabuncu, B. D. Singer, B. Conroy, R. E. Bryan, P. J. Ramadge, J. V. Haxby. Function-based intersubject alignment of human cortical anatomy. Cerebral Cortex, vol. 20, no. 1, pp. 130–140, 2009. DOI: https://doi.org/10.1093/cercor/bhp085.

    Article  Google Scholar 

  29. J. P. Dmochowski, P. Sajda, J. Dias, L. C. Parra. Correlated components of ongoing EEG point to emotionally laden attention-a possible marker of engagement? Frontiers in Human Neuroscience, vol. 6, Article number 112, 2012. DOI: https://doi.org/10.3389/fnhum.2012.00112.

  30. P. H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, vol. 31, no. 1, pp. 1–10, 1966. DOI: https://doi.org/10.1007/BF02289451.

    Article  MathSciNet  MATH  Google Scholar 

  31. H. Xu, A. Lorbert, P. J. Ramadge, J. S. Guntupalli, J. V. Haxby. Regularized hyperalignment of multi-set fMRI data. In Proceedings of IEEE Statistical Signal Processing Workshop, IEEE, Ann Arbor, USA, pp. 229–232, 2012. DOI: https://doi.org/10.1109/SSP.2012.6319668.

  32. P. H. Chen, J. S. Guntupalli, J. V. Haxby, P. J. Ramadge. Joint SVD-Hyperalignment for multi-subject FMRI data alignment. In Proceedings of IEEE International Workshop on Machine Learning for Signal Processing, IEEE, Reims, France, 2014. DOI: https://doi.org/10.1109/MLSP.2014.6958912.

    Book  Google Scholar 

  33. P. H. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. V. Haxby, P. J. Ramadge. A reduced-dimension fMRI shared response model. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 460–468, 2015.

  34. J. Sui, G. Pearlson, A. Caprihan, T. Adali, K. A. Kiehl, J. Y. Liu, J. Yamamoto, V. D. Calhoun. Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. NeuroImage, vol. 57, no. 3, pp. 839–855, 0011. DOI: https://doi.org/10.1016/j.neuroimage.2011.05.055.

    Article  Google Scholar 

  35. J. Sui, H. He, G. D. Pearlson, T. Adali, K. A. Kiehl, Q. B. Yu, V. P. Clark, E. Castro, T. White, B. A. Mueller, B. C. Ho, N. C. Andreasen, V. D. Calhoun. Three-way (N-way) fusion of brain imaging data based on mCCA+ jICA and its application to discriminating schizophrenia. NeuroImage, vol. 66, pp. 119–132, 2013. DOI: https://doi.org/10.1016/j.neuroimage.2012.10.051.

    Article  Google Scholar 

  36. P. H. Chen, X. Zhu, H. J. Zhang, J. S. Turek, J. Chen, T. L. Willke, U. Hasson, P. J. Ramadge. A convolutional autoencoder for multi-subject fMRI data aggregation. [Online], Available: https://arxiv.org/abs/1608.04846, 2016.

  37. U. Hasson, O. Landesman, B. Knappmeyer, I. Vallines, N. Rubin, D. J. Heeger. Neurocinematics: The neuroscience of film. Projections, vol. 2, no. 1, pp. 1–26, 2008. DOI: https://doi.org/10.3167/proj.2008.020102.

    Article  Google Scholar 

  38. J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, P. Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, vol. 293, no. 5539, pp. 2425–2430, 2001. DOI: https://doi.org/10.1126/science.1063736.

    Article  Google Scholar 

  39. J. V. Haxby, A. C. Connolly, J. S. Guntupalli. Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, vol. 37, pp. 435–456, 2014. DOI: https://doi.org/10.1146/annurev-neuro-062012-170325.

  40. C. Allfeld, J. D. Haynes. Mutt-voxel pattern analysis. Brain Mapping, vol. 1, pp. 641–646, 2015. DOI: https://doi.org/10.1016/B978-0-12-397025-1.00345-6.

    Article  Google Scholar 

  41. M. Yousefnezhad, D. Q. Zhang. Anatomical pattern analysis for decoding visual stimuli in human brains. Cognitive Computation, vol. 10, no. 2, pp. 284–295, 2018. DOI: https://doi.org/10.1007/s12559-017-9518-9.

    Article  Google Scholar 

  42. D. D. Wagner, R. S. Chavez, T. W. Broom. Decoding the neural representation of self and person knowledge with multivariate pattern analysis and data-driven approaches. Wiley Interdisciplinary Reviews: Cognitive Science, vol. 10, no. 1, Article number e1482, 2019. DOI: https://doi.org/10.1002/wcs.1482.

    Google Scholar 

  43. A. C. Connolly, J. S. Guntupalli, J. Gors, M. Hanke, Y. O. Halchenko, Y. C. Wu, H. Abdi, J. V. Haxby. The representation of biological classes in the human brain. Journal of Neuroscience, vol. 32, no. 8, pp. 2608–2618, 2012. DOI: https://doi.org/10.1523/JNEUROSCI.5547-11.2012.

    Article  Google Scholar 

  44. S. Ryali, K. Supekar, D. A. Abrams, V. Menon. Sparse logistic regression for whole-brain classification of fMRI data. NeuroImage, vol. 51, no. 2, pp. 752–764, 2010. DOI: https://doi.org/10.1016/j.neuroimage.2010.02.040.

    Article  Google Scholar 

  45. L. Grosenick, B. Klingenberg, S. Greer, J. Taylor, B. Knutson. Whole-brain sparse penalized discriminant analysis for predicting choice. NeuroImage, vol. 47, no. S1, Article number S58, 2009. DOI: https://doi.org/10.1016/S1053-8119(09)70232-0.

    Google Scholar 

  46. C. van Meel, A. Baeck, C. R. Gillebert, J. Wagemans, H. P. O. de Beeck. The representation of symmetry in multi-voxel response patterns and functional connectivity throughout the ventral visual stream. NeuroImage, vol. 191, pp. 216–224, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2019.02.030.

    Article  Google Scholar 

  47. N. Kriegeskorte, R. Goebel, P. Bandettini. Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 10, pp. 3863–3868, 2006. DOI https://doi.org/10.1073/pnas.0600244103.

    Article  Google Scholar 

  48. N. Kriegeskorte, M. Mur, P. Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, vol. 2, Article number 4, 2008. DOI: https://doi.org/10.3389/neuro.06.004.2008.

  49. M. Yuan, Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 68, no. 1, pp. 49–67, 2006. DOI: https://doi.org/10.1111/j.1467-9868.2005.00532.x.

    Article  MathSciNet  MATH  Google Scholar 

  50. A. E. Hoerl, R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, vol. 42, no. 1, pp. 80–86, 2000. DOI: https://doi.org/10.1080/00401706.2000.10485983.

    Article  MATH  Google Scholar 

  51. H. Zou, T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 67, no. 2, pp. 301–320, 2005. DOI: https://doi.org/10.1111/j.1467-9868.2005.00503.x.

    Article  MathSciNet  MATH  Google Scholar 

  52. A. G. Huth, S. Nishimoto, A. T. Vu, J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, vol. 76, no. 6, pp. 1210–1224, 2012. DOI: https://doi.org/10.1016/j.neuron.2012.10.014.

    Article  Google Scholar 

  53. L. Su, E. Fonteneau, W. Marslen-Wilson, N. Kriegeskorte. Spatiotemporal searchlight representational similarity analysis in EMEG source space. In Proceedings of the 2nd International Workshop on Pattern Recognition in NeuroImaging, IEEE, London, UK, pp. 97–100, 2012. DOI: https://doi.org/10.1109/PRNI.2012.26.

    Google Scholar 

  54. E. A. Wasserman, A. Chakroff, R. Saxe, L. Young. Illuminating the conceptual structure of the space of moral violations with searchlight representational similarity analysis. NeuroImage, vol. 159, pp. 371–387, 2017. DOI: https://doi.org/10.1016/j.neuroimage.2017.07.043

    Article  Google Scholar 

  55. M. V. Peelen, A. Caramazza Conceptual object representations in human anterior temporal cortex. Journal of Neuroscience, vol. 32, no. 45, pp. 15728–15736, 2012 DOI:https://doi.org/10.1523/JNEUROSCI.1953-12

    Article  Google Scholar 

  56. D. J. Kravitz, C. S. Peng, C. I. Baker Real-world scene representations in high-level visual cortex: It’s the spaces more than the places Journal of Neuroscience, vol. 31, no. 20, pp. 7322–7333, 2011 DOI: https://doi.org/10.1523/JNEUROSCI.4588-102011

    Article  Google Scholar 

  57. G. Handjaras, E. Ricciardi, A. Leo, A. Lenci, L. Cecchetti, M. Cosottini, G. Marotta, P. Pietrini How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge NeuroImage, vol. 135, pp. 232–242, 2016 DOI: https://doi.org/10.1016/j.neuroimage.2016.04.063

    Article  Google Scholar 

  58. D. I. Tamir, M. A. Thornton, J. M. Contreras, J. P. Mitchell Neural evidence that three dimensions organize mental state representation: Rationality, social impact, and valence Proceedings of the National Academy of Sciences of the United States of America, vol. 113, no. 1, pp. 194–199, 2016 DOI: https://doi.org/10.1073/pnas.1511905112.

    Article  Google Scholar 

  59. R. S. Chavez, T. F. Heatherton Representational similarity of social and valence information in the medial pFC Journal of Cognitive Neuroscience, vol. 27, no. 1, pp. 73–82, 2015. DOI: https://doi.org/10.1162/jocn_a_00697.

    Article  Google Scholar 

  60. B. Thirion, E. Duchesnay, E Hubbard, J. Dubois, J. B. Poline, D. Lebihan, S. Dehaene. Inverse retinotopy: Inferring the visual content of images from brain activation patterns. NeuroImage, vol. 33, no. 4, pp. 1104–1116, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2006.06.062.

    Article  Google Scholar 

  61. Y. Miyawaki, H. Uchida, O. Yamashita, M. A. Sato, Y. Morito, H. C. Tanabe, N. Sadato, Y. Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, vol. 60, no. 5, pp. 915–929, 2008. DOI: https://doi.org/10.1016/j.neuron.2008.11.004.

    Article  Google Scholar 

  62. T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, J. L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, vol. 63, no. 6, pp. 902–915, 2009. DOI: https://doi.org/10.1016/j.neuron.2009.09.006.

    Article  Google Scholar 

  63. S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, J. L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, vol. 21, no. 19, pp. 1641–1646, 2011. DOI: https://doi.org/10.1016/j.cub.2011.08.031.

    Article  Google Scholar 

  64. Y. Fujiwara, Y. Miyawaki, Y. Kamitani. Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Computation, vol. 25, no. 4, pp. 979–1005, 2013. DOI: https://doi.org/10.1162/NECO_a_00423.

    Article  MathSciNet  MATH  Google Scholar 

  65. A. S. Cowen, M. M. Chun, B. A. Kuhl. Neural portraits of perception: Reconstructing face images from evoked brain activity. NeuroImage, vol. 94, pp. 12–22, 2014. DOI: https://doi.org/10.1016/j.neuroimage.2014.03.018.

    Article  Google Scholar 

  66. C. D. Du, C. Y. Du, H. G. He. Sharing deep generative representation for perceived image reconstruction from human brain activity. In Proceedings of International Joint Conference on Neural Networks, IEEE, Anchorage, USA, pp. 1049–1056, 2017. DOI: https://doi.org/10.1109/IJCNN.2017.7965968.

    Google Scholar 

  67. D. P. Kingma, M. Welling. Auto-encoding variational Bayes. [Onliee], Aaailbble: https://arxiv.org/abs/1312.6114, 2014.

  68. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680, 2014.

  69. T. Horikawa, Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, vol. 8, Article number 15037, 2017. DOI: https://doi.org/10.1038/ncomms15037.

  70. G. St-Yves, T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, IEEE, Miyazaki, Japan, pp. 1054–1061, 2018. DOI: https://doi.org/10.1109/SMC.2018.00187.

    Google Scholar 

  71. Y. Güçlütürk, U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, M. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 4249–4260, 2017.

  72. K. Seeliger, U. Güçlü, L. Ambrogioni, Y Güçlütürk, M. A. J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, vol. 181, pp. 775–785, 2018. DOI: https://doi.org/10.1016/j.neuroimage.2018.07.043.

    Article  Google Scholar 

  73. R. VanRullen, L. Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology, vol. 2, no. 1, Article number 193, 2019. DOI: https://doi.org/10.1038/s42003-019-0438-y.

    Google Scholar 

  74. R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, M. Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019.

  75. D. Li, C. D. Du, H. G. He. Semi-supervised cross-modal image generation with generative adversarial networks. Pattern Recognition, vol. 100, Article number 107085, 2020. DOI: https://doi.org/10.1016/j.patcog.2019.107085.

  76. C. D. Du, C. Y. Du, H. Wang, J. P. Li, W. L. Zheng, B. L. Lu, H. G. He. Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In Proceedings of the 26th ACM international conference on Multimedia, ACM, Seoul, Republic of Korea, pp. 108–116, 2018. DOI: https://doi.org/10.1145/3240508.3240528.

    Google Scholar 

  77. X. Cai, F. P. Nie, W. D. Cai, H. Huang. Heterogeneous image features integration via multi-modal semi-supervised learning model. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, 2013. DOI: https://doi.org/10.1109/ICCV.2013.218.

    Book  Google Scholar 

  78. Z. X. Zhang, F. Ringeval, B. Dong, E. Coutinho, B. Schuller. Enhanced semi-supervised learning for multimodal emotion recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Shanghai, China, 2016. DOI: https://doi.org/10.1109/ICASSP.2016.7472666.

    Book  Google Scholar 

  79. T. Schonberg, C. R. Fox, J. A. Mumford, E. Congdon, C. Trepel, R. A. Poldrack. Decreasing ventromedial prefrontal cortex activity during sequential risk-taking: An fMRI investigation of the balloon analog risk task. Frontiers in Neuroscience, vol. 6, Article number 80, 2012. DOI: https://doi.org/10.3389/fnins.2012.00080.

  80. A. R. Aron, M. A. Gluck, R. A. Poldrack. Long-term test-retest reliability of functional MRI in a classification learning task. NeuroImage, vol. 29, no. 3, pp. 1000–1006, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2005.08.010.

    Article  Google Scholar 

  81. S. M. Tom, C. R. Fox, C. Trepel, R. A. Poldrack. The neural basis of loss aversion in decision-making under risk. Science, vol. 315, no. 5811, pp. 515–518, 2007. DOI: https://doi.org/10.1126/science.1134239.

    Article  Google Scholar 

  82. K. Foerde, B. J. Knowlton, and R. A. Poldrack. Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 31, pp. 11778–11783, 2006. DOI: https://doi.org/10.1073/pnas.0602659103.

    Article  Google Scholar 

  83. R. A. Poldrack, J. Clark, E. J. Paré-Blagoev, D. Shohamy, J. C. Moyano, C. Myers, M. A. Gluck. Interactive memory systems in the human brain. Nature, vol. 414, no. 6863, pp. 546–550, 2001. DOI: https://doi.org/10.1038/35107080.

    Article  Google Scholar 

  84. A. M. C. Kelly, L. Q. Uddin, B. B. Biswal, F. X. Castellanos, M. P. Milham. Competition between functional brain networks mediates behavioral variability. NeuroImage, vol. 39, no. 1, pp. 527–537, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2007.08.008.

    Article  Google Scholar 

  85. K. J. Duncan, C. Pattamadilok, I. Knierim, J. T. Devlin. Consistency and variability in unctional localisers. NeuroImage, vol. 46, no. 4, pp. 1018–1026, 2009. DOI: https://doi.org/10.1016/j.neuroimage.2009.03.014.

    Article  Google Scholar 

  86. J. M. Walz, R. I. Goldman, M. Carapezza, J. S. Muraskin, T. R. Brown, P. Sajda. Simultaneous EEG-MRI reveals temporal evolution of coupling between supramodal cortical attention networks and the brainstem. Journal of Neuroscience, vol. 33, no. 49, pp. 19212–19222, 2013. DOI: https://doi.org/10.1523/JNEUROSCI.2649-13.2013.

    Article  Google Scholar 

  87. T. D. Verstynen. The organization and dynamics of corticostriatal pathways link the medial orbitofrontal cortex to uture behavioral responses. Journal of Neurophysiology, vol. 112, no. 10, pp. 2457–2469, 2014. DOI: https://doi.org/10.1152/jn.00221.2014.

    Article  Google Scholar 

  88. M. G. Veldhuizen, R. K. Babbs, B. Patel, W. Fobbs, N. B. Kroemer, E. Garcia, M. R. Yeomans, D. M. Small. Integration of sweet taste and metabolism determines carbohydrate reward. Current Biology, vol. 27, no. 16, pp. 2476–2485.e6, 2017. DOI: https://doi.org/10.1016/j.cub.2017.07.018.

    Article  Google Scholar 

  89. M. Hanke, Y. O. Halchenko, P. B. Sederberg, S. J. Hanson, J. V. Haxby, S. Pollmann. PyMVPA: A python toolbox or multivariate pattern analysis of MRI data. Neuroinformatics, vol. 7, no. 1, pp. 37–53, 2009. DOI: https://doi.org/10.1007/s12021-008-9041-y.

    Article  Google Scholar 

  90. M. Hanke, Y. O. Halchenko, P. B. Sederberg, E. Olivetti, I. Fründ, J. W. Rieger, C. S. Herrmann, J. V. Haxby, S. JoséHanson, S. Pollmann. PyMVPA: A unifying approach to the analysis of neuroscientific data. Frontiers in Neuroinformatics, vol. 3, Article number 3, 2009. DOI: https://doi.org/10.3389/neuro.11.003.2009.

  91. H. J. Zhang, P. H. Chen, P. Ramadge. Transfer learning on fMRI datasets. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, Playa Blanca, Spain, pp. 595–603, 2018.

  92. M. L. Wang, D. Q. Zhang, J. S. Huang, P. T. Yap, D. G. Shen, M. X. Liu. Identifying autism spectrum disorder with multi-site fMRI via low-rank domain adaptation. IEEE Transactions on Medical Imaging, vol. 39, no. 3, pp. 644–655, 2020. DOI: https://doi.org/10.1109/TMI.2019.2933160.

    Article  Google Scholar 

  93. X. Zhang, Q. Yang. Transfer hierarchical attention network for generative dialog system. International Journal of Automation and Computing, vol. 16, no. 6, pp. 720–736, 2019. DOI: https://doi.org/10.1007/s11633-019-1200-0.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 61876082, 61861130366, 6173-2006 and 61902183), National Key Research and Development Program of China (Nos. 2018 YFC2001600, 2018YFC 2001602), the Royal Society-Academy of Medical Sciences Newton Advanced Fellowship (No. NAFR1-0371), and China Postdoctoral Science Foundation funded project (No. 2019M661831).

Author information

Author notes

  1. These authors contributed to the work equally and should be regarded as co-first authors

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Shuo Huang, Wei Shao, Mei-Ling Wang & Dao-Qiang Zhang

  2. MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, 211106, China

    Shuo Huang, Wei Shao, Mei-Ling Wang & Dao-Qiang Zhang

Authors
  1. Shuo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mei-Ling Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dao-Qiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dao-Qiang Zhang.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Shuo Huang received the B. Sc. degree in software engineering from Northeastern University, China in 2015. He is currently a Ph. D. degree candidate in software engineering in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), China.

His research interests include machine learning and human brain decoding.

Wei Shao received the B. Sc. and M. Sc. degrees in information and computing science from Nanjing University of Technology, China in 2009 and 2012, respectively, and the Ph. D. degree in software engineering from Nanjing University of Aeronautics and Astronautics, China in 2018.

His research interests include machine learning and bioinformatics.

Mei-Ling Wang received the M. Sc. degree in information and communication engineering from Nanjing University of Information Science and Technology, China in 2016. She is currently a Ph. D. degree candidate in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China.

Her research interests include machine learning and brain imaging genetics.

Dao-Qiang Zhang received the B. Sc. and Ph. D. degrees in computer science from Nanjing University of Aeronautics and Astronautics, China in 1999, and 2004, respectively. He joined Department of Computer Science and Engineering of NUAA as a lecturer in 2004, and is a professor at present. He has published over 200 scientific articles in refereed international journals such as IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Medical Imaging, IEEE Transactions on Image Processing, Neuroimage, Human Brain Mapping, Medical Image Analysis; and conference proceedings such as IJCAI, AAAI, NIPS, CVPR, MICCAI, KDD, with 12 000+ citations by Google Scholar. He was nominated for the National Excellent Doctoral Dissertation Award of China in 2006, won the Best Paper Award and the Best Student Award of several international conferences such as PRICAI’06, STMI’12 and BICS’16, etc. He has served as a program committee member for several international conferences such as IJCAI, AAAI, NIPS, MICCAI, SDM, PRICAI, ACML, etc. He is a member of the Machine Learning Society of the Chinese Association of Artificial Intelligence (CAAI), and the Artificial Intelligence & Pattern Recognition Society of the China Computer Federation (CCF).

His research interests include machine learning, pattern recognition, data minining and medical image analysis.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, S., Shao, W., Wang, ML. et al. fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review. Int. J. Autom. Comput. 18, 170–184 (2021). https://doi.org/10.1007/s11633-020-1263-y

Download citation

  • Received: 08 August 2020

  • Accepted: 19 October 2020

  • Published: 16 January 2021

  • Issue Date: April 2021

  • DOI: https://doi.org/10.1007/s11633-020-1263-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Functional magnetic resonance imaging (fMRI)
  • functional alignment
  • brain activity
  • brain decoding
  • visual stimuli reconstruction
Download PDF

Working on a manuscript?

Avoid the common mistakes

Advertisement

Over 10 million scientific documents at your fingertips

Switch Edition
  • Academic Edition
  • Corporate Edition
  • Home
  • Impressum
  • Legal information
  • Privacy statement
  • Your US state privacy rights
  • How we use cookies
  • Your privacy choices/Manage cookies
  • Accessibility
  • FAQ
  • Contact us
  • Affiliate program

Not affiliated

Springer Nature

© 2023 Springer Nature Switzerland AG. Part of Springer Nature.