One of the most significant challenges in the neuroscience community is to understand how the human brain works. Recent progress in neuroimaging techniques have validated that it is possible to decode a person’s thoughts, memories, and emotions via functional magnetic resonance imaging (i.e., fMRI) since it can measure the neural activation of human brains with satisfied spatiotemporal resolutions. However, the unprecedented scale and complexity of the fMRI data have presented critical computational bottlenecks requiring new scientific analytic tools. Given the increasingly important role of machine learning in neuroscience, a great many machine learning algorithms are presented to analyze brain activities from the fMRI data. In this paper, we mainly provide a comprehensive and up-to-date review of machine learning methods for analyzing neural activities with the following three aspects, i.e., brain image functional alignment, brain activity pattern analysis, and visual stimuli reconstruction. In addition, online resources and open research problems on brain pattern analysis are also provided for the convenience of future research.
J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, P. J. Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, vol. 72, no. 2, pp. 404–116, 2011. DOI: https://doi.org/10.1016/j.neuron.2011.08.026.
M. B. Cai, N. W. Schuck, J. W. Pillow, Y. Niv. A Bayesian method for reducing bias in neural representational similarity analysis. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 4958–4966, 2016.
J. H. Tao, J. Huang, Y. Li, Z. Lian, M. Y. Niu. Semi-supervised ladder networks for speech emotion recognition. International Journal of Automation and Computing, vol. 16, no. 4, pp. 437–448, 2019. DOI: https://doi.org/10.1007/s11633-019-1175-x.
A. M. Michael, M. Anderson, R. L. Miller, T. Adali, V. D. Calhoun. Preserving subject variability in group fMRI analysis: Performance evaluation of GICA vs. IVA. Frontiers in Systems Neuroscience, vol. 8, Article number 106, 2014. DOI: https://doi.org/10.3389/fnsys.2014.00106.
Z. F. Wen, T. Y. Yu, Z. L. Yu, Y. Q. Li. Grouped sparse Bayesian learning for voxel selection in multivoxel pattern analysis of fMRI data. NeuroImage, vol. 184, pp. 417–430, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2018.09.031.
D. Haputhanthri, G. Brihadiswaran, S. Gunathilaka, D. Meedeniya, S. Jayarathna, M. Jaime, C. Harshaw. Integration of facial thermography in EEG-based classification of ASD. International Journal of Automation and Computing, to be published. DOI: https://doi.org/10.1007/s11633-020-1231-6.
J. V. Haxby. Multivariate pattern analysis of fMRI: The early beginnings. NeuroImage, vol. 62, no. 2, pp. 852–855, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2012.03.016.
C. D. Du, C. Y. Du, L. J. Huang, H. G. He. Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 8, pp. 2310–2323, 2019. DOI: https://doi.org/10.1109/TNNLS.2018.2882456.
A. Lorbert, P. J. Ramadge. Kernel hyperalignment. In Proceedings of the 25th International Conference on Neural Information Processing Systems Lake Tahoe, USA, pp. 1790–1798, 2012.
Y. Zhan, J. C. Zhang, S. T. Song, L. Yao. Visual image reconstruction from fMRI activation using multi-scale support vector machine decoders. In Proceedings of the 15th International Conference on Human-Computer Interaction, Springer, Las Vegas, USA, pp. 491–497, 2013. DOI: https://doi.org/10.1007/978-3-642-39342-6_54.
Y. Kamitani, F. Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, vol. 8, no. 5, pp. 679–685, 2005. DOI: https://doi.org/10.1038/nn1444.
K. N. Kay, T. Naselaris, R. J. Prenger, J. L. Gallant. Identifying natural images from human brain activity. Nature, vol. 452, no. 7185, pp. 352–355, 2008. DOI: https://doi.org/10.1038/nature06713.
F. De Martino, G. Valente, N. Staeren, J. Ashburner, R. Goebel, E. Formisano. Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns. NeuroImage, vol. 43, no. 1, pp. 44–58, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.06.037.
O. Yamashita, M. A. Sato, T. Yoshioka, F. Tong, Y. Kamitani. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage, vol. 42, no. 4, pp. 1414–1429, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2008.05.050.
W. D. Li, M. X. Liu, F. Chen, D. Q. Zhang. Graph-based decoding model for functional alignment of unaligned fMRI data. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, USA, pp. 2653–2660, 2020.
M. Yousefnezhad, D. Q. Zhang. Local discriminant hyper-alignment for multi-subject fMRI data alignment. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, California, USA, pp. 59–65, 2017. DOI
M. Yousefnezhad, D. Q. Zhang. Deep hyperalignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 1603–1611, 2017.
J. S. Guntupalli, M. Hanke, Y. O. Halchenko, A. C. Connolly, P. J. Ramadge, J. V. Haxby. A model of representational spaces in human cortex. Cerebral Cortex, vol. 26, pp. 2919–2934, 2016. DOI: https://doi.org/10.1093/cercor/bhw068.
P. H. Chen. Multi-view Representation Learning with Applications to Functional Neuroimaging Data, Princeton University, Ph. D. dissertation, USA, 2017.
G. H. Shen, K. Dwivedi, K. Majima, T. Horikawa, Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, vol. 13, Article number 21, 2019. DOI: https://doi.org/10.3389/fncom.2019.00021.
T. Naselaris, K. N. Kay, S. Nishimoto, J. L. Gallant. Encoding and decoding in fMRI. NeuroImage, vol. 56, no. 2, pp. 400–410, 2011. DOI: https://doi.org/10.1016/j.neuroimage.2010.07.073.
C. D. Du, J. P. Li, L. J. Huang, H. G. He. Brain encoding and decoding in fMRI with bidirectional deep generative models. Engineering, vol. 5, no. 5, pp. 948–953, 2019. DOI: https://doi.org/10.1016/j.eng.2019.03.010.
K. Vakamudi, S. Posse, R. Jung, B. Cushnyr, M. O. Chohan. Real-time presurgical resting-state fMRI in patients with brain tumors: Quality control and comparison with task-fMRI and intraoperative mapping. Human Brain Mapping, vol. 41, pp. 797–814, 2020. DOI: https://doi.org/10.1002/hbm.24840.
J. Talairach P. Tournoux. 3-dimensional proportional system: An approach to cerebral imaging. Co-Planar Stereotaxic Atlas of the Human Brain. Thieme, 1988.
A. C. Evans, D. L. Collins, S. R. Mills, E. D. Brown, R. L. Kelly, T. M. Peters. 3D statistical neuroanatomical models from 305 MRI volumes. In Proceedings of IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference, IEEE, San Francisco, USA, pp. 1813–1817, 1993. DOI: https://doi.org/10.1109/NSSMIC.1993.373602.
W. Chau, A. R. McIntosh. The Talairach coordinate of a point in the MNI space: How to interpret it. NeuroImage, vol. 25, no. 2, pp. 408–416, 2005. DOI: https://doi.org/10.1016/j.neuroimage.2004.12.007.
B. R. Conroy, B. D. Singer, J. V. Haxby, P. J. Ramadge. fMRI-based inter-subject cortical alignment using functional connectivity. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 378–386, 2009.
M. R. Sabuncu, B. D. Singer, B. Conroy, R. E. Bryan, P. J. Ramadge, J. V. Haxby. Function-based intersubject alignment of human cortical anatomy. Cerebral Cortex, vol. 20, no. 1, pp. 130–140, 2009. DOI: https://doi.org/10.1093/cercor/bhp085.
J. P. Dmochowski, P. Sajda, J. Dias, L. C. Parra. Correlated components of ongoing EEG point to emotionally laden attention-a possible marker of engagement? Frontiers in Human Neuroscience, vol. 6, Article number 112, 2012. DOI: https://doi.org/10.3389/fnhum.2012.00112.
P. H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, vol. 31, no. 1, pp. 1–10, 1966. DOI: https://doi.org/10.1007/BF02289451.
H. Xu, A. Lorbert, P. J. Ramadge, J. S. Guntupalli, J. V. Haxby. Regularized hyperalignment of multi-set fMRI data. In Proceedings of IEEE Statistical Signal Processing Workshop, IEEE, Ann Arbor, USA, pp. 229–232, 2012. DOI: https://doi.org/10.1109/SSP.2012.6319668.
P. H. Chen, J. S. Guntupalli, J. V. Haxby, P. J. Ramadge. Joint SVD-Hyperalignment for multi-subject FMRI data alignment. In Proceedings of IEEE International Workshop on Machine Learning for Signal Processing, IEEE, Reims, France, 2014. DOI: https://doi.org/10.1109/MLSP.2014.6958912.
P. H. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. V. Haxby, P. J. Ramadge. A reduced-dimension fMRI shared response model. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 460–468, 2015.
J. Sui, G. Pearlson, A. Caprihan, T. Adali, K. A. Kiehl, J. Y. Liu, J. Yamamoto, V. D. Calhoun. Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. NeuroImage, vol. 57, no. 3, pp. 839–855, 0011. DOI: https://doi.org/10.1016/j.neuroimage.2011.05.055.
J. Sui, H. He, G. D. Pearlson, T. Adali, K. A. Kiehl, Q. B. Yu, V. P. Clark, E. Castro, T. White, B. A. Mueller, B. C. Ho, N. C. Andreasen, V. D. Calhoun. Three-way (N-way) fusion of brain imaging data based on mCCA+ jICA and its application to discriminating schizophrenia. NeuroImage, vol. 66, pp. 119–132, 2013. DOI: https://doi.org/10.1016/j.neuroimage.2012.10.051.
P. H. Chen, X. Zhu, H. J. Zhang, J. S. Turek, J. Chen, T. L. Willke, U. Hasson, P. J. Ramadge. A convolutional autoencoder for multi-subject fMRI data aggregation. [Online], Available: https://arxiv.org/abs/1608.04846, 2016.
U. Hasson, O. Landesman, B. Knappmeyer, I. Vallines, N. Rubin, D. J. Heeger. Neurocinematics: The neuroscience of film. Projections, vol. 2, no. 1, pp. 1–26, 2008. DOI: https://doi.org/10.3167/proj.2008.020102.
J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, P. Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, vol. 293, no. 5539, pp. 2425–2430, 2001. DOI: https://doi.org/10.1126/science.1063736.
J. V. Haxby, A. C. Connolly, J. S. Guntupalli. Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, vol. 37, pp. 435–456, 2014. DOI: https://doi.org/10.1146/annurev-neuro-062012-170325.
C. Allfeld, J. D. Haynes. Mutt-voxel pattern analysis. Brain Mapping, vol. 1, pp. 641–646, 2015. DOI: https://doi.org/10.1016/B978-0-12-397025-1.00345-6.
M. Yousefnezhad, D. Q. Zhang. Anatomical pattern analysis for decoding visual stimuli in human brains. Cognitive Computation, vol. 10, no. 2, pp. 284–295, 2018. DOI: https://doi.org/10.1007/s12559-017-9518-9.
D. D. Wagner, R. S. Chavez, T. W. Broom. Decoding the neural representation of self and person knowledge with multivariate pattern analysis and data-driven approaches. Wiley Interdisciplinary Reviews: Cognitive Science, vol. 10, no. 1, Article number e1482, 2019. DOI: https://doi.org/10.1002/wcs.1482.
A. C. Connolly, J. S. Guntupalli, J. Gors, M. Hanke, Y. O. Halchenko, Y. C. Wu, H. Abdi, J. V. Haxby. The representation of biological classes in the human brain. Journal of Neuroscience, vol. 32, no. 8, pp. 2608–2618, 2012. DOI: https://doi.org/10.1523/JNEUROSCI.5547-11.2012.
S. Ryali, K. Supekar, D. A. Abrams, V. Menon. Sparse logistic regression for whole-brain classification of fMRI data. NeuroImage, vol. 51, no. 2, pp. 752–764, 2010. DOI: https://doi.org/10.1016/j.neuroimage.2010.02.040.
L. Grosenick, B. Klingenberg, S. Greer, J. Taylor, B. Knutson. Whole-brain sparse penalized discriminant analysis for predicting choice. NeuroImage, vol. 47, no. S1, Article number S58, 2009. DOI: https://doi.org/10.1016/S1053-8119(09)70232-0.
C. van Meel, A. Baeck, C. R. Gillebert, J. Wagemans, H. P. O. de Beeck. The representation of symmetry in multi-voxel response patterns and functional connectivity throughout the ventral visual stream. NeuroImage, vol. 191, pp. 216–224, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2019.02.030.
N. Kriegeskorte, R. Goebel, P. Bandettini. Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 10, pp. 3863–3868, 2006. DOI https://doi.org/10.1073/pnas.0600244103.
N. Kriegeskorte, M. Mur, P. Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, vol. 2, Article number 4, 2008. DOI: https://doi.org/10.3389/neuro.06.004.2008.
M. Yuan, Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 68, no. 1, pp. 49–67, 2006. DOI: https://doi.org/10.1111/j.1467-9868.2005.00532.x.
A. E. Hoerl, R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, vol. 42, no. 1, pp. 80–86, 2000. DOI: https://doi.org/10.1080/00401706.2000.10485983.
H. Zou, T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 67, no. 2, pp. 301–320, 2005. DOI: https://doi.org/10.1111/j.1467-9868.2005.00503.x.
A. G. Huth, S. Nishimoto, A. T. Vu, J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, vol. 76, no. 6, pp. 1210–1224, 2012. DOI: https://doi.org/10.1016/j.neuron.2012.10.014.
L. Su, E. Fonteneau, W. Marslen-Wilson, N. Kriegeskorte. Spatiotemporal searchlight representational similarity analysis in EMEG source space. In Proceedings of the 2nd International Workshop on Pattern Recognition in NeuroImaging, IEEE, London, UK, pp. 97–100, 2012. DOI: https://doi.org/10.1109/PRNI.2012.26.
E. A. Wasserman, A. Chakroff, R. Saxe, L. Young. Illuminating the conceptual structure of the space of moral violations with searchlight representational similarity analysis. NeuroImage, vol. 159, pp. 371–387, 2017. DOI: https://doi.org/10.1016/j.neuroimage.2017.07.043
M. V. Peelen, A. Caramazza Conceptual object representations in human anterior temporal cortex. Journal of Neuroscience, vol. 32, no. 45, pp. 15728–15736, 2012 DOI:https://doi.org/10.1523/JNEUROSCI.1953-12
D. J. Kravitz, C. S. Peng, C. I. Baker Real-world scene representations in high-level visual cortex: It’s the spaces more than the places Journal of Neuroscience, vol. 31, no. 20, pp. 7322–7333, 2011 DOI: https://doi.org/10.1523/JNEUROSCI.4588-102011
G. Handjaras, E. Ricciardi, A. Leo, A. Lenci, L. Cecchetti, M. Cosottini, G. Marotta, P. Pietrini How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge NeuroImage, vol. 135, pp. 232–242, 2016 DOI: https://doi.org/10.1016/j.neuroimage.2016.04.063
D. I. Tamir, M. A. Thornton, J. M. Contreras, J. P. Mitchell Neural evidence that three dimensions organize mental state representation: Rationality, social impact, and valence Proceedings of the National Academy of Sciences of the United States of America, vol. 113, no. 1, pp. 194–199, 2016 DOI: https://doi.org/10.1073/pnas.1511905112.
R. S. Chavez, T. F. Heatherton Representational similarity of social and valence information in the medial pFC Journal of Cognitive Neuroscience, vol. 27, no. 1, pp. 73–82, 2015. DOI: https://doi.org/10.1162/jocn_a_00697.
B. Thirion, E. Duchesnay, E Hubbard, J. Dubois, J. B. Poline, D. Lebihan, S. Dehaene. Inverse retinotopy: Inferring the visual content of images from brain activation patterns. NeuroImage, vol. 33, no. 4, pp. 1104–1116, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2006.06.062.
Y. Miyawaki, H. Uchida, O. Yamashita, M. A. Sato, Y. Morito, H. C. Tanabe, N. Sadato, Y. Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, vol. 60, no. 5, pp. 915–929, 2008. DOI: https://doi.org/10.1016/j.neuron.2008.11.004.
T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, J. L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, vol. 63, no. 6, pp. 902–915, 2009. DOI: https://doi.org/10.1016/j.neuron.2009.09.006.
S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, J. L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, vol. 21, no. 19, pp. 1641–1646, 2011. DOI: https://doi.org/10.1016/j.cub.2011.08.031.
Y. Fujiwara, Y. Miyawaki, Y. Kamitani. Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Computation, vol. 25, no. 4, pp. 979–1005, 2013. DOI: https://doi.org/10.1162/NECO_a_00423.
A. S. Cowen, M. M. Chun, B. A. Kuhl. Neural portraits of perception: Reconstructing face images from evoked brain activity. NeuroImage, vol. 94, pp. 12–22, 2014. DOI: https://doi.org/10.1016/j.neuroimage.2014.03.018.
C. D. Du, C. Y. Du, H. G. He. Sharing deep generative representation for perceived image reconstruction from human brain activity. In Proceedings of International Joint Conference on Neural Networks, IEEE, Anchorage, USA, pp. 1049–1056, 2017. DOI: https://doi.org/10.1109/IJCNN.2017.7965968.
D. P. Kingma, M. Welling. Auto-encoding variational Bayes. [Onliee], Aaailbble: https://arxiv.org/abs/1312.6114, 2014.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680, 2014.
T. Horikawa, Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, vol. 8, Article number 15037, 2017. DOI: https://doi.org/10.1038/ncomms15037.
G. St-Yves, T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, IEEE, Miyazaki, Japan, pp. 1054–1061, 2018. DOI: https://doi.org/10.1109/SMC.2018.00187.
Y. Güçlütürk, U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, M. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 4249–4260, 2017.
K. Seeliger, U. Güçlü, L. Ambrogioni, Y Güçlütürk, M. A. J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, vol. 181, pp. 775–785, 2018. DOI: https://doi.org/10.1016/j.neuroimage.2018.07.043.
R. VanRullen, L. Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology, vol. 2, no. 1, Article number 193, 2019. DOI: https://doi.org/10.1038/s42003-019-0438-y.
R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, M. Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019.
D. Li, C. D. Du, H. G. He. Semi-supervised cross-modal image generation with generative adversarial networks. Pattern Recognition, vol. 100, Article number 107085, 2020. DOI: https://doi.org/10.1016/j.patcog.2019.107085.
C. D. Du, C. Y. Du, H. Wang, J. P. Li, W. L. Zheng, B. L. Lu, H. G. He. Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In Proceedings of the 26th ACM international conference on Multimedia, ACM, Seoul, Republic of Korea, pp. 108–116, 2018. DOI: https://doi.org/10.1145/3240508.3240528.
X. Cai, F. P. Nie, W. D. Cai, H. Huang. Heterogeneous image features integration via multi-modal semi-supervised learning model. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, 2013. DOI: https://doi.org/10.1109/ICCV.2013.218.
Z. X. Zhang, F. Ringeval, B. Dong, E. Coutinho, B. Schuller. Enhanced semi-supervised learning for multimodal emotion recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Shanghai, China, 2016. DOI: https://doi.org/10.1109/ICASSP.2016.7472666.
T. Schonberg, C. R. Fox, J. A. Mumford, E. Congdon, C. Trepel, R. A. Poldrack. Decreasing ventromedial prefrontal cortex activity during sequential risk-taking: An fMRI investigation of the balloon analog risk task. Frontiers in Neuroscience, vol. 6, Article number 80, 2012. DOI: https://doi.org/10.3389/fnins.2012.00080.
A. R. Aron, M. A. Gluck, R. A. Poldrack. Long-term test-retest reliability of functional MRI in a classification learning task. NeuroImage, vol. 29, no. 3, pp. 1000–1006, 2006. DOI: https://doi.org/10.1016/j.neuroimage.2005.08.010.
S. M. Tom, C. R. Fox, C. Trepel, R. A. Poldrack. The neural basis of loss aversion in decision-making under risk. Science, vol. 315, no. 5811, pp. 515–518, 2007. DOI: https://doi.org/10.1126/science.1134239.
K. Foerde, B. J. Knowlton, and R. A. Poldrack. Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences of the United States of America, vol. 103, no. 31, pp. 11778–11783, 2006. DOI: https://doi.org/10.1073/pnas.0602659103.
R. A. Poldrack, J. Clark, E. J. Paré-Blagoev, D. Shohamy, J. C. Moyano, C. Myers, M. A. Gluck. Interactive memory systems in the human brain. Nature, vol. 414, no. 6863, pp. 546–550, 2001. DOI: https://doi.org/10.1038/35107080.
A. M. C. Kelly, L. Q. Uddin, B. B. Biswal, F. X. Castellanos, M. P. Milham. Competition between functional brain networks mediates behavioral variability. NeuroImage, vol. 39, no. 1, pp. 527–537, 2008. DOI: https://doi.org/10.1016/j.neuroimage.2007.08.008.
K. J. Duncan, C. Pattamadilok, I. Knierim, J. T. Devlin. Consistency and variability in unctional localisers. NeuroImage, vol. 46, no. 4, pp. 1018–1026, 2009. DOI: https://doi.org/10.1016/j.neuroimage.2009.03.014.
J. M. Walz, R. I. Goldman, M. Carapezza, J. S. Muraskin, T. R. Brown, P. Sajda. Simultaneous EEG-MRI reveals temporal evolution of coupling between supramodal cortical attention networks and the brainstem. Journal of Neuroscience, vol. 33, no. 49, pp. 19212–19222, 2013. DOI: https://doi.org/10.1523/JNEUROSCI.2649-13.2013.
T. D. Verstynen. The organization and dynamics of corticostriatal pathways link the medial orbitofrontal cortex to uture behavioral responses. Journal of Neurophysiology, vol. 112, no. 10, pp. 2457–2469, 2014. DOI: https://doi.org/10.1152/jn.00221.2014.
M. G. Veldhuizen, R. K. Babbs, B. Patel, W. Fobbs, N. B. Kroemer, E. Garcia, M. R. Yeomans, D. M. Small. Integration of sweet taste and metabolism determines carbohydrate reward. Current Biology, vol. 27, no. 16, pp. 2476–2485.e6, 2017. DOI: https://doi.org/10.1016/j.cub.2017.07.018.
M. Hanke, Y. O. Halchenko, P. B. Sederberg, S. J. Hanson, J. V. Haxby, S. Pollmann. PyMVPA: A python toolbox or multivariate pattern analysis of MRI data. Neuroinformatics, vol. 7, no. 1, pp. 37–53, 2009. DOI: https://doi.org/10.1007/s12021-008-9041-y.
M. Hanke, Y. O. Halchenko, P. B. Sederberg, E. Olivetti, I. Fründ, J. W. Rieger, C. S. Herrmann, J. V. Haxby, S. JoséHanson, S. Pollmann. PyMVPA: A unifying approach to the analysis of neuroscientific data. Frontiers in Neuroinformatics, vol. 3, Article number 3, 2009. DOI: https://doi.org/10.3389/neuro.11.003.2009.
H. J. Zhang, P. H. Chen, P. Ramadge. Transfer learning on fMRI datasets. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, Playa Blanca, Spain, pp. 595–603, 2018.
M. L. Wang, D. Q. Zhang, J. S. Huang, P. T. Yap, D. G. Shen, M. X. Liu. Identifying autism spectrum disorder with multi-site fMRI via low-rank domain adaptation. IEEE Transactions on Medical Imaging, vol. 39, no. 3, pp. 644–655, 2020. DOI: https://doi.org/10.1109/TMI.2019.2933160.
X. Zhang, Q. Yang. Transfer hierarchical attention network for generative dialog system. International Journal of Automation and Computing, vol. 16, no. 6, pp. 720–736, 2019. DOI: https://doi.org/10.1007/s11633-019-1200-0.
This work was supported by National Natural Science Foundation of China (Nos. 61876082, 61861130366, 6173-2006 and 61902183), National Key Research and Development Program of China (Nos. 2018 YFC2001600, 2018YFC 2001602), the Royal Society-Academy of Medical Sciences Newton Advanced Fellowship (No. NAFR1-0371), and China Postdoctoral Science Foundation funded project (No. 2019M661831).
Colored figures are available in the online version at https://link.springer.com/journal/11633
Shuo Huang received the B. Sc. degree in software engineering from Northeastern University, China in 2015. He is currently a Ph. D. degree candidate in software engineering in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), China.
His research interests include machine learning and human brain decoding.
Wei Shao received the B. Sc. and M. Sc. degrees in information and computing science from Nanjing University of Technology, China in 2009 and 2012, respectively, and the Ph. D. degree in software engineering from Nanjing University of Aeronautics and Astronautics, China in 2018.
His research interests include machine learning and bioinformatics.
Mei-Ling Wang received the M. Sc. degree in information and communication engineering from Nanjing University of Information Science and Technology, China in 2016. She is currently a Ph. D. degree candidate in College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China.
Her research interests include machine learning and brain imaging genetics.
Dao-Qiang Zhang received the B. Sc. and Ph. D. degrees in computer science from Nanjing University of Aeronautics and Astronautics, China in 1999, and 2004, respectively. He joined Department of Computer Science and Engineering of NUAA as a lecturer in 2004, and is a professor at present. He has published over 200 scientific articles in refereed international journals such as IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Medical Imaging, IEEE Transactions on Image Processing, Neuroimage, Human Brain Mapping, Medical Image Analysis; and conference proceedings such as IJCAI, AAAI, NIPS, CVPR, MICCAI, KDD, with 12 000+ citations by Google Scholar. He was nominated for the National Excellent Doctoral Dissertation Award of China in 2006, won the Best Paper Award and the Best Student Award of several international conferences such as PRICAI’06, STMI’12 and BICS’16, etc. He has served as a program committee member for several international conferences such as IJCAI, AAAI, NIPS, MICCAI, SDM, PRICAI, ACML, etc. He is a member of the Machine Learning Society of the Chinese Association of Artificial Intelligence (CAAI), and the Artificial Intelligence & Pattern Recognition Society of the China Computer Federation (CCF).
His research interests include machine learning, pattern recognition, data minining and medical image analysis.
About this article
Cite this article
Huang, S., Shao, W., Wang, ML. et al. fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review. Int. J. Autom. Comput. 18, 170–184 (2021). https://doi.org/10.1007/s11633-020-1263-y
- Functional magnetic resonance imaging (fMRI)
- functional alignment
- brain activity
- brain decoding
- visual stimuli reconstruction