Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network

  • Siyu Yu
  • Nanning ZhengEmail author
  • Hao Wu
  • Ming Du
  • Badong Chen
Conference paper
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 559)


Brain-inspired computing is a research hotspot in artificial intelligence (AI). One of the key problems in this field is how to find the bridge between brain connectivity and data correlation in a connection-to-cognition model. Functional magnetic resonance imaging (fMRI) signals provide rich information about brain activities. Existing modeling approaches with fMRI focus on the strength information, but neglect structural information. In a previous work, we proposed a monolayer correlation network (CorrNet) to model the structural connectivity. In this paper, we extend the monolayer CorrNet to a hierarchical correlation network (HcorrNet) by analysing visual stimuli of natural images and fMRI signals in the entire visual cortex, that is, V1, V2 V3, V4, fusiform face area (FFA), the lateral occipital complex (LOC) and parahippocampal place area (PPA). Through the HcorrNet, the efficient connectivity of the brain can be inferred layer by layer. Then, the stimulus-sensitive activity mode of voxels can be extracted, and the forward encoding process of visual perception can be modeled. Both of them can guide the decoding process of fMRI signals, including classification and image reconstruction. In the experiments, we improved a dynamic evolving spike neuron network (SNN) as the classifier, and used Generative Adversarial Networks (GANs) to reconstruct image.


Brain-inspired computing Visual perception Functional magnetic resonance imaging (fMRI) Hierarchical correlation network (HcorrNet) Connection 


  1. 1.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223 (2017)Google Scholar
  2. 2.
    Eliasmith, C., et al.: A large-scale model of the functioning brain. Science 338(6111), 1202–1205 (2012)CrossRefGoogle Scholar
  3. 3.
    Fujiwara, Y., Miyawaki, Y., Kamitani, Y.: Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Comput. 25(4), 979–1005 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Gütig, R., Sompolinsky, H.: The tempotron: a neuron that learns spike timing-based decisions. Nature Neurosci. 9(3), 420–428 (2006)CrossRefGoogle Scholar
  5. 5.
    Hausfeld, L., Valente, G., Formisano, E.: Multiclass fMRI data decoding and visualization using supervised self-organizing maps. NeuroImage 96, 54–66 (2014)CrossRefGoogle Scholar
  6. 6.
    Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8, 15037 (2017)CrossRefGoogle Scholar
  7. 7.
    Hu, J., Tang, H., Tan, K.C., Li, H.: How the brain formulates memory: a spatio-temporal model research frontier. IEEE Comput. Intell. Mag. 11(2), 56–68 (2016)CrossRefGoogle Scholar
  8. 8.
    Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L.: Identifying natural images from human brain activity. Nature 452(7185), 352 (2008)CrossRefGoogle Scholar
  9. 9.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)
  10. 10.
    Kuang, D., Guo, X., An, X., Zhao, Y., He, L.: Discrimination of ADHD based on fMRI data with deep belief network. In: Huang, D.-S., Han, K., Gromiha, M. (eds.) ICIC 2014. LNCS, vol. 8590, pp. 225–232. Springer, Cham (2014). Scholar
  11. 11.
    Ma, Y., Wang, Z., Yu, S., Chen, B., Zheng, N., Ren, P.: A novel spiking neural network of receptive field encoding with groups of neurons decision. Front. Inf. Technol. Electron. Eng. 19(1), 139–150 (2018)CrossRefGoogle Scholar
  12. 12.
    Naselaris, T., Olman, C.A., Stansbury, D.E., Ugurbil, K., Gallant, J.L.: A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes. Neuroimage 105, 215–228 (2015)CrossRefGoogle Scholar
  13. 13.
    Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V.: Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10(9), 424–430 (2006)CrossRefGoogle Scholar
  14. 14.
    Park, H.J., Friston, K.: Structural and functional brain networks: from connections to cognition. Science 342(6158), 1238411 (2013)CrossRefGoogle Scholar
  15. 15.
    Penny, W.D., Friston, K.J., Ashburner, J.T., Kiebel, S.J., Nichols, T.E.: Statistical Parametric Mapping: the Analysis of Functional Brain Images. Academic Press, Cambridge (2011)Google Scholar
  16. 16.
    Shen, G., Dwivedi, K., Majima, K., Horikawa, T., Kamitani, Y.: End-to-end deep image reconstruction from human brain activity. BioRxiv p. 272518 (2018)Google Scholar
  17. 17.
    Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), e1006633 (2019)CrossRefGoogle Scholar
  18. 18.
    Wang, W., Arora, R., Livescu, K., Bilmes, J.: On deep multi-view representation learning. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 1083–1092 (2015)Google Scholar
  19. 19.
    Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex 28(12), 4136–4160 (2017)CrossRefGoogle Scholar
  20. 20.
    Yamashita, O., Sato, M., Yoshioka, T., Tong, F., Kamitani, Y.: Sparse estimation automatically selects voxels relevant for the decoding of FMRI activity patterns. NeuroImage 42(4), 1414–1429 (2008)CrossRefGoogle Scholar
  21. 21.
    Yu, S., Zheng, N., Ma, Y., Wu, H., Chen, B.: A novel brain decoding method: a correlation network framework for revealing brain connections. IEEE Trans. Cogn. Dev. Syst. 11, 95–106 (2018)CrossRefGoogle Scholar
  22. 22.
    Zeeman, E.C.: Topology of the brain (1965)Google Scholar
  23. 23.
    Zheng, N., et al.: Hybrid-augmented intelligence: collaboration and cognition. Front. Inf. Technol. Electron. Eng. 18(2), 153–179 (2017)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Siyu Yu
    • 1
    • 2
  • Nanning Zheng
    • 1
    • 2
    Email author
  • Hao Wu
    • 1
    • 2
  • Ming Du
    • 1
    • 2
  • Badong Chen
    • 1
    • 2
  1. 1.Institute of Artificial Intelligence and RoboticsXi’an Jiaotong UniversityXi’anPeople’s Republic of China
  2. 2.National Engineering Laboratory for Visual Information Processing and ApplicationsXi’an Jiaotong UniversityXi’anPeople’s Republic of China

Personalised recommendations