Skip to main content

Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception


The reappearance of human visual perception is a challenging topic in the field of brain decoding. Due to the complexity of visual stimuli and the constraints of fMRI data collection, the present decoding methods can only reconstruct the basic outline or provide similar figures/features of the perceived natural stimuli. To achieve a high-quality and high-resolution reconstruction of natural images from brain activity, this paper presents an end-to-end perception reconstruction model called the similarity-conditions generative adversarial network (SC-GAN), where visually perceptible images are reconstructed based on human visual cortex responses. The SC-GAN extracts the high-level semantic features of natural images and corresponding visual cortical responses and then introduces the semantic features as conditions of generative adversarial networks (GANs) to realize the perceptual reconstruction of visual images. The experimental results show that the semantic features extracted from SC-GAN play a key role in the reconstruction of natural images. The similarity between the presented and reconstructed images obtained by the SC-GAN is significantly higher than that obtained by a condition generative adversarial network (C-GAN). The model we proposed offers a potential perspective for decoding the brain activity of complex natural stimuli.

This is a preview of subscription content, access via your institution.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7


  1. Button, K. S., J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafò. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14(5):365, 2013.

    Article  CAS  Google Scholar 

  2. Debayle, J., Y. Gavet, and J. C. Pinoli. General adaptive neighborhood image restoration, enhancement and segmentation. In: International Conference Image Analysis and Recognition, Springer, Berlin, 2006, pp. 29–40.

  3. Engel, S. A., G. H. Glover, and B. A. Wandell. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex (New York, NY: 1991) 7(2):181–192, 1997.

    CAS  Google Scholar 

  4. Fang, F., and S. He. Cortical responses to invisible objects in the human dorsal and ventral pathways. Nat. Neurosci. 8(10):1380, 2005.

    Article  CAS  Google Scholar 

  5. Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al. Generative adversarial nets. In: Advances in Neural Information Processing Systems, 2014.

  6. Güçlütürk, Y., U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, and M. A. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems, 2017, pp. 4246–4257.

  7. Horikawa, T., and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8:15037, 2017.

    Article  CAS  Google Scholar 

  8. Huang, W., H. Yan, R. Liu, L. Zhu, H. Zhang, and H. Chen. F-score feature selection based Bayesian reconstruction of visual image from human brain activity. Neurocomputing 316:202–209, 2018.

    Article  Google Scholar 

  9. Kiehl, K. A., K. R. Laurens, T. L. Duty, B. B. Forster, and P. F. Liddle. An event-related fMRI study of visual and auditory oddball tasks. J. Psychophysiol. 15(4):221, 2001.

    Article  Google Scholar 

  10. Ledig, C., L. Theis, F. Huszár, J. Caballero, A. Cunningham, et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4681–4690.

  11. Ma, Y., X. Gu, and Y. Wang. Histogram similarity measure using variable bin size distance. Comput. Vis. Image Underst. 114(8):981–989, 2010.

    Article  Google Scholar 

  12. Nguyen, H. V., and L. Bai. Cosine similarity metric learning for face verification. In: Asian Conference on Computer Vision, Springer, Berlin, 2010, pp. 709–720.

  13. Norman, K. A., S. M. Polyn, G. J. Detre, and J. V. Haxby. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10(9):424–430, 2006.

    Article  Google Scholar 

  14. Pluim, J. P., J. A. Maintz, and M. A. Viergever. Mutual-information-based registration of medical images: a survey. IEEE Trans. Med. Imaging 22(8):986–1004, 2003.

    Article  Google Scholar 

  15. Reed, S., Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. 2016. arXiv preprint arXiv:1605.05396

  16. Seeliger, K., U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M. A. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181:775–785, 2018.

    Article  CAS  Google Scholar 

  17. Shen, G., T. Horikawa, K. Majima, and Y. Kamitani. Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1):e1006633, 2019.

    Article  Google Scholar 

  18. St-Yves, G., and T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2018, pp. 1054–1061.

  19. Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600–612, 2004.

    Article  Google Scholar 

  20. Yi, Z., H. Zhang, P. Tan, and Gong, M. Dualgan. Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2849–2857.

  21. Zhang, H., T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D. Metaxas, and N. Stackgan. Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5907–5915.

Download references


This work was supported in part by the National Natural Science Foundation of China (61773094, 61533006, U1808204, 31730039, 31671133 and 61876114), the Ministry of Science and Technology of China (2015CB351701), National Major Scientific Instruments and Equipment Development Project (ZDYZ2015-2) and Chinese Academy of Sciences Strategic Priority Research Program B grants (XDB32010300). The authors declared that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Hongmei Yan, Zhentao Zuo or Huafu Chen.

Additional information

Associate Editor Xiaoxiang Zheng oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (AVI 142,142 kb)

Supplementary material 2 (PDF 1347 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Huang, W., Yan, H., Wang, C. et al. Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception. Ann Biomed Eng 48, 2323–2332 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Visual decoding
  • Reconstruction
  • SC-GAN
  • Deep learning