Skip to main content
SpringerLink
Log in

Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception

  • Original Article
  • Published:
Annals of Biomedical Engineering Aims and scope Submit manuscript

Abstract

The reappearance of human visual perception is a challenging topic in the field of brain decoding. Due to the complexity of visual stimuli and the constraints of fMRI data collection, the present decoding methods can only reconstruct the basic outline or provide similar figures/features of the perceived natural stimuli. To achieve a high-quality and high-resolution reconstruction of natural images from brain activity, this paper presents an end-to-end perception reconstruction model called the similarity-conditions generative adversarial network (SC-GAN), where visually perceptible images are reconstructed based on human visual cortex responses. The SC-GAN extracts the high-level semantic features of natural images and corresponding visual cortical responses and then introduces the semantic features as conditions of generative adversarial networks (GANs) to realize the perceptual reconstruction of visual images. The experimental results show that the semantic features extracted from SC-GAN play a key role in the reconstruction of natural images. The similarity between the presented and reconstructed images obtained by the SC-GAN is significantly higher than that obtained by a condition generative adversarial network (C-GAN). The model we proposed offers a potential perspective for decoding the brain activity of complex natural stimuli.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7

Similar content being viewed by others

References

  1. Button, K. S., J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafò. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14(5):365, 2013.

    Article  CAS  Google Scholar 

  2. Debayle, J., Y. Gavet, and J. C. Pinoli. General adaptive neighborhood image restoration, enhancement and segmentation. In: International Conference Image Analysis and Recognition, Springer, Berlin, 2006, pp. 29–40.

  3. Engel, S. A., G. H. Glover, and B. A. Wandell. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex (New York, NY: 1991) 7(2):181–192, 1997.

    CAS  Google Scholar 

  4. Fang, F., and S. He. Cortical responses to invisible objects in the human dorsal and ventral pathways. Nat. Neurosci. 8(10):1380, 2005.

    Article  CAS  Google Scholar 

  5. Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al. Generative adversarial nets. In: Advances in Neural Information Processing Systems, 2014.

  6. Güçlütürk, Y., U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, and M. A. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems, 2017, pp. 4246–4257.

  7. Horikawa, T., and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8:15037, 2017.

    Article  CAS  Google Scholar 

  8. Huang, W., H. Yan, R. Liu, L. Zhu, H. Zhang, and H. Chen. F-score feature selection based Bayesian reconstruction of visual image from human brain activity. Neurocomputing 316:202–209, 2018.

    Article  Google Scholar 

  9. Kiehl, K. A., K. R. Laurens, T. L. Duty, B. B. Forster, and P. F. Liddle. An event-related fMRI study of visual and auditory oddball tasks. J. Psychophysiol. 15(4):221, 2001.

    Article  Google Scholar 

  10. Ledig, C., L. Theis, F. Huszár, J. Caballero, A. Cunningham, et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4681–4690.

  11. Ma, Y., X. Gu, and Y. Wang. Histogram similarity measure using variable bin size distance. Comput. Vis. Image Underst. 114(8):981–989, 2010.

    Article  Google Scholar 

  12. Nguyen, H. V., and L. Bai. Cosine similarity metric learning for face verification. In: Asian Conference on Computer Vision, Springer, Berlin, 2010, pp. 709–720.

  13. Norman, K. A., S. M. Polyn, G. J. Detre, and J. V. Haxby. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10(9):424–430, 2006.

    Article  Google Scholar 

  14. Pluim, J. P., J. A. Maintz, and M. A. Viergever. Mutual-information-based registration of medical images: a survey. IEEE Trans. Med. Imaging 22(8):986–1004, 2003.

    Article  Google Scholar 

  15. Reed, S., Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. 2016. arXiv preprint arXiv:1605.05396

  16. Seeliger, K., U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M. A. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181:775–785, 2018.

    Article  CAS  Google Scholar 

  17. Shen, G., T. Horikawa, K. Majima, and Y. Kamitani. Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1):e1006633, 2019.

    Article  Google Scholar 

  18. St-Yves, G., and T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2018, pp. 1054–1061.

  19. Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600–612, 2004.

    Article  Google Scholar 

  20. Yi, Z., H. Zhang, P. Tan, and Gong, M. Dualgan. Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2849–2857.

  21. Zhang, H., T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D. Metaxas, and N. Stackgan. Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5907–5915.

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61773094, 61533006, U1808204, 31730039, 31671133 and 61876114), the Ministry of Science and Technology of China (2015CB351701), National Major Scientific Instruments and Equipment Development Project (ZDYZ2015-2) and Chinese Academy of Sciences Strategic Priority Research Program B grants (XDB32010300). The authors declared that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author information

Authors and Affiliations

  1. MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, 610054, People’s Republic of China

    Wei Huang, Hongmei Yan, Chong Wang, Jiyi Li, Zhan Shen & Huafu Chen

  2. State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing, 100101, China

    Zhentao Zuo

  3. University of Chinese Academy of Sciences, Beijing, 100049, China

    Zhentao Zuo

  4. Department of Medical Information Engineering, Sichuan University, Chengdu, 610065, China

    Jiang Zhang

Authors
  1. Wei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongmei Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiyi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhentao Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhan Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Huafu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hongmei Yan, Zhentao Zuo or Huafu Chen.

Additional information

Associate Editor Xiaoxiang Zheng oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (AVI 142,142 kb)

Supplementary material 2 (PDF 1347 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, W., Yan, H., Wang, C. et al. Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception. Ann Biomed Eng 48, 2323–2332 (2020). https://doi.org/10.1007/s10439-020-02502-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10439-020-02502-3

Keywords

Search

Navigation