The reappearance of human visual perception is a challenging topic in the field of brain decoding. Due to the complexity of visual stimuli and the constraints of fMRI data collection, the present decoding methods can only reconstruct the basic outline or provide similar figures/features of the perceived natural stimuli. To achieve a high-quality and high-resolution reconstruction of natural images from brain activity, this paper presents an end-to-end perception reconstruction model called the similarity-conditions generative adversarial network (SC-GAN), where visually perceptible images are reconstructed based on human visual cortex responses. The SC-GAN extracts the high-level semantic features of natural images and corresponding visual cortical responses and then introduces the semantic features as conditions of generative adversarial networks (GANs) to realize the perceptual reconstruction of visual images. The experimental results show that the semantic features extracted from SC-GAN play a key role in the reconstruction of natural images. The similarity between the presented and reconstructed images obtained by the SC-GAN is significantly higher than that obtained by a condition generative adversarial network (C-GAN). The model we proposed offers a potential perspective for decoding the brain activity of complex natural stimuli.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Button, K. S., J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafò. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14(5):365, 2013.
Debayle, J., Y. Gavet, and J. C. Pinoli. General adaptive neighborhood image restoration, enhancement and segmentation. In: International Conference Image Analysis and Recognition, Springer, Berlin, 2006, pp. 29–40.
Engel, S. A., G. H. Glover, and B. A. Wandell. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb. Cortex (New York, NY: 1991) 7(2):181–192, 1997.
Fang, F., and S. He. Cortical responses to invisible objects in the human dorsal and ventral pathways. Nat. Neurosci. 8(10):1380, 2005.
Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al. Generative adversarial nets. In: Advances in Neural Information Processing Systems, 2014.
Güçlütürk, Y., U. Güçlü, K. Seeliger, S. Bosch, R. van Lier, and M. A. van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems, 2017, pp. 4246–4257.
Horikawa, T., and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8:15037, 2017.
Huang, W., H. Yan, R. Liu, L. Zhu, H. Zhang, and H. Chen. F-score feature selection based Bayesian reconstruction of visual image from human brain activity. Neurocomputing 316:202–209, 2018.
Kiehl, K. A., K. R. Laurens, T. L. Duty, B. B. Forster, and P. F. Liddle. An event-related fMRI study of visual and auditory oddball tasks. J. Psychophysiol. 15(4):221, 2001.
Ledig, C., L. Theis, F. Huszár, J. Caballero, A. Cunningham, et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4681–4690.
Ma, Y., X. Gu, and Y. Wang. Histogram similarity measure using variable bin size distance. Comput. Vis. Image Underst. 114(8):981–989, 2010.
Nguyen, H. V., and L. Bai. Cosine similarity metric learning for face verification. In: Asian Conference on Computer Vision, Springer, Berlin, 2010, pp. 709–720.
Norman, K. A., S. M. Polyn, G. J. Detre, and J. V. Haxby. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10(9):424–430, 2006.
Pluim, J. P., J. A. Maintz, and M. A. Viergever. Mutual-information-based registration of medical images: a survey. IEEE Trans. Med. Imaging 22(8):986–1004, 2003.
Reed, S., Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. 2016. arXiv preprint arXiv:1605.05396
Seeliger, K., U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M. A. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181:775–785, 2018.
Shen, G., T. Horikawa, K. Majima, and Y. Kamitani. Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1):e1006633, 2019.
St-Yves, G., and T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2018, pp. 1054–1061.
Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600–612, 2004.
Yi, Z., H. Zhang, P. Tan, and Gong, M. Dualgan. Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2849–2857.
Zhang, H., T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D. Metaxas, and N. Stackgan. Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5907–5915.
This work was supported in part by the National Natural Science Foundation of China (61773094, 61533006, U1808204, 31730039, 31671133 and 61876114), the Ministry of Science and Technology of China (2015CB351701), National Major Scientific Instruments and Equipment Development Project (ZDYZ2015-2) and Chinese Academy of Sciences Strategic Priority Research Program B grants (XDB32010300). The authors declared that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Associate Editor Xiaoxiang Zheng oversaw the review of this article.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (AVI 142,142 kb)
Rights and permissions
About this article
Cite this article
Huang, W., Yan, H., Wang, C. et al. Perception-to-Image: Reconstructing Natural Images from the Brain Activity of Visual Perception. Ann Biomed Eng 48, 2323–2332 (2020). https://doi.org/10.1007/s10439-020-02502-3
- Visual decoding
- Deep learning