Advertisement

Current Advances in Neural Decoding

  • Marcel A. J. van GervenEmail author
  • Katja Seeliger
  • Umut Güçlü
  • Yağmur Güçlütürk
Chapter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11700)

Abstract

Neural decoding refers to the extraction of semantically meaningful information from brain activity patterns. We discuss how advances in machine learning drive new advances in neural decoding. While linear methods allow for the reconstruction of basic stimuli from brain activity, more sophisticated nonlinear methods are required when reconstructing complex naturalistic stimuli. We show how deep neural networks and adversarial training yield state-of-the-art results. Ongoing advances in machine learning may one day allow the reconstruction of thoughts from brain activity patterns, providing a unique insight into the contents of the human mind.

Keywords

Neural decoding Visual Perception Functional Magnetic Resonance Imaging Deep Neural Networks Adversarial training 

References

  1. 1.
    Akbari, H., Khalighinejad, B., Herrero, J.L., Mehta, A.D., Mesgarani, N.: Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep. 9(1), 874 (2019)Google Scholar
  2. 2.
    Anumanchipalli, G.K., Chartier, J., Chang, E.F.: Speech synthesis from neural decoding of spoken sentences. Nature 568, 493–501 (2019)Google Scholar
  3. 3.
    Bahramisharif, A., van Gerven, M.A.J., Heskes, T., Jensen, O.: Covert attention allows for continuous control of brain-computer interfaces. Eur. J. Neurosci. 31(8), 1501–1508 (2010)Google Scholar
  4. 4.
    Bialek, W., Rieke, F., van Steveninck, R.R.D.R., Warland, D.: Reading a neural code. Science 252(5014), 1854–1857 (1991)Google Scholar
  5. 5.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  6. 6.
    Chang, L., Tsao, D.Y.: The code for facial identity in the primate brain. Cell 169(6), 1013–1028 (2017)Google Scholar
  7. 7.
    Cowen, A.S., Chun, M.M., Kuhl, B.A.: Neural portraits of perception: reconstructing face images from evoked brain activity. NeuroImage 94, 12–22 (2014)Google Scholar
  8. 8.
    Cox, D.D., Dean, T.: Neural networks and neuroscience-inspired computer vision. Curr. Biol. 24(18), PR921–R929 (2014)Google Scholar
  9. 9.
    Daugman, J.G.: Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J. Opt. Soc. Am. A: 2(7), 1160–1169 (1985)Google Scholar
  10. 10.
    Dayan, P., Abbott, L.F.: Theoretical Neuroscience. MIT Press, Cambridge (2005)zbMATHGoogle Scholar
  11. 11.
    Dijkstra, N., Bosch, S.E., van Gerven, M.A.J.: Vividness of visual imagery depends on the neural overlap with perception in visual areas. J. Neurosci. 37(5), 1367–1373 (2017)Google Scholar
  12. 12.
    Dijkstra, N., Mostert, P., de Lange, F.P., Bosch, S.E., van Gerven, M.A.J.: Differential temporal dynamics during visual imagery and perception. eLIFE, pp. 1–16 (2018)Google Scholar
  13. 13.
    Dijkstra, N., Zeidman, P., Ondobaka, S., van Gerven, M.A.J., Friston, K.: Distinct top-down and bottom-up brain connectivity during visual perception and imagery. Sci. Rep. 7(5677), 1–9 (2017)Google Scholar
  14. 14.
    Domingos, P.: Why does bagging work? a Bayesian account and its implications. In: Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, pp. 155–158 (1997)Google Scholar
  15. 15.
    Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Univ. Montreal 1341, 1–13 (2009)Google Scholar
  16. 16.
    Fukushima, K.: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193–202 (1980)zbMATHGoogle Scholar
  17. 17.
    van Gerven, M.A.J.: A primer on encoding models in sensory neuroscience. J. Math. Psychol. 76(B), 172–183 (2017)Google Scholar
  18. 18.
    van Gerven, M.A.J., Chao, Z.C., Heskes, T.: On the decoding of intracranial data using sparse orthonormalized partial least squares. J. Neural Eng. 9(2), 026017 (2012)Google Scholar
  19. 19.
    van Gerven, M.A.J., Kok, P., de Lange, F.P., Heskes, T.: Dynamic decoding of ongoing perception. NeuroImage 57, 950–957 (2011)Google Scholar
  20. 20.
    van Gerven, M.A.J., de Lange, F.P., Heskes, T.: Neural decoding with hierarchical generative models. Neural Comput. 22(12), 3127–3142 (2010)zbMATHGoogle Scholar
  21. 21.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NeurIPS) 2014, pp. 2672–2680 (2014)Google Scholar
  22. 22.
    Güçlü, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35(27), 10005–10014 (2015)Google Scholar
  23. 23.
    Güçlü, U., van Gerven, M.A.J.: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage 145, 329–336 (2017)Google Scholar
  24. 24.
    Güçlü, U., Thielen, J., Hanke, M., van Gerven, M.A.J.: Brains on beats. In: Advances in Neural Information Processing Systems (NeurIPS) 2016, pp. 1–12 (2016)Google Scholar
  25. 25.
    Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S.E., van Lier, R., van Gerven, M.A.J.: Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems (NeurIPS) 2017 (2017)Google Scholar
  26. 26.
    Hastie, T., Tibshirani, R.J., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2008).  https://doi.org/10.1007/978-0-387-84858-7CrossRefzbMATHGoogle Scholar
  27. 27.
    Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001)Google Scholar
  28. 28.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Horikawa, T., Kamitani, Y.: Hierarchical neural representation of dreamed objects revealed by brain decoding with deep neural network features. Front. Comput. Neurosci. 11, 1–11 (2017)Google Scholar
  30. 30.
    Horikawa, T., Tamaki, M., Miyawaki, Y., Kamitani, Y.: Neural decoding of visual imagery during sleep. Science 340(6132), 639–642 (2013)Google Scholar
  31. 31.
    Ienca, M., Haselager, P., Emanuel, E.J.: Brain leaks and consumer neurotechnology. Nat. Biotechnol. 36(9), 805–810 (2018)Google Scholar
  32. 32.
    Jones, J.P., Palmer, L.A.: An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. J. Neurophysiol. 58, 1233–1258 (1987)Google Scholar
  33. 33.
    Kamitani, Y., Tong, F.: Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8(5), 679–685 (2005)Google Scholar
  34. 34.
    Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L.: Identifying natural images from human brain activity. Nature 452, 352–355 (2008)Google Scholar
  35. 35.
    LeCun, Y., Bengio, Y., Hinton, G.E.: Deep learning. Nature 521(7553), 436 (2015)Google Scholar
  36. 36.
    Marčelja, S.: Mathematical description of the responses of simple cortical cells. J. Opt. Soc. Am. A: 70(11), 1297–1300 (1980)MathSciNetGoogle Scholar
  37. 37.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: International Conference on Learning Representations (ICLR) 2013. Cornell University Library (2013)Google Scholar
  38. 38.
    Miyawaki, Y., et al.: Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60(5), 915–929 (2008)Google Scholar
  39. 39.
    Mordvintsev, A., Olah, C., Tyka, M.: Inceptionism: going deeper into neural networks (2005). https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
  40. 40.
    Naselaris, T., Olman, C.A., Stansbury, D.E., Ugurbil, K., Gallant, J.L.: A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes. NeuroImage 105, 215–228 (2015)Google Scholar
  41. 41.
    Naselaris, T., Prenger, R.J., Kay, K.N., Oliver, M., Gallant, J.L.: Bayesian reconstruction of natural images from human brain activity. Neuron 63(6), 902–915 (2009)Google Scholar
  42. 42.
    Nishida, S., Nishimoto, S.: Decoding naturalistic experiences from human brain activity via distributed representations of words. NeuroImage 180, 232–242 (2018)Google Scholar
  43. 43.
    Nishimoto, S., Vu, A.T., Naselaris, T., Benjamini, Y., Yu, B., Gallant, J.L.: Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1–6 (2011)Google Scholar
  44. 44.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)Google Scholar
  45. 45.
    Pasley, B.N., et al.: Reconstructing speech from human auditory cortex. PLoS Biol. 10(1), e1001251 (2012)Google Scholar
  46. 46.
    Ponce, C.R., et al.: Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177, 999–1009 (2019)Google Scholar
  47. 47.
    Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat. Neurosci. 2(11), 1019–1025 (1999)Google Scholar
  48. 48.
    Roelfsema, P.R., Denys, D., Klink, P.C.: Mind reading and writing: the future of neurotechnology. Trends Cogn. Sci. 22(7), 1–13 (2018)Google Scholar
  49. 49.
    Roweis, S., Brody, C.: Linear heteroencoders. Technical report. GCNU TR 1999–002, Gatsby Computational Neuroscience Unit (1999)Google Scholar
  50. 50.
    Schoenmakers, S., Barth, M., Heskes, T., van Gerven, M.A.J.: Linear reconstruction of perceived images from human brain activity. NeuroImage 83, 951–961 (2013)Google Scholar
  51. 51.
    Schoenmakers, S., Güçlü, U., van Gerven, M.A.J., Heskes, T.: Gaussian mixture models and semantic gating improve reconstructions from human brain activity. Front. Comput. Neurosci. 8, 1–10 (2015)Google Scholar
  52. 52.
    Seeliger, K., et al.: Convolutional neural network-based encoding and decoding of visual object recognition in space and time. NeuroImage 180(A), 253–266 (2017)Google Scholar
  53. 53.
    Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., van Gerven, M.A.J.: Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181, 775–785 (2018)Google Scholar
  54. 54.
    Seeliger, K., Ambrogioni, L., Güçlütürk, Y., Güçlü, U., Gerven, M.A.J.: Neural system identification with neural information flow. bioRxiv (2019)Google Scholar
  55. 55.
    Senden, M., Emmerling, T.C., van Hoof, R., Frost, M.A., Goebel, R.: Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. Brain Struct. Funct. 224(3), 1167–1183 (2019)Google Scholar
  56. 56.
    Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M., Poggio, T.: Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 411–426 (2007)Google Scholar
  57. 57.
    Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), 1–23 (2019)Google Scholar
  58. 58.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  59. 59.
    Stanley, G.B., Li, F.F., Dan, Y.: Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J. Neurosci. 19(18), 8036–8042 (1999)Google Scholar
  60. 60.
    Thirion, B., et al.: Inverse retinotopy: inferring the visual content of images from brain activation patterns. NeuroImage 33(4), 1104–1116 (2006)Google Scholar
  61. 61.
    VanRullen, R., Reddy, L.: Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks. arXiv preprint arXiv:1810.03856 (2018)
  62. 62.
    Victor, J.D., Purpura, K., Katz, E., Mao, B.: Population encoding of spatial frequency, orientation, and color in macaque V1. J. Neurophysiol. 72(5), 2151–2166 (1994)Google Scholar
  63. 63.
    Vidaurre, D., van Gerven, M.A.J., Bielza, C., Larrañaga, P., Heskes, T.: Bayesian sparse partial least squares. Neural Comput. 25(12), 3318–3339 (2013)MathSciNetzbMATHGoogle Scholar
  64. 64.
    Wallis, J.D.: Decoding cognitive processes from neural ensembles. Trends Cogn. Sci. 22(12), 1091–1102 (2018)Google Scholar
  65. 65.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_53CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Marcel A. J. van Gerven
    • 1
    Email author
  • Katja Seeliger
    • 1
  • Umut Güçlü
    • 1
  • Yağmur Güçlütürk
    • 1
  1. 1.Donders Institute for Brain, Cognition and BehaviourNijmegenThe Netherlands

Personalised recommendations