Skip to main content
SpringerLink
Log in

Current Advances in Neural Decoding

  • Chapter
  • First Online:
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11700))

Abstract

Neural decoding refers to the extraction of semantically meaningful information from brain activity patterns. We discuss how advances in machine learning drive new advances in neural decoding. While linear methods allow for the reconstruction of basic stimuli from brain activity, more sophisticated nonlinear methods are required when reconstructing complex naturalistic stimuli. We show how deep neural networks and adversarial training yield state-of-the-art results. Ongoing advances in machine learning may one day allow the reconstruction of thoughts from brain activity patterns, providing a unique insight into the contents of the human mind.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Eq. (21.12) is also a standard result obtained in Bayesian linear regression when the roles of \(\mathbf {\varvec{B}}\) and \(\mathbf {\varvec{x}}\) are interchanged [5].

References

  1. Akbari, H., Khalighinejad, B., Herrero, J.L., Mehta, A.D., Mesgarani, N.: Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep. 9(1), 874 (2019)

    Google Scholar 

  2. Anumanchipalli, G.K., Chartier, J., Chang, E.F.: Speech synthesis from neural decoding of spoken sentences. Nature 568, 493–501 (2019)

    Google Scholar 

  3. Bahramisharif, A., van Gerven, M.A.J., Heskes, T., Jensen, O.: Covert attention allows for continuous control of brain-computer interfaces. Eur. J. Neurosci. 31(8), 1501–1508 (2010)

    Google Scholar 

  4. Bialek, W., Rieke, F., van Steveninck, R.R.D.R., Warland, D.: Reading a neural code. Science 252(5014), 1854–1857 (1991)

    Google Scholar 

  5. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)

    MATH  Google Scholar 

  6. Chang, L., Tsao, D.Y.: The code for facial identity in the primate brain. Cell 169(6), 1013–1028 (2017)

    Google Scholar 

  7. Cowen, A.S., Chun, M.M., Kuhl, B.A.: Neural portraits of perception: reconstructing face images from evoked brain activity. NeuroImage 94, 12–22 (2014)

    Google Scholar 

  8. Cox, D.D., Dean, T.: Neural networks and neuroscience-inspired computer vision. Curr. Biol. 24(18), PR921–R929 (2014)

    Google Scholar 

  9. Daugman, J.G.: Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J. Opt. Soc. Am. A: 2(7), 1160–1169 (1985)

    Google Scholar 

  10. Dayan, P., Abbott, L.F.: Theoretical Neuroscience. MIT Press, Cambridge (2005)

    MATH  Google Scholar 

  11. Dijkstra, N., Bosch, S.E., van Gerven, M.A.J.: Vividness of visual imagery depends on the neural overlap with perception in visual areas. J. Neurosci. 37(5), 1367–1373 (2017)

    Google Scholar 

  12. Dijkstra, N., Mostert, P., de Lange, F.P., Bosch, S.E., van Gerven, M.A.J.: Differential temporal dynamics during visual imagery and perception. eLIFE, pp. 1–16 (2018)

    Google Scholar 

  13. Dijkstra, N., Zeidman, P., Ondobaka, S., van Gerven, M.A.J., Friston, K.: Distinct top-down and bottom-up brain connectivity during visual perception and imagery. Sci. Rep. 7(5677), 1–9 (2017)

    Google Scholar 

  14. Domingos, P.: Why does bagging work? a Bayesian account and its implications. In: Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, pp. 155–158 (1997)

    Google Scholar 

  15. Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Univ. Montreal 1341, 1–13 (2009)

    Google Scholar 

  16. Fukushima, K.: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193–202 (1980)

    MATH  Google Scholar 

  17. van Gerven, M.A.J.: A primer on encoding models in sensory neuroscience. J. Math. Psychol. 76(B), 172–183 (2017)

    Google Scholar 

  18. van Gerven, M.A.J., Chao, Z.C., Heskes, T.: On the decoding of intracranial data using sparse orthonormalized partial least squares. J. Neural Eng. 9(2), 026017 (2012)

    Google Scholar 

  19. van Gerven, M.A.J., Kok, P., de Lange, F.P., Heskes, T.: Dynamic decoding of ongoing perception. NeuroImage 57, 950–957 (2011)

    Google Scholar 

  20. van Gerven, M.A.J., de Lange, F.P., Heskes, T.: Neural decoding with hierarchical generative models. Neural Comput. 22(12), 3127–3142 (2010)

    MATH  Google Scholar 

  21. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NeurIPS) 2014, pp. 2672–2680 (2014)

    Google Scholar 

  22. Güçlü, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35(27), 10005–10014 (2015)

    Google Scholar 

  23. Güçlü, U., van Gerven, M.A.J.: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage 145, 329–336 (2017)

    Google Scholar 

  24. Güçlü, U., Thielen, J., Hanke, M., van Gerven, M.A.J.: Brains on beats. In: Advances in Neural Information Processing Systems (NeurIPS) 2016, pp. 1–12 (2016)

    Google Scholar 

  25. Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S.E., van Lier, R., van Gerven, M.A.J.: Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems (NeurIPS) 2017 (2017)

    Google Scholar 

  26. Hastie, T., Tibshirani, R.J., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2008). https://doi.org/10.1007/978-0-387-84858-7

    Book  MATH  Google Scholar 

  27. Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001)

    Google Scholar 

  28. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)

    MathSciNet  MATH  Google Scholar 

  29. Horikawa, T., Kamitani, Y.: Hierarchical neural representation of dreamed objects revealed by brain decoding with deep neural network features. Front. Comput. Neurosci. 11, 1–11 (2017)

    Google Scholar 

  30. Horikawa, T., Tamaki, M., Miyawaki, Y., Kamitani, Y.: Neural decoding of visual imagery during sleep. Science 340(6132), 639–642 (2013)

    Google Scholar 

  31. Ienca, M., Haselager, P., Emanuel, E.J.: Brain leaks and consumer neurotechnology. Nat. Biotechnol. 36(9), 805–810 (2018)

    Google Scholar 

  32. Jones, J.P., Palmer, L.A.: An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. J. Neurophysiol. 58, 1233–1258 (1987)

    Google Scholar 

  33. Kamitani, Y., Tong, F.: Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8(5), 679–685 (2005)

    Google Scholar 

  34. Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L.: Identifying natural images from human brain activity. Nature 452, 352–355 (2008)

    Google Scholar 

  35. LeCun, Y., Bengio, Y., Hinton, G.E.: Deep learning. Nature 521(7553), 436 (2015)

    Google Scholar 

  36. Marčelja, S.: Mathematical description of the responses of simple cortical cells. J. Opt. Soc. Am. A: 70(11), 1297–1300 (1980)

    MathSciNet  Google Scholar 

  37. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: International Conference on Learning Representations (ICLR) 2013. Cornell University Library (2013)

    Google Scholar 

  38. Miyawaki, Y., et al.: Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60(5), 915–929 (2008)

    Google Scholar 

  39. Mordvintsev, A., Olah, C., Tyka, M.: Inceptionism: going deeper into neural networks (2005). https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html

  40. Naselaris, T., Olman, C.A., Stansbury, D.E., Ugurbil, K., Gallant, J.L.: A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes. NeuroImage 105, 215–228 (2015)

    Google Scholar 

  41. Naselaris, T., Prenger, R.J., Kay, K.N., Oliver, M., Gallant, J.L.: Bayesian reconstruction of natural images from human brain activity. Neuron 63(6), 902–915 (2009)

    Google Scholar 

  42. Nishida, S., Nishimoto, S.: Decoding naturalistic experiences from human brain activity via distributed representations of words. NeuroImage 180, 232–242 (2018)

    Google Scholar 

  43. Nishimoto, S., Vu, A.T., Naselaris, T., Benjamini, Y., Yu, B., Gallant, J.L.: Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1–6 (2011)

    Google Scholar 

  44. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)

    Google Scholar 

  45. Pasley, B.N., et al.: Reconstructing speech from human auditory cortex. PLoS Biol. 10(1), e1001251 (2012)

    Google Scholar 

  46. Ponce, C.R., et al.: Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177, 999–1009 (2019)

    Google Scholar 

  47. Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat. Neurosci. 2(11), 1019–1025 (1999)

    Google Scholar 

  48. Roelfsema, P.R., Denys, D., Klink, P.C.: Mind reading and writing: the future of neurotechnology. Trends Cogn. Sci. 22(7), 1–13 (2018)

    Google Scholar 

  49. Roweis, S., Brody, C.: Linear heteroencoders. Technical report. GCNU TR 1999–002, Gatsby Computational Neuroscience Unit (1999)

    Google Scholar 

  50. Schoenmakers, S., Barth, M., Heskes, T., van Gerven, M.A.J.: Linear reconstruction of perceived images from human brain activity. NeuroImage 83, 951–961 (2013)

    Google Scholar 

  51. Schoenmakers, S., Güçlü, U., van Gerven, M.A.J., Heskes, T.: Gaussian mixture models and semantic gating improve reconstructions from human brain activity. Front. Comput. Neurosci. 8, 1–10 (2015)

    Google Scholar 

  52. Seeliger, K., et al.: Convolutional neural network-based encoding and decoding of visual object recognition in space and time. NeuroImage 180(A), 253–266 (2017)

    Google Scholar 

  53. Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., van Gerven, M.A.J.: Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181, 775–785 (2018)

    Google Scholar 

  54. Seeliger, K., Ambrogioni, L., Güçlütürk, Y., Güçlü, U., Gerven, M.A.J.: Neural system identification with neural information flow. bioRxiv (2019)

    Google Scholar 

  55. Senden, M., Emmerling, T.C., van Hoof, R., Frost, M.A., Goebel, R.: Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. Brain Struct. Funct. 224(3), 1167–1183 (2019)

    Google Scholar 

  56. Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M., Poggio, T.: Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 411–426 (2007)

    Google Scholar 

  57. Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), 1–23 (2019)

    Google Scholar 

  58. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  59. Stanley, G.B., Li, F.F., Dan, Y.: Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J. Neurosci. 19(18), 8036–8042 (1999)

    Google Scholar 

  60. Thirion, B., et al.: Inverse retinotopy: inferring the visual content of images from brain activation patterns. NeuroImage 33(4), 1104–1116 (2006)

    Google Scholar 

  61. VanRullen, R., Reddy, L.: Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks. arXiv preprint arXiv:1810.03856 (2018)

  62. Victor, J.D., Purpura, K., Katz, E., Mao, B.: Population encoding of spatial frequency, orientation, and color in macaque V1. J. Neurophysiol. 72(5), 2151–2166 (1994)

    Google Scholar 

  63. Vidaurre, D., van Gerven, M.A.J., Bielza, C., Larrañaga, P., Heskes, T.: Bayesian sparse partial least squares. Neural Comput. 25(12), 3318–3339 (2013)

    MathSciNet  MATH  Google Scholar 

  64. Wallis, J.D.: Decoding cognitive processes from neural ensembles. Trends Cogn. Sci. 22(12), 1091–1102 (2018)

    Google Scholar 

  65. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands

    Marcel A. J. van Gerven, Katja Seeliger, Umut Güçlü & Yağmur Güçlütürk

Authors
  1. Marcel A. J. van Gerven
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katja Seeliger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umut Güçlü
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yağmur Güçlütürk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcel A. J. van Gerven .

Editor information

Editors and Affiliations

  1. Fraunhofer Heinrich Hertz Institute, Berlin, Germany

    Wojciech Samek

  2. Technische Universität Berlin, Berlin, Germany

    Grégoire Montavon

  3. University of Oxford, Oxford, UK

    Andrea Vedaldi

  4. Technical University of Denmark, Kgs. Lyngby, Denmark

    Lars Kai Hansen

  5. Sekretariat MAR 4-1, Technical University of Berlin, Berlin, Berlin, Germany

    Klaus-Robert Müller

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

van Gerven, M.A.J., Seeliger, K., Güçlü, U., Güçlütürk, Y. (2019). Current Advances in Neural Decoding. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L., Müller, KR. (eds) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science(), vol 11700. Springer, Cham. https://doi.org/10.1007/978-3-030-28954-6_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-28954-6_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-28953-9

  • Online ISBN: 978-3-030-28954-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation