Near-Instantaneous Classification of Perceptual States from Cortical Surface Recordings

  • Kai J. Miller
  • Gerwin Schalk
  • Dora Hermes
  • Jeffrey G. Ojemann
  • Rajesh P. N. Rao
Part of the SpringerBriefs in Electrical and Computer Engineering book series (BRIEFSELECTRIC)


Human visual processing is of such complexity that, despite decades of focused research, many basic questions remain unanswered. Although we know that the inferotemporal cortex is a key region in object recognition, we don’t fully understand its physiologic role in brain function, nor do we have the full set of tools to explore this question. Here we show that electrical potentials from the surface of the human brain contain enough information to decode a subject’s perceptual state accurately, and with fine temporal precision. Electrocorticographic (ECoG) arrays were placed over the inferotemporal cortical areas of seven subjects. Pictures of faces and houses were quickly presented while each subject performed a simple visual task. Results showed that two well-known types of brain signals—event-averaged broadband power and event-averaged raw potential—can independently or together be used to classify the presented image. When applied to continuously recorded brain activity, our decoding technique could accurately predict whether each stimulus was a face, house, or neither, with ~20 ms timing error. These results provide a roadmap for improved brain-computer interfacing tools to help neurosurgeons, research scientists, engineers, and, ultimately, patients.


Human vision Electrocorticography Broadband power Event-related potential Fusiform cortex 



We are grateful to the patients and staff at Harborview Hospital in Seattle. We are grateful for helpful discussions with Kalanit Grill-Spector and Bharathi Jagadeesh. This work was supported by National Aeronautics and Space Administration Graduate Student Research Program (KJM), the NIH (R01-NS065186 (KJM, JGO, RPNR), T32-EY20485 (DH), R01-EB006356 (GS), R01-EB00856 (GS) and P41-EB018783 (GS)), the NSF (EEC-1028725 (RPNR)), the US Army Research Office (W911NF-07-1-0415 (GS), W911NF-08-1-0216 (GS) and W911NF-14-1-0440 (GS)), and Fondazione Neurone (GS).


  1. G.K. Aguirre, E. Zarahn, M. D’Esposito, An area within human ventral cortex sensitive to “building” stimuli: evidence and implications. Neuron 21, 373–383 (1998)CrossRefGoogle Scholar
  2. C.M. Bishop, Neural Networks for Pattern Recognition (Oxford University Press, Oxford, 1995)zbMATHGoogle Scholar
  3. R.M. Cichy, D. Pantazis, A. Oliva, Resolving human object recognition in space and time. Nat. Neurosci. 17, 455–462 (2014)CrossRefGoogle Scholar
  4. S.M. Crouzet, H. Kirchner, S.J. Thorpe, Fast saccades toward faces: face detection in just 100 ms. J. Vis. 10(16), 11–17 (2010)Google Scholar
  5. A.D. Engell, G. McCarthy, The relationship of gamma oscillations and face-specific ERPs recorded subdurally from occipitotemporal cortex. Cereb. Cortex 21, 1213–1221 (2011)CrossRefGoogle Scholar
  6. A.D. Engell, G. McCarthy, Repetition suppression of face-selective evoked and induced EEG recorded from human cortex. Human Brain Map (2014). doi: 10.1002/hbm.22467
  7. A.D. Engell, S. Huettel, G. McCarthy, The fMRI BOLD signal tracks electrophysiological spectral perturbations, not event-related potentials. NeuroImage 59, 2600–2606 (2012)CrossRefGoogle Scholar
  8. R. Epstein, N. Kanwisher, A cortical representation of the local visual environment. Nature 392, 598–601 (1998)CrossRefGoogle Scholar
  9. C. Guger, B.Z. Allison, G.R. Müller-Putz, in Brain-Computer Interface Research: A State-of-the-Art Summary 4 (in press)Google Scholar
  10. D. Hermes, K.J. Miller, H.J. Noordmans, M.J. Vansteensel, N.F. Ramsey, Automated electrocorticographic electrode localization on individually rendered brain surfaces. J. Neurosci. Methods 185, 293–298 (2010)CrossRefGoogle Scholar
  11. C.P. Hung, G. Kreiman, T. Poggio, J.J. DiCarlo, Fast readout of object identity from macaque inferior temporal cortex. Science 310, 863–866 (2005)CrossRefGoogle Scholar
  12. C. Jacques, N. Witthoft, K.S. Weiner, B.L. Foster, V. Rangarajan, D. Hermes, K.J. Miller, J. Parvizi, K. Grill-Spector, Corresponding ECoG and fMRI category-selective signals in Human ventral temporal cortex. Neuropsychologia (2015)Google Scholar
  13. N. Kanwisher, J. McDermott, M.M. Chun, The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302–4311 (1997)Google Scholar
  14. R. Kiani, H. Esteky, K. Mirpour, K. Tanaka, Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J. Neurophysiol. 97, 4296–4309 (2007)CrossRefGoogle Scholar
  15. H. Kirchner, S.J. Thorpe, Ultra-rapid object detection with saccadic eye movements: visual processing speed revisited. Vis. Res. 46, 1762–1776 (2006)CrossRefGoogle Scholar
  16. H. Liu, Y. Agam, J.R. Madsen, G. Kreiman, Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 62, 281–290 (2009)CrossRefGoogle Scholar
  17. K.J. Miller, L.B. Sorensen, J.G. Ojemann, M. den Nijs, Power-law scaling in the brain surface electric potential. PLoS Comput. Biol. 5, e1000609 (2009)MathSciNetCrossRefGoogle Scholar
  18. K.J. Miller, C.J. Honey, D. Hermes, R.P. Rao, M. denNijs, J.G. Ojemann, Broadband changes in the cortical surface potential track activation of functionally diverse neuronal populations. NeuroImage 85(Pt 2), 711–720 (2014)CrossRefGoogle Scholar
  19. K.J. Miller, D. Hermes, N. Witthoft, R.P. Rao, J.G. Ojemann, The physiology of perception in human temporal lobe is specialized for contextual novelty. J. Neurophysiol. 114, 256–263 (2015a)CrossRefGoogle Scholar
  20. K.J. Miller, G. Schalk, D. Hermes, J.G. Ojemann, R.P.N. Rao, Spontaneous Decoding of the Timing and Content of Human Object Perception from Cortical Surface Recordings Reveals Complementary Information in the Event-Related Potential and Broadband Spectral Change (2015b)Google Scholar
  21. B. Porat, A Course in Digital Signal Processing (Wiley, New York, 1997)Google Scholar
  22. A. Puce, T. Allison, J.C. Gore, G. McCarthy, Face-sensitive regions in human extrastriate cortex studied by functional MRI. J. Neurophysiol. 74, 1192–1199 (1995)Google Scholar
  23. G. Schalk, J. Mellinger, A Practical Guide to Brain-Computer Interfacing with BCI2000 (Springer, London, 2010)CrossRefGoogle Scholar
  24. G. Schalk, D.J. McFarland, T. Hinterberger, N. Birbaumer, J.R. Wolpaw, BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51(6), 1034–1043 (2004)CrossRefGoogle Scholar
  25. I. Simanova, M. van Gerven, R. Oostenveld, P. Hagoort, Identifying object categories from event-related EEG: toward decoding of conceptual representations. PLoS ONE 5, e14465 (2010)CrossRefGoogle Scholar
  26. M.E. van de Nieuwenhuijzen, A.R. Backus, A. Bahramisharif, C.F. Doeller, O. Jensen, M.A. van Gerven, MEG-based decoding of the spatiotemporal dynamics of visual category perception. NeuroImage 83, 1063–1073 (2013)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2015

Authors and Affiliations

  1. 1.Department of NeurosurgeryStanford UniversityStanfordUSA
  2. 2.Program in Neurobiology and BehaviorUniversity of WashingtonSeattleUSA
  3. 3.National Center for Adaptive NeurotechnologiesWadsworth CenterAlbanyUSA
  4. 4.Department of PsychologyStanford UniversityStanfordUSA
  5. 5.Department of Neurological SurgeryUniversity of WashingtonSeattleUSA
  6. 6.Computer Science and EngineeringUniversity of WashingtonSeattleUSA
  7. 7.Center for Sensorimotor Neural EngineeringUniversity of WashingtonSeattleUSA

Personalised recommendations