Advertisement

The Potential for a Speech Brain–Computer Interface Using Chronic Electrocorticography

  • Qinwan RabbaniEmail author
  • Griffin Milsap
  • Nathan E. Crone
Review

Abstract

A brain–computer interface (BCI) is a technology that uses neural features to restore or augment the capabilities of its user. A BCI for speech would enable communication in real time via neural correlates of attempted or imagined speech. Such a technology would potentially restore communication and improve quality of life for locked-in patients and other patients with severe communication disorders. There have been many recent developments in neural decoders, neural feature extraction, and brain recording modalities facilitating BCI for the control of prosthetics and in automatic speech recognition (ASR). Indeed, ASR and related fields have developed significantly over the past years, and many lend many insights into the requirements, goals, and strategies for speech BCI. Neural speech decoding is a comparatively new field but has shown much promise with recent studies demonstrating semantic, auditory, and articulatory decoding using electrocorticography (ECoG) and other neural recording modalities. Because the neural representations for speech and language are widely distributed over cortical regions spanning the frontal, parietal, and temporal lobes, the mesoscopic scale of population activity captured by ECoG surface electrode arrays may have distinct advantages for speech BCI, in contrast to the advantages of microelectrode arrays for upper-limb BCI. Nevertheless, there remain many challenges for the translation of speech BCIs to clinical populations. This review discusses and outlines the current state-of-the-art for speech BCI and explores what a speech BCI using chronic ECoG might entail.

Key Words

Electrocorticography (ECoG) Brain–computer interface (BCI) Neural speech decoding Automatic speech recognition (ASR) Locked-in syndrome (LIS) Communication 

Notes

Acknowledgments:

The authors of this paper have been supported by the National Institutes of Health (R01 NS091139).

Required Author Forms

Disclosure forms provided by the authors are available with the online version of this article.

Supplementary material

13311_2018_692_MOESM1_ESM.pdf (452 kb)
ESM 1 (PDF 451 kb)

References

  1. 1.
    Gorno-Tempini ML, Hillis AE, Weintraub S, Kertesz A, Mendez M, Cappa SF, et al. Classification of primary progressive aphasia and its variants. Neurology 2011;76(11):1006–14.CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Smith E, Delargy M. Locked-in syndrome. BMJ 2005;330(7488):406–9.CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Indefrey P. The Spatial and Temporal Signatures of Word Production Components: A Critical Update. Front Psychol [Internet]. 2011;2. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3191502/
  4. 4.
    Friston KJ, Jezzard P, Turner R. Analysis of functional MRI time-series. Hum Brain Mapp 1994;1(2):153–71.CrossRefGoogle Scholar
  5. 5.
    Irani F, Platek SM, Bunce S, Ruocco AC, Chute D. Functional near infrared spectroscopy (fNIRS): an emerging neuroimaging technology with important applications for the study of brain disorders. Clin Neuropsychol 2007;21(1):9–37.CrossRefPubMedGoogle Scholar
  6. 6.
    Dassios G, Fokas AS, Kariotou F. On the non-uniqueness of the inverse MEG problem. Inverse Probl 2005;21(2):L1.CrossRefGoogle Scholar
  7. 7.
    Baillet S, Mosher JC, Leahy RM. Electromagnetic brain mapping. IEEE Signal Process Mag 2001;18(6):14–30.CrossRefGoogle Scholar
  8. 8.
    Gotman J. High frequency oscillations: the new EEG frontier? Epilepsia 2010;51(Suppl 1):63–5.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Michel CM, Murray MM. Towards the utilization of EEG as a brain imaging tool. NeuroImage 2012;61(2):371–85.CrossRefPubMedGoogle Scholar
  10. 10.
    Zelmann R, Lina JM, Schulze-Bonhage A, Gotman J, Jacobs J. Scalp EEG is not a blur: it can see high frequency oscillations although their generators are small. Brain Topogr 2014;27(5):683–704.CrossRefPubMedGoogle Scholar
  11. 11.
    Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, et al. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng 2018;15(3):031005.CrossRefPubMedGoogle Scholar
  12. 12.
    Maynard EM, Nordhausen CT, Normann RA. The Utah Intracortical Electrode Array: a recording structure for potential brain-computer interfaces. Electroencephalogr Clin Neurophysiol 1997;102(3):228–39.CrossRefPubMedGoogle Scholar
  13. 13.
    Biran R, Martin DC, Tresco PA. Neuronal cell loss accompanies the brain tissue response to chronically implanted silicon microelectrode arrays. Exp Neurol 2005;195(1):115–26.CrossRefPubMedGoogle Scholar
  14. 14.
    Prodanov D, Delbeke J. Mechanical and biological interactions of implants with the brain and their impact on implant design. Front Neurosci [Internet]. 2016;10 [cited 2018 Jun 5]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4746296/
  15. 15.
    Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 2006;442(7099):164–71.CrossRefPubMedGoogle Scholar
  16. 16.
    Bouton CE, Shaikhouni A, Annetta NV, Bockbrader MA, Friedenberg DA, Nielson DM, et al. Restoring cortical control of functional movement in a human with quadriplegia. Nature 2016;533(7602):247–50.CrossRefPubMedGoogle Scholar
  17. 17.
    Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 2013;381(9866):557–64.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Crone NE, Sinai A, Korzeniewska A. High-frequency gamma oscillations and human brain mapping with electrocorticography. In: Neuper C, Klimesch W, editors. Progress in Brain Research [Internet]. Elsevier; 2006. p. 275–95. (Event-Related Dynamics of Brain Oscillations; vol. 159). [cited 2018 Aug 30] Available from: http://www.sciencedirect.com/science/article/pii/S0079612306590193
  19. 19.
    Milsap G, Collard M, Coogan C, Crone NE. BCI2000Web and WebFM: browser-based tools for brain computer interfaces and functional brain mapping. Front Neurosci 2019Google Scholar
  20. 20.
    Slutzky MW, Jordan LR, Krieg T, Chen M, Mogul DJ, Miller LE. Optimal spacing of surface electrode arrays for brain machine interface applications. J Neural Eng 2010;7(2):26004.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Rupp K, Roos M, Milsap G, Caceres C, Ratto C, Chevillet M, et al. Semantic attributes are encoded in human electrocorticographic signals during visual object recognition. NeuroImage 2017;148:318–29.CrossRefPubMedGoogle Scholar
  22. 22.
    Huth AG, de Heer WA, Griffiths TL, Theunissen FE, Gallant JL. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 2016;532(7600):453–8.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Mines MA, Hanson BF, Shoup JE. Frequency of occurrence of phonemes in conversational English. Lang Speech 1978;21(3):221–41.CrossRefPubMedGoogle Scholar
  24. 24.
    Hickok G, Poeppel D. Chapter 25—neural basis of speech perception. In: Hickok G, Small SL, editors. Neurobiology of Language [Internet]. San Diego: Academic Press; 2016. p. 299–310. [cited 2018 Aug 30] Available from: http://www.sciencedirect.com/science/article/pii/B9780124077942000250 CrossRefGoogle Scholar
  25. 25.
    Titze IR. Principles of Voice Production. Prentice Hall; 1994. 390 p.Google Scholar
  26. 26.
    Gasser M. 3.3 Vowels. In: How Language Works: The Cognitive Science of Linguistics [Internet]. 2012 [cited 2017 Dec 24]. Available from: http://www.indiana.edu/~hlw/PhonUnits/vowels.html
  27. 27.
    Hickok G, Houde J, Rong F. Sensorimotor Integration in speech processing: computational basis and neural organization. Neuron 2011;69(3):407–22.CrossRefPubMedPubMedCentralGoogle Scholar
  28. 28.
    Conant D, Bouchard KE, Chang EF. Speech map in the human ventral sensory-motor cortex. Curr Opin Neurobiol 2014;24:63–7.CrossRefPubMedGoogle Scholar
  29. 29.
    Rubin P, Baer T, Mermelstein P. An articulatory synthesizer for perceptual research. J Acoust Soc Am 1981;70(2):321–8.CrossRefGoogle Scholar
  30. 30.
    Uria B, Murray I, Renals S, Richmond K. Deep architectures for articulatory inversion [Internet]. INTERSPEECH-2012; 2012. Available from: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.248.6643
  31. 31.
    Hotz-Boendemaker S, Funk M, Summers P, Brugger P, Hepp-Reymond M-C, Curt A, et al. Preservation of motor programs in paraplegics as demonstrated by attempted and imagined foot movements. NeuroImage 2008 Jan 1;39(1):383–94.CrossRefGoogle Scholar
  32. 32.
    Hermes D, Vansteensel MJ, Albers AM, Bleichner MG, Benedictus MR, Orellana CM, et al. Functional MRI-based identification of brain areas involved in motor imagery for implantable brain–computer interfaces. J Neural Eng 2011;8(2):025007.CrossRefPubMedGoogle Scholar
  33. 33.
    Liu H, Agam Y, Madsen JR, Kreiman G. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 2009;62(2):281–90.CrossRefPubMedPubMedCentralGoogle Scholar
  34. 34.
    Wang W, Degenhart AD, Sudre GP, Pomerleau DA, Tyler-Kabara EC. Decoding semantic information from human electrocorticographic (ECoG) signals. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2011. p. 6294–8.Google Scholar
  35. 35.
    Manning JR, Sperling MR, Sharan A, Rosenberg EA, Kahana MJ. Spontaneously reactivated patterns in frontal and temporal lobe predict semantic clustering during memory search. J Neurosci 2012;32(26):8871–8.CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    Chen Y, Shimotake A, Matsumoto R, Kunieda T, Kikuchi T, Miyamoto S, et al. The ‘when’ and ‘where’ of semantic coding in the anterior temporal lobe: temporal representational similarity analysis of electrocorticogram data. Cortex 2016;79:1–13.CrossRefPubMedPubMedCentralGoogle Scholar
  37. 37.
    Moerel M, De Martino F, Formisano E. An anatomical and functional topography of human auditory cortical areas. Front Neurosci [Internet]. 2014;8. [cited 2017 Nov 13]. Available from: https://www.frontiersin.org/articles/10.3389/fnins.2014.00225/full#h4
  38. 38.
    Humphries C, Liebenthal E, Binder JR. Tonotopic organization of human auditory cortex. NeuroImage 2010;50(3):1202–11.CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Langers DRM, van Dijk P. Mapping the tonotopic organization in human auditory cortex with minimally salient acoustic stimulation. Cereb Cortex 2012;22(9):2024–38.CrossRefPubMedGoogle Scholar
  40. 40.
    Su L, Zulfiqar I, Jamshed F, Fonteneau E, Marslen-Wilson W. Mapping tonotopic organization in human temporal cortex: representational similarity analysis in EMEG source space. Front Neurosci [Internet] 2014;e8. [cited 2017 Dec 25] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4228977/
  41. 41.
    Norman-Haignere S, Kanwisher N, McDermott JH. Cortical pitch regions in humans respond primarily to resolved harmonics and are located in specific tonotopic regions of anterior auditory cortex. J Neurosci 2013;33(50):19451–69.CrossRefPubMedPubMedCentralGoogle Scholar
  42. 42.
    Pantev C, Hoke M, Lutkenhoner B, Lehnertz K. Tonotopic organization of the auditory cortex: pitch versus frequency representation. Science 1989;246(4929):486–8.CrossRefPubMedGoogle Scholar
  43. 43.
    Mesgarani N, Cheung C, Johnson K, Chang EF. Phonetic feature encoding in human superior temporal gyrus. Science 2014;343(6174):1006–10.CrossRefPubMedPubMedCentralGoogle Scholar
  44. 44.
    Hullett PW, Hamilton LS, Mesgarani N, Schreiner CE, Chang EF. Human superior temporal gyrus organization of spectrotemporal modulation tuning derived from speech stimuli. J Neurosci 2016;36(6):2014–26.CrossRefPubMedPubMedCentralGoogle Scholar
  45. 45.
    Hamilton LS, Edwards E, Chang EF. A spatial map of onset and sustained responses to speech in the human superior temporal gyrus. Curr Biol 2018;28(12):1860–1871.e4.CrossRefPubMedGoogle Scholar
  46. 46.
    Yao B, Belin P, Scheepers C. Silent reading of direct versus indirect speech activates voice-selective areas in the auditory cortex. J Cogn Neurosci 2011;23(10):3146–52.CrossRefPubMedGoogle Scholar
  47. 47.
    Perrone-Bertolotti M, Kujala J, Vidal JR, Hamame CM, Ossandon T, Bertrand O, et al. How silent is silent reading? Intracerebral evidence for top-down activation of temporal voice areas during reading. J Neurosci 2012;32(49):17554–62.CrossRefPubMedGoogle Scholar
  48. 48.
    Perrone-Bertolotti M, Pichat C, Le Bas JF, Baciu A, Baciu M. Functional MRI evidence for modulation of cerebral activity by grapheme-to-phoneme conversion in French, and by the variable of gender. J Neurolinguistics 2011;24(4):507–20.CrossRefGoogle Scholar
  49. 49.
    Chan AM, Dykstra AR, Jayaram V, Leonard MK, Travis KE, Gygi B, et al. Speech-specific tuning of neurons in human superior temporal gyrus. Cereb Cortex N Y NY 2014;24(10):2679–93.CrossRefGoogle Scholar
  50. 50.
    Magrassi L, Aromataris G, Cabrini A, Annovazzi-Lodi V, Moro A. Sound representation in higher language areas during language generation. Proc Natl Acad Sci 2015;112(6):1868–73.CrossRefPubMedGoogle Scholar
  51. 51.
    Brumberg JS, Krusienski DJ, Chakrabarti S, Gunduz A, Brunner P, Ritaccio AL, et al. Spatio-temporal progression of cortical activity related to continuous overt and covert speech production in a reading task. PLoS One 2016 Nov 22;11(11):e0166872.CrossRefPubMedPubMedCentralGoogle Scholar
  52. 52.
    Martin S, Brunner P, Holdgraf C, Heinze H-J, Crone NE, Rieger J, et al. Decoding spectrotemporal features of overt and covert speech from the human cortex. Front Neuroengineering [Internet]. 2014;7. [cited 2017 Dec 19] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4034498/
  53. 53.
    Pei X, Leuthardt EC, Gaona CM, Brunner P, Wolpaw JR, Schalk G. Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition. NeuroImage 2011;54(4):2960–72.CrossRefPubMedGoogle Scholar
  54. 54.
    Leuthardt EC, Pei X-M, Breshears J, Gaona C, Sharma M, Freudenberg Z, et al. Temporal evolution of gamma activity in human cortex during an overt and covert word repetition task. Front Hum Neurosci [Internet]. 2012;6. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3342676/
  55. 55.
    Tian X, Poeppel D. Mental imagery of speech and movement implicates the dynamics of internal forward models. Front Psychol [Internet]. 2010;1. [cited 2018 Jul 11] Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2010.00166/full
  56. 56.
    Pei X, Barbour DL, Leuthardt EC, Schalk G. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J Neural Eng 2011;8(4):046028.CrossRefPubMedPubMedCentralGoogle Scholar
  57. 57.
    Ikeda S, Shibata T, Nakano N, Okada R, Tsuyuguchi N, Ikeda K, et al. Neural decoding of single vowels during covert articulation using electrocorticography. Front Hum Neurosci [Internet]. 2014;8. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3945950/
  58. 58.
    Penfield W, Boldrey E. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain. 1937;60(4):389–443.CrossRefGoogle Scholar
  59. 59.
    Schieber MH. Constraints on somatotopic organization in the primary motor cortex. J Neurophysiol 2001;86(5):2125–43.CrossRefPubMedGoogle Scholar
  60. 60.
    Donoghue JP, Nurmikko A, Black M, Hochberg LR. Assistive technology and robotic control using motor cortex ensemble-based neural interface systems in humans with tetraplegia. J Physiol 2007;579(3):603–11.CrossRefPubMedPubMedCentralGoogle Scholar
  61. 61.
    Hotson G, McMullen DP, Fifer MS, Johannes MS, Katyal KD, Para MP, et al. Individual finger control of the modular prosthetic limb using high-density electrocorticography in a human subject. J Neural Eng 2016;13(2):026017.CrossRefPubMedPubMedCentralGoogle Scholar
  62. 62.
    Sharma G, Annetta N, Friedenberg DA, Bockbrader M. Advances in BCI: a neural bypass technology to reconnect the brain to the body. In: Brain-Computer Interface Research [Internet]. Springer, Cham; 2017 [cited 2017 Nov 13]. p. 9–20. (SpringerBriefs in Electrical and Computer Engineering). Available from: https://link.springer.com/chapter/10.1007/978-3-319-64373-1_2
  63. 63.
    Bouchard KE, Mesgarani N, Johnson K, Chang EF. Functional organization of human sensorimotor cortex for speech articulation. Nature 2013;495(7441):327–32.CrossRefPubMedPubMedCentralGoogle Scholar
  64. 64.
    Toyoda G, Brown EC, Matsuzaki N, Kojima K, Nishida M, Asano E. Electrocorticographic correlates of overt articulation of 44 English phonemes: intracranial recording in children with focal epilepsy. Clin Neurophysiol 2014;125(6):1129–37.CrossRefPubMedGoogle Scholar
  65. 65.
    Lotte F, Brumberg JS, Brunner P, Gunduz A, Ritaccio AL, Guan C, et al. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci [Internet]. 2015;9. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4338752/
  66. 66.
    Cheung C, Hamilton LS, Johnson K, Chang EF. The auditory representation of speech sounds in human motor cortex. elife 2016;5:e12577.CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Arsenault JS, Buchsbaum BR. No evidence of somatotopic place of articulation feature mapping in motor cortex during passive speech perception. Psychon Bull Rev 2016;23(4):1231–40.CrossRefPubMedGoogle Scholar
  68. 68.
    Glanz Iljina O, Derix J, Kaur R, Schulze-Bonhage A, Auer P, Aertsen A, et al. Real-life speech production and perception have a shared premotor-cortical substrate. Sci Rep 2018;8(1):8898.CrossRefPubMedPubMedCentralGoogle Scholar
  69. 69.
    Tankus A, Fried I, Shoham S. Structured neuronal encoding and decoding of human speech features. Nat Commun 2012;3:ncomms1995.CrossRefGoogle Scholar
  70. 70.
    Mugler EM, Patton JL, Flint RD, Wright ZA, Schuele SU, Rosenow J, et al. Direct classification of all American English phonemes using signals from functional speech motor cortex. J Neural Eng 2014;11(3):035015.CrossRefPubMedPubMedCentralGoogle Scholar
  71. 71.
    Mugler EM, Goldrick M, Rosenow JM, Tate MC, Slutzky MW. Decoding of articulatory gestures during word production using speech motor and premotor cortical activity. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2015. p. 5339–42.Google Scholar
  72. 72.
    Bouchard KE, Chang EF. Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2014. p. 6782–5.Google Scholar
  73. 73.
    Kanas VG, Mporas I, Benz HL, Sgarbas KN, Bezerianos A, Crone NE. Joint spatial-spectral feature space clustering for speech activity detection from ECoG signals. IEEE Trans Biomed Eng 2014;61(4):1241–50.CrossRefPubMedPubMedCentralGoogle Scholar
  74. 74.
    Mugler EM, Goldrick M, Slutzky MW. Cortical encoding of phonemic context during word production. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2014. p. 6790–3.Google Scholar
  75. 75.
    Zhang D, Gong E, Wu W, Lin J, Zhou W, Hong B. Spoken sentences decoding based on intracranial high gamma response using dynamic time warping. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2012. p. 3292–5.Google Scholar
  76. 76.
    Crone NE, Sinai A, Korzeniewska A. High-frequency gamma oscillations and human brain mapping with electrocorticography. In: Neuper C, Klimesch W, editors. Progress in Brain Research [Internet]. Elsevier; 2006 [cited 2017 Dec 3]. p. 275–95. (Event-Related Dynamics of Brain Oscillations; vol. 159). Available from: http://www.sciencedirect.com/science/article/pii/S0079612306590193
  77. 77.
    Buzsáki G, Wang X-J. Mechanisms of gamma oscillations. Annu Rev Neurosci 2012;35:203–25.CrossRefPubMedPubMedCentralGoogle Scholar
  78. 78.
    Yazdan-Shahmorad A, Kipke DR, Lehmkuhle MJ. High gamma power in ECoG reflects cortical electrical stimulation effects on unit activity in layers V/VI. J Neural Eng 2013;10(6):066002.CrossRefPubMedPubMedCentralGoogle Scholar
  79. 79.
    Martin S, Iturrate I, del Millán JR, Knight RT, Pasley BN. Decoding inner speech using electrocorticography: progress and challenges toward a speech prosthesis. Front Neurosci [Internet]. 2018 ;12 [cited 2018 Jul 18]. Available from: https://www.frontiersin.org/articles/10.3389/fnins.2018.00422/full
  80. 80.
    Vansteensel MJ, Pels EGM, Bleichner MG, Branco MP, Denison T, Freudenburg ZV, et al. Fully implanted brain–computer interface in a locked-in patient with ALS. N Engl J Med 2016;375(21):2060–6.CrossRefPubMedPubMedCentralGoogle Scholar
  81. 81.
    Guenther FH, Brumberg JS, Wright EJ, Nieto-Castanon A, Tourville JA, Panko M, et al. A wireless brain-machine interface for real-time speech synthesis. PLoS One 2009;4(12):e8218.CrossRefPubMedPubMedCentralGoogle Scholar
  82. 82.
    Herff C, Heger D, de Pesters A, Telaar D, Brunner P, Schalk G, et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front Neurosci [Internet]. 2015;9. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4464168/
  83. 83.
    Herff C, de Pesters A, Heger D, Brunner P, Schalk G, Schultz T, et al. Towards continuous speech recognition for BCI. 2017;21–9.Google Scholar
  84. 84.
    Moses DA, Mesgarani N, Leonard MK, Chang EF. Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. J Neural Eng 2016;13(5):056004.CrossRefPubMedPubMedCentralGoogle Scholar
  85. 85.
    Moses DA, Leonard MK, Chang EF. Real-time classification of auditory sentences using evoked cortical activity in humans. J Neural Eng 2018;15(3):036005.CrossRefPubMedGoogle Scholar
  86. 86.
    Bruns A. Fourier-, Hilbert- and wavelet-based signal analysis: are they really different approaches? J Neurosci Methods 2004;137(2):321–32.CrossRefPubMedGoogle Scholar
  87. 87.
    Bocquelet F, Hueber T, Girin L, Savariaux C, Yvert B. Real-time control of an articulatory-based speech synthesizer for brain computer interfaces. PLoS Comput Biol 2016;12(11):e1005119.CrossRefPubMedPubMedCentralGoogle Scholar
  88. 88.
    Bouchard KE, Conant DF, Anumanchipalli GK, Dichter B, Chaisanguanthum KS, Johnson K, et al. High-resolution, non-invasive imaging of upper vocal tract articulators compatible with human brain recordings. PLoS One 2016;11(3):e0151327.CrossRefPubMedPubMedCentralGoogle Scholar
  89. 89.
    Conant DF, Bouchard KE, Leonard MK, Chang EF. Human sensorimotor cortex control of directly measured vocal tract movements during vowel production. J Neurosci 2018;38(12):2955–66.CrossRefPubMedPubMedCentralGoogle Scholar
  90. 90.
    Herff C, Johnson G, Diener L, Shih J, Krusienski D, Schultz T. Towards direct speech synthesis from ECoG: a pilot study. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf 2016;2016:1540–3.Google Scholar
  91. 91.
    Akbari H, Khalighinejad B, Herrero J, Mehta A, Mesgarani N. Reconstructing intelligible speech from the human auditory cortex. bioRxiv. 2018 19;350124.Google Scholar
  92. 92.
    Milsap G, Collard M, Coogan C, Rabbani Q, Wang Y, Nathan E. C. Keyword spotting using human electrocorticographic recordings. Front Neurosci. 2019Google Scholar
  93. 93.
    Yeager JD, Phillips DJ, Rector DM, Bahr DF. Characterization of flexible ECoG electrode arrays for chronic recording in awake rats. J Neurosci Methods 2008;173(2):279–85.CrossRefPubMedPubMedCentralGoogle Scholar
  94. 94.
    Chao ZC, Nagasaka Y, Fujii N. Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkey. Front Neuroengineering [Internet]. 2010;3. [cited 2017 Jul 8] Available from: http://journal.frontiersin.org/article/10.3389/fneng.2010.00003/full
  95. 95.
    Rouse AG, Williams JJ, Wheeler JJ, Moran DW. Cortical adaptation to a chronic micro-electrocorticographic brain computer interface. J Neurosci 2013;33(4):1326–30.CrossRefPubMedPubMedCentralGoogle Scholar
  96. 96.
    Degenhart AD, Eles J, Dum R, Mischel JL, Smalianchuk I, Endler B, et al. Histological evaluation of a chronically-implanted electrocorticographic electrode grid in a non-human primate. J Neural Eng 2016;13(4):046019.CrossRefPubMedPubMedCentralGoogle Scholar
  97. 97.
    Duckrow RB, Tcheng TK. Daily variation in an intracranial EEG feature in humans detected by a responsive neurostimulator system. Epilepsia 2007;48(8):1614–20.CrossRefPubMedGoogle Scholar
  98. 98.
    Cook MJ, O’Brien TJ, Berkovic SF, Murphy M, Morokoff A, Fabinyi G, et al. Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study. Lancet Neurol 2013;12(6):563–71.CrossRefPubMedGoogle Scholar
  99. 99.
    Spencer DC, Sun FT, Brown SN, Jobst BC, Fountain NB, Wong VSS, et al. Circadian and ultradian patterns of epileptiform discharges differ by seizure-onset location during long-term ambulatory intracranial monitoring. Epilepsia 2016;57(9):1495–502.CrossRefPubMedGoogle Scholar
  100. 100.
    Smart O, Rolston JD, Epstein CM, Gross RE. Hippocampal seizure-onset laterality can change over long timescales: a same-patient observation over 500 days. Epilepsy Behav Case Rep 2013;1:56–61.CrossRefPubMedPubMedCentralGoogle Scholar
  101. 101.
    Rao VR, Leonard MK, Kleen JK, Lucas BA, Mirro EA, Chang EF. Chronic ambulatory electrocorticography from human speech cortex. NeuroImage 2017;153:273–82.CrossRefPubMedPubMedCentralGoogle Scholar
  102. 102.
    Bundy DT, Zellmer E, Gaona CM, Sharma M, Szrama N, Hacker C, et al. Characterization of the effects of the human dura on macro- and micro-electrocorticographic recordings. J Neural Eng 2014;11(1):016006.CrossRefPubMedPubMedCentralGoogle Scholar
  103. 103.
    Flint RD, Rosenow JM, Tate MC, Slutzky MW. Continuous decoding of human grasp kinematics using epidural and subdural signals. J Neural Eng 2016;14(1):016005.CrossRefPubMedPubMedCentralGoogle Scholar
  104. 104.
    Leuthardt EC, Gaona C, Sharma M, Szrama N, Roland J, Freudenberg Z, et al. Using the electrocorticographic speech network to control a brain-computer interface in humans. J Neural Eng 2011;8(3):036004.CrossRefPubMedPubMedCentralGoogle Scholar
  105. 105.
    Vale FL, Pollock G, Dionisio J, Benbadis SR, Tatum WO. Outcome and complications of chronically implanted subdural electrodes for the treatment of medically resistant epilepsy. Clin Neurol Neurosurg Assen 2013;115(7):985–90.CrossRefGoogle Scholar
  106. 106.
    Hamer HM, Morris HH, Mascha EJ, Karafa MT, Bingaman WE, Bej MD, et al. Complications of invasive video-EEG monitoring with subdural grid electrodes. Neurology 2002;58(1):97–103.CrossRefPubMedGoogle Scholar
  107. 107.
    Hedegärd E, Bjellvi J, Edelvik A, Rydenhag B, Flink R, Malmgren K. Complications to invasive epilepsy surgery workup with subdural and depth electrodes: a prospective population-based observational study. J Neurol Neurosurg Psychiatry 2014;85(7):716–20.CrossRefPubMedGoogle Scholar
  108. 108.
    Wellmer J, von der Groeben F, Klarmann U, Weber C, Elger CE, Urbach H, et al. Risks and benefits of invasive epilepsy surgery workup with implanted subdural and depth electrodes. Epilepsia 2012;53(8):1322–32.CrossRefPubMedGoogle Scholar
  109. 109.
    Mathon B, Clemenceau S, Hasboun D, Habert M-O, Belaid H, Nguyen-Michel V-H, et al. Safety profile of intracranial electrode implantation for video-EEG recordings in drug-resistant focal epilepsy. J Neurol 2015;262(12):2699–712.CrossRefPubMedGoogle Scholar
  110. 110.
    Nagahama Y, Schmitt AJ, Nakagawa D, Vesole AS, Kamm J, Kovach CK, et al. Intracranial EEG for seizure focus localization: evolving techniques, outcomes, complications, and utility of combining surface and depth electrodes. J Neurosurg 2018;1(aop):1–13.Google Scholar
  111. 111.
    Voorhies JM, Cohen-Gadol A. Techniques for placement of grid and strip electrodes for intracranial epilepsy surgery monitoring: pearls and pitfalls. Surg Neurol Int [Internet] 2013;4. [cited 2017 Dec 3] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3740610/
  112. 112.
    Kourbeti IS, Vakis AF, Ziakas P, Karabetsos D, Potolidis E, Christou S, et al. Infections in patients undergoing craniotomy: risk factors associated with post-craniotomy meningitis. J Neurosurg 2014;122(5):1113–9.PubMedGoogle Scholar
  113. 113.
    Patel KS, Goldenberg B, Schwartz TH. Betadine irrigation and post-craniotomy wound infection. Clin Neurol Neurosurg 2014;118:49–52.CrossRefPubMedPubMedCentralGoogle Scholar
  114. 114.
    Weltman A, Yoo J, Meng E. Flexible, penetrating brain probes enabled by advances in polymer microfabrication. Micromachines 2016;7(10):180.CrossRefPubMedCentralGoogle Scholar
  115. 115.
    Matsuo T, Kawasaki K, Osada T, Sawahata H, Suzuki T, Shibata M, et al. Intrasulcal electrocorticography in macaque monkeys with minimally invasive neurosurgical protocols. Front Syst Neurosci [Internet]. 2011;5. [cited 2017 Dec 5] Available from: https://www.frontiersin.org/articles/10.3389/fnsys.2011.00034/full
  116. 116.
    Castagnola E, Maiolo L, Maggiolini E, Minotti A, Marrani M, Maita F, et al. PEDOT-CNT-coated low-impedance, ultra-flexible, and brain-conformable micro-ECoG arrays. IEEE Trans Neural Syst Rehabil Eng 2015;23(3):342–50.CrossRefPubMedGoogle Scholar
  117. 117.
    Kaiju T, Doi K, Yokota M, Watanabe K, Inoue M, Ando H, et al. High spatiotemporal resolution ECoG recording of somatosensory evoked potentials with flexible micro-electrode arrays. Front Neural Circuits [Internet]. 2017;11. [cited 2017 Dec 3] Available from: https://www.frontiersin.org/articles/10.3389/fncir.2017.00020/full
  118. 118.
    Xie K, Zhang S, Dong S, Li S, Yu C, Xu K, et al. Portable wireless electrocorticography system with a flexible microelectrodes array for epilepsy treatment. Sci Rep 2017;7(1):7808.CrossRefPubMedPubMedCentralGoogle Scholar
  119. 119.
    Wang PT, King CE, McCrimmon CM, Lin JJ, Sazgar M, Hsu FPK, et al. Comparison of decoding resolution of standard and high-density electrocorticogram electrodes. J Neural Eng 2016;13(2):026016.CrossRefPubMedGoogle Scholar
  120. 120.
    Muller L, Hamilton LS, Edwards E, Bouchard KE, Chang EF. Spatial resolution dependence on spectral frequency in human speech cortex electrocorticography. J Neural Eng 2016;13(5):056013.CrossRefPubMedPubMedCentralGoogle Scholar
  121. 121.
    Thongpang S, Richner TJ, Brodnick SK, Schendel A, Kim J, Wilson JA, et al. A micro-electrocorticography platform and deployment strategies for chronic BCI applications. Clin EEG Neurosci 2011;42(4):259–65.CrossRefPubMedPubMedCentralGoogle Scholar
  122. 122.
    Wodlinger B, Degenhart AD, Collinger JL, Tyler-Kabara EC, Wang W. The impact of electrode characteristics on electrocorticography (ECoG). In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2011. p. 3083–6.Google Scholar
  123. 123.
    Freeman WJ, Rogers LJ, Holmes MD, Silbergeld DL. Spatial spectral analysis of human electrocorticograms including the alpha and gamma bands. J Neurosci Methods 2000;95(2):111–21.CrossRefPubMedGoogle Scholar
  124. 124.
    Blackrock Microsystems [Internet]. [cited 2018 Jul 20]. Available from: http://blackrockmicro.com/
  125. 125.
    BrainGate [Internet]. BrainGate. [cited 2018 Nov 6]. Available from: https://www.braingate.org/
  126. 126.
    Revolutionizing prosthetics [Internet]. JHUAPL. [cited 2018 Nov 6]. Available from: https://www.jhuapl.edu/prosthetics/
  127. 127.
    NeuroLife neural bypass technology [Internet]. Battelle. [cited 2018 Nov 6]. Available from: https://www.battelle.org/government-offerings/health/medical-devices/neurotechnology/neurolife-neural-bypass-technology
  128. 128.
    Matsushita K, Hirata M, Suzuki T, Ando H, Ota Y, Sato F, et al. Development of an implantable wireless ECoG 128ch recording device for clinical brain machine interface. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2013. p. 1867–70.Google Scholar
  129. 129.
    Yoshimoto S, Araki T, Uemura T, Nezu T, Sekitani T, Suzuki T, et al. Implantable wireless 64-channel system with flexible ECoG electrode and optogenetics probe. In: 2016 IEEE Biomedical Circuits and Systems Conference (BioCAS). 2016. p. 476–9.Google Scholar
  130. 130.
    Reddy DR. Speech recognition by machine: a review. Proc IEEE 1976;64(4):501–31.CrossRefGoogle Scholar
  131. 131.
    Xiong W, Droppo J, Huang X, Seide F, Seltzer M, Stolcke A, et al. Achieving human parity in conversational speech recognition. ArXiv161005256 Cs [Internet]. 2016 [cited 2017 Dec 4]; Available from: http://arxiv.org/abs/1610.05256
  132. 132.
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521(7553):436–44.CrossRefGoogle Scholar
  133. 133.
    Na S, Yoo S. Allowable propagation delay for VoIP calls of acceptable quality. In: Advanced Internet Services and Applications [Internet]. Springer, Berlin, Heidelberg; 2002 [cited 2017 Dec 4]. p. 47–55. (Lecture Notes in Computer Science). Available from: https://link.springer.com/chapter/10.1007/3-540-45639-2_6
  134. 134.
    Stuart A, Kalinowski J, Rastatter MP, Lynch K. Effect of delayed auditory feedback on normal speakers at two speech rates. J Acoust Soc Am 2002;111(5):2237–41.CrossRefPubMedGoogle Scholar
  135. 135.
    Stuart A, Kalinowski J. Effect of delayed auditory feedback, speech rate, and sex on speech production. Percept Mot Skills 2015;120(3):747–65.CrossRefPubMedGoogle Scholar
  136. 136.
    Badian M, Appel E, Palm D, Rupp W, Sittig W, Taeuber K. Standardized mental stress in healthy volunteers induced by delayed auditory feedback (DAF). Eur J Clin Pharmacol 1979;16(3):171–6.CrossRefPubMedGoogle Scholar
  137. 137.
    Salthouse TA. Effects of age and skill in typing. J Exp Psychol Gen 1984;113(3):345–71.CrossRefPubMedGoogle Scholar
  138. 138.
    Norvig P. English Letter frequency counts: Mayzner Revisited or ETAOIN SRHLDCU [Internet]. [cited 2018 Jun 15]. Available from: http://norvig.com/mayzner.html
  139. 139.
    The 44 sounds (phonemes) of English [Internet]. DSF Literacy Resources; [cited 2017 Dec 17]. Available from: https://www.dyslexia-reading-well.com/support-files/the-44-phonemes-of-english.pdf
  140. 140.
    Yuan J, Liberman M, Cieri C. Towards an integrated understanding of speaking rate in conversation. In: Conversation Interspeech. 2006. p. 541–544.Google Scholar
  141. 141.
    Krauledat M, Schröder M, Blankertz B, Müller K-R. Reducing calibration time for brain-computer interfaces: a clustering approach. In: Advances in Neural Information Processing Systems 19. MIT Press; 2007. p. 753–760.Google Scholar
  142. 142.
    Brandman DM, Hosman T, Saab J, Burkhart MC, Shanahan BE, Ciancibello JG, et al. Rapid calibration of an intracortical brain–computer interface for people with tetraplegia. J Neural Eng 2018;15(2):026007.CrossRefPubMedGoogle Scholar
  143. 143.
    Verhoeven T, Hübner D, Tangermann M, Müller KR, Dambre J, Kindermans PJ. Improving zero-training brain-computer interfaces by mixing model estimators. J Neural Eng 2017;14(3):036021.CrossRefPubMedGoogle Scholar
  144. 144.
    Hedberg SR. Dictating this article to my computer: automatic speech recognition is coming of age. IEEE Expert 1997;12(6):9–11.CrossRefGoogle Scholar
  145. 145.
    Huang X, Baker J, Reddy R. A historical perspective of speech recognition. Commun ACM 2014;57(1):94–103.CrossRefGoogle Scholar
  146. 146.
    Protalinski E. Google’s speech recognition technology now has a 4.9% word error rate [Internet]. VentureBeat. 2017 [cited 2017 Dec 5]. Available from: https://venturebeat.com/2017/05/17/googles-speech-recognition-technology-now-has-a-4-9-word-error-rate/

Copyright information

© The American Society for Experimental NeuroTherapeutics, Inc. 2019

Authors and Affiliations

  • Qinwan Rabbani
    • 1
    Email author
  • Griffin Milsap
    • 2
  • Nathan E. Crone
    • 3
  1. 1.Department of Electrical EngineeringThe Johns Hopkins University Whiting School of EngineeringBaltimoreUSA
  2. 2.Department of Biomedical EngineeringThe Johns Hopkins University School of MedicineBaltimoreUSA
  3. 3.Department of NeurologyThe Johns Hopkins University School of MedicineBaltimoreUSA

Personalised recommendations