Skip to main content
SpringerLink
Log in

DCLL—A Deep Network for Possible Real-Time Decoding of Imagined Words

  • Conference paper
  • First Online:
International Symposium on Intelligent Informatics (ISI 2022)

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 333))

Included in the following conference series:

  • 166 Accesses

Abstract

We present a novel architecture for classifying imagined words from electroencephalogram (EEG) captured during speech imagery. The proposed architecture employs a sliding window with overlap for data augmentation (DA) and common spatial pattern (CSP) in order to derive the features. The dimensionality of features is reduced using linear discriminant analysis (LDA). Long short-term memory (LSTM) along with majority voting is used as the classifier. We call the proposed architecture the DCLL (DA-CSP-LDA-LSTM) architecture. On a publicly available imagined word dataset, the DCLL architecture achieves an accuracy of 85.2% for classifying the imagined words “in” and “cooperate”. Although this is around 7% less than the best result in the literature on this dataset, the DCLL architecture is roughly 300 times faster than the latter, making it a potential candidate for imagined word-based online BCI systems where the EEG signal needs to be classified in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. S.N. Abdulkader, A. Atia, M.S.M. Mostafa, Brain computer interfacing: applications and challenges. Egypt. Inf. J. 16(2), 213–230 (2015)

    Google Scholar 

  2. C. Herff, D. Heger, A. De Pesters, D. Telaar, P. Brunner, G. Schalk, T. Schultz, Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. Neurosci. 9, 217 (2015)

    Article  Google Scholar 

  3. C. Herff, G. Johnson, L. Diener, J. Shih, D. Krusienski, T. Schultz, Towards direct speech synthesis from ECoG: A pilot study, in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2016), pp. 1540–1543

    Google Scholar 

  4. E.M. Mugler, C.A. Ruf, S. Halder, M. Bensch, A. Kubler, Design and implementation of a p300-based brain-computer interface for controlling an internet browser. IEEE Trans. Neural Syst. Rehabil. Eng. 18(6), 599–609 (2010)

    Article  Google Scholar 

  5. M. Xu, X. Xiao, Y. Wang, H. Qi, T.P. Jung, D. Ming, A brain-computer interface based on miniature-event-related potentials induced by very small lateral visual stimuli. IEEE Trans. Biomed. Eng. 65(5), 1166–1175 (2018)

    Article  Google Scholar 

  6. J.S. Brumberg, E.J. Wright, D.S. Andreasen, F.H. Guenther, P.R. Kennedy, Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech motor cortex. Front. Neurosci. 5, 65 (2011)

    Google Scholar 

  7. P. Kennedy, A. Cervantes, C. Gambrell, P. Ehirim, Advances in the development of a speech prosthesis, in Direct and Indirect Benefits of Translingual Neurostimulation Technology for Neurorehabilitation of Chronic Stroke Symptoms (2017), p. 1

    Google Scholar 

  8. G.H. Wilson, S.D. Stavisky, F.R. Willett, D.T. Avansino, J.N. Kelemen, L.R. Hochberg, J.M. Henderson, S. Druckmann, K.V. Shenoy, Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J. Neural Eng. 17(6), 066007 (2020)

    Article  Google Scholar 

  9. D. Dash, A. Wisler, P. Ferrari, E.M. Davenport, J. Maldjian, J. Wang, MEG sensor selection for neural speech decoding. IEEE Access 8, 182320–182337 (2020)

    Article  Google Scholar 

  10. F. Destoky, M. Philippe, J. Bertels, M. Verhasselt, N. Coquelet, M. Vander Ghinst, V. Wens, X. De Tiège, M. Bourguignon, Comparing the potential of MEG and EEG to uncover brain tracking of speech temporal envelope. Neuroimage 184, 201–213 (2019)

    Article  Google Scholar 

  11. S.S. Yoo, T. Fairneny, N.K. Chen, S.E. Choo, L.P. Panych, H. Park, S.Y. Lee, F.A. Jolesz, Brain-computer interface using fMRI: spatial navigation by thoughts. Neuroreport 15(10), 1591–1595 (2004)

    Article  Google Scholar 

  12. K. Abe, T. Takahashi, Y. Takikawa, H. Arai, S. Kitazawa, Applying independent component analysis to detect silent speech in magnetic resonance imaging signals. Eur. J. Neurosci. 34(8), 1189–1199 (2011)

    Article  Google Scholar 

  13. C. Herff, F. Putze, D. Heger, C. Guan, T. Schultz, Speaking mode recognition from functional near infrared spectroscopy, in 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE, 2012), pp. 1715–1718

    Google Scholar 

  14. E.N. Kamavuako, U.A. Sheikh, S.O. Gilani, M. Jamil, I.K. Niazi, Classification of overt and covert speech for near-infrared spectroscopy-based brain computer interface. Sensors 18(9), 2989 (2018)

    Article  Google Scholar 

  15. A.R. Sereshkeh, R. Yousefi, A.T. Wong, T. Chau, Online classification of imagined speech using functional near-infrared spectroscopy signals. J. Neural Eng. 16(1), 016005 (2018)

    Article  Google Scholar 

  16. I.A. Fouad, F.E.Z.M. Labib, M.S. Mabrouk, A.A. Sharawy, A.Y. Sayed, Improving the performance of P300 BCI system using different methods. Netw. Model. Anal. Health Inf. Bioinform. 9(1), 1–13 (2020)

    Google Scholar 

  17. E.W. Sellers, D.J. Krusienski, D.J. McFarland, T.M. Vaughan, J.R. Wolpaw, A p300 event-related potential brain-computer interface (bci): the effects of matrix size and inter stimulus interval on performance. Biol. Psychol. 73(3), 242–252 (2006)

    Article  Google Scholar 

  18. J. Kevric, A. Subasi, Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomed. Signal Process. Control 31, 398–406 (2017)

    Article  Google Scholar 

  19. G. Onose, C. Grozea, A. Anghelescu, C. Daia, C. Sinescu, A. Ciurea, T. Spircu, A. Mirea, I. Andone, A. Spânu et al., On the feasibility of using motor imagery EEG-based brain-computer interface in chronic tetraplegics for assistive robotic arm control: a clinical test and long-term post-trial follow-up. Spinal Cord 50(8), 599–608 (2012)

    Article  Google Scholar 

  20. M.K. Ojha, M.K. Mukul, Detection of target frequency from SSVEP signal using empirical mode decomposition for SSVEP based BCI inference system. Wireless Personal Commun. 1–13 (2020)

    Google Scholar 

  21. G.R. Müller-Putz, R. Scherer, C. Brauneis, G. Pfurtscheller, Steady-state visual evoked potential (ssvep)-based communication: impact of harmonic frequency components. J. Neural Eng. 2(4), 123 (2005)

    Article  Google Scholar 

  22. C. Han, G. Xu, J. Xie, C. Chen, S. Zhang, Highly interactive brain-computer interface based on flicker-free steady-state motion visual evoked potential. Sci. Rep. 8(1), 1–13 (2018)

    Google Scholar 

  23. B. Allison, B. Graimann, A. Gräser, Why use a BCI if you are healthy, in ACE Workshop-Brain-Computer Interfaces and Games (2007), pp. 7–11

    Google Scholar 

  24. R. Bogue, Brain-computer interfaces: control by thought. Indus. Robot Int. J. (2010)

    Google Scholar 

  25. J.T. Panachakel, A. Ramakrishnan, Decoding covert speech from EEG-a comprehensive review. Front. Neurosci. 15, 392 (2021)

    Article  Google Scholar 

  26. C.H. Nguyen, G.K. Karavas, P. Artemiadis, Adaptive multi-degree of freedom brain computer interface using online feedback: Towards novel methods and metrics of mutual adaptation between humans and machines for BCI. PloS One 14(3), e0212620 (2019)

    Article  Google Scholar 

  27. A.R. Sereshkeh, R. Trott, A. Bricout, T. Chau, Online EEG classification of covert speech for brain-computer interfacing. Int. J. Neural Syst. 27(08), 1750033 (2017)

    Article  Google Scholar 

  28. N. Naseer, M.J. Hong, K.S. Hong, Online binary decision decoding using functional near-infrared spectroscopy for the development of brain-computer interface. Exp. Brain Res. 232(2), 555–564 (2014)

    Article  Google Scholar 

  29. G. Gallegos-Ayala, A. Furdea, K. Takano, C.A. Ruf, H. Flor, N. Birbaumer, Brain communication in a completely locked-in patient using bedside near-infrared spectroscopy. Neurology 82(21), 1930–1932 (2014)

    Article  Google Scholar 

  30. C.H. Nguyen, G.K. Karavas, P. Artemiadis, Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features. J. Neural Eng. 15(1), 016002 (2017)

    Article  Google Scholar 

  31. P. He, G. Wilson, C. Russell, Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med. Biol. Eng. Comput. 42(3), 407–412 (2004)

    Article  Google Scholar 

  32. E. Lashgari, D. Liang, U. Maoz, Data augmentation for deep-learning-based electroencephalography. J. Neurosci. Methods 108885 (2020)

    Google Scholar 

  33. A. O’Shea, G. Lightbody, G. Boylan, A. Temko, Neonatal seizure detection using convolutional neural networks, in 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP) (IEEE, 2017), pp. 1–6

    Google Scholar 

  34. N.S. Kwak, K.R. Müller, S.W. Lee, A convolutional neural network for steady state visual evoked potential classification under ambulatory environment. PloS One 12(2), e0172578 (2017)

    Article  Google Scholar 

  35. I. Ullah, M. Hussain, H. Aboalsamh et al., An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Syst. Appl. 107, 61–71 (2018)

    Article  Google Scholar 

  36. I. Majidov, T. Whangbo, Efficient classification of motor imagery electroencephalography signals using deep learning methods. Sensors 19(7), 1736 (2019)

    Article  Google Scholar 

  37. Y. Luo, B.L. Lu, EEG data augmentation for emotion recognition using a conditional Wasserstein GAN, in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2018), pp. 2535–2538

    Google Scholar 

  38. Z. Wei, J. Zou, J. Zhang, J. Xu, Automatic epileptic EEG detection using convolutional neural network with improvements in time-domain. Biomed. Signal Process. Control 53, 101551 (2019)

    Article  Google Scholar 

  39. S. Chang, H. Jun, Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives. J. Asian Architect. Build. Eng. 18(5), 381–391 (2019)

    Article  Google Scholar 

  40. J.T. Panachakel, G.P. Kumar, K. Sharma, A. Ramakrishnan, Automated classification of EEG into meditation and non-meditation epochs using common spatial pattern, linear discriminant analysis, and LSTM, in TENCON 2021-2021 IEEE Region 10 Conference (IEEE, 2021), pp. 1–6

    Google Scholar 

  41. P. Goel, R. Joshi, M. Sur, H.A. Murthy, A common spatial pattern approach for classification of mental counting and motor execution EEG, in International Conference on Intelligent Human Computer Interaction (Springer, 2018), pp. 26–35

    Google Scholar 

  42. J.T. Panachakel, N.N. Vinayak, M. Nunna, A.G. Ramakrishnan, K. Sharma, An improved EEG acquisition protocol facilitates localized neural activation, in Advances in Communication Systems and Networks (Springer, 2020), pp. 267–281

    Google Scholar 

  43. J.T. Panachakel, G.P. Kumar, K. Sharma, A. Ramakrishnan, Binary classification of meditative state from the resting state using EEG, in 2021 IEEE 18th India Council International Conference (INDICON) (IEEE, 2021), pp. 1–6

    Google Scholar 

  44. J.T. Panachakel, A. Ramakrishnan, Decoding imagined speech from EEG using transfer learning. IEEE Access (2021)

    Google Scholar 

  45. J.T. Panachakel, A. Ramakrishnan, T. Ananthapadmanabha, A novel deep learning architecture for decoding imagined speech from EEG. arXiv preprint arXiv:2003.09374 (2020)

  46. J.T. Panachakel, A. Ramakrishnan, A. Anusha, K. Sharma, Can we identify the category of imagined phoneme from EEG? in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (IEEE, 2021)

    Google Scholar 

  47. J.T. Panachakel, A. Ramakrishnan, Classification of phonological categories in imagined speech using phase synchronization measure, in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (IEEE, 2021)

    Google Scholar 

  48. V. Benzy, A. Vinod, R. Subasree, S. Alladi, K. Raghavendra, Motor imagery hand movement direction decoding using brain computer interface to aid stroke recovery and rehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 28(12), 3051–3062 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MILE Lab, Indian Institute of Science, Bengaluru, India

    Jerrin Thomas Panachakel & A. G. Ramakrishnan

Authors
  1. Jerrin Thomas Panachakel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. G. Ramakrishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jerrin Thomas Panachakel .

Editor information

Editors and Affiliations

  1. School of Computer Science & Engineering (SoCSE), Innovation and Technology (KUDSIT), Kerala University of Digital Sciences, Trivandrum, Kerala, India

    Sabu M. Thampi

  2. Dept of Computer Science & Engineering, Indian Inst of Technology Kharagpur, Kharagpur, India

    Jayanta Mukhopadhyay

  3. PAN, Systems Research Institute, Warszawa, Poland

    Marcin Paprzycki

  4. Department of Computer Science and Information Engineering (CSIE), Providence University, Taichung, Taiwan

    Kuan-Ching Li

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Panachakel, J.T., Ramakrishnan, A.G. (2023). DCLL—A Deep Network for Possible Real-Time Decoding of Imagined Words. In: Thampi, S.M., Mukhopadhyay, J., Paprzycki, M., Li, KC. (eds) International Symposium on Intelligent Informatics. ISI 2022. Smart Innovation, Systems and Technologies, vol 333. Springer, Singapore. https://doi.org/10.1007/978-981-19-8094-7_1

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-8094-7_1

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-8093-0

  • Online ISBN: 978-981-19-8094-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation