Skip to main content
SpringerLink
Log in

A hybrid deep learning framework for automated visual image classification using EEG signals

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In recent years, the concept of reading people's minds while performing specific tasks has grown in popularity, especially in the field of brain–computer interface systems. The goal of this research is to offer a new comprehensive framework for visual image classification utilizing EEG signals. Here, LSTM network is chosen to extract the feature of the EEG signal. To improve the classification accuracy in comparison to recent works in this field, a ResNet is used to extract the feature from the images, and a fuzzy regression is used to map the features extracted from the images on the features extracted from the EEG signal. Because medical data usually have uncertainty and ambiguity, and the number of data samples is small, in this research, interval type-2 fuzzy regression is used. This study makes use of data from Stanford University. Classification accuracy, precision, recall, and F1 score are the measures used to evaluate outcomes. In this case, the LSTM network is able to classify images based on EEG signals with 55.55% accuracy and 55.73% precision. The findings of utilizing ResNet and interval type-2 fuzzy regression suggest that regression can significantly enhance classification accuracy. Classification accuracy and precision obtained with interval type-2 fuzzy regression and SVM classifier are 66.67% and 66.66%, respectively, for all participants. These results outperform classification accuracy and precision obtained using type-1 fuzzy, neural network and polynomial regression. In addition, the mean accuracy for all participants obtained in this article is 4.09% greater than the best result reported in a previous article using the same database. This leads to the conclusion that fuzzy regression, especially interval type-2 fuzzy regression, performs better in high uncertainty environments. Therefore, it is concluded that the method proposed in this research can be used for decoding brain response to visual image stimuli and classifying the stimulus images with acceptable accuracy, thereby advancing the goal of mind reading.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

All EEG datasets used for classification are available from the Stanford Digital Repository, http://purl.stanford.edu/ bq914sc3730.

References

  1. Cox DD, Savoy RL (2003) Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19(2):261–270

    Article  Google Scholar 

  2. Nguyen T et al (2015) EEG signal classification for BCI applications by wavelets and interval type-2 fuzzy logic systems. Expert Syst Appl 42(9):4370–4380

    Article  Google Scholar 

  3. Mashatan S, Ghassemi F (2019) Brain functional connectivity in SSVEP BasedBCI speller. In: 2019 26th National and 4th international Iranian conference on biomedical engineering (ICBME). IEEE

  4. Behzadnia A et al (2018) The neural correlation of sustained attention in performing conjunctive continuous performance task: an event-related potential study. NeuroReport 29(11):954–961

    Article  Google Scholar 

  5. Taghizadeh-Sarabi M, Daliri MR, Niksirat KS (2015) Decoding objects of basic categories from electroencephalographic signals using wavelet transform and support vector machines. Brain Topogr 28:33–46

    Article  Google Scholar 

  6. Daliri MR et al (2013) EEG signature of object categorization from event-related potentials. J Med Signals Sens 3(1):37

    Article  Google Scholar 

  7. Jafakesh S et al (2016) Decoding of object categories from brain signals using cross frequency coupling methods. Biomed Signal Process Control 27:60–67

    Article  Google Scholar 

  8. Tafreshi TF, Daliri MR, Ghodousi M (2019) Functional and effective connectivity based features of EEG signals for object recognition. Cogn Neurodyn 13:555–566

    Article  Google Scholar 

  9. Spampinato C et al (2017) Deep learning human mind for automated visual classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  10. Zheng X et al (2020) Ensemble deep learning for automated visual classification using EEG signals. Pattern Recognit 102:107147

    Article  Google Scholar 

  11. Fares A, Zhong S, Jiang J (2018) Region level bi-directional deep learning framework for eeg-based image classification. In: 2018 IEEE international conference on bioinformatics and biomedicine (BIBM). IEEE

  12. McCartney B, Martinez-del-Rincon J, Devereux B, Murphy B (2019) Towards a real-world brain-computer interface for image retrieval. BioRxiv 576983.

  13. Kaneshiro B et al (2015) A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. PLoS ONE 10(8):e0135697

    Article  Google Scholar 

  14. Murphy B, Baroni M, Poesio M (2009) EEG responds to conceptual stimuli and corpus semantics. In: Proceedings of the 2009 conference on empirical methods in natural language processing

  15. McCartney B, Devereux B, Martinez-del-Rincon J (2022) A zero-shot deep metric learning approach to brain–computer interfaces for image retrieval. Knowl Based Syst 246:108556

    Article  Google Scholar 

  16. Wirawan IMA et al (2022) Modified weighted mean filter to improve the baseline reduction approach for emotion recognition. Emerg Sci J 6(6):1255–1273

    Article  Google Scholar 

  17. http://www.neurobs.com

  18. Makeig S et al (1995) Independent component analysis of electroencephalographic data. 8

  19. Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134(1):9–21

    Article  Google Scholar 

  20. Hochreiter, S., & Schmidhuber, J. (1996). LSTM can solve hard long time lag problems. Advances in neural information processing systems, 9.

  21. Gers F (2001) Long short-term memory in recurrent neural networks. Verlag nicht ermittelbar

  22. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 60(6):84–90

    Google Scholar 

  23. He K et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  24. Wen H et al (2018) Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Sci Rep 8(1):1–17

    MathSciNet  Google Scholar 

  25. Gu J et al (2018) Recent advances in convolutional neural networks. Pattern Recognit 77:354–377

    Article  Google Scholar 

  26. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  27. Piccialli F et al (2021) A survey on deep learning in medicine: why, how and when? Inf Fusion 66:111–137

    Article  Google Scholar 

  28. Wang F et al (2017) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  29. Dreiseitl S, Ohno-Machado L (2002) Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform 35(5–6):352–359

    Article  Google Scholar 

  30. Nasiri M, Shanbeh M, Tavanai H (2005) Comparison of statistical regression, fuzzy regression and artificial neural network modeling methodologies in polyester dyeing. In: International conference on computational intelligence for modelling, control and automation and international conference on intelligent agents, web technologies and internet commerce (CIMCA-IAWTIC'06). IEEE

  31. Phibanchon S et al (2007) An adaptive fuzzy regression model for the prediction of dichotomous response variables. In 2007 International conference on computational science and its applications (ICCSA 2007). IEEE

  32. Tu JV (1996) Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. J Clin Epidemiol 49(11):1225–1231

    Article  Google Scholar 

  33. Zadeh LA (1975) The concept of a linguistic variable and its application to approximate reasoning-III. Inf Sci 9(1):43–80

    Article  MathSciNet  MATH  Google Scholar 

  34. Coupland S, John R (2008) Type-2 fuzzy logic and the modelling of uncertainty. In: Fuzzy sets and their extensions: representation, aggregation and models. Springer, Berlin, Heidelberg, pp 3–22

  35. Hisdal E (1981) The IF THEN ELSE statement and interval-valued fuzzy sets of higher type. Int J Man Mach Stud 15(4):385–455

    Article  MathSciNet  MATH  Google Scholar 

  36. Takagi T, Sugeno M (1985) Fuzzy identification of systems and its applications to modeling and control. IEEE Trans Syst Man Cybern 1:116–132

    Article  MATH  Google Scholar 

  37. Wu D (2012) Twelve considerations in choosing between Gaussian and trapezoidal membership functions in interval type-2 fuzzy logic controllers. In: 2012 IEEE international conference on fuzzy systems. IEEE

  38. Li R et al (2020) The perils and pitfalls of block design for EEG classification experiments. IEEE Trans Pattern Anal Mach Intell 43(1):316–333

    Google Scholar 

  39. Li R, Johansen JS, Ahmed H, Ilyevsky TV, Wilbur RB, Bharadwaj HM, Siskind JM (2018) Training on the test set? an analysis of spampinato et al. [31]. arXiv preprint arXiv:1812.07697

  40. Du C et al (2018) Multi-view adversarially learned inference for cross-domain joint distribution matching. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining

  41. Kavasidis I et al (2017) Brain2image: converting brain signals into images. In: Proceedings of the 25th ACM international conference on multimedia

  42. Palazzo S et al (2020) Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Trans Pattern Anal Mach Intell 43(11):3833–3849

    Article  Google Scholar 

  43. Palazzo S et al (2017) Generative adversarial networks conditioned by brain signals. In: Proceedings of the IEEE international conference on computer vision

  44. Tirupattur P et al (2018) Thoughtviz: visualizing human thoughts using generative adversarial network. In: Proceedings of the 26th ACM international conference on multimedia

  45. Bisserier A, Boukezzoula R, Galichet SJ (2010) A revisited approach to linear fuzzy regression using trapezoidal fuzzy intervals. Inf Sci 180(19):3653–3673

    Article  MathSciNet  MATH  Google Scholar 

  46. Wardoyo R, Wirawan IMA, Pradipta IGA (2022) Oversampling approach using radius-SMOTE for imbalance electroencephalography datasets. Emerg Sci J 6(2):382–398

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Amirkabir University of Technology, Tehran, Iran

    Hajar Ahmadieh, Farnaz Gassemi & Mohammad Hasan Moradi

Authors
  1. Hajar Ahmadieh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Farnaz Gassemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Hasan Moradi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Farnaz Gassemi.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmadieh, H., Gassemi, F. & Moradi, M.H. A hybrid deep learning framework for automated visual image classification using EEG signals. Neural Comput & Applic 35, 20989–21005 (2023). https://doi.org/10.1007/s00521-023-08870-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08870-w

Keywords

Search

Navigation