A study of decoding human brain activities from simultaneous data of EEG and fMRI using MVPA

  • Raheel Zafar
  • Nidal Kamel
  • Mohamad Naufal
  • Aamir Saeed Malik
  • Sarat C. Dass
  • Rana Fayyaz Ahmad
  • Jafri M. Abdullah
  • Faruque Reza
Scientific Paper


Neuroscientists have investigated the functionality of the brain in detail and achieved remarkable results but this area still need further research. Functional magnetic resonance imaging (fMRI) is considered as the most reliable and accurate technique to decode the human brain activity, on the other hand electroencephalography (EEG) is a portable and low cost solution in brain research. The purpose of this study is to find whether EEG can be used to decode the brain activity patterns like fMRI. In fMRI, data from a very specific brain region is enough to decode the brain activity patterns due to the quality of data. On the other hand, EEG can measure the rapid changes in neuronal activity patterns due to its higher temporal resolution i.e., in msec. These rapid changes mostly occur in different brain regions. In this study, multivariate pattern analysis (MVPA) is used both for EEG and fMRI data analysis and the information is extracted from distributed activation patterns of the brain. The significant information among different classes is extracted using two sample t test in both data sets. Finally, the classification analysis is done using the support vector machine. A fair comparison of both data sets is done using the same analysis techniques, moreover simultaneously collected data of EEG and fMRI is used for this comparison. The final analysis is done with the data of eight participants; the average result of all conditions are found which is 65.7% for EEG data set and 64.1% for fMRI data set. It concludes that EEG is capable of doing brain decoding with the data from multiple brain regions. In other words, decoding accuracy with EEG MVPA is as good as fMRI MVPA and is above chance level.


EEG fMRI Visual decoding SVM DWT 



This research is supported by the Ministry of Education (MOE), Malaysia under the Grant of Higher Institution Centre of Excellence (HICoE) for CISIR (Ref: 0153CA-002).

Compliance with ethical standards

Conflict of interest

Aamir Saeed Malik has received the above grant and he declares that he has no conflict of interest.

Ethical approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of the Hospital Universiti Sains Malaysia (HUSM) and is already used in a recent study [34]. The study protocol was approved by the local ethics committee of HUSM.

Informed consent

Informed consent was obtained from all individual participants included in the study before the start of the experiment.


  1. 1.
    Penny WD, Friston KJ, Ashburner JT, Kiebel SJ, Nichols TE (2011) Statistical parametric mapping: the analysis of functional brain images. Academic Press, CambridgeGoogle Scholar
  2. 2.
    Kriegeskorte N (2011) Pattern-information analysis: from stimulus decoding to computational-model testing. Neuroimage 56:411–421CrossRefPubMedGoogle Scholar
  3. 3.
    Pereira F, Mitchell T, Botvinick M (2009) Machine learning classifiers and fMRI: a tutorial overview. Neuroimage 45:S199–S209CrossRefPubMedGoogle Scholar
  4. 4.
    O’Toole AJ, Jiang F, Abdi H, Pénard N, Dunlop JP, Parent MA (2007) Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data. J Cogn Neurosci 19:1735–1752CrossRefPubMedGoogle Scholar
  5. 5.
    Sadati N, Hosseinzadeh M, Dumont GA (2018) Multi-model robust control of depth of hypnosis. Biomed Signal Process Control 40:443–453CrossRefGoogle Scholar
  6. 6.
    Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430CrossRefPubMedGoogle Scholar
  7. 7.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587Google Scholar
  8. 8.
    Crouzet SM, Busch NA, Ohla K (2015) Taste quality decoding parallels taste sensations. Curr Biol 25:890–896CrossRefPubMedGoogle Scholar
  9. 9.
    Zafar R, Dass SC, Malik AS (2017) Electroencephalogram-based decoding cognitive states using convolutional neural network and likelihood ratio based score fusion. PLoS ONE 12:e0178410CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Kearney-Ramos TE, Lench DH, Hoffman M, Correia B, Dowdle LT, Hanlon CA (2018) Gray and white matter integrity influence TMS signal propagation: a multimodal evaluation in cocaine-dependent individuals. Sci Rep 8:3253CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Caballero-Gaudes C, Reynolds RC (2017) Methods for cleaning the BOLD fMRI signal. Neuroimage 154:128–149CrossRefPubMedGoogle Scholar
  12. 12.
    Mumford JA, Turner BO, Ashby FG, Poldrack RA (2012) Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses. NeuroImage 59:2636–2643CrossRefPubMedGoogle Scholar
  13. 13.
    Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL (2009) Bayesian reconstruction of natural images from human brain activity. Neuron 63:902CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Friston KJ, Jezzard P, Turner R (1994) Analysis of functional MRI time-series. Hum Brain Map 1:153–171CrossRefGoogle Scholar
  15. 15.
    Smith S (2005) EEG in neurological conditions other than epilepsy: when does it help, what does it add? J Neurol Neurosurg Psychiatry 76:ii8–ii12CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Rosa MJ, Seymour B (2014) Decoding the matrix: benefits and limitations of applying machine learning algorithms to pain neuroimaging. Pain 155:864–867CrossRefPubMedGoogle Scholar
  17. 17.
    Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, Gallant JL (2011) Reconstructing visual experiences from brain activity evoked by natural movies. Curr Biol 21:1641–1646CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Horikawa T, Tamaki M, Miyawaki Y, Kamitani Y (2013) Neural decoding of visual imagery during sleep. Science 340:639–642CrossRefPubMedGoogle Scholar
  19. 19.
    Zafar R, Kamel N, Naufal M, Malik AS, Dass SC, Ahmad RF, Abdullah JM, Reza F (2017) Decoding of visual activity patterns from fMRI responses using multivariate pattern analyses and convolutional neural network. J Integr Neurosc 16(3):275–289. CrossRefGoogle Scholar
  20. 20.
    Gallant JL (2016) The Gallant Lab at UC BerkeleyGoogle Scholar
  21. 21.
    Cox DD, Savoy RL (2003) Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19:261–270CrossRefPubMedGoogle Scholar
  22. 22.
    Kamitani Y, Tong F (2005) Decoding the visual and subjective contents of the human brain. Nat Neurosci 8:679–685CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Kay KN, Naselaris T, Prenger RJ, Gallant JL (2008) Identifying natural images from human brain activity. Nature 452:352–355CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Schoenmakers S, Barth M, Heskes T, van Gerven M (2013) Linear reconstruction of perceived images from human brain activity. NeuroImage 83:951–961CrossRefPubMedGoogle Scholar
  25. 25.
    Mumford JA, Davis T, Poldrack RA (2014) The impact of study design on pattern estimation for single-trial multivariate pattern analysis. Neuroimage 103:130–138CrossRefPubMedGoogle Scholar
  26. 26.
    Gardumi A, Ivanov D, Hausfeld L, Valente G, Formisano E, Uludağ K (2016) The effect of spatial resolution on decoding accuracy in fMRI multivariate pattern analysis. Neuroimage 132:32–42CrossRefPubMedGoogle Scholar
  27. 27.
    Mitchell TM, Shinkareva SV, Carlson A, Chang K-M, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science 320:1191–1195CrossRefPubMedGoogle Scholar
  28. 28.
    Tong F, Pratte MS (2012) Decoding patterns of human brain activity. Annu Rev Psychol 63:483–509CrossRefPubMedGoogle Scholar
  29. 29.
    Norman KA, Polyn SM, Detre GJ, Haxby JV (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn Sci 10:424–430CrossRefPubMedGoogle Scholar
  30. 30.
    Cruse D, Chennu S, Chatelle C, Bekinschtein TA, Fernández-Espejo D, Pickard JD, Laureys S, Owen AM (2012) Bedside detection of awareness in the vegetative state: a cohort study. Lancet 378:2088–2094CrossRefGoogle Scholar
  31. 31.
    Taghizadeh-Sarabi M, Daliri MR, Niksirat KS (2015) Decoding objects of basic categories from electroencephalographic signals using wavelet transform and support vector machines. Brain Topogr 28:33–46CrossRefPubMedGoogle Scholar
  32. 32.
    Douglas PK, Lau E, Anderson A, Head A, Kerr W, Wollner M, Moyer D, Li W, Durnhofer M, Bramen J (2013) Single trial decoding of belief decision making from EEG and fMRI data using independent components features. Front Hum Neurosci 7(392):1–10. CrossRefGoogle Scholar
  33. 33.
    Winters-Hilt S, Merat S (2007) SVM clustering. BMC Bioinform 8:S18-S18Google Scholar
  34. 34.
    Zafar R, Dass SC, Malik A, Kamel N, Rehman J, Ahmad RF, Abdullah JM, Reza F (2017) Prediction of human brain activity using likelihood ratio based score fusion. In: IEEE access, pp 1–1Google Scholar
  35. 35.
    Ashburner J, Barnes G, Chen C, Daunizeau J, Flandin G, Friston K, Gitelman D, Kiebel S, Kilner J, Litvak V (2008) SPM8 manual functional imaging laboratory. Institute of Neurology, London, p 41Google Scholar
  36. 36.
    Simanova I, Van Gerven M, Oostenveld R, Hagoort P (2010) Identifying object categories from event-related EEG: toward decoding of conceptual representations. PLoS ONE 5:e14465CrossRefPubMedPubMedCentralGoogle Scholar
  37. 37.
    Subasi A (2005) Automatic recognition of alertness level from EEG by using neural network and wavelet coefficients. Expert Syst Appl 28:701–711CrossRefGoogle Scholar
  38. 38.
    Kandaswamy A, Kumar CS, Ramanathan RP, Jayaraman S, Malmurugan N (2004) Neural classification of lung sounds using wavelet coefficients. Comput Biol Med 34:523–537CrossRefPubMedGoogle Scholar
  39. 39.
    Bibian S, Zikov T, Ries CR, Dumont GA, Huzmezan M (2004) The wavelet-based anesthetic value (WAV): a novel alternative to the bispectral index (BIS). Anesthesiology 101:A342Google Scholar
  40. 40.
    Lundqvist KK, Bibian S, Ries C, Yu P, Dumont G (2004) Can the wavelet-based anesthetic value (WAV) predict airway motor response to LMA insertion? Anesthesiology 101:A606Google Scholar
  41. 41.
    Messer SR, Agzarian J, Abbott D (2001) Optimal wavelet denoising for phonocardiograms. Microelectron J 32:931–941CrossRefGoogle Scholar
  42. 42.
    Yohanes RE, Ser W, Huang G-B (2012) Discrete wavelet transform coefficients for emotion recognition from EEG signals. In: Engineering in medicine and biology society (EMBC), 2012 annual international conference of the IEEE, pp 2251–2254Google Scholar
  43. 43.
    Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3:390CrossRefGoogle Scholar
  44. 44.
    Zafar R, Malik AS, Amin HU, Kamel N, Dass SC (2015) Discrimination of brain states using wavelet and power spectral density. In: Neural information processing. Springer, New York, pp 341–347CrossRefGoogle Scholar
  45. 45.
    Rafiee J, Rafiee M, Prause N, Schoen M (2011) Wavelet basis functions in biomedical signal processing. Expert Syst Appl 38:6190–6201CrossRefGoogle Scholar
  46. 46.
    McMillan DE (1989) Increased levels of acute-phase serum proteins in diabetes. Metabolism 38:1042–1046CrossRefPubMedGoogle Scholar
  47. 47.
    Wang L, Fu X (2006) Data mining with computational intelligence. Springer, New YorkGoogle Scholar
  48. 48.
    Chang C-C, Lin C-J (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27CrossRefGoogle Scholar
  49. 49.
    Liao K, Xiao R, Gonzalez J, Ding L (2014) Decoding individual finger movements from one hand using human EEG signals. PLoS ONE 9:e85192CrossRefPubMedPubMedCentralGoogle Scholar
  50. 50.
    Knerr S, Personnaz L, Dreyfus G (1990) Single-layer learning revisited: a stepwise procedure for building and training a neural network. In: Neurocomputing. Springer, Berlin, pp 41–50CrossRefGoogle Scholar
  51. 51.
    Friedman J (1996) Another approach to polychotomous classification, technical report. Department of Statistics, Stanford University, StanfordGoogle Scholar
  52. 52.
    Yin, Rosanna Soentpiet (1999) In: Schölkopf B, Burges CJ, Smola AJ (eds) Advances in Kernel Methods: Support Vector Learning, vol 1. MIT Press, p. 376Google Scholar
  53. 53.
    Phillips S, Takeda Y, Singh A (2012) Visual feature integration indicated by phase-locked frontal-parietal EEG signals. PLoS ONE 7:e32502CrossRefPubMedPubMedCentralGoogle Scholar
  54. 54.
    Kay KN, Winawer J, Mezer A, Wandell BA (2013) Compressive spatial summation in human visual cortex. J Neurophysiol 110:481–494CrossRefPubMedPubMedCentralGoogle Scholar
  55. 55.
    Teplan M (2002) Fundamentals of EEG measurement. Meas Sci Rev 2:1–11Google Scholar
  56. 56.
    CBS (2009). How technology may soon “read” your mind.

Copyright information

© Australasian College of Physical Scientists and Engineers in Medicine 2018

Authors and Affiliations

  1. 1.Department of EngineeringNational University of Modern LanguagesIslamabadPakistan
  2. 2.Centre for Intelligent Signal and Imaging Research (CISIR)Universiti Teknologi PETRONASPerakMalaysia
  3. 3.Center for Neuroscience Services and ResearchUniversiti Sains MalaysiaKota BharuMalaysia
  4. 4.Department of Neurosciences, School of Medical SciencesUniversiti Sains MalaysiaKota BharuMalaysia

Personalised recommendations