Skip to main content
SpringerLink
Log in

The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features

  • Scientific Paper
  • Published:
Physical and Engineering Sciences in Medicine Aims and scope Submit manuscript

Abstract

Recently, developing an accurate automatic emotion recognition system using a minimum number of bio-signals has become a challenging issue in “affective computing.” This study aimed to propose a reliable system by examining nonlinear dynamics of photoplethysmogram (PPG) and galvanic skin response (GSR). To address this goal, two strategies were adopted. First, the efficiency of each signal in valence/arousal based emotion categorization was examined. Then, the proficiency of a hybrid feature, by combining both GSR and PPG features was studied. Lyapunov exponents, lagged Poincare's measures, and approximate entropy were extracted to characterize the irregularity and chaotic behavior of the phase space. To discriminate two levels of arousal and two levels of the valence, a probabilistic neural network (PNN) with different sigma adjustment parameter was examined. The results showed that the phase space geometry and consequently, the signal dynamics are influenced by the emotional music video. Additionally, distinctive patterns of the phase space behavior were observed under the influence of different lags. For both signals, the most irregularity was observed during the high valence, and the least irregularity was seen during the low valence. Consequently, signals’ irregularity is affected by the valence dimension. The results showed that the fusion has more potential for emotion recognition than that of using each signal separately. For sigma = 0.1, the highest recognition rate was 100% in a subject-dependent mode. In a subject-independent mode, the maximum accuracies of 88.57 and 86.8% were obtained for arousal and valence dimensions, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhalehpour S, Akhtar Z, Erdem CE (2005) Multimodal emotion recognition based on peak frame selection from video. SIViP 10:827–834

    Google Scholar 

  2. Jing S, Mao X, Chen L (2018) Prominence features: effective emotional features for speech emotion recognition. Digit Signal Process 72:216–231

    Google Scholar 

  3. Gil E, Laguna P, Martinez JP, Barquero-Perez O, Garcia-Alberola A, Sornmo L (2013) Heart rate turbulence analysis based on photoplethysmography. IEEE Trans Biomed Eng 60:3149–3155

    PubMed  Google Scholar 

  4. Solosenko A, Petrenas A, Marozas V (2015) Photoplethysmography-based method for automatic detection of premature ventricular contractions. IEEE Trans Biomed Circuits Syst 9:662–669

    PubMed  Google Scholar 

  5. Solosenko A, Petrenas A, Marozas V, Sornmo L (2017) Modeling of the photoplethysmogram during atrial fibrillation. Comput Biol Med 81:130–138

    PubMed  Google Scholar 

  6. Hwang S, Seo J, Jebelli H, Lee S (2016) Feasibility analysis of heart rate monitoring of construction workers using a photoplethysmography (PPG) sensor embedded in a wristband-type activity tracker. Automat Constr 71:372–381

    Google Scholar 

  7. Boloursaz Mashhadi M, Asadi E, Eskandari M, Kiani S, Marvasti F (2016) Heart rate tracking using wrist-type photoplethysmographic (PPG) signals during physical exercise with simultaneous accelerometry. IEEE Signal Process Lett 23:227–231

    Google Scholar 

  8. Temko A (2017) Accurate heart rate monitoring during physical exercises using PPG. IEEE Trans Biomed Eng 64:2016–2024

    PubMed  Google Scholar 

  9. Firoozabadi R, Helfenbein ED, Babaeizadeh S (2017) Efficient noise-tolerant estimation of heart rate variability using single-channel photoplethysmography. J Electrocardiol 50:841–846

    PubMed  Google Scholar 

  10. Islam MS, Rabbi MSE, Dobaie AMA, Hasan MK (2017) PREHEAT: precision heart rate monitoring from intense motion artifact corrupted PPG signals using constrained RLS and wavelets. Biomed Signal Process Control 38:212–223

    Google Scholar 

  11. Zhao D, Sun Y, Wan S, Wang F (2017) SFST: a robust framework for heart rate monitoring from photoplethysmography signals during physical activities. Biomed Signal Process Control 33:316–324

    Google Scholar 

  12. Resit Kavsaoglu A, Polat K, Recep Bozkurt M (2014) A novel feature ranking algorithm for biometric recognition with PPG signals. Comput Biol Med 49:1–14

    CAS  PubMed  Google Scholar 

  13. Papapanagiotou V, Diou C, Zhou L, van den Boer J, Mars M, Delopoulos A (2017) A novel chewing detection system based on PPG, audio and accelerometry. IEEE J Biomed Health Inform 21:607–618

    PubMed  Google Scholar 

  14. Orphanidou C (2017) Derivation of respiration rate from ambulatory ECG and PPG using ensemble empirical mode decomposition: comparison and fusion. Comput Biol Med 81:45–54

    PubMed  Google Scholar 

  15. Hassani A, Foruzan AH (2019) Improved PPG-based estimation of the blood pressure using latent space features. SIViP 13:1141

    Google Scholar 

  16. Longmore SK, Lui GY, Naik G, Breen PP, Jalaludin B, Gargiulo GD (2019) A comparison of reflective photoplethysmography for detection of heart rate, blood oxygen saturation, and respiration rate at various anatomical locations. Sensors 19:1874

    Google Scholar 

  17. Lee H, Lee J, Shin M (2019) Using wearable ECG/PPG sensors for driver drowsiness detection based on distinguishable pattern of recurrence plots. Electronics 8:192

    Google Scholar 

  18. Zha YT, Liu G, Shang X, Wang F, Cai J, Wei X (2018) Non-invasive assessment of cerebral hemodynamics with CWNIRS-ICG and application of EEMD-SSE in PPG signal extraction. Optik 156:22–30

    Google Scholar 

  19. Kavsaoglua AR, Polat K, Hariharan M (2015) Non-invasive prediction of hemoglobin level using machine learning techniques with the PPG signal’s characteristics features. Appl Soft Comput 37:983–991

    Google Scholar 

  20. Sudheesh NN, Joseph KP (2000) Investigation into the effects of music and meditation on galvanic skin response. ITBM-RBM 21:158–163

    Google Scholar 

  21. Nagai Y, Goldstein LH, Fenwick PB, Trimble MR (2004) Clinical efficacy of galvanic skin response biofeedback training in reducing seizures in adult epilepsy: a preliminary randomized controlled study. Epilepsy Behav 5:216–223

    PubMed  Google Scholar 

  22. Fusar-Poli P, Landi P, O’Connor C (2009) Neurophysiological response to emotional faces with increasing intensity of fear: a skin conductance response study. J Clin Neurosci 16:981–982

    CAS  PubMed  Google Scholar 

  23. Khalfa S, Isabelle P, Jean-Pierre B, Manon R (2002) Event-related skin conductance responses to musical emotions in humans. Neurosci Lett 328:145–149

    CAS  PubMed  Google Scholar 

  24. Kim J, Andre E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30:2067–2083

    PubMed  Google Scholar 

  25. Lang P, Greenwald M, Bradely M, Hamm A (1993) Looking at pictures—affective, facial, visceral, and behavioral reactions. Psychophysiol 30:261–273

    CAS  Google Scholar 

  26. van Dooren M, de Vries JJG, Janssen JH (2012) Emotional sweating across the body: comparing 16 different skin conductance measurement locations. Physiol Behav 106:298–304

    PubMed  Google Scholar 

  27. Younessi Heravi MA, Khalilzadeh MA, Sarafan R, Azarnoosh M (2013) Lie detector system based on PhotoPlethysmoGraph (PPG) and galvanic skin response (GSR) signals by means of neural network. Signal Data Process 2:49–60 (Text in Persian)

    Google Scholar 

  28. Sariahmetoglu H, Soysal A, Sen A, Yuksel B, Celiker S, Ciftci-Kavaklioglu B, Arpaci B (2014) Forehead sympathetic skin responses in determining autonomic involvement in Parkinson’s disease. Clin Neurophysiol 125:2436–2440

    PubMed  Google Scholar 

  29. Sriramprakash S, Prasanna Vadana D, Ramana Murthy OV (2017) Stress detection in working people. Procedia Comput Sci 115:359–366

    Google Scholar 

  30. Martinez R, Irigoyen E, Arruti A, Martin JI, Muguerza J (2017) A real-time stress classification system based on arousal analysis of the nervous system by an F-state machine. Comput Methods Programs Biomed 148:81–90

    CAS  PubMed  Google Scholar 

  31. Salazar-Ramirez A, Irigoyen E, Martinez R, Zalabarria U (2018) An enhanced fuzzy algorithm based on advanced signal processing for identification of stress. Neurocomput 271:48–57

    Google Scholar 

  32. Chen LI, Zhao Y, Ye PF, Zhang J, Zou JZ (2017) Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers. Expert Syst Appl 85:279–291

    Google Scholar 

  33. Thomas GEC, Crutch SJ, Camic PM (2018) Measuring physiological responses to the arts in people with a dementia. Int J Psychophysiol 123:64–73

    PubMed  Google Scholar 

  34. Kim KH, Bang SW, Kim SR (2004) Emotion recognition system using short-term monitoring of physiological signals. Med Biol Eng Comput 42:419–427

    CAS  PubMed  Google Scholar 

  35. Verhoef T, Lisetti C, Barreto A, Ortega F, van der Zant T, Cnossen F (2009) Bio-sensing for emotional characterization without word labels. In: Jacko JA (eds) Human-computer interaction ambient, ubiquitous and intelligent interaction HCI 2009 lecture notes in computer science, vol 5612. Springer, Berlin

    Google Scholar 

  36. Soleymani M, Koelstra S, Patras I, Pun T, (2011) Continuous emotion detection in response to music videos. Automatic face & gesture recognition, and workshops (FG, (2011) IEEE International Conference, Santa Barbara

  37. Park MW, Kim CJ, Whang M, Lee EC (2013) Individual emotion classification between happiness and sadness by analyzing photoplethysmography and skin temperature software engineering (WCSE). Fourth World Congress. IEEE, Hong Kong

  38. Li C, Feng Z, Xu C (2014) Physiological-based emotion recognition with IRS model smart computing (SMARTCOMP). International Conference. IEEE, Hong Kong

  39. Cheng J, Liu G, Lai X (2014) Calculation of nonlinear features of SC for emotion recognition. J Comput Inform Syst 10:2331–2339

    Google Scholar 

  40. Khan AM, Lewo M (2016) Recognizing emotion from blood volume pulse and skin conductance sensor using machine learning algorithms. In: Kyriacou E, Christofides S, Pattichis C (eds) XIV Mediterranean conference on medical and biological engineering and computing. IFMBE Proceedings, vol 57. Springer, Cham

    Google Scholar 

  41. Goshvarpour A, Abbasi A, Goshvarpour A, Daneshvar S (2017) Discrimination between different emotional states based on the chaotic behavior of galvanic skin responses. SIViP 11:1347–1355

    Google Scholar 

  42. Goshvarpour A, Abbasi A, Goshvarpour A, Daneshvar S (2016) A novel signal-based fusion approach for accurate music emotion recognition. Biomed Eng Appl Basis Commun 28:1650040

    Google Scholar 

  43. Goshvarpour A, Abbasi A, Goshvarpour A (2017) An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomed J 40:355–368

    Google Scholar 

  44. Goshvarpour A, Goshvarpour A (2018) Poincaré's section analysis for PPG-based automatic emotion recognition. Chaos Soliton Fract 114:400–407

    Google Scholar 

  45. Domínguez-Jiménez JA, Campo-Landines KC, Martínez-Santos JC, Delahoz EJ, Contreras-Ortiz SH (2019) A machine learning model for emotion recognition from physiological signals. Biomed Signal Process Control 55:101646

    Google Scholar 

  46. Tsuda I (1992) Chaotic pulsation in human capillary vessels and its dependence on mental and physical conditions. Int J Bifurcat Chaos 2:313–324

    Google Scholar 

  47. Sumida T, Arimitu Y (2000) Mental conditions reflected by chaos of pulsation in the capillary vessels. Int J Bifurcat Chaos 10:2245–2255

    Google Scholar 

  48. Sviridova N, Sakai K (2015) Human photoplethysmogram: new insight into chaotic characteristics. Chaos Solitons Fractals 77:53–63

    Google Scholar 

  49. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput 3:18–31

    Google Scholar 

  50. Kumar N, Khaund K, Hazarika SM (2016) Bispectral analysis of EEG for emotion recognition. Procedia Comput Sci 84:31–35

    Google Scholar 

  51. Guo Y, Naik GR, Huang S, Abraham A, Nguyen HT (2015) Nonlinear multiscale maximal Lyapunov exponent for accurate myoelectric signal classification. Appl Soft Comput 36:633–640

    Google Scholar 

  52. Goshvarpour A, Goshvarpour A (2019) Do meditators and non-meditators have different HRV dynamics? Cogn Syst Res 54:21–36

    Google Scholar 

  53. Goshvarpour A, Goshvarpour A (2019) Gender and age classification using a new poincare section-based feature set of ECG. SIViP 13:531–539

    Google Scholar 

  54. Goshvarpour A, Abbasi A, Goshvarpour A (2017) Indices from lagged poincare plots of heart rate variability: an efficient nonlinear tool for emotion discrimination. Australas Phys Eng Sci Med 40:277–287

    Google Scholar 

  55. Goshvarpour A, Abbasi A, Goshvarpour A (2017) Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots. Australas Phys Eng Sci Med 40:617–629

    Google Scholar 

  56. Goshvarpour A, Abbasi A, Goshvarpour A (2017) Do men and women have different ECG responses to sad pictures? Biomed Signal Process Control 38:67–73

    Google Scholar 

  57. Rosenstein MT, Collins JJ, De Luca CJ (1993) A practical method for calculating largest Lyapunov exponents from small data sets. Physica D 65:117–134

    Google Scholar 

  58. Pincus SM (1991) Approximate entropy as a measure of system complexity. Proc Natl Acad Sci USA 88:2297–2301

    CAS  PubMed  Google Scholar 

  59. Demuth H, Beale M (2000) Neural network toolbox. The MathWorks Inc, Natick

    Google Scholar 

  60. Chung SY, Yoon HJ (2012) Affective classification using bayesian classifier and supervised learning. In: 12th International conference on control, automation and systems (ICCAS). IEEE, pp 1768–1771

  61. Zhang X, Hu B, Chen J, Moore P (2013) Ontology-based context modeling for emotion recognition in an intelligent web. World Wide Web 16:497–513

    Google Scholar 

  62. Liu Y, Sourina O (2014) Real-time subject-dependent EEG-based emotion recognition algorithm. In: Gavrilova ML, Tan CJK, Mao X, Hong L (eds) Transactions on Computer Science, vol XXIII. LNCS, Berlin, pp 199–223

    Google Scholar 

  63. Mert A, Akan A (2018) Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl 21:81–89

    Google Scholar 

  64. Goshvarpour A, Goshvarpour A (2019) EEG spectral powers and source localization in depressing, sad, and fun music videos focusing on gender differences. Cognitive Neurodyn 13:161–173

    Google Scholar 

Download references

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Faculty of Electrical Engineering, Sahand University of Technology, Tabriz, Iran

    Atefeh Goshvarpour

  2. Department of Biomedical Engineering, Imam Reza International University, Mashhad, Razavi Khorasan, Iran

    Ateke Goshvarpour

  3. Imam Reza International University, Rezvan Campus (Female Students), Phalestine Sq., PO. BOX 91735-553, Mashhad, Razavi Khorasan, Iran

    Ateke Goshvarpour

Authors
  1. Atefeh Goshvarpour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ateke Goshvarpour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ateke Goshvarpour.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article examined the signals of DEAP dataset [49], which is available in the public domain. This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study [49].

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goshvarpour, A., Goshvarpour, A. The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features. Phys Eng Sci Med 43, 119–134 (2020). https://doi.org/10.1007/s13246-019-00825-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13246-019-00825-7

Keywords

Search

Navigation