Skip to main content
SpringerLink
Log in

Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients

  • Conference paper
  • First Online:
Computer Vision – ACCV 2016 Workshops (ACCV 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10116))

Included in the following conference series:

Abstract

Alzheimer Disease (AD) early detection is considered of high importance for improving the quality of life of patients and their families. Amongst all the different approaches for AD detection, significant work has been focused on emotion analysis through facial expressions, body language or speech. Many studies also use the electroencephalogram in order to capture emotions that patients cannot physically express. Our work introduces an emotion recognition approach using facial expression and EEG signal analysis. A novel dataset was created specifically to remark the autobiographical memory deficits of AD patients. This work uses novel EEG features based on quaternions, facial landmarks and the combination of them. Their performance was evaluated in a comparative study with a state of the art methods that demonstrates the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alpher, A.: Advances in frobnication. J. Foo 12, 234–778 (2002)

    Google Scholar 

  2. Alpher, A., Fotheringham-Smythe, J.P.N.: Frobnication revisited. J. Foo 13, 234–778 (2003)

    Article  Google Scholar 

  3. Herman, S., Fotheringham-Smythe, J.P.N., Gamow, G.: Can a machine frobnicate? J. Foo 14, 234–778 (2004)

    Google Scholar 

  4. Smith, F.: The Frobnicatable Foo Filter. GreatBooks, Atown (2009)

    Google Scholar 

  5. Wills, H.: Frobnication tutorial. Technical report CS-1204, XYZ University, Btown (1999)

    Google Scholar 

  6. Rosler, A., Mapstone, M.E., Hays, A.K., Mesulam, M., Rademaker, A., Gitelman, D.R., Weintraub, S.: Alterations of visual search strategy in Alzheimer’s disease and aging. Neuropsychology 14(3), 398–408 (2000)

    Article  Google Scholar 

  7. Pereira, M.L., Camargo, M.V.Z.A., Aprahamian, I., Forlenza, O.V.: Eye movement analysis and cognitive processing: detecting indicators of conversion to Alzheimer’s disease. Neuropsychiatric Dis. Treat. 10, 1273–1285 (2014)

    Article  Google Scholar 

  8. Alpher, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

    Article  Google Scholar 

  9. Koelstra, S., Muehl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)

    Article  Google Scholar 

  10. Sapey-Triomphe, L.A., Heckemann, R.A., Boublay, N., Dorey, J.M., Hnaff, M.A., Rouch, I., Padovan, C.: Neuroanatomical correlates of recognizing face expressions in mild stages of Alzheimers disease. PLoS ONE 10(12) (2015)

    Google Scholar 

  11. Van den Stock, J., De Winter, F.L., de Gelder, B., Rangarajan, J.R., Cypers, G., Maes, F., Sunaert, S., Goffin, K., Vandenberghe, R., Vandenbulcke, M.: Impaired recognition of body expressions in the behavioral variant of frontotemporal dementia. Neuropsychologia 75, 496–504 (2015)

    Article  Google Scholar 

  12. Han, K.H., Zaytseva, Y., Bao, Y., Pppel, E., Chung, S.Y., Kim, J.W., Kim, H.T.: Impairment of vocal expression of negative emotions in patients with Alzheimers disease. Front. Aging Neurosci. 6(101), 1–6 (2014)

    Google Scholar 

  13. Irish, M., Hornberger, M., Lah, S., Miller, L., Pengas, G., Nestor, P.J., Hodges, J.R., Piguet, O.: Profiles of recent autobiographical memory retrieval in semantic dementia, behavioural-variant frontotemporal dementia, and Alzheimers disease. Neuropsychologia 49(9), 2694–2702 (2011)

    Article  Google Scholar 

  14. Seidl, U., Lueken, U., Thomann, P.A., Kruse, A., Schrder, J.: Facial expression in Alzheimers disease impact of cognitive deficits and neuropsychiatric symptoms. Am. J. Alzheimer’s Dis. Other Dementias 27(2), 100–106 (2012)

    Article  Google Scholar 

  15. American Psychiatric Association: Diagnostic and statistical manual of mental disorders (DSM-5). American Psychiatric Association Publishing (2013)

    Google Scholar 

  16. Bettadapura, V.: Face expression recognition and analysis: the state of the art. Tech. Report, pp. 1–27. arXiv:1203.6722 (2012)

  17. Ekman, P., Friesen, W.V.: The Facial Action Coding System: A Technique for The Measurement of Facial Movement. Consulting Psychologists Press, San Francisco (1978)

    Google Scholar 

  18. Weninger, F., Wllmer, M., Schuller, B.: Emotion recognition in naturalistic speech and language a survey. In: Emotion Recognition: A Pattern Analysis Approach, pp. 237–267 (2015)

    Google Scholar 

  19. Chowdhuri, M.A.D., Bojewar, S.: Emotion detection analysis through tone of user: a survey. Emotion 5(5), 859–861 (2016)

    Google Scholar 

  20. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7(1), 17–28 (2016)

    Article  Google Scholar 

  21. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)

    Article  Google Scholar 

  22. Nicolaou, M.A., Gunes, H., Pantic, M.: Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. Affect. Comput. 2(2), 92–105 (2011)

    Article  Google Scholar 

  23. McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schroder, M.: The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3(1), 5–17 (2012)

    Article  Google Scholar 

  24. Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Google Scholar 

  25. Lokannavar, S., Lahane, P., Gangurde, A., Chidre, P.: Emotion recognition using EEG signals. Emotion 4(5), 54–56 (2015)

    Google Scholar 

  26. Vijayan, A.E., Sen, D., Sudheer, A.P.: EEG-based emotion recognition using statistical measures and auto-regressive modeling. In: IEEE International Conference on Computational Intelligence and Communication Technology (CICT), vol. 14(1), pp. 587–591 (2015)

    Google Scholar 

  27. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

    Article  Google Scholar 

  28. Baltru, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10 (2016)

    Google Scholar 

  29. Sohaib, A.T., Qureshi, S., Hagelbäck, J., Hilborn, O., Jerčić, P.: Evaluating classifiers for emotion recognition using EEG. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS (LNAI), vol. 8027, pp. 492–501. Springer, Heidelberg (2013). doi:10.1007/978-3-642-39454-6_53

    Chapter  Google Scholar 

  30. Müller-Putz, G.R., Riedl, R., Wriessnegger, S.C.: Electroencephalography (EEG) as a research tool in the information systems discipline: foundations, measurement, and applications. Commun. Assoc. Inform. Syst. 37(46), 911–948 (2015)

    Google Scholar 

  31. Petrantonakis, P.C., Hadjileontiadis, L.J.: Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Trans. Affect. Comput. 1, 81–97 (2010)

    Article  Google Scholar 

  32. Valstar, M.F., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 3, pp. 76–84 (2005)

    Google Scholar 

  33. Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 258–264 (2003)

    Google Scholar 

  34. Hamilton, W.R.: On quaternions, or on a new system of imaginaries in algebra. Philos. Mag. 25(3), 489–495 (1844)

    Google Scholar 

  35. Chen, M., Meng, X., Wang, Z.: Quaternion fisher discriminant analysis for bimodal multi-feature fusion. In: Abraham, A., Jiang, X.H., Snášel, V., Pan, J.-S. (eds.) Intelligent Data Analysis and Applications. AISC, vol. 370, pp. 479–487. Springer, Cham (2015). doi:10.1007/978-3-319-21206-7_41

    Chapter  Google Scholar 

  36. Le Bihan, N., Sangwine, S.J.: Quaternion principal component analysis of color images. In: International Conference on Image Processing (ICIP), vol. 1, pp. I-809–I-812 (2003)

    Google Scholar 

  37. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53 (2000)

    Google Scholar 

  38. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, pp. 317–321 (2005)

    Google Scholar 

  39. Patras, I., Pantic, M.: Particle filtering with factorized likelihoods for tracking facial features. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 97–102 (2004)

    Google Scholar 

  40. Izard, C.E.: Human Emotions. Springer, New York (2013)

    Google Scholar 

  41. Nicolle, J., Rapp, V., Bailly, K., Prevost, L., Chetouani, M.: Robust continuous prediction of human emotions using multiscale dynamic cues. In: 14th ACM International Conference on Multimodal Interaction, pp. 501–508 (2012)

    Google Scholar 

  42. Littlewort, G.C., Bartlett, M.S., Lee, K.: Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27(12), 1797–1803 (2009)

    Article  Google Scholar 

  43. Huang, K.C., Huang, S.Y., Kuo, Y.H.: Emotion recognition based on a novel triangular facial feature extraction method. In: 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–6(2010)

    Google Scholar 

  44. Sariyanidi, E., Gunes, H., Gkmen, M., Cavallaro, A.: Local zernike moment representation for facial affect recognition. In: British Machine Vision Conference (2013)

    Google Scholar 

  45. Zhao, G., Pietikinen, M.: Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn. Lett. 30(12), 1117–1127 (2009)

    Article  Google Scholar 

  46. Wllmer, M., Eyben, F., Reiter, S., Schuller, B., Cox, C., Douglas-Cowie, E., Cowie, R.: Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies. Interspeech 12, 597–600 (2008)

    Google Scholar 

  47. Adali, T., Schreier, P.J., Scharf, L.L.: Complex-valued signal processing: the proper way to deal with impropriety. IEEE Trans. Sig. Process. (Overview Pap.) 59(11), 5101–5123 (2011)

    Article  MathSciNet  Google Scholar 

  48. Li, X.-L., Adali, T., Anderson, M.: Noncircular principal component analysis and its application to model selection. IEEE Sig. Proc. 59(10), pp. 4516i–4528i (2011)

    Google Scholar 

  49. Chai, Z., Ma, K.K., Liu, Z.: Complex wavelet-based face recognition using independent component analysis. In: Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Proceedings, pp. 832–835 (2009)

    Google Scholar 

  50. Bonita, J.D., Ambolode, II., L.C.C., Rosenberg, B.M., Cellucci, C.J., Watanabe, T.A.A., Rapp, P.E., Albano, A.M.: Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures. Cogn. Neurodyn. 8(1), 1–15 (2014)

    Google Scholar 

  51. Li, K., Sun, G., Zhang, B., Wu, S., Wu, G.: Correlation between forehead EEG and sensorimotor area EEG in motor imagery task. In: Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing, DASC 2009, pp. 430–435 (2009)

    Google Scholar 

  52. Abadi, M.K., Subramanian, R., Kia, S.M., Avesani, P., Patras, I., Sebe, N.: DECAF: MEG-based multimodal database for decoding affective physiological responses. IEEE Trans. Affect. Comput. 6(3), 209–222 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Kingston University, Kingston upon Thames, England

    Juan Manuel Fernandez Montenegro & Vasileios Argyriou

  2. Imperial College, London, England

    Athanasios Gkelias

Authors
  1. Juan Manuel Fernandez Montenegro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Athanasios Gkelias
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vasileios Argyriou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Manuel Fernandez Montenegro .

Editor information

Editors and Affiliations

  1. Institute of Information Science, Academia Sinica, Taipei, Taiwan

    Chu-Song Chen

  2. Tsinghua University , Beijing, China

    Jiwen Lu

  3. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Singapore

    Kai-Kuang Ma

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 12433 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Montenegro, J.M.F., Gkelias, A., Argyriou, V. (2017). Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients. In: Chen, CS., Lu, J., Ma, KK. (eds) Computer Vision – ACCV 2016 Workshops. ACCV 2016. Lecture Notes in Computer Science(), vol 10116. Springer, Cham. https://doi.org/10.1007/978-3-319-54407-6_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54407-6_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54406-9

  • Online ISBN: 978-3-319-54407-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation