CBR Tagging of Emotions from Facial Expressions

  • Paloma Lopez-de-Arenosa
  • Belén Díaz-Agudo
  • Juan A. Recio-García
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8765)


Mobility and context-awareness are two active research directions that open new potential to recommender systems. Usage of dynamically enriched information from the user context leads the system to find better solutions that are adapted to the specific situations. In this paper we focus on the difficult problem of dynamically acquiring the emotional context about the user during a recommendation process. We use the fact that emotions are tightly connected with facial expressions and it is difficult for people to hide emotions in facial expressions. We describe PhotoMood, a CBR system that uses gestures to identify emotions in faces, and present preliminary experiments with MadridLive, a mobile and context aware recommender system for leisure activities in Madrid. In the experiments, the momentary emotion of a user is dynamically detected from pictures of the facial expression taken unobtrusively with the front facing camera of the mobile device.


context-aware CBR emotional context image tagging 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adomavicius, G., Mobasher, B., Ricci, F., Tuzhilin, A.: Context-aware recommender systems. AI Magazine 32, 67–80 (2011)Google Scholar
  2. 2.
    Braunhofer, M., Kaminskas, M., Ricci, F.: Location-aware music recommendation. IJMIR 2, 31–44 (2013)Google Scholar
  3. 3.
    Benou, P., Bitos, V.: Context-aware query processing in ad-hoc environments of peers. JECO 6, 38–62 (2008)Google Scholar
  4. 4.
    Quijano-Sánchez, L., Recio-García, J.A., Díaz-Agudo, B., Jiménez-Díaz, G.: Social factors in group recommender systems. ACM Transactions on Intelligent Systems and Technology 4, Article 8 (2013)Google Scholar
  5. 5.
    Cohn, J.F.: Foundations of human computing: Facial expression and emotion. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds.) AI for Human Computing. LNCS (LNAI), vol. 4451, pp. 1–16. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Ortony, A., Turner, T.J.: What’s basic about basic emotions? Psychological Review 97(3), 315–331 (1990)CrossRefGoogle Scholar
  7. 7.
    Russell, J.A.: Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychological Bulletin 115, 102–141 (1994)CrossRefGoogle Scholar
  8. 8.
    Scollon, C.N., Kim-Prieto, C., Diener, E.: Experience sampling: Promises and pitfalls, strengths and weaknesses. Journal of Happiness Studies 4 (2003)Google Scholar
  9. 9.
    Eckman, P.: Facial expression and emotion. American Psychologist 48, 384–392 (1993)CrossRefGoogle Scholar
  10. 10.
    Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  11. 11.
    Feris, R.S., de Campos, T.E., Cesar Jr., R.M.: Detection and tracking of facial features in video sequences. In: Cairó, O., Sucar, L.E., Cantu, F.J. (eds.) MICAI 2000. LNCS, vol. 1793, pp. 127–135. Springer, Heidelberg (2000)Google Scholar
  12. 12.
    Solina, F., Peer, P., Batagelj, B., Juvan, S., Kovac, J.: Color-based face detection in the “15 seconds of fame” art installation. In: Proceedings of Mirage 2003 (INRIA Rocquencourt), pp. 38–47 (2003)Google Scholar
  13. 13.
    Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection. In: EUROCON 2003, Computer as a Tool. The IEEE Region 8, vol. 2, pp. 144–148. IEEE (2003)Google Scholar
  14. 14.
    Wang, J., Tan, T.: A new face detection method based on shape information. Pattern Recognition Letters 21, 463–471 (2000)CrossRefGoogle Scholar
  15. 15.
    Paul, S.K., Uddin, M.S., Bouakaz, S.: Extraction of facial feature points using cumulative histogram. CoRR abs/1203.3270 (2012)Google Scholar
  16. 16.
    Chawan, P.M., Jadhav, M.M.C., Mashruwala, J.B., Nehete, A.K., Panjari, P.A.: Real time emotion recognition through facial expressions for desktop devices. International Journal of Emerging Science and Engineering (IJESE) 1, 104–108 (2013)Google Scholar
  17. 17.
    Lin, C., Fan, K.-C.: Triangle-based approach to the detection of human face. Pattern Recognition 34, 1271–1284 (2001)CrossRefzbMATHGoogle Scholar
  18. 18.
    Yang, G., Huang, T.S.: Human face detection in a complex background. Pattern Recognition 27, 53–63 (1994)CrossRefGoogle Scholar
  19. 19.
    Draper, B.A., Baek, K., Bartlett, M.S., Beveridge, J.: Recognizing faces with PCA and ICA. Computer Vision and Image Understanding 91, 115–137 (2003) (Special Issue on Face Recognition)Google Scholar
  20. 20.
    Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20, 23–38 (1998)CrossRefGoogle Scholar
  21. 21.
    Sung, K.-K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20, 39–51 (1998)CrossRefGoogle Scholar
  22. 22.
    Lin, H.J., Yen, S.H., Yeh, J.P., Lin, M.J.: Face detection based on skin color segmentation and svm classification. In: SSIRI, pp. 230–231. IEEE Computer Society (2008)Google Scholar
  23. 23.
    Fragopanagos, N., Taylor, J.: Emotion recognition in human computer interaction. Neural Networks 18, 389–405 (2005) (Emotion and Brain)Google Scholar
  24. 24.
    Pantic, M., Rothkrantz, L.J.: Facial action recognition for facial expression analysis from static face images. Trans. Sys. Man Cyber. Part B 34, 1449–1461 (2004)CrossRefGoogle Scholar
  25. 25.
    Maglogiannis, I., Vouyioukas, D., Aggelopoulos, C.: Face detection and recognition of natural human emotion using markov random fields. Personal and Ubiquitous Computing 13, 95–101 (2009)CrossRefGoogle Scholar
  26. 26.
    Anderson, S., Conway, M.: Investigating the structure of autobiographical memories. Journal of Experimental Psychology: Learning, Memory, and Cognition 19, 1178–1196 (1993)Google Scholar
  27. 27.
    Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003) (Special Issue on Face Recognition)CrossRefGoogle Scholar
  28. 28.
    Teeters, A., El Kaliouby, R., Picard, R.: Self-cam: Feedback from what would be your social partner. In: ACM SIGGRAPH 2006 Research Posters. SIGGRAPH 2006. ACM, New York (2006)Google Scholar
  29. 29.
    Gruebler, A., Suzuki, K.: Analysis of social smile sharing using a wearable device that captures distal electromyographic signals. In: Stoica, A., Zarzhitsky, D., Howells, G., Frowd, C.D., McDonald-Maier, K.D., Erdogan, A.T., Arslan, T. (eds.) EST, pp. 178–181. IEEE Computer Society (2012)Google Scholar
  30. 30.
    Karapanos, E.: Modeling Users’ Experiences with Interactive Systems. SCI, vol. 436. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  31. 31.
    Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognitive Neuroscience 3, 71–86 (1991)CrossRefGoogle Scholar
  32. 32.
    Hyvärinen, A., Oja, E.: Independent component analysis: Algorithms and applications. Neural Netw. 13, 411–430 (2000)CrossRefGoogle Scholar
  33. 33.
    Liu, Q., Huang, R., Lu, H., Ma, S.: Face recognition using kernel-based fisher discriminant analysis. In: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 197–201 (2002)Google Scholar
  34. 34.
    Guo, G., Li, S.Z., Chan, K.L.: Support vector machines for face recognition. Image and Vision Computing 19, 631–638 (2001)CrossRefGoogle Scholar
  35. 35.
    Nefian, A.V., Hayess, M.H.: Hidden Markov Models for Face Recognition. In: Proc. International Conf. on Acoustics, Speech and Signal Processing (ICASSP 1998), vol. 5, pp. 2721–2724 (1998)Google Scholar
  36. 36.
    Degtyarev, N., Seredin, O.: Comparative testing of face detection algorithms. In: Elmoataz, A., Lezoray, O., Nouboud, F., Mammass, D., Meunier, J. (eds.) ICISP 2010. LNCS, vol. 6134, pp. 200–209. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Paloma Lopez-de-Arenosa
    • 1
  • Belén Díaz-Agudo
    • 1
  • Juan A. Recio-García
    • 1
  1. 1.Department of Software Engineering and Artificial IntelligenceUniversidad Complutense de MadridSpain

Personalised recommendations