Combining Features for Recognizing Emotional Facial Expressions in Static Images

  • Jiří Přinosil
  • Zdeněk Smékal
  • Anna Esposito
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5042)


This work approaches the problem of recognizing emotional facial expressions in static images focusing on three preprocessing techniques for feature extraction such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and Gabor filters. These methods are commonly used for face recognition and the novelty consists in combining features provided by them in order to improve the performance of an automatic procedure for recognizing emotional facial expressions. Testing and recognition accuracy were performed on the Japanese Female Facial Expression (JAFFE) database using a Multi-Layer Perceptron (MLP) Neural Network as classifier. The best classification accuracy on variations of facial expressions included in the training set was obtained combining PCA and LDA features (93% of correct recognition rate), whereas, combining PCA, LDA and Gabor filter features the net gave 94% of correct classification on facial expressions of subjects not included in the training set.


Principal Component Analysis Linear Discriminant Analysis Gabor filters facial features basic emotions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abbound, B., Davoine, F.: Facial Expression Recognition and Synthesis Based on an Appereance Model. Signal Processing: Image Communication 19, 723–740 (2004)Google Scholar
  2. 2.
    La Barre, W.: The Cultural basis of Emotions Gnd gestures. Journal of Personality 16, 49–68 (1947)CrossRefGoogle Scholar
  3. 3.
    Birdwhistell, R.: Kinesies and Context. University of Pennsylvania Press, Philadelphia (1970)Google Scholar
  4. 4.
    Cohen, I., Cozman, F.G., Sebe, N., Cirelo, M.C., Huang, T.S.: Semisupervised Learning of Classifiers: Theory, Algorithms and their Application to Human-Computer Interaction. IEEE Transactions on PAMI 26, 1553–1567 (2004)CrossRefGoogle Scholar
  5. 5.
    Darwin, C.: The Expression of the Emotions in Man and Animals. J. Murray, London (1872)CrossRefGoogle Scholar
  6. 6.
    Ekman, P., Friesen, W.: Constants across Cultures in the Face and Emotion. Journal of Personality and Social Psychology (1971)Google Scholar
  7. 7.
    Ekman, P., Friesen, W.: Emotional Facial Action Coding System. Unpublished manual (1978)Google Scholar
  8. 8.
    Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  9. 9.
    Ekman, P.: The Argument and Evidence about Universals in Facial Expressions of Emotion. In: Wagner, H., Manstead, A. (eds.) Handbook of Social Psychophysiology, pp. 143–164. Wiley, Chichester (1989)Google Scholar
  10. 10.
    Ekman, P.: Facial Expression of Emotion: New Findings. New Questions. Psychological Science 3, 34–38 (1992)CrossRefGoogle Scholar
  11. 11.
    Fasel, B., Luettin, J.: Automatic Facial Expression Analysis: A Survey. Pattern Recognition 36(1), 259–275 (2003)CrossRefzbMATHGoogle Scholar
  12. 12.
    Fisher, R.A.: The Statistical Utilization of Multiple Measurements. Annali of Eugenics 8, 376–386 (1938)CrossRefzbMATHGoogle Scholar
  13. 13.
    Fridlund, A.J.: The New Ethnology of Human Facial Expressions. In: Russell, J.A., Fernandez-Dols, J. (eds.) The Psychology of Facial Expressions, pp. 103–129. Cambridge University Press, Cambridge (1997)CrossRefGoogle Scholar
  14. 14.
    Hong, H., Neven, H., von der Malsburg, C.: Online Facial Expression Recognition Based on Personalized Galleries. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp. 354–359 (1998)Google Scholar
  15. 15.
    Huang, C.L., Huang, Y.M.: Facial Expression Recognition Using Model-Based Feature Extraction and Action Parameters Classification. Journal of Visual Communication and Image Representation 8(3), 278–290 (1997)CrossRefGoogle Scholar
  16. 16.
    Huang, S.H., Wu, Q.J., Lai, S.H.: Improved AdaBoost-based Image Retrieval with Relevance Feedback Via Paired Feature Learning. Multimedia Systems 12, 14–26 (2006)CrossRefGoogle Scholar
  17. 17.
    Izard, C.E., Dougherty, L.M., Hembree, E.A.: A System for Identifying Affect Expressions by Holistic Judgments. Unpublished manuscript. Available from Instructional Resource Center, University of Delaware (1983)Google Scholar
  18. 18.
    Izard, C.E.: Innate and Universal Facial Expressions: Evidence from Developmental and Cross-Cultural Research. Psychological Bulletin 115, 288–299 (1994)CrossRefGoogle Scholar
  19. 19.
    Jollife, I.T.: Principal Component Analysis, 2nd edn. Springer, New York (2002)Google Scholar
  20. 20.
    Kamachi, M., Lyons, M., Gyoba, J.: Japanese Female Facial Expression Database, Psychology Department in Kyushu University,
  21. 21.
    Klinerberg, O.: Emotional Expression in Chinese Literature. Journal of Abnormal and Social Psychology 33, 517–520 (1938)CrossRefGoogle Scholar
  22. 22.
    Lee, Y., Kim, I., Shim, J., Marshall, D.: 3D Facial Image Recognition Using a Nose Volume and Curvature Based Eigenface. In: Kim, M.-S., Shimada, K. (eds.) GMP 2006. LNCS, vol. 4077, pp. 616–622. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  23. 23.
    Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic Classification of Single Facial Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 21, 1357–1362 (1999)CrossRefGoogle Scholar
  24. 24.
    Martýnez, A.M.: Recognition of Partially Occluded and/or Imprecisely Localized Faces Using a Probabilistic Approach. In: Proceeding of the International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 712–717 (2000)Google Scholar
  25. 25.
    Martýnez, A.M.: PCA versus LDA. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 228–233 (2001)CrossRefGoogle Scholar
  26. 26.
    Moghaddam, B., Pentland, A.: Probabilistic Visual Learning for Object Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 696–710 (1997)CrossRefGoogle Scholar
  27. 27.
    Moon, H., Phillips, P.J.: Analysis of PCA-based Face Recognition Algorithms. In: Bowyer, K.J., Phillips, P.J. (eds.) Empirical Evaluation Techniques in Computer Vision. IEEE Computer Soceity, Los Alamitos (1998)Google Scholar
  28. 28.
    Pantic, M., Rothkrantz, J.M.: Automatic Analysis of Facial Expression: The State of the Art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  29. 29.
    Pantic, M., Rothkrantz, J.M.: Expert System for Automatic Analysis of Facial Expression. Image and Vision Computing Journal 18(11), 881–905 (2000)CrossRefGoogle Scholar
  30. 30.
    Papageorgiou, C., Oren, M., Poggio, T.: A General Framework for Object Detection. In: International Conference on Computer Vision, pp. 992–998 (1998)Google Scholar
  31. 31.
    Petkov, N., Wieling, M.B.: Gabor Filtering Augmented with Surround Inhibition for Improved Contour Detection by Texture Suppression. Perception 33, 68c (2004)Google Scholar
  32. 32.
    Phillips, P.J., Moon, H., Rauss, P., Rizvi, S.A.: The FERET Evaluation Methodology for Face-Recognition Algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1090–1104 (2000)CrossRefGoogle Scholar
  33. 33.
    Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W.: Overview of the Face Recognition Grand Challenge. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (2005)Google Scholar
  34. 34.
    Roth, D., Yang, M., Ahuja, N.: A SNoW-Based Face Detector. Advances in Neural Information Processing Systems, 855–861 (2000)Google Scholar
  35. 35.
    Russell, J.A.: A Circumplex Model of Affect. Journal of Personality and Social Psychology 39, 1161–1171 (1980)CrossRefGoogle Scholar
  36. 36.
    Ryu, H., Chun, S.S., Sull, S.: Multiple Classifiers Approach for Computational Efficiency in Multi-scale Search Based Face Detection. In: Jiao, L., Wang, L., Gao, X.-b., Liu, J., Wu, F. (eds.) ICNC 2006. LNCS, vol. 4221. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  37. 37.
    Samaria, F., Harter, A.: The ORL Database of Faces, AT&T Laboratories Cambridge University,
  38. 38.
    Schneiderman, H., Kanade, T.: A Statistical Method for 3D Object Detection Applied to Faces and Cars. In: International Conference on Computer and Pattern Recognition, vol. 1, pp. 746–751 (2000)Google Scholar
  39. 39.
    Schlosberg, H.: Three Dimensions of Emotion. The Psychological Review 61(2), 81–88 (1953)CrossRefGoogle Scholar
  40. 40.
    Simoncelli, E.P., Olshausen, B.A.: Natural Image Statistics and Neural Representation. Annual Review of Neuroscience 24, 1193–1216 (2001)CrossRefGoogle Scholar
  41. 41.
    Sung, K., Poggio, T.: Example-Based Learning for View-Based Face Detection. IEEE Transaction on Pattern Analyses and Machine Intelligence 20, 39–51 (1998)CrossRefGoogle Scholar
  42. 42.
    Tomkins, S.S.: Affect Theory. In: Scherer, K.R., Ekman, P. (eds.) Approaches to Emotion, pp. 163–196. Erlbaum, Hillsdale (1984)Google Scholar
  43. 43.
    Turk, M., Pentland, A.: Face Recognition Using Eigenfaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591 (1991)Google Scholar
  44. 44.
    Viola, A.P., Jones, M.J.: Robust Real-Time Face Detection. International Journal of Computer Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  45. 45.
    White, G.M.: Emotion Inside Out the Anthropology of Affect. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 29–40. Guilford Press, New York (1993)Google Scholar
  46. 46.
    Zhao, J., Kearney, G.: Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs. In: Proceedings of the International Conference on Neural Information Processing, vol. 1, pp. 454–457 (1996)Google Scholar
  47. 47.
    Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face Recognition: A Literature Survey. ACM, Computing Surveys 35(4), 399–458 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Jiří Přinosil
    • 1
  • Zdeněk Smékal
    • 1
  • Anna Esposito
    • 2
  1. 1.Brno University of TechnologyCzech Republic
  2. 2.Department of Psychology and IIASSSecond University of NaplesItaly

Personalised recommendations