Facial Action Point Based Emotion Recognition by Principal Component Analysis

  • Anisha Halder
  • Arindam Jati
  • Garima Singh
  • Amit Konar
  • Aruna Chakraborty
  • R. Janarthanan
Part of the Advances in Intelligent and Soft Computing book series (AINSC, volume 131)


This paper proposes an alternative approach to emotion recognition of a subject from selected 36 facial action points marked at specific locations on their faces. Facial expressions obtained by the subjects enacting them are recorded, and the corresponding changes in marked action points are measured. The measurements reveal that the action points have wider variations in facial expressions containing diverse emotions. Considering 10 instances for each facial expression, and carrying the same emotion, experimented over 10 subjects, we obtain a set of 100 distance matrices, representing the distance between any two selected action points. The 100 matrices for each individual emotion are averaged, and the first principal component, representing the most prominent features of the average distance matrix is evaluated. During the recognition phase, the first Principal component obtained from the distance matrix of an unknown facial expression is evaluated, and its Euclidean distance with the first Principal component of each emotion is determined. The unknown facial expression is classified into emotion class j, if the Euclidean distance between the obtained principal component and that of j-th emotion class is minimum. Classification of 120 facial images, containing equal number of samples for six emotion classes, reveals an average classification accuracy of 92.5%, the highest being in relax and disgust and the least in fear and anger.


Action points Emotion Recognition Principal Component Analysis (PCA) 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Konar, A., Chakraborty, A. (eds.): Advances in Emotion Recognition. Wiley-Blackwell (to appear, 2011)Google Scholar
  2. 2.
    Uwechue, O.A., Pandya, S.A.: Human Face Recognition Using Third-Order Synthetic Neural Networks. Kluwer, Boston (1997)CrossRefGoogle Scholar
  3. 3.
    Biswas, B., Mukherjee, A.K., Konar, A.: Matching of digital images using fuzzy logic, vol. 35(2), pp. 7–11. AMSE Publication (1995)Google Scholar
  4. 4.
    Bhavsar, A., Patel, H.M.: Facial Expression Recognition Using Neural Classifier and Fuzzy Mapping. In: IEEE Indicon, Conference, Chennai, India (2005)Google Scholar
  5. 5.
    Guo, Y., Gao, H.: Emotion Recognition System in Images Based On Fuzzy Neural Network and HMM. In: Proc. 5th IEEE Int. Conf. on Cognitive Informatics (ICCI 2006). IEEE (2006)Google Scholar
  6. 6.
    Rizon, M., Karthigayan, M., Yaacob, S., Nagarajan, R.: Japanese face emotions classification using lip features. In: Geometric Modelling and Imaging (GMAI 2007), Universiti Malaysia Perlis, Jejawi, Perlis, Malaysia. IEEE (2007)Google Scholar
  7. 7.
    Kobayashi, H., Hara, F.: Measurement of the strength of six basic facial expressions by neural network. Trans. Jpn. Soc. Mech. Eng. (C) 59(567), 177–183 (1993)Google Scholar
  8. 8.
    Kobayashi, H., Hara, F.: Recognition of mixed facial expressions by neural network. Trans. Jpn. Soc. Mech. Eng. (C) 59(567), 184–189 (1993)Google Scholar
  9. 9.
    Kobayashi, H., Hara, F.: The recognition of basic facial expressions by neural network. Trans. Soc. Instrum. Contr. Eng. 29(1), 112–118 (1993)Google Scholar
  10. 10.
    Tsai, H., Lai, Y., Zhang, Y.: Using SVM to design facial expression recognition for shape and texture features. In: Proceedings of the Ninth International Conference on Machine Learning and Cybernetics, Qingdao, July 11-14. IEEE (2010)Google Scholar
  11. 11.
    Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions From Facial Clues. Prentice-Hall, Englewood Cliffs (1975)Google Scholar
  12. 12.
    Zhao, H., Wang, Z., Men, J.: Facial Complex Expression Recognition Based on Fuzzy Kernel Clustering and Support Vector Machines. In: Third International Conference on Natural Computation (ICNC 2007). IEEE (2007)Google Scholar
  13. 13.
    Lee, Y., Han, C.W., Shim, J.: Fuzzy Neural Networks and Fuzzy Integral Approach to Curvature-based Component Range Facial Recognition. In: 2007 International Conference on Convergence Information Technology. IEEE (2007)Google Scholar
  14. 14.
    Sun, J.M., Pei, X.S., Zhou, S.S.: Facial Emotion Recognition in modern distant system using SVM. In: Proceedings of the Seventh International Conference on Machine Learning and Cybernetics, Kunming. IEEE (July 2008)Google Scholar
  15. 15.
    Das, S., Halder, A., Bhowmik, P., Chakraborty, A., Konar, A., Nagar, A.K.: Voice and Facial Expression Based Classification of Emotion Using Linear Support Vector. In: 2009 Second International Conference on Developments in eSystems Engineering. IEEE (2009)Google Scholar
  16. 16.
    Picard, R.W., Vyzas, E., Healey, J.: Toward Machine Emotional Intelligence: Analysis of Affective Physiological State. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  17. 17.
    Pantic, M., Patras, I.: Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments From Face Profile Image Sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B 36(2), 433–449 (2006)CrossRefGoogle Scholar
  18. 18.
    Pantic, M., Patras, I., Rothkrantz, L.J.M.: Facial action recognition in face profile image sequences. In: Proc. IEEE Int. Conf. ICME, pp. 37–40 (2002)Google Scholar
  19. 19.
    Valstar, M.F., Pantic, M.: Combined Support Vector Machines and Hidden Markov Models for modelling Facial Action Temporal Dynamics. In: Article in a Conference ProceedingsGoogle Scholar
  20. 20.
    Valstar, M., Pantic, M.: Fully Automatic Facial Action Unit Detection and Temporal Analysis. In: Proceedings of the IEEE Computer society conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2006) (2006)Google Scholar
  21. 21.
    Ekman, P., Friesen, W.: Facial Action Coding System. Consulting Psychologist Press, Palo Alto (1978)Google Scholar
  22. 22.
    Ekman, P.: Methods For Measuring Facial Action. In: Scherer, K.R., Ekman, P. (eds.) Handbook of Methods In Nonverbal Bahavior Research, pp. 45–135. Cambridge University Press, New York (1982)Google Scholar
  23. 23.
    Chakraborty, A., Konar, A., Chakraborty, U.K., Chatterjee, A.: Emotion Recognition From Facial Expressions and Its Control Using Fuzzy Logic. IEEE Transactions on Systems, Man and Cybernetics (2009)Google Scholar
  24. 24.
    Khademi, M., Manzuri-Shalmani, M.T., Kiapour, M.H., Kiaei, A.A.: Recognizing Combinations of Facial Action Units with Different Intensity Using a Mixture of Hidden Markov Models and Neural Network. In: El Gayar, N., Kittler, J., Roli, F. (eds.) MCS 2010. LNCS, vol. 5997, pp. 304–313. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  25. 25.
    Tong, Y., Liao, W., Ji, Q.: Inferring Facial Action Units with Causal Relations. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR 2006) (2006)Google Scholar
  26. 26.
    Tian, Y.-L., Kanade, T., Cohn, J.F.: Evaluation of Gabor-Wavelet-Based Facial Action Unit Recognition in Image Sequences of Increasing Complexity. Article in a JournalGoogle Scholar
  27. 27.
    Cohn, J.F., Zlochower, A.J., Lien, J., Kanade, T.: Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. Psychophysiology 36(1), 35–43 (1999)CrossRefGoogle Scholar
  28. 28.
    Bartlett, M.S., Viola, P.A., Sejnowski, T.J., Golomb, B.A., Larsen, J., Hager, J.C., Ekman, P.: Classifying Facial Action. In: Touretzky, D., Mozer, M., Hasselmo, M. (eds.) Advances in Neural Information Processing Systems 8 (NIPS 1996), pp. 823–829. MIT Press (1996)Google Scholar
  29. 29.
    Huang, C.-L., Huang, Y.-M.: Facial Expression Recognition using Model- Based Feature Extraction and Action Parameters Classification. Journal of Visual Communication and Image Representation 8(3), 278–290 (1997)CrossRefGoogle Scholar
  30. 30.
    Konar, A.: Computational Intelligence: Principles, Techniques and Applications. Springer, Heidelberg (2005)MATHGoogle Scholar
  31. 31.
    Demsar, J.: Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7, 1–30 (2006)MathSciNetMATHGoogle Scholar
  32. 32.
    Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning AlgorithmsGoogle Scholar

Copyright information

© Springer India Pvt. Ltd. 2012

Authors and Affiliations

  • Anisha Halder
    • 1
  • Arindam Jati
    • 1
  • Garima Singh
    • 1
  • Amit Konar
    • 1
  • Aruna Chakraborty
    • 2
  • R. Janarthanan
    • 3
  1. 1.Department of Electronics and Tele-Communication EngineeringJadavpur UniversityKolkataIndia
  2. 2.Department of Computer Science and EngineeringSt. Thomas’ College of Engineering and TechnologyKolkataIndia
  3. 3.Department of ITJaya Engg. CollegeChennaiIndia

Personalised recommendations