Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

  • Aruna Chakraborty
  • Amit Konar
Part of the Studies in Computational Intelligence book series (SCI, volume 234)


The chapter examines the scope of fuzzy relational approach to human emotion recognition from facial expressions, and its control. Commercial audio-visual movies pre-selected for exciting specific emotions have been presented before subjects to arouse their emotions. The video clips of their facial expressions describing the emotions are recorded and analyzed by segmenting and localizing the individual frames into regions of interest. Selected facial features such as eye-opening, mouth-opening and the length of eyebrow-constriction are next extracted from the localized regions. These features are then fuzzified, and mapped on to an emotion space by employing Mamdani type relational model. A scheme for the validation of the system parameters is also presented. The later part of the chapter provides a fuzzy scheme for controlling the transition of emotion dynamics toward a desired state using suitable audio-visual movies. Experimental results and computer simulations indicate that the proposed scheme for emotion recognition and control is simple and robust with a good level of experimental accuracy.


Membership Function Facial Expression Facial Image Emotion Recognition Fuzzy Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bezdek, J.C.: Fuzzy Mathematics in Pattern Classification, Ph.D. Thesis, Applied Mathematics Center. Cornell University, Ithaca (1973)Google Scholar
  2. 2.
    Biswas, B., Mukherjee, A.K.: Template Matching with Fuzzy Descriptors. J. of Inst. of Engineers (1997)Google Scholar
  3. 3.
    Black, M.T., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Com. Vis. 25, 23–48 (1997)CrossRefGoogle Scholar
  4. 4.
    Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proc. of ICMI 2004, Pennsylvania (October 2004)Google Scholar
  5. 5.
    Cohen, I., Garg, A., Huang, T.S.: Emotion recognition using multilevel HMM. In: Proc. of the NIPS Workshop on Affective Computing, Colorado (2000)Google Scholar
  6. 6.
    Cohen, I.: Facial Expression Recognition from Video Sequences, MS Thesis, Univ. of Illinois at Urbana-Champaign, Dept. of Electrical Engg. (2000)Google Scholar
  7. 7.
    Conati, C., Zhou, X.: Modeling students’ emotions from cognitive appraisal in educational games. In: Proc. of the Sixth Int. Conf. On Intelligent Tutoring System, France (2002)Google Scholar
  8. 8.
    Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Anal. Machine Intell. 21, 974–989 (1999)CrossRefGoogle Scholar
  9. 9.
    Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Prentice-Hall, New Jersey (1975)Google Scholar
  10. 10.
    Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. Pattern Anal. Machine Intell. 19, 757–763 (1997)CrossRefGoogle Scholar
  11. 11.
    Fellenz, W.A., Taylor, J.G., Cowie, R., Douglas-Cowie, E., Piat, F., Kollias, S., Orovas, C., Apolloni, B.: On emotion recognition of faces and of speech using neural networks, fuzzy logic and the ASSESS Systems. In: Proc. of the IEEE -INNS-ENNS Int. Joint Conf. Neural Networks, p. 2093 (2000)Google Scholar
  12. 12.
    Fernandez-Dols, J.M., Wallbotl, H., Sanchez, F.: Emotion Category Accessibility and the Decoding of Emotion from Facial Expression and Context. Journal of Nonverbal Behavior 15 (1991)Google Scholar
  13. 13.
    Gao, Y., Leung, M.K.H., Hui, S.C., Tananda, M.W.: Facial expression recognition from line-based caricatures. IEEE Trans. Systems, Man and Cybernetics- Part A: Systems and Humans 33(3) (May 2003)Google Scholar
  14. 14.
    Gordon, R.N.: The Structure of Emotions: Investigations in Cognitive Philosophy. Cambridge Studies in Philosophy. Cambridge University Press, Cambridge (1990)Google Scholar
  15. 15.
    Izumitani, K., Mikami, T., Inoue, K.: A Model of Expression Grade for Face Graphs Using Fuzzy Integral. System and Control 28(10), 590–596 (1984)Google Scholar
  16. 16.
    Kawakami, F., Morishima, S., Yamada, H., Harashima, H.: Construction of 3-D Emotion Space Using Neural Network. In: Proc. of the 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing, Iizuka, pp. 309–310 (1994)Google Scholar
  17. 17.
    Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice- Hall, New Jersey (1995)zbMATHGoogle Scholar
  18. 18.
    Kobayashi, H., Hara, F.: The Recognition of basic Facial Expressions by neural network. Trans. on the society of Instrument and Control Engineers 29(1), 112–118 (1993)Google Scholar
  19. 19.
    Kobayashi, H., Hara, F.: Measurement of the Strength of Six Basic Facial Expressions by Neural Network. Trans. of the Japan Society of Mechanical Engineers (C) 59(567), 177–183 (1993)Google Scholar
  20. 20.
    Kobayashi, H., Hara, F.: Recognition of Mixed Facial Expressions by Neural Network, ibid., pp. 184–189 (1993)Google Scholar
  21. 21.
    Konar, A.: Computational Intelligence: Principles, Techniques and Applications. Springer, Heidelberg (2005)zbMATHGoogle Scholar
  22. 22.
    Krammer, A.F., Sirevaag, E.J., Braune, R.: A psycho-physiological assessment of operator workload during simulated flight missions. Human Factors 29(2), 145–160 (1987)Google Scholar
  23. 23.
    Lanitis, A., Taylor, C.J., Cootes, T.F.: Automatic interpretation and coding of face images using flexible models. IEEE Trans. Pattern Anal. Machine Intell. 19, 743–756 (1997)CrossRefGoogle Scholar
  24. 24.
    Li, H., Roivainan, P., Forchheimer, R.: 3D Motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Machine Intell. 15, 545–555 (1993)CrossRefGoogle Scholar
  25. 25.
    Li, X., Ji, Q.: Active affective state detection and user assistance with dynamic Bayesian networks. IEEE Trans. Systems, Man and Cybernetics-Part A: Systems and Humans 35(1) (January 2005)Google Scholar
  26. 26.
    Mase, K.: Recognition of facial expression from optical flow. Proc. IEICE Trans., Special Issue Coput. Vis. And Its Applications 74(10), 3474–3483 (1991)Google Scholar
  27. 27.
    Pantic, M., Rothkrantz, L.: Automatic analysis of facial expressions: State of the Art. IEEE Trans. Pattern Anal. Machine Intell. 22(2), 1424–1445 (2000)CrossRefGoogle Scholar
  28. 28.
    Pedrycz, W., Gomide, F.: An Introduction to Fuzzy Sets: Analysis and Design. MIT Press, Massachusetts (1998)zbMATHGoogle Scholar
  29. 29.
    Pentland, E.A.P.: Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence 19(7), 757–763 (1997)CrossRefGoogle Scholar
  30. 30.
    Picard, R.: Affective Computing. MIT Press, Cambridge (1997)Google Scholar
  31. 31.
    Picard, R.W., Vyzas, E., Healy, J.: Toward machine emotional intelligence: analysis of affective psychological states. IEEE Trans. on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  32. 32.
    Rani, P., Sarkar, N., Adams, J.: Anxiety-based affective communication for implicit human-machine interaction. Advanced Engineering, Informatics 21(3), 323–334 (2007)CrossRefGoogle Scholar
  33. 33.
    Rani, P., Sarkar, N., Smith, C., Kirby, L.: Anxiety detecting robotic systems-towards implicit human-robot collaboration. Robotica 22(1), 83–93 (2004)CrossRefGoogle Scholar
  34. 34.
    Rosenblum, M., Yacoob, Y., Davis, L.: Human expression recognition from motion using a radial basis function network architecture. IEEE Trans. Neural Networks 7, 1121–1138 (1996)CrossRefGoogle Scholar
  35. 35.
    Scheirer, J., Fernadez, R., Klein, J., Picard, R.: Frustrating the user on purpose: a step toward building an affective computer. Interacting with Computers 14(2), 93–118 (2002)Google Scholar
  36. 36.
    Simon, H.: Motivational and Emotional Control of Cognition. In: Models of Thought, pp. 29–38. Yale University Press, New Haven (1979)Google Scholar
  37. 37.
    Terzopoulus, D., Waters, K.: Analysis and Synthesis of facial image sequences using physical and anatomical models. IEEE Trans. Pattern Anal. Machine Intell. 15, 569–579 (1993)CrossRefGoogle Scholar
  38. 38.
    Tian, Y., Kanade, T., Cohn, J.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Machine Intell. 23(2), 97–116 (2001)CrossRefGoogle Scholar
  39. 39.
    Ueki, N., Morishima, S., Harashima, H.: Expression Analysis/Synthesis System Based on Emotion Space Constructed by Multilayered Neural Network. Systems and Computers in Japan 25(13) (1994)Google Scholar
  40. 40.
    Uwechue, O.A., Pandya, S.A.: Human Face Recognition Using Third-Order Synthetic Neural Networks. Kluwer Academic publishers, Boston (1997)Google Scholar
  41. 41.
    Vanger, P., Honlinger, R., Haykin, H.: Applications of Synergetic in Decoding Facial Expressions of Emotions. In: Proc. of Int. Workshop on Automatic face and Gesture recognition, Zurich, pp. 24–29 (1995)Google Scholar
  42. 42.
    Vasilakos, A., Pedrycz, W.: Ambient Intelligence, Wireless Networking and Ubiquitous Computing. Artech House, Norwood (June 2006)Google Scholar
  43. 43.
    Yacoob, Y., Davis, L.: Computing spatio-temporal representations of human faces. In: Proc. of Computer Vision and Pattern Recognition, pp. 70–75. IEEE Computer Society Conference, Los Alamitos (1994)CrossRefGoogle Scholar
  44. 44.
    Yacoob, Y., Davis, L.: Recognizing human facial expression from long image sequences using optical flow. IEEE Trans. Pattern Anal. Machine Intell. 16, 636–642 (1996)CrossRefGoogle Scholar
  45. 45.
    Yamada, H.: Visual Information for categorizing Facial expression of Emotion. Applied Cognitive Psychology 7, 257–270 (1993)CrossRefGoogle Scholar
  46. 46.
    Zeng, Z., Fu, Y., Roisman, G.I., Wen, Z., Hu, Y., Huang, T.S.: Spontaneous emotional facial expression detection. J. of Multimedia 1(5) (August 2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Aruna Chakraborty
    • Amit Konar

      There are no affiliations available

      Personalised recommendations