Cognitive Processing

, Volume 20, Issue 4, pp 391–403 | Cite as

Fuzzy emotion: a natural approach to automatic facial expression recognition from psychological perspective using fuzzy system

  • Dewi Yanti LilianaEmail author
  • T. Basaruddin
  • M. Rahmat Widyanto
  • Imelda Ika Dian Oriza
Research Article


Many studies in automatic facial expression recognitions merely limit their focus on recognizing basic emotions, ignoring the fact that humans show various emotions in their daily life. Moreover, from psychological perspective humans express multiple emotions simultaneously. Up to now, researchers recognize two basic emotions at the same time, called mixed emotions. Nevertheless, the mixed emotion still does not reflect how humans express the emotion naturally. This paper advances the concept of mixed emotion into a generalized fuzzy emotion. Fuzzy emotion captures multiple emotions in a single image using fuzzy inference engine. We propose a fuzzy emotion framework which consists of processing system and knowledge system. The processing system extracts facial expression parameters, and the knowledge system employs a fuzzy knowledge-based engine, elicited from the psychologist knowledge to recognize facial expressions. Some advantages are offered: (1) no facial template comparison; (2) no training efforts needed; (3) moreover, fuzzy emotion can recognize ambiguous facial expressions adaptively. The experiment gives a recognition result with the highest accuracy rate of 0.90. A research agenda for future study of mixed emotion recognition is proposed.


Artificial intelligence Affective computing Emotion recognition Facial expression Fuzzy emotion Fuzzy system 



The authors would like to thank all the contributors as well as participants in this research. The first author would also like to thank the Indonesia Endowment Fund for Education (LPDP) for the doctoral study sponsorship.


This study was funded by the Indonesia Ministry of Research Technology and Higher Education through the Postgraduate Team Grant number 120/SP2H/PTNBH/DRPM/2018.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. Aly A, Tapus A (2015) An online fuzzy-based approach for human emotions detection: an overview on the human cognitive model of understanding and generating multimodal actions. Spring Tracts Adv Robot 106:185–212. CrossRefGoogle Scholar
  2. Chakrabarti D, Dutta D (2013) Facial expression recognition using eigenspaces. Proc Technol 10:755–761. CrossRefGoogle Scholar
  3. Chakraborty A, Konar A, Chakraborty UK, Chatterjee A (2009) Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans Syst Man Cybern Part A Syst Humans 39(4):726–743. CrossRefGoogle Scholar
  4. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23:681–685. CrossRefGoogle Scholar
  5. Du S, Tao Y, Martinez AM (2014) Compound facial expressions of emotion. Proc Natl Acad Sci 111:E1454–E1462. CrossRefPubMedGoogle Scholar
  6. Ekman P (1992) An argument for basic emotions. Cogn Emot 6:169–200. CrossRefGoogle Scholar
  7. Ekman P, Friesen WV (1975) Unmasking the face: a guide to recognizing emotions from facial clues. Prentice-Hall, OxfordGoogle Scholar
  8. Ilbeygi M, Shah-Hosseini H (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146. CrossRefGoogle Scholar
  9. Izard CE (1992) Basic emotions, relations among emotions, and emotion-cognition relations. Psychol Rev 99:561–565. CrossRefPubMedGoogle Scholar
  10. Kamachi M, Lyons M, Gyoba J (1997) The Japanese female facial expression (jaffe) database. Accessed 3 Oct 2018
  11. Larsen JT, McGraw A (2014) The case for mixed emotions. Soc Personal Psychol Compass 8:263–274. CrossRefGoogle Scholar
  12. Liliana DY, Widyanto MR, Basaruddin T (2016) Human emotion recognition based on active appearance model and semi-supervised fuzzy C-means. In: 2016 international conference on advanced computer science and information systems (ICACSIS), pp 439–445Google Scholar
  13. Liliana DY, Basaruddin C, Widyanto MR (2017) Mix emotion recognition from facial expression using SVM-CRF sequence classifier. In: Proceedings of international conference on algorithms, computing and systems - ICACS’17, pp 27–31.
  14. Loconsole C, Miranda CR, Augusto G et al (2014) Real-time emotion recognition—novel method for geometrical facial features extraction. In: Proceedings of the 9th international conference on computer vision theory and applications, pp 378–385Google Scholar
  15. Lucey P, Cohn JF, Kanade T et al (2010) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, CVPRW, pp 94–101.
  16. Mamdani EH (1974) Application of fuzzy algorithms for control of simple dynamic plant. Proc Inst Electr Eng 121:1585–1588. CrossRefGoogle Scholar
  17. Mavadati S, Mahoor M, Bartlett K, Trinh P, Cohn J (2013) DISFA: A Spontaneous Facial Action Intensity Database. IEEE Trans Affect Comput 4(2):151–160. CrossRefGoogle Scholar
  18. Nicolai A, Choi A (2015) Facial emotion recognition using fuzzy systems. In: 2015 IEEE international conference on systems, man, and cybernetics, pp 2216–2221.
  19. Sadeghi H, Raie A-A, Mohammadi M-R (2013) Facial expression recognition using geometric normalization and appearance representation. In: 2013 8th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 159–163Google Scholar
  20. Siddiqi MH, Ali R, Khan AM et al (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24:1386–1398. CrossRefPubMedGoogle Scholar
  21. Sujono, Gunawan AAS (2015) Face expression detection on Kinect using active appearance model and fuzzy logic. Proc Comput Sci 59:268–274. CrossRefGoogle Scholar
  22. Tzimiropoulos G, Pantic M (2013) Optimization problems for fast AAM fitting in-the-wild. In: 2013 IEEE international conference on computer vision, pp 593–600Google Scholar
  23. Vinciarelli A, Salamin H, Pantic M (2009) Social signal processing: understanding social actions through nonverbal behaviour analysis. In: 2009 IEEE computer society conference on computer vision and pattern recognition workshops, vol 3, pp 42–49.
  24. Zadeh LA (1965) Fuzzy sets. Inf Control 8:338–353. CrossRefGoogle Scholar

Copyright information

© Marta Olivetti Belardinelli and Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Computer ScienceUniversitas IndonesiaDepokIndonesia
  2. 2.Faculty of PsychologyUniversitas IndonesiaDepokIndonesia

Personalised recommendations