Multimedia Tools and Applications

, Volume 78, Issue 11, pp 15033–15052 | Cite as

Smile intensity recognition in real time videos: fuzzy system approach

  • Vinola C.Email author
  • Vimala Devi K.


Facial emotion is a significant way of understanding or interpreting one’s inner thoughts. Real time video at any instant exhibits the emotion which serves as the input to the emotion recognition system. Many literatures propose different strategies in identifying the emotions by working on different features in the facial components, including geometrical, appearance and motion features. This paper considers the geometrical features as a prime component in deciding the intensity of the smile expressed in the real time videos of the AM-FED (Affectiva-MIT Facial Expression Dataset). Geometrical features considered in the work are the normalized Euclidean distance between the contributing LandMarkPoints (LMPs) of the eyes and the lip portions of the face. Fuzzy logic is applied to the system to effectively classify the intensity of the emotion, i.e., happiness or smile as Maximum, Moderate, Less and neutral. Being a Land mark based assessment, evaluating the normalized values of the Euclidean distance between LMPs for each frame of the video and then mapping the values of all the frames in a range helps the fuzzy decision making stage to relate the mapped values to the smile intensity of each frame. The average recognition rate obtained is 86.54%. The system contributes a less complex but nearly accurate smile intensity recognition model when compared to other computation intensive decision making models, with a practical significance in the customer or client’s mood/satisfactory identification in the online marketing and/or communication of the intensity of the smile of the conversing person to a visually challenged person.


Smile intensity LandMarkPoints Normalized Euclidean Distance Percentage Fuzzy system 



  1. 1.
    Baltrusaitis T et al (2011) Real-time inference of mental states from facial expressions and upper body gestures. In: Proc. of the IEEE Int'l Conference on Automatic Face and Gesture Recognition and Workshops, p 909–914.
  2. 2.
    Chakraborty A et al Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans Syst Man Cybern A Syst Hum 39(4):726–743.
  3. 3.
    Chen S, Tian YL, Liu Q, Metaxas DN (2013) Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. Image Vis Comput 31:175–185. CrossRefGoogle Scholar
  4. 4.
    Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183. CrossRefGoogle Scholar
  5. 5.
    Dornaika F, Moujahid A, Raducanu B (2013) Facial expression recognition using tracked facial actions: classifier performance analysis. Eng Appl Artif Intell 26:467–477. CrossRefGoogle Scholar
  6. 6.
    Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists PressGoogle Scholar
  7. 7.
    Esau N et al (2007) Real-time facial expression recognition using a fuzzy emotion model. Fuzzy Systems Conference. FUZZ-IEEE 2007. IEEE International, p 1–6.
  8. 8.
    Freire-Obregón D, Castrillón-Santana M (2015) An evolutive approach for smile recognition in video sequences. Int J Patt Recogn Artif Intell.
  9. 9.
    Guo G, Guo R, Li X (2013) Facial expression recognition influenced by human aging. IEEE Trans Affect Comput 4:291–298. CrossRefGoogle Scholar
  10. 10.
    Hablani R, Chaudari N, Tanwani S (2013) Recognition of facial expressions using local binary patterns of important facial parts. International Journal of Image Processing 7:163–170Google Scholar
  11. 11.
    Happy SL, Aurobinda R (2015) Automatic facial expression recognition using features of salient face patches. IEEE Trans Affect Comput 6(1):1–11. CrossRefGoogle Scholar
  12. 12.
    Kauser N, Sharma J Facial expression recognition using LBP template of facial parts and multilayer neural network. In: proc. of International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), p 445–449.
  13. 13.
    Kudiri KM, Said AMd, Nayan MY Human emotion detection through speech and facial expressions. International Conference on Computer and Information Sciences (ICCOINS), p 351–356.
  14. 14.
    Li P, Phung SL, Bouzerdom A, Tivive FHC (2010) Automatic recognition of smiling and neutral facial expressions. Digital Image Computing: Techniques and Applications, p 581–586.
  15. 15.
    Liu Z et al (2017) A facial expression emotion recognition based human-robot interaction system. IEEE/CAA J. Autom. Sinica 4(4):668–676. CrossRefGoogle Scholar
  16. 16.
    Liu Y et al Facial expression recognition with PCA and LBP features extracting from active facial patches. IEEE International Conference on Real-time Computing and Robotics (RCAR), p 368–373.
  17. 17.
    Mamdani EH, Assilian S (1975) An experiment in linguistic synthesis with a fuzzy logic controller. Int J Man Mach Stud 7(1):1–13. CrossRefzbMATHGoogle Scholar
  18. 18.
    Matlab 2016 Help Documentation-mapminmax algorithm in NeuralNetwork ToolboxGoogle Scholar
  19. 19.
    McDuff D, el Kaliouby R, Demirdjian D, Picard R (2013) Predicting online media effectiveness based on smile responses gathered over the internet. 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition.
  20. 20.
    McDuff D, el Kaliouby R, Senechal T, Amr M, Cohn J, Picard R (2013) Affectiva-MIT Facial Expression Dataset (AMFED): naturalistic and spontaneous facial expressions collected “In-the-Wild”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition WorkshopsGoogle Scholar
  21. 21.
    McDuff D et al Automatic measurement of ad preferences from facial responses gathered over the internet. Image Vis Comput:630–640.
  22. 22.
    Nicolle J, K Bailly, Chetouani M (2015) Facial action unit intensity prediction via hard multi-task metric learning for Kernel regression. 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition.
  23. 23.
    Patil JV, Bailke P Real time facial expression recognition using RealSense camera and ANN. In: Proc of International conference on Inventive Computation Technologies (ICICT), p 1–6.
  24. 24.
    Peihe W, Dekai Z Convexity of level sets of minimal graph on space form with nonnegative curvature. J Differ Equ 262:5534–5564.
  25. 25.
    Peihe W, Lingling Z Some geometrical properties of convex level sets of minimal graph on 2-dimensional Riemannian mani-folds. Nonlinear Anal Theory Methods Appl 130(1):1–13.
  26. 26.
    Qiao Y, Zeng K, Xu L (2016) A smartphone-based driver fatigue detection using fusion of multiple real-time facial features. In: proc of Consumer Communications & Networking Conference (CCNC), 2016 13th IEEE Annual, p 230–235.
  27. 27.
    Riu B et al Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018.
  28. 28.
    Shimada K et al Appearance-based smile intensity estimation by cascaded support vector machines. Pattern Recogn Lett:13–21.
  29. 29.
    Whitehill J, Littlewort G, Fasel I, Bartlett M (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106–2111CrossRefGoogle Scholar
  30. 30.
    Yang Z et al (2018) “Intermediate data caching optimization for multistage and parallel big data frameworks,” ARXivGoogle Scholar
  31. 31.
    Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2:219–229. CrossRefGoogle Scholar
  32. 32.
    Zhang K et al. (2016) Gender and smile classification using deep convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition Workshops, p 34–38.
  33. 33.
    Zimmermann HJ Fuzzy set theory-and its applications, Fourth Edition. Springer Science+Business Media, LLC.
  34. 34.
    Zisheng, Jun-Ichi, Masahide (2009) Facial-component-based bag of words and PHOG descriptor for facial expression recognition. In: Proc. of IEEE Int'l conference on Systems, Man and Cybernetics, p 1353–1358.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringFrancis Xavier Engineering CollegeTirunelveliIndia
  2. 2.Department of Computer Science and EngineeringVelammal Engineering CollegeChennaiIndia

Personalised recommendations