Spontaneous Facial Expression Analysis Using Optical Flow Technique

  • L. SidavongEmail author
  • S. Lal
  • T. Sztynda
Part of the Smart Sensors, Measurement and Instrumentation book series (SSMI, volume 29)


Investigation of emotions manifested through facial expressions has valuable applications in predictive behavioural studies. A potential application may be to impart intelligence to surveillance systems such as Closed-Circuit Television (CCTV) systems for recognition of emotional facial expressions. A facial recognition program tailored to evaluating facial behaviour for real time application can be met if patterns of emotions can be detected. An exploratory analysis of optical flow data was conducted with an aim to detect patterns and trends to differentiate between the emotional facial expressions: amusement, sadness and fear from the frontal and profile facial orientations. Analysis was in the form of emotion maps constructed from feature vectors obtained by using the Lucas-Kanade implementation of optical flow. Classification of individual emotions showed recognition of amusement was much greater in comparison to the recognition of the negative emotions, sadness and fear. Recognition was not negatively affected using reduced set of feature vectors derived from the emotion maps. Further investigation is necessary to assess the utility of emotion maps to visualise feature representations of emotional expression.



This research is supported by an Australian Government Research Training Program Scholarship and UTS Science Faculty Research funds. Thanks to Dr. Budi Jap who proposed the feature extraction methods.


  1. 1.
    D. Wilson, A. Sutton, Open-street CCTV in Australia. Australian Institute of Criminology, Trends and Issues in Crime and Criminal Justice (2003), Accessed 5 June (2016)
  2. 2.
    F.M. Donald, C.H.M. Donald, Task disengagement and implications for vigilance performance in CCTV surveillance. Cogn. Technol. Work 17, 121–130 (2015)CrossRefGoogle Scholar
  3. 3.
    A. Isnard, Can surveillance cameras be successful in preventing crime and controlling anti-social behaviours, in The Character, Impact and Prevention of Crime in Regional Australia Conference, Townsville, Queensland, Australia (2001)Google Scholar
  4. 4.
    S. Freud, J. Strachey, The Ego and the Id (Norton, New York, 1962)Google Scholar
  5. 5.
    C. Darwin, The Expression of the Emotions in Man and Animals (Oxford University Press, New York, 1872)CrossRefGoogle Scholar
  6. 6.
    S.S. Tomkins, V.E. Demos, Exploring Affect: The Selected Writings of Silvan S Tomkins (Cambridge University Press, 1995)Google Scholar
  7. 7.
    M.D. Matsumoto, P. Ekman, Facial expression analysis. Scholarpedia 3, 4237–4248 (2008)CrossRefGoogle Scholar
  8. 8.
    B. Fasel, J. Luettin, Automatic facial expression analysis: a survey. Pattern Recogn. 36, 259–275 (2003)CrossRefGoogle Scholar
  9. 9.
    J.T. Cacioppo, R.E. Petty, Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. J. Pers. Soc. Psychol. 50, 260–268 (1986)CrossRefGoogle Scholar
  10. 10.
    U. Dimberg, Facial electromyography and emotional reactions. Psychophysiology 27, 481–494 (1990)CrossRefGoogle Scholar
  11. 11.
    R.W. Buck, V.J. Savin, Communicating of affect through facial expressions in humans. J. Personal. Soc. Psychol. 23, 362–371 (1972)CrossRefGoogle Scholar
  12. 12.
    P. Ekman, G. Roper, G. C. Hager, Deliberate facial movement. Development, 886–891 (1980)Google Scholar
  13. 13.
    P. Ekman, W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement (Consulting Psychological Press, Palo Alto, California, 1978)Google Scholar
  14. 14.
    P. Ekman, W.V. Friesen, J.C. Hager, Facial Action Coding System: The Manual on CD Rom, ed. by Human Face, Salt Lake City (2002)Google Scholar
  15. 15.
    Y.L. Tian, T. Kanade, J. Cohn, Recognising action units for facial expression analysis, in Handbook of Face Recognition (Springer, New York, 2003) pp. 247–275Google Scholar
  16. 16.
    M.S. Bartlett, J.R. Movellan, G.C. Littlewort, B. Braathen, M.G. Frank, T.J. Sejnowski, Towards automatic recognition of spontaneous facial actions, in What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS), ed. by P. Ekman, E. Rosenberg, 2nd edn. (Oxford University Press, NY, 2005), pp. 393–412Google Scholar
  17. 17.
    C.P. Sumanthi, T. Santhanam, M. Mahadevi, Automatic facial expression analysis: a survey. Int. J. Comput. Sci. Eng. Survey 3, 47–59 (2012)Google Scholar
  18. 18.
    A. Sanchez, J.V. Ruiz, A.B. Moreno, A.S. Montemayor, J. Hernandez, J.J. Pantrigo, Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74, 1272–1282 (2011)Google Scholar
  19. 19.
    A. Psaltis, Optical Flow for Dynamic Facial Expression Recognition (Faculty of Science, Utrecht University, Master, 2013)Google Scholar
  20. 20.
    F. Zhang, Y. Gao, J.D. Bakos, Lucas-Kanade optical flow estimation on the TI C66x digital signal processor, in IEEE High Performance Extreme Computing Conference (HPEC) (2014), pp. 1–6Google Scholar
  21. 21.
    A.R. Naghsh-Nilchi, M. Roshanzamir, An efficient algorithm for motion detection based facial expression recognition using optical flow. Int. Sch. Sci. Res. Innov. 2, 2725–2729 (2008)Google Scholar
  22. 22.
    M.Z. Uddin, T.S. Kim, B.C. Song, An optical flow feature-based robust facial expression recognition with HMM from video. Int. J. Innov. Comput. Inf. Control 9, 1409–1421 (2013)Google Scholar
  23. 23.
    D. Patel, S. Upadhyay, Optical flow measurement using Lucas Kanade method. Int. J. Comput. Appl. 61, 6–10 (2013)Google Scholar
  24. 24.
    T. Kanade, J. Cohn, Y.L. Tian, Comprehensive database for facial expression analysis, in Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, (2000), pp. 46–53Google Scholar
  25. 25.
    C. Grossard, O. Grynspan, S. Serret, A. Jouen, K. Bailly, D. Cohen, Serious games to teach social interactions and emotions to individuals with autism spectrum disorders (ASD). Comput. Educ. 113, 195–211 (2017)CrossRefGoogle Scholar
  26. 26.
    P. Werner, A. Al-Hamadi, R. Niese, S. Walter, S. Gruss, H.C. Traue, Towards pain monitoring: facial expression, head pose, a new database, an automatic system and remaining challenges, in Proceedings of the British Machine Vision Conference (BMVA Press, 2013), pp. 1–13.
  27. 27.
    A.B. Ashraf, S. Lucey, J.F. Cohn, T. Chen, Z. Ambadar, K.M. Prkachin, P.E. Solomon, The painful face—pain expression recognition using active appearance models. Image Vis. Comput. 27, 1788–1796 (2009)CrossRefGoogle Scholar
  28. 28.
    T. Lawson, R. Rogerson, M. Barnacle, A comparison between the cost effectiveness of CCTV and improved street lighting as a means of crime reduction. Comput. Environ. Urban Syst. (2017).
  29. 29.
    J.H. Yin, S.A. Velastin, A.C. Davies, Image processing techniques for crowd density estimation using a reference image, in ACCV (1995), pp. 489–498Google Scholar
  30. 30.
    C.C. Chibelushi, F. Bourel, A.A. Low, Robust facial expression recognition using a state-based model of spatially-localised facial dynamics, in Proceedings of Fifth IEEE International Conference on Automatic Face and Gesture Recognition (2002), pp. 106–111Google Scholar
  31. 31.
    I. Kotsia, I. Buciu, I. Pitas, An analysis of facial expression recognition under partial facial image occlusion. Image Vis. Comput. 26, 1052–1067 (2008)CrossRefGoogle Scholar
  32. 32.
    H.K. Ekenel, R. Stiefelhagen, Why is facial occlusion a challenging problem? in Proceedings of Third International Conference on Advances in Biometrics, ICB 2009, Alghero, Italy, 2–5 June, 2009, ed. by M. Tistarelli, M.S. Nixon (Springer, Berlin, Heidelberg)Google Scholar
  33. 33.
    M. Dyck, M. Winbeck, S. Leigberg, R.C. Gur, K. Mathiak, Recognition profile of emotions in natural and virtual faces PLoS ONE 3:e3628 (2008).
  34. 34.
    C. Anthica, M.K. Venkatesha, B. Suryanarayana Adiga, A survey on facial expression databases. Int. J. Eng. Sci. Technol. 2, 5158–5174 (2010)Google Scholar
  35. 35.
    M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, J. Movellan, Fully automatic facial action recognition in spontaneous behavior, in 7th International Conference on Automatic Face and Gesture Recognition, FGR 2006 (2006), pp. 223–230Google Scholar
  36. 36.
    T. Pfister, X. Li, G. Zhao, M. Pietikainen, Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework, in ICCV Workshops (2011), pp. 868–875Google Scholar
  37. 37.
    J.J. Gross, R.W. Levenson, Emotion elicitation using films, Cogn. Emot. 87–108 (1995)Google Scholar
  38. 38.
    L. Sidavong, S. Lal, T. Sztynda, Spontaneous facial expression analysis using optical flow, in 2017 Eleventh International Conference on Sensing Technology (ICST), Sydney, NSW, Australia (2017).
  39. 39.
    B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in Proceedings of the 7th Joint Conference on Artificial Intelligence (Vancouver British Coloumbia, Canada, 1981)Google Scholar
  40. 40.
    R.W. Levensen, The autonomic nervous system and emotion. Emot. Rev. 6, 110–112 (2014)Google Scholar
  41. 41.
    A. Greco, A. Lanata, L. Citi, N. Vanello, G. Valenza, E.P. Scilingo, Skin admittance measurement for emotion recognition: a study over frequency sweep. Electronics 5, 46 (2016)CrossRefGoogle Scholar
  42. 42.
    P.C. Ellsworth, K. Scherer, Appraisal processes in emotion, in Handbook of Affective Sciences, ed. by R.J. Davidson, et al. (Oxford University Press, Oxford New York, 2003)Google Scholar
  43. 43.
    P.C. Ellsworth, Appraisal theory: old and new questions. Emot. Rev. 5, 125–131 (2013)CrossRefGoogle Scholar
  44. 44.
    B. Xia, Which Facial Expressions Can Reveal Your Gender? A Study With 3D Faces, arXiv:1805.00371 (2018)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Life SciencesUniversity of Technology SydneySydneyAustralia

Personalised recommendations