Skip to main content
SpringerLink
Log in

An Efficient Local Feature-Based Facial Expression Recognition System

  • Research Article - Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

In this paper, a novel approach to recognizing certain facial expressions using time-sequential depth videos is proposed. Local directional pattern features are extracted from time-sequential depth faces. Robust local features are then applied with hidden Markov models for successfully obtaining facial expressions. The proposed approach shows superior recognition performance compared to conventional approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen Z.-X., Liu C.-Y., Chang F.-L., Han X.-Z.: Fast face detection algorithm based on improved skin-color model. Arab. J. Sci. Eng. 38(3), 629–635 (2013)

    Article  MATH  Google Scholar 

  2. Kim, D.-S.; Jeon, I.-J.; Lee, S.-Y.; Rhee, P.-K.; Chung, D.-J.: Embedded face recognition based on fast genetic algorithm for intelligent digital photography. IEEE Trans. Consum. Electron. 52(3), 726–734 (2006)

  3. Padgett, C.; Cottrell, G.: Representation face images for emotion classification. In: Advances in Neural Information Processing Systems, vol. 9, pp. 855–858. MIT Press, Cambridge (1997)

  4. Mitra S., Acharya T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 37(3), 311–324 (2007)

    Article  Google Scholar 

  5. Donato G., Bartlett M.S., Hagar J.C., Ekman P., Sejnowski T.J.: Classifying facial actions. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 974–989 (1999)

    Article  Google Scholar 

  6. Ekman P., Priesen W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  7. Meulders M., Boeck P.D., Mechelen I.V., Gelman A.: Probabilistic feature analysis of facial perception of emotions. Appl. Stat. 54(4), 781–793 (2005)

    MATH  Google Scholar 

  8. Calder A.J., Burton A.M., Miller P., Young A.W., Akamatsu S.: A principal component analysis of facial expressions. Vis. Res. 41, 1179–1208 (2001)

    Article  Google Scholar 

  9. Dubuisson S., Davoine F., Masson M.: A solution for facial expression representation and recognition. Signal Process. Image Commun. 17, 657–673 (2002)

    Article  Google Scholar 

  10. Buciu, I.; Kotropoulos, C.; Pitas, I.: ICA and Gabor representation for facial expression recognition. In: Proceedings of the IEEE, pp. 855–858 (2003)

  11. Chen F., Kotani K.: Facial expression recognition by supervised independent component analysis using MAP estimation. IEICE Trans. Inf. Syst. E91-D(2), 341–350 (2008)

    Article  Google Scholar 

  12. Hyvarinen A., Karhunen J., Oja E.: Independent Component Analysis. Wiley, New York (2001)

    Book  Google Scholar 

  13. Karklin Y., Lewicki M.S: Learning higher-order structures in natural images. Netw. Comput. Neural Syst. 14, 483–499 (2003)

    Article  Google Scholar 

  14. Bartlett, M.S.; Donato, G.; Movellan, J.R.; Hager, J.C.; Ekman, P.; Sejnowski, T.J.: Face image analysis for expression measurement and detection of deceit. In: Proceedings of the Sixth Joint Symposium on Neural Computation, pp. 8–15 (1999)

  15. Chao-Fa C., Shin F.Y.: Recognizing facial action units using independent component analysis and support vector machine. Pattern Recognit. 39, 1795–1798 (2006)

    Article  MATH  Google Scholar 

  16. Calder A.J., Young A.W., Keane J.: Configural information in facial expression perception. J. Exp. Psychol. Hum. Percept. Perform. 26(2), 527–551 (2000)

    Article  Google Scholar 

  17. Lyons, M.J.; Akamatsu, S.; Kamachi, M.; Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205 (1998)

  18. Bartlett M.S., Movellan J.R., Sejnowski T.J.: Face ecognition by independent component analysis. IEEE Trans. Neural Netw. 13(6), 1450–1464 (2002)

    Article  Google Scholar 

  19. Liu C.: Enhanced independent component analysis and its application to content based face image retrieval. IEEE Trans. Syst. Man Cybern. Part B Cybern. 34(2), 1117–1127 (2004)

    Article  Google Scholar 

  20. Phillips P.J., Wechsler H., Huang J., Rauss P.: The FERET database and evaluation procedure for face-recognition algorithms. Image Vis. Comput. 16, 295–306 (1998)

    Article  Google Scholar 

  21. Uddin M.Z., Lee J.J., Kim T.-S.: An enhanced independent component-based human facial expression recognition from video. IEEE Trans. Consum. Electron. 55(4), 2216–2224 (2009)

    Article  Google Scholar 

  22. Ojala T., Pietikäïnen M., Mäenpää T.: Multiresolution gray scale and rotation invariant texture analysis with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell 24, 971–987 (2002)

    Article  Google Scholar 

  23. Shan, C.; Gong, S.; McOwan, P.: Robust facial expression recognition using local binary patterns. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp. 370–373 (2005)

  24. Shan C., Gong S., McOwan P.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput 27, 803–816 (2009)

    Article  Google Scholar 

  25. Jabid, T.; Kabir, M.H.; Chae, O.: Local directional pattern (LDP) a robust image descriptor for object recognition. In: Proceedings of the IEEE Advanced Video and Signal Based Surveillance (AVSS), pp. 482–487 (2010)

  26. Uddin, M.Z.; Lee, J.J.; Kim, T.-S.: Independent Shape Component-based Human Activity Recognition via Hidden Markov Model. Appl. Intell. 33(2), 193–206 (2010)

  27. Zhu Y., De Silva L.C., Ko C.C.: Using moment invariants and HMM in facial expression recognition. Pattern Recognit. Lett. 23(1–3), 83–91 (2002)

    Article  MATH  Google Scholar 

  28. Cohen I., Sebe N., Garg A., Chen L.S., Huang T.S.: Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91, 160–187 (2003)

    Article  Google Scholar 

  29. Aleksic P.S., Katsaggelos A.K.: Automatic facial expression recognition using facial animation parameters and multistream HMMs. IEEE Trans. Inf. Secur. 1, 3–11 (2006)

    Article  Google Scholar 

  30. Uddin M.Z.: A depth video-based facial expression recognition system. IETE Tech. Rev. 29(2), 169–177 (2012)

    Article  Google Scholar 

  31. Li., W, Zhang, Z.; Liu, Z.: Action recognition based on a bag of 3d points. In: Proceedings of Workshop on Human Activity Understanding from 3D Data, pp. 9–14 (2010)

  32. Li W., Zhang Z., Liu Z.: Expandable data-driven graphical modeling of human actions based on salient postures. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1499–1510 (2008)

    Article  Google Scholar 

  33. Oreifej, O.; Liu, Z.: Hon4d: histogram of oriented 4d normals for activity recognition from depth sequences. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 716–723 (2013)

  34. Vieira, A.; Nascimento, E.; Oliveira, G.; Liu, Z.; Campos, M.: Stop: space-time occupancy patterns for 3d action recognition from depth map sequences. In: Proceedings of Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 252–259 2012

  35. Wang., J.; Liu, Z.; Chorowski, J.; Chen, Z.; Wu, Y.: Robust 3d action recognition with random occupancy patterns. In: Proceedings of European Conference on Computer Vision, pp. 872–885 (2012)

  36. Yang, X.; Zhang, C.; Tian, Y: Recognizing actions using depth motion maps based histograms of oriented gradients. In: Proceedings of ACM International Conference on Multimedia, pp. 1057–1060 (2012)

  37. Lei, J.; Ren, X.; Fox, D.: Fine-grained kitchen activity recognition using rgb-d. In: Proceedings of ACM Conference on Ubiquitous Computing, pp. 208–211 (2012)

  38. Jalal A., Uddin M.Z., Kim J.T., Kim T.S.: Recognition of human home activities via depth silhouettes and transformation for smart homes. Indoor Built Environ. 21(1), 184–190 (2011)

    Article  Google Scholar 

  39. Wang, Y.; Huang, K.; Tan, T.: Human activity recognition based on r transform. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  40. Koppula H.S., Gupta R., Saxena A.: Human activity learning using object affordances from rgb-d videos. Int. J. Robot. Res. 32(8), 951–970 (2013)

    Article  Google Scholar 

  41. Yang, X.; Tian, Y.: Eigenjoints-based action recognition using Naive-Bayes-nearest-neighbor. In: Proceedings of Workshop on Human Activity Understanding from 3D Data, pp. 14–19 (2012)

  42. Sung, J.; Ponce, C.; Selman, B.; Saxena, A.: Unstructured human activity detection from RGBD images. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 842–849 (2012)

  43. McCallum, A.; Freitag, D.; Pereira, F.C.N.: Maximum entropy markov models for information extraction and segmentation. In: Proceedings of International Conference on Machine Learning, pp. 591–598 (2000)

  44. Hamer, H.; Schindler, K.; Koller-Meier, E.; Van Gool, L.: Tracking a hand manipulating an object. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1475–1482 (2009)

  45. Oikonomidis, I.; Kyriazis, N.; Argyros, A.A.: Tracking the articulated motion of two strongly interacting hands. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1862–1869 (2012)

  46. Hamer, H.; Gall, J.; Weise, T.; Van Gool, L.: An object-dependent hand pose prior from sparse training data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 671–678 (2010)

  47. Luong, D.D.; Lee, S.; Kim, T.-S.: Human computer interface using the recognized finger parts of hand depth silhouette via random forests. In: Proceedings of 13th International Conference on Control, Automation and Systems, pp. 905–909 (2013)

  48. Iddan, G.J.; Yahav, G.: 3D imaging in the studio (and elsewhere…). In: Proceedings of SPIE, vol. 4298, pp. 48–55 (2001)

  49. Ong S., Ranganath S.: Automatic sign language analysis: a survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27(6), 873–891 (2005)

    Article  Google Scholar 

  50. Pei, T.; Starner, T.; Hamilton, H.; Essa, I.; Rehg, J.: Learning the basic units in american sign language using discriminative segmental feature selection. In: Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4757–4760 (2009)

  51. Yang H.D., Sclaroff S., Lee S.W.: Sign language spotting with a threshold model based on conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 31(7), 1264–1277 (2009)

    Article  Google Scholar 

  52. Dreuw, P.; Ney, H.; Martinez, G.; Crasborn, O.; Piater, J.; Moya, J.M.; Wheatley, M.: The signspeak project—bridging the gap between signers and speakers. In: Proceedings of International Conference on Language Resources and Evaluation, pp. 476–481 (2010)

  53. Liu, X.; Fujimura, K.: Hand gesture recognition using depth data. In: Proceedings of International Conference on Automatic Face and Gesture Recognition, pp. 529–534 (2004)

  54. Mo, Z.; Neumann, U.: Real-time hand pose recognition using low-resolution depth images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1499–1505 (2006)

  55. Breuer, P.; Eckes, C.; Muller, S.: Hand gesture recognition with a novel IR time-of-flight range camera: a pilot study. In: Proceedings of the 3rd International Conference on Computer Vision/Computer Graphics Collaboration Techniques, pp. 247–260 (2007)

  56. Soutschek, S.; Penne, J.; Hornegger, J.; Kornhuber J.: 3-d Gesture-based scene navigation in medical imaging applications using time-of-flight cameras. In: Proceedings of Workshop on Time of Flight Camera based Computer Vision, pp. 1–6 (2008)

  57. Kollorz E., Penne J., Hornegger J., Barke A.: Gesture recognition with a time-of-flight camera. Int. J. Intell. Syst. Technol. Appl. 5, 334–343 (2008)

    Google Scholar 

  58. Penne, J.; Soutschek, S.; Fedorowicz, L.; Hornegger, J.: Robust real-time 3d time-of-flight based gesture navigation. In: Proceedings of International Conference on Automatic Face and Gesture Recognition, pp. 1–2 (2008)

  59. Li, Z.; Jarvis, R.: Real time hand gesture recognition using a range camera. In: Proceedings of Australasian Conference on Robotics and Automation (2009)

  60. Takimoto, H.; Yoshimori, S.; Mitsukura, Y.; Fukumi M.: Classification of hand postures based on 3d vision model for human–robot interaction. In: Proceedings of International Symposium on Robot and Human Interactive Communication, pp. 292–297 (2010)

  61. Van den Bergh, M.; Van Gool, L.: Combining RGB and ToF cameras for real-time 3d hand gesture interaction. In: Proceedings of IEEE Workshop on Applications of Computer Vision, pp. 66–72 (2011)

  62. Marnik J.: The polish finger alphabet hand postures recognition using elastic graph matching. Comput. Recognit. Syst. 2 45, 454–461 (2007)

    Article  Google Scholar 

  63. Breitenstein, M.D.; Kuettel, D.; Weise, T.; Van Gool, L.; Pfister, H.: Real-time face pose estimation from single range images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  64. Cai, Q.; Gallup, D.; Zhang, C.; Zhang, Z.: 3d deformable face tracking with a commodity depth camera. In: Proceeding of European Conference on Computer Vision, pp. 229–242 (2010)

  65. Morency, L.P.; Sundberg, P.; Darrell, T.: Pose estimation using 3d view-based eigenspaces. In: Proceedings of IEEE International Workshop on Analysis and Modeling of Faces and Gestures, pp. 45–52 (2003)

  66. Seemann, E.; Nickel, K.; Stiefelhagen, R.: Head pose estimation using stereo vision for human–robot interaction. In: Proceedings of Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 626–631 (2004)

  67. Mian, A.; Bennamoun, M.; Owens, R.: Automatic 3d face detection, normalization and recognition. In: Proceedings of Third International Symposium on 3D Data Processing, Visualization, and Transmission, pp. 735–742 (2006)

  68. Lu, X.; Jain, A.K.: Automatic feature extraction for multiview 3d face recognition. In: Proceedings of 7th International Conference on Automatic Face and Gesture Recognition, pp. 585–590 (2006)

  69. Weise, T.; Leibe, B.; Van Gool, L.: Fast 3d scanning with automatic motion compensation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  70. Weise, T.; Bouaziz, S.; Li, H.; Pauly, M.: Realtime performance-based facial animation. ACM Trans. Graph. 30(4), article 77, 1–10 (2011)

  71. Breitenstein, M.D.; Jensen, J.; Hoilund, C.; Moeslund, T.B.; Van Gool, L.: Head pose estimation from passive stereo images. In Proceedings of 16th Scandinavian Conference on Image Analysis, pp. 219–228 (2009)

  72. Fanelli, G.; Gall, J.; Van Gool, L.: Real time head pose estimation with random regression forests. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 617–624 (2011)

  73. Sun, Y.; Yin, L.: Automatic pose estimation of 3d facial models. In: Proceedings of International Conference on Pattern Recognition, pp. 1–4, (2008)

  74. Segundo M., Silva L., Bellon O., Queirolo C.: Automatic face segmentation and facial landmark detection in range images. IEEE Trans. Syst. Man Cybern. Part B Cybern. 40(5), 1319–1330 (2010)

    Article  Google Scholar 

  75. Chang K.I., Bowyer K.W., Flynn P.J.: Multiple nose region matching for 3d face recognition under varying facial expression. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1695–1700 (2006)

    Article  Google Scholar 

  76. Nair P., Cavallaro A.: 3-d Face detection, landmark localization, and registration using a point distribution model. IEEE Trans. Multimed. 11(4), 611–623 (2009)

    Article  Google Scholar 

  77. Rabiner L.R.: A tutorial on hidden Markov modes and selected application in speech recognition. Proc. IEEE 77, 257–286 (1989)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Education, Sungkyunkwan University, Seoul, Republic of Korea

    Md. Zia Uddin

Authors
  1. Md. Zia Uddin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md. Zia Uddin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Uddin, M.Z. An Efficient Local Feature-Based Facial Expression Recognition System. Arab J Sci Eng 39, 7885–7893 (2014). https://doi.org/10.1007/s13369-014-1396-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-014-1396-9

Keywords

Search

Navigation