Skip to main content
SpringerLink
Log in

Facial Expression Recognition Method Based on a Part-Based Temporal Convolutional Network with a Graph-Structured Representation

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2020 (ICANN 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12396))

Included in the following conference series:

Abstract

Facial expressions are controlled by facial muscles and can be regarded as appearance and shape variations in key parts. A key challenge in facial expression recognition is capturing effective information from a facial image. In this paper, we propose a basic graph contour that is based on key parts for facial expression recognition. Each node on the graph contour represents a landmark, and each edge represents the connection between the two selected nodes. To further investigate the graph representation and to make the graphs more distinctive, we use a Gabor filter to extract appearance variations around the graph nodes while applying an affine transformation to capture the shape variations from graphs without expression in graphs with expression. Then, to serve as an efficient network for processing in which the graph extracts the appearance and shape representations, we introduce the temporal convolutional network (TCN). Finally, we propose a part-based temporal convolutional network (PTCN) that emphasizes the key facial parts. The experimental results demonstrate that this method realizes significant improvements over state-of-the-art methods utilizing three widely used facial databases: Oulu-CASIA, CK+, and MMI.

L. Zhong and C. Bai have contributed equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Yao, A., Cai, D., Hu, P., Wang, S., Sha, L., Chen, Y.: HoloNet: towards robust emotion recognition in the wild. In: ACM International Conference on Multimodal Interaction, pp. 472–478. ACM (2016)

    Google Scholar 

  2. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 143–157. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_10

    Chapter  Google Scholar 

  3. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)

    Article  MathSciNet  Google Scholar 

  4. Angelopoulo, E., Molana, R., Daniilidis, K.: Multispectral skin color modeling. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 2, pp. II–II. IEEE (2001)

    Google Scholar 

  5. Zhong, L., Bai, C., Li, J., Chen, T., Li, S., Liu, Y.: A Graph-Structured Representation with BRNN for Static-based Facial Expression Recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE, May 2019

    Google Scholar 

  6. Wang, Y., Yu, H., Stevens, B., Liu, H.: Dynamic facial expression recognition using local patch and LBP-TOP. In: International Conference on Human System Interactions. IEEE (2015)

    Google Scholar 

  7. Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing 6(1), 1–12 (2015)

    Article  Google Scholar 

  8. http://www.cs.cmu.edu/afs/cs/project/face/www/facs.ht

  9. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3444–3451. IEEE, June 2013

    Google Scholar 

  10. Daugman, J.G.: Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. JOSA A 2(7), 1160–1169 (1985)

    Article  Google Scholar 

  11. Cowen, A., Abdel-Ghaffar, S., Bishop, S.: Using structural and semantic voxel-wise encoding models to investigate face representation in human cortex. J. Vis. 15(12), 422 (2015)

    Article  Google Scholar 

  12. Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling (2018)

    Google Scholar 

  13. Zhang, W., Zhang, Y., Ma, L., Guan, J., Gong, S.: Multimodal learning for facial expression recognition. Pattern Recogn. 48(10), 3191–3202 (2015)

    Article  Google Scholar 

  14. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE, June 2010

    Google Scholar 

  15. Taini, M., Zhao, G., Li, S. Z., Pietikainen, M.: Facial expression recognition from near-infrared video sequences. In: 19th International Conference on Pattern Recognition, 2008. ICPR 2008, pp. 1–4. IEEE, December 2008

    Google Scholar 

  16. Valstar, M., Pantic, M.: Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proceedings 3rd International Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, p. 65, May 2010

    Google Scholar 

  17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1414.6980 (2014)

  18. Guo, Y., Zhao, G., Pietikäinen, M.: Dynamic facial expression recognition using longitudinal facial expression atlases. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 631–644. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_45

    Chapter  Google Scholar 

  19. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  20. Yu, Z., Liu, Q., Liu, G.: Deeper cascaded peak-piloted network for weak expression recognition. The Visual Computer 34(12), 1691–1699 (2017). https://doi.org/10.1007/s00371-017-1443-0

    Article  Google Scholar 

  21. Zhao, X., et al.: Peak-piloted deep network for facial expression recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 425–442. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_27

    Chapter  Google Scholar 

  22. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  23. Liu, M., Shan, S., Wang, R., Chen, X.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1749–1756. IEEE, June 2014

    Google Scholar 

  24. Cai, J., Meng, Z., Khan, A. S., Li, Z., O’Reilly, J., Tong, Y.: Island loss for learning discriminative features in facial expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 302–309. IEEE, May 2018

    Google Scholar 

  25. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M.: Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273, 643–649 (2018)

    Article  Google Scholar 

  26. Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 558–565. IEEE, May 2017

    Google Scholar 

  27. Liu, X., Kumar, B.V.K.V., You, J., Jia, P.: Adaptive deep metric learning for identity-aware facial expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 522–531. IEEE Computer Society (2017)

    Google Scholar 

  28. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D.N.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2562–2569. IEEE, June 2012

    Google Scholar 

  29. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3D convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2278–2288. IEEE, July 2017

    Google Scholar 

  30. Kim, D.H., Baddar, W., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. 10, 223–236 (2017)

    Article  Google Scholar 

  31. Sun, N., Li, Q., Huan, R., Liu, J., Han, G.: Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn. Lett. 119, 49–61 (2017)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities (XDJK2020C016).

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Southwest University, Chongqing, China

    Lei Zhong, Changmin Bai, Jianfeng Li, Tong Chen & Shigang Li

  2. Chongqing Key Laboratory of Nonlinear Circuit and Intelligent Information Processing, Southwest University, Chongqing, China

    Lei Zhong, Changmin Bai, Jianfeng Li & Tong Chen

  3. Hiroshima City University, Hiroshima, Japan

    Shigang Li

Authors
  1. Lei Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Changmin Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianfeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shigang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianfeng Li .

Editor information

Editors and Affiliations

  1. Department of Applied Informatics, Comenius University in Bratislava, Bratislava, Slovakia

    Igor Farkaš

  2. Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kgs. Lyngby, Denmark

    Paolo Masulli

  3. Department of Informatics, University of Hamburg, Hamburg, Germany

    Stefan Wermter

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhong, L., Bai, C., Li, J., Chen, T., Li, S. (2020). Facial Expression Recognition Method Based on a Part-Based Temporal Convolutional Network with a Graph-Structured Representation. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12396. Springer, Cham. https://doi.org/10.1007/978-3-030-61609-0_48

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61609-0_48

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61608-3

  • Online ISBN: 978-3-030-61609-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation