Skip to main content
SpringerLink
Log in

Quantized Separable Residual Network for Facial Expression Recognition on FPGA

  • Conference paper
  • First Online:
Cognitive Systems and Signal Processing (ICCSIP 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1397))

Included in the following conference series:

Abstract

Facial expression recognition plays an important role in human machine interaction, and thus becomes an important task in cognitive science and artificial intelligence. In vision fields, facial expression recognition aims to identify facial expressions through images or videos, but there is rare work towards real-world applications. In this work, we propose a hardware-friendly quantized separable residual network and developed a real-world facial expression recognition system on a field programming gate array. The proposed network is first trained on devices with graphical processing units, and then quantized to speed up inference. Finally, the quantized algorithm is deployed on a high-performance edge device - Ultra96-V2 field programming gate array board. The complete system involves capturing images, detecting faces, and recognizing expressions. We conduct exhaustive experiments for comparing the performance with various deep learning models and show superior results. The overall system has also demonstrated satisfactory performance on FPGA, and could be considered as an important milestone for facial expression recognition applications in the real world.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020)

    Google Scholar 

  2. Gennari, R., et al.: Children’s emotions and quality of products in participatory game design. Int. J. Hum. Comput. Stud. 101, 45–61 (2017)

    Article  Google Scholar 

  3. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 94–101. IEEE (2010)

    Google Scholar 

  4. Filntisis, P.P., Efthymiou, N., Koutras, P., Potamianos, G., Maragos, P.: Fusing body posture with facial expressions for joint recognition of affect in child-robot interaction. IEEE Robot. Autom. Lett. 4(4), 4011–4018 (2019)

    Article  Google Scholar 

  5. Wang, Y., Ai, H., Wu, B., Huang, C.: Real time facial expression recognition with Adaboost. In: International Conference on Pattern Recognition, vol. 3, pp. 926–929. IEEE (2004)

    Google Scholar 

  6. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  7. LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)

    Article  Google Scholar 

  8. Han, S., Meng, Z., Khan, A.S., Tong, Y.: Incremental boosting convolutional neural network for facial action unit recognition. arXiv preprint arXiv:1707.05395 (2017)

  9. Fernandez, P.D.M., Pena, F.A.G., Ren, T., et al.: FERAtt: facial expression recognition with attention net. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 837–846 (2019)

    Google Scholar 

  10. Balahur, A., Hermida, J.M., Montoyo, A., Muñoz, R.: EmotiNet: a knowledge base for emotion detection in text built on the appraisal theories. In: Muñoz, R., Montoyo, A., Métais, E. (eds.) NLDB 2011. LNCS, vol. 6716, pp. 27–39. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22327-3_4

    Chapter  Google Scholar 

  11. Phan-Xuan, H., Le-Tien, T., Nguyen-Tan, S.: FPGA platform applied for facial expression recognition system using convolutional neural networks. Procedia Comput. Sci. 151, 651–658 (2019)

    Article  Google Scholar 

  12. Vinh, P.T., Vinh, T.Q.: Facial expression recognition system on SoC FPGA. In: International Symposium on Electrical and Electronics Engineering, pp. 1–4. IEEE (2019)

    Google Scholar 

  13. Viola, P., Jones, M.: Robust real-time object detection. Int. J. Comput. Vis. 4(34–47), 4 (2001)

    Google Scholar 

  14. Wang, W., Chang, F., Zhao, J., Chen, Z.: Automatic facial expression recognition using local binary pattern. In: World Congress on Intelligent Control and Automation, pp. 6375–6378. IEEE (2010)

    Google Scholar 

  15. Yang, J., Zhang, D., Frangi, A.F., et al.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004)

    Article  Google Scholar 

  16. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3359–3368 (2018)

    Google Scholar 

  17. Han, Y., Chen, B., Zhang, X.: Sex difference of saccade patterns in emotional facial expression recognition. In: Sun, F., Liu, H., Hu, D. (eds.) ICCSIP 2016. CCIS, vol. 710, pp. 144–154. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-5230-9_16

    Chapter  Google Scholar 

  18. Bailey, D.G.: Design for Embedded Image Processing on FPGAs. Wiley, New York (2011)

    Google Scholar 

  19. Kumar, A., Hansson, A., Huisken, J., et al.: An FPGA design flow for reconfigurable network-based multi-processor systems on chip. In: Design, Automation and Test in Europe Conference and Exhibition, pp. 1–6. IEEE (2007)

    Google Scholar 

  20. Kathail, V.: Xilinx vitis unified software platform. In: ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 173–174 (2020)

    Google Scholar 

  21. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  22. Howard, A.G., Zhu, M., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  23. Zhu, Y., Bai, L., Peng, W., Zhang, X., Luo, X.: Depthwise separable convolution feature learning for ihomogeneous rock image classification. In: Sun, F., Liu, H., Hu, D. (eds.) ICCSIP 2018. CCIS, vol. 1005, pp. 165–176. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-7983-3_15

    Chapter  Google Scholar 

  24. Qiu, J., Wang, J., Yao, S., et al.: Going deeper with embedded FPGA platform for convolutional neural network. In: ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 26–35 (2016)

    Google Scholar 

  25. Yang, Y., et al.: Synetgy: algorithm-hardware co-design for convnet accelerators on embedded FPGA. In: Proceedings of ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 23–32 (2019)

    Google Scholar 

  26. Wu, T., Liu, W., Jin, Y.: An end-to-end solution to autonomous driving based on xilinx FPGA. In: International Conference on Field-Programmable Technology, pp. 427–430. IEEE (2019)

    Google Scholar 

  27. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H.J., Hawk, S.T., Van Knippenberg, A.D.: Presentation and validation of the Radboud faces database. Cogn. Emot. 24(8), 1377–1388 (2010)

    Article  Google Scholar 

  28. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

Download references

Acknowledgement

This work is supported by the Hong Kong Innovation and Technology Commission and City University of Hong Kong (Project 7005230).

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China

    Xinqi Fan, Mingjie Jiang & Hong Yan

  2. College of Petrochemical and Environmental Engineering, Zhejiang Ocean University, Zhejiang, China

    Huaizhi Zhang

  3. College of Information Science and Technology, Jinan University, Guangdong, China

    Yang Li

Authors
  1. Xinqi Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingjie Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huaizhi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hong Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingjie Jiang .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Fuchun Sun

  2. Tsinghua University, Beijing, China

    Huaping Liu

  3. Tsinghua University, Beijing, China

    Bin Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fan, X., Jiang, M., Zhang, H., Li, Y., Yan, H. (2021). Quantized Separable Residual Network for Facial Expression Recognition on FPGA. In: Sun, F., Liu, H., Fang, B. (eds) Cognitive Systems and Signal Processing. ICCSIP 2020. Communications in Computer and Information Science, vol 1397. Springer, Singapore. https://doi.org/10.1007/978-981-16-2336-3_1

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-2336-3_1

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-2335-6

  • Online ISBN: 978-981-16-2336-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation