Light-Weight Deep Convolutional Network-Based Approach for Recognizing Emotion on FPGA Platform

  • Thuong Le-TienEmail author
  • Hanh Phan-Xuan
  • Sy Nguyen-Tan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11814)


Emotion being a subjective thing, leveraging knowledge and science behind labeled data and extracting the components that constitute it. With the development of deep learning in computer vision, emotion recognition has become a widely-tackled research problem. For mobility and privacy reasons, the required image processing should be local on embedded computer platforms with performance requirements and energy constraints. For this purpose, in this work, we propose a Field Programmable Gate Array (FPGA) architecture applied for this task using independent method called convolutional neural network (CNN [1]). The design flow is evaluated by implementing the previously trained CNN to recognize facial emotions from face image implemented in python on a PC. The project explains the process of porting the CNN algorithm from python to C/C++ and then executing it on a ZYNQ FPGA board. Once we have trained a network, weights from the Tensorflow model will be convert as C-arrays. After having the weights as C arrays, they can be implemented to FPGA system. This method was trained on the posed-emotion dataset (FER2013). The results show that with more fine-tuning and depth, the CNN model can outperform the state-of-the-art methods for emotion recognition. The bottleneck of the CNN [2] is the convolutional layers and that is why different solutions for that accelerator are analyzed and the performance of each solution is tested.


Convolutional neural network Tensorflow model Vivado FPGA FER2013 


  1. 1.
  2. 2.
  3. 3.
    Ververidis, D., Kotropoulos, C.: Automatic speech classification to five emotional states based on gender information. In: Proceedings of the EUSIPCO2004 Conference, Austria, pp. 341–344 (2004)Google Scholar
  4. 4.
    Cowie, R., et al.: Emotion recognition in human computer interaction. IEEE Signal Process. Mag. 18(1), 32–80 (2001)CrossRefGoogle Scholar
  5. 5.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep Learning. Nature 521 (2015). Scholar
  6. 6.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, PMLR9, pp. 249–256 (2010)Google Scholar
  7. 7.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 211–252 (2012) MathSciNetCrossRefGoogle Scholar
  8. 8.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, pp. 1440–1448 (2015)Google Scholar
  9. 9.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmenta. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015. Boston, MA, USA, pp. 3431–3440 (2015)Google Scholar
  10. 10.
    Zhang, Y., et al.: Towards end-to-end speech recognition with deep convolutional neural networks. In: Proceedings of Interspeech Conference, San Francisco, USA (2016)Google Scholar
  11. 11.
    Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). Scholar
  12. 12.
    Martinez, B., Valstar, M.F.: Advances, challenges, and opportunities in automatic facial expression recognition. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 63–100. Springer, Cham (2016). Scholar
  13. 13.
  14. 14.
  15. 15.
    Phan-Van, H.: Neural Network Lecture, Danang University of Science and Technology (2013)Google Scholar
  16. 16.
  17. 17.
    Ma, Y., et al.: Optimizing loop operation and dataflow in FPGA acceleration of deep convolutional neural networks. Arizona University (2017)Google Scholar
  18. 18.
    Horseman, A.: SVM for Facial Expression Recognition. A demonstrate project using SVM (2007).
  19. 19.
    Shima, A., Azar, F.: Convolutional neural networks for facial expression recognition. In: Proceedings of the Stanford University report (2016).
  20. 20.
  21. 21.
    Szegedy, C., et al.: Going deeper with convolutionsGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Electrical and Electronics EngineeringHo Chi Minh City University of TechnologyHo Chi Minh CityVietnam
  2. 2.Ho Chi Minh City University of TechnologyHo Chi Minh CityVietnam

Personalised recommendations