Bio-Inspired Spiking Neural Networks for Facial Expression Recognition: Generalisation Investigation

  • Esma Mansouri-BenssassiEmail author
  • Juan Ye
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11324)


Facial expression recognition is a popular research topic to a wide range of applications in human-computer interaction, social robotics, and affective computing. Various attempts have been made to improve the techniques and accuracies of FER. However, one of the main challenges still persists – how to generalise across different datasets, deal with small data and reduce classifiers bias. In this paper, we explore the application of bio-inspired Spiking Neural Networks (SNN) with unsupervised learning using spike timing dependent plasticity (STDP) for FER. We have evaluated our approach on two publicly available, third-party, facial expression datasets. The results have shown that our approach has achieved consistently high accuracies (92%) in cross-dataset evaluation and exhibited a significant improvement compared with the state-of-the-art CNN and HOG feature extraction techniques. The results suggest that our approach can learn more effective feature representations, which lead to good generalisation across subjects in different ethnic groups with different facial dimensions and characteristics.


Neural networks FER Unsupervised learning 



We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Quadro M5000 GPU used for this research.


  1. 1.
    Carcagnì, P., Coco, M.D., Leo, M., Distante, C.: Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1), 645 (2015)CrossRefGoogle Scholar
  2. 2.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection, vol. 5, pp. 886–893 (2005)Google Scholar
  3. 3.
    Diehl, P., Cook, M.: Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci. 9, 99 (2015)CrossRefGoogle Scholar
  4. 4.
    Filip, P., Andrzej, K.: Introduction to spiking neural networks: information processing. Learn. Appl. 71, 409–433 (2011)Google Scholar
  5. 5.
    Gilani, S.Z., Mian, A., Shafait, F., Reid, I.: Dense 3D face correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 40(7), 1584–1598 (2017)CrossRefGoogle Scholar
  6. 6.
    Goodman, D., Brette, R.: Brain: a simulator for spiking neural networks in python. Front. Neuroinformatics 2, 5 (2008)Google Scholar
  7. 7.
    Hodgkin, A.L., Huxley, A.F.: A quantitative description of membrane current and its application to conduction and excitation in nerve. Bull. Math. Biol. 52, 25–71 (1990)CrossRefGoogle Scholar
  8. 8.
    Jose, J.T., Amudha, J., Sanjay, G.: A survey on spiking neural networks in image processing, pp. 107–115 (2015)Google Scholar
  9. 9.
    Kheradpisheh, S., Ganjtabesh, M., Thorpe, S., Masquelier, T.: Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks pp. 56–67 (2017)CrossRefGoogle Scholar
  10. 10.
    Khorrami, P., Paine, T.L., Huang, T.S.: Do deep neural networks learn facial action units when doing expression recognition? CoRR 2015 (2015)Google Scholar
  11. 11.
    Kim, B.K., Dong, S.Y., Roh, J., Kim, G., Lee, S.Y.: Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach, pp. 1499–1508, June 2016Google Scholar
  12. 12.
    Liu, Y., et al.: Facial expression recognition with PCA and LBP features extracting from active facial patches, pp. 368–373, June 2016Google Scholar
  13. 13.
    Lopes, A.T., de Aguiar, E., Souza, A.D., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)CrossRefGoogle Scholar
  14. 14.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression, pp. 94–101, June 2010Google Scholar
  15. 15.
    Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21, 162–8828 (1999)CrossRefGoogle Scholar
  16. 16.
    Maass, W.: Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10, 1659–1671 (1997)CrossRefGoogle Scholar
  17. 17.
    Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans. Cybern. 99, 1–12 (2016)Google Scholar
  18. 18.
    Mansouri-Benssassi, E.: A decentralised multimodal integration of social signals: a bio-inspired approach. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI, pp. 633–637 (2017)Google Scholar
  19. 19.
    Marr, D., Hildreth, E.: Theory of edge detection. Proc. Royal Soc. London Ser. B 23, 187–217 (1980)CrossRefGoogle Scholar
  20. 20.
    Mishra, B., et al.: Facial expression recognition using feature based techniques and model based techniques: a survey, pp. 589–594 (2015)Google Scholar
  21. 21.
    Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks, pp. 1–10, March 2016Google Scholar
  22. 22.
    Tie, Y., Guan, L.: A deformable 3-D facial expression model for dynamic human emotional state recognition. IEEE Trans. Circ. Syst. Video Technol. 23, 142–157 (2013)CrossRefGoogle Scholar
  23. 23.
    van der Walt, S., et al.: scikit-image: image processing in python. PeerJ 2, e453 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Computer Science, University of St AndrewsSt AndrewsUK

Personalised recommendations