Using Facial Expression Recognition for Crowd Monitoring

  • Ross Philip Holder
  • Jules-Raymond Tapamo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10749)


In recent years, Crowd Monitoring techniques have attracted emerging interest in the field of computer vision due to their ability to monitor groups of people in crowded areas, where conventional image processing methods would not suffice. Existing Crowd Monitoring techniques focus heavily on analyzing a crowd as a single entity, usually in terms of their density and movement pattern. While these techniques are well suited for the task of identifying dangerous and emergency situations, they are very limited when it comes to identifying emotion within a crowd. In this work, we propose a novel Crowd Monitoring algorithm based on estimating crowd emotion using Facial Expression Recognition (FER). By isolating different types of emotion within a crowd, we aim to predict the mood of a crowd even in scenes of non-panic. To validate the effectiveness of the proposed algorithm, a series of cross-validation tests are performed using a novel Crowd Emotion dataset with known ground-truth emotions. The results show that the algorithm presented is able to accurately and efficiently predict multiple classes of crowd emotion even in non-panic situations where movement and density information may be incomplete.


  1. 1.
    Ahmed, F., Hossain, E.: Automated facial expression recognition using gradient-based ternary texture patterns. Chin. J. Eng. 2013, 1–8 (2013)CrossRefGoogle Scholar
  2. 2.
    Ali, S., Shah, M.: A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–6 (2007)Google Scholar
  3. 3.
    Andrade, E.L., Blunsden, S., Fisher, R.B.: Modelling crowd scenes for event detection. In: 18th International Conference on Pattern Recognition, pp. 175–178 (2006)Google Scholar
  4. 4.
    Baig, M.W., Barakova, E.I., Marcenaro, L., Rauterberg, M., Regazzoni, C.S.: Crowd emotion detection using dynamic probabilistic models. In: del Pobil, A.P., Chinellato, E., Martinez-Martin, E., Hallam, J., Cervera, E., Morales, A. (eds.) SAB 2014. LNCS (LNAI), vol. 8575, pp. 328–337. Springer, Cham (2014). Google Scholar
  5. 5.
    Barry, E.: Protests in Moldova Explode, with a Call to Arms on Twitter. The New York Times, p. A1, 7 April 2009Google Scholar
  6. 6.
    Boghossian, B.A., Velastin, S.A.: Motion-based machine vision techniques for the management of large crowds. In: The 6th IEEE International Conference on Electronics, Circuits and Systems, vol. 2, pp. 961–964 (1999)Google Scholar
  7. 7.
    Brostow, G.J., Cipolla, R.: Unsupervised Bayesian detection of independent motion in crowds. In: Computer Society Conference on Computer Vision and Pattern Recognition, pp. 594–601 (2006)Google Scholar
  8. 8.
    Cheriyadat, A.M., Radke, R.: Detecting dominant motions in dense crowds. IEEE J. Sel. Topics Sig. Process. 2(4), 568–581 (2008)CrossRefGoogle Scholar
  9. 9.
    Cikara, M., Jenkins, A.C., Dufour, N., Saxe, R.: Reduced self-referential neural response during intergroup competition predicts competitor harm. NeuroImage 96, 36–43 (2014)CrossRefGoogle Scholar
  10. 10.
    Davies, A.C., Yin, J.H., Velastin, S.A.: Crowd monitoring using image processing. Electron. Commun. Eng. J. 7(1), 37–47 (1995)CrossRefGoogle Scholar
  11. 11.
    Dhall, A.: Context based facial expression analysis in the wild. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), September 2013Google Scholar
  12. 12.
    Ekman, P.: Strong evidence for universals in facial expressions: a reply to russell’s mistaken critique. Psychol. Bull. 115(2), 268–287 (1994)CrossRefGoogle Scholar
  13. 13.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Hassner, T., Itcher, Y., Kliper-Gross, O.: Violent flows: real-time detection of violent crowd behavior. In: 3rd IEEE International Workshop on Socially Intelligent Surveillance and Monitoring (SISM) at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Rhode Island, June 2012Google Scholar
  15. 15.
    Holder, R.P., Tapamo, J.R.: Improved gradient local ternary patterns for facial expression recognition. EURASIP J. Image Video Process. 2017(1), 42 (2017). CrossRefGoogle Scholar
  16. 16.
    Hsu, C.W., Lin, C.J.: A comparison on methods for multiclass support vector machines. IEEE Trans. Neural Netw. 13(2), 415–425 (2002)CrossRefGoogle Scholar
  17. 17.
    Khryashchev, V., Ganin, A., Golubev, M., Shmaglit, L.: Audience analysis system on the basis of face detection, tracking and classification techniques. In: Proceedings of the International MultiConference of Engineers and Computer Scientists, vol. 1, Hong Kong, March 2013Google Scholar
  18. 18.
    Kumari, J., Rajesh, R., Pooja, K.M.: Facial expression recognition: a survey. In: Second International Symposium on Computer Vision and the Internet, vol. 58, pp. 486–491 (2015)Google Scholar
  19. 19.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior Analysis, San Francisco, CA, USA, pp. 94–101. IEEE (2010)Google Scholar
  20. 20.
    Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: IEEE Computer Vision and Pattern Recognition (2010)Google Scholar
  21. 21.
    Mahto, S., Yadav, Y.: A survey on various facial expression recognition techniques. Int. J. Adv. Res. Electr. Electron. Instrum. Eng. 3(11), 13028–13031 (2014)Google Scholar
  22. 22.
    Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: IEEE Computer Vision and Pattern Recognition (2009)Google Scholar
  23. 23.
    Mishra, S., Dhole, A.: A survey on facial expression recognition techniques. Int. J. Sci. Res. (IJSR) 4(4), 1247–1250 (2015)Google Scholar
  24. 24.
    Rabaud, V., Belongie, S.: Counting crowded moving objects. In: Computer Society Conference on Computer Vision and Pattern Recognition, pp. 705–711 (2006)Google Scholar
  25. 25.
    Rabiee, H.R., Haddadnia, J., Mousavi, H., Nabi, M., Murino, V., Sebe, N.: Emotion-based crowd representation for abnormality detection, pp. 1–7. CoRR abs/1607.07646 (2016)Google Scholar
  26. 26.
    Solmaz, B., Moore, B.E., Shah, M.: Identifying behaviors in crowd scenes using stability analysis for dynamical systems. IEEE Trans. Pattern Anal. Mach. Intell. 34(10), 2064–2070 (2012)CrossRefGoogle Scholar
  27. 27.
    Tarjan, R.E.: Minimum spanning trees. In: Data Structures and Network Algorithms, CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 44, chap. 6, pp. 72–77. Bell Laboratories, Murray Hill (1983)Google Scholar
  28. 28.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kauai, HI, USA, pp. 511–518. IEEE (2001)Google Scholar
  29. 29.
    Wang, X., Ma, X., Grimson, W.E.L.: Unsupervised activity perception in crowded and complicated scenes using hierarchical Bayesian models. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 539–555 (2009)CrossRefGoogle Scholar
  30. 30.
    Wu, X., Liang, G., Lee, K.K., Xu, Y.: Crowd density estimation using texture analysis and learning, pp. 214–219 (2006)Google Scholar
  31. 31.
    Zhan, B., Monekosso, D.N., Remagnino, P., Velastin, S.A., Xu, L.Q.: Crowd analysis: a survey. Mach. Vis. Appl. 19(5), 345–357 (2008)CrossRefGoogle Scholar
  32. 32.
    Zhang, D., Tong, C., Lu, Y., Liu, Z.: Dominant motions detection in dense crowds based on particle video. Int. J. Digit. Content Technol. Appl. (JDCTA) 6(10), 294–301 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of EngineeringUniversity of KwaZulu-NatalDurbanSouth Africa

Personalised recommendations