Skip to main content
SpringerLink
Log in

A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videos

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Monitoring continuously captured surveillance videos is a challenging and time consuming task. To assist this issue, a new framework is introduced that applies anomaly detection, semantic annotation and provides a concept-based search interface. In particular, novel optical flow based features are used for abnormal crowd behaviour detection. Then, processed surveillance videos are annotated using a new semantic metadata model based on multimedia standards using Semantic Web technologies. In this way, globally inter-operable metadata about abnormal crowd behaviours are generated. Finally, for the first time, based on crowd behaviours, a novel concept-based semantic search interface is proposed. In the proposed interface, along with search results (video segments), statistical data about crowd behaviours are also presented. With extensive user studies, it is demonstrated that the proposed concept-based semantic search interface enables efficient search and analysis of abnormal crowd behaviours. Although there are existing works to achieve (a) crowd anomaly detection, (b) semantic annotation and (c) semantic search interface, none of the existing works combine these three system components in a novel framework like the one proposed in this paper. In each system component, we introduce contributions to the field as well as use the Semantic Web technologies to combine and standardize output of different system components; output of the anomaly detection is automatically annotated with metadata and stored to a semantic database. When continuous surveillance videos are processed, only the semantic database is updated. Finally, the user interface queries the updated database for searching/analyzing surveillance videos without changing any coding. Thus, the framework supports re-usability. This paper explains and evaluates different components of the framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. http://open-biomed.sourceforge.net/opmv/ns.html

  2. http://multimedialab.elis.ugent.be/organon/ontologies/ninsuna

  3. A demo of the proposed concept-based semantic video search interface is available at

    https://www.youtube.com/watch?v=8f4UVgYBmHs

References

  1. A demo of the semantic search interface is available at https://www.youtube.com/watch?v=8f4UVgYBmHs

  2. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors. IEEE Transactions on Pattern Analysis and Machine Intelligence 30:555–560

    Article  Google Scholar 

  3. Aggarwal K, Ryoo MS (2011) Human Activity Analysis: A Review, ACM Computing Surveys, 43(3):16

  4. Ali S, Shah M (2007) A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis. IEEE Conference on Computer Vision and Pattern Recognition, 1–6

  5. Arndt R, Troncy R, Staab S, Hardman L, Vacura M (2007) COMM: Designing a Well-Founded Multimedia Ontology for the Web. In 6th International Semantic Web Conference (ISWC'2007)

  6. Brooke J (1996) SUS – A quick and dirty usability scale. Usability evaluation in industry 189(194):4–7

    Google Scholar 

  7. Brostow G, Cipolla R (2006) Unsupervised Bayesian Detection of Independent Motion in Crowds. IEEE Computer Vision and Pattern Recognition 1:594–601

    Google Scholar 

  8. Chen D, Huang P (2011) “Motion-based unusual event detection in human crowds”, in Journal of Visual Communication and Image Representation, 22(2), pages: 178–186

  9. Colque RM et al (2017) Histograms of optical flow orientation and magnitude to detect anomalous events in videos. IEEE Trans. Circuits Syst. Video Technol. 27(3):673–682

    Article  Google Scholar 

  10. Cong Y, Yuan J, Liu J (2013) Abnormal Event Detection in Crowded Scenes using Sparse Representation. Pattern Recognition 46(7):1851–1864

    Article  Google Scholar 

  11. Dee HM, Caplier A (2010) Crowd behaviour analysis using histograms of motion direction, IEEE International Conference on Image Processing, pages: 1545–1548

  12. Direkoglu C, Sah M, O’Connor NE (2017) Abnormal crowd behaviour detection using novel optical flow-based features. IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS2017)

  13. Ernesto SB, Andrade L, Fisher RB (2006) Modelling Crowd Scenes for Event Detection. IEEE International Conference on Pattern Recognition 1:175–178

    Google Scholar 

  14. Fang Z, Fei F, Fang Y, Lee C, Xiong N, Shu L, Chen S (2016) Abnormal event detection in crowded scenes based on deep learning. Multimedia Tools and Applications 75(22):14617–14639

    Article  Google Scholar 

  15. Feng Y, Yuan Y, Lu X (2016) Deep Representation for Abnormal Event Detection in Crowded Scenes. In Proceedings of ACM on Multimedia Conference (MM '16). Pages:591–595

  16. Fernandez C, Baige P, Roca X, Gonzalez J (2007) Semantic Annotation of Complex Human Scenes for Multimedia Surveillance. Artificial Intelligence and Human-Oriented Computing 4733:698–709

    Google Scholar 

  17. Fernandez J, Calavia L, Baladron C, Aguiar JM, Carro B, Sanchez-esguevillas A, Alonso-lopez JA, Smilansky Z (2013) An Intelligent Surveillance Platform for Large Metropolitan Areas with Dense Sensor Deployment. Sensors 13(6):7414–7442

    Article  Google Scholar 

  18. Garcia R, Celma O (2005) Semantic Integration and Retrieval of Multimedia Metadata . In Proc. of the 5th International Workshop on Knowledge Markup and Semantic Annotation (SemAnnot 2005)

  19. Gnouma M, Ejbali R, Zaied M (2018) Abnormal events’ detection in crowded scenes. Multimedia Tools Appl. 77(19):24843–24864

    Article  Google Scholar 

  20. Greco L, Ritrovato P, Saggese A, Vento M (2016) Abnormal Event Recognition: A Hybrid Approach Using Semantic Web Technologies. CVPR Workshops

  21. Greco L, Ritrovato P, Vento M (2017) Advanced video analytics: an ontology-based approach. International Conference on Web Intelligence, Mining and Semantics

  22. J.S. Hare, P.A.S Sinclair, P.H. Lewis, K. Martinez, Kirk, P.G.B. Enser, and C.J. Sandom (2006) Bridging the Semantic Gap in Multimedia Information Retrieval: Top-down and Bottom-up approaches. At Mastering the Gap: From Information Extraction to Semantic Representation, European Semantic Web Conference

  23. Hawkins S, He H, Williams G, Baxter R (2002) Outlier Detection Using Replicator Neural Networks. International Conference in Data Warehousing and Knowledge Discovery

  24. Hunter J (2001) Adding Multimedia to the Semantic Web - Building an MPEG-7 Ontology. In International Semantic Web Working Symposium (SWWS 2001)

  25. Kazi Tani MY, Lablack A, Ghomari A, Bilasco IM (2015) Events Detection Using a Video-Surveillance Ontology and a Rule-Based Approach. ECCV 2014 Workshops. ECCV 2014. Lecture Notes in Computer Science, vol 8926

  26. Ko T (2011) A Survey on Behaviour Analysis in Video Surveillance Applications, Video Surveillance

  27. Kok VJ, Lim MK, Chan CS. Crowd behaviour analysis: A review where physics meets biology.

  28. Kratz L, Nishino K (2009) Anomaly Detection in Extremely Crowded Scenes using Spatio-Temporal Motion Pattern Models. IEEE Conference on Computer Vision and Pattern Recognition, pages:1446–1453

  29. W. Lee, W. Bailer, T. Bürger, P.A. Champin, J.P. Evain, V. Malaisé, T. Michel, F. Sasaki, J. Söderberg, F. Stegmaier, J. Strassner, Ontology for Media Resources 1.0, W3C Recommendation, February 2012. Available at http://www.w3.org/TR/mediaont-10/

  30. Li T, Chang H, Wang M, Ni B, Hong R, Yan S (2015) Crowded Scene Analysis: A Survey, in IEEE Transactions on Circuits and Systems for Video Technology, 25:(3):367–386

  31. Lucas BD, Kanade T (1981) “An Iterative Image Registration Technique with an Application to Stereo Vision”, in International Joint Conference on Artificial Intelligence, pages 674–679

  32. F. Manola, E. Miller and B. McBride. Resource Description Framework (RDF), W3C Recommendation, February 2004. Available at http://www.w3.org/TR/2004/REC-rdf-primer-20040210/

  33. Marques JS, Jorge PM, Abrantes AJ, Lemos JM (2003) Tracking Groups of Pedestrians in Video Sequences. Computer Vision and Pattern Recognition Workshop 9:101

    Google Scholar 

  34. D. L. McGuinness, F. Van Harmelen. OWL Web Ontology Language, W3C Recommendation, February 2004. Available at http://www.w3.org/TR/owl-features/ (last accessed at 28 December 2016).

  35. Mehran R, Oyama A, Shah M (2009) Abnormal Crowd Behaviour Detection using Social Force Model. IEEE Conference on Computer Vision and Pattern Recognition, pages:935–942

  36. MPEG-7 Multimedia Content Description Standard (ISO/IEC 15938). Available at http://mpeg.chiariglione.org/standards/mpeg-7 (last accessed at 28 December 2016).

  37. Open Annotation Extension Specification, W3C Community Darft, Avaibale at http://www.openannotation.org/spec/extension/

  38. Pan L, Zhou H, Liu Y, Wang M (2019) Global event influence model: integrating crowd motion and social psychology for global anomaly detection in dense crowds. Journal of Electronic Imaging 28(2):023033

    Article  Google Scholar 

  39. Patil N, Biswas PK (2018) Global abnormal events detection in crowded scenes using context location and motion-rich spatio-temporal volumes. IET Image Process. 12(4):596–604

    Article  Google Scholar 

  40. Protégé Ontology Editor. Available at http://protege.stanford.edu/

  41. E. Prud'hommeaux and A. Seaborne. SPARQL Query Language for RDF. W3C Recommendation, January 2008. Available at http://www.w3.org/TR/rdf-sparql-query/

  42. Ranasinghe S, Al Machot F, Mayr HC (2016) A review on applications of activity recognition systems with regard to performance and evaluation. International Journal of Distributed Sensor Networks, Vol. 12(8)

  43. Ravanbakhsh M et al. (2017) Abnormal event detection in videos using generative adversarial nets. 2017 IEEE International Conference on Image Processing (ICIP). IEEE

  44. Ravanbakhsh M, Nabi M, Mousavi H, Sangineto E, Sebe N (2018) Plug-and-Play CNN for Crowd Motion Analysis: An Application in Abnormal Event Detection, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, pp. 1689–1698.

  45. Sah M, Direkoglu C (2017) Semantic Annotation of Surveillance Videos for Abnormal Crowd Behaviour Search and Analysis, IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS2017)

  46. R. Sanderson, P. Ciccarese, B. Young. Web Annotation Vocabulary, W3C Candidate Recommendation, November 2016, Available at http://www.w3.org/TR/annotation-vocab/

  47. SanMiguel JC, Martinez JM, Garcia A (2009) An Ontology for Event Detection and its Application in Surveillance Video. IEEE International Conference on Advanced Video and Signal Based Surveillance, 220–225

  48. Shadbolt N, Berners-Lee T, Hall W (2006) The Semantic Web Revisited. IEEE Intelligent Systems 21(3):96–101

    Article  Google Scholar 

  49. Sjekavica T, Gledec G, Horvat M (2014) Advantages of Semantic Web Technologies Usage in the Multimedia Annotation and Retrieval. International Journal of Computers and Communications 8:41–48

    Google Scholar 

  50. Snidaro L, Belluz M, Foresti GL (2007) Representing and recognizing complex events in surveillance applications. IEEE Conference on Advanced Video and Signal Based Surveillance, 493–498

  51. Stamou G, van Ossenbruggen J, Pan JZ, Schreiber G, Smith JR (2006) Multimedia annotations on the Semantic Web. IEEE Multimedia 13(1):86–90

    Article  Google Scholar 

  52. Swathi HY, Shivakumar G, Mohana HS (2017) Crowd Behavior Analysis: A Survey . IEEE International Conference on Recent Advances in Electronics and Communication Technology (ICRAECT)

  53. Tripathi G, Singh K, Vishwakarma DK (2019) Convolutional neural networks for crowd behaviour analysis: a survey. Visual Computing 35:753–776. https://doi.org/10.1007/s00371-018-1499-5

    Article  Google Scholar 

  54. Tsinaraki C, Polydoros P, Christodoulakis S (2004) Interoperability support for Ontology-based Video Retrieval Applications. In Proc. of 3rd International Conference on Image and Video Retrieval (CIVR 2004)

  55. Tu P, Sebastian T, Doretto G, Krahnstoever N, Rittscher J, Yu T (2008) Unified Crowd Segmentation. European Conference on Computer Vision 5305:691–704

    Google Scholar 

  56. University of Minnesota, available from http://mha.cs.umn.edu/movies/crowdactivity-all.avi.

  57. University of Reading, PETS 2009 Dataset S3 Rapid Dispersion, available from http://www.cvg.rdg.ac.uk/PETS2009/a.html#s2l1

  58. Vishwakarma S, Agrawal A (2013) A Survey on Activity Recognition and Behavior Understanding in Video Surveillance. Visual Computing 29(10):983–1009

    Article  Google Scholar 

  59. Wang X, Loy C-C (2017) Deep Learning for Scene Independent Crowd Analysis. Group and Crowd Behavior for Computer Vision, 209–252

  60. Weixin L, Mahadevan V, Vasconcelos N (2014) Anomaly Detection and Localization in Crowded Scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(1):18–32

    Article  Google Scholar 

  61. Weiya R, Guo-Hui L, Jun C, Hao-Zhe L (2012) Abnormal crowd behavior detection using behavior entropy model. International Conference on Wavelet Analysis and Pattern Recognition. 212–221

  62. Wu S, Moore BE, Shah M (2010) Chaotic Invariants of Lagrangian Particle Trajectories for Anomaly Detection in Crowded Scenes. IEEE Conference on Computer Vision and Pattern Recognition, pages:2054–2060

  63. Wu S, Wong HS, Yu Z (2014) A Bayesian Model for Crowd Escape Behaviour Detection. IEEE Transactions on Circuits and Systems for Video Technology 24(1):85–98

    Article  Google Scholar 

  64. Xue M, Zheng S, Zhang C (2012) Ontology-based surveillance video archive and retrieval system. IEEE International Conference on Advanced Computational Intelligence (ICACI), 84–89

  65. Zhang X, Yu Q, Yu H (2018) Physics Inspired Methods for Crowd Video Surveillance and Analysis: A Survey. IEEE Access 6:66816–66830

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, Near East University, Near East Boulevard, via Mersin 10, Nicosia, North Cyprus, Turkey

    Eren Hatirnaz & Melike Sah

  2. Electrical and Electronics Engineering Department, Middle East Technical University – Northern Cyprus Campus, Kalkanli, via Mersin 10, Guzelyurt, North Cyprus, Turkey

    Cem Direkoglu

Authors
  1. Eren Hatirnaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melike Sah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cem Direkoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melike Sah.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hatirnaz, E., Sah, M. & Direkoglu, C. A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videos. Multimed Tools Appl 79, 17579–17617 (2020). https://doi.org/10.1007/s11042-020-08659-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08659-2

Keywords

Search

Navigation