Skip to main content
SpringerLink
Log in

A novel approach for suspicious activity detection with deep learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Suspicious human activities like fighting, shooting, fire have got serious security concern in public places because of a steep surge in these types of cases all around. CCTV cameras are generally installed at public places like malls, railway stations; but evidences suggest that only availability of these cameras are not very effective unless the video feeds are constantly monitored. Therefore, we intend to build an automated and intelligent video surveillance system to detect the suspicious human activities, followed by an alert generation. In this article, we propose a deep neural network-based solution to identify suspicious human activities. Here, the deep Inception V3 model extracts the salient discriminative activity-specific features from video streams. Furthermore, we feed these features into a recurrent neural network, namely Long Short Term Memory (LSTM) network, which is used to develop a temporal relation between features extracted from consecutive frames in order to distinguish suspicious human activities accurately. Added to it, the proposed system is evaluated for diverse data by collecting activities from eleven benchmark databases: KTH action database, WEIZMANN database, JHMDB database, HMDB database, UCF-Crime database, UCF101 database, MIVIA database, UCF database, FIRESENSE database, VISILAB database, and SHAKEFIVE2 database. The proposed approach achieved a recognition rate of 98.87%, showing significant improvement as compared to the state-of-the-art (SOTA) methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availability

All data analyzed during this study are summarized in the Tables 2 and 3. Furthermore, hyperlinks of all the datasets are summarized in the Table 9. Requests for any other material related to this paper should be made to the corresponding author.

References

  1. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans Pattern Anal Mach Intell 30(3):555–60

    Article  Google Scholar 

  2. Ainsworth T (2002) Buyer beware. Security Oz 19:18–26

    Google Scholar 

  3. Ali S, Shah M (2007) A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. IEEE Int Conf Comput Vis Pattern Recognit (CVPR)

  4. Bao L, Intille SS (2004) Activity recognition from user-annotated acceleration data. Int Conf Pervasive Comput:1–17

  5. Blank M, Gorelick L, Shechtman E, Irani M, Basri R (2005) Actions as space-time shapes. IEEE Int Conf Comput Vis (ICCV) 2:1395–1402

    Google Scholar 

  6. Carletti V, Foggia P, Percannella G, Saggese A, Vento M (2013) Recognition of human actions from rgb-d videos using a reject option. Int Conf Image Anal Process:436–445

  7. Dimitropoulos K, Barmpoutis P, Grammalidis N (2014) Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection. IEEE Trans Circuits Syst Video Technol 25(2):339–51

    Article  Google Scholar 

  8. Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. Proc IEEE Conf Comput Vis Pattern Recognit

  9. Does video surveillance deter crime? (2014) Accessed 2 April 2021. https://evidencebasedliving.human.cornell.edu/2014/10/03/does-video-surveillance-deter-crime/

  10. Elhamod M, Levine MD (2012) Automated real-time detection of potentially suspicious behavior in public transport areas. IEEE Trans Intell Transp Syst 14(2):688–99

    Article  Google Scholar 

  11. Fan Y, Levine MD, Wen G, Qiu S (2017) A deep neural network for real-time detection of falling humans in naturally occurring scenes. Neurocomputing 260:43–58

    Article  Google Scholar 

  12. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proc IEEE Conf Comput Vis Pattern Recognit

  13. Foucher S, Lalonde M, Gagnon L (2011) A system for airport surveillance: detection of people running, abandoned objects, and pointing gestures. Vis Inf Process XX 8056:805610

    Google Scholar 

  14. Gholamrezaii M, AlModarresi SMT (2021) A time-efficient convolutional neural network model in human activity recognition. Multimed Tools Appl 80 (13):19361–19376

    Article  Google Scholar 

  15. Gouaillier V, Fleurant A (2009) Intelligent video surveillance: promises and challenges, technological and commercial intelligence report. CRIM Tech Defence Security:456–468

  16. Jhuang H, Gall J, Zuffi S, Schmid C, Black MJ (2013) Towards understanding action recognition. IEEE Int Conf Comput Vis:3192–3199

  17. Ji S, Xu W, Yang M, Yu K (2012) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35:221–231

    Article  Google Scholar 

  18. Kapoor S (2021) UP: lakhs Of rupees jewelery looted by showing weapons in Ashiana area of lucknow. News Bust, 09 March 2021. Accessed on 28 March 2021

  19. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition. Int Conf Comput Vis:2556–2563

  20. Kumar KPS, Bhavani R (2020) Human activity recognition in egocentric video using HOG, GiST and color features. Multimed Tools Appl 79:3543–3559

    Article  Google Scholar 

  21. Kushwaha A, Khare A, Khare M (2021) Human activity recognition algorithm in video sequences based on integration of magnitude and orientation. Inf Optical Flow Int J Image Graph

  22. Ladjailia A, Bouchrika I, Merouani HF, Harrati N, Mahfouf Z (2020) Human activity recognition via optical flow: decomposing activities into basic actions. Neural Comput Appl 32(21):16387–16400

    Article  Google Scholar 

  23. Ling H, Wu J, Huang J, Chen J, Li P (2020) Attention-based convolutional neural network for deep face recognition. Multimed Tools Appl 79(9):5595–5616

    Article  Google Scholar 

  24. Nogueira TDC, Vinhal CDN, Junior DCG, Ullmann MRD (2020) Reference-based model using multimodal gated recurrent units for image captioning. Multimed Tools Appl 79(41):30615–30635

    Article  Google Scholar 

  25. Phillips Iii W, Shah M, da Vitoria Lobo N (2012) Flame recognition in video. Pattern Recognit Lett 23(1-3):319–327

    Article  MATH  Google Scholar 

  26. Pu S, Song T, Zhang Y, Xie D (2017) Estimation of crowd density in surveillance scenes based on deep convolutional neural network. Proc Comput Sci 111:154–159

    Article  Google Scholar 

  27. Rashwan HA, Garcia MA, Abdulwahab S, Puig D (2020) Action representation and recognition through temporal co-occurrence of flow fields and convolutional neural networks. Multimed Tools Appl 79:34141–34158

    Article  Google Scholar 

  28. Ravi N, Nikhil D, Prreetham M, Littman ML (2005) Activity recognition from accelerometer data. Aaai 5(2005):1541–1546

    Google Scholar 

  29. Rueda FM, Grzeszick R, Fink GA, Feldhorst SS, Hompel MT (2018) Convolutional neural networks for human activity recognition using body-worn sensors. Informatics 5(2):26

    Article  Google Scholar 

  30. Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. Proc 17th Int Conf Pattern Recognit (ICPR). IEEE:32–36

  31. Serrano I, Déniz O, García GB (2013) VISILAB At MediaEval fight detection. MediaEval

  32. Sharif MH, Djeraba C (2012) An entropy approach for abnormal activities detection in video streams. Pattern Recognit 45(7):2543–2561

    Article  Google Scholar 

  33. Shen J, Robertson N (2021) BBAS: towards large scale effective ensemble adversarial attacks against deep neural network learning. Inf Sci 569:469–478

    Article  Google Scholar 

  34. Soomro K, Zamir AR, Shah M (2012) UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402

  35. Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. Proc IEEE Conf Comput Vis Patt Recognit

  36. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  37. Tam NN, Ly NQ (2017) Abnormal activity detection based on dense spatial-temporal features and improved one-class learning. Proc Eighth Int Symp Inf Commun Technol:370–377

  38. Tarassenko L, Nairac A, Townsend N, Buston I, Cowley P (2000) Novelty detection for the identification of abnormalities. Int J Syst Sci 31 (11):1427–1439

    Article  MATH  Google Scholar 

  39. Tripathi V, Gangodkar D, Latta V, Mittal A (2015) Robust abnormal event recognition via motion and shape analysis at ATM installations. J Electr Comput Eng

  40. (2021). Two on bike snatch gold chain from businessman in Greater Noida market. https://timesofindia.indiatimes.com/city/noida/2-on-bike-snatch-gold-chain-from-bizman-in-city-market/articleshow/79345480.cms. 22 Nov 2020. Accessed on 28 March 2021

  41. Van Gemeren C, Poppe R, Veltkamp RC (2016) Spatio-temporal detection of fine-grained dyadic human interactions. Proc Human Behavior Understand Workshop (ECCV-HBU):116–133

  42. Veeriah V, Zhuang N, Qi GJ (2015) Differential recurrent neural networks for action recognition. IEEE Int Conf Comput Vis:4041–4049

  43. Velastin SA, Boghossian BA, Vicencio-Silva MA (2006) A motion-based image processing system for detecting potentially dangerous situations in underground railway stations. Trans Res Part C Emerging Technol 14(2):96–113

    Article  Google Scholar 

  44. Wang L, Qian X, Zhang Y, Shen J, Cao X (2019) Enhancing sketch-based image retrieval by cnn semantic re-ranking. IEEE Trans Cybern 50 (7):3330–3342

    Article  Google Scholar 

  45. Wiliem A, Madasu V, Boles W, Yarlagadda P (2012) A suspicious behaviour detection using a context space model for smart surveillance systems. Comput Vis Image Underst 116(2):194–209

    Article  Google Scholar 

  46. Yuan X, Li L, Wang Y (2019) Nonlinear dynamic soft sensor modeling with supervised long short-term memory network. IEEE Trans Industr Inf 16 (5):3168–3176

    Article  MathSciNet  Google Scholar 

  47. Zhang T, Jia W, He X, Yang J (2016) Discriminative dictionary learning with motion weber local descriptor for violence detection. IEEE Trans Circuits Syst Video Technol 27(3):696–709

    Article  Google Scholar 

  48. Zhong H, Jianbo S, Mirko V (2004) Detecting unusual activity in video. IEEE Comput Society Conf Comput Vis Pattern Recognit (CVPR) 2:II–II

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Motilal Nehru National Institute of Technology Allahabad, Prayagraj, Prayagraj, India

    Neelam Dwivedi, Dushyant Kumar Singh & Dharmender Singh Kushwaha

Authors
  1. Neelam Dwivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dushyant Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dharmender Singh Kushwaha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Neelam Dwivedi.

Ethics declarations

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dwivedi, N., Singh, D.K. & Kushwaha, D.S. A novel approach for suspicious activity detection with deep learning. Multimed Tools Appl 82, 32397–32420 (2023). https://doi.org/10.1007/s11042-023-14445-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14445-7

Keywords

Search

Navigation