Skip to main content
SpringerLink
Log in

A hybrid CNN and LSTM-based deep learning model for abnormal behavior detection

  • 1195: Deep Learning for Multimedia Signal Processing and Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Traditional cameras can only record videos passively. If the camera can further automatically recognize human behavior and activity, it can immediately issue an alarm to notify the monitor or guards when abnormal behavior is detected. Hence, the monitor or guard can quickly take relevant actions based on the detected behavior. In this paper, we propose a deep learning model for abnormal behavior detection, which use object detection technology YOLOv3 to detect pedestrians, and then use hybrid Deep-SORT algorithm to track pedestrians to obtain tracking trajectories from the sequence frames. Then, the convolutional neural network (CNN) is used to extract the action characteristics of each tracked trajectory, and the long short-term memory network (LSTM) is used to build anomalous behavior identification model to predict abnormal behavior, such as falling, kicking, punching, etc. The experimental results show that the proposed method has a good recognition rate in different behavior data sets, and it can also meet the needs of real-time monitoring.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Aliet S, Shah M (2010) Human action recognition in videos using kinematic features and multiple instance learning. IEEE Trans Pattern Anal Mach Intell 32(2):288–303

    Article  Google Scholar 

  2. Alwan M et al (2006) A smart and passive floor-vibration based fall detector for elderly. In: 2nd International Conference on Information and Communication Technologies

  3. Arifoglu D, Bouchach A (2017) Activity recognition and abnormal behaviour detection with recurrent neural networks. In: Proc. of MobiSPC. pp 86–93

  4. Belshaw M, Taati B, Giesbercht D, Mihailidis A (2011) Intelligent vision-based fall detection system: preliminary results from a real world deployment. RESNA/ICTA 2011: Advancing Rehabilitation Technologies for an Aging Society

  5. Belshaw M, Taati B, Snoek J, Mihailidis A (2011) Towards a single sensor passive solution for automated fall detection. Conf Proc IEEE Eng Med Biol Soc. pp 1773–1776

  6. Bera A, Kim S, Manocha D (2016) Realtime anomaly detection using trajectory-wise crowd behavior learning. Proceedings of International Conference on Computer Vision and Pattern Recognition, CVPRW’13. pp 50–57

  7. Bermejo NE, Deniz SO, Bueno GG, Rahul S (2011) Violence detection in video using computer vision techniques. In: Proc. of the 14th international conference on computer analysis of images and patterns, CAIP2011, pp 332–339

  8. Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. In: Proc. of IEEE International Conference on Image Processing

  9. Bourke AK, O’Brien JV, Lyons GM (2007) Evaluation of a threshold-based tri-axial accelerometer fall detection algorithm. Gait Posture 26(2):194–199

    Article  Google Scholar 

  10. Bourke AK, van de Ven PW, Chaya AE, OLaighin GM, Nelson J (2008) Testing of a long-term fall detection system incorporated into a custom vest for the elderly. Proc. Of IEEE Conf. on Eng Med Biol Soc. pp 2844–2847

  11. Burkard R, Dell'Amico M, Martello S (2012) Assignment problems. Revised reprint. SIAM - Society of Industrial and Applied Mathematics, Philadelphia

    Book  Google Scholar 

  12. Chaker R, Aghbari ZA, Junejo IN (2017) Social network model for crowd anomaly detection and localization. Pattern Recogn 61:266–281

    Article  Google Scholar 

  13. Charfi I, Miteran J, Dubois J, Atri M, Tourki R (2013) Optimised spatio-temporal descriptors for real-time fall detection: comparison of SVM and Adaboost based classification. J Electron Imaging 22(4):17

    Article  Google Scholar 

  14. Fan Y, Levine MD, Wen G, Qiu S (2017) A deep neural network for real-time detection of falling humans in naturally occurring scenes. Neurocomputing 260:43–58

    Article  Google Scholar 

  15. Feng Y, Yuan Y, Lu X (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  16. Girshick R (2015) Fast R-CNN. In: The IEEE International Conference on Computer Vision, Santiago, Chile. pp 1440–1448

  17. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, pp 580–587.

  18. Gomez HF, Martinez TR, Arias TS, Fernandez CA, Sylvie R, Gonzalez EA et al (2015) Identification of loitering human behaviour in video surveillance environments. Artif Comput Biol Med 9107:516–525

    Article  Google Scholar 

  19. Guang S, Fu G, Li P, Geng H (2014) Violent behavior detection based on SVM in the elevator. Int J Secur Appl 8:31–40

    Google Scholar 

  20. Harrou F, Zerrouki N, Sun Y, Houacine A (2017)Vision-based fall detection system for improving safety of elderly people. IEEE Instrum Meas Mag 20:19–55

    Article  Google Scholar 

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc of IEEE Conference on Computer Vision and Pattern Recognition. pp 770–778

  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  23. Hu Y (2020) Design and implementation of abnormal behavior detection based on deep intelligent analysis algorithms in massive video surveillance. J Grid Comput 18:227–237

    Article  Google Scholar 

  24. Hu X, Dai J et al (2019) A weakly supervised framework for abnormal behavior detection and localization in crowded scenes. Neurocomputing 383:270–281

    Article  Google Scholar 

  25. Huang Z, Niu Q, Xiao S (2020) Human behavior recognition based on motion data analysis. Int J Pattern Recognit Artif Intell 34(9):2056005

    Article  Google Scholar 

  26. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  27. Kangas M et al (2009) Sensitivity and specificity of fall detection in people aged 40 years and over. Gait Posture 29(4):571–574

    Article  Google Scholar 

  28. Kim Y, Bang H (2018) Introduction to Kalman filter and its applications. Open access peer-reviewed chapter. https://doi.org/10.5772/intechopen.80600

  29. Kim J, Grauman K (2009) Observe locally, infer globally: a space–time MRF for detecting abnormal activities with incremental updates. Proc. IEEE Conf. Comput. Vis. Pattern Recognition. pp 2921–2928

  30. Laxhammar R, Falkman G (2014) Online learning and sequential anomaly detection in trajectories. IEEE Trans Pattern Anal Mach Intell 36(6):1158–1173

    Article  Google Scholar 

  31. Li Y, Zeng Z, Popescu M, Ho KC (2010) Acoustic fall detection using a circular microphone array. Conf Proc IEEE Eng Med Biol Soc pp 2242–2245

  32. Lin T-Y, Dollar P et al (2017) Feature pyramid networks for object detection. Proc. of IEEE Conference on Computer Vision and Pattern Recognition

  33. Mabrouk AB, Zagrouba E (2018) Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Syst Appl 91:480–491

    Article  Google Scholar 

  34. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. Proc. IEEE Conf. Comput. Vis. Pattern Recognition. pp 935–942

  35. Nannan L, Xinyu W et al (2015) Anomaly detection in video surveillance via gaussian process, International Journal of Pattern Recognition and Artificial Intelligence. 29(6):1555011

  36. Nguyen VD, Le MT et al (2014) An efficient camera-based surveillance for fall detection of elderly people. In: IEEE 9th conference on Industrial electronics and applications (ICIEA). pp 994–997

  37. Nunezez-Marcos A, Azkune G, Arganda-Carreras I (2017)Vision-based fall detection with convolutional neural networks. Wirel Commun Mob Comput 2017:1–16

    Article  Google Scholar 

  38. Olah C (2015) Understanding LSTM networks. [Online]. Available: https://colah.github.io/posts/2015-08-Understanding-LSTMs/. Accessed 26 June 2019

  39. Pan H et al (2018) Fighting detection based on pedestrian pose estimation. In: 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). pp 1–5

  40. Popescu M, Li Y, Skubic M, Rantz M (2008) An acoustic fall detector system that uses sound height information to reduce the false alarm rate. Conf Proc IEEE Eng Med Biol Soc. pp 4628–4631

  41. Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767, 2018.

  42. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. pp 779–788

  43. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149

    Article  Google Scholar 

  44. Ryoo MS, Aggarwal JK (2009)Spatio-temporal relationship match: video structure comparison for recognition of complex human activities. In: IEEE 12th International Conference on Computer Vision, Kyoto, Japan

  45. Stone EE, Skubic M (2015) Fall detection in homes of older adults using the Microsoft Kinect. IEEE J Biomed Health Inform 19:290–301

    Article  Google Scholar 

  46. Tani MK, Lablack A, Ghomari A, Bilasco IM (2015) Events detection using a video-surveillance ontology and a rule-based approach. Computer vision - ECCV 2014 workshops. pp 299–308

  47. Uijlings J, van de Sande K, Gevers T, Smeulders A (2013) Selective search for object. Int J Comput Vis 104(2):154–171

    Article  Google Scholar 

  48. Van Beeck K, Van Engeland K, Vennekens J, Goedemé T (2017) Abnormal behavior detection in LWIR surveillance of railway platforms. 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). pp 1–6

  49. Wang XF, He XH et al (2016) A classification method based on streak flow for abnormal crowd behaviors. Optik 127(4):2386–2392

    Article  Google Scholar 

  50. Weiya R, Guohui L, Boliang S, Kuihua H (2015) Unsupervised kernel learning for abnormal events detection. Vis Comput 31:245–255

    Article  Google Scholar 

  51. Welch G, Bishop G (2006) An introduction to the Kalman filter. http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf

  52. Wojke N, Bewley A, Paulus D (2017) Deep SORT: simple online and realtime tracking with a deep association metric. arXiv preprint arXiv:1703.07402

  53. Wu S, Moore BE, Shah M (2010) Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. Proc. IEEE Conf. Comput. Vis. Pattern Recognition. pp 2054–2060

  54. Zhang J, Wu C, Wang Y (2020) Human fall detection based on body posture spatio-temporal evolution. Sensors 20(3):946

    Article  Google Scholar 

  55. Zhao Z-Q, Zheng P, Xu S-t, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Lear Syst 99:1–21

    Google Scholar 

  56. Zhao R, Wang Y, Jia P, Li C, Ma Y, Zhang Z (2021) Abnormal human behavior recognition based on image processing technology. In: IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). pp 1924–1928

  57. Zhu Y, Wang Z (2016)Real-time abnormal behavior detection in elevator. Intelligent Visual Surveillance: 4th Chinese Conference. pp 154–161

Download references

Acknowledgments

This work was supported by the Ministry of Science and Technology, Taiwan, under the grants MOST 109-2637-E-167-004 -.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Chin-Yi University of Technology, No.57, Sec. 2, Zhongshan Rd., Taiping Dist., Taichung, 41170, Taiwan

    Chuan-Wang Chang

  2. Department of Computer Science and Information Engineering, National Yunlin University of Science and Technology, No. 123 University Rd., Sec. 3, Douliou, Yunlin, 64002, Taiwan

    Chuan-Yu Chang & You-Ying Lin

Authors
  1. Chuan-Wang Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuan-Yu Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. You-Ying Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chuan-Yu Chang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chang, CW., Chang, CY. & Lin, YY. A hybrid CNN and LSTM-based deep learning model for abnormal behavior detection. Multimed Tools Appl 81, 11825–11843 (2022). https://doi.org/10.1007/s11042-021-11887-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11887-9

Keywords

Search

Navigation