Advertisement

World Wide Web

, Volume 22, Issue 2, pp 689–715 | Cite as

Abnormal event detection in tourism video based on salient spatio-temporal features and sparse combination learning

  • Yue Geng
  • Junping DuEmail author
  • Meiyu Liang
Article
  • 212 Downloads
Part of the following topical collections:
  1. Special Issue on Deep vs. Shallow: Learning for Emerging Web-scale Data Computing and Applications

Abstract

With the booming development of tourism, travel security problems are becoming more and more prominent. Congestion, stampedes, fights and other tourism emergency events occurred frequently, which should be a wake-up call for tourism security. Therefore, it is of great research value and application prospect to real-time monitor tourists and detect abnormal events in tourism surveillance video by using computer vision and video intelligent processing technology, which can realize the timely forecast and early warning of tourism emergencies. At present, although most of the video-based abnormal event detection methods work well in simple scenes, there are often problems such as low detection rate and high false positive rate in complex motion scenarios, and the detection of abnormal events can’t be processed in real time. To tackle these issues, we propose an abnormal event detection model in tourism video based on salient spatio-temporal features and sparse combination learning, which has good robustness and timeliness in complex motion scenarios and can be adapted to real-time anomaly detection in practical applications. Specifically, spatio-temporal gradient model is combined with foreground detection to extract 3D gradient features on the foreground target of video sequence as the salient spatio-temporal features, which can eliminate the interference of the background. Sparse combination learning algorithm is used to establish the abnormal event detection model, which can realize the real-time detection of abnormal events. In addition, we construct a new ScenicSpot dataset with 18 video clips (5964 frames) containing both normal and abnormal events. The experimental results on ScenicSpot dataset and two standard benchmark datasets show that our method can realize the automatic detection and recognition of tourists’ abnormal behavior, and has better performance compared with the classical methods.

Keywords

Abnormal event detection Gaussian mixture model Salient spatio-temporal features PCA dimensional reduction Sparse combination learning 

Notes

Funding

This work was supported by the National Natural Science Foundation of China (No. 61320106006, No. 61502042, No. 61532006, No. 61772083) and the Fundamental Research Funds for the Central University (No. 2017RC39).

References

  1. 1.
    Benezeth, Y., Jodoin, P.M., Saligrama, V., Rosenberger, C.: Abnormal events detection based on spatio-temporal co-occurences. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2458–2465 (2009).  https://doi.org/10.1109/CVPRW.2009.5206686
  2. 2.
    Chen, H., Zhao, X., Wang, T., Tan, M., Sun, S.: Spatial-temporal context-aware abnormal event detection based on incremental sparse combination learning. In: 2016 12th World Congress on Intelligent Control and Automation (WCICA), pp. 640–644 (2016).  https://doi.org/10.1109/WCICA.2016.7578533 CrossRefGoogle Scholar
  3. 3.
    Chen, T., Hou, C., Wang, Z., Chen, H.: Anomaly detection in crowded scenes using motion energy model. Multimedia Tools and Applications. 3, (2017).  https://doi.org/10.1007/s11042-017-5020-3
  4. 4.
    Colque, R.V.H.M., Schwartz, W.R.: Histograms of optical flow orientation and magnitude to detect anomalous events in videos. IEEE Transactions on Circuits and Systems for Video Technology. 99 (2017).  https://doi.org/10.1109/SIBGRAPI.2015.21
  5. 5.
    Cong, Y., Yuan, J., Liu, J.: Abnormal event detection in crowded scenes using sparse representation. Pattern Recogn. 46(7), 1851–1864 (2013).  https://doi.org/10.1016/j.patcog.2012.11.021 CrossRefGoogle Scholar
  6. 6.
    Cui, X., Liu, Q., Gao, M., Metaxas, D.N.: Abnormal detection using interaction energy potentials. The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 42(7). 3161–3167 (2011).  https://doi.org/10.1109/CVPR.2011.5995558
  7. 7.
    Cui, J., Liu, W., Xing, W.: Crowd behaviors analysis and abnormal detection based on surveillance data. J. Vis. Lang. Comput. 25(6), 628–636 (2014).  https://doi.org/10.1016/j.jvlc.2014.10.032 CrossRefGoogle Scholar
  8. 8.
    Du, D., Qi, H., Huang, Q., Zeng, W., Zhang, C.: Abnormal event detection in crowded scenes based on structural multi-scale motion interrelated patterns. IEEE International Conference on Multimedia and Expo (ICME). 1–6 (2013).  https://doi.org/10.1109/ICME.2013.6607499
  9. 9.
    Feng, Y., Yuan, Y., Lu, X.: Learning deep event models for crowd anomaly detection. Neurocomputing. 219, 548–556 (2017).  https://doi.org/10.1016/j.neucom.2016.09.063 CrossRefGoogle Scholar
  10. 10.
    Gao, L., Song, J., Liu, X., Shao, J., Liu, J., Shao, J.: Learning in high-dimensional multimedia data: the state of the art. Multimedia Systems. 23(3), 1–11 (2017).  https://doi.org/10.1007/s00530-015-0494-1 CrossRefGoogle Scholar
  11. 11.
    Gao, L., Guo, Z., Zhang, H., Xu, X., Shen, H.T.: Video captioning with attention-based LSTM and semantic consistency. IEEE Transactions on Multimedia. 19(9), 2045–2055 (2017).  https://doi.org/10.1109/TMM.2017.2729019 CrossRefGoogle Scholar
  12. 12.
    Gogoi, P., Bhattacharyya, D.K., Borah, B., Kalita, J.K.: A survey of outlier detection methods in network anomaly identification. Comput. J. 54(4), 570–588 (2011).  https://doi.org/10.1093/comjnl/bxr026 CrossRefGoogle Scholar
  13. 13.
    Hu, D.H., Zhang, X.X., Yin, J., Zheng, V.W., Yang, Q.: Abnormal activity recognition based on HDP-HMM models. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI), pp. 1715–1720 (2009)Google Scholar
  14. 14.
    Huang, D., Hu, W., Wu, X., et al.: The algorithm of video foreground extraction via improved single gauss model and merge of broken targets. J. Signal Process. 3, 299–307 (2015).  https://doi.org/10.3969/j.issn.1003-0530.2015.03.007 Google Scholar
  15. 15.
    Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A. 374(2065), 20150202 (2016).  https://doi.org/10.1098/rsta.2015.0202 MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Kong, L., Guo, L., Wang, Q., Han, Y.: Improvement of linear filter in image denoising. In: International Conference on Intelligent Earth Observing and Applications, Pp. 98083F (2015).  https://doi.org/10.1117/12.2207241 Google Scholar
  17. 17.
    Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1446–1453 (2009).  https://doi.org/10.1109/CVPR.2009.5206771
  18. 18.
    Leyva, R., Sanchez, V., Li, C.T.: Video anomaly detection with compact feature sets for online performance. IEEE Trans. Image Process. 26, 99–3478 (2017).  https://doi.org/10.1109/TIP.2017.2695105 MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Li, A., Miao, Z., Cen, Y., Liang, Q.: Abnormal event detection based on sparse reconstruction in crowded scenes. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1786–1790 (2016).  https://doi.org/10.1109/ICASSP.2016.7471984
  20. 20.
    Li, X., Zhou, Z., Chen, L., Gao, L.: Residual attention-based LSTM for video captioning. World Wide Web: Internet and Web Information Systems. 9, 1–16 (2018).  https://doi.org/10.1007/s11280-018-0531-z Google Scholar
  21. 21.
    Liu, Z., Feng, X., Zhang, J.: Action recognition based on deep convolution neural network and depth sequence. Journal of Chongqing University (Natural Science Edition). 40(11), 99–106 (2017).  https://doi.org/10.11835/j.issn.1000-582X.2017.11.012 Google Scholar
  22. 22.
    Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 FPS in MATLAB. In: IEEE International Conference on Computer Vision (ICCV), pp. 2720–2727 (2014).  https://doi.org/10.1109/ICCV.2013.338 Google Scholar
  23. 23.
    Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1975–1981 (2010).  https://doi.org/10.1109/CVPR.2010.5539872
  24. 24.
    Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 935–942 (2009).  https://doi.org/10.1109/CVPR.2009.5206641
  25. 25.
    Meng, L.I., Chen, K., Guo, C., Fei, L.I., Peipei, J.I.: Abnormal crowd event detection by fusing saliency information and social force model. Opto-Electron. Eng. (2016)Google Scholar
  26. 26.
    Miao, Y., Song, J.: Abnormal event detection based on SVM in video surveillance. Advanced Research and Technology in Industry Applications. 1379–1383 (2014).  https://doi.org/10.1109/WARTIA.2014.6976540
  27. 27.
    Mittal, S., Prasad, T., Saurabh, S., Fan, X., Shin, H.: Pedestrian detection and tracking using deformable part models and Kalman filtering. In: Soc Design Conference, 10(7), pp. 960–966 (2013). doi:  https://doi.org/10.1109/ISOCC.2012.6407106 Google Scholar
  28. 28.
    Nallaivarothayan, H., Fookes, C., Denman, S., Sridharan, S.: An MRF based abnormal event detection approach using motion and appearance features. IEEE International Conference on Advanced Video and Signal Based Surveillance. 343–348 (2014).  https://doi.org/10.1109/AVSS.2014.6918692
  29. 29.
    Pathan, S.S., Al-Hamadi, A., Michaelis, B.: Using conditional random field for crowd behavior analysis. In: Asian Conference on Computer Vision (ACCV). 6468, 370–379 (2010)Google Scholar
  30. 30.
    Pennisi, A., Bloisi, D.D., Locchi, L.: Online real-time crowd behavior detection in video sequences. Comput. Vis. Image Underst. 144, 166–176 (2016).  https://doi.org/10.1016/j.cviu.2015.09.010 CrossRefGoogle Scholar
  31. 31.
    Rao, A.S., Gubbi, J., Marusic, S., Palaniswami, M.: Crowd event detection on optical flow manifolds. IEEE Transactions on Cybernetics. 46(7), 1524–1537 (2016).  https://doi.org/10.1109/TCYB.2015.2451136 CrossRefGoogle Scholar
  32. 32.
    Ren, H., Moeslund, T.B.: Abnormal event detection using local sparse representation. IEEE International Conference on Advanced Video and Signal Based Surveillance. 125–130 (2014).  https://doi.org/10.1109/AVSS.2014.6918655
  33. 33.
    Shen, Y., Wang, X.: Video moving target detection method based on background subtraction and interframe difference method. Automation & Instrumentation. 4, 122–124 (2017).  https://doi.org/10.14016/j.cnki.1001-9227.2017.04.122 Google Scholar
  34. 34.
    Wali, A., Alimi, A.M.: Event detection from video surveillance data based on optical flow histogram and high-level feature extraction. International Workshop on Database and Expert Systems Application. 221–225 (2009).  https://doi.org/10.1109/DEXA.2009.81
  35. 35.
    Wang, T., Snoussi, H.: Histograms of optical flow orientation for visual abnormal events detection. IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance. 13–18 (2012).  https://doi.org/10.1109/AVSS.2012.39
  36. 36.
    Wang, J., Schweitzer, J., Tilmann, F., White, R.S., Soosalu, H.: Application of the multichannel wiener filter to regional event detection using NORSAR seismic-array data. Bull. Seismol. Soc. Am. 101(6), 2887–2896 (2011).  https://doi.org/10.1785/0120110003 CrossRefGoogle Scholar
  37. 37.
    Wang, S.M., Fang, L.Y., Deng, F.: Research on the evaluation model of urban tourism management efficiency with uncertain linguistic information. Journal of Control Science and Engineering. 2, 12–14 (2014).  https://doi.org/10.1155/2014/582454 zbMATHGoogle Scholar
  38. 38.
    Wang, M., Li, X., Chen, Q., et al.: Surveillance event detection based on CNN. Acta Automat. Sin. 42(6), 892–903 (2016).  https://doi.org/10.16383/j.aas.2016.c150729 Google Scholar
  39. 39.
    Wang, C., Yao, H., Sun, X.: Anomaly detection based on spatio-temporal sparse representation and visual attention analysis. Multimedia Tools and Applications. 76, 1–17 (2016).  https://doi.org/10.1007/s11042-015-3199-8 Google Scholar
  40. 40.
    Wang, X., Gao, L., Wang, P., Sun, X., Liu, X.: Two-stream 3D convNet fusion for action recognition in videos with arbitrary size and length. IEEE Transactions on Multimedia. PP. 644(99), 1–1 (2017).  https://doi.org/10.1109/TMM.2017.2749159 Google Scholar
  41. 41.
    Wang, X., Gao, L., Song, J., Zhen, X., Sebe, N., Shen, H.T.: Deep appearance and motion learning for egocentric activity recognition. Neurocomputing. 275, 438–447 (2018).  https://doi.org/10.1016/j.neucom.2017.08.063 CrossRefGoogle Scholar
  42. 42.
    Wen, Y., Du, J., Lee, J.M.: Abnormal event detection based on social force model combined with crowd violent flow. International Conference on Cloud Computing and Intelligence Systems. 440–446 (2016).  https://doi.org/10.1109/CCIS.2016.7790299
  43. 43.
    Wriggers, W., Stafford, K.A., Shan, Y., Piana, S., Maragakis, P., Lindorff-Larsen, K., Miller, P.J., Gullingsrud, J., Rendleman, C.A., Eastwood, M.P., Dror, R.O., Shaw, D.E.: Automated event detection and activity monitoring in long molecular dynamics simulations. J. Chem. Theory Comput. 5(10), 2595–2605 (2009)CrossRefGoogle Scholar
  44. 44.
    Wu, C., Li, M., Liu, M., Zheng, Z., Zhang, Y.: Adaptive motion detection based on median background model. Journal of Shenyang Jianzhu University. (2008)Google Scholar
  45. 45.
    Wu, X., Guo, H., Li, N., et al.: Survey on the video-based abnormal event detection in crowd scenes. Journal of Electronic Measurement and Instrument. 28(6), 575–584 (2014).  https://doi.org/10.13382/j.jemi.2014.06.001 Google Scholar
  46. 46.
    Xing, H.U., Shiqiang, H.U., Luo, L., Guoxiang, L.I.: Abnormal event detection in crowded scenes via bag-of-atomic-events-based topic model. Turk. J. Electr. Eng. Comput. Sci. 24, 2638–2653 (2016).  https://doi.org/10.3906/elk-1310-191 CrossRefGoogle Scholar
  47. 47.
    Xu, D., Ricci, E., Yan, Y., et al.: Detecting anomalous events in videos by learning deep representations of appearance and motion. Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv. 1510, 01553–01127 (2015).  https://doi.org/10.1016/j.cviu.2016.10.010 Google Scholar
  48. 48.
    Yang, H., Cao, Y., Wu, S., Lin, W.: Abnormal crowd behavior detection based on local pressure model. In: Signal and Information Processing Association Summit and Conference (APSIPA ASC), pp. 1–4 (2014)Google Scholar
  49. 49.
    Ye, R., Li, X.: Collective representation for abnormal event detection. J. Comput. Sci. Technol. 32(3), 470–479 (2017).  https://doi.org/10.1007/s11390-017-1737-8 MathSciNetCrossRefGoogle Scholar
  50. 50.
    Yin, C., Xiang, J.Y., Han, J.D.: Small target detection based on mean background model in IR images. Infrared Technology. (2004)Google Scholar
  51. 51.
    Yong, S.C., Yong, H.T.: Abnormal event detection in videos using spatiotemporal autoencoder. International Symposium on Neural Networks. 189–196 (2017).  https://doi.org/10.1007/978-3-319-59081-3_23
  52. 52.
    Yu, B., Liu, Y., Sun, Q.: A content-adaptively sparse reconstruction method for abnormal events detection with low-rank property. IEEE Transactions on Systems Man and Cybernetics Systems. 99, 1–13 (2016).  https://doi.org/10.1109/TSMC.2016.2638048 Google Scholar
  53. 53.
    Yu, Y., Shen, W., Huang, H., Zhang, Z.: Abnormal event detection in crowded scenes using two sparse dictionaries with saliency. Journal of Electronic Imaging. 26(3), 033013 (2017).  https://doi.org/10.1117/1.JEI.26.3.033013 CrossRefGoogle Scholar
  54. 54.
    Zhang, D., Gaticaperez, D., Bengio, S., Mccowan, I.: Semi-supervised adapted HMMs for unusual event detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 1, 611–618 (2005).  https://doi.org/10.1109/CVPR.2005.316 Google Scholar
  55. 55.
    Zhang, R., Zhou, M., Gong, X., He, X., Qian, W., Qin, S., Zhou, A.: Detecting anomaly in data streams by fractal model. World Wide Web. 18(5), 1419–1441 (2015).  https://doi.org/10.1007/s11280-014-0296-y CrossRefGoogle Scholar
  56. 56.
    Zhang, Z., Liu, S., Zhang, Z.: Consistent sparse representation for abnormal event detection. IEICE Trans. Inf. Syst. E98.D(10), 1866–1870 (2015).  https://doi.org/10.1587/transinf.2015EDL8113 CrossRefGoogle Scholar
  57. 57.
    Zhong, C.: Xu, G.: movement pedestrian detection method combined with foreground subtraction and deep learning. Computer and digital. Engineering. 44(12), 2396–2399 (2016).  https://doi.org/10.3969/j.issn.1672-9722.2016.12.023 Google Scholar
  58. 58.
    Zhou, S., Shen, W., Zeng, D., Fang, M., Wei, Y., Zhang, Z.: Spatial-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Processing Image Communication. 47, 358–368 (2016).  https://doi.org/10.1016/j.image.2016.06.007 CrossRefGoogle Scholar
  59. 59.
    Zhu, J., Jiang, W., Liu, A., Liu, G., Zhao, L.: Effective and efficient trajectory outlier detection based on time-dependent popular route. World Wide Web. 20(1), 111–134 (2017).  https://doi.org/10.1007/s11280-016-0400-6 CrossRefGoogle Scholar
  60. 60.
    Zhu, Y., Zhang, X., Wang, R., Zheng, W., Zhu, Y.: Self-representation and PCA embedding for unsupervised feature selection. World Wide Web. 1, 1–14 (2017).  https://doi.org/10.1007/s11280-017-0497-2 Google Scholar
  61. 61.
    Zou, Y.H., Guo, C.S.: Video abnormal event detection based on HMM cascaded with LDA. In: Journal of Hangzhou Dianzi University (2013)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Beijing Key Lab of Intelligent Telecommunication Software and Multimedia, School of Computer ScienceBeijing University of Posts and TelecommunicationsBeijingChina

Personalised recommendations