Fast and accurate detection and localization of abnormal behavior in crowded scenes

  • Mohammad Sabokrou
  • Mahmood Fathy
  • Zahra Moayed
  • Reinhard Klette
Original Paper

Abstract

This paper presents a novel video processing method for accurate and fast anomaly detection and localization in crowded scenes. We propose a cubic-patch-based method based on a cascade of classifiers, which combines the results of two types of video descriptors. Based on the low likelihood of an anomaly occurrence and the redundancy of structures in normal patches of a video, we introduce two efficient feature sets for describing the spatial and temporal video context, which are named “local” and “global” descriptors. The local description is based on the relation of a patch with its neighbors, and the global description is provided by a sparse auto-encoder. Two reference models, using the local and global descriptions of the training normal patches, are learned as two one-class classifiers. For being fast and accurate, these two classifiers are combined as a cascaded classifier. First, the local classifier, which is faster than the global one, is used for early identification of “many” normal cubic patches. Then, the remaining patches are checked “carefully” by the global classifier. Also, we propose a technique for learning from small patches and inferring from larger patches; this leads to an improved performance. It is shown that the proposed method performs comparable to, or even better than top-performing detection and localization methods on standard benchmarks but with a substantial improvement in speed.

Keywords

Anomaly detection Unsupervised deep learning Cascade classifier Local and global descriptors 

References

  1. 1.
    Cong, Y., et al.: Sparse reconstruction cost for abnormal event detection. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 3449–3456 (2011)Google Scholar
  2. 2.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012)Google Scholar
  3. 3.
    Girshick, R., et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 580–587Google Scholar
  4. 4.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 568–576. Curran Associates, Inc. (2014)Google Scholar
  5. 5.
    Yang, Y., et al.: Semi-supervised learning of feature hierarchies for object detection in a video. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1650–1657 (2013)Google Scholar
  6. 6.
    Giusti, A., et al.: Fast image scanning with deep max-pooling convolutional neural networks. In: IEEE International Conference on Image Processing, pp. 4034–4038 (2013)Google Scholar
  7. 7.
    Bertini, M., Del Bimbo, A., Seidenari, L.: Multi-scale and real-time non-parametric approach for anomaly detection and localization. Comput. Vis. Image Underst. 116(3), 320–329 (2012)CrossRefGoogle Scholar
  8. 8.
    Sabokrou, M., et al.: Real-time anomaly detection and localization in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition Workshops, pp. 320–329 (2015)Google Scholar
  9. 9.
    Coates, A., et al.: An analysis of single-layer networks in unsupervised feature learning. In: Conference on Artificial Intelligence Statistics, pp. 215–223 (2011)Google Scholar
  10. 10.
    Statistical Visual Computing Lab, UC San Diego: UCSD Anomaly Detection Dataset. www.svcl.ucsd.edu/projects/anomaly/dataset.html. Accessed Feb 2013 (2013)
  11. 11.
    Artificial Intelligence, Robotics and Vision Laboratory University of Minnesota, Dep. Computer Science and Engineering: Monitoring Human Activity. http://mha.cs.umn.edu/Movies/Crowd-Activity-All.avimha.cs.umn.edu/Movies/Crowd-Activity-All.avi. Accessed Feb 2013 (2006)
  12. 12.
    Jianga, F., Yuan, J., Tsaftarisa, S.A., Katsaggelosa, A.K.: Anomalous video event detection using spatiotemporal context. Comput. Vis. Image Underst. 115(3), 323–333 (2011)CrossRefGoogle Scholar
  13. 13.
    Wu, S., et al.: Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2054–2060 (2010)Google Scholar
  14. 14.
    Piciarelli, C., Micheloni, C., Foresti, G.L.: Trajectory-based anomalous event detection. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1544–1554 (2008)CrossRefGoogle Scholar
  15. 15.
    Antonakaki, P., Kosmopoulos, D., Perantonis, S.J.: Detecting abnormal human behaviour using multiple cameras. Signal Process. 89(9), 1723–1738 (2009)CrossRefMATHGoogle Scholar
  16. 16.
    Piciarelli, C., Foresti, G.L.: On-line trajectory clustering for anomalous events detection. Pattern Recognit. Lett. 27(15), 1835–1842 (2006)CrossRefGoogle Scholar
  17. 17.
    Hu, W., Xiao, X., Fu, Z., et al.: A system for learning statistical motion patterns. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1450–1464 (2006)CrossRefGoogle Scholar
  18. 18.
    Calderara, S., Heinemann, U., Prati, A., et al.: Detecting anomalies in people’s trajectories using spectral graph analysis. Comput. Vis. Image Underst. 115(8), 1099–1111 (2011)CrossRefGoogle Scholar
  19. 19.
    Morris, B.T., Trivedi, M.M.: Trajectory learning for activity understanding: unsupervised, multilevel, and long-term adaptive approach. IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2287–2301 (2011)CrossRefGoogle Scholar
  20. 20.
    Tung, F., Zelek, J.S., Clausi, D.A.: Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis. Comput. 29(4), 230–240 (2011)CrossRefGoogle Scholar
  21. 21.
    Biswas, S., Rabu, v: Anomaly detection via short local trajectories. Neurocomputing 242, 63–72 (2017)CrossRefGoogle Scholar
  22. 22.
    Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: Robust real-time unusual event detection using multiple fixed location monitors. IEEE Trans. Pattern Anal. Mach. Intell. 30(3), 555–560 (2008)CrossRefGoogle Scholar
  23. 23.
    Boiman, O., Irani, M.: Detecting irregularities in images and in video. Int. J. Comput. Vis. 74(1), 17–31 (2007)CrossRefGoogle Scholar
  24. 24.
    Mahadevan, V., et al.: Anomaly detection in crowded scenes. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1975–1981 (2010)Google Scholar
  25. 25.
    Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 18–32 (2014)CrossRefGoogle Scholar
  26. 26.
    Kim, J., Grauman, K.: Observe locally, infer globally: a space–time MRF for detecting abnormal activities with incremental updates. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2921–2928 (2009)Google Scholar
  27. 27.
    Benezeth, Y., et al.: Abnormal events detection based on spatio-temporal co-occurrences. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1446–1453 (2009)Google Scholar
  28. 28.
    Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 1446–1453 (2009)Google Scholar
  29. 29.
    Mehran, R., et al.: Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 935–942 (2009)Google Scholar
  30. 30.
    Zaharescu, A., Wildes, R.: Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing. European Conference on Computer Vision, LNCS, pp. 563–576 (2010)Google Scholar
  31. 31.
    Saligrama, V., Chen, Z.: Video anomaly detection based on local statistical aggregates. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2112–2119 (2012)Google Scholar
  32. 32.
    Antić, B., Ommer, B.: Video parsing for abnormality detection. In: International Conference on Computer Vision, pp. 2415–2422 (2011)Google Scholar
  33. 33.
    Reddy, V., et al.: Improved anomaly detection in crowded scenes via cell-based analysis of foreground speed, size and texture. In: CVPR Workshops, pp. 55–61 (2011)Google Scholar
  34. 34.
    Ullah, H., Conci, N.: Crowd motion segmentation and anomaly detection via multi-label optimization. In: ICPR Workshop Pattern Recognition Crowd Analysis (2012)Google Scholar
  35. 35.
    Roshtkhari, M.J., Levine, M.D.: An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput. Vis. Image Underst. 117(10), 1436–1452 (2013)CrossRefGoogle Scholar
  36. 36.
    Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans. Inf. Forensics Secur. 8(10), 1590–1599 (2013)CrossRefGoogle Scholar
  37. 37.
    Zhu, Y., et al.: Context-aware modeling and recognition of activities in video. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2491–2498 (2013)Google Scholar
  38. 38.
    Roshtkhari, M.J., Levine, M.D.: Online dominant and anomalous behavior detection in videos. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2611–2618 (2013)Google Scholar
  39. 39.
    Nakahata, M.T., Thomaz, L.A., da Silva, A.F., da Silva, E.A.B., Netto, S.L.: Anomaly detection with a moving camera using spatio-temporal codebooks. Multidim. Syst. Signal Process. (2017). doi:10.1007/s11045-017-0486-8
  40. 40.
    Xu, D., Song, R., Wu, X., et al.: Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts. Neurocomputing 143, 144–152 (2014)CrossRefGoogle Scholar
  41. 41.
    Cheng, K.W., et al.: Video anomaly detection and localization using hierarchical feature representation and Gaussian process regression. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 2909–2917 (2015)Google Scholar
  42. 42.
    Ullah, H., Tenuti, L., Conci, N.: Gaussian mixtures for anomaly detection in crowded scenes. In: IS&T/SPIE Electronic Imaging, vol. 8663 (2013)Google Scholar
  43. 43.
    Ullah, H., Tenuti, L., Conci, N.: Real-time anomaly detection in dense crowded scenes. In: IS&T/SPIE Electronic Imaging, vol. 9026 (2014)Google Scholar
  44. 44.
    Ullah, H., Tenuti, L., Conci, N.: Dominant motion analysis in regular and irregular crowd scenes. In: ECCV Workshop Human Behaviour Understanding, pp. 62–72 (2014)Google Scholar
  45. 45.
    Mousavi, H., et al.: Analyzing tracklets for the detection of abnormal crowd behavior. In: IEEE Winter Conference on Applications Computer Vision, pp. 148–155 (2015)Google Scholar
  46. 46.
    Mousavi, H., et al.: Abnormality detection with improved histogram of oriented tracklets. In: International Conference on Image Analysis and Processing, vol. 9280, pp. 722–732 (2015)Google Scholar
  47. 47.
    Rabiee, H., et al.: Detection and localization of crowd behavior using a novel tracklet-based mode. Int. J. Mach. Learn. Cybern. 8(36), 1–12 (2017)Google Scholar
  48. 48.
    Yuan, Y., Fang, J., Wang, Q.: Online anomaly detection in crowd scenes via structure analysis. IEEE Trans. Cybern. 45(3), 562–575 (2015)CrossRefGoogle Scholar
  49. 49.
    Lee, D., Suk, H.I., Park, S.K., et al.: Motion influence map for unusual human activity detection and localization in crowded scenes. IEEE Trans. Circuits Syst. Video Technol. 25(10), 1612–1623 (2015)CrossRefGoogle Scholar
  50. 50.
    Zhang, D., et al.: Semi-supervised adapted HMMS for unusual event detection. IEEE Conf. Computer Vis. Pattern Recognit. 1, 611–618 (2005)Google Scholar
  51. 51.
    Lu, C., et al.: Abnormal event detection at 150 fps in MATLAB. In: IEEE Internatioanal Conference on Computer Vision, pp. 2720–2727 (2013)Google Scholar
  52. 52.
    Xiao, T., Zhang, C., Zha, H.: Learning to detect anomalies in surveillance video. IEEE Signal Process. Lett. 22(9), 1477–1481 (2015)CrossRefGoogle Scholar
  53. 53.
    Li, N., Wu, X., Xu, D., et al.: Spatio-temporal context analysis within video volumes for anomalous-event detection and localization. Neurocomputing 155, 309–319 (2015)CrossRefGoogle Scholar
  54. 54.
    Xu, D., et al.: Learning deep representations of appearance and motion for anomalous event detection. In: British Machine Vision Conference, pp. 8.1–8.12 (2015)Google Scholar
  55. 55.
    Sabokrou, M., et al.: Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron. Lett. 52, 1122–1124 (2016)CrossRefGoogle Scholar
  56. 56.
    Sabokrou, M., et al.: Deep-cascade: cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26(4), 1992–2004 (2017)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Vincent, P. et al.: Extracting and composing robust features with denoising autoencoders. In: ACM International Conference on Machine Learning, pp. 1096–1103 (2008)Google Scholar
  58. 58.
    Sabokrou, M., et al.: Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Arxiv e-prints arXiv:1609.00866 (2017)
  59. 59.
    Ravanbakhsh, M., et al.: Plug-and-play CNN for crowd motion analysis: an application in abnormal event detection. arXiv preprint arXiv:1610.00307 (2016)
  60. 60.
    Zhou, S., et al.: Spatio-temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process. Image Commun. 47, 358–368 (2016)CrossRefGoogle Scholar
  61. 61.
    Hu, X., et al.: Video anomaly detection using deep incremental slow feature analysis network. IET Comput. Vis. 10(4), 258–265 (2016)CrossRefGoogle Scholar
  62. 62.
    Fang, Z., et al.: Abnormal event detection in crowded scenes based on deep learning. Multimed. Tools Appl. 75, 14617–14639 (2016)CrossRefGoogle Scholar
  63. 63.
    Munwar, A., Vinayavekhin, P., Magistris, G.: Spatio-temporal anomaly detection for industrial robots through prediction in unsupervised feature space. In: IEEE Winter Conference on Applications Computer Vision, pp. 1017–1025 (2017)Google Scholar
  64. 64.
    Brunet, D., Vrscay, E.R., Wang, Z.: On the mathematical properties of the structural similarity index. IEEE Trans. Image Process. 21(4), 1488–1499 (2012)MathSciNetCrossRefGoogle Scholar
  65. 65.
    Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.School of Computer ScienceInstitute for Research in Fundamental Sciences (IPM)TehranIran
  2. 2.Iran University of Science and TechnologyNarmak, TehranIran
  3. 3.Auckland University of TechnologyAucklandNew Zealand

Personalised recommendations