Advertisement

AVID: Adversarial Visual Irregularity Detection

  • Mohammad SabokrouEmail author
  • Masoud Pourreza
  • Mohsen Fayyaz
  • Rahim Entezari
  • Mahmood Fathy
  • Jürgen Gall
  • Ehsan Adeli
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11366)

Abstract

Real-time detection of irregularities in visual data is very invaluable and useful in many prospective applications including surveillance, patient monitoring systems, etc. With the surge of deep learning methods in the recent years, researchers have tried a wide spectrum of methods for different applications. However, for the case of irregularity or anomaly detection in videos, training an end-to-end model is still an open challenge, since often irregularity is not well-defined and there are not enough irregular samples to use during training. In this paper, inspired by the success of generative adversarial networks (GANs) for training deep models in unsupervised or self-supervised settings, we propose an end-to-end deep network for detection and fine localization of irregularities in videos (and images). Our proposed architecture is composed of two networks, which are trained in competing with each other while collaborating to find the irregularity. One network works as a pixel-level irregularity \(\mathcal {I}\)npainter, and the other works as a patch-level \(\mathcal {D}\)etector. After an adversarial self-supervised training, in which \(\mathcal {I}\) tries to fool \(\mathcal {D}\) into accepting its inpainted output as regular (normal), the two networks collaborate to detect and fine-segment the irregularity in any given testing video. Our results on three different datasets show that our method can outperform the state-of-the-art and fine-segment the irregularity.

Notes

Acknowledgements

This research was in part supported by a grant from IPM (No. CS1396-5-01). Mohsen Fayyaz and Juergen Gall have been financially supported by the DFG project GA 1927/4-1 (Research Unit FOR 2535) and the ERC Starting Grant ARCA (677650).

References

  1. 1.
    Boiman, O., Irani, M.: Detecting irregularities in images and in video. Int. J. Comput. Vis. 74, 17–31 (2007)CrossRefGoogle Scholar
  2. 2.
    Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1975–1981. IEEE (2010)Google Scholar
  3. 3.
    Bertini, M., Del Bimbo, A., Seidenari, L.: Multi-scale and real-time non-parametric approach for anomaly detection and localization. Comput. Vis. Image Underst. 116, 320–329 (2012)CrossRefGoogle Scholar
  4. 4.
    Colque, R.V.H.M., Caetano, C., de Andrade, M.T.L., Schwartz, W.R.: Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans. Circ. Syst. Video Technol. 27, 673–682 (2017)CrossRefGoogle Scholar
  5. 5.
    Xia, Y., Cao, X., Wen, F., Hua, G., Sun, J.: Learning discriminative reconstructions for unsupervised outlier removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1511–1519 (2015)Google Scholar
  6. 6.
    Morris, B.T., Trivedi, M.M.: Trajectory learning for activity understanding: unsupervised, multilevel, and long-term adaptive approach. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2287–2301 (2011)CrossRefGoogle Scholar
  7. 7.
    Sabokrou, M., Fathy, M., Hoseini, M.: Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron. Lett. 52, 1122–1124 (2016)CrossRefGoogle Scholar
  8. 8.
    You, C., Robinson, D.P., Vidal, R.: Provable self-representation based outlier detection in a union of subspaces. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  9. 9.
    Sabokrou, M., Fayyaz, M., Fathy, M., Klette, R.: Deep-cascade: cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. 26, 1992–2004 (2017)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Sabokrou, M., Fayyaz, M., Fathy, M., Moayedd, Z., et al.: Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Underst. 172, 88–97 (2018)CrossRefGoogle Scholar
  11. 11.
    Lawson, W., Bekele, E., Sullivan, K.: Finding anomalies with generative adversarial networks for a patrolbot. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 12–13 (2017)Google Scholar
  12. 12.
    Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection. In: CVPR (2018)Google Scholar
  13. 13.
    Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 146–157. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-59050-9_12CrossRefGoogle Scholar
  14. 14.
    Ravanbakhsh, M., Sangineto, E., Nabi, M., Sebe, N.: Training adversarial discriminators for cross-channel abnormal event detection in crowds. arXiv preprint arXiv:1706.07680 (2017)
  15. 15.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  16. 16.
    Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., Sebe, N.: Abnormal event detection in videos using generative adversarial nets. arXiv preprint arXiv:1708.09644 (2017)
  17. 17.
    Odena, A.: Semi-supervised learning with generative adversarial networks. In: Data Efficient Machine Learning workshop, ICML (2016)Google Scholar
  18. 18.
    Do-Omri, A., Wu, D., Liu, X.: A self-training method for semi-supervised GANs. In: ICLR (2018)Google Scholar
  19. 19.
    Piciarelli, C., Foresti, G.L.: On-line trajectory clustering for anomalous events detection. Pattern Recogn. Lett. 27, 1835–1842 (2006)CrossRefGoogle Scholar
  20. 20.
    Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Trans. Pattern Anal. Mach. Intell. 30, 555–560 (2008)CrossRefGoogle Scholar
  21. 21.
    Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Tran. Inf. Forensics Secur. 8, 1590–1599 (2013)CrossRefGoogle Scholar
  22. 22.
    Benezeth, Y., Jodoin, P.M., Saligrama, V., Rosenberger, C.: Abnormal events detection based on spatio-temporal co-occurences. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 2458–2465. IEEE (2009)Google Scholar
  23. 23.
    Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 935–942. IEEE (2009)Google Scholar
  24. 24.
    Kratz, L., Nishino, K.: Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 1446–1453. IEEE (2009)Google Scholar
  25. 25.
    Kim, J., Grauman, K.: Observe locally, infer globally: a space-time MRF for detecting abnormal activities with incremental updates. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 2921–2928. IEEE (2009)Google Scholar
  26. 26.
    Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. In: BMVC (2015)Google Scholar
  27. 27.
    Sabokrou, M., Fathy, M., Hoseini, M., Klette, R.: Real-time anomaly detection and localization in crowded scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 56–62 (2015)Google Scholar
  28. 28.
    Feng, Y., Yuan, Y., Lu, X.: Learning deep event models for crowd anomaly detection. Neurocomputing 219, 548–556 (2017)CrossRefGoogle Scholar
  29. 29.
    Fang, Z., et al.: Abnormal event detection in crowded scenes based on deep learning. Multimed. Tools Appl. 75, 14617–14639 (2016)CrossRefGoogle Scholar
  30. 30.
    Cong, Y., Yuan, J., Liu, J.: Sparse reconstruction cost for abnormal event detection. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3449–3456. IEEE (2011)Google Scholar
  31. 31.
    Antić, B., Ommer, B.: Video parsing for abnormality detection. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2415–2422. IEEE (2011)Google Scholar
  32. 32.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  33. 33.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)Google Scholar
  34. 34.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  35. 35.
    Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection-a new baseline. arXiv preprint arXiv:1712.09867 (2017)
  36. 36.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  37. 37.
    Nie, D., Wang, L., Adeli, E., Lao, C., Lin, W., Shen, D.: 3-D fully convolutional networks for multimodal isointense infant brain image segmentation. IEEE Trans. Cybern. (2018)Google Scholar
  38. 38.
    LeCun, Y., Cortes, C., Burges, C.J.: MNIST handwritten digit database. AT&T Labs, vol. 2 (2010). http://yann.lecun.com/exdb/mnist
  39. 39.
    Divakar, N., Babu, R.V.: Image denoising via CNNs: an adversarial approach. In: New Trends in Image Restoration and Enhancement, CVPR Workshop (2017)Google Scholar
  40. 40.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  41. 41.
    Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 221–231 (2013)CrossRefGoogle Scholar
  42. 42.
    Paszke, A., et al.: Automatic differentiation in pytorch (2017)Google Scholar
  43. 43.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  44. 44.
    Xu, D., Song, R., Wu, X., Li, N., Feng, W., Qian, H.: Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts. Neurocomputing 143, 144–152 (2014)CrossRefGoogle Scholar
  45. 45.
    Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 36, 18–32 (2014)CrossRefGoogle Scholar
  46. 46.
    Wu, S., Oreifej, O., Shah, M.: Action recognition in videos acquired by a moving camera using motion decomposition of Lagrangian particle trajectories. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1419–1426. IEEE (2011)Google Scholar
  47. 47.
    Saligrama, V., Chen, Z.: Video anomaly detection based on local statistical aggregates. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2112–2119. IEEE (2012)Google Scholar
  48. 48.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)Google Scholar
  49. 49.
    Liu, S., Bousquet, O., Chaudhuri, K.: Approximation and convergence properties of generative adversarial learning. In: Advances in Neural Information Processing Systems, pp. 5551–5559 (2017)Google Scholar
  50. 50.
    Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. In: International Conference on Learning Representations (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mohammad Sabokrou
    • 1
    Email author
  • Masoud Pourreza
    • 2
  • Mohsen Fayyaz
    • 3
  • Rahim Entezari
    • 4
  • Mahmood Fathy
    • 1
  • Jürgen Gall
    • 3
  • Ehsan Adeli
    • 5
  1. 1.Institute for Research in Fundamental Sciences (IPM)TehranIran
  2. 2.AI & ML Center of PartTehranIran
  3. 3.University of BonnBonnGermany
  4. 4.Complexity Science Hub, ViennaViennaAustria
  5. 5.Stanford UniversityStanfordUSA

Personalised recommendations