International Journal of Computer Vision

, Volume 122, Issue 3, pp 502–523 | Cite as

Spatiotemporal Deformable Prototypes for Motion Anomaly Detection

  • Robert BenschEmail author
  • Nico Scherf
  • Jan Huisken
  • Thomas Brox
  • Olaf Ronneberger


This paper presents an approach for motion-based anomaly detection, where a prototype pattern is detected and elastically registered against a test sample to detect anomalies in the test sample. The prototype model is learned from multiple sequences to define accepted variations. “Supertrajectories” based on hierarchical clustering of dense point trajectories serve as an efficient and robust representation of motion patterns. An efficient hashing approach provides transformation hypotheses that are refined by a spatiotemporal elastic registration. We propose a new method for elastic registration of 3D+time trajectory patterns that induces spatial elasticity from trajectory affinities. The method is evaluated on a new motion anomaly dataset of juggling patterns and performs well in detecting subtle anomalies. Moreover, we demonstrate the applicability to biological motion patterns.


Anomaly detection Motion patterns Point trajectories Elastic registration 



We thank J. Koch, A. Krämer, T. Paxian and D. Mai who contributed their juggling expertise and agreed to perform diverse juggling patterns in front of our Kinect camera. This study was supported by the Excellence Initiative of the German Federal and State Governments (BIOSS Centre for Biological Signalling Studies EXC 294 to R.B., T.B. and O.R.). N.S. and J.H. have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant agreement No. 647885).

Supplementary material

Supplementary material 1 (m4v 22775 KB)


  1. Adam, A., Rivlin, E., Shimshoni, I., & Reinitz, D. (2008). Robust real-time unusual event detection using multiple fixed-location monitors. IEEE Transactions Pattern Analysis and Machine Intelligence, 30(3), 555–560.CrossRefGoogle Scholar
  2. Antic, B. & Ommer, B. (2011). Video parsing for abnormality detection. In D. N. Metaxas, L. Quan, A. Sanfeliu, & L. J. V. Gool (Eds.), IEEE international conference on computer vision (ICCV) (pp. 2415–2422).Google Scholar
  3. Benezeth, Y., Jodoin, P. M., Saligrama, V., & Rosenberger, C. (2009). Abnormal events detection based on spatio-temporal co-occurences. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2458–2465).Google Scholar
  4. Bensch, R., Brox, T., & Ronneberger, O. (2015). Spatiotemporal deformable prototypes for motion anomaly detection. In M. W. J. Xianghua Xie & G. K. L. Tam (Eds.), British machine vision conference (BMVC) (pp. 189.1–189.12). London: BMVA Press.Google Scholar
  5. Besl, P., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions Pattern Analysis and Machine Intelligence, 14(2), 239–256.CrossRefGoogle Scholar
  6. Boiman, O. & Irani, M. (2007b). Similarity by composition. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems 19 (NIPS) (pp. 177–184). Cambridge, MA: MIT Press.Google Scholar
  7. Boiman, O., & Irani, M. (2007a). Detecting irregularities in images and in video. International Journal of Computer Vision, 74(1), 17–31.CrossRefGoogle Scholar
  8. Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(3), 500–513.CrossRefGoogle Scholar
  9. Byrd, R. H., Lu, P., Nocedal, J., & Zhu, C. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5), 1190–1208.MathSciNetCrossRefzbMATHGoogle Scholar
  10. Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 15:1–15:58.CrossRefGoogle Scholar
  11. Cong, Y., Yuan, J., & Liu, J. (2011). Sparse reconstruction cost for abnormal event detection. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 3449–3456).Google Scholar
  12. Cong, Y., Yuan, J., & Tang, Y. (2013). Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Transactions on Information Forensics and Security, 8(10), 1590–1599.CrossRefGoogle Scholar
  13. Dee, H. M. & Hogg, D. C. (2004). Detecting inexplicable behaviour. In British machine vision conference (BMVC) (pp. 477–486).Google Scholar
  14. Everitt, B., Landau, S., Leese, M., & Stahl, D. (2011). Cluster analysis. Wiley series in probability and statistics New York: Wiley.Google Scholar
  15. Hu, W., Xiao, X., Fu, Z., Xie, D., Tan, T., & Maybank, S. (2006). A system for learning statistical motion patterns. IEEE Transactions Pattern Analysis and Machine Intelligence, 28(9), 1450–1464.CrossRefGoogle Scholar
  16. Kim, J. & Grauman, K. (2009). Observe locally, infer globally: A space-time MRF for detecting abnormal activities with incremental updates. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2921–2928).Google Scholar
  17. Kratz, L. & Nishino, K. (2009). Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1446–1453).Google Scholar
  18. Li, C., Han, Z., Ye, Q., & Jiao, J. (2013). Visual abnormal behavior detection based on trajectory sparse reconstruction analysis. Neurocomputing, 119,94–100. Intelligent Processing Techniques for Semantic-based Image and Video Retrieval.Google Scholar
  19. Mahadevan, V., Li, W., Bhalodia, V., & Vasconcelos, N. (2010). Anomaly detection in crowded scenes. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1975–1981).Google Scholar
  20. Mehran, R., Oyama, A., & Shah, M. (2009). Abnormal crowd behavior detection using social force model. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 935–942).Google Scholar
  21. Nait-Charif, H. & McKenna, S. J. (2004). Activity summarisation and fall detection in a supportive home environment. In IEEE international conference on pattern recognition (ICPR) (Vol. 4, pp. 323–326).Google Scholar
  22. Piciarelli, C., Micheloni, C., & Foresti, G. L. (2008). Trajectory-based anomalous event detection. IEEE Transactions on Circuits and Systems for Video Technology, 18(11), 1544–1554.CrossRefGoogle Scholar
  23. Popoola, O. P., & Wang, K. (2012). Video-based abnormal human behavior recognition—A review. IEEE Transactions on Circuits and Systems for Video Technology, 42(6), 865–878.Google Scholar
  24. Roshtkhari, M. J., & Levine, M. D. (2013). An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Computer Vision and Image Understanding, 117(10), 1436–1452.CrossRefGoogle Scholar
  25. Saligrama, V. & Chen, Z. (2012). Video anomaly detection based on local statistical aggregates. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2112–2119).Google Scholar
  26. Saligrama, V., Konrad, J., & Jodoin, P. M. (2010). Video anomaly identification. IEEE Signal Processing Magazine, 27(5), 18–33.CrossRefGoogle Scholar
  27. Schmid, B., Shah, G., Scherf, N., Weber, M., Thierbach, K., Campos, C. P., Roeder, I., Aanstad, P., & Huisken, J. (2013). High-speed panoramic light-sheet microscopy reveals global endodermal cell dynamics. Nature Communications, 4. doi: 10.1038/ncomms3207.
  28. Sillito, R. R. & Fisher, R. B. (2008). Semi-supervised learning for anomalous trajectory detection. In British machine vision conference (BMVC) (pp. 1035–1044).Google Scholar
  29. Sundaram, N., Brox, T., & Keutzer, K. (2010). Dense point trajectories by GPU-accelerated large displacement optical flow. In European conference on computer vision (ECCV). Lecture notes in computer science (pp. 438–451). New York: Springer.Google Scholar
  30. Umeyama, S. (1991). Least-squares estimation of transformation parameters between two point patterns. IEEE Signal Processing Magazine, 13(4), 376–380.Google Scholar
  31. Winkelbach, S., Molkenstruck, S., & Wahl, F. M. (2006). Low-cost laser range scanner and fast surface registration approach. In Pattern Recognition (Proc. DAGM). Lecture notes in computer science (pp. 718–728). New York: Springer.Google Scholar
  32. Wu, S., Moore, B. E., & Shah, M. (2010). Chaotic invariants of Lagrangian particle trajectories for anomaly detection in crowded scenes. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2054–2060).Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of FreiburgFreiburgGermany
  2. 2.BIOSS Centre for Biological Signalling StudiesUniversity of FreiburgFreiburgGermany
  3. 3.Max Planck Institute of Molecular Cell Biology and GeneticsDresdenGermany
  4. 4.Faculty of Medicine Carl Gustav Carus, Institute for Medical Informatics and BiometryTU DresdenDresdenGermany

Personalised recommendations