Knowledge and Information Systems

, Volume 32, Issue 2, pp 243–279 | Cite as

Real-time spatio-temporal analysis of dynamic scenes

Regular Paper


We propose a set of tools for spatio-temporal real-time analysis of dynamic scenes. It is designed to improve the grounding situation of autonomous agents in (simulated) physical domains. We introduce a knowledge processing pipeline ranging from relevance-driven compilation of a qualitative scene description to a knowledge-based detection of complex event and action sequences, conceived as a spatio-temporal pattern-matching problem. A methodology for the formalization of motion patterns and their inner composition is introduced and applied to capture human expertise about domain-specific motion situations. We present extensive experimental results from a challenging environment: 3D soccer simulation. It substantiates real-time applicability of our approach under tournament conditions, based on a 5-Hz (a) precise and (b) noisy/incomplete perception. The approach is not limited to robot soccer. Instead, it can also be applied in other fields such as experimental biology and logistic processes.


Analysis of dynamic scenes Spatio-temporal pattern matching Qualitative knowledge 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal CC (2009) On classification and segmentation of massive audio data streams. Knowl Inf Syst 20(2): 137–156MathSciNetCrossRefGoogle Scholar
  2. 2.
    Allen J (1983) Maintaining knowledge about temporal intervals. Commun ACM 26(11): 832–843CrossRefMATHGoogle Scholar
  3. 3.
    Allen JF (1984) Towards a general theory of action and time. Artif Intell 23(2): 123–154CrossRefMATHGoogle Scholar
  4. 4.
    Allen JF, Hayes PJ (1990) Moments and points in an interval-based temporal logic. Comput Intell 5(4): 225–238Google Scholar
  5. 5.
    Allen J, Ferguson G (1994) Actions and events in interval temporal logic. J Logic Comput 4(5): 531–579MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Baker C, Dolan J (2008) Traffic interaction in the urban challenge: putting Boss on its best behavior. In: International conference on intelligent robots and systems (IROS 2008), pp 1752–1758Google Scholar
  7. 7.
    Balch T, Dellaert F, Feldman A et al (2006) How multirobot systems research will accelerate our understanding of social animal behavior. Proc IEEE 94(7): 1445–1463CrossRefGoogle Scholar
  8. 8.
    Beetz M, Kirchlechner B, Lames M (2005) Computerized real-time analysis of football games. IEEE Perv Comput 4(3): 33–39CrossRefGoogle Scholar
  9. 9.
    Beetz M, Hoyningen-Huene N, Bandouch J et al (2006) Camera-based Observation of football games for analyzing multi-agent activities. In: ‘AAMAS ’06: proceedings of the fifth international joint conference on autonomous agents and multiagent systems. ACM, New York, NY, USA, pp 42–49Google Scholar
  10. 10.
    Bertrand O, Carle P, Choppy C (2009) Modelling chronicle recognition for distributed simulation processing with coloured petri nets. In: ‘Simutools ’09: proceedings of the 2nd international conference on simulation tools and techniques’, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering). ICST, Brussels, Belgium, Belgium, pp 1–2Google Scholar
  11. 11.
    Choppy C, Bertrand O, Carle P (2009) Coloured petri nets for chronicle recognition. In: Ada-Europe ’09: proceedings of the 14th Ada-Europe international conference on reliable software technologies. Springer, Berlin, pp 266–281Google Scholar
  12. 12.
    Divakaran A, Peker K, Radharkishnan R et al (2003) Video summarization using MPEG- 7 motion activity and audio descriptors. Video Mining, p 91Google Scholar
  13. 13.
    Dousson C, Le Maigat P (2007) Chronicle recognition improvement using temporal focusing and hierarchization. In: IJCAI’07: proceedings of the 20th international joint conference on artifical intelligence, pp 324–329. Morgan Kaufmann, San FranciscoGoogle Scholar
  14. 14.
    Feldman A, Balch T (2004) Representing honey bee behavior for recognition using human trainable models. Adapt Behav 12(3–4): 241–250CrossRefGoogle Scholar
  15. 15.
    Ferman AM, Tekalp AM (2002) Efficient filtering and clustering methods for temporal video segmentation and visual summarization. J Vis Commun Image Represent 9(4): 336–351CrossRefGoogle Scholar
  16. 16.
    Fox D, Thrun S, Burgard W et al (2001) Particle filters for mobile robot localization. In: Doucet A, Freitas N, Gordon N (eds) Sequential Monte Carlo methods in practice. Springer, New York, pp 470–498Google Scholar
  17. 17.
    Fraser G, Steinbauer G et al (2004) Application of qualitative reasoning to robotic soccer. In: 18th International workshop on qualitative reasoning, USAGoogle Scholar
  18. 18.
    Freksa C (1992) Temporal reasoning based on semi-intervals. In: Artificial intelligence, vol 54, pp 199–227Google Scholar
  19. 19.
    Gehrke JD (2005) Qualitative Szenenrepräsentation für intelligente Fahrzeuge. Master’s thesis, Universität BremenGoogle Scholar
  20. 20.
    Ghallab M (1996) On chronicles: representation, on-line recognition and learning. In: KR, pp 597–606Google Scholar
  21. 21.
    Gottfried B (2007) Representing short-term observations of moving objects by a simple visual language. J Vis Lang Comput 19(3): 321–342CrossRefGoogle Scholar
  22. 22.
    Gregorčič G, Lightbody G (2009) Gaussian process approach for modelling of nonlinear systems. Eng Appl Artif Intell 22(4–5): 522–533Google Scholar
  23. 23.
    Heintz F, Doherty P (2004) DyKnow: a framework for processing dynamic knowledge and object structures in autonomous systems. In: Proceedings of the international workshop on monitoring, security, and rescue techniques in multiagent systems (MSRAS)Google Scholar
  24. 24.
    Herzog G (1995) Utilizing interval-based event representations for incremental high level scene analysis. Technical report 91, Universität des SaarlandesGoogle Scholar
  25. 25.
    Intille S, Bobick A (2001) Recognizing planned, multiperson action. Comput Vis Image Understand 81(3): 414–445CrossRefMATHGoogle Scholar
  26. 26.
    Jeannin S, Divakaran A (2001) MPEG-7 visual motion descriptors. IEEE Trans Circuits Syst Video Tech 11(6): 720–724CrossRefGoogle Scholar
  27. 27.
    Jenkins OC, Matarić MJ (2004) A spatio-temporal extension to isomap nonlinear dimension reduction. In: The international conference on machine learning (ICML 2004), Banff, AB, Canada, pp 441–448Google Scholar
  28. 28.
    Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of 18th international conference on machine learning, Morgan Kaufmann, San Francisco, CA, pp 282–289Google Scholar
  29. 29.
    Lattner AD (2007) Temporal pattern mining in dynamic environments. Dissertation, Universität Bremen, number 309 in ‘DISKI’, Akademische Verlagsgesellschaft Aka GmbH, Berlin, GermanyGoogle Scholar
  30. 30.
    Lattner AD, Visser U, Warden T et al (2006) Virtual Werder 3D team documentation. Technical report 36, TZI - Center for Computing Technologies, Universität BremenGoogle Scholar
  31. 31.
    Miene A (2004) Räumlich-zeitliche analyse von dynamischen Szenen. Dissertation, Universität Bremen, number 279 in ‘DISKI’, Akademische Verlagsgesellschaft Aka GmbH, Berlin, GermanyGoogle Scholar
  32. 32.
    Miene A, Visser U, Herzog O (2004) Recognition and prediction of motion situations based on a qualitative motion description. In: Polani D, Browning B, Bonarini A, Yoshida K (eds) RoboCup 2003: robot soccer world cup VII, vol 3020 of LNCS. Springer, Berlin, pp 77–88CrossRefGoogle Scholar
  33. 33.
    Mohanty S (2009) Gaussian process time series model for life prognosis of metallic structures. J Intell Mater Syst Struct 20(8): 887–896CrossRefGoogle Scholar
  34. 34.
    Müller UT (2002) Beschreiben und Erkennen von Verhaltensmustern beim simulierten Fußballspiel. Master’s thesis, Humboldt-Universität zu Berlin, Berlin, GermanyGoogle Scholar
  35. 35.
    Musto A (2000) Qualitative Repräsentation von Bewegungsverläufen. PhD thesis, Technische Universität MünchenGoogle Scholar
  36. 36.
    Nair R, Milind T et al (2004) Automated assistants for analyzing team behaviors. J Autonom Agents Multi-Agent Syst 8(1): 69–111CrossRefGoogle Scholar
  37. 37.
    Neumann B, Möller R (2006) On scene interpretation with description logics. In: Christensen H, Nagel H-H (eds) Cognitive vision systems: sampling the spectrum of approaches, number 3948. Lect Notes Comput Sci, Springer, Berlin, pp 247–248Google Scholar
  38. 38.
    Obeid N, Rao R (2010) On integrating event definition and event detection. Knowl Inf Syst 22: 129–158CrossRefGoogle Scholar
  39. 39.
    Papapetrou P, Kollios G, Sclaroff S, Gunopulos D (2009) Mining frequent arrangements of temporal intervals. Knowl Inf Syst 21(2): 133–171CrossRefGoogle Scholar
  40. 40.
    Pokrajac D, Hoskinson RL, Obradovic Z (2003) Modeling spatial-temporal data with a short observation history. Knowl Inf Syst 5(3): 368–386CrossRefGoogle Scholar
  41. 41.
    Rabiner LR (1990) A tutorial on hidden Markov models and selected applications in speech recognition, pp 267–296Google Scholar
  42. 42.
    Rasmussen CE, Williams C (2006) Gaussian processes for machine learning. MIT Press, CambridgeMATHGoogle Scholar
  43. 43.
    Stanton C, Williams M-A et al (2004) Grounding robot sensory and symbolic information using the semantic web. In: Polani D (eds) RoboCup 2003: robot soccer world cup VII. Springer, Berlin, pp 757–764CrossRefGoogle Scholar
  44. 44.
    Steinbauer G, Weber J, Wotawa F (2005) From real-world to its qualitative representation—practical lessons learned. In: 18th International workshop on qualitative reasoning, Graz, pp 186–191Google Scholar
  45. 45.
    Thrun S, Montemerlo M, Becker J et al (2008) Junior: the stanford entry in the urban challenge. J Field Robot 25(9): 569–597CrossRefGoogle Scholar
  46. 46.
    Vail DL, Veloso MM, Lafferty JD (2007) Conditional random fields for activity recognition. In: AAMAS ’07: proceedings of the 6th international joint conference on autonomous agents and multiagent systems. ACM, New York, NY, USA, pp 1–8Google Scholar
  47. 47.
    Visser U, Burkhard H-D (2007) RoboCup: 10 years of achievements and future challenges. AI Mag 28(2): 115–132Google Scholar
  48. 48.
    Warden T (2007) Spatio-temporal real-time analysis of dynamic scenes in the RoboCup 3D soccer simulation league. Master’s thesis, Universität BremenGoogle Scholar
  49. 49.
    Wendler J (2003) Automatisches Modellieren von Agenten-Verhalten: Erkennen, Verstehen und Vorhersagen von Verhalten in komplexen Multi-Agenten-Systemen. PhD thesis, Humboldt-Universität zu BerlinGoogle Scholar
  50. 50.
    Wendler J, Bach J et al (2004) Recognizing and Predicting agent behavior with case based reasoning. In: Polani D (eds) RoboCup 2003: Robot Soccer World Cup VII’, LNAI. Springer, Berlin, pp 729–738CrossRefGoogle Scholar
  51. 51.
    Zhou G., Dong N, Wang Y (2009) Non-linear dynamic texture analysis and synthesis using constrained Gaussian process latent variable model. Pacific-Asia conference on circuits, communications and systems, pp 27–30Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.Center for Computing and Communication TechnologiesUniversity of BremenBremenGermany
  2. 2.Department of Computer ScienceUniversity of MiamiCoral GablesUSA

Personalised recommendations