Deep Semantic Abstractions of Everyday Human Activities

On Commonsense Representations of Human Interactions
  • Jakob SuchanEmail author
  • Mehul Bhatt
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 693)


We propose a deep semantic characterisation of space and motion categorically from the viewpoint of grounding embodied human-object interactions. Our key focus is on an ontological model that would be adept to formalisation from the viewpoint of commonsense knowledge representation, relational learning, and qualitative reasoning about space and motion in cognitive robotics settings. We demonstrate key aspects of the space & motion ontology and its formalisation as a representational framework in the backdrop of select examples from a dataset of everyday activities. Furthermore, focussing on human-object interaction data obtained from RGBD sensors, we also illustrate how declarative (spatio-temporal) reasoning in the (constraint) logic programming family may be performed with the developed deep semantic abstractions.



We acknowledge funding by the Germany Research Foundation (DFG) via the Collaborative Research Center (CRC) EASE – Everyday Activity Science and Engineering ( This paper builds on, and is a condensed version of, a workshop contribution [31] at the ICCV 2017 conference. We also acknowledge the support of Vijayanta Jain in preparation of part of the overall activity dataset; toward this, the help of Omar Moussa, Hubert Kloskoski, Thomas Hudkovic, Vyyom Kelkar as subjects is acknowledged.


  1. 1.
    Aiello, M., Pratt-Hartmann, I.E., van Benthem, J.F.: Handbook of Spatial Logics. Springer, New York (2007)CrossRefzbMATHGoogle Scholar
  2. 2.
    Bartels, G., Kresse, I., Beetz, M.: Constraint-based movement representation grounded in geometric features. In: Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Atlanta, Georgia, USA, 15–17 October 2013Google Scholar
  3. 3.
    Bennett, B., Cohn, A.G., Torrini, P., Hazarika, S.M.: Describing rigid body motions in a qualitative theory of spatial regions. In: Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on on Innovative Applications of Artificial Intelligence, pp. 503–509 (2000)Google Scholar
  4. 4.
    Bennett, B., Cohn, A.G., Torrini, P., Hazarika, S.M.: A foundation for region-based qualitative geometry. In: Proceedings of the 14th European Conference on Artificial Intelligence, pp. 204–208 (2000)Google Scholar
  5. 5.
    Bhatt, M., Guesgen, H., Wölfl, S., Hazarika, S.: Qualitative spatial and temporal reasoning: Emerging applications, trends, and directions. Spat. Cogn. Comput. 11(1), 1–14 (2011)Google Scholar
  6. 6.
    Bhatt, M.: Reasoning about space, actions and change: A paradigm for applications of spatial reasoning. In: Qualitative Spatial Representation and Reasoning: Trends and Future Directions. IGI Global, USA (2012)Google Scholar
  7. 7.
    Bhatt, M.: Between sense and sensibility: Declarative narrativisation of mental models as a basis and benchmark for visuo-spatial cognition and computation focussed collaborative cognitive systems (2013). CoRR abs/1307.3040
  8. 8.
    Bhatt, M., Dylla, F., Hois, J.: Spatio-terminological inference for the design of ambient environments. In: Proceedings of the 9th International Conference on Spatial Information Theory, COSIT 2009. Lecture Notes in Computer Science, Aber Wrac’h, France, 21–25 September 2009, vol. 5756, pp. 371–391. Springer (2009)Google Scholar
  9. 9.
    Bhatt, M., Kersting, K.: Semantic Interpretation of Multimodal Human Behaviour Data: Making Sense of Events, Activities. Processes, KI - Künstliche Intelligenz/Artificial Intelligence (2017)Google Scholar
  10. 10.
    Bhatt, M., Lee, J.H., Schultz, C.P.L.: CLP(QS): A declarative spatial reasoning framework. In: Proceedings of the 10th International Conference on Spatial Information Theory, COSIT 2011, Belfast, ME, USA, 12–16 September 2011, pp. 210–230 (2011)Google Scholar
  11. 11.
    Bhatt, M., Loke, S.: Modelling dynamic spatial systems in the situation calculus. Spat. Cogn. Comput. 8(1–2), 86–130 (2008). Google Scholar
  12. 12.
    Bhatt, M., Schultz, C., Freksa, C.: The ‘Space’ in spatial assistance systems: conception, formalisation and computation. In: Tenbrink, T., Wiener, J., Claramunt, C. (eds.) Representing Space in Cognition: Interrelations of Behavior, Language, and Formal Models. Explorations in Language and Space. Oxford University Press (2013). ISBN 978-0-19-967991-1Google Scholar
  13. 13.
    Cohn, A.G., Renz, J.: Qualitative spatial reasoning. In: van Harmelen, F., Lifschitz, V., Porter, B. (eds.) Handbook of Knowledge Representation. Elsevier (2007)Google Scholar
  14. 14.
    Davis, E.: Pouring liquids: A study in commonsense physical reasoning. Artif. Intell. 172(12–13), 1540–1578 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Davis, E.: How does a box work? a study in the qualitative dynamics of solid objects. Artif. Intell. 175(1), 299–345 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Davis, E.: Qualitative spatial reasoning in interpreting text and narrative. Spat. Cogn. Comput. 13(4), 264–294 (2013)Google Scholar
  17. 17.
    Dubba, K.S.R., Bhatt, M., Dylla, F., Hogg, D.C., Cohn, A.G.: Interleaved inductive-abductive reasoning for learning complex event models. In: Muggleton, S., Tamaddoni-Nezhad, A., Lisi, F.A. (eds.) 21st International Conference on Inductive Logic Programming, ILP 2011, Revised Selected Papers. Lecture Notes in Computer Science, Windsor Great Park, UK, 31 July–3 August 2011, vol. 7207, pp. 113–129. Springer (2011)Google Scholar
  18. 18.
    Dubba, K.S.R., Cohn, A.G., Hogg, D.C., Bhatt, M., Dylla, F.: Learning relational event models from video. J. Artif. Intell. Res. (JAIR) 53, 41–90 (2015)zbMATHMathSciNetGoogle Scholar
  19. 19.
    Eppe, M., Bhatt, M.: Narrative based postdictive reasoning for cognitive robotics. In: 11th International Symposium on Logical Formalizations of Commonsense Reasoning, COMMONSENSE 2013 (2013)Google Scholar
  20. 20.
    Eschenbach, C., Schill, K.: Studying spatial cognition - a report on the DFG workshop on “The representation of motion”. KI 13(3), 57–58 (1999)Google Scholar
  21. 21.
    Guesgen, H.W.: Spatial reasoning based on Allen’s temporal logic. Technical Report TR-89-049, International Computer Science Institute Berkeley (1989)Google Scholar
  22. 22.
    Hayes, P.J.: Naive physics I: ontology for liquids. In: Hubbs, J.R., Moore, R.C. (eds.) Formal Theories of the Commonsense World. Ablex Publishing Corporation, Norwood (1985)Google Scholar
  23. 23.
    Hazarika, S.M.: Qualitative Spatial Change: Space-Time Histories and Continuity. Ph.D. thesis, The University of Leeds, School of Computing, supervisor - Anthony Cohn (2005)Google Scholar
  24. 24.
    Hernández, D., Clementini, E., Di Felice, P.: Qualitative distances. Springer (1995)Google Scholar
  25. 25.
    Moratz, R.: Representing relative direction as a binary relation of oriented points. In: ECAI, pp. 407–411 (2006)Google Scholar
  26. 26.
    Randell, D.A., Cui, Z., Cohn, A.G.: A spatial logic based on regions and connection. In: KR 1992, pp. 165–176 (1992)Google Scholar
  27. 27.
    Renz, J., Nebel, B.: Qualitative spatial reasoning using constraint calculi. In: Handbook of Spatial Logics, pp. 161–215 (2007)Google Scholar
  28. 28.
    Scivos, A., Nebel, B.: The finest of its class: the natural, point-based ternary calculus LR for qualitative spatial reasoning. In: Freksa, C., et al. International Conference Spatial Cognition on Spatial Cognition IV. Reasoning, Action, Interaction. Lecture Notes in Computer Science, Vol. 3343, pp. 283–303. Springer, Heidelberg (2004)Google Scholar
  29. 29.
    Spranger, M., Suchan, J., Bhatt, M.: Robust natural language processing - combining reasoning, cognitive semantics and construction grammar for spatial language. In: IJCAI 2016: 25th International Joint Conference on Artificial Intelligence. AAAI Press (2016)Google Scholar
  30. 30.
    Spranger, M., Suchan, J., Bhatt, M., Eppe, M.: Grounding dynamic spatial relations for embodied (robot) interaction. In: Proceedings of the PRICAI 2014: Trends in Artificial Intelligence, Gold Coast, QLD, Australia, 1–5 December 2014, pp. 958–971 (2014).
  31. 31.
    Suchan, J., Bhatt, M.: Commonsense scene semantics for cognitive robotics: towards grounding embodied visuo-locomotive interactions. In: ICCV 2017 Workshop: Vision in Practice on Autonomous Robots (ViPAR), International Conference on Computer Vision (ICCV) (2017)Google Scholar
  32. 32.
    Suchan, J., Bhatt, M., Santos, P.E.: Perceptual narratives of space and motion for semantic interpretation of visual data. In: Proceedings of the Computer Vision - ECCV 2014 Workshops - Zurich, Switzerland, 6–7 and 12 September 2014, Part II, pp. 339–354 (2014)Google Scholar
  33. 33.
    Tellex, S.: Natural language and spatial reasoning. Ph.D. thesis, Massachusetts Institute of Technology (2010)Google Scholar
  34. 34.
    Tyler, A., Evans, V.: The Semantics of English Prepositions: Spatial Scenes, Embodied Meaning and Cognition. Cambridge University Press, Cambridge (2003)CrossRefGoogle Scholar
  35. 35.
    Walega, P.A., Bhatt, M., Schultz, C.P.L.: ASPMT(QS): Non-monotonic spatial reasoning with answer set programming modulo theories. In: Proceedings of the 13th International Conference on Logic Programming and Nonmonotonic Reasoning, LPNMR 2015, Lexington, KY, USA, 27–30 September 2015, pp. 488–501 (2015)Google Scholar
  36. 36.
    Worgotter, F., Aksoy, E.E., Kruger, N., Piater, J., Ude, A., Tamosiunaite, M.: A simple ontology of manipulation actions based on hand-object relations (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Spatial Reasoning, EASE CRC: Everyday Activity Science and EngineeringUniversity of BremenBremenGermany
  2. 2.Machine Perception and Interaction Laboratory, Centre for Applied Autonomous Sensor Systems (AASS)Örebro UniversityÖrebroSweden

Personalised recommendations