Modelling Language, Action, and Perception in Type Theory with Records

  • Simon Dobnik
  • Robin Cooper
  • Staffan Larsson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8114)


Formal models of natural language semantics using TTR (Type Theory with Records) attempt to relate natural language to perception, modelled as classification of objects and events by types which are available as resources to an agent. We argue that this is better suited for representing the meaning of spatial descriptions in the context of agent modelling than traditional formal semantic models which do not relate spatial concepts to perceptual apparatus. Spatial descriptions include perceptual, conceptual and discourse knowledge which we represent all in a single framework. Being a general framework for modelling both linguistic and non-linguistic cognition, TTR is more suitable for the modelling of situated conversational agents in robotics and virtual environments where interoperability between language, action and perception is required. The perceptual systems gain access to abstract conceptual meaning representations of language while the latter can be justified in action and perception.


language action perception formal semantics spatial descriptions learning and classification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Blackburn, P., Bos, J.: Representation and inference for natural language. A first course in computational semantics. CSLI (2005)Google Scholar
  2. 2.
    Winograd, T.: Understanding Natural Language. Edinburgh University Press (1976)Google Scholar
  3. 3.
    Miller, G.A., Johnson-Laird, P.N.: Language and perception. Cambridge University Press, Cambridge (1976)Google Scholar
  4. 4.
    Gapp, K.P.: Basic meanings of spatial relations: Computation and evaluation in 3d space. In: Hayes-Roth, B., Korf, R.E. (eds.) AAAI, pp. 1393–1398. AAAI Press/The MIT Press (1994)Google Scholar
  5. 5.
    Logan, G.D., Sadler, D.D.: A computational analysis of the apprehension of spatial relations. In: Bloom, P., Peterson, M.A., Nadel, L., Garrett, M.F. (eds.) Language and Space, pp. 493–530. MIT Press, Cambridge (1996)Google Scholar
  6. 6.
    Siskind, J.M.: Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. Journal of Artificial Intelligence Research 15, 31–90 (2001)zbMATHGoogle Scholar
  7. 7.
    Roy, D.: Semiotic schemas: a framework for grounding language in action and perception. Artificial Intelligence 167, 170–205 (2005)CrossRefGoogle Scholar
  8. 8.
    Guerra-Filho, G., Aloimonos, Y.: A language for human action. Computer 40, 42–51 (2007)CrossRefGoogle Scholar
  9. 9.
    Matuszek, C., Herbst, E., Zettlemoyer, L., Fox, D.: Learning to parse natural language commands to a robot control system. In: Proc. of the 13th International Symposium on Experimental Robotics, ISER (2012)Google Scholar
  10. 10.
    Dobnik, S.: Teaching mobile robots to use spatial words. PhD thesis, University of Oxford: Faculty of Linguistics, Philology and Phonetics and The Queen’s College, Oxford, United Kingdom (2009)Google Scholar
  11. 11.
    Cooper, R.: Austinian truth, attitudes and type theory. Research on Language and Computation 3, 333–362 (2005)CrossRefGoogle Scholar
  12. 12.
    Cooper, R.: Type theory and semantics in flux. In: Kempson, R., Asher, N., Fernando, T. (eds.) Handbook of the Philosophy of Science. General editors: Gabbay, D.M., Thagard, P., Woods, J., vol. 14. Elsevier BV (2012)Google Scholar
  13. 13.
    Davidson, D.: The logical form of action sentences. In: Rescher, N., Anderson, A.R. (eds.) The Logic of Decision and Action. University of Pittsburgh Press, Pittsburgh (1967)Google Scholar
  14. 14.
    Barwise, J.: The situation in logic. CSLI Publications, Stanford (1989)Google Scholar
  15. 15.
    Cooper, R.: Records and record types in semantic theory. Journal of Logic and Computation 15, 99–112 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Larsson, S., Cooper, R.: Towards a formal view of corrective feedback. In: Alishahi, A., Poibeau, T., Villavicencio, A. (eds.) Proceedings of the Workshop on Cognitive Aspects of Computational Language Acquisition, EACL, pp. 1–9 (2009)Google Scholar
  17. 17.
    Cooper, R., Larsson, S.: Compositional and ontological semantics in learning from corrective feedback and explicit definition. In: Edlund, J., Gustafson, J., Hjalmarsson, A., Skantze, G. (eds.) Proceedings of SemDial 2009 (DiaHolmia): Workshop on the Semantics and Pragmatics of Dialogue, Stockholm, Sweden, Royal Institute of Technology (KTH), pp. 59–66 (2009)Google Scholar
  18. 18.
    Ginzburg, J.: The interactive stance: meaning for conversation. Oxford University Press, Oxford (2012)Google Scholar
  19. 19.
    Larsson, S.: The TTR perceptron: Dynamic perceptual meanings and semantic coordination. In: Artstein, R., Core, M., DeVault, D., Georgila, K., Kaiser, E., Stent, A. (eds.) SemDial 2011 (Los Angelogue): Proceedings of the 15th Workshop on the Semantics and Pragmatics of Dialogue, Los Angeles, California, pp. 140–148 (2011)Google Scholar
  20. 20.
    Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1778–1785 (2009)Google Scholar
  21. 21.
    Herskovits, A.: Language and spatial cognition: an interdisciplinary study of the prepositions in English. Cambridge University Press, Cambridge (1986)Google Scholar
  22. 22.
    Talmy, L.: Toward a cognitive semantics: concept structuring systems, vol. 1 and 2. MIT Press, Cambridge (2000)Google Scholar
  23. 23.
    Regier, T., Carlson, L.A.: Grounding spatial language in perception: an empirical and computational investigation. Journal of Experimental Psychology: General 130, 273–298 (2001)CrossRefGoogle Scholar
  24. 24.
    Coventry, K.R., Prat-Sala, M., Richards, L.: The interplay between geometry and function in the apprehension of Over, Under, Above and Below. Journal of Memory and Language 44, 376–398 (2001)CrossRefGoogle Scholar
  25. 25.
    Levinson, S.C.: Space in language and cognition: explorations in cognitive diversity. Cambridge University Press, Cambridge (2003)CrossRefGoogle Scholar
  26. 26.
    Maillat, D.: The semantics and pragmatics of directionals: a case study in English and French. PhD thesis, Committee for Comparative Philology and General Linguistics, University of Oxford, Oxford, UK (2003)Google Scholar
  27. 27.
    Steels, L., Loetzsch, M.: Perspective alignment in spatial language. In: Coventry, K.R., Tenbrink, T., Bateman, J.A. (eds.) Spatial Language and Dialogue. Oxford University Press (2009)Google Scholar
  28. 28.
    Dobnik, S.: Spatial descriptions in discourse: choosing a perspective. In: Brown-Schmidt, S., Jonathan Ginzburg, S.L. (eds.) Proceedings of SemDial 2012 (SeineDial): The 16th Workshop on the Semantics and Pragamtics of Dialogue, Université Paris-Diderot (Paris 7), Paris Sorbonne-Cité, pp. 151–152 (2012)Google Scholar
  29. 29.
    Dissanayake, M.W.M.G., Newman, P.M., Durrant-Whyte, H.F., Clark, S., Csorba, M.: A solution to the simultaneous localization and map building (SLAM) problem. IEEE Transactions on Robotic and Automation 17, 229–241 (2001)CrossRefGoogle Scholar
  30. 30.
    Shieber, S.: An Introduction to Unification-Based Approaches to Grammar. CSLI Publications, Stanford (1986)Google Scholar
  31. 31.
    Fernando, T.: A finite-state approach to events in natural language semantics. Journal of Logic and Computation 14, 79–92 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Lauria, S., Bugmann, G., Kyriacou, T., Bos, J., Klein, E.: Training personal robots using natural language instruction. IEEE Intelligent Systems 16, 38–45 (2001)Google Scholar
  33. 33.
    Witten, I.H., Frank, E., Hall, M.A.: Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, Burlington (2011)Google Scholar
  34. 34.
    Clark, A., Lappin, S.: Linguistic nativism and the poverty of the stimulus. Wiley-Blackwell, Chichester (2011)CrossRefGoogle Scholar
  35. 35.
    Tenenbaum, J.B., Kemp, C., Griffiths, T.L., Goodman, N.D.: How to grow a mind: Statistics, structure, and abstraction. Science 331, 1279–1285 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Pulman, S.G., Liakata, M.: Learning domain theories. In: Nicolov, N., Bontcheva, K., Angelova, G., Mitkov, R. (eds.) RANLP. Current Issues in Linguistic Theory (CILT), vol. 260, pp. 29–44. John Benjamins, Amsterdam/Philadelphia (2003)Google Scholar
  37. 37.
    Baroni, M., Murphy, B., Barbu, E., Poesio, M.: Strudel: A corpus-based semantic model based on properties and types. Cognitive Science 34, 222–254 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Simon Dobnik
    • 1
  • Robin Cooper
    • 1
  • Staffan Larsson
    • 1
  1. 1.Department of Philosophy, Linguistics and Theory of ScienceUniversity of GothenburgGöteborgSweden

Personalised recommendations