Modeling the Directionality of Attention During Spatial Language Comprehension

  • Thomas KluthEmail author
  • Michele Burigo
  • Pia Knoeferle
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10162)


It is known that the comprehension of spatial prepositions involves the deployment of visual attention. For example, consider the sentence “The salt is to the left of the stove”. Researchers [29, 30] have theorized that people must shift their attention from the stove (the reference object, RO) to the salt (the located object, LO) in order to comprehend the sentence. Such a shift was also implicitly assumed in the Attentional Vector Sum (AVS) model by [35], a cognitive model that computes an acceptability rating for a spatial preposition given a display that contains an RO and an LO. However, recent empirical findings showed that a shift from the RO to the LO is not necessary to understand a spatial preposition ([3], see also [15, 38]). In contrast, these findings suggest that people perform a shift in the reverse direction (i.e., from the LO to the RO). Thus, we propose the reversed AVS (rAVS) model, a modified version of the AVS model in which attention shifts from the LO to the RO. We assessed the AVS and the rAVS model on the data from [35] using three model simulation methods. Our simulations show that the rAVS model performs as well as the AVS model on these data while it also integrates the recent empirical findings. Moreover, the rAVS model achieves its good performance while being less flexible than the AVS model. (This article is an updated and extended version of the paper [23] presented at the 8th International Conference on Agents and Artificial Intelligence in Rome, Italy. The authors would like to thank Holger Schultheis for helpful discussions about the additional model simulation.)


Spatial language Spatial prepositions Cognitive modeling Model flexibility Visual attention 



This research was supported by the Cluster of Excellence Cognitive Interaction Technology ‘CITEC’ (EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG). The authors would also like to thank two anonymous reviewers for their useful comments and suggestions.


  1. 1.
    Burigo, M.: On the role of informativeness in spatial language comprehension. Ph.D. thesis, School of Psychology, University of Plymouth (2008)Google Scholar
  2. 2.
    Burigo, M., Coventry, K.R., Cangelosi, A., Lynott, D.: Spatial language and converseness. Q. J. Exp. Psychol. 69(12), 2319–2337 (2016). doi: 10.1080/17470218.2015.11248942016 CrossRefGoogle Scholar
  3. 3.
    Burigo, M., Knoeferle, P.: Visual attention during spatial language comprehension. PLoS ONE 10(1), e0115758 (2015)CrossRefGoogle Scholar
  4. 4.
    Burigo, M., Sacchi, S.: Object orientation affects spatial language comprehension. Cogn. Sci. 37(8), 1471–1492 (2013)CrossRefGoogle Scholar
  5. 5.
    Cangelosi, A., Coventry, K.R., Rajapakse, R., Joyce, D., Bacon, A., Richards, L., Newstead, S.N.: Grounding language in perception: a connectionist model of spatial terms and vague quantifiers. Prog. Neural Process. 16, 47 (2005)Google Scholar
  6. 6.
    Canty, A., Ripley, B.: Boot: Bootstrap R (S-Plus) Functions (2015). R package version 1.3-15Google Scholar
  7. 7.
    Carlson, L.A., Logan, G.D.: Attention and spatial language. In: Itti, L., Rees, G., Tsotsos, J.K. (eds.) Neurobiology of Attention, pp. 330–336. Elsevier, Amsterdam (2005). Chap. 54CrossRefGoogle Scholar
  8. 8.
    Carlson, L.A., Regier, T., Lopez, W., Corrigan, B.: Attention unites form and function in spatial language. Spat. Cogn. Comput. 6(4), 295–308 (2006)Google Scholar
  9. 9.
    Carlson-Radvansky, L.A., Covey, E.S., Lattanzi, K.M.: “What” effects on “where”: functional influences on spatial relations. Psychol. Sci. 10(6), 516–521 (1999)CrossRefGoogle Scholar
  10. 10.
    Carrasco, M.: Visual attention: the past 25 years. Vis. Res. 51(13), 1484–1525 (2011)CrossRefGoogle Scholar
  11. 11.
    CGAL: Computational geometry algorithms library.
  12. 12.
    Coventry, K.R., Garrod, S.C.: Saying, Seeing, and Acting: The Psychological Semantics of Spatial Prepositions. Essays in Cognitive Psychology, Psychology Press, Taylor and Francis, Hove and New York (2004)Google Scholar
  13. 13.
    Coventry, K.R., Lynott, D., Cangelosi, A., Monrouxe, L., Joyce, D., Richardson, D.C.: Spatial language, visual attention, and perceptual simulation. Brain Lang. 112(3), 202–213 (2010)CrossRefGoogle Scholar
  14. 14.
    Coventry, K.R., Prat Sala, M., Richards, L.: The interplay between geometry and function in the comprehension of over, under, above, and below. J. Mem. Lang. 44(3), 376–398 (2001)CrossRefGoogle Scholar
  15. 15.
    Franconeri, S.L., Scimeca, J.M., Roth, J.C., Helseth, S.A., Kahn, L.E.: Flexible visual processing of spatial relationships. Cognition 122(2), 210–227 (2012)CrossRefGoogle Scholar
  16. 16.
    Gapp, K.-P.: An empirically validated model for computing spatial relations. In: Wachsmuth, I., Rollinger, C.-R., Brauer, W. (eds.) KI 1995. LNCS, vol. 981, pp. 245–256. Springer, Heidelberg (1995). doi: 10.1007/3-540-60343-3_41 CrossRefGoogle Scholar
  17. 17.
    Georgopoulos, A.P., Schwartz, A.B., Kettner, R.E.: Neuronal population coding of movement direction. Science 233, 1416–1419 (1986)CrossRefGoogle Scholar
  18. 18.
    Hörberg, T.: Influences of form and function on the acceptability of projective prepositions in Swedish. Spat. Cogn. Comput. 8(3), 193–218 (2008)Google Scholar
  19. 19.
    Kelleher, J.D., Kruijff, G.J.M., Costello, F.J.: Proximity in context: an empirically grounded computational model of proximity for processing topological spatial expressions. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 745–752. Association for Computational Linguistics (2006)Google Scholar
  20. 20.
    Kim, W., Navarro, D.J., Pitt, M.A., Myung, I.J.: An MCMC-based method of comparing connectionist models in cognitive science. Adv. Neural Inf. Process. Syst. 16, 937–944 (2004)Google Scholar
  21. 21.
    Kluth, T.: A C++ implementation of the reversed Attentional Vector Sum (rAVS) model. Bielefeld University (2016). doi: 10.4119/unibi/2900103
  22. 22.
    Kluth, T., Burigo, M., Knoeferle, P.: Investigating the parameter space of cognitive models of spatial language comprehension. In: 5. Interdisziplinärer Workshop Kognitive Systeme, Bochum (2016)Google Scholar
  23. 23.
    Kluth, T., Burigo, M., Knoeferle, P.: Shifts of attention during spatial language comprehension: a computational investigation. In: Proceedings of the 8th International Conference on Agents and Artificial Intelligence, vol. 2, pp. 213–222. SCITEPRESS (2016). doi: 10.5220/0005851202130222
  24. 24.
    Kluth, T., Burigo, M., Schultheis, H., Knoeferle, P.: Distinguishing cognitive models of spatial language understanding. In: Proceedings of the International Conference on Cognitive Modeling (2016). Poster presented at the ICCM 2016Google Scholar
  25. 25.
    Kluth, T., Burigo, M., Schultheis, H., Knoeferle, P.: Testing the predictions of cognitive models of spatial language comprehension (in preparation)Google Scholar
  26. 26.
    Kluth, T., Schultheis, H.: Attentional distribution and spatial language. In: Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. (eds.) Spatial Cognition 2014. LNCS (LNAI), vol. 8684, pp. 76–91. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-11215-2_6 Google Scholar
  27. 27.
    Lee, C., Rohrer, W.H., Sparks, D.L.: Population coding of saccadic eye movements by neurons in the superior colliculus. Nature 332, 357–360 (1988)CrossRefGoogle Scholar
  28. 28.
    Logan, G.D.: Spatial attention and the apprehension of spatial relations. J. Exp. Psychol. Hum. Percept. Perform. 20(5), 1015 (1994)CrossRefGoogle Scholar
  29. 29.
    Logan, G.D.: Linguistic and conceptual control of visual spatial attention. Cogn. Psychol. 28(2), 103–174 (1995)CrossRefGoogle Scholar
  30. 30.
    Logan, G.D., Sadler, D.D.: A computational analysis of the apprehension of spatial relations. In: Bloom, P., Peterson, M.A., Nadel, L., Garrett, M.F. (eds.) Language and Space, pp. 493–530. The MIT Press, Cambridge (1996). Chap. 13Google Scholar
  31. 31.
    Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–1092 (1953)CrossRefGoogle Scholar
  32. 32.
    Navarro, D.J., Pitt, M.A., Myung, I.J.: Assessing the distinguishability of models and the informativeness of data. Cogn. Psychol. 49(1), 47–84 (2004)CrossRefGoogle Scholar
  33. 33.
    Pitt, M.A., Kim, W., Navarro, D.J., Myung, J.I.: Global model analysis by parameter space partitioning. Psychol. Rev. 113(1), 57–83 (2006)CrossRefGoogle Scholar
  34. 34.
    Pitt, M.A., Myung, I.J.: When a good fit can be bad. Trends Cogn. Sci. 6(10), 421–425 (2002)CrossRefGoogle Scholar
  35. 35.
    Regier, T., Carlson, L.A.: Grounding spatial language in perception: an empirical and computational investigation. J. Exp. Psychol.: Gen. 130(2), 273–298 (2001)CrossRefGoogle Scholar
  36. 36.
    Richter, M., Lins, J., Schneegans, S., Sandamirskaya, Y., Schöner, G.: Autonomous neural dynamics to test hypotheses in a model of spatial language. In: Bello, P., Guarini, M., Mc-Shane, M., Scassellati, B. (eds.) Proceedings of the 36th Annual Conference of the Cognitive Science Society, pp. 2847–2852. Cognitive Science Society, Austin (2014)Google Scholar
  37. 37.
    Roberts, S., Pashler, H.: How persuasive is a good fit? A comment on theory testing. Psychol. Rev. 107(2), 358–367 (2000)CrossRefGoogle Scholar
  38. 38.
    Roth, J.C., Franconeri, S.L.: Asymmetric coding of categorical spatial relations in both language and vision. Front. Psychol. 3, Article No. 464 (2012)Google Scholar
  39. 39.
    Schultheis, H., Carlson, L.A.: Mechanisms of reference frame selection in spatial term use: computational and empirical studies. Cogn. Sci. (2015). doi: 10.1111/cogs.12327
  40. 40.
    Schultheis, H., Singhaniya, A., Chaplot, D.S.: Comparing model comparison methods. In: Proceedings of the 35th Annual Conference of the Cognitive Science Society, pp. 1294–1299. Cognitive Science Society, Austin (2013)Google Scholar
  41. 41.
    Veksler, V.D., Myers, C.W., Gluck, K.A.: Model flexibility analysis. Psychol. Rev. 122(4), 755–769 (2015)CrossRefGoogle Scholar
  42. 42.
    Wagenmakers, E.J., Ratcliff, R., Gomez, P., Iverson, G.J.: Assessing model mimicry using the parametric bootstrap. J. Math. Psychol. 48(1), 28–50 (2004)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Language & Cognition Group, CITEC (Cognitive Interaction Technology Excellence Cluster)Bielefeld UniversityBielefeldGermany
  2. 2.Department of German Language and LinguisticsHumboldt UniversityBerlinGermany

Personalised recommendations