Cognitive Processing

, Volume 12, Issue 1, pp 67–94 | Cite as

Interpreting spatial language in image captions

  • Mark M. HallEmail author
  • Philip D. Smart
  • Christopher B. Jones
Research Report


The map as a tool for accessing data has become very popular in recent years, but a lot of data do not have the necessary spatial meta-data to allow for that. Some data such as photographs however have spatial information in their captions and if this could be extracted, then they could be made available via map-based interfaces. Towards this goal, we introduce a model and spatio-linguistic reasoner for interpreting the spatial information in image captions that is based upon quantitative data about spatial language use acquired directly from people. Spatial language is inherently vague, and both the model and reasoner have been designed to incorporate this vagueness at the quantitative level and not only qualitatively.


Spatial language Vagueness Natural language processing Spatial reasoning Field-based modelling Geographic information retrieval 



We would like to gratefully acknowledge contributors to Geograph British Isles (see, whose work is made available under the following Creative Commons Attribution-ShareAlike 2.5 Licence ( This material is based upon work supported by the European Community in the TRIPOD (FP6 cr \(\hbox{n}^\circ045335\)) project. We would also like to thank the two reviewers, whose comments and suggestions helped focus the ideas presented in this paper.


  1. Ahlqvist O, Keukelaar J, Oukbir K (1998) Using rough classification to represent uncertainty in spatial data. In: Proceedings of the SIRC Colloquium, pp 1–9Google Scholar
  2. Altman D (1994) Fuzzy set theoretic approaches for handling imprecision in spatial analysis. Int J Geogr Inf Sci 8(3):271–289CrossRefGoogle Scholar
  3. Andogah G, Bouma G, Nerbonne J, Koster E (2008) Placename ambiguity resolution. In: LREC workshop on methodologies and resources for processing spatial languageGoogle Scholar
  4. Bennet B (2001) Application of supervaluation semantics to vaguely defined spatial concepts. In: Spatial information theory. foundations of geographic information science : international conference, COSIT 2001 Morro Bay, CA, USA, September 19–23, 2001. Proceedings, pp 108–123Google Scholar
  5. Bennett B, Agarwal P (2007) Semantic categories underlying the meaning of ’place’. In: Spatial information theory, 8th international conference, COSIT 2007, Melbourne, Australia, September 19–23, 2007, Proceedings, pp 78–95Google Scholar
  6. Bittner T, Stell J (2003) Stratified rough sets and vagueness. In: Spatial information theory, Springer, Berlin/Heidelberg, pp 270–286Google Scholar
  7. Bowerman M, Choi S (2003) Space under construction: language-specific spatial categorization in first language acquisition. In: Gentner D, Goldin-Meadow S (eds) Language in mind. MIT, Cambridge, pp 387–428Google Scholar
  8. Brown P (1994) The ins and ons of tzeltal locative expressions. Linguistics 32:743–790CrossRefGoogle Scholar
  9. Burghardt D (2005) Controlled line smoothing by snakes. GeoInformatica 9(3):237–252CrossRefGoogle Scholar
  10. Buscaldi D, Rosso P (2008) Map-based vs. knowledge-based toponym disambiguation. In: Proceeding of the 2nd international workshop on geographic information retrieval.GIR’08, pp 19–22Google Scholar
  11. Chomsky N (1965) Aspects of the theory of syntax. MIT, CambridgeGoogle Scholar
  12. Clementini E, Felice PD (1996) An algebraic model for spatial objects with indeterminate boundaries. In: Geographic objects with indeterminate boundaries. Taylor and Francis, London, pp 155–169Google Scholar
  13. Clementini E, Felice PD (1997) Approximate topological relations. Int J Approx Reason 16(2):173–204CrossRefGoogle Scholar
  14. Cohn A, Gotts N (1996a) The ’egg-yolk’ representation of regions with indeterminate boundaries. In: Proceedings, GISDATA specialist meeting on geographical objects with undetermined boundaries, Francis Taylor, pp 171–187Google Scholar
  15. Cohn A, Gotts N (1996b) Representing spatial vagueness: a merological approach. In: KR’96: principles of knowledge representation and reasoning. Morgan Kaufmann, San Mateo, pp 230–241Google Scholar
  16. Couclelis H (1992) People manipulate objects (but cultivate fields): Beyond the raster-vector debate in gis. In: Theories and methods of spatio-temporal reasoning in geographic space, vol 639/1992, Springer, Berlin/Heidelberg, pp 65–77Google Scholar
  17. Couclelis H, Gottsegen J (1997) What maps mean to people: Denotation, connotation, and geographic visualization in land-use debates. In: Spatial information theory: a theoretical basis for GIS (COSIT’97), vol 1329/1997, Springer, Berlin/Heidelberg, pp 151–162Google Scholar
  18. Coventry K, Prat-Sala M, Richards L (2001) The interplay between geometry and function in the comprehension of over, under, above and below. J Memory Lang 44(3):376–398CrossRefGoogle Scholar
  19. Cunningham H, Maynard D, Bontcheva K, Tablan V (2002) Gate: a framework and graphical development environment for robust nlp tools and applications. In: Proceedings of the 40th anniversary meeting of the association for computational linguistics, pp 168–175Google Scholar
  20. Edwardes A, Purves R (2007) A theoretical grounding for semantic descriptions of place. Lect Notes Comput Sci 4857:106Google Scholar
  21. Egenhofer M (1991) Reasoning about binary topological relations. In: Second symposium on large spatial databases, lecture notes in computer science, vol 525. Springer, pp 143–160Google Scholar
  22. Erwig M, Schneider M (1997) Partition and conquer. In: COSIT ’97: Proceedings of the international conference on spatial information theory. Springer, London, pp 389–407Google Scholar
  23. Fabrikant S, Buttenfield B (2001) Formalizing semantic spaces for information access. Ann Assoc Am Geogr 91(2):263–280CrossRefGoogle Scholar
  24. Fisher P (2000) Sorites paradox and vague geographies. Fuzzy Sets Syst 113(1):7–18CrossRefGoogle Scholar
  25. Fisher P, Wood J, Cheng T (2004) Where is helvellyn? Fuzziness of multi-scale landscape morphometry. Trans Inst Br Geogr 29(1):106–128CrossRefGoogle Scholar
  26. Fisher PF, Orf TM (1991) An investigation of the meaning of near and close on a university campus. Comput Environ Urban Syst 15(1-2):23–35. doi: 10.1016/0198-9715(91)90043-D CrossRefGoogle Scholar
  27. Frank AU, Raubal M (1999) Formal specification of image schemata—a step towards interoperability in geographic information systems. Spatial Cogn Comput 1(1):67–101CrossRefGoogle Scholar
  28. Friedman C, Kra P, Yu H, Krauthammer M, Rhzetsky A (2001) Genies: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinformatics 17(1):74–82Google Scholar
  29. Fuhr T, Socher G, Scheering C, Sagerer G (1995) A three-dimensional spatial model for the interpretation of image data. In: IJCAI-95 Workshop on the representation and processing of spatial expressions, pp 93–102Google Scholar
  30. Gahegan M (1995) Proximity operators for qualitative spatial reasoning. In: Spatial information theory: a theoretical basis for GIS. Springer, Berlin/Heidelberg, pp 31–44Google Scholar
  31. Gapp K (1994) Basic meanings of spatial relations: Computation and evaluation in 3d space. In: National conference on artificial intelligence, pp 1393–1398Google Scholar
  32. Gärdenfors P (2000) Conceptual spaces: the geometry of thought. MIT Press, CambridgeGoogle Scholar
  33. Garrod S, Ferrier G, Campbell S (1999) In and on: investigating the functional geometry of spatial prepositions. Cogn Tech Work 72(2):167–189Google Scholar
  34. Goodchild M (1992) Geographical data modeling. Comput Geosci 18(4):401–408CrossRefGoogle Scholar
  35. Goodchild M, Hill L (2008) Introduction to digital gazetteer research. Int J Geogr Inf Sci 22(10):1039–1044CrossRefGoogle Scholar
  36. Guo Q, Liu Y, Wieczorek J (2008) Georeferencing locality descriptions and computing associated uncertainty using a probabilistic approach. Int J Geogr Inf Sci 22(10):1067–1090CrossRefGoogle Scholar
  37. Güting R, Schneider M (1993) Realms: A foundation for spatial data types in database systems. In: Proceedings of the 3rd international symposium on large databases, pp 33–44Google Scholar
  38. Hall M, Jones C (2008) Quantifying spatial prepositions: an experimental study. In: Proceedings of the ACM GIS’08, pp 451–454Google Scholar
  39. Hall MM, Jones CB (2009) Initialising and terminating active contours for vague field crisping. In: GISRUK 2009, pp 395–397Google Scholar
  40. Hall S (1980) Encoding/decoding. In: For Contemporary Cultural Studies C (ed) Culture, media, language: working papers in cultural studies 1972–79. Hutchinson, London, pp. 128–138Google Scholar
  41. Hengl T (2007) A practical guide to geostatistical mapping of environmental variablesGoogle Scholar
  42. Herskovits A (1986) Language and spatial cognition: an interdisciplinary study of prepositions in English. Cambridge University Press, CambridgeGoogle Scholar
  43. Horvath P, Jermyn I, Kato Z, Zerubia J (2009) A higher-order active contour model of a ’gas of circles’ and its application to tree crown extraction. Pattern Recogn Lett 42(5):699–709Google Scholar
  44. Hwang S, Thill JC (2005) Modeling localities with fuzzy sets and gis. Fuzzy modeling with spatial information for geographic problems, pp 71–104Google Scholar
  45. Johnson M (1987) The body in the mind. University of Chicago Press, ChicagoGoogle Scholar
  46. Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour models. Int J Comput Vis 1(4):321–331CrossRefGoogle Scholar
  47. Kemmerer D (2006) The semantics of space: integrating linguistic typology and cognitive neuroscience. Neuropsychologia 44(9):1607–1621CrossRefPubMedGoogle Scholar
  48. Kemmerer D, Tranel D (2000) A double dissociation between linguistic and perceptual representations of spatial relationships. Cogn Neuropsychol 17(5):393–414CrossRefPubMedGoogle Scholar
  49. Klippel A, Montello D (2007) Linguistic and nonlinguistic turn direction concepts. In: Spatial information theory, 8th international conference, COSIT 2007, Melbourne, Australia, September 19–23, 2007, Proceedings, pp 354–372Google Scholar
  50. Klir G, Yuan B (1995) Fuzzy sets and fuzzy logic: theory and applications. Prentice-Hall, Englewood CliffsGoogle Scholar
  51. Krige D (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J Chem Metallur Min Soc 52:119–1139)Google Scholar
  52. Kuhn W (2002) Modeling the semantics of geographic categories through conceptual integration. In: Geographic information science: second international conference, GIScience, pp 108–118Google Scholar
  53. Kulik L (2001) A geometric theory of vague boundaries based on supervaluation. In: Spatial information theory. Foundations of geographic information science : international conference, COSIT 2001. Springer, Berlin/Heidelberg, pp 44–59Google Scholar
  54. Lakoff G, Johnson M (1980) Metaphors we live by. The University of Chicago Press, ChicagoGoogle Scholar
  55. Lam KM, Yan H (1994) Fast greedy algorithm for active contours. Electron Lett 30(1):21–23CrossRefGoogle Scholar
  56. Landau B, Jackendoff R (1993) “What” and “where” in spatial language and spatial cognition. Behav Brain Sci 16(2):217–238CrossRefGoogle Scholar
  57. Laurini R, Pariente D (1996) Towards a field-oriented language: First specifications. In: Geographic objects with indeterminate boundaries. Taylor and Francis, London, pp 225–236Google Scholar
  58. Leidner J (2007) Toponym resolution in text: annotation, evaluation and applications of spatial grounding of place names. PhD thesis, School of Informatics, Edinburgh, UKGoogle Scholar
  59. Levinson S (2003) Space in language and cognition: explorations in cognitive diversity. CUP, CambridgeCrossRefGoogle Scholar
  60. Levinson S, Kita S, Haun D, Rasch B (2002) Returning the tables: language affects spatial reasoning. Cogn Tech Work 84(2):155–188Google Scholar
  61. Li P, Gleitman L (2002) Turning the tables: language and spatial reasoning. Cogn Tech Work 83(3):265–294Google Scholar
  62. Liu Y, Goodchild M, Guo Q, Tian Y, Wu L (2008) Towards a general field model and its order in gis. Int J Geogr Inf Sci 22(6):623–643CrossRefGoogle Scholar
  63. Liu Y, Guo Q, Wieczorek J, Goodchild M (2009) Positioning localities based on spatial assertions. Int J Geogr Inf Sci 23(11):1471–1501CrossRefGoogle Scholar
  64. Liu Y, Yuan Y, Xiao D, Zhang Y, Hu J (2010) A point-set-based approximation for areal objects: a case study of representing localities. Comput Environ Urban Syst 34(1):28–39CrossRefGoogle Scholar
  65. Lodge D (1984) Small world. Penguin Books, New YorkGoogle Scholar
  66. Mark D (1989) Cognitive image-schemata for geographic information: relations to user views and gis interfaces. In: Proceedings GIS/LIS’89, pp 551–560Google Scholar
  67. Mark D, Frank A (1995) Experiential and formal models of geographic space. Environ Plan 23:3–24Google Scholar
  68. Mark D, Turk A, Stea D (2007) Progress on yindjibarndi ethnophysiography. In: Spatial information theory, 8th International Conference, COSIT 2007, Melbourne, Australia, September 19-23, 2007, Proceedings, pp 1–19Google Scholar
  69. Matheron G (1962) Traité de géostatistique appliquée. Mémoires du Bureau de Recherches Géologiques et Minières 14Google Scholar
  70. Miller G, Johnson-Laird P (1976) Language and Perception. Cambridge University Press, CambridgeGoogle Scholar
  71. Morrow D, Clark H (1988) Interpreting words in spatial descriptions. Lang Cogn Process 3:275–291CrossRefGoogle Scholar
  72. Mukerjee A, Gupta K, Nautiyal S, Singh M, Mishra N (2000) Conceptual description of visual scenes from linguistic models. Image Vis Comput 18(2):173–187CrossRefGoogle Scholar
  73. Parsons S (1996) Current approaches to handling imperfect information in data and knowledge bases. Knowl Data Eng 3(8):353–372CrossRefGoogle Scholar
  74. Pfoser D, Tryfona N, Jensen C (2005) Indeterminacy and spatiotemporal data: basic definitions and case study. GeoInformatica 9(3):211–236CrossRefGoogle Scholar
  75. Power C, Simms A, White R (2001) Hierarchical fuzzy pattern matching for the regional comparison of land use maps. Int J Geogr Sci 15:77–100CrossRefGoogle Scholar
  76. Purves R, Clough P, Joho H (2005) Identifying imprecise regions for geographic information retrieval using the web. In: GISRUK’05, pp 313–318Google Scholar
  77. Randell D, Cui Z, Cohn A (1992) A spatial logic based on regions and connection. In: KR’92. Principles of knowledge representation and reasoning: proceedings of the third international conference. Morgan Kaufmann, pp 165–176Google Scholar
  78. Raubal M, Worboys M (1999) A formal model of the process of wayfinding in built environments. In: Spatial information theory. cognitive and computational foundations of geographic information science: international conference COSIT’99, Stade, Germany, August 1999. Proceedings, Springer, Berlin/Heidelberg. Lect Notes Comput Sci 1661:748Google Scholar
  79. Ravin Y, Wacholder N (1996) Extracting names from natural-language text. Tech. Rep. Report 20338, IBM ResearchGoogle Scholar
  80. Robinson V (2000) Individual and multipersonal fuzzy spatial relations acquired using human–machine interaction. Fuzzy Sets Syst 113(1):133–145CrossRefGoogle Scholar
  81. Robinson V (2003) A perspective on the fundamentals of fuzzy sets and their use in geographic information systems. Trans GIS 7(1):3–30CrossRefGoogle Scholar
  82. Sapir E (1929) The status of linguistics as a science. Language 5Google Scholar
  83. Schneider M (1996) Modelling spatial objects with undetermined boundaries using the realm/rose approach. In: Geographic objects with indeterminate boundaries, vol 2. Taylor and Francis, London, pp 141–152Google Scholar
  84. Schneider M (2000) Finite resolution crisp and fuzzy spatial objects. In: International symposium on spatial data handling, pp 3–17Google Scholar
  85. Schneider M (2001) A design of topological predicates for complex crisp and fuzzy regions. In: Conceptual modeling—ER 2001, Springer, Berlin/Heidelberg. Lect Notes Comput Sci 2224:103Google Scholar
  86. Schockaert S, de Cock M, Kerre E (2008) Location approximation for local search services using natural language hints. Int J Geogr Inf Sci 22(3):315–336CrossRefGoogle Scholar
  87. Smart P, Jones C, Twaroch F (2010) Multi-source toponym data integration and mediation for a meta-gazetteer service. In: GISCIENCE 2010 (forthcoming)Google Scholar
  88. Smith B, Varzi A (1997) Fiat and bona fide boundaries: towards an ontology of spatially extended objects. In: Spatial Information theory: a theoretical basis for GIS (COSIT’97). Lect Notes Comput Sci 1329:103–119Google Scholar
  89. Smith D, Crane G (2001) Disambiguating geographic names in a historical digital library. In: Research and advanced technology for digital libraries: fifth European conference (ECDL 2001), pp 127–136Google Scholar
  90. Srihari R, Rapaport W (1990) Combining linguistic and pictorial information: using captions to interpret newspaper photographs. In: Current trends in SNePS—semantic network processing system, no. 437/1990 in Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, pp 85–96Google Scholar
  91. Steiniger S, Meier S (2004) Snakes: a technique for line smoothing and displacement in map generalisation. In: ICA workshop on generalisation and multiple representationGoogle Scholar
  92. Talmy L (1983) How language structures space. In: Spatial orientation. Plenum, New York, pp 225–282Google Scholar
  93. Tang X (2004) Spatial object model[l]ing in fuzzy topological spaces : with applications to land cover change. PhD thesis, University of Twente, EnschedeGoogle Scholar
  94. Terzopoulos D (1986) Regularization of inverse visual problems involving disontinuities. IEEE Transactions PAMI-8, p 413Google Scholar
  95. Tversky B, Lee P (1998) How space structures language. In: Spatial cognition: an interdisciplinary approach to representing and processing spatial knowledge, Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, p 157Google Scholar
  96. Vorwerg C, Rickheit G (1998) Typicality effects in the categorization of spatial relations. In: Spatial cognition: an interdisciplinary approach to representing and processing spatial knowledge, Lecture Notes in Computer Science, vol 1404, Springer Berlin/Heidelberg, pp 203–222Google Scholar
  97. Wang F, Hall G (1996) Fuzzy representation of geographical boundaries in gis. Int J Geogr Inf Sci 10(5):573–590CrossRefGoogle Scholar
  98. Whorf B, Carroll J, Chase S (1956) Language, thought, and reality: selected writings of Benjamin Lee Whorf. MIT, CambridgeGoogle Scholar
  99. Wieczorek J, Guo Q, Hiimans R (2004) The point-radius method for georeferencing locality descriptions and calculating associated uncertainty. Int J Geogr Inf Sci 18(8):745–767CrossRefGoogle Scholar
  100. Winter S (2000) Uncertain topological relations between imprecise regions. Int J Geogr Inf Sci 14(5):411–430CrossRefGoogle Scholar
  101. Worboys M (2001) Nearness relations in environmental space. Int J Geogr Inf Sci 15(7):633–651CrossRefGoogle Scholar
  102. Worboys M, Duckham M, Kulik L (2004) Commonsense notions of proximity and direction in environmental space. Spatial Cogn Comput 4(4):285–312CrossRefGoogle Scholar
  103. Xie X, Mirmehdi M (2006) Magnetostatic field for the active contour model: a study in convergence. In: Proceedings of the 17th British machine vision conference, pp 127–136Google Scholar
  104. Yamada A, Yamamoto T, Ikeda H, Nishida T, Doshita S (1992) Reconstructing spatial image from natural language texts. In: Proceedings COLING-92, vol 4, pp 1279–1283Google Scholar
  105. Zadeh L (1965) Fuzzy sets. Inf Control 8:338–353CrossRefGoogle Scholar

Copyright information

© Marta Olivetti Belardinelli and Springer-Verlag 2010

Authors and Affiliations

  • Mark M. Hall
    • 1
    Email author
  • Philip D. Smart
    • 1
  • Christopher B. Jones
    • 1
  1. 1.Cardiff School of Computer Science & InformaticsCardiff UniversityCardiffUK

Personalised recommendations