# Interpreting spatial language in image captions

- 215 Downloads
- 3 Citations

## Abstract

The map as a tool for accessing data has become very popular in recent years, but a lot of data do not have the necessary spatial meta-data to allow for that. Some data such as photographs however have spatial information in their captions and if this could be extracted, then they could be made available via map-based interfaces. Towards this goal, we introduce a model and spatio-linguistic reasoner for interpreting the spatial information in image captions that is based upon quantitative data about spatial language use acquired directly from people. Spatial language is inherently vague, and both the model and reasoner have been designed to incorporate this vagueness at the quantitative level and not only qualitatively.

## keywords

Spatial language Vagueness Natural language processing Spatial reasoning Field-based modelling Geographic information retrieval## Notes

### Acknowledgments

We would like to gratefully acknowledge contributors to Geograph British Isles (see http://www.geograph.org.uk/credits/2007-02-24), whose work is made available under the following Creative Commons Attribution-ShareAlike 2.5 Licence (http://creativecommons.org/licenses/by-sa/2.5/). This material is based upon work supported by the European Community in the TRIPOD (FP6 cr \(\hbox{n}^\circ045335\)) project. We would also like to thank the two reviewers, whose comments and suggestions helped focus the ideas presented in this paper.

## References

- Ahlqvist O, Keukelaar J, Oukbir K (1998) Using rough classification to represent uncertainty in spatial data. In: Proceedings of the SIRC Colloquium, pp 1–9Google Scholar
- Altman D (1994) Fuzzy set theoretic approaches for handling imprecision in spatial analysis. Int J Geogr Inf Sci 8(3):271–289CrossRefGoogle Scholar
- Andogah G, Bouma G, Nerbonne J, Koster E (2008) Placename ambiguity resolution. In: LREC workshop on methodologies and resources for processing spatial languageGoogle Scholar
- Bennet B (2001) Application of supervaluation semantics to vaguely defined spatial concepts. In: Spatial information theory. foundations of geographic information science : international conference, COSIT 2001 Morro Bay, CA, USA, September 19–23, 2001. Proceedings, pp 108–123Google Scholar
- Bennett B, Agarwal P (2007) Semantic categories underlying the meaning of ’place’. In: Spatial information theory, 8th international conference, COSIT 2007, Melbourne, Australia, September 19–23, 2007, Proceedings, pp 78–95Google Scholar
- Bittner T, Stell J (2003) Stratified rough sets and vagueness. In: Spatial information theory, Springer, Berlin/Heidelberg, pp 270–286Google Scholar
- Bowerman M, Choi S (2003) Space under construction: language-specific spatial categorization in first language acquisition. In: Gentner D, Goldin-Meadow S (eds) Language in mind. MIT, Cambridge, pp 387–428Google Scholar
- Brown P (1994) The ins and ons of tzeltal locative expressions. Linguistics 32:743–790CrossRefGoogle Scholar
- Burghardt D (2005) Controlled line smoothing by snakes. GeoInformatica 9(3):237–252CrossRefGoogle Scholar
- Buscaldi D, Rosso P (2008) Map-based vs. knowledge-based toponym disambiguation. In: Proceeding of the 2nd international workshop on geographic information retrieval.GIR’08, pp 19–22Google Scholar
- Chomsky N (1965) Aspects of the theory of syntax. MIT, CambridgeGoogle Scholar
- Clementini E, Felice PD (1996) An algebraic model for spatial objects with indeterminate boundaries. In: Geographic objects with indeterminate boundaries. Taylor and Francis, London, pp 155–169Google Scholar
- Clementini E, Felice PD (1997) Approximate topological relations. Int J Approx Reason 16(2):173–204CrossRefGoogle Scholar
- Cohn A, Gotts N (1996a) The ’egg-yolk’ representation of regions with indeterminate boundaries. In: Proceedings, GISDATA specialist meeting on geographical objects with undetermined boundaries, Francis Taylor, pp 171–187Google Scholar
- Cohn A, Gotts N (1996b) Representing spatial vagueness: a merological approach. In: KR’96: principles of knowledge representation and reasoning. Morgan Kaufmann, San Mateo, pp 230–241Google Scholar
- Couclelis H (1992) People manipulate objects (but cultivate fields): Beyond the raster-vector debate in gis. In: Theories and methods of spatio-temporal reasoning in geographic space, vol 639/1992, Springer, Berlin/Heidelberg, pp 65–77Google Scholar
- Couclelis H, Gottsegen J (1997) What maps mean to people: Denotation, connotation, and geographic visualization in land-use debates. In: Spatial information theory: a theoretical basis for GIS (COSIT’97), vol 1329/1997, Springer, Berlin/Heidelberg, pp 151–162Google Scholar
- Coventry K, Prat-Sala M, Richards L (2001) The interplay between geometry and function in the comprehension of over, under, above and below. J Memory Lang 44(3):376–398CrossRefGoogle Scholar
- Cunningham H, Maynard D, Bontcheva K, Tablan V (2002) Gate: a framework and graphical development environment for robust nlp tools and applications. In: Proceedings of the 40th anniversary meeting of the association for computational linguistics, pp 168–175Google Scholar
- Edwardes A, Purves R (2007) A theoretical grounding for semantic descriptions of place. Lect Notes Comput Sci 4857:106Google Scholar
- Egenhofer M (1991) Reasoning about binary topological relations. In: Second symposium on large spatial databases, lecture notes in computer science, vol 525. Springer, pp 143–160Google Scholar
- Erwig M, Schneider M (1997) Partition and conquer. In: COSIT ’97: Proceedings of the international conference on spatial information theory. Springer, London, pp 389–407Google Scholar
- Fabrikant S, Buttenfield B (2001) Formalizing semantic spaces for information access. Ann Assoc Am Geogr 91(2):263–280CrossRefGoogle Scholar
- Fisher P (2000) Sorites paradox and vague geographies. Fuzzy Sets Syst 113(1):7–18CrossRefGoogle Scholar
- Fisher P, Wood J, Cheng T (2004) Where is helvellyn? Fuzziness of multi-scale landscape morphometry. Trans Inst Br Geogr 29(1):106–128CrossRefGoogle Scholar
- Fisher PF, Orf TM (1991) An investigation of the meaning of near and close on a university campus. Comput Environ Urban Syst 15(1-2):23–35. doi: 10.1016/0198-9715(91)90043-D CrossRefGoogle Scholar
- Frank AU, Raubal M (1999) Formal specification of image schemata—a step towards interoperability in geographic information systems. Spatial Cogn Comput 1(1):67–101CrossRefGoogle Scholar
- Friedman C, Kra P, Yu H, Krauthammer M, Rhzetsky A (2001) Genies: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinformatics 17(1):74–82Google Scholar
- Fuhr T, Socher G, Scheering C, Sagerer G (1995) A three-dimensional spatial model for the interpretation of image data. In: IJCAI-95 Workshop on the representation and processing of spatial expressions, pp 93–102Google Scholar
- Gahegan M (1995) Proximity operators for qualitative spatial reasoning. In: Spatial information theory: a theoretical basis for GIS. Springer, Berlin/Heidelberg, pp 31–44Google Scholar
- Gapp K (1994) Basic meanings of spatial relations: Computation and evaluation in 3d space. In: National conference on artificial intelligence, pp 1393–1398Google Scholar
- Gärdenfors P (2000) Conceptual spaces: the geometry of thought. MIT Press, CambridgeGoogle Scholar
- Garrod S, Ferrier G, Campbell S (1999) In and on: investigating the functional geometry of spatial prepositions. Cogn Tech Work 72(2):167–189Google Scholar
- Goodchild M (1992) Geographical data modeling. Comput Geosci 18(4):401–408CrossRefGoogle Scholar
- Goodchild M, Hill L (2008) Introduction to digital gazetteer research. Int J Geogr Inf Sci 22(10):1039–1044CrossRefGoogle Scholar
- Guo Q, Liu Y, Wieczorek J (2008) Georeferencing locality descriptions and computing associated uncertainty using a probabilistic approach. Int J Geogr Inf Sci 22(10):1067–1090CrossRefGoogle Scholar
- Güting R, Schneider M (1993) Realms: A foundation for spatial data types in database systems. In: Proceedings of the 3rd international symposium on large databases, pp 33–44Google Scholar
- Hall M, Jones C (2008) Quantifying spatial prepositions: an experimental study. In: Proceedings of the ACM GIS’08, pp 451–454Google Scholar
- Hall MM, Jones CB (2009) Initialising and terminating active contours for vague field crisping. In: GISRUK 2009, pp 395–397Google Scholar
- Hall S (1980) Encoding/decoding. In: For Contemporary Cultural Studies C (ed) Culture, media, language: working papers in cultural studies 1972–79. Hutchinson, London, pp. 128–138Google Scholar
- Hengl T (2007) A practical guide to geostatistical mapping of environmental variablesGoogle Scholar
- Herskovits A (1986) Language and spatial cognition: an interdisciplinary study of prepositions in English. Cambridge University Press, CambridgeGoogle Scholar
- Horvath P, Jermyn I, Kato Z, Zerubia J (2009) A higher-order active contour model of a ’gas of circles’ and its application to tree crown extraction. Pattern Recogn Lett 42(5):699–709Google Scholar
- Hwang S, Thill JC (2005) Modeling localities with fuzzy sets and gis. Fuzzy modeling with spatial information for geographic problems, pp 71–104Google Scholar
- Johnson M (1987) The body in the mind. University of Chicago Press, ChicagoGoogle Scholar
- Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour models. Int J Comput Vis 1(4):321–331CrossRefGoogle Scholar
- Kemmerer D (2006) The semantics of space: integrating linguistic typology and cognitive neuroscience. Neuropsychologia 44(9):1607–1621CrossRefPubMedGoogle Scholar
- Kemmerer D, Tranel D (2000) A double dissociation between linguistic and perceptual representations of spatial relationships. Cogn Neuropsychol 17(5):393–414CrossRefPubMedGoogle Scholar
- Klippel A, Montello D (2007) Linguistic and nonlinguistic turn direction concepts. In: Spatial information theory, 8th international conference, COSIT 2007, Melbourne, Australia, September 19–23, 2007, Proceedings, pp 354–372Google Scholar
- Klir G, Yuan B (1995) Fuzzy sets and fuzzy logic: theory and applications. Prentice-Hall, Englewood CliffsGoogle Scholar
- Krige D (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J Chem Metallur Min Soc 52:119–1139)Google Scholar
- Kuhn W (2002) Modeling the semantics of geographic categories through conceptual integration. In: Geographic information science: second international conference, GIScience, pp 108–118Google Scholar
- Kulik L (2001) A geometric theory of vague boundaries based on supervaluation. In: Spatial information theory. Foundations of geographic information science : international conference, COSIT 2001. Springer, Berlin/Heidelberg, pp 44–59Google Scholar
- Lakoff G, Johnson M (1980) Metaphors we live by. The University of Chicago Press, ChicagoGoogle Scholar
- Lam KM, Yan H (1994) Fast greedy algorithm for active contours. Electron Lett 30(1):21–23CrossRefGoogle Scholar
- Landau B, Jackendoff R (1993) “What” and “where” in spatial language and spatial cognition. Behav Brain Sci 16(2):217–238CrossRefGoogle Scholar
- Laurini R, Pariente D (1996) Towards a field-oriented language: First specifications. In: Geographic objects with indeterminate boundaries. Taylor and Francis, London, pp 225–236Google Scholar
- Leidner J (2007) Toponym resolution in text: annotation, evaluation and applications of spatial grounding of place names. PhD thesis, School of Informatics, Edinburgh, UKGoogle Scholar
- Levinson S (2003) Space in language and cognition: explorations in cognitive diversity. CUP, CambridgeCrossRefGoogle Scholar
- Levinson S, Kita S, Haun D, Rasch B (2002) Returning the tables: language affects spatial reasoning. Cogn Tech Work 84(2):155–188Google Scholar
- Li P, Gleitman L (2002) Turning the tables: language and spatial reasoning. Cogn Tech Work 83(3):265–294Google Scholar
- Liu Y, Goodchild M, Guo Q, Tian Y, Wu L (2008) Towards a general field model and its order in gis. Int J Geogr Inf Sci 22(6):623–643CrossRefGoogle Scholar
- Liu Y, Guo Q, Wieczorek J, Goodchild M (2009) Positioning localities based on spatial assertions. Int J Geogr Inf Sci 23(11):1471–1501CrossRefGoogle Scholar
- Liu Y, Yuan Y, Xiao D, Zhang Y, Hu J (2010) A point-set-based approximation for areal objects: a case study of representing localities. Comput Environ Urban Syst 34(1):28–39CrossRefGoogle Scholar
- Lodge D (1984) Small world. Penguin Books, New YorkGoogle Scholar
- Mark D (1989) Cognitive image-schemata for geographic information: relations to user views and gis interfaces. In: Proceedings GIS/LIS’89, pp 551–560Google Scholar
- Mark D, Frank A (1995) Experiential and formal models of geographic space. Environ Plan 23:3–24Google Scholar
- Mark D, Turk A, Stea D (2007) Progress on yindjibarndi ethnophysiography. In: Spatial information theory, 8th International Conference, COSIT 2007, Melbourne, Australia, September 19-23, 2007, Proceedings, pp 1–19Google Scholar
- Matheron G (1962) Traité de géostatistique appliquée. Mémoires du Bureau de Recherches Géologiques et Minières 14Google Scholar
- Miller G, Johnson-Laird P (1976) Language and Perception. Cambridge University Press, CambridgeGoogle Scholar
- Morrow D, Clark H (1988) Interpreting words in spatial descriptions. Lang Cogn Process 3:275–291CrossRefGoogle Scholar
- Mukerjee A, Gupta K, Nautiyal S, Singh M, Mishra N (2000) Conceptual description of visual scenes from linguistic models. Image Vis Comput 18(2):173–187CrossRefGoogle Scholar
- Parsons S (1996) Current approaches to handling imperfect information in data and knowledge bases. Knowl Data Eng 3(8):353–372CrossRefGoogle Scholar
- Pfoser D, Tryfona N, Jensen C (2005) Indeterminacy and spatiotemporal data: basic definitions and case study. GeoInformatica 9(3):211–236CrossRefGoogle Scholar
- Power C, Simms A, White R (2001) Hierarchical fuzzy pattern matching for the regional comparison of land use maps. Int J Geogr Sci 15:77–100CrossRefGoogle Scholar
- Purves R, Clough P, Joho H (2005) Identifying imprecise regions for geographic information retrieval using the web. In: GISRUK’05, pp 313–318Google Scholar
- Randell D, Cui Z, Cohn A (1992) A spatial logic based on regions and connection. In: KR’92. Principles of knowledge representation and reasoning: proceedings of the third international conference. Morgan Kaufmann, pp 165–176Google Scholar
- Raubal M, Worboys M (1999) A formal model of the process of wayfinding in built environments. In: Spatial information theory. cognitive and computational foundations of geographic information science: international conference COSIT’99, Stade, Germany, August 1999. Proceedings, Springer, Berlin/Heidelberg. Lect Notes Comput Sci 1661:748Google Scholar
- Ravin Y, Wacholder N (1996) Extracting names from natural-language text. Tech. Rep. Report 20338, IBM ResearchGoogle Scholar
- Robinson V (2000) Individual and multipersonal fuzzy spatial relations acquired using human–machine interaction. Fuzzy Sets Syst 113(1):133–145CrossRefGoogle Scholar
- Robinson V (2003) A perspective on the fundamentals of fuzzy sets and their use in geographic information systems. Trans GIS 7(1):3–30CrossRefGoogle Scholar
- Sapir E (1929) The status of linguistics as a science. Language 5Google Scholar
- Schneider M (1996) Modelling spatial objects with undetermined boundaries using the realm/rose approach. In: Geographic objects with indeterminate boundaries, vol 2. Taylor and Francis, London, pp 141–152Google Scholar
- Schneider M (2000) Finite resolution crisp and fuzzy spatial objects. In: International symposium on spatial data handling, pp 3–17Google Scholar
- Schneider M (2001) A design of topological predicates for complex crisp and fuzzy regions. In: Conceptual modeling—ER 2001, Springer, Berlin/Heidelberg. Lect Notes Comput Sci 2224:103Google Scholar
- Schockaert S, de Cock M, Kerre E (2008) Location approximation for local search services using natural language hints. Int J Geogr Inf Sci 22(3):315–336CrossRefGoogle Scholar
- Smart P, Jones C, Twaroch F (2010) Multi-source toponym data integration and mediation for a meta-gazetteer service. In: GISCIENCE 2010 (forthcoming)Google Scholar
- Smith B, Varzi A (1997) Fiat and bona fide boundaries: towards an ontology of spatially extended objects. In: Spatial Information theory: a theoretical basis for GIS (COSIT’97). Lect Notes Comput Sci 1329:103–119Google Scholar
- Smith D, Crane G (2001) Disambiguating geographic names in a historical digital library. In: Research and advanced technology for digital libraries: fifth European conference (ECDL 2001), pp 127–136Google Scholar
- Srihari R, Rapaport W (1990) Combining linguistic and pictorial information: using captions to interpret newspaper photographs. In: Current trends in SNePS—semantic network processing system, no. 437/1990 in Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, pp 85–96Google Scholar
- Steiniger S, Meier S (2004) Snakes: a technique for line smoothing and displacement in map generalisation. In: ICA workshop on generalisation and multiple representationGoogle Scholar
- Talmy L (1983) How language structures space. In: Spatial orientation. Plenum, New York, pp 225–282Google Scholar
- Tang X (2004) Spatial object model[l]ing in fuzzy topological spaces : with applications to land cover change. PhD thesis, University of Twente, EnschedeGoogle Scholar
- Terzopoulos D (1986) Regularization of inverse visual problems involving disontinuities. IEEE Transactions PAMI-8, p 413Google Scholar
- Tversky B, Lee P (1998) How space structures language. In: Spatial cognition: an interdisciplinary approach to representing and processing spatial knowledge, Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, p 157Google Scholar
- Vorwerg C, Rickheit G (1998) Typicality effects in the categorization of spatial relations. In: Spatial cognition: an interdisciplinary approach to representing and processing spatial knowledge, Lecture Notes in Computer Science, vol 1404, Springer Berlin/Heidelberg, pp 203–222Google Scholar
- Wang F, Hall G (1996) Fuzzy representation of geographical boundaries in gis. Int J Geogr Inf Sci 10(5):573–590CrossRefGoogle Scholar
- Whorf B, Carroll J, Chase S (1956) Language, thought, and reality: selected writings of Benjamin Lee Whorf. MIT, CambridgeGoogle Scholar
- Wieczorek J, Guo Q, Hiimans R (2004) The point-radius method for georeferencing locality descriptions and calculating associated uncertainty. Int J Geogr Inf Sci 18(8):745–767CrossRefGoogle Scholar
- Winter S (2000) Uncertain topological relations between imprecise regions. Int J Geogr Inf Sci 14(5):411–430CrossRefGoogle Scholar
- Worboys M (2001) Nearness relations in environmental space. Int J Geogr Inf Sci 15(7):633–651CrossRefGoogle Scholar
- Worboys M, Duckham M, Kulik L (2004) Commonsense notions of proximity and direction in environmental space. Spatial Cogn Comput 4(4):285–312CrossRefGoogle Scholar
- Xie X, Mirmehdi M (2006) Magnetostatic field for the active contour model: a study in convergence. In: Proceedings of the 17th British machine vision conference, pp 127–136Google Scholar
- Yamada A, Yamamoto T, Ikeda H, Nishida T, Doshita S (1992) Reconstructing spatial image from natural language texts. In: Proceedings COLING-92, vol 4, pp 1279–1283Google Scholar
- Zadeh L (1965) Fuzzy sets. Inf Control 8:338–353CrossRefGoogle Scholar