Abstract
Referring Expression Generation (reg) algorithms, a core component of systems that generate text from non-linguistic data, seek to identify domain objects using natural language descriptions. While reg has often been applied to visual domains, very few approaches deal with the problem of fuzziness and gradation. This paper discusses these problems and how they can be accommodated to achieve a more realistic view of the task of referring to objects in visual scenes.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Reiter, E., Dale, R.: Building Natural Language Generation Systems. Cambridge University Press, Cambridge (2000)
Kacprzyk, J., Zadrozny, S.: Computing with words is an implementable paradigm: Fuzzy queries, linguistic data summaries, and natural-language generation. IEEE Trans. Fuzzy Syst. 18(3), 461–472 (2010)
Marín, N., Sánchez, D.: On generating linguistic descriptions of time series. Fuzzy Sets Syst. 285, 6–30 (2016)
Ramos-Soto, A., Bugarín, A., Barro, S.: On the role of linguistic descriptions of data in the building of natural language generation systems. Fuzzy Sets Syst. 285, 31–51 (2016)
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Han, X., Berg, T., Daume III., H.: Midge: Generating Image Descriptions From Computer Vision Detections. In: EACL 2012, Avignon, France, pp. 747–756. Association for Computational Linguistics (2012)
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: Baby talk: Understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013)
Yatskar, M., Galley, M., Vanderwende, L., Zettlemoyer, L.: See No Evil, Say No Evil : Description Generation from Densely Labeled Images. In: Proceedings of the Third Joint Conference on Lexical and Computation Semantics (*SEM) (2014)
Reiter, E., Dale, R.: A fast algorithm for the generation of referring expressions. In: COLING 1992, pp. 232–238 (1992)
Krahmer, E., van Deemter, K.: Computational generation of referring expressions: A survey. Comput. Linguist. 38(1), 173–218 (2012)
Elsner, M., Rohde, H., Clarke, A.D.F.: Information structure prediction for visual-world referring expressions. In: EACL 2014, Gothenburg, Sweden, pp. 520–529. Association for Computational Linguistics (2014)
Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.L.: ReferItGame: referring to objects in photographs of natural scenes. In: EMNLP 2014, Doha, Qatar, pp. 787–798. Association for Computational Linguistics (2014)
van Deemter, K.: Generating referring expressions that involve gradable properties. Comput. Linguist. 32(2), 195–222 (2006)
Turner, R., Sripada, S., Reiter, E., Davy, I.P.: Selecting the content of textual descriptions of geographically located events in spatio-temporal weather data. In: Ellis, R., Allen, T., Petridis, M. (eds.) Applications and Innovations in Intelligent Systems XV, pp. 75–88. Springer, London (2008)
Reiter, E., Sripada, S., Hunter, J., Yu, J., Davy, I.: Choosing words in computer-generated weather forecasts. Artif. Intell. 167(1–2), 137–169 (2005)
Cadenas, J.T., Marín, N., Vila, M.A.: Context-aware fuzzy databases. Appl. Soft Comput. 25, 215–233 (2014)
Castillo-Ortega, R., Chamorro-Martínez, J., Marín, N., Sánchez, D., Soto-Hidalgo, J.M.: Describing images via linguistic features and hierarchical segmentation. In: FUZZ-IEEE 2010, pp. 1–8 (2010)
Soto-Hidalgo, J.M., Chamorro-Martínez, J., Sánchez, D.: A new approach for defining a fuzzy color space. In: FUZZ-IEEE 2010, pp. 1–6 (2010)
Chamorro-Martínez, J., Medina, J.M., Barranco, C.D., Galán-Perales, E., Soto-Hidalgo, J.M.: Retrieving images in fuzzy object-relational databases using dominant color descriptors. Fuzzy Sets Syst. 158(3), 312–324 (2007)
Chamorro-Martínez, J., Martínez-Jiménez, P.M., Soto-Hidalgo, J.M., León-Salas, A.: A fuzzy approach for modelling visual texture properties. Inf. Sci. 313, 1–21 (2015)
Chamorro-Martínez, J., Martínez-Jiménez, P.M., Soto-Hidalgo, J.M., Prados-Suárez, B.: Fuzzy sets on 2D spaces for fineness representation. Int. J. Approx. Reasoning 62, 46–60 (2015)
Prados-Suárez, B., Chamorro-Martínez, J., Sánchez, D., Abad, J.: Region-based fit of color homogeneity measures for fuzzy image segmentation. Fuzzy Sets Syst. 158(3), 215–229 (2007)
Hudelot, C., Atif, J., Bloch, I.: Fuzzy spatial relation ontology for image interpretation. Fuzzy Sets Syst. 159(15), 1929–1951 (2008)
Buck, A.R., Keller, J.M., Skubic, M.: A memetic algorithm for matching spatial configurations with the histograms of forces. IEEE Trans. Evol. Comput. 17(4), 588–604 (2013)
Dale, R.: Cooking up referring expressions. In: ACL 1989, Vancouver, BC, pp. 68–75. Association for Computational Linguistics (1989)
Grice, H.P.: Logic and conversation. In: Cole, P., Morgan, J.L. (eds.) Syntax and Semantics 3: Speech Acts, pp. 41–58. Elsevier, Amsterdam (1975)
Dale, R., Reiter, E.: Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Sci. 19(2), 233–263 (1995)
Gatt, A., Belz, A.: Introducing shared tasks to NLG: the TUNA shared task evaluation challenges. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS, vol. 5790, pp. 264–293. Springer, Heidelberg (2010)
Jordan, P.W., Walker, M.A.: Learning content selection rules for generating object descriptions in dialogue. J. Artif. Intell. Res. 24, 157–194 (2005)
Janarthanam, S., Lemon, O.: Adaptive generation in dialogue systems using dynamic user modeling. Computat. Linguist. 40(4), 883–920 (2014)
Castillo-Ortega, R., Marín, N., Sánchez, D., Tettamanzi, A.G.B.: Quality assessment in linguistic summaries of data. In: Greco, S., Bouchon-Meunier, B., Coletti, G., Fedrizzi, M., Matarazzo, B., Yager, R.R. (eds.) IPMU 2012, Part II. CCIS, vol. 298, pp. 285–294. Springer, Heidelberg (2012)
Bugarín, A., Marín, N., Sánchez, D., Triviño, G.: Aspects of quality evaluation in linguistic descriptions of data. In: FUZZ-IEEE 2015, pp. 1–8 (2015)
Gardent, C.: Generating minimal definite descriptions. In: ACL 2002, pp. 96–103, July 2002
Frank, M.C., Goodman, N.D.: Predicting pragmatic reasoning in language games. Science (New York, N.Y.) 336(6084), 998 (2012)
Kelleher, J.D., Kruijff, G.: Incremental generation of spatial referring expressions in situated dialog. In: COLING-ACL 2006, Sydney, Australia, pp. 1041–1048. Association for Computational Linguistics (2006)
Pechmann, T.: Incremental speech production and referential overspecification. Linguistics (1989)
Belke, E., Meyer, A.S.: Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during “same”-“different” decisions. Eur. J. Cognit. Psychol. 14(2), 237–266 (2002)
Viethen, J., Goudbeek, M., Krahmer, E.: The impact of colour difference and colour codability on reference production. In: CogSci 2012, Austin, TX, pp. 1084–1089. Cognitive Science Society (2012)
van Gompel, R.P., Gatt, A., Krahmer, E., van Deemter, K.: Overspecification in reference: Modelling size contrast effects. In: AMLAP 2014 (2014)
Itti, L., Koch, C.: Computational modelling of visual attention. Nat. Rev. Neurosci. 2(3), 194–203 (2001)
Erdem, E., Erdem, A.: Visual saliency estimation by nonlinearly integrating features using region covariances. J. Vis. 13(4:11), 1–20 (2013)
Oliva, A., Torralba, A.: The role of context in object recognition. Trends Cogn. Sci. 11(12), 520–527 (2007)
Torralba, A., Oliva, A., Castelhano, M.S., Henderson, J.M.: Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychol. Rev. 113(4), 766–786 (2006)
Kanan, C., Tong, M.H., Zhang, L., Cottrell, F.W.: SUN: Top-down saliency using natural statistics. Vis. Cogn. 17(6–7), 979–1003 (2009)
Sedivy, J.C.: Pragmatic versus form-based accounts of referential contrast: evidence for effects of informativity expectations. J. Psycholinguist. Res. 32(1), 3–23 (2003)
Westerbeek, H., Koolen, R., Maes, A.: Stored object knowledge and the production of referring expressions: the case of color typicality. Frontiers Psychol. 6(July), 1–12 (2015)
Dale, R., Haddock, N.: Generating referring expressions involving relations. In: EACL 1991, Berlin, Germany, pp. 161–166 (1991)
Areces, C., Koller, A., Striegnitz, K.: Referring expressions as formulas of description logic. In: INLG 2008, pp. 42–49 (2008)
Gatt, A., Portet, F.: Multilingual generation of uncertain temporal expressions from data: A study of a possibilistic formalism and its consistency with human subjective evaluations. Fuzzy Sets Syst. 285, 73–93 (2016)
Acknowledgments
This work has been partially supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (FEDER) under project TIN2014-58227-P.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Gatt, A., Marín, N., Portet, F., Sánchez, D. (2016). The Role of Graduality for Referring Expression Generation in Visual Scenes. In: Carvalho, J., Lesot, MJ., Kaymak, U., Vieira, S., Bouchon-Meunier, B., Yager, R. (eds) Information Processing and Management of Uncertainty in Knowledge-Based Systems. IPMU 2016. Communications in Computer and Information Science, vol 610. Springer, Cham. https://doi.org/10.1007/978-3-319-40596-4_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-40596-4_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-40595-7
Online ISBN: 978-3-319-40596-4
eBook Packages: Computer ScienceComputer Science (R0)