Abstract
Dependency parsing is a popular approach for syntactic analysis of natural language utterances. It concerns building a dependency tree of the linguistic input relying only on a model of syntactic regularities. The cognitive process of human language processing, however, has also access to other sources of knowledge, like visual clues that can be used to improve language understanding.
In this paper, we approach integrating visual context and linguistic information to improve the reliability of dependency parsing. To achieve this goal, we modify a state-of-the-art dependency parser to make it accept visual information as extra features in addition to the original linguistic input. All these inputs (features) are considered in the learning process of the trained model. Experiments have been carried out to investigate the contribution of this additional multimodal information on ambiguity resolution and parsing quality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baumgärtner, C.: On-line cross-modal context integration for natural language parsing. Ph.D. thesis Universität Hamburg, Hamburg (2013)
Bohnet, B.: Very high accuracy and fast dependency parsing is not a contradiction. In: Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pp. 89–97. Beijing (2010)
Chu, Y.J., Liu, T.H.: On the shortest arborescence of a directed graph. Sci. Sinica 14, 1396–1400 (1965)
Crammer, K., Dekel, O., Shalev-Shwartz, S., Singer, Y.: Online passive-aggressive algorithms. J. Mach. Learn. Res. 7, 551–585 (2006)
de Marneffe, M.-C., Manning, C.D.: Stanford typed dependencies manual, 1 April 2015. Retrieved from http://nlp.stanford.edu/pubs/dependencies-coling08.pdf
Knoeferle, P.: The role of visual scenes in spoken language comprehension: evidence from eye-tracking. Saarlandes: Ph.D. thesis Universität des Saarlandes (2005)
Lei, T., Xin, Y., Zhang, Y., Barzilay, R., Jaakkola, T.: Low-rank tensors for scoring dependency structures. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL (2014)
Lei, T., Zhang, Y., Barzilay, R., Jaakkola, T.: RBGParser, 15 October 2015. Retrieved from github: https://github.com/taolei87/RBGParser
McCrae, P.: A model for the cross-modal influence of visual context upon language processing. In: The International Conference Recent Advances in Natural Language Processing, pp. 230–235 (2009)
Nivre, J.: Incrementality in deterministic dependency parsing. In: Proceeding Increment Parsing 2004 Proceedings of the Workshop on Incremental Parsing, pp. 50–57. Stroudsburg, PA, USA (2004)
Nivre, J., Hall, J., Nilsson, J.: Memory-based dependency parsing. In: Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL), Boston (2004)
Thomson, S., Kong, L., Martins, A.: ARK Syntactic & Semantic Parsing Demo, 1 December 2014. Retrieved from http://demo.ark.cs.cmu.edu/parse
Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguist. 2, 67–78 (2014)
Zhang, Y., Lei, T., Barzilay, R., Jaakkola, T., Globerson, A.: Steps to excellence: simple inference with refined scoring of dependency trees. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL, pp. 197–207 (2014a)
Zhang, Y., Lei, T., Barzilay, R., Jaakkola, T.: Greedis goodif randomized: new inference for dependency parsing. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1013–1024. Doha, Qatar: Association for Computational Linguistics (2014b)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Salama, A.R., Menzel, W. (2017). Multimodal Graph-Based Dependency Parsing of Natural Language. In: Hassanien, A., Shaalan, K., Gaber, T., Azar, A., Tolba, M. (eds) Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2016. AISI 2016. Advances in Intelligent Systems and Computing, vol 533. Springer, Cham. https://doi.org/10.1007/978-3-319-48308-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-48308-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-48307-8
Online ISBN: 978-3-319-48308-5
eBook Packages: EngineeringEngineering (R0)