Multimodal Graph-Based Dependency Parsing of Natural Language

  • Amr Rekaby SalamaEmail author
  • Wolfgang Menzel
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 533)


Dependency parsing is a popular approach for syntactic analysis of natural language utterances. It concerns building a dependency tree of the linguistic input relying only on a model of syntactic regularities. The cognitive process of human language processing, however, has also access to other sources of knowledge, like visual clues that can be used to improve language understanding.

In this paper, we approach integrating visual context and linguistic information to improve the reliability of dependency parsing. To achieve this goal, we modify a state-of-the-art dependency parser to make it accept visual information as extra features in addition to the original linguistic input. All these inputs (features) are considered in the learning process of the trained model. Experiments have been carried out to investigate the contribution of this additional multimodal information on ambiguity resolution and parsing quality.


Multimodal integration Graph-based dependency parsing RBG parser 


  1. Baumgärtner, C.: On-line cross-modal context integration for natural language parsing. Ph.D. thesis Universität Hamburg, Hamburg (2013)Google Scholar
  2. Bohnet, B.: Very high accuracy and fast dependency parsing is not a contradiction. In: Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pp. 89–97. Beijing (2010)Google Scholar
  3. Chu, Y.J., Liu, T.H.: On the shortest arborescence of a directed graph. Sci. Sinica 14, 1396–1400 (1965)MathSciNetzbMATHGoogle Scholar
  4. Crammer, K., Dekel, O., Shalev-Shwartz, S., Singer, Y.: Online passive-aggressive algorithms. J. Mach. Learn. Res. 7, 551–585 (2006)MathSciNetzbMATHGoogle Scholar
  5. de Marneffe, M.-C., Manning, C.D.: Stanford typed dependencies manual, 1 April 2015. Retrieved from
  6. Knoeferle, P.: The role of visual scenes in spoken language comprehension: evidence from eye-tracking. Saarlandes: Ph.D. thesis Universität des Saarlandes (2005)Google Scholar
  7. Lei, T., Xin, Y., Zhang, Y., Barzilay, R., Jaakkola, T.: Low-rank tensors for scoring dependency structures. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL (2014)Google Scholar
  8. Lei, T., Zhang, Y., Barzilay, R., Jaakkola, T.: RBGParser, 15 October 2015. Retrieved from github:
  9. McCrae, P.: A model for the cross-modal influence of visual context upon language processing. In: The International Conference Recent Advances in Natural Language Processing, pp. 230–235 (2009)Google Scholar
  10. Nivre, J.: Incrementality in deterministic dependency parsing. In: Proceeding Increment Parsing 2004 Proceedings of the Workshop on Incremental Parsing, pp. 50–57. Stroudsburg, PA, USA (2004)Google Scholar
  11. Nivre, J., Hall, J., Nilsson, J.: Memory-based dependency parsing. In: Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL), Boston (2004)Google Scholar
  12. Thomson, S., Kong, L., Martins, A.: ARK Syntactic & Semantic Parsing Demo, 1 December 2014. Retrieved from
  13. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguist. 2, 67–78 (2014)Google Scholar
  14. Zhang, Y., Lei, T., Barzilay, R., Jaakkola, T., Globerson, A.: Steps to excellence: simple inference with refined scoring of dependency trees. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL, pp. 197–207 (2014a)Google Scholar
  15. Zhang, Y., Lei, T., Barzilay, R., Jaakkola, T.: Greedis goodif randomized: new inference for dependency parsing. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1013–1024. Doha, Qatar: Association for Computational Linguistics (2014b)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of InformaticsUniversity of HamburgHamburgGermany

Personalised recommendations