Learning to Parse Natural Language Commands to a Robot Control System

  • Cynthia MatuszekEmail author
  • Evan Herbst
  • Luke Zettlemoyer
  • Dieter Fox
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 88)


As robots become more ubiquitous and capable of performing complex tasks, the importance of enabling untrained users to interact with them has increased. In response, unconstrained natural-language interaction with robots has emerged as a significant research area. We discuss the problem of parsing natural language commands to actions and control structures that can be readily implemented in a robot execution system. Our approach learns a parser based on example pairs of English commands and corresponding control language expressions. We evaluate this approach in the context of following route instructions through an indoor environment, and demonstrate that our system can learn to translate English commands into sequences of desired actions, while correctly capturing the semantic intent of statements involving complex control structures. The procedural nature of our formal representation allows a robot to interpret route instructions online while moving through a previously unknown environment.


Natural Language Lexical Item Robot Control Statistical Machine Translation Categorial Grammar 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Artzi, Y., Zettlemoyer, L.S.: Bootstrapping semantic parsers from conversations. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2011)Google Scholar
  2. 2.
    Beetz, M., Arbuckle, T., Belker, T., Bennewitz, M., Burgard, W., Cremers, A.B., Fox, D., Grosskreutz, H., Hähnel, D., Schulz, D.: Integrated plan-based control of autonomous service robots in human environments. IEEE Intelligent Systems 16(5) (2001)Google Scholar
  3. 3.
    Boutilier, C., Reiter, R., Soutchanski, M., Thrun, S.: Decision-theoretic, high-level agent programming in the situation calculus. In: Proc. of the National Conference on Artificial Intelligence, AAAI (2000)Google Scholar
  4. 4.
    Branavan, S.R.K., Zettlemoyer, L., Barzilay, R.: Reading between the lines: Learning to map high-level instructions to commands. In: ACL, pp. 1268–1277 (2010)Google Scholar
  5. 5.
    Burgard, W., Cremers, A.B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S.: Experiences with an interactive museum tour-guide robot. Artificial Intelligence 114(1-2), 3–55 (1999)zbMATHCrossRefGoogle Scholar
  6. 6.
    Chen, D.L.: Fast online lexicon learning for grounded language acquisition. In: Proc. of the Annual Meetings of the Association for Computational Linguistics, ACL (2012)Google Scholar
  7. 7.
    Chen, D.L., Mooney, R.: Learning to interpret natural language navigation instructions from observations. In: Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI 2011), pp. 859–865 (2011)Google Scholar
  8. 8.
    Dzifcak, J., Scheutz, M., Baral, C., Schermerhorn, P.: What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In: Proc. of the 2009 IEEE Int’l Conf. on Robotics and Automation, ICRA 2009 (2009)Google Scholar
  9. 9.
    Ferrein, A., Lakemeyer, G.: Logic-based robot control in highly dynamic domains. Robotics and Autonomous Systems 56(11) (2008)Google Scholar
  10. 10.
    Friedman, S., Pasula, H., Fox, D.: Voronoi random fields: Extracting topological structure of indoor environments via place labeling. In: IJCAI, pp. 2109–2114 (2007)Google Scholar
  11. 11.
    Hähnel, D., Burgard, W., Lakemeyer, G.: GOLEX - bridging the gap between logic (GOLOG) and a real robot. In: Proc. of the German Conference on Artificial Intelligence (KI), Germany (1998)Google Scholar
  12. 12.
    Ingrand, F., Chatila, R., Alami, R., Robert, F.: PRS: A high level supervision and control language for autonomous mobile robots. In: Proc. of the IEEE International Conference on Robotics & Automation, ICRA (1996)Google Scholar
  13. 13.
    Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: Proc. of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, CHI 2008. ACM (2008)Google Scholar
  14. 14.
    Kollar, T., Tellex, S., Roy, D., Roy, N.: Toward understanding natural language directions. In: HRI 2010: Proc. of the 5th Int’l Conf. on Human-Robot Interaction. ACM Press (2010)Google Scholar
  15. 15.
    Kress-Gazit, H., Fainekos, G., Pappas, G.: Translating structured english to robot controllers. Advanced Robotics 22(12) (2008)Google Scholar
  16. 16.
    Kress-Gazit, H., Wongpiromsarn, T., Topcu, U.: Correct, reactive robot control from abstraction and temporal logic specifications. IEEE Robotics and Automation Magazine, special issue on Formal Methods for Robotics and Automation 18(3), 65–74 (2011)CrossRefGoogle Scholar
  17. 17.
    Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., Steedman, M.: Inducing probabilistic CCG grammars from logical form with higher-order unification. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2010)Google Scholar
  18. 18.
    Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., Steedman, M.: Lexical generalization in CCG grammar induction for semantic parsing. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2011)Google Scholar
  19. 19.
    Lai, K., Bo, L., Ren, X., Fox, D.: A scalable tree-based approach for joint object and pose recognition. In: The AAAI Conference on Artificial Intelligence, AAAI (2011)Google Scholar
  20. 20.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  21. 21.
    Lison, P., Kruijff, G.-J.M.: An integrated approach to robust processing of situated spoken dialogue. In: Proc. of SRSL 2009, the 2nd Workshop on Semantic Representation of Spoken Language, Athens, Greece, pp. 58–65. Association for Computational Linguistics (March 2009)Google Scholar
  22. 22.
    Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., Fox, D.: A Joint Model of Language and Perception for Grounded Attribute Learning. In: Proc. of the 2012 International Conference on Machine Learning, Edinburgh, Scotland (June 2012)Google Scholar
  23. 23.
    Matuszek, C., Fox, D., Koscher, K.: Following directions using statistical machine translation. In: ACM/IEEE International Conference on Human-Robot Interaction, HRI (2010)Google Scholar
  24. 24.
    Mooney, R.: Learning to connect language and perception. In: Fox, D., Gomes, C.P. (eds.) Proc. of the Twenty-Third AAAI Conf. on Artificial Intelligence, AAAI 2008, Chicago, Illinois, pp. 1598–1601. AAAI Press (2008)Google Scholar
  25. 25.
    Shimizu, N., Haas, A.: Learning to Follow Navigational Route Instructions. In: Int’l Joint Conf. on Artificial Intelligence, IJCAI (2009)Google Scholar
  26. 26.
    Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In: EMNLP, pp. 254–263 (2008)Google Scholar
  27. 27.
    Steedman, M.: The Syntactic Process. MIT Press (2000)Google Scholar
  28. 28.
    Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., Roy, N.: Approaching the symbol grounding problem with probabilistic graphical models. AI Magazine 32(4) (2012)Google Scholar
  29. 29.
    Tsuruoka, Y., Tsujii, J.: Iterative CKY parsing for probabilistic context-free grammars. In: Su, K.-Y., Tsujii, J., Lee, J.-H., Kwong, O.Y. (eds.) IJCNLP 2004. LNCS (LNAI), vol. 3248, pp. 52–60. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  30. 30.
    Wei, Y., Brunskill, E., Kollar, T., Roy, N.: Where to go: Interpreting natural directions using global inference. In: Proc. of the IEEE International Conference on Robotics & Automation, ICRA (2009)Google Scholar
  31. 31.
    Zettlemoyer, L.S., Collins, M.: Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In: Proc. of the Conf. on Uncertainty in Artificial Intelligence (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Cynthia Matuszek
    • 1
    Email author
  • Evan Herbst
    • 1
  • Luke Zettlemoyer
    • 1
  • Dieter Fox
    • 1
  1. 1.University of WashingtonSeattleUSA

Personalised recommendations