Synthesizing Robot Programs with Interactive Tutor Mode
With the rapid development of the robotic industry, domestic robots have become increasingly popular. As domestic robots are expected to be personal assistants, it is important to develop a natural language-based human-robot interactive system for end-users who do not necessarily have much programming knowledge. To build such a system, we developed an interactive tutoring framework, named “Holert”, which can translate task descriptions in natural language to machine-interpretable logical forms automatically. Compared to previous works, Holert allows users to teach the robot by further explaining their intentions in an interactive tutor mode. Furthermore, Holert introduces a semantic dependency model to enable the robot to “understand” similar task descriptions. We have deployed Holert on an open-source robot platform, Turtlebot 2. Experimental results show that the system accuracy could be significantly improved by 163.9% with the support of the tutor mode. This system is also efficient. Even the longest task session with 10 sentences can be handled within 0.7 s.
KeywordsHuman-robot interaction semantic parsing program synthesis intelligent robotic systems natural language understanding
Unable to display preview. Download preview PDF.
This work was supported by Tsinghua University Initiative Scientific Research Program (No. 20141081140).
- C. Zhou, M. H. Jin, Y. C. Liu, Z. Zhang, Y. Liu, H. Liu. Singularity robust path planning for real time base attitude adjustment of free-floating space robot. International Journal of Automation and Computing, vol. 14, no. 2, pp. 169–178, 2017. DOI: https://doi.org/10.1007/s11633-017-1055-1.CrossRefGoogle Scholar
- K. C. D. Fu, Y. Nakamura, T. Yamamoto, H. Ishiguro. Analysis of motor synergies utilization for optimal movement generation for a human-like robotic arm. International Journal of Automation and Computing, vol. 10, no. 6, pp. 515–524, 2013. DOI: https://doi.org/10.1007/s11633-013-0749-2.CrossRefGoogle Scholar
- C. Datta, C. Jayawardena, I. H. Kuo, B. A. MacDonald. RoboStudio: A visual programming environment for rapid authoring and customization of complex services on a personal service robot. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal, pp. 2352–2357, 2012. DOI: https://doi.org/10.1109/IROS.2012.6386105.Google Scholar
- J. Dzifcak, M. Scheutz, C. Baral, P. Schermerhorn. What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In Proceedings of IEEE International Conference on Robotics and Automation, Kobe, Japan, pp. 4163–4168, 2009. DOI: https://doi.org/10.1109/ROBOT.2009.5152776.Google Scholar
- C. Matuszek, E. Herbst, L. Zettlemoyer, D. Fox. Learning to parse natural language commands to a robot control system. In Proceedings of the 13th International Symposium on Experimental Robotics, Springer, Heidelberg, Germany, 403–415, 2013. DOI: https://doi.org/10.1007/978-3-319-00065-7_28.CrossRefGoogle Scholar
- R. F. Ge, R. J. Mooney. A statistical semantic parser that integrates syntax and semantics. In Proceedings of the 9th Conference on Computational Natural Language Learning, Association for Computational Linguistics, Ann Arbor, USA, 9–16, 2005. DOI: https://doi.org/10.3115/1706543.1706546.Google Scholar
- Y. Artzi, L. Zettlemoyer. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Edinburgh, UK, 421–432, 2011.Google Scholar
- J. Berant, A. Chou, R. Frostig, P. Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of Conference on Empirical Methods in Natural Language Processing, Washington, USA, 2013.Google Scholar
- Y. N. Dauphin, D. Z. Hakkani-Tur, G. Tur, L. P. Heck. Deep learning for semantic parsing including semantic utterance classification, USA. Patent 20150310862, October 2015.Google Scholar
- M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng. ROS: An open-source robot operating system. In Proceedings of ICRA Workshop on Open Source Software, Kobe, Japan, 2009.Google Scholar
- A. Voutilainen. Part-of-speech tagging. The Oxford Handbook of Computational Linguistics, R. Mitkov, Ed., Oxford, UK: Oxford University Press, 219–232, 2003.Google Scholar
- D. Q. Chen, C. Manning. A fast and accurate dependency parser using neural networks. In Proceedings of Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Doha, Qatar, 2014.Google Scholar
- G. Look, B. Kottahachchi, R. Laddaga, H. Shrobe. A location representation for generating descriptive walking directions. In Proceedings of the 10th International Conference on Intelligent User Interfaces, ACM, San Diego, USA, 122–129, 2005. DOI: https://doi.org/10.1145/1040830.1040862.CrossRefGoogle Scholar
- G. Bugmann, E. Klein, S. Lauria, T. Kyriacou. Corpusbased robotics: A route instruction example. In Proceedings of the 8th International Conference on Intelligent Autonomous Systems, Amsterdam, Netherlands, 96–103, 2004.Google Scholar
- M. MacMahon, B. Stankiewicz, B. Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of National Conference on Artificial Intelligence, AAAI, Austin, UK, 2006.Google Scholar
- J. Thomason, S. Q. Zhang, R. J. Mooney, P. Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the 24th International Conference on Artificial Intelligence, AAAI, Buenos Aires, Argentina, 1923–1929, 2015.Google Scholar
- L. B. She, S. H. Yang, Y. Cheng, Y. Y. Jia, J. Y. Chai, N. Xi. Back to the blocks world: Learning new actions through situated human-robot dialogue. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Philadelphia, USA, 89–97, 2014.CrossRefGoogle Scholar
- E. Grefenstette, P. Blunsom, N. de Freitas, K. M. Hermann. A deep architecture for semantic parsing. In Proceedings of the ACL Workshop on Semantic Parsing, Association for Computational Linguistics, Baltimore, USA, 2014.Google Scholar
- B. Biggio, B. Nelson, P. Laskov. Support vector machines under adversarial label noise. In Proceedings of the 3rd Asian Conference on Machine Learning, Taoyuan, China, 97–112, 2011.Google Scholar