An agent for learning new natural language commands

  • Amos AzariaEmail author
  • Shashank Srivastava
  • Jayant Krishnamurthy
  • Igor Labutov
  • Tom M. Mitchell


Teaching via natural language is an intuitive way for end users to add functionality to a virtual assistant, enabling them to personalize their assistant with new commands without requiring the intervention of the system developer, who cannot possibly anticipate all of an end user’s needs. In this paper we introduce our Learning by Instruction Agent (LIA), the first virtual assistant, for an email domain, that is capable of learning how to perform new commands taught by end users in natural language. LIA grounds the semantics of each command in terms of primitive executable procedures. When a user provides LIA with a command that it does not understand, it prompts the user to explain the command through a sequence of natural language steps. From this input, LIA learns the meaning of the new command and how to generalize the command to novel situations. For example, having been taught how to “forward an email to Alice”, it can correctly understand “forward this email to Bob”. We show that users that were assigned to interact with LIA completed the task quicker than users assigned to interact with a non-learning agent. These results demonstrate the potential of natural language teaching to improve the capabilities of intelligent personal assistants. We annotated 4759 natural language statements with their associated computer readable execution commands (logical forms) to form a dataset (which we publicize in this paper). We present the performance of several different parser methods on this dataset.


Human–agent interaction Human–computer interaction Agents learning from humans Natural language processing Machine learning 



This work was supported in part by Samsung GRO, Verizon (Yahoo!) through CMU’s InMind project [7] and, the Ministry of Science Technology & Space, Israel, and DARPA under Contract No. FA8750-13-2-0005.


  1. 1.
    Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning (p. 1). ACM.Google Scholar
  2. 2.
    Allen, J., Chambers, N., Ferguson, G., Galescu, L., Jung, H., Swift, M., & Taysom, W. (2007). Plow: A collaborative task learning agent. In Proceedings of the national conference on artificial intelligence (Vol. 22, p. 1514). Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.Google Scholar
  3. 3.
    Amir, O., & Gal, Y. K. (2013). Plan recognition and visualization in exploratory learning environments. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(3), 16.Google Scholar
  4. 4.
    Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009). A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5), 469–483.CrossRefGoogle Scholar
  5. 5.
    Artzi, Y., Lee, K., & Zettlemoyer, L. (2015). Broad-coverage ccg semantic parsing with AMR. In EMNLP (pp. 1699–1710).Google Scholar
  6. 6.
    Azaria, A., Gal, Y., Kraus, S., & Goldman, C. V. (2016). Strategic advice provision in repeated human-agent interactions. Autonomous Agents and Multi-agent Systems, 30(1), 4–29.CrossRefGoogle Scholar
  7. 7.
    Azaria, A., & Hong, J. (2016). Recommender system with personality. In RecSys (pp. 207–210).Google Scholar
  8. 8.
    Azaria, A., Krishnamurthy, J., & Mitchell, T. M. (2016). Instructable intelligent personal agent. In Thirtieth AAAI conference on artificial intelligence.Google Scholar
  9. 9.
    Azaria, A., Richardson, A., & Kraus, S. (2015). An agent for deception detection in discussion based environments. In Proceedings of the 18th ACM conference on computer supported cooperative work and social computing (pp. 218–227). ACM.Google Scholar
  10. 10.
    Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the international conference on learning representations (ICLR).Google Scholar
  11. 11.
    Berant, J., Chou, A., Frostig, R., & Liang, P. (2013). Semantic parsing on freebase from question-answer pairs. In EMNLP (Vol. 2, p. 6).Google Scholar
  12. 12.
    Biermann, A. W. (1983). Natural language programming. Berlin: Springer.CrossRefGoogle Scholar
  13. 13.
    Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 1371–1394). Berlin: Springer. Google Scholar
  14. 14.
    Branavan, S. R., Chen, H., Zettlemoyer, L. S., & Barzilay, R. (2009). Reinforcement learning for mapping instructions to actions. In Proceedings of the joint conference of the 47th annual meeting of the ACL and the 4th international joint conference on natural language processing of the AFNLP: Volume 1-volume 1 (pp. 82–90). Association for Computational Linguistics.Google Scholar
  15. 15.
    Brewer, R., Morris, M. R., & Piper, A. M. (2016). Why would anybody do this?: Understanding older adults’ motivations and challenges in crowd work. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2246–2257). ACM.Google Scholar
  16. 16.
    Buntain, C., Azaria, A., & Kraus, S. (2014). Leveraging fee-based, imperfect advisors in human-agent games of trust. In Twenty-eighth AAAI conference on artificial intelligence.Google Scholar
  17. 17.
    Calinon, S., Guenter, F., & Billard, A. (2007). On learning, representing, and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 37(2), 286–298.CrossRefGoogle Scholar
  18. 18.
    Cantrell, R., Benton, J., Talamadupula, K., Kambhampati, S., Schermerhorn, P., & Scheutz, M. (2012). Tell me when and why to do it! run-time planner model updates via natural language instruction. In 2012 7th ACM/IEEE international conference on human–robot interaction (HRI) (pp. 471–478). IEEE.Google Scholar
  19. 19.
    Chalamish, M., Sarne, D., & Kraus, S. (2007). Mass programmed agents for simulating human strategies in large scale systems. In Proceedings of the 6th international joint conference on autonomous agents and multiagent systems (p. 135). ACM.Google Scholar
  20. 20.
    Chen, D. L., & Mooney, R. J. (2008). Learning to sportscast: A test of grounded language acquisition. In Proceedings of the 25th international conference on machine learning (pp. 128–135). ACM.Google Scholar
  21. 21.
    Chen, D. L., & Mooney, R. J. (2011). Learning to interpret natural language navigation instructions from observations (pp. 859–865). San Francisco, CA.Google Scholar
  22. 22.
    Chkroun, M., & Azaria, A. (2019). Lia: A virtual assistant that can be taught new commands by speech. International Journal of Human-Computer Interaction, 35(17), 1596–1607.CrossRefGoogle Scholar
  23. 23.
    Clancey, W. J. (1983). The epistemology of a rule-based expert system-a framework for explanation. Artificial intelligence, 20(3), 215–251.CrossRefGoogle Scholar
  24. 24.
    Clarke, J., Goldwasser, D., Chang, M.-W., & Roth, D. (2010). Driving semantic parsing from the world’s response. In CoNLL (pp. 18–27). Association for Computational Linguistics.Google Scholar
  25. 25.
    Druck, G., Mann, G., & McCallum, A. (2008). Learning from labeled features using generalized expectation criteria. In Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval (pp. 595–602). ACM.Google Scholar
  26. 26.
    Haas, N., & Hendrix, G. G. (1980). An approach to acquiring and applying knowledge. In AAAI (pp. 235–239).Google Scholar
  27. 27.
    Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018). A data-driven analysis of workers’ earnings on amazon mechanical turk. In Proceedings of the 2018 CHI conference on human factors in computing systems (p. 449). ACM.Google Scholar
  28. 28.
    Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1), 335–346.CrossRefGoogle Scholar
  29. 29.
    Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121.CrossRefGoogle Scholar
  30. 30.
    Knox, W. B., & Stone, P. (2009). Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture (p. 9–16). ACM.Google Scholar
  31. 31.
    Koedinger, K. R., Aleven, V., Heffernan, N., McLaren, B., & Hockenberry, M. (2004). Opening the door to non-programmers: Authoring intelligent tutor behavior by demonstration. In J. C. Lester, R. M. Vicari, & F. Paraguaçu (Eds.), Intelligent tutoring systems (pp. 162–174). Berlin: Springer.CrossRefGoogle Scholar
  32. 32.
    Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques. Cambridge: MIT press.zbMATHGoogle Scholar
  33. 33.
    Krishnamurthy, J., & Kollar, T. (2013). Jointly learning to parse and perceive: Connecting natural language to the physical world. Transactions of Association for Computational Linguistics, 1, 193–206.CrossRefGoogle Scholar
  34. 34.
    Krishnamurthy, J., & Mitchell, T. M. (2012). Weakly supervised training of semantic parsers. In EMNLP-CoNLL (pp. 754–765).Google Scholar
  35. 35.
    Kuklinski, K., Fischer, K., Marhenke, I., Kirstein, F., Solvason, D., Kruger, N., Savarimuthu, T. R., et al. (2014). Teleoperation for learning by demonstration: Data glove versus object manipulation for intuitive robot control. In 2014 6th international congress on Ultra modern telecommunications and control systems and workshops (ICUMT) (pp. 346–351). IEEE.Google Scholar
  36. 36.
    Kwiatkowski, T., Choi, E., Artzi, Y., & Zettlemoyer, L. S. (2013). Scaling semantic parsers with on-the-fly ontology matching. In ACL.Google Scholar
  37. 37.
    Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., & Steedman, M. (2010). Inducing probabilistic ccg grammars from logical form with higher-order unification. In EMNLP (pp. 1223–1233).Google Scholar
  38. 38.
    Le, V., Gulwani, S., & Su, Z. (2013). Smartsynth: Synthesizing smartphone automation scripts from natural language. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services (pp. 193–206). ACM.Google Scholar
  39. 39.
    Li, T. J.-J., Azaria, A., & Myers, B. A. (2017). Sugilite: Creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 6038–6049). ACM.Google Scholar
  40. 40.
    Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014). The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: System demonstrations (pp. 55–60).Google Scholar
  41. 41.
    Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A joint model of language and perception for grounded attribute learning. arXiv preprint arXiv:1206.6423.
  42. 42.
    Myers, B. A., Cypher, A., Maulsby, D., Smith, D. C., & Shneiderman, B. (1991). Demonstrational interfaces: Coming soon? In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 393–396). ACM.Google Scholar
  43. 43.
    Nakaoka, S., Nakazawa, A., Kanehiro, F., Kaneko, K., Morisawa, M., Hirukawa, H., et al. (2007). Learning from observation paradigm: Leg task models for enabling a biped humanoid robot to imitate human dances. The International Journal of Robotics Research, 26(8), 829–844.CrossRefGoogle Scholar
  44. 44.
    Pasupat, P., & Liang, P. (2015). Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305.
  45. 45.
    Price, B., & Boutilier, C. (2003). Accelerating reinforcement learning through implicit imitation. Journal of Artificial Intelligence Research, 19, 569–629.CrossRefGoogle Scholar
  46. 46.
    Quirk, C., Mooney, R., & Galley, M. (2015). Language to code: Learning semantic parsers for if-this-then-that recipes. In Proceedings of the 53rd annual meeting of the association for computational linguistics (ACL-15) (pp. 878–888). Beijing, China.Google Scholar
  47. 47.
    Quirk, C., Mooney, R. J., & Galley, M. (2015). Language to code: Learning semantic parsers for if-this-then-that recipes. ACL, 1, 878–888.Google Scholar
  48. 48.
    Reed, A. (2010). Creating interactive fiction with inform 7. Boston: Cengage Learning.Google Scholar
  49. 49.
    Rosenfeld, A., Cohen, M., Taylor, M. E., & Kraus, S. (2018). Leveraging human knowledge in tabular reinforcement learning: A study of human subjects. The Knowledge Engineering Review, 33, e14.CrossRefGoogle Scholar
  50. 50.
    Rosenfeld, A., & Kraus, S. (2018). Predicting human decision-making: From prediction to action. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(1), 1–150.CrossRefGoogle Scholar
  51. 51.
    Rosenfeld, A., Zuckerman, I., Segal-Halevi, E., Drein, O., & Kraus, S. (2016). Negochat-a: A chat-based negotiation agent with bounded rationality. Autonomous Agents and Multi-agent Systems, 30(1), 60–81.CrossRefGoogle Scholar
  52. 52.
    Sarafian, E., Tamar, A., & Kraus, S. (2019). Safe policy learning from observations. In The multi-disciplinary conference on reinforcement learning and decision making.Google Scholar
  53. 53.
    Segal, A., David, Y. B., Williams, J. J., Gal, K., & Shalom, Y. (2018). Combining difficulty ranking with multi-armed bandits to sequence educational content. In International conference on artificial intelligence in education (pp. 317–321). Springer.Google Scholar
  54. 54.
    She, L., Yang, S., Cheng, Y., Jia, Y., Chai, J., & Xi, N. (2014). Back to the blocks world: Learning new actions through situated human-robot dialogue. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL) (pp. 89–97).Google Scholar
  55. 55.
    Spielberg, Y., & Azaria, A. (2019). The concept of criticality in reinforcement learning. In International conference on tools with artificial intelligence.Google Scholar
  56. 56.
    Srivastava, S., Azaria, A., & Mitchell, T. (2017). Parsing natural language conversations using contextual cues. In IJCAI 2017.Google Scholar
  57. 57.
    Steedman, M., & Baldridge, J. (2011). Combinatory categorial grammar. In R. D. Borsley & K. Börjars (Eds.), Non-transformational syntax: Formal and explicit models of grammar (pp. 181–224). Wiley Online Library. Google Scholar
  58. 58.
    Thomason, J., Zhang, S., Mooney, R., & Stone, P. (2015). Learning to interpret natural language commands through human-robot dialog. In Proceedings of the twenty-fourth international joint conference on artificial intelligence (IJCAI).Google Scholar
  59. 59.
    Wang, Y., Berant, J., & Liang, P. (2015). Building a semantic parser overnight. In ACL.Google Scholar
  60. 60.
    Winograd, T. (1973). A procedural model of language understanding. In R. Schank & K. Colby (Eds.), Computer models of thought and language (pp. 152–186). New York: WH Freeman.Google Scholar
  61. 61.
    Wong, Y. W., & Mooney, R. J. (2007). Learning synchronous grammars for semantic parsing with lambda calculus. In ACL (Vol. 45, p. 960).Google Scholar
  62. 62.
    Zettlemoyer, L. S., & Collins, M. (2005). Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI ’05, proceedings of the 21st conference in uncertainty in artificial intelligence.Google Scholar
  63. 63.
    Zettlemoyer, L. S., & Collins, M. (2007). Online learning of relaxed ccg grammars for parsing to logical form. In EMNLP-CoNLL (pp. 678–687).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceAriel UniversityArielIsrael
  2. 2.Data Science CenterAriel UniversityArielIsrael
  3. 3.Microsoft ResearchRedmondUSA
  4. 4.Semantic MachinesBerkeleyUSA
  5. 5.LAER AINew YorkUSA
  6. 6.Machine Learning DepartmentCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations