Autonomous Robots

, Volume 38, Issue 1, pp 89–105 | Cite as

Provably correct reactive control from natural language

  • Constantine LignosEmail author
  • Vasumathi Raman
  • Cameron Finucane
  • Mitchell Marcus
  • Hadas Kress-Gazit


This paper presents an integrated system for generating, troubleshooting, and executing correct-by-construction controllers for autonomous robots using natural language input, allowing non-expert users to command robots to perform high-level tasks. This system unites the power of formal methods with the accessibility of natural language, providing controllers for implementable high-level task specifications, easy-to-understand feedback on those that cannot be achieved, and natural language explanation of the reason for the robot’s actions during execution. The natural language system uses domain-general components that can easily be adapted to cover the vocabulary of new applications. Generation of a linear temporal logic specification from the user’s natural language input uses a novel data structure that allows for subsequent mapping of logical propositions back to natural language, enabling natural language feedback about problems with the specification that are only identifiable in the logical form. We demonstrate the robustness of the natural language understanding system through a user study where participants interacted with a simulated robot in a search and rescue scenario. Automated analysis and user feedback on unimplementable specifications is demonstrated using an example involving a robot assistant in a hospital.


Natural language Formal methods High-level control Synthesis 



We would like to thank Taylor Turpen, Israel Geselowitz, and Kenton Lee for their assistance with software development and data collection. This work was supported in part by: ARO MURI (SUBTLE) W911NF-07-1-0216, NSF  CAREER  CNS-0953365, DARPA N66001-12-1-4250, and TerraSwarm, one of six centers of STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA.


  1. Berant, J., & Liang, P. (2014). Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1415–1425).Google Scholar
  2. Bhatia, A., Kavraki, L. E., & Vardi, M.Y. (2010). Sampling-based motion planning with temporal goals. In IEEE International Conference on Robotics and Automation (ICRA), IEEE (pp. 2689–2696).Google Scholar
  3. Biere, A. (2008). PicoSAT essentials. Journal on Satisfiability Boolean Modeling and Computation (JSAT), 4, 75–97.zbMATHGoogle Scholar
  4. Bikel, D. M. (2004). Intricacies of Collins’ parsing model. Computational Linguistics, 30(4), 479–511.CrossRefzbMATHGoogle Scholar
  5. Bobadilla, L., Sanchez, O., Czarnowski, J., Gossman, K., & LaValle, S. (2011). Controlling wild bodies using linear temporal logic. In Robotics: Science and Systems (RSS).Google Scholar
  6. Brooks, D., Lignos, C., Finucane, C., Medvedev, M., Perera, I., Raman, V., Kress-Gazit, H., Marcus, M., & Yanco, H. (2012). Make it so: Continuous, flexible natural language interaction with an autonomous robot. In Proceedings of the Grounding Language for Physical Systems Workshop at the 76th AAAI Conference on Artificial Intelligence.Google Scholar
  7. Chen, D. L., & Mooney, R.J. (2011). Learning to interpret natural language navigation instructions from observations. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 859–865).Google Scholar
  8. Cizelj, I., & Belta, C. (2013). Negotiating the probabilistic satisfaction of temporal logic motion specifications. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4320–4325).Google Scholar
  9. Clarke, E. M., Grumberg, O., & Peled, D. A. (1999). Model checking. Cambridge, MA: MIT Press.Google Scholar
  10. Dzifcak, J., Scheutz, M., Baral, C., & Schermerhorn, P. (2009). What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 4163–4168).Google Scholar
  11. Fainekos, G. E. (2011). Revising temporal logic specifications for motion planning. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 40–45).Google Scholar
  12. Finucane, C., Jing, G., & Kress-Gazit, H. (2010). LTLMoP: Experimenting with language, temporal logic and robot control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1988–1993).Google Scholar
  13. Gabbard, R., Marcus, M., & Kulick, S. (2006). Fully parsing the Penn Treebank. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (NAACL HLT) (pp. 184–191).Google Scholar
  14. Karaman, Frazzoli. (2009). Sampling-based motion planning with deterministic \(\mu \)-calculus specifications. In IEEE Conference on Decision and Control (CDC) (pp. 2222–2229).Google Scholar
  15. Kim, K., Fainekos, G. E., & Sankaranarayanan, S. (2012). On the revision problem of specification automata. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5171–5176).Google Scholar
  16. Kloetzer, M., & Belta, C. (2008). A fully automated framework for control of linear systems from temporal logic specifications. IEEE Transactions on Automatic Control, 53(1), 287–297.CrossRefMathSciNetGoogle Scholar
  17. Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2008). Translating structured english to robot controllers. Advanced Robotics, 22(12), 1343–1359.CrossRefGoogle Scholar
  18. Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2009). Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics, 25(6), 1370–1381.CrossRefGoogle Scholar
  19. Matuszek, C., Fox, D., & Koscher, K. (2010). Following directions using statistical machine translation. In Human-Robot Interaction (HRI) (pp. 251–258).Google Scholar
  20. Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A joint model of language and perception for grounded attribute learning. In Proceedings of the 29th International Conference on Machine Learning (ICML) (pp. 1671–1678).Google Scholar
  21. Matuszek, C., Herbst, E., Zettlemoyer, L., & Fox, D. (2013). Learning to parse natural language commands to a robot control system. Experimental Robotics, 88, 403–415.CrossRefGoogle Scholar
  22. Piterman, N., Pnueli, A., & Sa’ar, Y. (2006). Synthesis of reactive(1) designs. In Verification, Model Checking, and Abstract Interpretation (VMCAI) (pp. 364–380).Google Scholar
  23. Poon, H., & Domingos, P. (2009). Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1–10).Google Scholar
  24. Raman, V., & Kress-Gazit, H. (2011). Analyzing unsynthesizable specifications for high-level robot behavior using LTLMoP. In Computer Aided Verification (CAV) (pp. 663–668).Google Scholar
  25. Raman, V., & Kress-Gazit, H. (2013a). Explaining impossible high-level robot behaviors. IEEE Transactions on Robotics, 29, 94–104.CrossRefGoogle Scholar
  26. Raman, V., & Kress-Gazit, H. (2013b). Towards minimal explanations of unsynthesizability for high-level robot behaviors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 757–762).Google Scholar
  27. Raman, V., & Kress-Gazit, H. (2014). Unsynthesizable cores: Minimal explanations for unsynthesizable high-level robot behaviors. arXiv:1409.1455.
  28. Raman, V., Lignos, C., Finucane, C., Lee, KCT., Marcus, M., & Kress-Gazit, H. (2013). Sorry Dave, I’m afraid I can’t do that: Explaining unachievable robot tasks using natural language. In Robotics: Science and Systems (RSS).Google Scholar
  29. Schuler, K. (2005). Verbnet: A broad-coverage, comprehensive verb lexicon. PhD thesis, University of Pennsylvania.Google Scholar
  30. Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A. G., Teller, S. J., & Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 1507–1514).Google Scholar
  31. Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL HLT) - Volume 1 (pp. 173–180).Google Scholar
  32. Wongpiromsarn, T., Topcu, U., & Murray, R. M. (2010). Receding horizon control for temporal logic specifications. In Hybrid Systems: Computation and Control (HSCC) (pp. 101–110).Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Constantine Lignos
    • 1
    Email author
  • Vasumathi Raman
    • 2
  • Cameron Finucane
    • 3
  • Mitchell Marcus
    • 1
  • Hadas Kress-Gazit
    • 3
  1. 1.Department of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.Department of Computing and Mathematical SciencesCalifornia Institute of TechnologyPasadenaUSA
  3. 3.Sibley School of Mechanical and Aerospace EngineeringCornell UniversityIthacaUSA

Personalised recommendations