Skip to main content
SpringerLink
Log in

Provably correct reactive control from natural language

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper presents an integrated system for generating, troubleshooting, and executing correct-by-construction controllers for autonomous robots using natural language input, allowing non-expert users to command robots to perform high-level tasks. This system unites the power of formal methods with the accessibility of natural language, providing controllers for implementable high-level task specifications, easy-to-understand feedback on those that cannot be achieved, and natural language explanation of the reason for the robot’s actions during execution. The natural language system uses domain-general components that can easily be adapted to cover the vocabulary of new applications. Generation of a linear temporal logic specification from the user’s natural language input uses a novel data structure that allows for subsequent mapping of logical propositions back to natural language, enabling natural language feedback about problems with the specification that are only identifiable in the logical form. We demonstrate the robustness of the natural language understanding system through a user study where participants interacted with a simulated robot in a search and rescue scenario. Automated analysis and user feedback on unimplementable specifications is demonstrated using an example involving a robot assistant in a hospital.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. This arguably unintuitive translation is due to specifications in LTLMoP being restricted to the GR(1) fragment of LTL.

  2. Even though it was the role of the operator, not the robot, to rescue hostages, we label these examples as tagging errors because a command was given to the system and it was not properly understood. The desired response in this situation is to understand the requested action but report that the robot cannot perform it.

References

  • Berant, J., & Liang, P. (2014). Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1415–1425).

  • Bhatia, A., Kavraki, L. E., & Vardi, M.Y. (2010). Sampling-based motion planning with temporal goals. In IEEE International Conference on Robotics and Automation (ICRA), IEEE (pp. 2689–2696).

  • Biere, A. (2008). PicoSAT essentials. Journal on Satisfiability Boolean Modeling and Computation (JSAT), 4, 75–97.

    MATH  Google Scholar 

  • Bikel, D. M. (2004). Intricacies of Collins’ parsing model. Computational Linguistics, 30(4), 479–511.

    Article  MATH  Google Scholar 

  • Bobadilla, L., Sanchez, O., Czarnowski, J., Gossman, K., & LaValle, S. (2011). Controlling wild bodies using linear temporal logic. In Robotics: Science and Systems (RSS).

  • Brooks, D., Lignos, C., Finucane, C., Medvedev, M., Perera, I., Raman, V., Kress-Gazit, H., Marcus, M., & Yanco, H. (2012). Make it so: Continuous, flexible natural language interaction with an autonomous robot. In Proceedings of the Grounding Language for Physical Systems Workshop at the 76th AAAI Conference on Artificial Intelligence.

  • Chen, D. L., & Mooney, R.J. (2011). Learning to interpret natural language navigation instructions from observations. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 859–865).

  • Cizelj, I., & Belta, C. (2013). Negotiating the probabilistic satisfaction of temporal logic motion specifications. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4320–4325).

  • Clarke, E. M., Grumberg, O., & Peled, D. A. (1999). Model checking. Cambridge, MA: MIT Press.

    Google Scholar 

  • Dzifcak, J., Scheutz, M., Baral, C., & Schermerhorn, P. (2009). What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 4163–4168).

  • Fainekos, G. E. (2011). Revising temporal logic specifications for motion planning. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 40–45).

  • Finucane, C., Jing, G., & Kress-Gazit, H. (2010). LTLMoP: Experimenting with language, temporal logic and robot control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1988–1993).

  • Gabbard, R., Marcus, M., & Kulick, S. (2006). Fully parsing the Penn Treebank. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (NAACL HLT) (pp. 184–191).

  • Karaman, Frazzoli. (2009). Sampling-based motion planning with deterministic \(\mu \)-calculus specifications. In IEEE Conference on Decision and Control (CDC) (pp. 2222–2229).

  • Kim, K., Fainekos, G. E., & Sankaranarayanan, S. (2012). On the revision problem of specification automata. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5171–5176).

  • Kloetzer, M., & Belta, C. (2008). A fully automated framework for control of linear systems from temporal logic specifications. IEEE Transactions on Automatic Control, 53(1), 287–297.

    Article  MathSciNet  Google Scholar 

  • Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2008). Translating structured english to robot controllers. Advanced Robotics, 22(12), 1343–1359.

    Article  Google Scholar 

  • Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2009). Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics, 25(6), 1370–1381.

    Article  Google Scholar 

  • Matuszek, C., Fox, D., & Koscher, K. (2010). Following directions using statistical machine translation. In Human-Robot Interaction (HRI) (pp. 251–258).

  • Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A joint model of language and perception for grounded attribute learning. In Proceedings of the 29th International Conference on Machine Learning (ICML) (pp. 1671–1678).

  • Matuszek, C., Herbst, E., Zettlemoyer, L., & Fox, D. (2013). Learning to parse natural language commands to a robot control system. Experimental Robotics, 88, 403–415.

    Article  Google Scholar 

  • Piterman, N., Pnueli, A., & Sa’ar, Y. (2006). Synthesis of reactive(1) designs. In Verification, Model Checking, and Abstract Interpretation (VMCAI) (pp. 364–380).

  • Poon, H., & Domingos, P. (2009). Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1–10).

  • Raman, V., & Kress-Gazit, H. (2011). Analyzing unsynthesizable specifications for high-level robot behavior using LTLMoP. In Computer Aided Verification (CAV) (pp. 663–668).

  • Raman, V., & Kress-Gazit, H. (2013a). Explaining impossible high-level robot behaviors. IEEE Transactions on Robotics, 29, 94–104.

    Article  Google Scholar 

  • Raman, V., & Kress-Gazit, H. (2013b). Towards minimal explanations of unsynthesizability for high-level robot behaviors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 757–762).

  • Raman, V., & Kress-Gazit, H. (2014). Unsynthesizable cores: Minimal explanations for unsynthesizable high-level robot behaviors. arXiv:1409.1455.

  • Raman, V., Lignos, C., Finucane, C., Lee, KCT., Marcus, M., & Kress-Gazit, H. (2013). Sorry Dave, I’m afraid I can’t do that: Explaining unachievable robot tasks using natural language. In Robotics: Science and Systems (RSS).

  • Schuler, K. (2005). Verbnet: A broad-coverage, comprehensive verb lexicon. PhD thesis, University of Pennsylvania.

  • Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A. G., Teller, S. J., & Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 1507–1514).

  • Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL HLT) - Volume 1 (pp. 173–180).

  • Wongpiromsarn, T., Topcu, U., & Murray, R. M. (2010). Receding horizon control for temporal logic specifications. In Hybrid Systems: Computation and Control (HSCC) (pp. 101–110).

Download references

Acknowledgments

We would like to thank Taylor Turpen, Israel Geselowitz, and Kenton Lee for their assistance with software development and data collection. This work was supported in part by: ARO MURI (SUBTLE) W911NF-07-1-0216, NSF  CAREER  CNS-0953365, DARPA N66001-12-1-4250, and TerraSwarm, one of six centers of STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA.

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, University of Pennsylvania, 3330 Walnut St., Philadelphia, PA, 19104, USA

    Constantine Lignos & Mitchell Marcus

  2. Department of Computing and Mathematical Sciences, California Institute of Technology, 1200 E California Blvd, Pasadena, CA, 91125, USA

    Vasumathi Raman

  3. Sibley School of Mechanical and Aerospace Engineering, Cornell University, 105 Upson Hall, Ithaca, NY, 14853, USA

    Cameron Finucane & Hadas Kress-Gazit

Authors
  1. Constantine Lignos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vasumathi Raman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cameron Finucane
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mitchell Marcus
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hadas Kress-Gazit
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Constantine Lignos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lignos, C., Raman, V., Finucane, C. et al. Provably correct reactive control from natural language. Auton Robot 38, 89–105 (2015). https://doi.org/10.1007/s10514-014-9418-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-014-9418-8

Keywords

Search

Navigation