Cloud-Based Probabilistic Knowledge Services for Instruction Interpretation

Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 3)


As the tasks of autonomous manipulation robots get more complex, the tasking of the robots using natural-language instructions becomes more important. Executing such instructions in the way they are meant often requires robots to infer missing, and disambiguate given information using lots of common and commonsense knowledge. In this work, we report on Probabilistic Action Cores (Prac) (Nyga and Beetz, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012) – a framework for learning of and reasoning about action-specific probabilistic knowledge bases that can be learned from hand-labeled instructions to address this problem. In Prac, knowledge about actions and objects is compactly represented by first-order probabilistic models, which are used to learn a joint probability distribution over the ways in which instructions for a given action verb are formulated. These joint probability distributions are then used to compute the plan instantiation that has the highest probability of producing the intended action given the natural language instruction. Formulating plan interpretation as a conditional probability is a promising approach because we can at the same time infer the plan that is most appropriate for performing the instruction, the refinement of the parameters of the plan on the basis of the information given in the instruction, and automatically fill in missing parameters by inferring their most probable value from the distribution. Prac has been implemented as a web-based online service on the cloud-robotics platform openEASE [7].



This work is supported by the EU FP7 Projects RoboHow (grant number 288533) and ACAT (grant number 600578).


  1. 1.
    Anderson, J.: Constraint-directed improvisation for everyday activities. Ph.D. thesis (1995)Google Scholar
  2. 2.
    Artzi, Y., Zettlemoyer, L.: Weakly supervised learning of semantic parsers for mapping instructions to actions. Trans. Assoc. Comput. Linguist. 1(1), 49–62 (2013)Google Scholar
  3. 3.
    Bailey, D.: When push comes to shove: A computational model of the role of motor control in the acquisition of action verbs. Ph.D. thesis, University of California (1997)Google Scholar
  4. 4.
    Baker, C.F., Fillmore, C.J., Lowe, J.B.: The berkeley framenet project. Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics -. ACL ’98, vol. 1, pp. 86–90. Association for Computational Linguistics, Stroudsburg, PA, USA (1998)Google Scholar
  5. 5.
    Barker, R.: Ecological Psychology: Concepts and Methods for Studying the Environment of Human Behavior. Stanford University Press, Stanford (1968)Google Scholar
  6. 6.
    Beetz, M., Jain, D., Mösenlechner, L., Tenorth, M., Kunze, L., Blodow, N., Pangercic, D.: Cognition-enabled autonomous robot control for the realization of home chore task intelligence. Proc. IEEE 100(8), 2454–2471 (2012)CrossRefGoogle Scholar
  7. 7.
    Beetz, M., Tenorth, M., Winkler, J.: Open-EASE – a knowledge processing service for robots and robotics/ai researchers. In: IEEE International Conference on Robotics and Automation (ICRA), Seattle, Washington, USA (2015). (Finalist for the Best Cognitive Robotics Paper Award)Google Scholar
  8. 8.
    De Marneffe, M., MacCartney, B., Manning, C.: Generating typed dependency parses from phrase structure parses. Proc. LREC 6, 449–454 (2006)Google Scholar
  9. 9.
    de Marneffe, M.-C., Manning, C.D.: The stanford typed dependencies representation. In: Coling 2008: Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation, CrossParser ’08, pp. 1–8. Stroudsburg, PA, USA (2008) (Association for Computational Linguistics)Google Scholar
  10. 10.
    Dzifcak, J., Scheutz, M., Baral, C., Schermerhorn, P.: What to do and how to do it: translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In: IEEE International Conference on Robotics and Automation, 2009. ICRA’09, pp. 4163–4168. IEEE (2009)Google Scholar
  11. 11.
    Feldman, J., Narayanan, S.: Embodied meaning in a neural theory of language. Brain Lang. 89(2), 385–392 (2004) (Language and MotorIntegration)Google Scholar
  12. 12.
    Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  13. 13.
    Firby, J.: Adaptive execution in complex dynamic Worlds. Technical report 672, Yale University, Department of Computer Science (1989)Google Scholar
  14. 14.
    Georgeff, M., Ingrand, F.: Decision making in an embedded reasing system. In: Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, pp. 972–978. Detroit, MI (1989)Google Scholar
  15. 15.
    Kim, J., Mooney, R.J.: Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 433–444. Association for Computational Linguistics (2012)Google Scholar
  16. 16.
    Matuszek, C., Fox, D., Koscher, K.: Following directions using statistical machine translation. In: Proceeding of the 5th ACM/IEEE International Conference on Human-robot Interaction, pp. 251–258. ACM (2010)Google Scholar
  17. 17.
    Minsky, M.: A framework for representing knowledge. Technical Report Memo 306, MIT-AI Laboratory (1974)Google Scholar
  18. 18.
    Misra, D.K., Sung, J., Lee, K., Saxena, A.: Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. In: Proceedings of Robotics Science and Systems, Berkeley, USA (2014)Google Scholar
  19. 19.
    Mösenlechner, L., Beetz, M.: Parameterizing actions to have the appropriate effects. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), San Francisco, CA, USA (2011). Accessed 25–30 Sept 2011Google Scholar
  20. 20.
    Neo, E., Sakaguchi, T., Yokoi, K.: A natural language instruction system for humanoid robots integrating situated speech recognition, visual recognition and on-line whole-body motion generation. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2008. AIM 2008, pp. 1176–1182. IEEE (2008)Google Scholar
  21. 21.
    Nyga, D., Beetz, M.: Everything robots always wanted to know about housework (but were afraid to ask). In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal (2012). Accessed 7–12 Oct 2012Google Scholar
  22. 22.
    Nyga, D., Beetz, M.: Reasoning about unmodelled concepts – incorporating class taxonomies in probabilistic relational models (2015).
  23. 23.
    Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62(1–2), 107–136 (2006)CrossRefGoogle Scholar
  24. 24.
    Ryu, J., Jung, Y., Kim, K., Myaeng, S.: Automatic extraction of human activity knowledge from method-describing web articles. In: Proceedings of the 1st Workshop on Automated Knowledge Base Construction, p. 16 (2010)Google Scholar
  25. 25.
    Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., Roy, N.: Understanding natural language commands for robotic navigation and mobile manipulation. In: Proceedings of the National Conference on Artificial Intelligence (AAAI) (2011)Google Scholar
  26. 26.
    Tenenbaum, J.B., Kemp, C., Griffiths, T.L., Goodman, N.D.: How to grow a mind: statistics, structure, and abstraction. Science 331(6022), 1279–1285 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Tenorth,M., Nyga, D., Beetz, M.: Understanding and executing instructions for everyday manipulation tasks from the world wide web. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1486–1491. Anchorage, AK, USA (2010). Accessed 3–8 May 2010Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Institute for Artificial Intelligence, University of BremenBremenGermany

Personalised recommendations