Extracting Planning Operators from Instructional Texts for Behaviour Interpretation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11117)


Recent attempts at behaviour understanding through language grounding have shown that it is possible to automatically generate planning models from instructional texts. One drawback of these approaches is that they either do not make use of the semantic structure behind the model elements identified in the text, or they manually incorporate a collection of concepts with semantic relationships between them. To use such models for behaviour understanding, however, the system should also have knowledge of the semantic structure and context behind the planning operators. To address this problem, we propose an approach that automatically generates planning operators from textual instructions. The approach is able to identify various hierarchical, spatial, directional, and causal relations between the model elements. This allows incorporating context knowledge beyond the actions being executed. We evaluated the approach in terms of correctness of the identified elements, model search complexity, model coverage, and similarity to handcrafted models. The results showed that the approach is able to generate models that explain actual tasks executions and the models are comparable to handcrafted models.


Planning operators Behaviour understanding Natural language processing 


  1. 1.
    Babeş-Vroman, M., et al.: Learning to interpret natural language instructions. In: Proceedings of the Workshop on Semantic Interpretation in an Actionable Context, Stroudsburg, PA, USA, pp. 1–6 (2012)Google Scholar
  2. 2.
    Baker, C., Saxe, R., Tenenbaum, J.: Action understanding as inverse planning. Cognition 113(3), 329–349 (2009)CrossRefGoogle Scholar
  3. 3.
    Benotti, L., Lau, T., Villalba, M.: Interpreting natural language instructions using language, vision, and behavior. ACM Trans. Interact. Intell. Syst. 4(3), 13:1–13:22 (2014)CrossRefGoogle Scholar
  4. 4.
    Branavan, S., Kushman, N., Lei, T., Barzilay, R.: Learning high-level planning from text. In: Proceedings of the Annual Meeting of Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 126–135 (2012)Google Scholar
  5. 5.
    Branavan, S., Zettlemoyer, L., Barzilay, R.: Reading between the lines: learning to map high-level instructions to commands. In: Proceedings of the Annual Meeting of Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 1268–1277 (2010)Google Scholar
  6. 6.
    Chen, D., Mooney, R.: Learning to interpret natural language navigation instructions from observations. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 859–865, August 2011Google Scholar
  7. 7.
    Diebold, F., Witman, K., Hanseman, D., Lysne, L., Moore, T.: Elements of Forecasting, 2nd edn. Cengage Learning, Boston (2000)Google Scholar
  8. 8.
    Goldwasser, D., Roth, D.: Learning from natural instructions. Mach. Learn. 94(2), 205–232 (2014)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Granger, C.: Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37(3), 424–438 (1969)CrossRefGoogle Scholar
  10. 10.
    Kirste, T., Krüger, F.: CCBM-a tool for activity recognition using computational causal behavior models. Technical report CS-01-12. Institut für Informatik, Universität Rostock, Rostock, Germany, May 2012Google Scholar
  11. 11.
    Kollar, T., Tellex, S., Roy, D., Roy, N.: Grounding verbs of motion in natural language commands to robots. In: Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics, vol. 79, pp. 31–47. Springer, Heidelberg (2014). Scholar
  12. 12.
    Krüger, F., Nyolt, M., Yordanova, K., Hein, A., Kirste, T.: Computational state space models for activity and intention recognition. A feasibility study. PLoS ONE 9(11), e109381 (2014)CrossRefGoogle Scholar
  13. 13.
    Li, X., Mao, W., Zeng, D., Wang, F.-Y.: Automatic construction of domain theory for attack planning. In: IEEE International Conference on Intelligence and Security Informatics, pp. 65–70, May 2010Google Scholar
  14. 14.
    Lindsay, A., Read, J., Ferreira, J., Hayton, T., Porteous, J., Gregory, P.: Framer: planning models from natural language action descriptions. In: International Conference on Automated Planning and Scheduling (2017)Google Scholar
  15. 15.
    Miller, G.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)CrossRefGoogle Scholar
  16. 16.
    Nguyen, T.A., Kambhampati, S., Do, M.: Synthesizing robust plans under incomplete domain models. In: Advances in Neural Information Processing Systems 26, pp. 2472–2480. Curran Associates Inc. (2013)Google Scholar
  17. 17.
    Philipose, M., Fishkin, K., Perkowitz, M., Patterson, D., Fox, D., Kautz, H., Hahnel, D.: Inferring activities from interactions with objects. IEEE Pervasive Comput. 3(4), 50–57 (2004)CrossRefGoogle Scholar
  18. 18.
    Ramirez, M., Geffner, H.: Goal recognition over POMDPs: inferring the intention of a POMDP agent. In: Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI 2011, Barcelona, Spain, vol. 3, pp. 2009–2014 (2011)Google Scholar
  19. 19.
    Sil, A., Yates, A.: Extracting strips representations of actions and events. In: Recent Advances in Natural Language Processing, Hissar, Bulgaria, pp. 1–8, September 2011Google Scholar
  20. 20.
    Tenorth, M., Nyga, D., Beetz, M.: Understanding and executing instructions for everyday manipulation tasks from the world wide web. In: IEEE International Conference on Robotics and Automation, pp. 1486–1491, May 2010Google Scholar
  21. 21.
    Veloso, M., Perez, A., Carbonell, J.: Nonlinear planning with parallel resource allocation. In: Proceedings of the DARPA Workshop of Innovative Approaches to Planning, Scheduling and Control (1990)Google Scholar
  22. 22.
    Vogel, A., Jurafsky, D.: Learning to follow navigational directions. In: Proceedings of the Annual Meeting of Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 806–814 (2010)Google Scholar
  23. 23.
    Webber, B., Badler, N., Eugenio, B., Geib, C., Levison, L., Moore, M.: Instructions, intentions and expectations. Artif. Intell. 73(1), 253–269 (1995)CrossRefGoogle Scholar
  24. 24.
    Ye, J., Dobson, S., McKeever, S.: Situation identification techniques in pervasive computing: a review. Pervasive Mob. Comput. 8(1), 36–66 (2012)CrossRefGoogle Scholar
  25. 25.
    Yordanova, K.: Discovering causal relations in textual instructions. In: Recent Advances in Natural Language Processing, Hissar, Bulgaria, pp. 714–720, September 2015Google Scholar
  26. 26.
    Yordanova, K.: Automatic generation of situation models for plan recognition problems. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, Varna, Bulgaria, pp. 823–830. INCOMA Ltd., September 2017Google Scholar
  27. 27.
    Yordanova, K.: A simple model for improving the performance of the Stanford Parser for action detection in textual instructions. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, Varna, Bulgaria, pp. 831–838. INCOMA Ltd., September 2017Google Scholar
  28. 28.
    Zhang, Z., Webster, P., Uren, V., Varga, A., Ciravegna, F.: Automatically extracting procedural knowledge from instructional texts using natural language processing. In: Proceedings of the International Conference on Language Resources and Evaluation, Istanbul, Turkey, pp. 520–527, May 2012Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of RostockRostockGermany

Personalised recommendations