Advertisement

User Modeling and User-Adapted Interaction

, Volume 1, Issue 2, pp 173–202 | Cite as

Recognizing intentions, interactions, and causes of plan failures

  • Gudula Retz-Schmidt 
Article

Abstract

Natural language systems for the description of image sequences have been developed (e.g. Neumann and Novak, 1986; Herzog et al., 1989). Even though these systems were used to verbalize the behaviour of human agents, they were limited in that they could only describe the purely visual, i.e. spatio-temporal, properties of the behaviour observed. For many applications of such systems (e.g. co-driver systems in traffic, expert systems in high-performance sports, tutorial systems that give “apprentices” instructions in construction tasks, etc.), it seems useful to extend their capabilities to cover a greater part of the performance of a human observer and thus make the system more helpful to the user. In particular, an interpretation process ought to be modelled that yields hypotheses about intentional entities from spatio-temporal information about agents. Its results should be verbalized in natural language.

This article presents an integrated approach for the recognition and natural language description of plans, intentions, interactions between multiple agents, plan failures, and causes of plan failures. The system described takes observations from image sequences as input. This type of input poses specific problems for the recognition process. After moving objects have been extracted from the image sequences by a vision system and spatio-temporal entities (such as spatial relations and events) have been recognized by an event-recognition system, a focussing process selects interesting agents to be concentrated on in the plan-recognition process. The set of plan hypotheses can be reduced by a hypothesis-selection component. Plan recognition serves as the basis for intention recognition, interaction recognition, and plan-failure analysis. The recognized intentional entities are described in natural language. The system is designed to extend the range of capabilities of the system SOCCER, which verbalizes real-world image sequences of soccer games in natural language.

Key words

observation of actions plan recognition intention recognition multiple agents recognition of interactions recognition and analysis of plan failures natural language description of intentional entities 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Allen, J. and D. Litman: 1986, ‘Plans, Goals, and Language’.Proceedings of the IEEE,74(7):939–947.Google Scholar
  2. Allen, J. and C. Perrault: 1980, ‘Analyzing Intention in Utterances’.Artificial Intelligence,15:143–179.Google Scholar
  3. Allen, J.: 1983,Recognizing Intentions from Natural Language Utterances. In: M. Brady and R. Berwick (eds.),Computational Models of Discourse, Cambridge, MA: MIT Press, pp. 107–166.Google Scholar
  4. Allen, J.: 1984, ‘Towards a General Theory of Action and Time’.Artificial Intelligence,23:123–154.Google Scholar
  5. André, E., G. Herzog and T. Rist: 1988, ‘On the Simultaneous Interpretation of Real World Image Sequences and their Natural Language Description: The System SOCCER’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 449–454.Google Scholar
  6. Anscombe, G.: 1957,Intention. Oxford: Basil Blackwell.Google Scholar
  7. Anscombe, G.: 1986, ‘Intention’. In: A. White (ed.),The Philosophy of Action, London: Oxford University Press, pp. 144–152.Google Scholar
  8. Appelt, D.: 1985,Planning English Sentences. Cambridge, England: Cambridge University Press.Google Scholar
  9. Audi, R.: 1973, ‘Intending’.Journal of Philosophy,70(13):387–403.Google Scholar
  10. Azarewicz, J. G. Fala, R. Fink and C. Heithecker: 1986, ‘Plan Recognition for Airborne Tactical Decision Making’. In:Proc. of the 5th National Conference of the American Association for Artificial Intelligence, Philadelphia, PA, pp. 805–811.Google Scholar
  11. Azarewicz, J. G. Fala, I. Hayslip and C. Heithecker: 1987, ‘Multi-Agent Plan Recognition in an Adversarial Domain’. In:Proc. of the Expert Systems in Government (ESIG) Conference, pp. 188–193.Google Scholar
  12. Baier, A.: 1970, ‘Act and Intent’.Journal of Philosophy,67(19):648–658.Google Scholar
  13. Baier, A.: 1976, ‘Intention, Practical Knowledge and Representation’. In: M. Brand and D. Walton (eds.),Action Theory, pp. 27–43, Dordrecht: Reidel.Google Scholar
  14. Balkanski, C.: 1990,Modeling Act-Type Relations in Collaborative Activity. Technical Report 23–90, Aiken Computation Laboratory, Center for Research in Computing Technology, Harvard University, Cambridge, MA.Google Scholar
  15. Benoit, J. E. Dombroski, P. Jordan and S. Laskowski: 1988, ‘Expert Systems for Multi-Agent Plan Recognition in Uncertain Environments (Extended Abstract)’. In:Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.Google Scholar
  16. Bratman, M., D. Israel and M. Pollack: 1988, 'Plans and Resource-Bounded Practical Reasoning.Computational Intelligence,4(4):349–355.Google Scholar
  17. Bratman, M.: 1987,Intention, Plans, and Practical Reason. Cambridge, MA: Harvard University Press.Google Scholar
  18. Brennenstuhl, W.: 1975,Handlungstheorie und Handlungslogik. Vorbereitung zur Entwicklung einer sprachadäquaten Handlungslogik. Kronberg/Taunus: Skriptor.Google Scholar
  19. Brewer, W. and D. Dupree: 1983, ‘Use of Plan Schemata in the Recall and Recognition of Goal-Directed Actions’.Journal of Experimental Psychology: Learning, Memory, and Cognition,9:117–129.Google Scholar
  20. Broverman, C. K. Huff and V. Lesser: 1986, ‘The Role of Plan Recognition in Design of an Intelligent User Interface’. In:Proc. of the IEEE Conference on Man, Machine, and Cybernetics, pp. 863–868.Google Scholar
  21. Bruce, B. and D. Newman: 1978, ‘Interacting Plans’.Cognitive Science,2:195–233.Google Scholar
  22. Bruce, B.: 1981, ‘Plans and Social Actions’. In: R. Spiro, B. Bruce, and W. Brewer (eds.),Theoretical Issues in Reading Comprehension, Chapter 15, pp. 367–384, Hillsdale, NJ: Erlbaum.Google Scholar
  23. Carberry, S.: 1989, ‘Plan Recognition and its Use in Understanding Dialogue’. In: A. Kobsa and W. Wahlster (eds.),User Models in Dialog Systems, Berlin: Springer, pp. 133–162.Google Scholar
  24. Carberry, S.: 1990a, ‘A Model of Plan Recognition that Facilitates Default Inferences’. In:Proc. of the Second International Workshop on User Modeling, Honolulu, HI.Google Scholar
  25. Carberry, S.: 1990b, ‘Incorporating Default Inferences into Plan Recognition’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 471–478.Google Scholar
  26. Carbonell, J.: 1979,The Counterplanning Process: A Model of Decision-Making in Adverse Situations. Technical Report, Computer Science Department, Carnegie-Mellon University, Pittsburg, PA.Google Scholar
  27. Carver,, N. V. Lesser and D. McCue: 1984, ‘Focusing in Plan Recognition’. In:Proc. of the 4th National Conference of the American Association for Artificial Intelligence, Austin, TX, pp. 42–48.Google Scholar
  28. Castelfranchi, C.: 1990, ‘No More Cooperation, Please! Controversial Points about the Social Structure of Verbal Interaction’. In:Proc. of the NATO Advanced Research Workshop on Computational Theories of Communication and their Applications: Problems and Prospects. Castel Ivano, Trentino, Italy.Google Scholar
  29. Chapman, D.: 1989,A Framework for External Reference. Internal Report, MIT Artificial Intelligence Laboratory, Cambridge, MA.Google Scholar
  30. Charniak, E. and D. McDermott: 1985,Introduction to Artificial Intelligence. Reading, MA: Addison-Wesley.Google Scholar
  31. Cohen, P. and H. Levesque: 1987, ‘Intention = Choice + Commitment’. In:Proc. of the 6th National Conference of the American Association for Artificial Intelligence, Seattle, WA, pp. 410–415.Google Scholar
  32. Cohen, P. C. Perrault and J. Allen: 1982, ‘Beyond Question Answering’. In: W. Lehnert and M. Ringle (eds.),Strategies for Natural Language Processing, Hillsdale, NJ: Erlbaum, pp. 245–274.Google Scholar
  33. Cohen, P.: 1986, ‘Communication as Rational Interaction’.CSLI Monthly,2(2).Google Scholar
  34. Desmarais, M. S. Larochelle and L. Giroux: 1987, ‘A Text Editor Consultant’. In:Proc. of the IEEE MONTEX' 87 Conferences: COMBINT 87, New York, NY, pp. 217–219.Google Scholar
  35. Dickman, H.: 1963, ‘The Perception of Behavioral Units’. In: R. Barker (ed.),The Stream of Behavior, New York, NY: Appleton-Century-Crofts, pp. 23–41.Google Scholar
  36. Finin, T.: 1983, ‘Providing Help and Advice in Task Oriented Systems’. In:Proc. of the 8th International Joint Conference on Artificial Intelligence, Karlsruhe, West Germany, pp. 176–178.Google Scholar
  37. Fischer, G. A. Lemke and T. Schwab: 1985, ‘Knowledge-Based Help Systems’. In:Proc. of the CHI' 85 Conference on Human Factors in Computing Systems, San Francisco, CA. Special Issue of the SIGCHI Bulletin, pp. 161–167.Google Scholar
  38. Foss, C. and G. Bower: 1986, ‘Understanding Actions in Relation to Goals’. In: N. Sharkey (ed.),Advances in Cognitive Science 1, Chichester: Ellis Horwood, pp. 94–124.Google Scholar
  39. Genesereth, M.: 1979, ‘The Role of Plans in Automated Consultation’. In:Proc. of the 6th International Joint Conference on Artificial Intelligence, Tokyo, Japan, pp. 311–319.Google Scholar
  40. Georgeff, M.: 1983, ‘Communication and Interaction in Multi-Agent Planning’. In:Proc. of the 3rd National Conference of the American Association for Artificial Intelligence, Washington, DC, pp. 125–129.Google Scholar
  41. Goldman, R. and E. Charniak: 1988, ‘A Probabilistic ATMS for Plan Recognition’. In;Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.Google Scholar
  42. Goldman, A.: 1970,A Theory of Human Action. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  43. Goodman, B. and D. Litman: 1989,Design Interfaces and Plan Recognition. Technical Report 7103, BBN Systems and Technologies Corp., Cambridge, MA.Google Scholar
  44. Greer, J. and G. McCalla: 1988,A Computational Framework for Granularity and its Application to Educational Diagnosis. Research Report 88–4, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, SK.Google Scholar
  45. Grosz, B. and C. Sidner: 1990, ‘Plans for Discourse’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 417–444.Google Scholar
  46. Hecking, M.: 1987, ‘How to Use Plan Recognition to Improve the Abilities of the Intelligent Help System SINIX Consultant’. In:Proc. of the INTERACT ' 87.Google Scholar
  47. Hecking, M.: 1990, ‘A Logical Treatment of Distributed and Multi-Level Plan Recognition’. In:Proc. of the AAAI-90 Workshop on Adversarial Reasoning.Google Scholar
  48. Hertzberg, J.: 1989,Planen. Einführung in die Planerstellungsmethoden der Künstlichen Intelligenz. Mannheim: BI-Wissenschaftsverlag.Google Scholar
  49. Herzog, G. and G. Retz-Schmidt: 1990, ‘Das System SOCCER: Simultane Interpretation und natürlichsprachliche Beschreibung zeitveränderlicher Szenen’. In: J. Perl (ed.),Sport und Informatik, Schorndorf: Hofmann, pp. 95–119.Google Scholar
  50. Herzog, G. C.-K. Sung, E. André, W. Enkelmann, H.-H. Nagel, T. Rist and W. Wahlster: 1989, ‘Incremental Natural Language Description of Dynamic Imagery’. In: C. Freksa and W. Brauer (eds.),Wissensbasierte Systeme. 3. Internationaler GI-Kongreβ, Berlin: Springer, pp. 153–162.Google Scholar
  51. Huff, K. and V. Lesser: 1982,Knowledge-Based Command Understanding: An Example for the Software Development Environment. COINS Technical Report 82–6, Computer and Information Science Department, University of Massachusetts at Amherst, Amherst, MA.Google Scholar
  52. Kautz, H. and J. Allen: 1986, ‘Generalized Plan Recognition’. In:Proc. of the 5th National Conference of the American Association for Artificial Intelligence, Philadelphia, PA, pp. 32–37.Google Scholar
  53. Kautz, H.: 1987,A Formal Theory of Plan Recognition. PhD thesis, Department of Computer Science, University of Rochester, Rochester, NY.Google Scholar
  54. Kautz, H.: 1990, ‘A Circumscriptive Theory of Plan Recognition’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 105–133.Google Scholar
  55. Kenny, A.: 1963,Action, Emotion and Will. London: Routledge & Kegan Paul.Google Scholar
  56. Kim, J.: 1976, ‘Intention and Practical Inference’. In: J. Manninen and R. Tuomela (eds.),Essays on Explanation and Understanding, Dordrecht: Reidel, pp. 249–269.Google Scholar
  57. Konolige, K. and M. Pollack: 1989, ‘Ascribing Plans to Agents. Preliminary Report’. In:Proc. of the 11th International Joint Conference on Artificial Intelligence, Detroit, MI, pp. 924–930.Google Scholar
  58. Lames, M. J. Perl, H.-J. Schröder, T. Uthman, and W.-D. Miethling: 1990, ‘Der Einsatz von Expertensystemen im Sport am Beispiel des TEnnis-Simulations-SYstems TESSY’. In: J. Perl (ed.),Sport und Informatik, Schorndorf: Hofmann, pp. 75–91.Google Scholar
  59. Levesque,, H. P. Cohen and J. Nunes: 1990, ‘On Acting Together’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 94–99.Google Scholar
  60. Lichtenstein, E. and W. Brewer: 1980, ‘Memory for Goal-Directed Actions’.Cognitive Psychology,12: 412–445.Google Scholar
  61. Litman, D. and J. Allen: 1987, ‘A Plan Recognition Model for Subdialogues in Conversations’.Cognitive Science,11: 163–200.Google Scholar
  62. Litman, D. and J. Allen: 1990, ‘Discourse Processing and Commonsense Plans’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 365–388.Google Scholar
  63. Lochbaum,, K. B. Grosz and C. Sidner: 1990, ‘Models of Plans to Support Communication: An Initial Report’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 485–490.Google Scholar
  64. London, R. and W. Clancey: 1982, ‘Plan Recognition Strategies in Student Modeling: Prediction and Description’. In:Proc. of the 2nd National Conference of the American Association for Artificial Intelligence, Pittsburgh, PA, pp. 335–338.Google Scholar
  65. London, R.: 1991, ‘Student Modeling to Support Multiple Instructional Approaches’.User Modeling and User-Adapted Interaction,1. Forthcoming.Google Scholar
  66. Mahesh,, K. K. Eiselt and A. Rau: 1990, ‘A Theory of Representation for Understanding Stories in Multi-Agent Adversarial Domains’. In:Proc. of the AAAI-90 Workshop on Adversarial Reasoning.Google Scholar
  67. McCue, D. and V. Lesser: 1983,Focusing and Constraint Management in Intelligent Interface Design. COINS Technical Report 83–36, Computer and Information Science Department, University of Massachusetts at Amherst, Amherst, MA.Google Scholar
  68. Miller, G. and P. Johnson-Laird: 1976,Language and Perception. Cambridge, England: Cambridge University Press.Google Scholar
  69. Miller, G. E. Galanter and K. Pribram: 1960,Plans and the Structure of Behaviour. New York, NY: Holt, Rinehart, and Winston.Google Scholar
  70. Nagel, H.-H.: 1988, ‘From Image Sequences Towards Conceptual Descriptions’.Image and Vision Computing,6(2):59–74.Google Scholar
  71. Neumann, B. and H.-J. Novak: 1986, ‘NAOS: Ein System zur natürlichsprachlichen Beschreibung zeitveränderlicher Szenen’.Informatik. Forschung und Entwicklung,1(1):83–92.Google Scholar
  72. Pleines, J.: 1975, ‘Kausale Relationen und Intentionalität’. In: V. Ehrich and P. Finke (eds.),Beiträge zur Grammatik und Pragmatik, Kronberg/Taunus: Scriptor, pp. 55–70.Google Scholar
  73. Pollack, M.: 1986, ‘A Model of Plan Inference that Distinguishes between the Beliefs of Actors and Observers’. In:Proc. of the 24th Annual Meeting of the ACL,Columbia University,New York, NY, pp. 207–214.Google Scholar
  74. Pollack, M.: 1989, Plan Recognition Beyond STRIPS. In:Proc. of the IJCAI' 89 Workshop on Plan Recognition.Google Scholar
  75. Pollack, M.: 1990, ‘Plans as Complex Mental Attitudes’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 77–103.Google Scholar
  76. Retz-Schmidt, G.: 1988, ‘A REPLAI of SOCCER: Recognizing Intentions in the Domain of Soccer Games’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 455–457.Google Scholar
  77. Retz-Schmidt, G.: 1991,Die Interpretation des Verhaltens mehrerer Akteure in Szenenfolgen. PhD thesis, Fachbereich Informatik, Universität des Saarlandes, Saarbrücken.Google Scholar
  78. Rosenschein, J.: 1986,Rational Interaction: Cooperation among Intelligent Agents. PhD thesis, Department of Computer Science, Stanford University.Google Scholar
  79. Ross, P. and J. Lewis: 1988, ‘Plan Recognition for Intelligent Tutoring Systems’. In: Ercoli and Lewis (eds.),Artificial Intelligence Tools in Education', New York, NY: Elsevier, pp. 29–37.Google Scholar
  80. Sandewall, E. G. Adorni, H.-H. Nagel and M. Thonnat: 1990, ‘PROMETHEUS Session’. In:Proc. of the 9th European Conference on Artificial Intelligence, Stockhom, Sweden, pp. 779–780.Google Scholar
  81. Schank, R. and R. Abelson: 1977,Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Hillsdale, NJ: Erlbaum.Google Scholar
  82. Schmidt, C. N. Sridharan and J. Goodson: 1978, ‘The Plan Recognition Problem: An Intersection of Psychology and Artificial Intelligence’.Artificial Intelligence,11:45–83.Google Scholar
  83. Searle, J.: 1990, ‘Collective Intentions and Actions’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 401–415.Google Scholar
  84. Sidner, C.: 1985, ‘Plan Parsing for Intended Response Recognition in Discourse’.Computational Intelligence,1(1): 1–10.Google Scholar
  85. Strömberg, D. D. Driankov, N. Olander and G. Petterson: 1989, ‘Plan Recognition for a Fighter Aircraft’. In:Proc. of the 9th International Conference on Expert Systems of Applications, pp. 277–286.Google Scholar
  86. Tuomela, R. and K. Miller: 1988, ‘We-Intentions’.Philosophical Studies,53:367–389.Google Scholar
  87. Werner, E.: 1988a, ‘Social Intentions’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 719–723.Google Scholar
  88. Werner, E.: 1988b, ‘Toward a Theory of Communication and Cooperation for Multiagent Planning’. In: M. Vardi (ed.),Proc. of the 2nd Conference on Theoretical Aspects of Reasoning about Knowledge, Los Altos, CA: Morgan Kaufmann, pp. 129–143.Google Scholar
  89. Werner, E.: 1989, ‘Cooperating Agents: A Unified Theory of Communication and Social Structure’. In: M. Huhns and L. Gasser (eds.),Distributed Artificial Intelligence. Vol. 2, Los Altos, CA: Morgan Kaufmann.Google Scholar
  90. Werner, E.: 1990, ‘What can Agents do Together? A Semantics for Reasoning about Cooperative Ability’. In:Proc. of the 9th European Conference on Artificial Intelligence, Stockhom, Sweden, pp. 694–701.Google Scholar
  91. Wilensky, R.: 1978,Understanding Goal-Based Stories. Technical Report 140, Department of Computer Science, Yale University, New Haven, CT.Google Scholar
  92. Wilensky, R.: 1983,Planning and Understanding: A Computational Approach to Human Reasoning. Reading, MA: Addison-Wesley.Google Scholar
  93. Wobcke, W: 1988, ‘A Logical Theory of Plan Recognition’. In:Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.Google Scholar
  94. Woodroffe, M.: 1988, ‘Plan Recognition and Intelligent Tutoring Systems’. In: J. Self (ed.),Intelligent Computer-Aided Instruction, London: Chapman & Hall, pp. 212–225.Google Scholar

Copyright information

© Kluwer Academic Publishers 1991

Authors and Affiliations

  • Gudula Retz-Schmidt 
    • 1
  1. 1.Dept. of Computer ScienceUniversity of HamburgHamburgGermany

Personalised recommendations