Applied Intelligence

, Volume 37, Issue 3, pp 305–320 | Cite as

Focus tree: modeling attentional information in task-oriented human-machine interaction

  • Milan GnjatovićEmail author
  • Marko Janev
  • Vlado Delić


This paper introduces a new model of attentional state in task-oriented human-machine interaction. It integrates three lines of research: (i) neurocognitive understanding of the focus of attention in working memory, (ii) the notion of attention related to the theory of discourse structure in the field of computational linguistics, and (iii) investigation of a corpus that comprises recordings of spontaneous speech-based human-machine interaction. The underlying idea was to make a computationally appropriate representation of attentional information that imitates the function of a focus of attention in human perception. The introduced model addresses both the research questions of storage and processing of attentional information. Finally, the paper illustrates the model for concrete interaction domains, and discusses its implementation within a prototype spoken dialogue system.


Focus tree Attentional information Human-machine interaction Cognition Utterance processing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Awh E, Jonides J (2001) Overlapping mechanisms of attention and spatial working memory. Trends Cogn Sci 5(3):119–126 CrossRefGoogle Scholar
  2. 2.
    Atkinson RC, Shiffrin RM (1968) Human memory: A proposed system and its control processes. In: Spence KW, Spence JT (eds) The psychology of learning and motivation: Advances in research and theory, vol 2. Academic Press, New York, pp 89–195 Google Scholar
  3. 3.
    Baddeley AD (1993) Working memory or working attention? In: Baddeley AD, Weiskrantz L (eds) Attention: Selection, awareness, and control. A tribute to Donald Broadbent. Oxford University Press, New York, pp 152–170 Google Scholar
  4. 4.
    Baddeley AD, Hitch GJ (1974) Working memory. In: Bower GH (ed) The psychology of learning and motivation: Advances in research and theory, vol 8. Academic Press, New York, pp 47–89 Google Scholar
  5. 5.
    Baddeley AD, Logie RH (1999) Working memory: The multiple component model. In: Miyake A, Shah P (eds) Models of working memory. Cambridge University Press, New York, pp 28–61 Google Scholar
  6. 6.
    Bledowski C, Rahm B, Rowe BR (2009) What ‘works’ in working memory? Separate systems for selection and updating of critical information. J Neurosci 29:13735–13741 CrossRefGoogle Scholar
  7. 7.
    Bledowski C, Kaiser J, Rahm B (2010) Basic operations in working memory: Contributions from functional imaging studies. Behav Brain Res 214(2):172–179 CrossRefGoogle Scholar
  8. 8.
    Broadbent DE (1958) Perception and communication. Pergamon Press, New York CrossRefGoogle Scholar
  9. 9.
    Campbell N (2006) On the structure of spoken language. In: Proceedings of the 3rd international conference on speech prosody 2006, Dresden, Germany, 4 pages Google Scholar
  10. 10.
    Campbell N (2007) Towards conversational speech synthesis; lessons learned from the expressive speech processing project. In: Proceedings of the sixth ISCA workshop on speech synthesis (SSW6), Bonn, Germany, pp 22–27 Google Scholar
  11. 11.
    Cowan N (1988) Evolving conceptions of memory storage selective attention, and their mutual constraints within the information processing system. Psychol Bull 104(2):163–191 CrossRefGoogle Scholar
  12. 12.
    Cowan N (2001) The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav Brain Sci 24:87–185 CrossRefGoogle Scholar
  13. 13.
    Cowan N, Fristoe NM, Elliot EM, Brunner RP, Saults JS (2006) Scope of attention, control of attention, and intelligence in children and adults. Mem Cogn 34:1754–1768 CrossRefGoogle Scholar
  14. 14.
    Engle RW, Kane MJ (2004) Executive attention, working memory capacity, and a two-factor theory of cognitive control. In: Ross B (ed) The psychology of learning and motivation, vol 44. Elsevier, New York, pp 145–199 Google Scholar
  15. 15.
    Engle RW, Cantor J, Carullo JJ (1992) Individual differences in working memory and comprehension: A test of four hypotheses. J Exp Psychol Learn Mem Cogn 18:972–992 CrossRefGoogle Scholar
  16. 16.
    Ericsson KA, Kintsch W (1995) Long-term working memory. Psychol Rev 102:211–245 CrossRefGoogle Scholar
  17. 17.
    Gnjatović M (2009) Adaptive dialogue management in human-machine interaction. Verlag Dr Hut, Munich Google Scholar
  18. 18.
    Gnjatović M, Rösner D (2007) An approach to processing of user’s commands in human-machine interaction. In: Proceedings of the 3rd language and technology conference (LTC’07), Adam Mickiewicz University, Poznan, Poland, pp 152–156 Google Scholar
  19. 19.
    Gnjatović M, Rösner D (2008) Adaptive dialogue management in the NIMITEK prototype system. In: Proceedings of the 4th IEEE tutorial and research workshop perception and interactive technologies for speech-based systems (PIT’08), Lecture notes in computer science, vol 5078. Springer, Berlin, pp 14–25 Google Scholar
  20. 20.
    Gnjatović M, Rösner D (2010) Inducing genuine emotions in simulated speech-based human-machine interaction: The nimitek corpus. IEEE Trans Affect Comput 1:132–144 CrossRefGoogle Scholar
  21. 21.
    Gnjatović M, Pekar D, Delić V (2011) Naturalness, adaptation and cooperativeness in spoken dialogue systems. In: Toward autonomous, adaptive, and context-aware multimodal interfaces, theoretical and practical issues, Lecture notes in computer science, vol 6456. Springer, Berlin, pp 298–304 CrossRefGoogle Scholar
  22. 22.
    Griffin IC, Nobre AC (2003) Orienting attention to locations in internal representations. J Cogn Neurosci 15(8):1176–1194 CrossRefGoogle Scholar
  23. 23.
    Grosz B, Sidner C (1986) Attention, intentions, and the structure of discourse. Comput Linguist 12(3):175–204 Google Scholar
  24. 24.
    Halliday M (1994) An introduction to functional grammar, 2nd edn. Edward Arnold, London Google Scholar
  25. 25.
    Hovy E, McCoy K (1989) Focusing your RST: A step toward generating coherent, multisentential text. In: Proceedings of the 11th annual conference of the cognitive science society, Ann Arbor Google Scholar
  26. 26.
    Jokinen K (2009) Constructive dialogue modelling: Speech interaction and rational agents. Wiley, New York Google Scholar
  27. 27.
    Jokinen K, Tanaka H, Yokoo A (1998) Context management with topics for spoken dialogue systems. In: Proceedings of COLING-ACL 1998, pp 631–637 Google Scholar
  28. 28.
    Just MA, Carpenter PA (1992) A capacity theory of comprehension: Individual differences in working memory. Psychol Rev 99:122–149 CrossRefGoogle Scholar
  29. 29.
    Kane MJ, Conway ARA, Hambrick DZ, Engle RW (2007) Variation in working memory capacity as variation in executive attention and control. In: Conway ARA, Jarrold C, Kane MJ, Miyake A, Towse JN (eds) Variation in working memory. Oxford University Press, Oxford, pp 21–48 Google Scholar
  30. 30.
    Kari L, Rozenberg G (2008) The many facets of natural computing. Commun ACM 51(10):72–83 CrossRefGoogle Scholar
  31. 31.
    Kirschner M (2007) Applying a focus tree model of dialogue context to interactive question answering. In: Proceedings of the ESSLLI’07 student session, Dublin, Ireland, pp 135–147 Google Scholar
  32. 32.
    Lecœuche R, Robertson D, Barry C, Mellish C (2000) Evaluating focus theories for dialogue management. Int J Hum-Comput Stud 52(1):23–76 CrossRefGoogle Scholar
  33. 33.
    Li S-T, Tsai F-C (2010) Constructing tree-based knowledge structures from text corpus. Appl Intell 33(1):67–78 MathSciNetCrossRefGoogle Scholar
  34. 34.
    McCoy K, Cheng J (1991) Focus of attention: Constraining what can be said next. In: Paris CL, Swartout WR, Moore WC (eds) Natural language generation in artificial intelligence and computational linguistics. Kluwer Academic, Norwell, pp 103–124 Google Scholar
  35. 35.
    Mendonça M, de Arruda LVR, Neves F (2011) Autonomous navigation system using event driven-fuzzy cognitive maps. Appl Intell. doi: 10.1007/s10489-011-0320-1 Google Scholar
  36. 36.
    Miller GA (1956) The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychol Rev 63:81–97 CrossRefGoogle Scholar
  37. 37.
    Moeller J-U (1996) Domain related focus-shifting constraints in dialogues with knowledge based systems. In: Adorni G, Zock M (eds) Trends in natural language generation: An artificial intelligence perspective. Springer, Berlin, pp 188–204 CrossRefGoogle Scholar
  38. 38.
    Mogle JA, Lovett BJ, Stawski RS, Sliwinski MJ (2008) What’s so special about working memory? An examination of the relationship among working memory, secondary memory, and fluid intelligence. Psychol Sci 19:1071–1077 CrossRefGoogle Scholar
  39. 39.
    Mohammad Y, Nishida T (2010) Controlling gaze with an embodied interactive control architecture. Appl Intell 32(2):148–163 CrossRefGoogle Scholar
  40. 40.
    Moubaiddin A, Obeid N (2009) Partial information basis for agent-based collaborative dialogue. Appl Intell 30(2):142–167 CrossRefGoogle Scholar
  41. 41.
    Nobre CA, Coull TJ, Maquet P, Frith DC, Vandenberghe R, Mesulam MM (2004) Orienting attention to locations in perceptual versus mental representations. J Cogn Neurosci 16(3):363–373 CrossRefGoogle Scholar
  42. 42.
    Oberauer K (2002) Access to information in working memory: Exploring the focus of attention. J Exp Psychol Learn Mem Cogn 28(3):411–421 CrossRefGoogle Scholar
  43. 43.
    Oberauer K, Lange EB (2009) Activation and binding in verbal working memory: A dual-process model for the recognition of nonwords. Cogn Psychol 58(1):102–136 CrossRefGoogle Scholar
  44. 44.
    O’Reilly RC, Braver TS, Cohen JD (1999) A biologically-based computational model of working memory. In: Miyake A, Shah P (eds) Models of working memory. Cambridge University Press, New York, pp 375–411 Google Scholar
  45. 45.
    Rahwan I, McBurney P (2007) Argumentation technology. IEEE Intell Syst 22(6):21–23 CrossRefGoogle Scholar
  46. 46.
    Roulet E (1992) On the structure of conversation as negotiation. In: Mey JL, Parret H, Verschueren J (eds) (On) Searle on conversation. John Benjamins, Philadelphia, pp 91–99 Google Scholar
  47. 47.
    Salthouse TA (1996) The processing-speed theory of adult age differences in cognition. Psychol Rev 103:403–428 CrossRefGoogle Scholar
  48. 48.
    Searle J (1992) Conversation. In: Mey JL, Parret H, Verschueren J (eds) (On) Searle on conversation. John Benjamins, Philadelphia, pp 7–29 Google Scholar
  49. 49.
    Shah P, Miyake A (1999) Models of working memory: An introduction. In: Miyake A, Shah P (eds) Models of working memory: Mechanisms of active maintenance and executive control. Cambridge University Press, New York, pp 1–26 CrossRefGoogle Scholar
  50. 50.
    Stede M, Schlangen D (2004) Information-seeking chat: Dialogue management by topic structure. In: Proceedings of the 8th workshop on semantics and pragmatics of dialogue, CATALOG 04, Barcelona Google Scholar
  51. 51.
    Stoltzfus ER, Hasher L, Zacks RT (1996) Working memory and aging: Current status of the inhibitory view. In: Richardson JTE, Engle RW, Hasher L, Logie RH, Stoltzfus ER, Zacks RT (eds) Working memory and human cognition. Oxford University Press, New York, pp 66–88 CrossRefGoogle Scholar
  52. 52.
    Unsworth N, Engle RW (2007) The nature of individual differences in working memory capacity: Active maintenance in primary memory and controlled search from secondary memory. Psychol Rev 114:104–132 CrossRefGoogle Scholar
  53. 53.
    Unsworth N, Spillers GJ (2010) Working memory capacity: Attention control secondary memory, or both? A direct test of the dual-component model. J Mem Lang 62(4):392–406 CrossRefGoogle Scholar
  54. 54.
    Wheeler ME, Treisman AM (2002) Binding in short-term visual memory. J Exp Psychol Gen 131:48–64 CrossRefGoogle Scholar
  55. 55.
    Xu Y, Ohmoto Y, Okada S, Ueda K, Komatsu T, Okadome T, Kamei K, Sumi Y, Nishida T (2010) Formation conditions of mutual adaptation in human-agent collaborative interaction. Appl Intell. doi: 10.1007/s10489-010-0255-y Google Scholar
  56. 56.
    Zhong N, Liu J, Yao Y (2010) Introduction to brain informatics. Cogn Syst Res 11(1):1–2 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Faculty of Technical SciencesUniversity of Novi SadNovi SadSerbia
  2. 2.Mathematical Institute of the Serbian Academy of Sciences and ArtsBeogradSerbia

Personalised recommendations