The synthetic teammate project

Abstract

The main objective of the Synthetic Teammate project is to develop language and task enabled synthetic agents capable of being integrated into team training simulations. To achieve this goal, the agents must be able to closely match human behavior. The initial application for the synthetic teammate research is creation of an agent able to perform the functions of a pilot for an Unmanned Aerial Vehicle (UAV) simulation as part of a three-person team. The agent, or synthetic teammate, is being developed in the ACT-R cognitive architecture. The major components include: language comprehension and generation, dialog management, agent-environment interaction, and situation assessment. Initial empirical results suggest that the agent-environment interaction is a good approximation to human behavior in the UAV environment, and we are planning further empirical tests of the synthetic teammate operating with human teammates. This paper covers the project’s modeling approach, challenges faced, progress made toward an integrated synthetic teammate, and lessons learned during development.

This is a preview of subscription content, log in to check access.

References

  1. Altmann G, Steedman M (1988) Interaction with context during human sentence processing. Cognition 30:191–238

    Article  Google Scholar 

  2. Anderson JR (2007) How can the human mind occur in the physical Universe? Oxford University Press, New York

    Google Scholar 

  3. Anderson JR, Fincham JM, Douglass S (1997) The role of examples and rules in the acquisition of a cognitive skill. J Exp Psychol Learn Mem Cogn 23:932–945

    Article  Google Scholar 

  4. Anderson JR, Bothell D, Byrne M, Douglass S, Lebiere C, Qin Y (2004) An integrated theory of the mind. Psychol Rev 111(4):1036–1060

    Article  Google Scholar 

  5. Ball J (1991). PM, propositional model, a computational psycholinguistic model of language comprehension based on a relational analysis of written english. UMI Dissertation Information Service, Ann Arbor, MI

  6. Ball J (2004a). A cognitively plausible model of language comprehension. In: Proceedings of the 13th conference on behavior representation in modeling and simulation, pp 305–316

  7. Ball J (2004b) Software agents with natural language capabilities–where are we? In: Symposium conducted at the 13th conference on behavior representation in modeling and simulation, Arlington, VA

  8. Ball J (2006) Can NLP systems be a cognitive black box? (Is cognitive science relevant to AI problems?) In: Paper presented at the AAAI spring symposium: between a rock and a hard place, cognitive science principles meet AI hard problems (Technical Report SS-06-02). AAAI Press, Menlo Park

    Google Scholar 

  9. Ball J (2007a) Construction-driven language processing. In: Vosniadou S, Kayser D, Protopapas A (eds) Proceedings of the 2nd european cognitive science conference. LEA, New York, pp 722–727

    Google Scholar 

  10. Ball J (2007b) A bi-polar theory of nominal and clause structure and function. Ann Rev Cogn Linguist 5(1):27–54

    Google Scholar 

  11. Ball J (2008) A naturalistic, functional approach to modeling language comprehension. In: Paper presented at the AAAI Fall Symposium, Naturally Inspired Artificial Intelligence (Technical Report FS-08-06). AAAI Press, Menlo Park

    Google Scholar 

  12. Ball J (2010) Simplifying the mapping from referring expression to referent in a conceptual semantics of reference. In: Proceedings of the 32nd annual meeting of the cognitive science society

  13. Ball J, Heiberg A, Silber R (2007) Toward a large-scale model of language comprehension in ACT-R 6. In: Lewis R, Polk T, Laird J (eds) Proceedings of the 8th international conference on cognitive modeling. Psychology Press, New York, pp 173–179

    Google Scholar 

  14. Boersma P, Hayes B (2001) Empirical tests of the gradual learning algorithm. Linguist Inq 32:45–86

    Article  Google Scholar 

  15. Byrne MD (2001) ACT-R/PM and menu selection: applying a cognitive architecture to HCI. Int J Human-Comput Stud 55(1):41–84

    Article  Google Scholar 

  16. Byrne MD, Kirlik A (2005) Using computational cognitive modeling to diagnose possible sources of aviation error. Int J Aviat Psychol 15(2):135–155

    Article  Google Scholar 

  17. Byrne MD, Wood SD, Sukaviriya P, Foley JD, Kieras DE (1994) Automating interface evaluation. In: Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence. ACM, New York, pp 232–237

    Google Scholar 

  18. Cassimatis N, Bello P, Langley P (2008) Ability, breadth, and parsimony in computational models of higher-order cognition. Cogn Sci 32:1304–1322

    Article  Google Scholar 

  19. Christianson K, Hollingsworth A, Halliwell J, Ferreira F (2001) Thematic roles assigned along the garden path linger. Cogn Psychol 42:368–407

    Article  Google Scholar 

  20. Colmerauer A, Roussel P (1996) The birth of Prolog. In: Bergin T, Gibson R (eds) History of programming languages II. ACM Press/Addison-Wesley, New York, pp 331–367

    Google Scholar 

  21. Cooke N, Shope S (2005) Synthetic task environments for teams: CERTT’s UAV-STE. Handbook on human factors and ergonomics methods, vol 46. CRC Press, Boca Raton

    Google Scholar 

  22. Cooke NJ, Kiekel PA, Helm EE (2001) Measuring team knowledge during skill acquisition of a complex task. Int J Cogn Ergon 5(3):297–315. Special section on knowledge acquisition

    Article  Google Scholar 

  23. Cooke NJ, Gorman JC, Duran JL, Taylor AR (2007) Team cognition in experienced command-and-control teams. J Exp Psychol Appl 13(3):146–157. Special issue on capturing expertise across domains

    Article  Google Scholar 

  24. Cooke NJ, Gorman JC, Kiekel PA (2008) Communication as team-level cognitive processing. In: Letsky M, Warner N, Fiore S, Smith CAP (eds) Macrocognition in teams: theories and methodologies. Ashgate, Hants, pp 51–64

    Google Scholar 

  25. Core MG, Allen JF (1997) Coding dialogs with the DAMSL annotation scheme. Paper presented at the AAAI fall symposium on communicative action in humans and machines, November 8–10, 1997, Cambridge, MA

  26. Douglass SA (2007) A computational model of situated action. Carnegie Mellon University, Pittsburgh (Doctoral dissertation)

    Google Scholar 

  27. Douglass SA (2010) Rule & Automata Modeling Language (RaAML) (in preparation)

  28. Douglass S, Ball J, Rodgers S (2009) Large declarative memories in ACT-R. In: Proceedings of the 9th international conference on cognitive modeling 2009, Manchester, UK

  29. Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37(1):32–64

    Article  Google Scholar 

  30. Ericsson S (2004) Dynamic optimisation of information enrichment in dialogue. In: Proceedings of the 8th international workshop on formal semantics and pragmatics of dialogue. Catalog, Barcelona

    Google Scholar 

  31. Freiman M, Ball J (2008) Computational cognitive modeling of reading comprehension at the word level. In: Proceedings of the 38th western conference on linguistics. University of California, Davis, Davis, pp 34–45

    Google Scholar 

  32. Freiman M, Ball J (2010) Improving the reading rate of Double-RLanguage. In: Proceedings of the 10th international conference on cognitive modeling (to appear)

  33. Gallagher HL, Frith CD (2003) Functional imaging of ‘theory of mind’. Trends Cogn Sci 7(2):77–83

    Article  Google Scholar 

  34. Gibson E, Pearlmutter NJ (1998) Constraints on sentence comprehension. Trends Cogn Sci 2(7):262–268

    Article  Google Scholar 

  35. Gorman JC, Cooke NJ, Winner JL (2006) Measuring team situation awareness in decentralized command and control systems. Ergonomics 49:1312–1325

    Article  Google Scholar 

  36. Grodner D, Gibson E, Argaman V, Babyonyshev M (2003) Against repair-based reanalysis in sentence comprehension. J Psycholinguist Res 32(2):141–166

    Article  Google Scholar 

  37. Hobbs JR (1985) Ontological promiscuity. In: Proceedings of the 23rd annual meeting of the association for computational linguistics. Chicago, IL, pp 61–69

  38. Hobbs JR (2003) Discourse and inference. Retrieved from http://www.isi.edu/~hobbs/disinf-tc.html

  39. Jones RM, Laird JE, Nielsen PE, Coulter KJ, Kenny P, Koss FV (1999) Automated intelligent pilots for combat flight simulation. AI Mag 20(1):27–41

    Google Scholar 

  40. Kamp H, Ryle U (1993) From discourse to logic: introduction to model-theoretic semantics of natural language, formal logic and discourse representation theory. Studies in linguistics and philosophy. Kluwer Academic, Dordrecht

    Google Scholar 

  41. Kieras DE (1988) Towards a practical GOMS model methodology for user interface design. In: Helander M (ed) The handbook of human-computer interaction. North-Holland, Amsterdam, pp 135–158

    Google Scholar 

  42. Kieras D, Meyer DE (1997) An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Hum-Comput Interact 12(4):391–438

    Article  Google Scholar 

  43. Kintsch W (1998) Comprehension, a paradigm for cognition. Cambridge University Press, New York

    Google Scholar 

  44. Klahr D, Chase WG, Lovelace EA (1983) Structure and process in alphabetic retrieval. J Exp Psychol Learn Mem Cogn 9(3):462–477

    Article  Google Scholar 

  45. Kosslyn S (2006) The case for mental imagery. Oxford University Press, New York

    Google Scholar 

  46. Laird JE, Jones RM (1998) Building advanced autonomous AI systems for large scale real time simulations. In: Proceedings of the 1998 computer game developers’ conference. Freeman, Long Beach, pp 365–378

    Google Scholar 

  47. Landauer T, Dumais S (1997) A solution to Plato’s problem: the latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychol Rev 104(2):211–240

    Article  Google Scholar 

  48. Langacker RW (1987) Foundations of cognitive grammar, vol I: theoretical prerequisites. Stanford University Press, Stanford

    Google Scholar 

  49. Langacker RW (1991) Foundations of cognitive grammar, vol II: descriptive application. Stanford University Press, Stanford

    Google Scholar 

  50. Lebiere C, Wray R (eds) (2006) Between a rock and a hard place: cognitive science principles meet AI-hard problems. AAAI Press, Menlo Park. Papers from the AAAI spring symposium

    Google Scholar 

  51. Lovett MC (1998) Choice. In: Anderson JR, Lebiere C (eds) The atomic components of thought. Erlbaum, Mahwah, pp 255–296

    Google Scholar 

  52. Marcus M, Badler N, Joshi A, Pappas G, Pereira F, Romero M, McCallum A, Potts C, Yanco H (2008) SUBTLE (Situation Understanding Bot Through Language and Environment) project program review. University of Massachusetts, Amherst

    Google Scholar 

  53. Matessa M (2000) Simulating adaptive communication. Carnegie Mellon University, Pittsburgh (Doctoral dissertation)

    Google Scholar 

  54. McClelland JL, Rumelhart DE (1981) An interactive activation model of context effects in letter perception. Part I. An account of basic findings. Psychol Rev 88(5):375–407

    Article  Google Scholar 

  55. McDonald D (1999) A rational reconstruction of Genaro. In: Proceedings of the RAGS Workshop, Edinburgh

  56. Miller G (1995) WordNet: a lexical database for English. Commun ACM 38(11):39–41

    Article  Google Scholar 

  57. Myers CW (2009) An account of model inspiration, integration, & sub-task validation. In: Proceedings of the 9th international conference on cognitive modeling, Manchester, UK

  58. Prince A, Smolensky P (1993/2004) Optimality theory: constraint interaction in generative grammar. Wiley-Blackwell, New York

    Google Scholar 

  59. Ritter FE, Van Rooy D, St Amant R (2002) A user modelling design tool based on a cognitive architecture for comparing interfaces. In: Computer-aided design of user interfaces III. Proceedings of the 4th international conference on computer-aided design of user interfaces CADUI’2002, Valenciennes, France, 15–17 May 2002. Kluwer Academics, Dordrecht, pp 111–118

    Google Scholar 

  60. Scolaro J, Santarelli T (2002) Cognitive modeling teamwork, taskwork, and instructional behavior in synthetic teammates. In: Proceedings of the 11th conference on computer generated forces and behavioral representation. Institute for Simulation and Training, Orlando

    Google Scholar 

  61. Seidenberg MS, McClelland JL (1989) A distributed, developmental model of word recognition and naming. Psychol Rev 96(4):523–568

    Article  Google Scholar 

  62. Stokes J (2001) Speech interaction and human behavior representations (HBRs). In: Proceedings of 10th conference on computer generated forces and behavioral representation. SISO, Inc, Norfolk, pp 467–476

    Google Scholar 

  63. Tambe M, Johnson WL, Jones RM, Koss F, Laird JE, Rosenbloom PS, Schwamb K (1995) Intelligent agents for interactive simulation environments. AI Mag 16(1):15–40

    Google Scholar 

  64. Tanenhaus MK, Spivey-Knowlton MJ, Eberhard KM, Sedivy JC (1995) Integration of visual and linguistic information in spoken language comprehension. Science 268(5217):1632–1634

    Article  Google Scholar 

  65. Traum DR, Allen JF (1994) Discourse obligations in dialogue processing. In: Proceedings of the 32nd annual meeting of the association for computational linguistics. Association for Computational Linguistics, Las Cruces, Morristown

    Google Scholar 

  66. Traum DR, Rickel J, Gratch J, Marsella S (2003) Negotiation over tasks in hybrid human-agent teams for simulation-based training. In: Proceedings of the second international joint conference on Autonomous agents and multiagent systems. ACM, Melbourn, Australia, New York, pp 441–448.

    Google Scholar 

  67. van Dijk T, Kintsch W (1983) Strategies of discourse comprehension. Academic Press, New York

    Google Scholar 

  68. Varges S (2006) Overgeneration and ranking for spoken dialogue systems. In: Proceedings of the 4th international natural language generation conference, Sydney, Australia, July 2006. Association for Computational Linguistics, pp 20–22

  69. Vosse T, Kempen G (2000) Syntactic structure assembly in human parsing: a computational model based on competitive inhibition and a lexicalist grammar. Cognition 75:105–143

    Article  Google Scholar 

  70. Walker MA, Whittaker SJ, Stent A, Maloor P, Moore J, Johnston M, Vasireddy G (2004) Generation and evaluation of user tailored responses in multimodal dialogue. Cogn Sci 28(5):811–840

    Article  Google Scholar 

  71. Yang Y, Bello P (2005) Some empirical results concerning deontic reasoning: models, schema, or both? In: Proceedings of the 27th annual meeting of the cognitive science society. Erlbaum, Mahway, pp 2393–2398

    Google Scholar 

  72. Zachary W, Santarelli T, Lyons D, Bergondy M, Johnston J (2001) Using a community of intelligent synthetic entities to support operational team training. In: Proceedings of the tenth conference on computer generated forces and behavioral representation. Orlando, FL: Institute for Simulation and Training, University of Central Florida, pp 215–224

  73. Zwann R, Radvansky G (1998) Situation models in language comprehension and memory. Psychol Bull 123(2):162–185

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jerry Ball.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ball, J., Myers, C., Heiberg, A. et al. The synthetic teammate project. Comput Math Organ Theory 16, 271–299 (2010). https://doi.org/10.1007/s10588-010-9065-3

Download citation

Keywords

  • Synthetic teammate
  • Language comprehension/generation
  • Dialog management
  • Situation model
  • Agent-environment interaction