Experimental Brain Research

, Volume 211, Issue 3–4, pp 613–630 | Cite as

What should I do next? Using shared representations to solve interaction problems

  • Giovanni PezzuloEmail author
  • Haris Dindo
Research Article


Studies on how “the social mind” works reveal that cognitive agents engaged in joint actions actively estimate and influence another’s cognitive variables and form shared representations with them. (How) do shared representations enhance coordination? In this paper, we provide a probabilistic model of joint action that emphasizes how shared representations help solving interaction problems. We focus on two aspects of the model. First, we discuss how shared representations permit to coordinate at the level of cognitive variables (beliefs, intentions, and actions) and determine a coherent unfolding of action execution and predictive processes in the brains of two agents. Second, we discuss the importance of signaling actions as part of a strategy for sharing representations and the active guidance of another’s actions toward the achievement of a joint goal. Furthermore, we present data from a human-computer experiment (the Tower Game) in which two agents (human and computer) have to build together a tower made of colored blocks, but only the human knows the constellation of the tower to be built (e.g., red-blue-red-blue-\(\ldots\)). We report evidence that humans use signaling strategies that take another’s uncertainty into consideration, and that in turn our model is able to use humans’ actions as cues to “align” its representations and to select complementary actions.


Joint action Shared representations Bayesian model Signaling Motor simulation 



The authors thank Guenther Knoblich, Natalie Sebanz and their research group for fruitful discussions, and two anonymous reviewers for helpful comments.


  1. Aarts H, Gollwitzer P, Hassin R (2004) Goal contagion: perceiving is for pursuing. J Pers Soc Psychol 87:23–37PubMedCrossRefGoogle Scholar
  2. Austin JL (1962) How to do things with words. Oxford University Press, New YorkGoogle Scholar
  3. Bacharach M (2006). In: Gold N, Sugden R (eds) Beyond individual choice. Princeton University Press, PrincetonGoogle Scholar
  4. Baker CL, Saxe R, Tenenbaum JB (2009) Action understanding as inverse planning. Cognition 113(3):329–349PubMedCrossRefGoogle Scholar
  5. Bicho E, Erlhagen W, Louro L, Silva ECE (2011) Neuro-cognitive mechanisms of decision making in joint action: a human-robot interaction study. Hum Mov SciGoogle Scholar
  6. Botvinick MM (2008) Hierarchical models of behavior and prefrontal function. Trends Cogn Sci 12(5):201–208PubMedCrossRefGoogle Scholar
  7. Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conflict monitoring and cognitive control. Psychol Rev 108(3):624–652PubMedCrossRefGoogle Scholar
  8. Bratman ME (1992) Shared cooperative activity. Philos Rev 101:327–341CrossRefGoogle Scholar
  9. Camazine S, Franks NR, Sneyd J, Bonabeau E, Deneubourg JL, Theraula G (2001) Self-organization in biological systems. Princeton University Press, PrincetonGoogle Scholar
  10. Clark H, Krych M (2004) Speaking while monitoring addressees for understanding. J Mem Lang 50(1):62–81CrossRefGoogle Scholar
  11. Clark HH (1996) Using language. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  12. Conte R, Castelfranchi C (1995) Cognitive and social action. University College London, LondonGoogle Scholar
  13. Csibra G, Gergely G (2007) "Obsessed with goals": functions and mechanisms of teleological interpretation of actions in humans. Acta Psychol 124:60–78CrossRefGoogle Scholar
  14. Cuijpers RH, van Schie HT, Koppen M, Erlhagen W, Bekkering H (2006) Goals and means in action observation: a computational approach. Neural Netw 19(3):311–322PubMedCrossRefGoogle Scholar
  15. Demiris Y, Khadhouri B (2005) Hierarchical attentive multiple models for execution and recognition (hammer). Robot Auton Syst J 54:361–369CrossRefGoogle Scholar
  16. Desmurget M, Grafton S (2000) Forward modeling allows feedback control for fast reaching movements. Trends Cogn Sci 4:423–431PubMedCrossRefGoogle Scholar
  17. Dindo H, Zambuto D, Pezzulo G (2011) Motor simulation via coupled internal models using sequential monte carlo. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI)Google Scholar
  18. Doucet A, Godsill S, Andrieu C (2000) On sequential monte carlo sampling methods for bayesian filtering. Stat Comput 10(3):197–208CrossRefGoogle Scholar
  19. Friston K (2008) Hierarchical models in the brain. PLoS Comput Biol 4(11):e1000211PubMedCrossRefGoogle Scholar
  20. Frith CD, Frith U (2006) How we predict what other people are going to do. Brain Res 1079(1):36–46PubMedCrossRefGoogle Scholar
  21. Frith CD, Frith U (2008) Implicit and explicit processes in social cognition. Neuron 60(3):503–510PubMedCrossRefGoogle Scholar
  22. Galantucci B (2005) An experimental study of the emergence of human communication systems. Cogn Sci 29:737–767CrossRefGoogle Scholar
  23. Garrod S, Pickering MJ (2004) Why is conversation so easy? Trends Cogn Sci 8(1):8–11PubMedCrossRefGoogle Scholar
  24. Garrod S, Pickering MJ (2009) Joint action, interactive alignment, and dialog. Top Cogn Sci 1(2):292–304CrossRefGoogle Scholar
  25. Georgiou I, Becchio C, Glover S, Castiello U (2007) Different action patterns for cooperative and competitive behaviour. Cognition 102(3):415–433PubMedCrossRefGoogle Scholar
  26. Gergely G, Csibra G (2003) Teleological reasoning in infancy: the naive theory of rational action. Trends Cogn Sci 7:287–292PubMedCrossRefGoogle Scholar
  27. Grice HP (1975) Logic and conversation. In: Cole P, Morgan JL (eds) Syntax and semantics. vol 3, Academic Press, New YorkGoogle Scholar
  28. Grosz BJ (1996) Collaborative systems. AI Mag 17(2):67–85Google Scholar
  29. Grush R (2004) The emulation theory of representation: motor control, imagery, and perception. Behav Brain Sci 27(3):377–396PubMedGoogle Scholar
  30. Hamilton AFdC, Grafton ST (2007) The motor hierarchy: from kinematics to goals and intentions. In: Haggard P, Rossetti Y, Kawato M (eds) Sensorimotor foundations of higher cognition. Oxford University Press, OxfordGoogle Scholar
  31. Hoffman G, Breazeal C (2007) Cost-based anticipatory action selection for human–robot fluency. IEEE Trans Robot 23(5):952–961CrossRefGoogle Scholar
  32. Howard R (1966) Information value theory. IEEE Trans Syst Sci Cybern 2(1):22–26CrossRefGoogle Scholar
  33. James W (1890) The principles of psychology. Dover Publications, New YorkCrossRefGoogle Scholar
  34. Jeannerod M (2001) Neural simulation of action: a unifying mechanism for motor cognition. NeuroImage 14:103–109CrossRefGoogle Scholar
  35. Jeannerod M (2006) Motor Cognition. Oxford University Press, OxfordCrossRefGoogle Scholar
  36. Kilner JM, Friston KJ, Frith CD (2007) Predictive coding: an account of the mirror neuron system. Cogn Process 8(3)Google Scholar
  37. Kirsh D (1999) Distributed cognition, coordination and environment design. In: Proceedings of the European conference on cognitive science, pp 1–11Google Scholar
  38. Knoblich G, Sebanz N (2008) Evolving intentions for social interaction: from entrainment to joint action. Philos Trans R Soc Lond B Biol Sci 363(1499):2021–2031PubMedCrossRefGoogle Scholar
  39. Konvalinka I, Vuust P, Roepstorff A, Frith CD (2010) Follow you, follow me: continuous mutual prediction and adaptation in joint tapping. Q J Exp Psychol (Colchester) 63(11):2220–2230CrossRefGoogle Scholar
  40. Lange FPD, Spronk M, Willems RM, Toni I, Bekkering H (2008) Complementary systems for understanding action intentions. Curr Biol 18(6):454–457PubMedCrossRefGoogle Scholar
  41. Levinson SC (2006) On the human "interaction engine". In: Enfield NJ, Levinson SC (eds) Roots of human sociality: culture, cognition and interaction. Berg, pp 39–69Google Scholar
  42. Maynard-Smith J, Harper D (2003) Animal Signals. Oxford University Press, OxfordGoogle Scholar
  43. Murphy KP (2002) Dynamic bayesian networks: representation, inference and learning. PhD thesis, UC Berkeley, Computer Science DivisionGoogle Scholar
  44. Newman-Norlund RD, van Schie HT, van Zuijlen AMJ, Bekkering H (2007) The mirror neuron system is more active during complementary compared with imitative action. Nat Neurosci 10(7):817–818PubMedCrossRefGoogle Scholar
  45. Newman-Norlund RD, Bosga J, Meulenbroek RGJ, Bekkering H (2008) Anatomical substrates of cooperative joint-action in a continuous motor task: virtual lifting and balancing. Neuroimage 41(1):169–177PubMedCrossRefGoogle Scholar
  46. Noordzij ML, Newman-Norlund SE, de Ruiter JP, Hagoort P, Levinson SC, Toni I (2009) Brain mechanisms underlying human communication. Front Hum Neurosci 3:14PubMedCrossRefGoogle Scholar
  47. Pacherie E (2008) The phenomenology of action: a conceptual framework. Cognition 107:179–217PubMedCrossRefGoogle Scholar
  48. Pezzulo G (2008) Coordinating with the future: the anticipatory nature of representation. Mind Mach 18(2):179–225CrossRefGoogle Scholar
  49. Pezzulo G (2011) Grounding procedural and declarative knowledge in sensorimotor anticipation. Mind Lang 26:78–114Google Scholar
  50. Pezzulo G (submitted) Shared representations as coordination tools for interactionGoogle Scholar
  51. Pezzulo G, Castelfranchi C (2009) Thinking as the control of imagination: a conceptual framework for goal-directed systems. Psychol Res 73(4):559–577PubMedCrossRefGoogle Scholar
  52. Prinz W (1990) A common coding approach to perception and action. In: Neumann O, Prinz W (eds) Relationships between perception and action. Springer, Berlin, pp 167–201Google Scholar
  53. Prinz W (1997) Perception and action planning. Eur J Cogn Psychol 9:129–154CrossRefGoogle Scholar
  54. Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2(1):79–87PubMedCrossRefGoogle Scholar
  55. Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192PubMedCrossRefGoogle Scholar
  56. Searle J (1990) Collective intentions and actions. In: Cohen JMPR M, Pollack E (eds) Intentions in communication. MIT Press, Cambridge, pp 401–416Google Scholar
  57. Sebanz N, Knoblich G (2009) Prediction in joint action: what, when, and where. Top Cogn Sci 1:353–367CrossRefGoogle Scholar
  58. Sebanz N, Bekkering H, Knoblich G (2006) Joint action: bodies and minds moving together. Trends Cogn Sci 10(2):70–76PubMedCrossRefGoogle Scholar
  59. Tomasello M, Carpenter M, Call J, Behne T, Moll H (2005) Understanding and sharing intentions: the origins of cultural cognition. Behav Brain Sci 28(5):675–691PubMedGoogle Scholar
  60. Tucker M, Ellis R (2004) Action priming by briefly presented objects. Acta Psychol 116:185–203CrossRefGoogle Scholar
  61. Vesper C, Butterfill S, Knoblich G, Sebanz N (2010) A minimal architecture for joint action. Neural Netw 23(8-9):998–1003PubMedCrossRefGoogle Scholar
  62. Van der Wel R, Knoblich G, Sebanz N (2010) Let the force be with us: Dyads exploit haptic coupling for coordination. J Exp Psychol Human Percept PerformGoogle Scholar
  63. Wilson M, Knoblich G (2005) The case for motor involvement in perceiving conspecifics. Psychol Bull 131:460–473PubMedCrossRefGoogle Scholar
  64. Wolpert D, Flanagan J (2001) Motor prediction. Curr Biol 11:729–732CrossRefGoogle Scholar
  65. Wolpert DM, Doya K, Kawato M (2003) A unifying computational framework for motor control and social interaction. Philos Trans R Soc Lond B Biol Sci 358(1431):593–602PubMedCrossRefGoogle Scholar
  66. Yoshida W, Dolan RJ, Friston KJ (2008) Game theory of mind. PLoS Comput Biol 4(12):e1000, 254+CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Istituto di Linguistica Computazionale “Antonio Zampolli”, CNRPisaItaly
  2. 2.Istituto di Scienze e Tecnologie della Cognizione, CNRRomaItaly
  3. 3.Computer Science EngineeringUniversity of PalermoPalermoItaly

Personalised recommendations