Experimental Brain Research

, Volume 230, Issue 3, pp 345–357 | Cite as

Supramodal and modality-sensitive representations of perceived action categories in the human brain

  • Richard Ramsey
  • Emily S. Cross
  • Antonia F. de C. Hamilton
Research Article
First Online:
Received:
Accepted:

Abstract

Seeing Suzie bite an apple or reading the sentence ‘Suzie munched the apple’ both convey a similar idea. But is there a common neural basis for action comprehension when generated through video or text? The current study used functional magnetic resonance imaging to address this question. Participants observed videos or read sentences that described two categories of actions: eating and cleaning. A conjunction analysis of video and sentence stimuli revealed that cleaning actions (compared to eating actions) showed a greater response in dorsal frontoparietal regions, as well as within the medial fusiform gyrus. These findings reveal supramodal representations of perceived actions in the human brain, which are specific to action categories and independent of input modality (video or written words). In addition, some brain regions associated with cleaning and eating actions showed an interaction with modality, which was manifested as a greater sensitivity for video compared with sentence stimuli. Together, this pattern of results demonstrates both supramodal and modality-sensitive representations of action categories in the human brain, a finding with implications for how we understand other people’s actions from video and written sources.

Keywords

Language Reading Motor system Action observation Social cognition 
This is a preview of subscription content, log in to check access.

Supplementary material

221_2013_3659_MOESM1_ESM.docx (2 mb)
Supplementary material (DOCX 2046 kb)

References

  1. Adolphs R (2009) The social brain: neural basis of social knowledge. Annu Rev Psychol 60(1):693–716. doi: 10.1146/annurev.psych.60.110707.163514 PubMedCrossRefGoogle Scholar
  2. Andersen RA (1997) Neural mechanisms of visual motion perception in primates. Neuron 18(6):865–872PubMedCrossRefGoogle Scholar
  3. Aziz-Zadeh L, Damasio A (2008) Embodied semantics for actions: findings from functional brain imaging. J Physiol Paris 102(1–3):35–39PubMedCrossRefGoogle Scholar
  4. Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M (2006) Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Curr Biol 16(18):1818–1823PubMedCrossRefGoogle Scholar
  5. Barrós-Loscertales A, González J, Pulvermüller F, Ventura-Campos N, Bustamante JC, Costumero Vc, Parcet MaA, Ávila Cs (2011) Reading salt activates gustatory brain regions: fMRI evidence for semantic grounding in a novel sensory modality. Cereb Cortex. doi: 10.1093/cercor/bhr324
  6. Baumgaertner A, Buccino G, Lange R, McNamara A, Binkofski F (2007) Polymodal conceptual processing of human biological actions in the left inferior frontal lobe. Eur J Neurosci 25(3):881–889. doi: 10.1111/j.1460-9568.2007.05346.x PubMedCrossRefGoogle Scholar
  7. Beauchamp MS, Martin A (2007) Grounding Object Concepts in Perception and Action: evidence from FMRI Studies of Tools. Cortex; a journal devoted to the study of the nervous system and behavior 43(3):461–468PubMedCrossRefGoogle Scholar
  8. Binkofski F, Buccino G, Zilles K, Fink GR (2004) Supramodal representation of objects and actions in the human inferior temporal and ventral premotor cortex. Cortex J Devot Study Nerv Syst Behav 40(1):159–161CrossRefGoogle Scholar
  9. Boulenger V, Hauk O, Pulvermuller F (2009) Grasping ideas with the motor system: semantic somatotopy in idiom comprehension. Cereb Cortex 19(8):1905–1914. doi: 10.1093/cercor/bhn217 PubMedCrossRefGoogle Scholar
  10. Brass M, Schmitt RM, Spengler S, Gergely G (2007) Investigating action understanding: inferential processes versus action simulation. Curr Biol 17(24):2117–2121PubMedCrossRefGoogle Scholar
  11. Braze D, Mencl WE, Tabor W, Pugh KR, Todd Constable R, Fulbright RK, Magnuson JS, Van Dyke JA, Shankweiler DP (2011) Unification of sentence processing via ear and eye: an fMRI study. Cortex J Devot Study Nerv Syst Behav 47(4):416–431CrossRefGoogle Scholar
  12. Brewer AA, Liu J, Wade AR, Wandell BA (2005) Visual field maps and stimulus selectivity in human ventral occipital cortex. Nat Neurosci 8(8):1102–1109PubMedCrossRefGoogle Scholar
  13. Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13(2):400–404PubMedGoogle Scholar
  14. Buccino G, Riggio L, Melli G, Binkofski F, Gallese V, Rizzolatti G (2005) Listening to action-related sentences modulates the activity of the motor system: a combined TMS and behavioral study. Cognit Brain Res 24(3):355–363CrossRefGoogle Scholar
  15. Carota F, Moseley R, Pulverm√ºller F (2012) Body-part-specific representations of semantic noun categories. J Cognit Neurosci 24(6):1492–1509. doi: 10.1162/jocn_a_00219 CrossRefGoogle Scholar
  16. Caspers S, Zilles K, Laird AR, Eickhoff SB (2010) ALE meta-analysis of action observation and imitation in the human brain. NeuroImage 50(3):1148–1167PubMedCrossRefGoogle Scholar
  17. Cross ES, Kraemer DJM, Hamilton AFDC, Kelley WM, Grafton ST (2009) Sensitivity of the action observation network to physical and observational learning. Cereb Cortex 19(2):315–326Google Scholar
  18. de Lange FP, Spronk M, Willems RM, Toni I, Bekkering H (2008) Complementary systems for understanding action intentions. Curr Biol 18(6):454–457 Google Scholar
  19. Downing PE, Peelen MV (2011) The role of occipitotemporal body-selective regions in person perception. Cogn Neurosci 2:1–24Google Scholar
  20. Duvernoy HM (1999) The human brain: surface, blood supply, and three-dimensional sectional anatomy. Springer, New YorkGoogle Scholar
  21. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, et al (2005) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage 25(4):1325–1335Google Scholar
  22. Friston KJ, Worsley KJ, Frackowiak RSJ, Mazziotta JC, Evans AC (1994) Assessing the significance of focal activations using their spatial extent. Hum Brain Mapp 1(3):210–220CrossRefGoogle Scholar
  23. Gallese V, Lakoff G (2005) The brain’s concepts: the role of the sensory-motor system in conceptual knowledge. Cognit Neuropsychol 22(3–4):455–479. doi: 10.1080/02643290442000310 CrossRefGoogle Scholar
  24. Gallese V, Keysers C, Rizzolatti G (2004) A unifying view of the basis of social cognition. Trends Cognit Sci 8(9):396–403CrossRefGoogle Scholar
  25. Gazzola V, Keysers C (2009) The observation and execution of actions share motor and somatosensory voxels in all tested subjects: single-subject analyses of unsmoothed fMRI data. Cereb Cortex 19(6):1239–1255. doi: 10.1093/cercor/bhn181 PubMedCrossRefGoogle Scholar
  26. Glenberg AM, Sato M, Cattaneo L, Riggio L, Palumbo D, Buccino G (2008) Processing abstract language modulates motor system activity. Q J Exp Psychol 61(6):905–919CrossRefGoogle Scholar
  27. Goldberg RF, Perfetti CA, Schneider W (2006a) Distinct and common cortical activations for multimodal semantic categories. Cognit Affect Behav Neurosci 6(3):214–222. doi: 10.3758/cabn.6.3.214 CrossRefGoogle Scholar
  28. Goldberg RF, Perfetti CA, Schneider W (2006b) Perceptual knowledge retrieval activates sensory brain regions. J Neurosci 26(18):4917–4921. doi: 10.1523/jneurosci.5389-05.2006 PubMedCrossRefGoogle Scholar
  29. Graziano MSA, Cooke DF (2006) Parieto-frontal interactions, personal space, and defensive behavior. Neuropsychologia 44(6):845–859PubMedCrossRefGoogle Scholar
  30. Grèzes J, Decety J (2001) Functional anatomy of execution, mental simulation, observation, and verb generation of actions: a meta-analysis. Hum Brain Mapp 12(1):1–19PubMedCrossRefGoogle Scholar
  31. Grèzes J, Frith C, Passingham RE (2004a) Brain mechanisms for inferring deceit in the actions of others. J Neurosci 24(24):5500–5505PubMedCrossRefGoogle Scholar
  32. Grèzes J, Frith CD, Passingham RE (2004b) Inferring false beliefs from the actions of oneself and others: an fMRI study. Neuroimage 21(2):744–750PubMedCrossRefGoogle Scholar
  33. Hamilton AF, Grafton ST (2006) Goal representation in human anterior intraparietal sulcus. J Neurosci 26(4):1133–1137PubMedCrossRefGoogle Scholar
  34. Hauk O, Johnsrude I, Pulvermüller F (2004) Somatotopic representation of action words in human motor and premotor cortex. Neuron 41(2):301–307PubMedCrossRefGoogle Scholar
  35. Haynes J-D, Rees G (2006) Decoding mental states from brain activity in humans. Nat Rev Neurosci 7(7):523–534PubMedCrossRefGoogle Scholar
  36. Hein G, Knight RT (2008) Superior temporal sulcus—it's my area: or is it? J Cogn Neurosci 20:2125–2136Google Scholar
  37. Kamitani Y, Tong F (2006) Decoding seen and attended motion directions from activity in the human visual cortex. Curr Biol 16(11):1096–1102PubMedCrossRefGoogle Scholar
  38. Keysers C, Gazzola V (2007) Integrating simulation and theory of mind: from self to social cognition. Trends Cognit Sci 11(5):194–196CrossRefGoogle Scholar
  39. Killgore WDS, Young AD, Femia LA, Bogorodzki P, Rogowska J, Yurgelun-Todd DA (2003) Cortical and limbic activation during viewing of high-versus low-calorie foods. NeuroImage 19(4):1381–1394PubMedCrossRefGoogle Scholar
  40. Kilner JM, Neal A, Weiskopf N, Friston KJ, Frith CD (2009) Evidence of mirror neurons in human inferior frontal gyrus. J Neurosci 29(32):10153–10159. doi: 10.1523/jneurosci.2668-09.2009 PubMedCrossRefGoogle Scholar
  41. Mahon BZ, Caramazza A (2008) A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. J Physiol Paris 102(1–3):59–70PubMedCrossRefGoogle Scholar
  42. Makin TR, Holmes NP, Ehrsson HH (2008) On the other hand: dummy hands and peripersonal space. Behav Brain Res 191(1):1–10PubMedCrossRefGoogle Scholar
  43. Marsh LE, Hamilton AFdC (2011) Dissociation of mirroring and mentalising systems in autism. NeuroImage 56(3):1511–1519. doi: 10.1016/j.neuroimage.2011.02.003 PubMedCrossRefGoogle Scholar
  44. Martin A (2007) The representation of object concepts in the brain. Annu Rev Psychol 58(1):25–45. doi: 10.1146/annurev.psych.57.102904.190143 PubMedCrossRefGoogle Scholar
  45. Mitchell JP, Heatherton TF, Macrae CN (2002) Distinct neural systems subserve person and object knowledge. Proc Natl Acad Sci USA 99:15238PubMedCrossRefGoogle Scholar
  46. Mitchell JP, Banaji MR, Macrae CN (2005) General and specific contributions of the medial prefrontal cortex to knowledge about mental states. NeuroImage 28(4):757–762PubMedCrossRefGoogle Scholar
  47. Nichols T, Brett M, Andersson J, Wager T, Poline J-B (2005) Valid conjunction inference with the minimum statistic. NeuroImage 25(3):653–660PubMedCrossRefGoogle Scholar
  48. Noppeney U, Josephs O, Kiebel S, Friston KJ, Price CJ (2005) Action selectivity in parietal and temporal cortex. Cognit Brain Res 25(3):641–649CrossRefGoogle Scholar
  49. Peelen MV, Atkinson AP, Vuilleumier P (2010) Supramodal representations of perceived emotions in the human brain. J Neurosci 30(30):10127–10134. doi: 10.1523/jneurosci.2161-10.2010 PubMedCrossRefGoogle Scholar
  50. Pulvermuller F (2005) Brain mechanisms linking language and action. Nat Rev Neurosci 6(7):576–582PubMedCrossRefGoogle Scholar
  51. Ramsey R, Hamilton AF (2010) Understanding actors and object-goals in the human brain. NeuroImage 50:1142–1147PubMedCrossRefGoogle Scholar
  52. Ramsey R, Hamilton AF (2012) How does your own knowledge influence the perception of other people’s actions in the human brain? Soc Cognit Affect Neurosci 7:242–251CrossRefGoogle Scholar
  53. Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192PubMedCrossRefGoogle Scholar
  54. Rueschemeyer S-A, Pfeiffer C, Bekkering H (2010) Body schematics: on the role of the body schema in embodied lexical-semantic representations. Neuropsychologia 48(3):774–781PubMedCrossRefGoogle Scholar
  55. Shipp S, Zeki S (1995) Segregation and convergence of specialised pathways in macaque monkey visual cortex. J Anat 187:547–562PubMedGoogle Scholar
  56. Simmons WK, Martin A, Barsalou LW (2005) Pictures of appetizing foods activate gustatory cortices for taste and reward. Cereb Cortex 15(10):1602–1608. doi: 10.1093/cercor/bhi038 PubMedCrossRefGoogle Scholar
  57. Speer NK, Reynolds JR, Swallow KM, Zacks JM (2009) Reading stories activates neural representations of visual and motor experiences. Psychol Sci 20(8):989–999. doi: 10.1111/j.1467-9280.2009.02397.x PubMedCrossRefGoogle Scholar
  58. Spunt RP, Lieberman MD (2012) Dissociating modality-specific and supramodal neural systems for action understanding. J Neurosci 32(10):3575–3583PubMedCrossRefGoogle Scholar
  59. Spunt RP, Falk EB, Lieberman MD (2010) Dissociable neural systems support retrieval of how and why action knowledge. Psychol Sci 21(11):1593–1598PubMedCrossRefGoogle Scholar
  60. Spunt RP, Lieberman, Satpute MD (2011) Identifying the what, why, and how of an observed action: an fMRI study of mentalizing and mechanizing during action observation. J Cognit Neurosci 23(1):63–74CrossRefGoogle Scholar
  61. Toni I, de Lange FP, Noordzij ML, Hagoort P (2008) Language beyond action. J Physiol Paris 102(1–3):71–79PubMedCrossRefGoogle Scholar
  62. Uddin LQ, Iacoboni M, Lange C, Keenan JP (2007) The self and social cognition: the role of cortical midline structures and mirror neurons. Trends Cognit Sci 11(4):153–157CrossRefGoogle Scholar
  63. Vigneau M, Beaucousin V, Hervé PY, Duffau H, Crivello F, Houdé O, et al (2006) Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. NeuroImage 30:1414–1432Google Scholar
  64. Vigneau M, Beaucousin V, Hervé P-Y, Jobard G, Petit L, Crivello F, et al (2011) What is right-hemisphere contribution to phonological, lexico-semantic, and sentence processing?: insights from a meta-analysis. NeuroImage 54:577–593Google Scholar
  65. Willems R, Hagoort P (in press) Cortical motor contributions to language understanding. In: Hermer L (ed) Contributions of primary sensory and motor cortices to higher cognitive processes. Research Signpost press, Scarborough, OntarioGoogle Scholar
  66. Wunderlich G, Marshall JC, Amunts K, Weiss PH, Mohlberg H, Zafiris O, Zilles K, Fink GR (2002) The importance of seeing it coming: a functional magnetic resonance imaging study of motion-in-depth towards the human observer. Neuroscience 112(3):535–540PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Richard Ramsey
    • 1
  • Emily S. Cross
    • 1
    • 2
  • Antonia F. de C. Hamilton
    • 3
    • 4
  1. 1.Wales Institute of Cognitive Neuroscience, School of PsychologyBangor UniversityBangor, GwyneddUK
  2. 2.Behavioural Science Institute, Donders Institute for Brain, Cognition and BehaviourRadboud University NijmegenNijmegenThe Netherlands
  3. 3.School of PsychologyUniversity of NottinghamUniversity Park, NottinghamUK
  4. 4.Institute of Cognitive NeuroscienceUniversity College LondonLondonUK

Personalised recommendations