For a cognitive neuroscience of concepts: Moving beyond the grounding issue
Cognitive neuroscience research on conceptual knowledge often is discussed with respect to “embodiment” or “grounding.” We tried to disentangle at least three distinct claims made using these terms. One of these, the view that concepts are entirely reducible to sensory-motor representations, is untenable and diminishing in the literature. A second is the view that concepts and sensory-motor representations “interact,” and a third view addresses the question of how concepts are neurally organized—the neural partitions among concepts of different kinds, and where these partitions are localized in cortex. We argue that towards the second and third issues, much fruitful research can be pursued, but that no position on them is specifically related to “grounding.” Furthermore, to move forward on them, it is important to precisely distinguish different kinds of representations—conceptual vs. sensory-motor—from each other theoretically and empirically. Neuroimaging evidence often lacks such specificity. We take an approach that distinguishes conceptual from sensory-motor representations by virtue of two properties: broad generality and tolerance to the absence of sensory-motor associations. We review three of our recent experiments that employ these criteria in order to localize neural representations of several specific kinds of nonsensory attributes: functions, intentions, and belief traits. Building on past work, we find that neuroimaging evidence can be used fruitfully to distinguish interesting hypotheses about neural organization. On the other hand, most such evidence does not speak to any clear notion of “grounding” or “embodiment,” because these terms do not make clear, specific, empirical predictions. We argue that cognitive neuroscience will proceed most fruitfully by relinquishing these terms.
KeywordsConcepts Embodied cognition Neuroimaging Semantic memory
What does it mean to study concepts?
When cognitive scientists launched a campaign to peer into the black box of the human mind, they aimed for a computational account—that is, one that would describe distinct components of cognition by the nature of their representations and the algorithms that operate on them. That some of these processing components also have a separable neural basis allowed cognitive neuropsychology to propel this enterprise forward, by delineating components that, due to neurological damage, led to selective impairments to certain kinds of representations and algorithms (Caramazza, 1992; Shallice, 1988).
The existence of an independent conceptual processing component at the neural level was identified using just this kind of approach. In 1975, Elizabeth Warrington described a set of patients who, despite intact abilities to match pictures of different viewpoints of an object, could neither name nor describe those objects, nor match names to descriptions (Warrington, 1975). These dissociations brought Warrington to the conclusion that “perceptual classification and semantic classification are hierarchically organized systems and can be differentially impaired” (p. 652), a conclusion supported by later work (Hodges, 1992; Hillis & Caramazza 1995). The neural separability of perceptual1 and semantic representations allows cognitive neuroscience to specifically probe the properties of semantic, or conceptual, components. Our purpose in this review was to describe how neuroimaging research can contribute to this enterprise.
It is hard to imagine any empirical fact that could undo Warrington’s conclusions. Yet, this seems to be the very thrust of the embodiment program, extreme versions of which claim that concepts are entirely reducible to modality-specific sensory or motor representations (Barsalou, 2008; Pulvermüller & Fadiga, 2010). This divergence of claims might have arisen, because the computational approach to the mind has fallen by the wayside of current research in cognitive neuroscience. The color and light show provided by neuroimaging makes it easy to watch the brain as it responds to words or pictures, rather than to distinguish carefully the types of representations computed by the various processing components engaged in these complex tasks. We argue that issues in the embodiment debate can be clarified by returning to a more computational approach.
Criteria for individuating conceptual-level representations
In order to target conceptual representations with fMRI, it is important to have criteria for determining whether a neural signal indicates the operation of a conceptual level of representation—such as those Warrington developed to interpret patients’ behavior. One can start with a core assumption: that the conceptual level of representation is one that allows generalization across a wide range of specifics; broader even than across multiple viewpoints of the same object. Beyond objects, a concept can span tokens with nothing perceivable in common (atheists, beauty) or indeed nothing perceivable at all (ideas, truth). It has no particular time or place, allowing us to generalize and predict properties of novel instances. These properties distinguish conceptual representations from both episodic and sensory-motor ones.
In this framework, what a representation refers to and what kind of representation it is are distinct (Caramazza, Hillis, Rapp, & Romani, 1990). A concept can refer to a perceivable thing, but the concept square is distinct from the sensory representation square. The content of the former refers to a wider range of things and participates in certain kinds of cognitive operations that the latter, by virtue of being specific, cannot—and vice versa (Jackendoff, 1987; Smith & Medin, 1981). Importantly, both sensory-motor and conceptual representations can be retrieved from memory. This is the key difference between our approach, which characterizes representations by virtue of properties such as generality, and approaches that characterize representations by the way they are accessed (e.g., retrieved from memory vs. encoded bottom-up as a percept). Because memory retrieval can engage either kind of representation—sensory and conceptual—as we define here, it is essential to apply additional criteria to distinguish them.
Another distinct property between conceptual and sensory-motor representations is that the acquisition and use of the latter is strictly tied to the operation of a specific sensory channel. Without vision, one cannot acquire the representations necessary to match different views of a couch. But one can understand the meanings of the word couch and determine its synonyms: a cardinal ability enabled by semantics (Chierchia, 2006). This is the ability that allows the blind to wield rich concepts, even those about vision—such as shiny or bright—of which they have no sensory experience (Koster-Hale, Bedny, & Saxe, 2014; Landau & Gleitman, 1988; Shepard & Cooper, 1992). Thus, one can separate conceptual from sensory aspects of knowledge by looking at populations where the sensory component is not available—as in the case of concepts in the blind. Following similar logic, in populations with normal sensory experience, one can use concepts that do not refer to specific sensory qualities (less like square and more like beauty or gravity) to tease apart these types of representation.
In sum, the extent of generalization and tolerance to absence of sensory experience are two criteria, either of which one can use to tease apart sensory-motor from conceptual representations. In section 4, we show examples of how this can be achieved with fMRI.
Versions of embodiment: what is really at stake?
Among theoretical issues about concepts, embodiment has been particularly influenced by neuroimaging evidence. The embodied stance proposes that concepts are “grounded” in sensory-motor or modality-specific systems, rather than being “amodal” in some way (Barsalou, Kyle Simmons, Barbey, & Wilson, 2003; Simmons et al., 2007; Tranel, Kemmerer, Adolphs, Damasio, & Damasio, 2003). The nature of the claim relies on the interpretation of these quoted terms. Because they have been used in multiple ways, various distinct empirical claims have been conflated. In its most extreme form, “grounding” is a denial of any division between conceptual and sensory levels of representation: it is a negative claim that there is no independent level of conceptual representation, offering an exactly opposite view to the one we have just outlined. More commonly, “grounding” refers to the idea that concepts and percepts interact. Finally, evidence about the neural organization of conceptual knowledge has been taken to weigh in on the question of “grounding,” which give the term a yet different sense. Each sense of “grounding” is thus a distinct claim, and only the first of these is a claim about the nature of concepts. We take up each sense of “grounding” in turn to illustrate this point and to suggest that the term has become meaningless (in current discussions of the representation of concepts). We suggest pivoting the discussion in a new direction: investigation of concepts using neuroimaging should principally focus on the distinct question of neural organization, and that the term “grounding” should be either recast or forgotten.
“Theories generally assume that knowledge resides in a modular semantic system separate from episodic memory and modality-specific systems for perception, action and emotion. These theories further assume that conceptual representations are amodal—unlike representations in modality-specific systems—and operate according to different principles [ . . . ] Increasingly, researchers propose that conceptual representations are grounded in the modalities.” (Barsalou et al., 2003, p. 84).
We thus infer that “grounded,” according to Barsalou, means a lack of separation between semantic and sensory-motor systems and entails failure to find any representations meeting our criteria for concepts in the brain: that is, representations that have distinct properties from sensory-motor ones. All concepts, under this view, consist entirely and only of sensory-motor knowledge. Positive evidence against this view is plentiful: the kind of evidence we opened with, and that distinct kinds of neuropsychological damage give rise to distinct disorders, some modality-specific and some modality-general (see Caramazza & Mahon, 2006 for an extensive review). This claim also offers no clear, testable account of concepts that do not refer to concrete aspects of the world. Thus, either reductionism cannot be right or the account needs to better specify what is meant by the terms “modal” and “amodal” (see Caramazza et al., 1990, for detailed discussion)—the reason it has been severely criticized for being circular (Aydede, 1999).
Much other research has recently fallen under the umbrella of “embodied” or “grounded” cognition that is not—and cannot be—used to support a reductionist claim. In these cases, “grounded” appears to denote the idea that conceptual and sensory-motor representations interact: that retrieving one rapidly engages the other, or that both are evoked by retrieving the meanings of words (Meteyard, Cuadrado, Bahrami, & Vigliocco, 2012, for recent review).2 This view is clearly distinct from reductionism because it does not oppose, but rather presupposes, the idea that the levels are distinct—such that they could interact to some or other degree. Support for this view does not imply reducibility: representations do not become reducible by virtue of interacting. A visual representation (e.g., a pen) that is tightly associated with a motor representation (e.g., the physical movement of writing) is not itself motor by virtue of this fact. Similarly, one might possess a concept pen having never physically used one, but acquiring the motor experience of writing does not change the original concept into a motor representation. Thus, questions about how such associative links affect cognition are orthogonal to those about the reducibility of concepts to sensory-motor representations, and also to the nature of concepts.
Much of the evidence put forward as supporting embodiment supports interaction, not reducibility. For example, evidence of tightly coupled conceptual and motor representations (Pulvermüller, Hauk, Nikulin, & Ilmoniemi, 2005) is not evidence that concepts are themselves motoric; nor are demonstrations of fast or automatic interaction between verbal or conceptual and motoric representations or tasks (Andres, Finocchiaro, Buiatti, & Piazza, 2015; Hauk, Davis, Kherif, & Pulvermüller, 2008; Wheatley, Weisberg, Beauchamp, & Martin, 2005; Yee, Chrysikou, Hoffman, & Thompson-Schill, 2013) evidence for motoric representations of concepts. Likewise, the activation of neural regions associated with sensory or motor information during conceptual retrieval (e.g., Hauk, Johnsrude, & Pulvermüller, 2004) is evidence about interaction, and not about reducibility (even granting that those regions are purely sensory-motor, which is not always evident). There is extensive treatment of this argument elsewhere (Bedny & Caramazza, 2011; Binder & Desai, 2011; Chatterjee, 2010; Leshinskaya & Caramazza, 2014b; Mahon & Caramazza, 2008; Mahon, 2014).
Taking the issue of interactivity as its own claim, however, makes it clear that there is little at stake. Although authors generally have framed the question as a continuum of interactivity (Binder & Desai 2011), it is unclear what theoretical position about concepts would deny that conceptual and sensory-motor representations make contact with each other. Even under a view that sensory-motor representations are highly modular, the simple recognition of an object from vision requires interactive links between sensory and conceptual knowledge. It is therefore unclear just how much interaction is needed to make concepts “grounded.” This ambiguity makes “grounding” either meaningless (where the opposite of grounding would mean we could not form associative links between any representations of different kinds) or based on a false assumption that by interacting with sensory-motor representations, concepts have a different nature than if they did not do so.
Principles of neural organization
Finally, “grounding” has been used as part of a claim about how concepts are neurally organized—a claim also distinct from reducibility, as it takes seriously the idea that there are conceptual (i.e., general) representations distinct from sensory-motor ones. The question of organization instead consists in two parts: the kinds of partitions that exist within the conceptual system (e.g., dissociable components that might be content-selective) and where these partitions are localized in cortex. This opens a wide hypothesis space, which has only begun to be explored. Yet, many reviews are complicit with the idea that what we do know about neural organization of concepts is to some extent consistent with “grounding” (McRae & Jones, 2013; Binder & Desai, 2011; Watson & Chatterjee, 2011; Patterson, Nestor, & Rogers, 2007; Martin, 2007; Thompson-Schill, 2003). In what way?
One proposal along these lines is the “sensory-motor” view. This view holds that even among conceptual (not just sensory-motor) representations, there is organization according to sensory-motor modality. That is, there are divisions among the neural substrates of concepts, and each component is characterized by the sensory or motor modality to which the concepts refer (Thompson-Schill, 2003). Thus, concepts of colors are stored in one subdivision, whereas concepts of sounds are stored in other subdivisions. In this way, partitions in semantic memory correspond to, or are analogous to, partitions in sensory and motor systems—even if not identical to them. This is consistent with neuropsychological data demonstrating conceptual deficits selective for color knowledge, for instance (Luzzatti & Davidoff, 1994; Miceli et al., 2001; Stasenko, Garcea, Dombovy, & Mahon, 2014).
This view also makes claims and predictions about the specific locations in which these attribute-selective effects occur. Color concepts are expected to reside near to sensory representations of color—regions that, when lost, lead to achromatopsia, deficits in color vision. Depending on how one interprets “near to,” this tends to hold up: color perception relies on the lingual gyrus, whereas color concepts are lost due to damage to a (substantially) more anterior region in the fusiform gyrus, corresponding to neuroimaging activations during color knowledge retrieval (Miceli et al., 2001; Simmons et al., 2007).
While this account explains certain empirical phenomena, it is unclear to what extent it supports any claim about the nature of conceptual representations, including that they are “grounded.” Yet reviews of such evidence do suggest that overlap and adjacency can be interpreted with respect to whether conceptual representations are embodied (Thompson-Schill, 2003; Meteyard et al, 2012).
We do not see what neural organizing principles—either the nature of divisions or their location—can tell us about the nature of those representations. If concepts about colors are clustered together in cortical space, it does not mean they are any more sensory than if they were scattered or grouped with other kinds of concepts. Neither does being cortically “close” to—or even overlapping with—sensory representations of color imbue them with sensory properties. This is because neural location cannot itself indicate the nature of a representation: even with good probabilistic knowledge of what kinds of representations might reside in an area, one cannot exclude the possibility that additional, distinct representations also reside there (Poldrack, 2006). In short, even if concepts were neurally organized by virtue of the modality of the content they referred to, they would still be concepts, under no threat of being reduced to sensory-motor representations (Caramazza, Anzellotti, Strnad, & Lingnau, 2014; Caramazza et al., 1990). It is unclear in what way concepts might become more “grounded” than they would be otherwise.
A different take on the implications of neural location have been expressed by Martin and colleagues (Martin, 2007; Martin & Chao, 2001; Simmons et al, 2007). For example, the adjacency between conceptual and sensory representations of color (among other phenomena) yielded the conclusion that “information about a particular object property, like its typical color, is stored in the same neural system active when that property is perceived” (Martin, 2007, p. 32) and that such evidence entails that “object concepts are grounded in perception and action” (p. 27). Thus, neural localization is taken as relevant to the question of “grounding,” and grounding is used here to mean that concepts participate in the cognitive processes of perceiving and acting on the world (at least, on our reading of this position). In this sense, it would be hard to disagree. Humans interpret the world using their rich conceptual knowledge of it, and no existing account of the mind claims that concepts are not used in perception or action. The alternative would mean, for example, that we plan actions without reference to abstract knowledge about goals or that we do not interpret the meaning of the visual world during perception. Nonetheless, neural separability—or overlap—between conceptual representations and sensory-motor ones is just as orthogonal to the issue of interactivity as it is to reducibility: neurally distant representations can interact in the brain, too.
It also is possible to interpret neural overlap evidence (as described above) as being not about distinct types of representations in memory (conceptual and sensory-motor), but rather, about memory retrieval and perceptual encoding. Such evidence may illustrate that the retrieval of content from memory relies on nearby neural areas as bottom-up encoding of similar content from an externally presented stimulus. With respect to the distinction made here, between conceptual and sensory-motor representations, this question is orthogonal, and would apply to both kinds of representations equally; under this interpretation, the term “concept” is simply employed in a different way than here.
In summary, although evidence about neural organization has been used to support the “grounding” of concepts, it is unclear what “grounding” means and how the evidence supports it. Localization does not tell us about the nature of representations (Caramazza, 1992). Patterns of adjacency and overlap could perhaps imply interactivity, but then it must be clarified what specifically about interaction entails “grounding”—why it is that interactions among distinct representations types (both in terms of content and levels of representations) is anything more than a general property of hierarchically and componentially structured cognitive systems.
Despite the varieties of claims put forward under the embodiment umbrella—about reducibility, interaction, and neural organization—only the first, reduction, truly makes a substantive claim about the nature of concepts: that concepts consist only and entirely of sensory-motor representations. This position is untenable on both theoretical and empirical grounds and is increasingly less prevalent in the literature. The extent to which concepts and sensory-motor representations interact, and how conceptual representations are neurally organized, are orthogonal to the core embodiment issue of the nature of concepts themselves. These issues are interesting and important in their own right, but do not impinge on any clearly defined view of “grounding,” other than the idea that conceptual representations participate in perception and action and that they communicate with sensory-motor representations. These claims are difficult to deny by any cognitive theory. There may be disagreement about the conditions or the nature of such interaction, but it is unclear what such differences entail about the nature of concepts and which kind of interactions count as “grounding” and which do not In short, whereas theories of interaction can be usefully fleshed out, the term “grounding” is uninformative and should be buried.
In contrast to the embodiment stalemate, the issue of the neural organization of concepts is one in which the field can move forward. Indeed, the questions of how the semantic system is partitioned, and where those partitions reside, are suited perfectly to neuroimaging. Answers to these questions could shed light on the broader principles of cortical organization, its evolutionary and developmental origins, and its cognitive implications. Thus, we devote the rest of the paper to this topic.
The sensory-motor view suggests that partitions of semantic memory are based on the sensory or motor quality of information those concepts refer to, such as color. But it is not clear that this characterizes all such partitions.3 The concepts examined in most of the prior literature have been those that possess specific sensory or motor associations; the sensory-motor view thus has never needed to confront evidence it cannot explain. However, it is clear that this view is unnecessarily restrictive: it has nothing to say about concepts not associated with any specific sensory-motor system—or, if it does, it is only at a very abstract level, in which action concepts of all kinds (including abstract ones such as protect or communicate) are termed “motor.” Thus, there may be additional, or broader, principles that explain the full space of concepts and attributes.
One possibility is that partitions exist in the conceptual system that are not characterized by referring to any sensory-motor modality. This would suggest that sensory-motor principles are not a general account of semantic organization, but a local account under a broader umbrella. Furthermore, it is possible that partitions previously thought to be characterized by a modality are not in fact limited to concepts of that modality, but also represent nonconcrete attribute types. Thus, to elucidate the general, or core set of, principles behind these partitions, a broad range of attribute types should be neurally localized—especially ones that do not correspond to particular sensory-motor modalities.
Nonsensory attributes: making room for other organizational principles
Whereas fMRI offers a useful tool for asking this question, extra effort is required to ensure that observations one makes with this instrument are specific to the conceptual system. In our recent work, we attempt to overcome both issues by localizing neural representations of a variety of specific nonsensory concepts: goals, beliefs, and functions. The localization patterns we find reinforce the idea that semantic memory is neurally organized into areas with preferential encoding of certain types of contents. They extend this idea by demonstrating that these contents can include nonsensory-motor attributes and that their neural loci can be individuated (Kellenbach, Brett, & Patterson, 2003). In so doing, these findings present additional insights into localization principles of content-selective semantic components—why they are localized as they are in the brain.
Selective activation in response to retrieving specific kinds of sensory attributes—such as form, color, size, action, taste, or sound (Goldberg, Perfetti, & Schneider, 2006; Kable, Kan, Wilson, Thompson-Schill, & Chatterjee, 2005; Kable, Lease-Spellmeyer, & Chatterjee, 2002; Kellenbach, Brett, & Patterson, 2001; Martin et al., 1995; Phillips, Noppeney, Humphreys, & Price, 2002; Simmons, Martin, & Barsalou, 2005)—have been used as evidence of partitions in semantic memory that are based on the sensory-motor content they refer to. However, these results are not specific to conceptual representations—they cannot distinguish between concepts of sensory-motor attributes and the sensory-motor representations coactivated with them. Thus, to the extent that a set of concepts is systematically associated with sensory-motor (or other) properties, selective response in some region to this class of concepts is not sufficient to establish which kind of representation it reflects.
On the other hand, targeting nonsensory-motor concepts—i.e., those that denote qualities or distinctions not correlated with any particular sensory-motor property—allows one to be more confident that they are localizing a conceptual representation, rather than associated sensory-motor knowledge. Furthermore, one can measure not just the selectivity, but also the generality of a representation explicitly, using techniques such as adaptation or pattern-similarity analysis (Grill-Spector & Malach, 2001; Haxby, Gobbini, & Furey, 2001; Mur, Bandettini, & Kriegeskorte, 2009). We apply both considerations in our experiments.
In the studies that we performed, we used specific rather than general nonsensory concepts because of the wealth of evidence that semantic memory has content-selective, neurally dissociable components (Blundo, Ricci, & Miller, 2006; Caramazza & Shelton, 1998; Hillis & Caramazza, 1991; Laiacona, Capitani, & Barbarotto, 1997; Lambon-Ralph, Howard, Nightingale, & Ellis, 1998; Miceli et al., 2001; Ochipa, Rothi, & Heilman, 1989; Vandenbulcke, Peeters, Fannes, & Vandenberghe, 2006; Warrington & Shallice, 1984). Given the distributed nature of concrete concepts, it is possible that nonsensory-motor concepts are likewise not all stored in a single neural locus. This might explain why experiments that contrast activation to “abstract concepts” and “concrete concepts” tend to find many, but inconsistent, regions (Binder, Westbury, McKiernan, Possing, & Medler, 2005; Bright, Moss, Longe, Stamatakis, & Tyler, 2007; Cappa, Perani, Schnur, Tettamanti, & Fazio, 1998; Goldberg, Perfetti, Fiez, & Schneider, 2007; Noppeney & Price, 2002; Rodríguez-Ferreiro, Gennari, Davies, & Cuetos, 2011; Skipper & Olson, 2013). If nonsensory-motor concepts are localized in multiple, content-selective partitions, then the particular regions observed in a single study will depend on which kinds are included in its stimulus set. This idea is supported by fMRI findings that show spatially distinct neural representations of abstract concepts, such as arithmetic and convince (Wilson-Mendelhall, 2013) and crime and time (Breining & Rapp, 2011). It therefore would be more fruitful to target each partition of nonconcrete concepts individually. The question is what these partitions might be. We drew on prior behavioral, neuroimaging, and patient data to formulate our hypotheses.
Intention attributes of action concepts
One nonsensory-motor property important to action concepts is their intention or purpose. The meaning of an action concept is not exhausted by the body movement it implies—indeed, some action concepts, such as imagine, denote no body movement at all. Many action concepts form categories of similar intended outcomes rather than similar movements. If someone said I was teaching the students, they were not necessarily expressing the fact that they stood in front of a room and waved their arms, but that they tried to impart some students with information, and this meaning would not change had they sat down, and not waved their arms, etc. The intended outcome is thus a central part of the core meaning of this concept.
Prior fMRI research has compared task conditions in which subjects think about the intention of an action vs. its body movements, and found greater activation in areas resembling those involved in theory of mind (de Lange, Spronk, Willems, Toni, & Bekkering, 2008; Hesse, Sparing, & Fink, 2009; Saxe & Kanwisher, 2003; Spunt & Lieberman, 2012; Spunt, Satpute, & Lieberman, 2011). These effects are likely due to explicit inference of the actors’ mental states and are interpreted as such (Spunt & Adolphs, 2014).
To find regions that respond to the retrieval of the semantic knowledge of the typical intentions of actions, independently of having to think about the mind of another person, we asked subjects to read names of actions conjugated in first person (e.g., I teach) and to judge how often they themselves generally perform this action (Leshinskaya & Caramazza, 2014a). To look for neural regions that responded to the presence of intention information, we contrasted trials where subjects read names of actions typically done on purpose with those that are typically performed by accident (e.g., teach vs. misunderstand). We orthogonally varied the extent to which the actions had associated body movements (e.g., kick vs. teach). We found that a posterior portion of the inferior parietal lobule, around the angular gyrus, responded to information about intentions: it showed greater activation to intentional actions than accidental ones, equally for bodily and mental actions. This suggests that this region is sensitive to a nonsensory-motor distinction among action concepts, perhaps representing the intention attributes of action concepts (what those actions are typically for). It therefore is an attribute-selective component of semantic, not perceptual, memory.4
Neural organization is a question of both the partitions of semantic memory and their locations. The location of intention-sensitivity in the angular gyrus was adjacent to, but nonoverlapping with, activation in response to a standard theory of mind task in the same participants. This is in line with the previously proposed principle that content-selective semantic effects tend to be found adjacent to other, content-related systems (Thompson-Schill, 2003). However, in this case, content-relatedness is not by virtue of a common sensory-motor modality. The source of the relatedness may be that action concepts, by denoting intentions, are instrumental to reasoning about others’ mental states (Koster-Hale, Saxe, Dungan, & Young, 2013; Young & Saxe, 2009). Precisely which commonality is the reason behind this topographic organization is a question we hope to tackle empirically; but sensory-motor modality—being neither a property of intentions nor of theory of mind—is not, in our view, a likely candidate.5
Concepts of belief traits
A similar picture emerges for concepts that refer to psychological concepts. Building on past work (Mitchell, Heatherton, & Macrae, 2002), we recently found that the right precuneus explicitly represents concepts of belief traits (Leshinskaya, Contreras, Caramazza, & Mitchell, in preparation). In this study, participants viewed names of social groups (e.g., atheists, evangelicals, economists) while comparing them pairwise in terms of one of two types of belief traits: political orientation (liberal vs. conservative), or spiritualism (spiritualist vs. materialist). Using a searchlight representational similarity analysis (Kriegeskorte, Goebel, & Bandettini, 2006; Mur et al., 2009), we found that the patterns of activation in the right precuneus captured a categorical distinction between conservative and liberal social groups, when participants focused on the political orientation dimension, and spiritual vs. materialist groups when participants focused on the spiritualism dimension, independently of the particular social groups that these traits applied to. This suggests that this part of right precuneus explicitly represents these belief trait categories and thus represents aspects of semantics that at minimum include ones that do not refer to any sensory-motor information. In other words, part of precuneus represents concepts, which cannot be characterized by any particular sensory-motor modality. Curiously, we found that this part of precuneus was adjacent to, but not overlapping with, theory of mind activations nearby, as localized in the same participants. This suggests that belief concepts follow a neural adjacency principle that is in line with the content—not sensory-motor modality—of those representations.
Abstract categories of functions
Mental properties are non-sensory-motor. Other concepts are irreducible to sensory-motor properties not by virtue of being invisible or inaudible, but because they form categories whose members are heterogeneous in their sensory-motor qualities and cannot be distinguished from nonmembers by virtue of any such property. This is true of many categories of object functions, which group together abstractly defined outcomes allowed by objects (such as decorate) that do not all share any common physical movement or sensory event.
In a recent study, we looked for regions that encode the distinction between two categories of functions: Decorate and Protect (Leshinskaya & Caramazza, in press). Participants attended to one of four functions as they looked at a common set of objects: keeping their body warm; protecting objects from water; decorating their house; and dressing up for a night out. These functions cohered into the broader categories of Decorate and Protect. The functions in each category had conceptually related outcomes but were distinct in terms of the physical manner in which they could be accomplished, given the objects being viewed. We used a searchlight representational similarity analysis to find brain regions whose patterns of activation captured these broad categories, by virtue of being more similar for pairs of same-category conditions (e.g., decorate house and dress up) than for different-category conditions.
We found such representations in anterior IPL, near the supramarginal gyrus (SMG). This finding was surprising: the SMG is most commonly found in studies of grasping, tool-use, and manual actions (Newman-Norlund, van Schie, van Hoek, Cuijpers, & Bekkering, 2010; Peeters et al., 2009; Peeters, Rizzolatti, & Orban, 2013) and interpreted as being the locus of kinematic representations rather than abstract concepts (Buxbaum, Shapiro, & Coslett, 2014). However, it also has shown cross-modal (motor to visual) classification of concrete actions (Hamilton & Grafton, 2006; Jastorff, Begliomini, Fabbri-Destro, Rizzolatti, & Orban, 2010; Oosterhof, Tipper, & Downing, 2013), demonstrating that its response is not specific to stimuli in a single modality. Our findings further extend the repertoire of this region to include distinctions that have no common reference to motor sensory qualities. We therefore suggest that this region contains a component of semantic memory that captures a specific kind of nonsensory-motor conceptual distinctions—the functions of objects. This repertoire can include more concrete knowledge, but the presence of nonsensory-motor information rules out the possibility that its content can be described, as a whole, as referring to knowledge about a specific sensory-motor modality. This finding has implications not only for our understanding of the neural basis of semantic knowledge, but also for our understanding of the regions of the brain that can be said to be “motor”: it directly illustrates that evidence showing lower-level representation in a region does not rule out the possibility of high-level representations in the same or a very close region. This result demonstrates the challenges associated with labeling regions and the problems with using the location of an activation to reverse-infer the operations or representations participants must have engaged during a task.
The surprising location of function concepts can help elucidate the principles of topography in the semantic system. Rather than being localized with other abstract concepts, or even with action intentions, abstract categories of object functions were found in adjacent or overlapping locations to those typically found to represent object manipulation. Yet, function categories are not necessarily formed over the same kind of information as categories of movements: functions often are, as in this case, defined by a common outcome—certain kinds of events— rather than grouping together common body movements. It is not necessary or obvious that they share a common sensory-motor modality. Rather, a more salient relationship between these representations is their common participation in the process of selecting and using objects and movements to achieve goals. We suggest that this may serve as an account of their relative adjacency in cortex.
We explored a variety of specific, nonsensory concepts—functions, intentions, and beliefs—and used analyses that made it fully possible to discover that these concept types (e.g., beliefs and functions) overlapped. However, this is not what we found. Instead, each kind of distinction was localized in a different part of cortex.6 These findings demonstrate that it is indeed possible to use fMRI to find localized effects of nonsensory conceptual distinctions—even if sensory properties sometimes seem to “speak louder” (Kellenbach et al., 2003). They demonstrate the existence of content-selective partitions within semantic memory that are not each characterized by a sensory or motor modality. Their localizations with respect to other neural systems appear to be governed by factors other than modality.
These results open up further questions. Why are different kinds of non-sensory concepts represented in distinct locations, and why those particular locations? A sensory-motor account could be true of concrete concepts, but it cannot work for the rest. Instead, in analogous fashion to the connectivity hypothesis of category-selective areas in the ventral stream (Mahon & Caramazza, 2011), we suggest that content-selectivity in the semantic system is driven by shared participation in computations with a common goal—a principle broader, and more precise, than a common sensory modality. This predicts that concepts are localized near (or best connected to) other processing components that rely on those concepts as inputs. Sometimes, these computational congruencies align with modality: for example, the goal of recognizing known objects by virtue of their form or color, but not always. The congruency could be the goal of understanding someone else’s state of mind, or using an object to achieve a goal. This principle is distinct from one based on the modality in which the concept was learned or to which the concept refers. However, our hypothesis must remain speculative until we are able to establish independently the role of different kinds of concepts in certain aspects of cognition and trace their participation in these aspects via the neural pathways between their respective neural substrates. Moving forward, one might formulate hypotheses about other partitions within the semantic system—and their localization—on the basis of common cognitive utility.
Summary and conclusions
We have described an approach to studying the semantic system that takes seriously the distinction between conceptual and sensory-motor levels of representation and identifies conceptual representations by virtue of their unique properties: broad generalization and content that does not correlate with any specific sensory properties.7 Distinguishing these levels explicitly makes it more clear what “grounding” could mean: that one could be reduced to the other; that they interact; or that their neural basis has some relation to sensory-motor systems. We have argued that only the first of these interpretations makes a claim about the nature of concepts, the core issue in the embodiment debate. But it is untenable given the evidence. The others speak to orthogonal issues: how concepts interact with other systems, and how they are spatially arranged in cortex. It is these orthogonal issues that are the most fruitful terrain for moving forward the study of concepts—but their resolution will not speak to the issue of “grounding.” In short, “grounding” has no currency in the current research landscape, and thus, neither does the embodiment debate.
The enterprise of probing the neural organization of semantic memory—understanding how it is divided, and the principles by which these partitions are spatially distributed in the brain—should move forward independently of the embodiment debate. Although such organization often is assumed to mirror those of perceptual systems, we argue that this does not have to be the case. Indeed, the human brain contains many new, expanded cortical areas that exhibit distinct circuit properties from evolutionarily older areas (Buckner & Krienen, 2013). It is highly likely that semantic representations reside, at least in part, in these new areas, whose organization could be radically different from that of sensory-motor systems. Our principal argument is that our understanding of the organizing principles of semantic memory is far from complete: both because of methodological challenges in localizing semantic representations and because of the truncated space of hypotheses that have been put to test.
We described a series of experiments that illustrate our contribution to this research goal. This work showed that distinct kinds of nonsensory concepts can be localized with fMRI and that they appear to be localized in distinct parts of cortex. In other words, there is selectivity in the semantic system to contents and distinctions that are not sensory-motor in nature: attribute types are not just modality types. The big, open question before us is why semantic memory has these divisions, and not others, and why they are placed in those particular cortical locations. We speculate that the interaction between concepts and other cognitive operations plays a key role in driving this cortical landscape.
We will primarily refer to the distinction between sensory and conceptual representations to make clear that the process of perception involves both kinds of representations.
One interpretation of this claim is that a word (a lexical representation) points to distinct kinds of representations, where some of these are concepts, while some others are sensory-motor memories. It is difficult to distinguish between the case where these components are tightly associated and where both contribute to what the word “means,” and so we do not address this issue here.
The question of which localization principles govern semantic representations should not be confused with the question of whether semantic memory is distributed at all, as opposed to being localized in a single part of the brain. Reviews often collapse these orthogonal questions into a single distinction between distributed, attribute-selective and modality-specific vs. single-locus and amodal semantic systems (Patterson, Nestor, & Rogers, 2007; Watson & Chatterjee, 2011). The question of whether semantic knowledge is distributed and attribute-selective, however, is independent of whether it is distributed by modality. The alternative to this claim is the possibility that semantic content is distributed by principles and attributes other than modality instead of, or in addition to, modality.
We further ruled out that the effects were due to differences in argument structure or emotional valence.
It may be worth noting that the adjacency/neural location evidence here is not used to infer the nature of the representations we localized, and thus is not a form of reverse inference. Rather, it is an observation about the neural location of two representations whose properties were inferred from their respective experiments.
Moving forward, it is important to ensure that this isn’t a trivial outcome of methodological factors. However, most of our findings converge with prior evidence from different methodologies. Furthermore, a substantial number of the same participants were involved in each experiment reviewed, making it unlikely to be due to individual differences. Nonetheless, the various contributions of task, content, and participant factors should be evaluated simultaneously in future research (Fedorenko, Duncan, & Kanwisher, 2013).
Further criteria may be needed once such regions are found with fMRI. It will be important to verify, using TMS for example, that the areas also are necessary for the use of a concept (such as synonymy judgment).
- Breining, B., & Rapp, B. (2011). Neural distinctions between categories of abstract and concrete words: A multi-voxel pattern analysis. In Poster presented at theThird Annual Neurobiology of Language Meeting, November 10-11, 2011. Annapolis, MD, USAGoogle Scholar
- Buxbaum, L. J., Shapiro, A. D., & Coslett, H. B. (2014). Critical brain regions for tool-related and imitative actions: A componential analysis. Brain: A Journal of Neurology. doi:10.1093/brain/awu111
- Caramazza, A. (1992). Is cognitive neuropsychology possible? Journal of Cognitive Neuroscience, 4(1).Google Scholar
- Chierchia, G. (2006). Formal semantics. In Encyclopedia of language & linguistics. Elsevier.Google Scholar
- Hodges, J. R. (1992). Semantic dementia: Progressive fluent aphasia with temporal lobe atrophy. Brain, 1783–1806. doi:10.1007/s12031-011-9586-3
- Kellenbach, M. L., Brett, M., & Patterson, K. (2003). Actions speak louder than functions: The importance of manipulability and action in tool representation. Journal of Cognitive Neuroscience, 30–46.Google Scholar
- Koster-Hale, J., Saxe, R., Dungan, J., & Young, L. L. (2013). Decoding moral judgments from neural representations of intentions. Proceedings of the National Academy of Sciences, 110(14), 8437–8439. doi:10.1073/pnas.1207992110/-/DCSupplemental.www.pnas.org/cgi/doi/10.1073/pnas.1207992110 CrossRefGoogle Scholar
- Laiacona, M., Capitani, E., & Barbarotto, R. (1997). Semantic category dissociations: A longitudinal study of two cases. Riabilitazione, 3, 441–461.Google Scholar
- Lambon-Ralph, M. A., Howard, D., Nightingale, G., & Ellis, A. (1998). Are living and non-living category-specific deficits causally linked to impaired perceptual or associative knowledge? Evidence from a category-specific double dissociation. Neurocase, 4(4), 311–338. doi:10.1080/13554799808410630 CrossRefGoogle Scholar
- Landau, B., & Gleitman, L. R. (1988). Language and experience: Evidence from the blind child. Cambridge, MA: Harvard University Press.Google Scholar
- Leshinskaya, A., & Caramazza, A. (2014b). Organization and structure of conceptual representations. In V. Ferreira, M. Goldrick, & M. Miozzo (Eds.), Oxford handbook of language production (pp. 118–133). Oxford: Oxford University Press.Google Scholar
- Leshinskaya, A., & Caramazza, A. (in press). Abstract categories of functions in anterior parietal lobe. Neuropsychologia. Doi: 10.1016/j.neuropsychologia.2015.01.014
- Martin, A., & Chao, L. L. (2001). Semantic memory and the brain: Structure and process. Current Opinion in Neurobiology, 11, 194–201.Google Scholar
- McRae, K., & Jones, M. (2013). Semantic memory. In D. Reisberg (Ed.), Oxford handbook of cognitive psychology. Oxford: Oxford University Press.Google Scholar
- Newman-Norlund, R., van Schie, H. T., van Hoek, M. E. C., Cuijpers, R. H., & Bekkering, H. (2010). The role of inferior frontal and parietal areas in differentiating meaningful and meaningless object-directed actions. Brain Research, 1315, 63–74. doi:10.1016/j.brainres.2009.11.065 CrossRefPubMedGoogle Scholar
- Oosterhof, N. N., Tipper, S. P., & Downing, P. E. (2013). Crossmodal and action-specific: Neuroimaging the human mirror neuron system. Trends in Cognitive Sciences, 1–8. doi:10.1016/j.tics.2013.04.012
- Peeters, R., Simone, L., Nelissen, K., Fabbri-Destro, M., Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2009). The representation of tool use in humans and monkeys: Common and uniquely human features. Journal of Neuroscience, 29(37), 11523–11539. doi:10.1523/JNEUROSCI.2040-09.2009 CrossRefPubMedGoogle Scholar
- Skipper, L. M., & Olson, I. R. (2013). Connectivity of cortical networks supporting social and nonsocial, abstract and concrete conceptual knowledge. In Poster presented at the Annual Meeting of the Society for Cognitive Neuroscience, 13-16 April 2013. San Francisco.Google Scholar
- Wilson-Mendenhall, C. D., Simmons, W. K., Martin, A., & Barsalou, L. W. (2013). Contextual processing of abstract concepts reveals neural representations of non-linguistic semantic content. Journal of Cognitive Neuroscience, 25, 920--935.Google Scholar