The amodal brain and the offloading hypothesis
- 1.5k Downloads
In this article, I argue that a growing body of evidence shows that concepts are amodal and I provide a novel interpretation of the body of evidence that was taken to support neo-empiricist theories of concepts: the offloading hypothesis in the 1990s and 2000s.
KeywordsConcepts and categories Semantics
A large empirical literature would be expected to assess whether knowledge is modal or amodal, given the importance of the issue. Surprisingly, however, researchers have adopted amodal approaches largely for theoretical reasons—relatively little direct empirical evidence exists for them. Amodal theories have been attractive theoretically because they implement important conceptual functions, such as the type-token distinction, categorical inference, productivity, and propositions. Amodal theories also are formalizable and can be implemented in computer hardware. Conversely, indirect empirical evidence has accumulated for modality-specific representations in working memory, long-term memory, language, and thought.
In previous work, I have raised questions about the strength of the evidence for neo-empiricist theories of concepts (Machery, 2006, 2007; see also Dove, 2009, 2011; Mahon & Caramazza, 2008), and I have questioned whether these theories can really explain how concepts “implement important conceptual functions,” in particular how thought can express “propositions” (Machery, 2014). In this article, I will challenge the claim that there is little evidence for amodal theories of concepts, arguing that in fact a growing body of evidence shows that concepts are amodal. In Section “Introduction”, the difficult and often unacknowledged problem of clarifying the distinction between empiricist and amodal theories of concepts is examined. In Section “Perceptual vs. non-perceptual representational codes”, a growing body of behavioral, neuroimaging, and neuropsychological findings supporting amodal theories of concepts is reviewed. Section “Evidence for amodal concepts” provides a novel interpretation of the body of evidence, which in the 1990s and 2000s was taken to support neo-empiricist theories of concepts: the offloading hypothesis.
Perceptual vs. non-perceptual representational codes
In the present context, neo-empiricist and amodal theories of concepts disagree, not about the origins of concepts (whether concepts are innate or acquired) or about the content of concepts (what concepts represent), but rather about the representational code, or format, of concepts. I will not attempt to define “representational code,” but some examples may be useful. Binary and decimal coding of numbers are two different representational codes despite encoding the same content. A picture and its digitized representation can carry the same information despite being put in different codes.
Neo-empiricists hold that concepts and perceptual or motor representations have the same code; amodal theories deny it. This way of distinguishing neo-empiricist and amodal theories of concepts raises the following question: What distinguishes a non-perceptual representational code from perceptual or motor representational codes? While, as we will see, this remains a vexed question, it turns out that we can make progress without defining what makes a code perceptual.
Early-modern philosophers, such as Descartes and Hume, and 19th- and early 20th-century introspectionist psychologists debating about the format of concepts did not have to answer this difficult question. Because they took perceptual representations and conceptual thought to be conscious, they relied on introspection to determine whether concepts and perceptual representations were of the same kind (Külpe, 1902; Ogden, 1911). By contrast, contemporary philosophers and psychologists acknowledge that conceptual thought and perceptual representations need not be conscious and that they cannot simply rely on introspection to determine whether concepts and perceptual representations are of the same kind.
During the past two decades, two proposals for distinguishing non-perceptual and perceptual representational codes have been put forward. Barsalou (1999, 578) has proposed that perceptual representations are analogue (or “analogical” as Barsalou puts it). A representation is analogue if and only if there is an isomorphism between its content (what it represents) and some properties of its vehicle. A mercury thermometer is an analogue representation of temperature, because the column of mercury expands and contracts as a function of the temperature. A map is an analogue representation of an area, because greater distances on the map represent greater distances in the represented area. First, not all analogue representations are perceptual1: Analogue computers use analogue representations, and it would be a stretch to attribute perceptual representations to them just for that reason. Some neural systems (e.g., the system involved in numerosity estimation, which is discussed in the next section) also manipulate analogue representations, but are typically viewed as amodal. In addition, in contrast to Barsalou’s claim, it is unsatisfying to hold that perceptual representations are analogue. It is an unsettled empirical question whether the representations involved in perceptual processing are analogue and, if some are, whether all are.
Prinz (2002, chapter 5) holds that perceptual representations are simply the representations that occur in perceptual systems (mutatis mutandis for motor representations). This proposal is however uninformative, because perceptual systems are not delineated in a principled and uncontroversial manner. Where, in neurobiological terms, do perceptual systems stop? Are the temporal poles parts of the visual systems? Does the frontal cortex belong to any perceptual system?
The target of the criticism here is merely Prinz’s attempt to define perceptual representations by appealing to perceptual systems. As shown below, this is compatible with appealing to clear cases of perceptual systems to provide evidence that some representations are perceptual.
The dialectical situation may seem problematic: Neo-empiricists and amodal theorists disagree about whether the format of concepts is perceptual, but the main proposals for characterizing perceptual representational formats are unsatisfying. Furthermore, we should not hope to be in a position to identify the essential features of perceptual representational formats anytime soon. It is an empirical question what these features are, and we are unlikely to be able to identify them until we have a much better scientific understanding of perceptual processing.
Fortunately, we need not know how to characterize perceptual representational formats to make progress in the debate between neo-empiricism and amodal theories. There are several plausible empirical tests that provide evidence for the involvement of perceptual/motor or non-perceptual/motor representations in solving some experimental task. While the significance of these tests for the issue at hand may be subject to discussion when taken independently, convergence between their results provides strong evidence about the format of the representations involved. Space is lacking to review and discuss all of them in detail here, but it is useful to mention some.
First, one has evidence that a task recruits perceptual or motor representations when performance is influenced—either improved or degraded—by specific perceptual experiences or actions. For instance, if entertaining a concept x involves entertaining a motor representation of a specific action (say, pushing away), then judging that something is an x when doing the opposite action (say, pulling toward oneself) is likely to be difficult. This phenomenon is known as “modal interference” (Kashak et al., 2005) and the opposite phenomenon is known as “modal facilitation” (Vermeulen, Mermillod, Godefroid, & Corneille, 2009). Furthermore, if entertaining a concept involves reenacting a perceptual representation in a particular perceptual modality, performance is likely to be worse when participants have to then reenact a perceptual representation in another modality compared with when they reenact a perceptual representation in the same modality, because in the latter case the perceptual system is already active (a form of priming). For instance, if concepts of perceptual properties (e.g., colors or sounds) are tokened in the relevant perceptual systems (respectively, V4 and BA 41 and 42), it should be more difficult to entertain the concept of a sound (e.g., thinking about how a motorbike sounds) after having entertained the concept of a color (e.g., having thought about the color of a fire truck) than after having entertained the concept of another sound (e.g., having thought about how a cow sounds). This greater difficulty may manifest itself experimentally in various ways (longer reaction times, greater probability of making a mistake, etc.). This phenomenon is known as “intermodal transfer costs” (Pecher, Zeelenberg, & Barsalou, 2003, 2004). Absence of modal interference/facilitation or modal transfer costs provides evidence that a task recruits amodal representations.
In neuroimaging, evidence that a task recruits uncontroversial perceptual areas (V1, V2, V4, MT, BA 41, and 42, etc.) provides evidence that perceptual representations are involved in solving this task. Naturally, activation in these areas may be incidental to the task of interest. For instance, Garcea, Dombovy, and Mahon (2013) have compellingly argued that activation in posterior parietal regions in tool naming tasks results from the automatic computation of object-directed action information but is not necessary for tool naming, as evidenced by patients who are still able to name tools despite damage in posterior parietal regions. However, while activation can be incidental, neuroimaging evidence that converges with other forms of evidence can provide strong evidence about the format of representations. Furthermore, if entertaining the concept of x (say of dogs) consists in reenacting seeing, hearing, etc., x, then brain activation should be distributed across the perceptual streams. Finding that entertaining the concept of x activates a single area (unicity of activation) would then be evidence that the concept is amodal, provided that this area is not a clear case of a perceptual system.
Evidence for amodal concepts
A growing body of evidence now supports the claim that there are many kinds of amodal concepts and, by enumerative induction, that concepts are amodal. In this section, I will review some of the most striking findings, focusing on behavioral, neuroimaging, and neuropsychological results. Together, these findings provide strong evidence that many kinds of concepts are amodal and that perhaps all concepts are amodal.
The most striking behavioral findings supporting the existence of amodal concepts come from the research on the number sense (Cantlon, 2012; Dehaene, 2011; see Dove, 2009 and Machery, 2007 for further discussion). Visual estimation of the number of objects in a set of objects and auditory estimation of the number of sounds in an auditory sequence obey Weber’s law: The standard deviation of participants’ estimations linearly increases with the mean number of objects and sounds (Dehaene, Dehaene-Lambertz, & Cohen, 1998). The best and widely accepted explanation of these findings is that a single system is involved in both visual and auditory estimations of numerosity rather than two systems that just happen to obey the same law. Furthermore, there is little to no intermodal transfer costs in tasks that involve manipulating representations of numbers. In particular, performance is similar when participants are asked to add their estimations of the number of objects in two visually presented sets and when they are asked to add their estimation of the number of objects in a visually presented set and their estimation of the number of sounds in an auditory sequence (Barth et al., 2006). Finally, newborns are able to match visual and auditory representations of numbers (Izard, Sann, Spelke, & Streri, 2009). For instance, when familiarized with a sequence of 4 sounds, 2-day old newborns prefer to look at 4 objects rather than 12 objects. Again, the best explanation of these findings is that visual and auditory estimations of numerosity recruit a single amodal system rather than two systems that just happen to be coordinated at birth.
Neo-empiricists may respond that the system involved in numerosity estimation is a perceptual system, just not a modality-specific one; rather, numerosity estimation involves a multimodal perceptual system. The data reviewed above do not distinguish between this hypothesis and the claim that numerosity estimation is amodal, but this hypothesis comes with a theoretical cost for neo-empiricists: It considerably undermines the parsimonious nature of their approach—which was supposed to be a virtue of their approach—because neo-empiricists now need to appeal to multimodal systems that mimic amodal systems. This neo-empiricist response would be more compelling if neo-empiricists explained how multimodal systems are to be distinguished empirically from amodal systems, but this difficult theoretical challenge has not been met.
Neuroimaging results provide further support for amodal theories of concepts. Neuroimaging provides compelling evidence that a single system, located in part in the right and left intraparietal sulcus, is involved in estimation of numerosity, independent of the stimuli’s perceptual modality. Piazza, Mechelli, Price, and Butterworth (2006) elicited very similar activation in a right lateralized frontoparietal cortical network (including the rIPS) for sets of visually presented objects and sequences of sounds. At this point, it seems beyond doubt, in fact it is widely accepted, that an amodal system underlies magnitude estimation.
Further neuroimaging results show that other kinds of representation are amodal too. A growing body of findings suggests that concepts of actions are amodal. Bedny and colleagues (2008) presented subjects with verbs and nouns whose meaning was associated more or less strongly with motion. No difference was found in MT and STS—areas involved, respectively, with motion and biological motion processing—between verbal stimuli (verbs and nouns) semantically associated with motion and those that are not. Bedny, Caramazza, Pascual-Leone, and Saxe (2012) found that activation is very similar in sighted and congenitally blind individuals when presented with verbs and nouns whose meaning is more or less strongly associated with motion. If entertaining a concept of an action involves reenacting perceiving this action, one would expect activation to differ in sighted people, who have extensive visual acquaintance with the relevant actions, and blind people, who are not so acquainted. In response, neo-empiricists could respond that these studies have focused on the left middle temporal gyrus (involved in the perception of actions) instead of motor regions (involved in actions). The neuropsychological data reviewed next address this shortcoming (see also Mahon, 2015 for further discussion).
Finally, concepts of mental states also seem to be amodal. Activation of the right temporoparietal junction, an area involved in mindreading, does not seem to depend on the stimuli’s perceptual modality. The rTPJ is activated by verbal stimuli (psychological words or stories eliciting mindreading) and by drawings. Consistent with this observation, Bedny, Pascual-Leone, and Saxe (2009) have provided evidence that the network involved in mindreading (including the rTPJ) is similar in congenitally blind and sighted adults.
Finally, neuropsychological findings provide further evidence for the amodal nature of concepts. Apraxia is a deficit in using objects such as tools. If entertaining the concept of a tool involves reenacting its use, then in people with apraxia, tool concepts should be impaired, and this impairment should manifest itself in other uses of these concepts, such as tool recognition. However, Negri and colleagues (2007) showed that apraxia and impairment of object recognition can be doubly dissociated (see also Garcea et al., 2013; see Garcea & Mahon, 2012 for a similar dissociation with healthy adults). Some patients are unable to use some objects but can recognize them, whereas other patients are able to use some objects but are unable to recognize them (for further review, see Mahon & Caramazza, 2005). Semantic dementia provides more general evidence about the amodal nature of concepts (McCaffrey & Machery, 2012). This form of frontotemporal dementia is characterized by bilateral degeneration of tissue around the anterior temporal lobe (ATL) and a progressive loss of conceptual knowledge. Patients experience debilitating impairments in their ability to recognize, name, and access relevant information about objects. Neo-empiricism predicts that patients with semantic dementia should lose the capacity to represent the features in a given perceptual modality (e.g., all visual features) across all concepts, whereas the capacity for representing the features in other modalities should be preserved (a “modality-specific” conceptual loss). For instance, patients with semantic dementia should be unable to reenact visual experiences of dogs, apples, hammers, and cars, while being able to reenact auditory or olfactory experiences of these objects. Neo-empiricism makes this prediction because conceptual knowledge is meant to be distributed across perceptual systems, which are localized in different brain regions. When semantic dementia impairs a brain region, only the modality-specific knowledge “housed” in this region should be destroyed. In contrast, amodal theories lead us to expect conceptual loss independently of perceptual modality. Patients with semantic dementia should be entirely unable to think about, e.g., dogs (what dogs look like, smell like, sound like, do, etc.), not just be unable to reenact visual or auditory experiences of dogs. Semantic dementia patients exhibit highly multimodal deficits related to specific affected concepts: Patients lose all feature knowledge (visual, auditory, tactile, etc.) for specific concepts. This pattern of deficits (a modality-general conceptual loss) is challenging for neo-empiricism but consistent with amodal theories of concepts.
In summary, a growing body of behavioral, neuroimaging, and neuropsychological evidence suggests that concepts are actually amodal.
The offloading hypothesis
While the growing body of evidence just reviewed suggests that concepts are amodal, there also is a large body of evidence allegedly supporting the neo-empiricist consensus (for a recent review, see Kiefer & Pulvermüller, 2012). How can this empirical conflict be resolved? In the last section of this article, I describe several possible resolutions.
First, it may have been a mistake to interpret some findings as supporting the neo-empiricist consensus. Neo-empiricists sometimes underestimate the resources of amodal models of concepts and concept processing (e.g., concept combination) and mischaracterize the behavioral predictions these make (Machery, 2007). The evidential significance of neuroimaging results alleged to support neo-empiricism can be unclear since activation of perceptual areas may be a by-product of activation elsewhere (Mahon & Caramazza, 2008). It is however dubious that all findings alleged to support the neo-empiricist consensus can be explained away by reinterpreting the significance of empirical results since some findings (e.g., the findings in Landy & Goldstone, 2007a, b reported below) do show that low-level perceptual features influence performance in tasks meant to tap into conceptual knowledge.
Second, the format of concepts may vary across domains: Concepts in some domains may be amodal, while having a perceptual format in other domains (Dove, 2009; Machery, 2007). However, this hypothesis does not fully resolve the empirical conflict between neo-empiricist and amodal bodies of evidence since concepts within the same domain (e.g., concepts of action) sometimes give rise to contradictory findings.
A third possibility is that categories, substances, and events can be represented by several concepts, which could have a different format (in line with Machery, 2009, 2011; Rice, forthcoming). That is, cats (or water or dogs) could be represented by two distinct concepts of cats (or water or dogs): one with an amodal format, one with a perceptual format. A drawback of this resolution of the empirical conflict between neo-empiricist and amodal bodies of evidence is that it is not parsimonious. Even if one is already committed to the view that categories, substances, and events are typically represented by several distinct types of concepts, one takes on the additional theoretical commitment that perceptual representations are among these kinds of concepts. In addition, it becomes mysterious why paradigmatic concept-involving tasks appear to recruit both amodal and perceptual or motor representations. In contrast, the off-loading hypothesis (to be presented next) explains this phenomenon.
In the remainder of this section, I would like to put forward the most plausible resolution of this empirical conflict: “the off-loading hypothesis.” It is useful to contrast the off-loading hypothesis with neo-empiricism. As we have seen, according to neo-empiricism, we solve such tasks by manipulating perceptual and motor representations because our conceptual representations themselves have a perceptual or motor format: Thinking is reenacting perceiving or acting. In contrast, according to the offloading hypothesis, we often offload the solution of tasks on perceptual and motor systems: While concepts themselves are amodal, we often manipulate perceptual and motor representations to solve tasks. One may wonder why we have amodal concepts at all if we can offload cognitive tasks onto perceptual systems. A plausible response is that not all tasks can offloaded.
The offloading hypothesis calls for empirical investigation and specification. Why do we offload the solution of tasks on perceptual and motor systems? Offloading may happen when the conceptual system does not encode the information needed for solving a given task (e.g., information about perceptual details), while perceptual representations stored in memory do. Offloading also may happen for tasks that can be efficiently solved this way. Perceptual cues may underlie perceptual heuristics that allow people to solve tasks reliably and quickly. For instance, Goldstone and colleagues have shown that simple visual cues are used to structure and solve arithmetical equations (Landy & Goldstone, 2007a, b). Given the way that equations are typically written (i.e., given the environment), these perceptual cues reliably lead to correct answers. A second open question is whether offloading happens automatically or (at least sometimes) intentionally. A third open question is whether offloading is done strategically and flexibly, with people relying on different perceptual or motor heuristics in different contexts (perhaps because these heuristics are most efficient in these contexts), or whether it is an inflexible strategy that people follow whether or not it is efficient.
The offloading hypothesis makes sense of the empirical conflict between neo-empiricist and amodal bodies of evidence: The former body of evidence shows that concepts are amodal, the second that people often solve tasks by manipulating perceptual and motor representations. If this interpretation of the empirical conflict is correct, the influential and important research guided by neo-empiricist ideas has shown not so much that concepts have a perceptual or motor representational format, but rather that offloading is a fundamental heuristic for our cognitive life. What we have learned is the unexpected extent to which people offload to solve conceptual tasks.
Viewing the past 20 years of neo-empiricist research through the lens of the offloading hypothesis has the potential to revigorate this area of research. Instead of looking for further effects showing that perceptual and motor variables influence performance in tasks tapping into higher cognition in an often atheoretical manner, psychologists and cognitive neuroscientists could focus on addressing the questions raised earlier in this section—the why and when of offloading, its efficiency given the nature of the environment, the control we have over it, and its strategic nature.
Descartes may well have been right and Hume wrong: A growing body of evidence suggests that concepts are amodal and that thinking does not essentially consist in reenacting perceiving and acting. On the other hand, we have learned over the past 20 years that perceptual and motor reenactment plays a much larger role in our cognitive life—we offload much of our cognitive duties on perceptual and motor systems—than amodal theorists would have expected two generations ago.
Note that Barsalou does not assert all are.
- Dehaene, S. (2011). The number sense: How the mind creates mathematics. Oxford: Oxford University Press.Google Scholar
- Kaschak, M. P., Madden, C. J., Therriault, D. J., Yaxley, R. H., Aveyard, M., Blanchard, A. A., & Zwaan,R. A. (2005). Perception of motion affects language processing. Cognition, 94, B79–B89.Google Scholar
- Landy, D., & Goldstone, R. L. (2007a). How abstract is symbolic thought? Journal of Experimental Psychology: Learning, Memory, & Cognition, 33, 720–733.Google Scholar
- Machery, E. (2014). Neo-empiricism and the structure of thoughts. In P. Calvo & J. Symons (Eds.), The architecture of cognition: Rethinking Fodor and Pylyshyn's systematicity challenge (pp. 335–352). Cambridge: MIT Press.Google Scholar
- Prinz, J. J. (2002). Furnishing the mind. Concepts and their perceptual basis. Cambridge: MIT Press.Google Scholar
- Rice, C. (Forthcoming). Concepts as pluralistic hybrids. Philosophy and Phenomenological Research.Google Scholar