Where does fitness fit in theories of perception?
Interface theory asserts that neither our perceptual experience of the world nor the scientific constructs used to describe the world are veridical. The primary argument used to uphold this claim is that (1) evolution is driven by a process of natural selection that favors fitness over veridicality, and (2) payoffs do not vary monotonically with truth. I argue that both the arguments used to bolster this claim and the conclusions derived from it are flawed. Interface theory assumes that perception evolved to directly track fitness but fails to consider the role of adaptation on ontogenetic time scales. I argue that the ubiquity of nonmonotonic payoff functions requires that (1) perception tracks “truth” for species that adapt on ontogenetic time scales and (2) that perception should be distinct from utility. These conditions are required to pursue an adaptive strategy to mitigate homeostatic imbalances. I also discuss issues with the interface metaphor, the particular formulation of veridicality that is considered, and the relationship of interface theory to the history of ideas on these topics.
KeywordsVisual perception Comparative psychology
The “interface theory of perception” (hereafter, IT) is a theoretical framework that has been refined by Hoffman and colleagues for more than a decade, the main results of which are summarized in the target article. The main premise of IT is that evolution favors fitness over veridicality, and hence, there is no reason to believe that any aspect of our experience reflects the world “as it is.” Rather, perception should be construed as an adaptive interface, similar to the user interface that allows humans to interact with computers. I am in agreement with the central role of fitness and evolution in shaping perceptual systems, but I believe that there are issues with the arguments presented, the metaphor itself, and inferences that the authors draw based on these considerations. I begin with the metaphor.
The core concept of IT is that there is nothing in perception that resembles or reveals veridical properties of the world. Instead, just as the icons of a computer interface obscure the inner workings of the computer, our perceptual interface obscures “truth” about the world, providing us with merely an adaptive tool for reproduction. This is a provocative assertion, as there are some fundamental differences between computer interfaces and the world. For a realist, the interface metaphor fails because a computer interface does not reveal anything directly about either what the computer does or its structure. It is a tool engineered by man, and the interface is an artificial construct designed to allow humans to interact effectively with a tool. By contrast, a realist assumes that the structure that forms the basis of sensory input is created by the world. For vision, it is the interaction of light sources with the surfaces and materials of objects and substances that populate the world; for audition, it is the pattern of pressure waves created by mechanical disturbances; for the chemical senses, it is the distribution of and identity of chemicals; and so on. The bold claim of IT is that our concepts of the physical world in the preceding sentences are themselves merely adaptive interfaces and have nothing but a weak probabilistic bearing on the nature of the objective world as it “truly is.” The core argument offered by Hoffman et al. (hereafter HSP) in support of this radical view is that evolution favors fitness (“payoffs”), and because payoffs do not vary monotonically with truth there is no reason to believe that perceptual experience tracks veridical properties of the world. In what follows, I argue that this argument constitutes the central flaw of the theory and that the conclusion that evolutionary games demonstrate that interface strategies drive realist strategies (“veridicality”) to extinction only follows for the special case that they have considered.
Evolutionary games, veridicality, and fitness
In order to evaluate different theoretical views of perception, HSP articulate different perceptual “strategies” that compete in evolutionary games. For simplicity, I will consider all of the realist strategies as a single category. Each strategy is characterized by how it maps “world structure” onto perception. Their main claim is that “natural selection tunes perception to payoffs, not to truth” and that “payoffs and truth are different, unless payoffs happen to vary monotonically with truth.” The main argument that they make is that payoffs will not generically vary monotonically with truth, so there is no reason to believe that perception tracks truth. To make this argument, they construct hypothetical resource–payoff graphs (HSP's Figs. 2, 3, and 5). Figure 2 corresponds to what they define as a “weak critical realist.” In this figure, “resource quantity” is mapped onto one of four colors (presumably, color serves as an arbitrary categorical representation of a perceptual response). The four colors each map onto a different range of the resource quantity (for clarity, let’s say salinity). This is contrasted with the interface strategy in Fig. 3, where the same four colors are mapped monotonically onto payoffs (each of the four colors are chosen in such a way that they map onto the same range of payoff values on either side of the Gaussian). The evolutionary game is played using the currency of payoffs, not resources, so it is relatively trivial to see why the interface strategy would prove more successful in a game of this kind. Put simply, the strategy that better track payoffs will win a game in which payoffs determine winners. The question is whether this example provides the conceptual force needed to support HSP’s broader claim in favor of interface strategies over realist strategies or whether the example considered is, at best, a special case.
"Such a nonlinear payoff function is quite common: Not enough water and one dies of thirst; too much and one drowns; somewhere in between is just right. Similarly for salt and a variety of other resources" (p. 19).
It is here where the authors’ conflation of payoffs (utility) and perception is revealed. Consider the example of salinity. It is true that too little or too much salt can have deleterious effects on an animal and hence on their fitness. But this does not imply that the mapping of salinity (or any other resource critical for homeostasis) onto perception should track payoffs directly (and hence, non-monotonically). Indeed, I will argue that precisely the opposite is true. Consider what IT implies if the perception of salinity varied as diagrammed in Fig. 3. If the different colors that encode different magnitudes of a resource quantity were perceptual states (experiences), such as “amount of salt,” then their figure implies that each response that has the same color should generate the same experience. This would imply that a very small amount of salt (say, the amount encoded as the red color as the left end of the Gaussian) is experienced in exactly the same way as an excessive abundance of salt (the red color on the right end of the Gaussian). The critical point is that if perception is tuned directly to the payoff structure in Fig. 3, an animal will only how good or poor a payoff is; if it is poor, it provides no information about why the payoff is poor (i.e., whether it’s too little or too much) and hence provides no clue to the adaptive course of action for an animal.
It is here where the main flaw of HSP’s argument, which maps perceptual states directly onto payoffs, becomes apparent. An interface strategy of the type advocated in Fig. 3 provides no information about what an organism should do to achieve homeostasis; it makes the problem of homeostasis (or cooking) a complete guessing game. Do I need more or less salt to achieve homeostasis? Is there too little or too much salt in this dish? The demands of homeostasis do not imply that perception should be a nonmonotonic function of any resource quantity simply because the payoff (utility) function is nonmonotonic; it only requires that the preference for a particular amount of a given resource (like salt) be nonmonotonic. It is actually much worse than this; IT theory gets the argument exactly backwards: The separation of perception and utility is actually required for an animal to act adaptively within its lifetime. Like the cook, an animal needs to know the direction of the homeostatic imbalance to pursue an adaptive course of action. The interface strategy depicted in Fig. 3 is completely useless in this regard; all that an animal could know is that its resource level is “off” by some amount; there is no information about which direction it is off. In order to provide information that will allow an animal to pursue adaptive strategies within its lifetime, perceptual responses need to track resource quantities monotonically. HSP’s claim that “…we cannot expect, in general, that payoff functions vary monotonically with truth because…the ubiquitous biological need for homeostasis militate against them” (p. 23) is not simply incorrect, the precise opposite is true for animals that can adjust the behavior to mitigate such imbalances. The need for homeostasis requires that animal’s know why there is a current imbalance, not merely that there is an imbalance.
The crucial flaw in HSP’s argument is the authors have only considered the problem of adaptation over evolutionary time scales; they have ignored the need for (and demonstrated capacity of) animals to adjust their behavior to achieve homeostasis within ontogenetic time scales. For the games they considered, the only “force” of adaptation was through natural selection. In such cases, perception should indeed track utility directly. But the same doesn’t hold if animals can adjust their behavior to meet homeostatic demands on ontogenetic time scales. If the “payoff” function of homeostasis is nonmonotonic, as is typical, then the perceptual response needs to track resources monotonically so that an animal can know how to adapt its behavior to achieve homeostasis. When the concept of fitness is broadened to include adaptation both across and within generations, it becomes apparent why perception should not track utility (payoffs) directly. The implications of this seem clear. It is precisely the ubiquity of nonmonotonic payoff functions that require (1) dissociation between what an animal perceives and its utility and (2) perceptual states that vary monotonically as a function of resources (“truth”). Both conditions are required so that the difference signals between the perception states and payoffs provide information about an adaptive course of action. Thus, there is nothing in the structure of their argument that supports a general argument against veridical perception; but this also depends on how the concept of “veridical” is defined. I turn to this next.
Veridicality and illusion
HSP’s frame their discussion of veridicality by considering how different perceptual strategies map world states onto perceptual experience. This way of defining perceptual strategies presumes the very “God’s eye view” that HSP explicitly reject. The question is whether it makes any sense to frame the problem this way or whether it characterizes (or exhausts) realist views of perception and/or science. I contend that it does neither.
The problem with HSP’s characterization of perceptual strategies is that there is never any means by which to articulate the “objective world” states contained in W (HSP say remarkably little about what the “objective world” means, just that it exists). The states of W can only be considered in the abstract; there is no way to ascertain what those states are (they are not observables), and hence, no way of determining what counts as an isomorphism, homomorphism, or any other mapping from W into or onto any other set apart from the hypothetical world of imagined mappings. This misses a fundamental point about what constitutes both the objects of science and experience; the fundamental elements of system description in science are observables, not unspecifiable “world states.” The concept of veridicality considered (and rightfully rejected) by HSP is one in which states can be specified without reference to any sense of observation, what Koenderink (2014) has dubbed a “God’s eye view.” Such concepts have no meaning, and are rightfully rejected. The relationship between “world states” and “observables” has been thoroughly examined in Rosen’s (1978) seminal work on measurement and is beyond the scope of the present discussion (although it is of central importance for HSP’s thesis). For present purposes, I will focus on the consequence of replacing “world states” with “observables” in understanding the meaning of “veridicality” and the arguments offered by HSP.
HSP’s view is that veridicality entails a particular mapping between (inherently unspecifiable) world states and our experience. Because we have no access to the “true” world states, it is impossible to ever determine whether perception is veridical (in their sense) or in error, which leads them to the conclusion that such distinctions can only be defined in terms of adaptive behavior. However, if the fundamental “elements” to be compared in assessing veridicality are observables, the issue of veridicality becomes determining whether the equivalence classes generated by perceptual experience map onto (or into) the set of observables generated by some other given procedure of measurement. In this context, veridicality is a measure of the congruence between different sets of observables; it is not a measure of discordancy between perception and truth, because the latter has no meaning apart from an inaccessible God’s eye view. This is a very different understanding of what it would mean for perception to be “veridical” and is very different from the versions of veridicality and realism considered by HSP.
The preceding has a significant impact on our understanding of illusion and hallucination. The naive concept of illusion is defined by forming a difference between a perceptual state and something regarded as truth. This construal of the problem biases the solution and is responsible for why the concept of illusion remains muddled. The preceding arguments suggest that there is no single method for defining what counts as truth, because the “objective world” is at best a theoretical construct; all that can be done is to compare the output of perceptual “measurement” with some other procedure of measurement. This is why the problem of establishing a satisfactory definition of illusion is so elusive; it is unclear what measurements should be compared with perception to assess veridicality. It is admittedly common for perceptual scientists to simply choose a particular physically measurable property of the world (lightness, shape, depth, etc.) and ask how well the visual system “recovers” that property, treating any discordancy between physical measurements and perception as “error” or “illusion.” Such conclusions and choices are mere hubris; it presumes that the physically chosen observables are what the perceptual system tracks (i.e., what natural selection shaped the perceptual system to track; see Anderson, 2014). To demonstrate that perception is nonveridical, it must be shown that there is no set of physically specifiable observables that map onto the set of equivalence classes generated by perceptual states. This is exceedingly difficult to achieve in practice (if not impossible), which is why the concept of illusion is so difficult to define. But it does not follow from these pragmatic difficulties that perception can only be distinguished from “hallucination” on the basis of fitness. It is perfectly coherent (and sensible) to identify hallucination with a dissociation between an individual’s experience of what s/he believes to exist and other methods of observation, which is how the term is typically used.
Computational evolutionary psychology
HSP suggest that the standard Bayesian approach to perception be replaced by what they dub “computational evolutionary psychology” (CEP). The main difference between CEP and the traditional Bayesian view is that it makes world states inherently inaccessible. The world is replaced with an “interpretation space,” which is generated by a “pushdown measure” of the world onto two “representational spaces.” It is unclear what ontological status is to be given to this formalism, i.e., how X and Y are to be interpreted—an issue HSP equivocate on themselves. However, it is worth considering whether CEP has any pragmatic consequence for science if it proves true.
The striking metaphysical claim of IT is that our experience, and all other means of measuring the world, provide no information about the world “as it is,” but merely reveal properties of an adaptive interface. So what would change if this were true? There is nothing in CEP that ever allows us to establish what the true states of W are; the world is therefore inherently inaccessible in this framework. This generates skepticism about the ultimate veracity of the properties we ascribe to physical reality but is otherwise seemingly inconsequential. If the accessible world only consists of properties of an adaptive interface, then it is the lawful dynamics of the interface that are the objects of study and explanation. On this point I am in agreement with HSP: science will continue unaffected, even if IT proved to be correct. But the same is true with idealism: All may indeed be an illusion, adaptive or otherwise; but if so, the success of science demonstrates that the science of illusion is indistinguishable from the science of realism.
Given the importance evolutionary theory has played in our understanding of natural systems, it seems odd that it would take this long to get evolution “right,” to paraphrase HSP (who chide others for getting evolution “wrong”). In this section, I will consider how HSP contextualize their theoretical contribution. My intent is not to be exhaustive but to illustrate some of the historical antecedents to the ideas expressed in IT.
The PDA loop
"If we can assume no simple (e.g., isomorphic or homomorphic) correspondence between our representations and the objective world, how can we explain our successful interactions with the world? …in short the answer is that this is possible because we do not simply passively view the world, but also act on it, and moreover we perceive the consequences of those actions. In other words, it is possible to interact with a fundamentally unknown world if (i) the perceptual channels are fixed; (ii) there is a regularity in the consequences of our actions in the objective world; and (iii) these perceptions and actions are coherently linked." (p. 50)
"John Dewey was a leading proponent of the American school of thought known as pragmatism.… On this view, inquiry should not be understood as consisting of a mind passively observing the world and drawing from this ideas that if true correspond to reality, but rather as a process which initiates with a check or obstacle to successful human action, proceeds to active manipulation of the environment to test hypotheses, and issues in a re-adaptation of organism to environment that allows once again for human action to proceed. With this view as his starting point, Dewey developed a broad body of work encompassing virtually all of the main areas of philosophical concern in his day." (The Internet Encyclopedia of Philosophy, http://www.iep.utm.edu/dewey/)
“A propertied thing X (e.g., a crevice) affords an activity Y (e.g., crawling into) for a propertied thing Z (e.g., a lizard) if and only if certain properties of X (e.g., the spatial extent of the crevice in the horizontal dimension) are dually complemented by certain properties of Z (e.g., the substantial width of the lizard in the horizontal dimension), where dual complementation of properties translates approximately as properties that are related by a symmetrical transformation or duality T such that: T(P1 ) → P2 and T(P2) → P1 …..
A propertied thing Z (an organism) can effect activity Y with respect to a propertied thing X (an environmental situation) if and only if certain properties of Z are dually complemented by certain properties of X. In other words, affordances and effectivities are dispositional properties of things referring to a thing’s potentialities-to what can happen. As such they are to be distinguished from occurrent properties, the properties that a thing is currently exhibiting be they discernible by naked observation or aided observation (say, through a microscope)…” (p. 261).
It is somewhat ironic that previous theories of perception that have been grounded in concepts of adaption, fitness, and ecology are so divergent metaphysically and epistemologically from that proposed by IT. It should be clear that it is possible to embrace the significance of fitness in shaping evolution without arriving at the epistemological and metaphysical conclusions of IT.
The main assertion of interface theory is that our perceptual systems evolved to track payoffs rather than truth. HSP base this assertion on evidence from evolutionary games in which strict interface strategies putatively drive realist strategies to extinction. I have argued that such games are oversimplified and that the conclusion is false. Interface strategies that map nonmonotonic utility functions onto a monotonic perceptual scale provide an unsigned difference signal that provides no information about how an animal should adjust its behavior to achieve homeostatic balance. It also flies in the face of experience; there are many perceptual scales that vary monotonically with physical resources, including the sensory laws that spawned the birth of psychology as a natural science. The problem of understanding precisely what it is about the world that our perceptual systems evolved to sense is nontrivial, and many aspects of our experience may not map onto our a priori conceptions of what the world “is” or how it is defined by physicists. But remarkable theories require remarkable evidence, and there is currently insufficient evidence to take the metaphysical leap proposed by interface theory.
- Anderson, B. L. (2014). Can computational goals inform theories of vision? Topics in Cognitive Science, (in press).Google Scholar
- Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.Google Scholar
- Koenderink, J. J. (2014). The all seeing eye? Perception, 43, 1–6.Google Scholar
- Rosen, R. (1978). Fundamentals of measurement and representation of natural systems. New York: Elsevier North-Holland, Inc.Google Scholar