Psychonomic Bulletin & Review

, Volume 22, Issue 6, pp 1541–1545 | Cite as

Are icons sense data?

Brief Report


We argue that Hoffman, Singh, and Prakash (Psychon Bull Rev, this issue) have not made the case that "the language of space-time and physical objects is the wrong language for describing the true structure of the objective world." Further, we contend that, contrary to what Hoffman et al. claim, the perceptual icons posited by interface theory seem best taken to be sense data.


The interface theory of perception Sense datum theory 
As Hoffman, Singh, and Prakash (2014) point out, Palmer expresses the orthodox view in vision theory when he states:

Evolutionarily speaking, visual perception is useful only if it is reasonably accurate…Indeed, vision is useful precisely because it is so accurate. By and large, what you see is what you get. When this is true, we have what is called veridical perception. (1999, 6)

Like Hoffman, Singh, and Prakash (hereafter, HSP), we are quite skeptical of that claim. First, if nonveridical perception confers greater reproductive success on members of a population P than veridical perception does, then the former will proliferate in P. Second, we think that the evolutionary game simulations HSP discuss provide reason to believe that nonveridical perception is better tuned to fitness than veridical perception, at least under certain circumstances. Third, we think there is positive reason to believe that normal perception is typically nonveridical in certain respects. Palmer (1999, 95) himself comments at one point that objects don’t actually have the colors that we see them as having. If that is so, then visual experience is systematically inaccurate. So, we are with HSP in challenging the orthodoxy in question.

We part company with HSP, however, when they tell us: “the language of space-time and physical objects is the wrong language for describing the true structure of the objective world.”. The language of space-time and physical objects is the language of physics. We see nothing in HSP’s paper that should raise doubts about physicists’ deployment of the language of space-time and physical objects in characterizing the objective world.

HSP take themselves to be led to this view about the language of space-time and physical objects by their interface theory of perception (henceforth, ITP). They introduce ITP by drawing an analogy:

Our perception of space-time is analogous to the desktop, and our perception of objects and their properties is analogous to the icons on the desktop. Just as the language of desktops and icons is the wrong language for describing the structure of the computer, so also the language of space-time and physical objects is the wrong language for describing the true structure of the objective world.

The language of desktops and icons is indeed the wrong language for describing the structure of the computer. But it doesn’t follow from this that the language of desktops and icons doesn’t successfully describe anything. There are desktops and they often have icons on them. Moreover, space-time exists and there are physical objects in space-time. Were there no such thing as space-time, then general relativity theory would be false, or at least inapplicable to “the objective world.” Were there no physical objects, there would be no molecules, atoms, electrons, neutrons, or protons. HSP’s paper gives us no reason to doubt there are such things.

Perhaps HSP mean just that there are no “middle-sized physical objects.” (We say this because they write at one point as if there are electrons. We can’t tell whether they think there are planets, stars, galaxies, and so on. We ourselves think that Galileo saw the moons of Jupiter when he looked through the telescope, not his perception of the moons of Jupiter. The moons of Jupiter were there before he looked, and remain there to this day, whether or not anyone is seeing them.) If, however, there are no middle-sized physical objects, then there are no animal bodies or indeed organisms of any sort. If there are no organisms of any sort, then, trivially, there are no organisms that have evolved. If there are no organisms that have evolved, then it is utterly irrelevant for HSP to bring in evolution.

One page after HSP appear to deny the existence of physical objects, they tell us:

We must take our perceptions seriously not because they reveal the true structure of the world, but because they are tuned, by natural selection, to fitness. Fitness is a function of the objective world. However, a fitness function depends not just on the objective world, but also on the organism, its state, and action.

If there is fitness and it depends on organisms, their states, and their actions, then organisms, their states, and their actions exist. Aren’t organisms complex physical objects? Aren’t they wholly composed of cells, that are wholly composed of molecules, that are wholly composed of atoms, that are wholly composed of electrons, protons, and (save the hydrogen atom) neutrons?

HSP say things at various places that presuppose that they accept the existence of visual computations (of the sorts involved in, say, depth perception, perceptual organization, and shape perception). We believe in visual computations as well, but we believe that they are implemented by processes in the brain. The brain, of course, is a complex middle-sized physical object. As such, HSP must hold that it does not exist. But if the brain does not exist, then where are visual computations carried out? HSP offer no answer to this question.

HSP tell us, you’ll recall: “our perception of objects and their properties is analogous to the icons on the desktop.” To try to comprehend what they might mean here, let’s look at their example of seeing a long, brown rattlesnake. They say:

[W]hen one sees a long brown rattlesnake, this perception does not represent that something in the objective world is long and brown. Not because the perception is false or misleading or illusory, but because the snake perception is there to adaptively guide your behavior, not to distract you with irrelevant details about the true structure of the world.

If rattlesnakes are not part of the objective world, then one would think that organisms are not part of the objective world. But as we noted, if organisms are not part of the objective world, then biological evolution isn’t a process in the objective world. If it isn’t, what is the point of appealing to it? The above passage gives the impression that they think that you and your behavior are part of the objective world and that your snake perception is too. Indeed, your snake perception is, they say, “there to adaptively guide your behavior.” It is thus clear that they think that your snake perception causally influences your behavior. But if you are part of the objective world, then why isn’t the rattlesnake part of the objective world too? Indeed, isn’t the rattlesnake a cause of your perception?

Of course, quantum mechanics doesn’t talk about rattlesnakes as such (that is, using the term “rattlesnakes”). But the relevant issue is not whether quantum mechanics talks about rattlesnakes as such, but rather whether every rattlesnake will fall under some (enormously complex, of course) description or other that is couched in the vocabulary of quantum mechanics. It should be noted that it would be a mistake for a quantum theorist to deny the reality of middle-sized objects. As John Bell pointed out, there must be be ables (Bell 1988; see also Maudlin 2007). Quantum mechanics is the most highly confirmed physical theory we’ve ever had. But it is not possible to describe the evidence we have for it without saying things that entail the existence of various kinds of physical objects. We can’t justifiably appeal to evidence obtained from particle accelerators, for instance, without presupposing that there are particle accelerators. If there are no particle accelerators, then, trivially, there is no evidence obtainable from particle accelerators. We can’t justifiably appeal to evidence involving light beams passing through slits in walls, if there are no walls, and so on.

We have a strong sense that in a number of places, HSP are not expressing their own view correctly. In any case, be that as it may, we’ll now just focus on ITP.

There are, we think, rattlesnakes. You and I can, moreover, see the same rattlesnake. We can take a picture of it and send it to our friends to let them know what we encountered on our desert trip. We can also, unfortunately, even be bitten by the same rattlesnake. And we can shoot the same rattlesnake. When that happens, it’s not the case that you shoot your snake-perception with your gun-perception and I shoot my snake-perception with my gun-perception. How can one even shoot a perception or shoot with a perception?

It is a truism that there are rattlesnakes and we can see numerically one and the same rattlesnake. But HSP deny such truisms. Consider what they say about cars:

When you and I both look at your car, the car I experience is not numerically identical to the car you experience. (…) But the objective reality is not a car and doesn’t remotely resemble a car; moreover, the car of your experience is distinct from the car of my experience.

Why is the car that you visually experience, that you see, not numerically the same car that I visually experience, that I see? Can’t we see exactly the same car, for instance, my car sitting in the driveway? (I might want to buy your car. I certainly don’t want to buy my perception of your car.) We think that HSP are missing a familiar basic distinction here. There is a distinction between an experience (or perception) and what it is an experience (or perception) of. As G.E. Moore pointed out in his famous 1903 paper, “The Refutation of Idealism,” failure to draw that distinction will lead one down the garden path to idealism—the view that the whole of reality is fundamentally mental. (Moore rejected idealism and embraced sense datum theory.) We talk about shared experiences, but what we share are types of experiences. Your particular visual experience is not numerically identical to my visual experience. But it patently doesn’t follow from that fact that you and I can’t see, can’t visually experience, numerically one and the same car. The objects of experiences (or perceptions) are not identical with the experiences (or perceptions) themselves. Think otherwise, and you’ll be led down the garden path to idealism.

If HSP maintained that icons are sense data, they could avoid idealism and say much of what they want to say without denying that the language of space-time and physical objects correctly characterizes objective reality. HSP, however, vigorously deny that ITP is a sense datum theory. But independently of the fact that sense datum theory is superior to idealism, various of their claims suggest that ITP is a kind of sense datum theory. According to sense datum theory, to have a sense experience is to be directly aware of a sense datum, where a sense datum is a kind of mental entity. Sense data stand in sense experiences for things in the objective world. They are essentially private: you have your sense data and I have mine. Our perceptual access to the objective world is indirect: it occurs only via our awareness of our own sense data. Our acts of awareness of sense data are caused by happenings in the objective world, but things in the objective world don’t in any way resemble sense data. Visual sense data, for instance, are colored. But things in the objective world don’t have colors; at least not in the same sense. A sense datum can have the property of redness. Nothing in the objective world has the property of redness. Sense datum theorists allow, however, that things in the objective world can count as red in a derivative sense, namely in virtue of being disposed to cause us, in appropriate circumstances, to be aware of a red sense datum. Thus, they say that it is true that the fire engine in red. It’s true that the fire engine is red, because the fire engine is disposed to cause a normal perceiver to be aware of a red sense datum under normal circumstances.

HSP’s icons look to us to be sense data. So, let’s look at the four reasons that HSP give for denying that ITP is a sense datum theory.
  1. (1)

    They claim that, pace sense datum theory, ITP is compatible with the view that our awareness of perceptual icons depends upon computational operations performed on proximal input.

  2. (2)

    They claim that, pace sense datum theory, “the interface theory does not entail that perception is an act whose objects are sense data, or that sense data are an incorrigible foundation for an edifice of verified knowledge.”

  3. (3)

    They claim that the formal mathematical structure of ITP is in fact metaphysically neutral.

  4. (4)

    They claim that ITP rejects the view, constitutive of sense datum theory, that undergoing a given phenomenal state involves participating in a two-place, act-object relation.


As concerns (1), we agree that ITP is compatible with a computational approach to visual processing. But so is sense datum theory. It is perfectly compatible with sense datum theory to maintain that our mental acts of direct awareness of sense data are the causal result of computational processes. What makes the mental acts ones of direct awareness of sense data is not that they are uncaused or that vision doesn’t involve computational processes. In fact, sense datum theorists hold the causal theory of perception. Moreover, the view is perfectly compatible with the claim that vision involves computational processes. What makes the mental acts ones of direct awareness of sense data is rather that when we are aware of sense data, we are not aware of them in virtue of being aware of something else. (Notice that the relevant notion of directness is not the one at issue between computationalists and Gibsonians.) It is wide open to a sense datum theorist to say that that act of direct awareness is the result of underlying computational processes. That doesn’t even run against the spirit of sense data theory.

Claim (2) has two parts: first, that ITP is not committed to perception’s involving an awareness of sense data, and second, that ITP does not entail that sense data form an incorrigible foundation for knowledge. We agree with the second part, but this claim is not essential to a sense datum view. Sense datum theorists can allow that we don’t have incorrigible knowledge of the properties of sense data. Indeed, later sense datum theorists (see Jackson 1977) rejected the blanket view that we have incorrigible knowledge of the properties of sense data. As concerns the first part, on the other hand, it is precisely what is up for debate.

As concerns (3), we completely agree that the formal structure of ITP is metaphysically neutral. But what is relevant here is the interpretation of the formal notions involved in ITP. Specifically, what are the members of the set X of possible experiences? Our contention is that HSP’s remarks on this issue seem to commit them to the view that the members of X are episodes of being aware of perceptual icons, and that perceptual icons are mental entities. This looks to be sense datum theory.

Finally, turn to (4). If (4) is true, then ITP is not a sense datum theory. However, if ITP is not an act-object account of perceptual experience, then it is hard to know how to interpret the desktop metaphor by which ITP is introduced. When viewing a desktop, we seem to stand in a relation of perceptual awareness to the icons on the desktop. If we don’t stand in something like this type of relation to perceptual icons, then how is the desktop metaphor to be understood? Moreover, the rejection of act-object structure seems to conflict with what HSP write in other places. For example, the act-object conception is implicit in speaking, as they do, of “the car I experience,” and of “the space-time and physical objects of our perceptions.” When they speak of seeing a rattlesnake, they suggest that the rattlesnake that we see is an icon. Prima facie, these descriptions purport to denote objects to which we stand in perceptual or experiential relations. Moreover, consider the following. Either we are aware of icons or we are not. If we are not aware of icons during perceptual experience, then in what sense do we encounter them? What role do they play in perceptual experience? If, however, we are aware of icons, then we are either directly aware of them or only indirectly aware of them (that is, aware of them only by means of being aware of something that is not an icon). If we are only indirectly aware of icons, then what are we directly aware of? If we are aware of anything, there must be something that we are directly aware of. If we are directly aware of icons, and icons are, as HSP seem to hold, mental entities, then icons are sense data. It seems to us that HSP in fact hold that we are directly aware of icons, and that icons are mental entities. It thus looks to us as if HSP in fact hold a two-place act-object account of perceptual experience.

In arguing that ITP eschews the act-object account, HSP suggest that their view is similar to Coates’s (2007) critical realist position. We lack space to discuss Coates’s complex account in detail here, but we will make a few remarks. Coates states that perceptual experiences have two components: (1) A “phenomenal, non-conceptual component,” and (2) “The exercise of conceptual states, possibly at a low level…in virtue of which the subject represents aspects of the physical surroundings” (2007, 16). What is relevant is the first component, component (1). Coates suggests (2007, 203–204) that component (1) is distinct from a sense datum, because sense data are not constituents of one’s consciousness, one’s conscious states. He is mistaken, however, in claiming that if the phenomenal component is a constituent of one’s conscious state, then it is not a sense datum. Sense datum theorists offered an act-object account of conscious states. On the sense datum view, to be in a conscious state is to be directly aware of a mental entity, a sense datum, an entity that could not exist in a world without minds (see Jackson 1977). Since conscious states, on this account, are relations between subjects and sense data, sense data are constituents of conscious states. There is, however, a distinction between properties or attributes of conscious states and constituents of conscious states. Perhaps Coates wants to hold that the phenomenal component is an aspect of the conscious state, an attribute of it, a property of it, rather than an entity, and so deny that there are phenomenal constituents. We have no objection to that. But if phenomenal components are attributes of conscious states, then they are not icons. The icons on a desktop are not attributes, not properties. They are entities. We do not know how to understand HSP’s desktop metaphor unless perceptual icons are also entities, mental entities that are constituents of conscious states. If perceptual icons are indeed mental entities that are constituents of conscious states, then they are sense data.

In any case, it is clear, as HSP themselves note, that ITP is not Coates’s view. While Coates allows that the phenomenal, non-conceptual “components” of experience are “immediately present” in experience (2007, 215), he explicitly denies that we see or perceive them (see 2007, 17). The objects of perception, for Coates, are external objects. Coates would maintain that a snake is a physical object and that you and I can see numerically one and the same snake.

Two of the main objections to sense datum theory are ones that should not concern HSP at all. One objection is that sense data won’t play the epistemological role of providing an incorrigible foundation for empirical knowledge. As we noted, later sense datum theorists acknowledged that, but maintained that sense datum theory nevertheless provides the correct theory of perceptual experience. The second objection is that sense datum theory is incompatible with physicalism. If, for instance, when one has a red, round after-image, there exists something red and round that one is aware of, then physicalism is false (since there is no relevant physical thing that is red and round). Sense datum theorists, of course, embrace that consequence: they maintain that physicalism is false. But HSP seem to reject physicalism as well. If they indeed reject physicalism, then, surely, they won’t take the fact that sense datum theory is incompatible with physicalism to be an objection to sense datum theory.

There are additional issues that arise as concerns sense datum theory. But, whether or not icons are sense data, those very same issues arise for icons. First, where are perceptual icons located? Are they located in the brain, or, failing this, within some other structure in objective reality? If so, then why do they not count as parts of objective reality? If not, are they instead located in a type of private mental space? If so, what are private mental spaces? Second, if perceptual icons are indeed located in a private mental space, can they interact with constituents of objective reality? There is reason to think that they must. HSP’s PDA loop posits a mapping (D) from perceptions (X) to actions (G), and a further mapping (A) from actions to states of the world (W). This indicates that they allow that our actions can exert influence on the objective world, and that our actions are guided by our perceptions (and hence by our awareness of perceptual icons). Thus, perceptual icons must be capable of interacting with objective reality. But how can elements of a private mental space interact with things from outside that space? Third, assuming that perceptual icons in fact interact with constituents of objective reality, will a completed physics need to take icons into account? If so, why don’t such icons count as parts of objective reality after all? If not, then must we admit that physics is not a causally complete theory, that there are factors that cause motion that physics does not take into account?

These questions arise for sense data and icons alike. Our suggestion, as it happens, is that HSP simply endorse a type of sense datum theory, and then either offer answers to these questions, or else explain why they do not require answers. We don’t here maintain that these questions cannot be adequately answered. They provide a challenge to—not a refutation of—ITP. By endorsing sense datum theory, HSP can avail themselves of the sense datum theorist’s explanation of how we can have knowledge of the objective world, even though we are only ever directly aware of our own, private sense data. We can have knowledge of the objective world, because we can infer that the objective world has certain features (including a space-time structure) as the best explanation of our sense data. By endorsing the theory, they can as well avail themselves of the techniques sense datum theorists developed for showing how the fact that we are directly aware only of our sense data coheres with our ordinary, everyday views about there being trees, rattlesnakes, deserts, cameras, driveways, cars, and the like. The moral that a philosopher sympathetic with HSP’s discussion might well draw is that we should all take a look again at sense datum theory. But to leap instead to idealism, which it seems to us HSP come perilously close to doing, would be to make the error that Moore pointed out over a hundred years ago.


  1. Bell, J. S. (1988). Speakable and Unspeakable in Quantum Mechanics. Cambridge: Cambridge University Press.Google Scholar
  2. Coates, P. (2007). The Metaphysics of Perception: Wilfrid Sellars, Critical Realism, and the Nature of Experience. London: Routledge.Google Scholar
  3. Hoffman, D., Singh, M., Prakash, C. (2014) The interface theory of perception. Psychonomic Bulletin and Review. this issue.Google Scholar
  4. Jackson, F. (1977). Perception: A Representative Theory. Cambridge: Cambridge University Press.Google Scholar
  5. Maudlin, T. (2007). Completeness, supervenience, and ontology. Journal of Physics A: Mathematical and Theoretical, 40, 3151–3171.Google Scholar
  6. Moore, G. E. (1903). The Refutation of Idealism. Mind, 12(48), 433–453.CrossRefGoogle Scholar
  7. Palmer, S. (1999). Vision Science: Photons to Phenomenology. Cambridge: Bradford Book.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  1. 1.Department of PhilosophyRutgers UniversityNew BrunswickUSA

Personalised recommendations