Abstract
Detecting visual changes can be based on perceiving, whereby one can identify a specific detail that has changed, on sensing, whereby one knows that there is a change but is unable to identify what changed, or on unconscious change detection, whereby one is unaware of any change even though the change influences one’s behavior. Prior work has indicated that the processes underlying these different types of change detection are functionally and neurally distinct, but the attentional mechanisms that are related to these different types of change detection remain largely unknown. In the current experiment, we examined eye movements during a change detection task in globally manipulated scenes, and participants indicated their change detection confidence on a scale that allowed us to isolate perceiving, sensing, and unconscious change detection. For perceiving-based change detection, but not sensing-based or unconscious change detection, participants were more likely to preferentially revisit highly changed scene regions across the first and second presentation of the scene (i.e., resampling). This increase in resampling started within 250 ms of the test scene onset, suggesting that the effect began within the first two fixations. In addition, changed scenes were related to more clustered (i.e., less dispersed) eye movements than unchanged scenes, particularly when the subjects were highly confident that no change had occurred – providing evidence for change detection outside of conscious awareness. The results indicate that perceiving, sensing, and unconscious change detection responses are related to partially distinct patterns of eye movements.
Similar content being viewed by others
Introduction
Noticing that something has changed can be the result of different types of thought processes. For example, when deleting redundant photos from your phone, you may realize that two similar family photos are slightly different. Perhaps you notice that your uncle has a particularly artificial smile in one photo and not the other. On the other hand, you may have a vague sense that something is different between the two photos, but even after careful inspection you cannot identify exactly what is different. Alternatively, you may think the photos look exactly the same, but your behavior may be altered in subtle ways that you are not aware of, such as differences in your pattern of eye movements while viewing each photo. These distinct routes to change detection not only involve different subjective experiences, but they can have very different behavioral consequences. Despite this, the perceptual and attentional mechanisms that are related to these different types of change detection have not been directly contrasted. In the present study, we test the hypothesis that these different types of change detection may be related to differences in how attention is deployed, and we do this by examining eye movements during a change detection task in scenes.
Research examining conscious visual change detection indicates that these perceptual discriminations can be characterized by at least two separable underlying processes: being able to identify the specific change (i.e., perceiving), and having a vague sense that something changed but failing to specifically identify the change (i.e., sensing; Aly & Yonelinas, 2012; Rensink, 2000). Behavioral studies have indicated that perceiving responses are thresholded in that they involve high levels of response confidence and occur abruptly during viewing, whereas sensing responses are strength-based in that they vary from weak to confident and gradually increase as changes are repeatedly viewed over multiple presentations (Aly & Yonelinas, 2012). In addition, perceiving responses occur most often when changes involve local, discrete alterations such as an object in a scene disappearing, whereas sensing responses occur most often when changes involve global alterations such as subtle spatial distortions to scenes (Aly & Yonelinas, 2012). Moreover, sensing-based responses are associated with activity throughout the ventral stream including the lateral occipital complex, fusiform gyrus, parahippocampal gyrus, and hippocampus, whereas perceiving-based responses most strongly relate to lateral parietal activity (Aly et al., 2013; Aly, Ranganath et al., 2014a; Aly, Wansard, et al., 2014b).
Despite the clear differences between perceiving and sensing, it is not yet known how they relate to attention. Prior work has indicated that when subjects detect a change, they are more likely to have looked at the changed regions within the scene (Hollingworth et al., 2001), and to have sampled fewer regions overall (Ryan & Cohen, 2004). In addition, when changes occur during saccadic eye movements, subjects are more likely to report those changes if they had previously fixated on the changed region (Henderson & Hollingworth, 1999; Hollingworth & Henderson, 2002). These results suggest that successful change detection is related to differences in visual attention as measured by eye movements (i.e., a less dispersed viewing pattern that is focused on the changed regions within the scenes). However, these studies only examined overall change detection accuracy, and so it is not known whether these eye movements reflect perceiving or sensing-based change detection.
In addition to the open question of the role of attention in perceiving and sensing, there is debate surrounding the role of unconscious processes in eye movements related to change detection. That is, whereas conscious change detection is related to increased viewing of changed regions, changes in overall viewing (e.g., fewer regions sampled and longer fixation durations) have been found even when subjects report that no change has occurred (Henderson & Hollingworth, 2003; Ryan & Cohen, 2004), suggesting that these eye movements may reflect a form of unconscious change detection. However, others contend that differences in eye movement patterns only occur when change detection is conscious (e.g., Smith et al., 2006; Smith & Squire, 2008). One potential concern – acknowledged by those authors – is that when measuring only a dichotomous (i.e., change/same) response, it is difficult to know whether participants were truly unaware of the change or whether they were just not sufficiently confident to make a “change” response. Thus, it is not known whether the eye movements that have been linked to unconscious change detection reflect truly unconscious effects or just low levels of sensing-based change detection.
Current research
To address these questions, we used a confidence-based change detection scale to isolate perceiving, sensing, and unconscious change detection, and examined how these processes relate to the deployment of visual attention. In the present experiment, participants viewed pairs of scenes presented sequentially while their eye movements were tracked. Every scene had a global distortion applied to it that subtly altered the relationship between scene elements (Fig. 1B and C). Global changes, rather than local changes, were used because prior work has established that global changes elicit high levels of both perceiving and sensing, whereas local changes elicit relatively little sensing-based change detection (Aly & Yonelinas, 2012). Each trial consisted of an initial scene presentation (i.e., study), a brief blank delay, and a second scene presentation (i.e., test), during which it either changed (i.e., changed trials) or remained the same (i.e., identical trials). On each trial, participants were asked to indicate whether they detected a change using a subjective report procedure (Aly & Yonelinas, 2012; Ramey et al., 2019), which allowed us to tease apart change detection based on perceiving, sensing, and unconscious memory. Participants responded “perceive” if they detected a change and could identify some aspect of the scene that had changed; otherwise, they rated their confidence on a scale from 1 (“sure the scene is the same”) to 5 (“sure the scene changed”). The “perceive” and 5 responses indicated that subjects were highly confident there was a change, and they were used to assess perceiving and sensing-based change detection, respectively. To index unconscious change detection, trials in which subjects were highly confident that there had been no change (i.e., the 1 responses) were examined.
To assess visual attention, we examined participants’ tendency to revisit regions between study and test (i.e., resampling) that were highly changed (i.e., high in change magnitude), as well as the overall spatial distribution of eye movements across scene regions (i.e., dispersion).
Although the role of eye movements in perceiving and sensing-based change detection has not yet been examined, prior findings led us to assess several general predictions. Namely, the link between perceiving and detecting local feature changes (Aly & Yonelinas, 2012) suggests that eye movements may be especially important for perceiving-based change detection: In particular, perceiving may rely on fixating the same highly changed regions before and after the change (Hollingworth & Henderson, 2002). Sensing, on the other hand, has been proposed to involve assessments of the strength of global match between two stimuli (Aly & Yonelinas, 2012), and therefore may be more dependent on gist or parafoveal vision than direct fixation. As a contrasting possibility, however, prior work has shown that the hippocampus is more involved in sensing than perceiving (Aly et al., 2013), and eye movements during scene perception are strongly related to hippocampal activity (Henderson & Choi, 2015; Liu et al., 2017, 2020; also see Ryan et al., 2000) – indirectly suggesting that sensing may be more tightly linked with eye movement behavior. Lastly, if prior unconscious change detection effects (e.g., participants viewing fewer regions in changed scenes; Ryan & Cohen, 2004) were truly due to change detection outside of awareness, then we would expect to observe change-related eye movement effects even when participants are highly confident there is no change – that is, eye movements that differ between changed and unchanged scenes even when participants confidently report that there is no change.
Method
Participants
Thirty undergraduates from the University of California, Davis completed the experiment for course credit. A sample size of 26 participants was needed to provide 95% power to detect the smallest analogous effect in a prior eye tracking study (i.e., the effect of recollection on first saccade accuracy; Ramey et al., 2019), using dz=.75 in a two-tailed paired-samples t-test. The quality of each participant’s eye tracking data was assessed by computing the mean percent signal across all trials to determine whether there was excessive track loss due to blinks or calibration loss, with a preselected criterion of 75% signal (Henderson & Hayes, 2017). All participants had greater than the signal criterion (M = 96.6%; range 88–99%), and thus all were retained for analysis.
Materials
Stimuli were 160 photographs of buildings. All scenes were presented in color at 1024 × 768 pixels subtending a visual angle of approximately 25° × 19° at presentation. For each scene, minor distortions were introduced using the “pinch” and “spherize” filters, at 5% strength, in Photoshop CS5. One pinched and one spherized version of each scene was created. To ensure that any effects were not due to the scenes appearing distorted, all scenes presented to participants were either pinched or spherized; that is, all scenes that the participants saw were manipulated in Photoshop in one way or another.
Apparatus
Participants’ eye movements were recorded using an SR Research EyeLink 1000+ tower mount eye tracker, sampling at 1000 hz. Stimuli were displayed on a monitor 85 cm from the subject’s eyes.
Procedure
Participants completed a change detection task consisting of 160 trials. Each trial began with a central fixation cross, followed by a study scene presentation for 2 s, a 600-ms blank gray screen with central fixation cross, a test scene presentation for 2 s, and a confidence scale that remained on the screen until the subject made a response (Fig. 1A). In half of the trials, the scene changed between study and test (i.e., the changed condition): the scene either changed from a pinched to spherized version of the same scene, or vice versa. In the other half of trials, the scene was unchanged between study and test (i.e., the identical condition). Half of the identical scenes were pinched and half were spherized. The scenes were counterbalanced such that each scene appeared in the changed condition for half the participants, and the identical condition for the other half.
For the confidence response, participants were asked to respond on a scale to indicate whether they thought the scene changed between study and test. Response options fell on a 1–5 plus “perceive” scale consisting of “I’m sure they’re the same,” “Maybe they’re the same,” “I don’t know,” “Maybe they’re different,” “I’m sure they’re different,” and “I perceive a difference” (i.e., a perceive response). A perceive response indicated that a participant not only was sure that there was a change, but they could also point to a specific region of the scene that changed. A response of “I’m sure they’re different” indicated high confidence comparable to that of a perceive response, but without the ability to point to a specific change. Participants were explicitly instructed that both “I’m sure they’re different” and perceive responses indicated equally high confidence, and differed only in the ability to specifically identify a changed region. In a previous study (Aly & Yonelinas, 2012), these self-report measures of perceiving were validated by requiring subjects to also verbally identify the specific feature that had changed. Subjective reports of perceiving, but not sensing, were directly related to objective measures of change identity.
Data reduction and analysis
Resampling
We computed resampling as the extent to which participants revisited the same regions between the study and test presentations of a scene (see Fig. 1E–G; for specific method, see Ramey, Henderson, et al., 2020a). We correlated the spatial locations of the fixations made during the study phase presentation of the scene with the spatial locations of the fixations made during the test phase presentation of the scene. To do this, we applied a Gaussian kernel with a cutoff frequency of -6 dB (to account for the fall-off in visual acuity from the fovea) to the matrix of fixation locations for each scene for each subject. This yielded two smoothed heatmaps of fixation locations for each trial: one at study and one at test (Fig. 1E and F). The two maps were then correlated to yield a single resampling value for each trial, with higher correlations indicating more resampling for that trial.
Change magnitude of resampled regions
To determine how change detection was related to resampling scene regions that were more highly changed as a result of the pinching and spherizing manipulations, we created a change map for each scene. To do this, we used MATLAB to compare the two versions of each scene pixel by pixel, in terms of the RGB values of each pixel. That is, a pixel was considered highly changed if the pixel at that coordinate in the other version of the scene was very different in terms of its RGB value. For example, if the pixel at a given (x,y) coordinate was green in both versions of a scene, it would have a very low change value; but if it was green in the pinched scene and purple in the spherized version, it would have a high change value. Note that this change measure not only accounted for differences in color, but in brightness as well. For each scene, this yielded a heatmap of the degree to which pixels were changed between the two versions of the scene, such that higher values indicate more change (Fig. 1B–D). We applied a Gaussian kernel with a cutoff frequency of -6 dB to the change maps, again to account for the fall-off in visual acuity from the fovea. The change map for a given scene was used to compute the average change magnitude (i.e., intensity of the change as computed by MATLAB’s imfuse function using the “diff” method) within the regions that were resampled between study and test. Specifically, the average change magnitude was computed for the regions that were overlapping between the study and test fixation heatmaps isolated in the trial-level resampling measure described above (Fig. 1G).
Dispersion
To quantify attentional dispersion, we calculated the number of regions that were fixated in a scene, as well as the distance between those regions. Specifically, we created a clustering algorithm (using silhouette with k-means) to group fixations into the mathematically optimal number of clusters based on their spatial relation to each other (for specific method, see Ramey, Henderson, et al., 2020a). To compute dispersion scores, we multiplied the number of clusters by the average distance between the medoids of those clusters (i.e., the most representative fixation in the cluster), to capture the extent to which fixations were distributed across the display. Higher values indicate that eye movements were more distributed across a scene. A prior study found that participants visited fewer regions when a scene was changed (Ryan & Cohen, 2004), and the present measure of dispersion takes the number of regions into account as well as how spatially distributed those regions are.
Statistical models
Statistical analyses were conducted using linear mixed effects models, which allowed us to harness trial-by-trial (i.e., within-subjects) data while controlling for individual differences and stimulus effects (for equations, see Appendix). To do this, we used crossed random intercepts of subject and image for all analyses. This allowed us to control for subject-related differences (e.g., some subjects may be more likely to perceive changes than others) and image-related differences (e.g., some images may be inherently more likely to elicit dispersed eye movements). The dependent variable in each model was the eye movement measure of interest. The fixed factors were response and trial type (i.e., changed vs. identical); see the Results section for the specific factors used in each analysis. The models were estimated using the lmerTest package in R (Kuznetsova et al., 2017). The model summaries provided degrees of freedom and t values, which are reported in the Results. The degrees of freedom were computed using the Satterthwaite approximation, and were rounded to the nearest integer in the Results. Effect sizes were calculated as classical Cohen’s d, as 2t/√df (Rosenthal & Rosnow, 1991).
Results
The behavioral response counts for the changed and identical scenes (Table 1) were similar to previous reports (Aly & Yonelinas, 2012) and showed that subjects were more likely to identify changed scenes as “different” and identical scenes as “same.”
Resampling of changed regions
To determine whether perceiving, sensing, and unconscious change detection were related to different patterns of eye movements, we first examined the extent to which subjects resampled regions between study and test that were highly changed as a result of the global manipulation in changed scenes. To do this, we examined whether the combination of increased resampling and increased viewing of changed regions differentiated between perceiving, confidence-matched sensing (i.e., “sure different” responses), and “sure same” responses. Note that for illustrative purposes we dichotomized change magnitude in Fig. 2A, but all continuous variables were treated as continuous in analysis.
Figure 2A illustrates that when participants perceived a change, they were more likely to resample more highly changed scene regions, suggesting that revisiting a changed region between study and test may be important for perceiving-based change detection. In contrast, when subjects accurately sensed a change, they were equally likely to resample high- and low-change regions in the scene. This difference between sensing and perceiving was reflected in a significant interaction between response type (i.e., sensing vs. perceiving) and change magnitude in predicting resampling, t(903) = 2.28, p = .023, d = .15 (Eq. A1). This also held when controlling for the number of fixations made at both study and test (included as two separate variables), p = .025. The results were also similar when the probability of making a perceive over a high-confidence sensing response was predicted from the eye movements using a mixed-effects logistic regression model, B = .20, p = .014 (Eq. A2, Appendix), which confirms that the specific combination of high resampling and high viewing of changed regions uniquely predicted perceiving. There were no main effects of resampling nor change magnitude on their own, ps > .12, indicating that there were no overall differences in the amount of resampling nor change magnitude between the perceiving and sensing responses.
Figure 2A also shows that when changes were not consciously detected (i.e., “sure same” trials), performance was similar to the sensing responses in that subjects were equally likely to resample high- and low-change regions. Supporting these observations, there was no significant interaction between change magnitude and response type (i.e., unconscious vs. sensing) in predicting resampling, p = .80. The results held when logistic regression was used with response type as the outcome, and there were no main effects of resampling nor change magnitude, ps > .18. Thus, sensing and unconscious change detection were not related to increased resampling of changed regions.
As an example of the resampling behavior that was observed for perceiving and sensing responses, Fig. 2C shows an individual resampling map for a perceiving response and a sensing response overlaid on the change map of that scene. These examples illustrate that for perceiving trials, resampling occurred more often at regions of high change (lighter regions) than at regions of low change, whereas for sensing trials, resampling occurred equally at regions of high and low change.
To determine how early in the test trial the preferential resampling of high-change regions occurred, we computed the change magnitude at each fixation location for trials with different levels of resampling. Figure 2B shows that for perceiving trials, change magnitude was greater for high than for low resampling trials for fixations that occurred between approximately 250 and 1,200 ms after stimulus onset, whereas no such effect was apparent for sensing responses. To identify the onset of the perceiving effect, we examined the relationship between resampling and change magnitude in 50-ms windows. We found that resampling was correlated with change magnitude in perceiving trials beginning in the 250- to 300-ms time window, r = .27, p < .0001, indicating that the tendency for perceiving trials to have preferential resampling of high-change regions occurred within the first one or two fixations of the test trial (the average fixation duration in the current study was 254 ms).
Dispersion
To determine whether perceiving, sensing, and unconscious change detection were related to different spatial distributions of eye movements, we examined the dispersion of eye movements during study and test for both changed and identical scenes. During study, there were no significant differences in dispersion between perceiving and high-confidence sensing trials, high-confidence sensing and “sure same” trials, changed and identical trials, nor an interaction between these factors, indicating that dispersion during presentation of the first image in the trial did not significantly impact performance, ps > .10.
However, during the presentation of the test image, eye movements did differ across the different change detection conditions. As illustrated in Fig. 3, eye movements were less dispersed for changed compared to identical scenes for the sensing and unconscious trials, and this effect was particularly apparent for the unconscious trials (i.e., when participants were sure there had been no change). A comparison of perceiving and high-confidence sensing indicates that eye movements were generally less dispersed for perceiving than sensing responses, t(1144) = -3.64, p < .001, d = -.22, but there was no difference in dispersion between changed and identical trials overall, nor an interaction between trial type (changed vs. identical) and response (perceive vs. sensing; Eq. A3, Appendix), ps > .20. This indicates that dispersion differences between perceiving and sensing were related to confidence but not veridical change detection. In contrast, in a model examining dispersion in high-confidence sensing trials and unconscious (“sure same”) trials, we found that dispersion was significantly lower for the changed than identical trials, t(269) = -4.59, p < .001, d = -.56. In addition, dispersion was higher in unconscious than sensing trials, t(1486) = -7.71, p < .0001, d = -.40. Although the difference in dispersion between changed and identical trials was larger for the unconscious than the sensing trials, the interaction between trial type and response type was not significant, p = .12 (Eq. A3, Appendix).
To further examine this difference in dispersion between changed and identical trials, we examined each of the six change detection confidence responses individually (i.e., the 1, 2, 3, 4, 5, and perceive response conditions). Interestingly, it was only for the “sure same” responses (i.e., “1”) that dispersion was significantly lower for changed than identical trials, t(272) = -2.95, p = .004, d = -.36 (other response ps > .31). However, when all response types were examined together (i.e., all of the data was used), dispersion was significantly lower for changed than identical trials overall, t(316) = -5.59, p < .0001, d = -.63. This suggests that the effect of trial type (i.e., changed vs. identical) on dispersion was strongest for the scenes that participants were sure had not changed – such that they had no conscious awareness of a change.
Discussion
In the present study, we examined how eye movements during change detection in scenes are related to perceiving, sensing, and unconscious detection of changes. We found that in changed scenes, when participants revisited the same highly changed scene regions before and after the change, they were more likely to perceive rather than sense a change. This effect emerged as early as 250 ms after the onset of the test phase scene, such that it began within the first one or two fixations. Moreover, we also found evidence for unconscious change detection in the form of less dispersed viewing for changed than unchanged scenes, even when participants indicated that they were sure there was no change.
The presently identified eye movement correlates of change detection processes provide a window through which we can begin to understand how these processes relate to attention. These results indicate that visiting the same, highly changed scene regions before and after the change uniquely predicts perceiving-based change detection. This effect may in part account for findings that perceiving-based responding is increased when changes are made to specific, local scene features compared to subtle global distortions of scenes (Aly & Yonelinas, 2012), such that local attention may be critical for perceiving. The finding that the perceiving-related increase in resampling of changed regions began as early as 250 ms indicates that the effect begins early in viewing. In fact, this early, relatively temporally constrained effect may be a possible mechanism underlying prior findings that perceiving tends to happen rather abruptly in a trial, whereas sensing involves a more gradual increase in the strength of global mismatch (Aly & Yonelinas, 2012). That is, perhaps change detection confidence remains low until a highly changed region is resampled (around 250 ms in the present study), at which point the participant suddenly perceives a change.
These results further indicate that successful sensing-based change detection is not as strongly related to precise resampling of highly changed regions across the two images. This result is consistent with earlier work suggesting that sensing is more gist-based, and involves a gradual build-up of a global scene representation for making judgments of global match rather than identifying specific changed regions (Aly et al., 2013). It is also consistent with the finding that sensing-based responding is increased when stimulus changes involve subtle distortions rather than specific local changes (Aly & Yonelinas, 2012). However, we did find that dispersion of viewing decreased as sensing confidence increased for both changed and unchanged scenes, which suggests that stronger sensing may relate to less active search irrespective of whether a change occurred. Thus, the present findings of differences in overt attentional patterns suggest that sensing and perceiving may be underpinned by distinct underlying attentional mechanisms.
The present study is the first to our knowledge to examine the possibility of unconscious change detection using confidence-based measurements. The fact that participants exhibited less dispersed patterns of viewing in changed than identical scenes, even when they were highly confident that there was no change, is generally in agreement with findings of change-related differences in global viewing patterns in the absence of awareness (Ryan & Cohen, 2004). Interestingly, when responses were examined separately, the difference in dispersion between changed and identical trials only remained significant when participants were sure there was no change; similarly, Ryan et al. (2000) reported that preferential viewing occurred primarily when participants reported no change. This suggests that unconscious change detection may be expressed primarily when conscious change detection is absent, in contrast to analogous work in long-term memory in which unconscious and conscious memory effects on eye movements are present simultaneously and appear to function in parallel (Ramey et al., 2019). Nonetheless, even for the sensing responses, changed scenes were related to numerically less dispersed viewing than unchanged scenes, and this difference in dispersion was not significantly greater in unconscious than sensing responses. Thus, we cannot rule out the possibility that both unconscious and sensing-based veridical change detection may be related to dispersion.
The present study extends a recently developed method for isolating the different processes underlying recognition memory, namely recollection, familiarity, and unconscious memory (Ramey et al., 2019; Ramey, Henderson, et al., 2020a). The fact that the analogous processes underlying change detection (i.e., perceiving, sensing, and unconscious change detection) were also able to be isolated in this way, such that they dissociated along the lines of eye movement behavior, highlights the versatility of this trial-level confidence-based scale. Other studies using this method have found that memory processes isolated in this way also (1) relate to different patterns of eye movements – including dispersion and resampling – during search, encoding, and retrieval (Ramey et al., 2019; Ramey, Henderson, et al., 2020a), (2) show differential interactions with schema knowledge (Ramey et al., 2022; Ramey, Yonelinas, et al., 2020b), and (3) predict real-world outcomes such as creative ability (Ramey & Zabelina, 2021). Thus, the present findings add to a growing body of work showing that this recently developed method can be fruitfully applied to isolate dissociable cognitive processes in a variety of settings.
The current results show that change detection processes differentially relate to the deployment of attention via eye movements. It should be noted, however, that because the present results are correlational, it is not yet established whether eye movements causally influence how changes will be detected, or whether an early experience of change detection could instead influence subsequent eye movements. For example, it could be that when participants happen to resample a changed region in a pair of scenes, this facilitates successful comparison of the two scenes and causes participants to perceive that change (i.e., they become consciously aware of the change and can report on it). Thus, different eye movements could be causing different types of change detection. Alternatively, however, it could be that different types of change detection cause different types of eye movements. For example, participants may begin to perceive a change in their peripheral vision, and that could drive their attention to resample that changed region to verify whether it changed. However, given prior findings that fixating changes causally produces change detection (Henderson & Hollingworth, 2003; Hollingworth et al., 2001; Hollingworth & Henderson, 2002), it seems likely that the present effects are at least in part driven by eye movements leading to perceiving and sensing. It is also possible that the effects are bidirectional, such that both of the above directions of causality are true; this would be consistent with findings of bidirectional resampling effects in long-term memory (Foulsham & Kingstone, 2013; Holm & Mantyla, 2007). Future work that causally manipulates eye movements with respect to perceiving and sensing will be critical in addressing this question.
Finally, there are limitations to the generalizability of the present results that should be noted. First, our stimuli consisted of global changes rather than local changes, so it is not clear whether these same patterns of eye movements would relate to change detection processes in the same way for local changes. Second, it is possible that the perceiving-specific viewing pattern of resampling changed regions was driven by differences in overall change detection strength rather than the distinction between perceiving and sensing per se. However, this possibility seems unlikely given that we controlled for strength, and given prior work finding both behavioral and neural dissociations between perceiving and sensing that are not simply driven by strength (Aly et al., 2013; Aly, Ranganath, & Yonelinas, 2014a; Aly, Wansard, et al., 2014b; Aly & Yonelinas, 2012). Third, while the present results indicate that overt attentional deployment patterns dissociate between perceiving and sensing-based change detection, the question remains of whether there may also be distinct covert or underlying attentional mechanisms that relate to perceiving and sensing.
Paralleling the growing body of literature showing that eye movements can uniquely index different processes underlying recognition memory (Holm & Mantyla, 2007; Kafkas & Montaldi, 2011; Ramey et al., 2019; Ramey, Henderson, et al., 2020a; Schwedes & Wentura, 2019), the present findings indicate that processes underlying perceptual judgments, too, exhibit unique relationships with patterns of attentional deployment. Not only do these findings open up a variety of avenues for future research – which may allow for manipulations that alter the tendency to perceive changes – but they also highlight the utility of eye movements for tapping into different cognitive processes.
Data Availability
The data are available here: https://osf.io/q9gd4/?view_only=dc32c729b14b401f9158eddd1fea5b64.
References
Aly, M., & Yonelinas, A. P. (2012). Bridging consciousness and cognition in memory and perception: Evidence for both state and strength processes. PLoS ONE, 7(1), e30231.
Aly, M., Ranganath, C., & Yonelinas, A. P. (2013). Detecting Changes in Scenes: The Hippocampus Is Critical for Strength-Based Perception. Neuron, 78(6), 1127–1137.
Aly, M., Ranganath, C., & Yonelinas, A. P. (2014a). Neural Correlates of State- and Strength-based Perception. Journal of Cognitive Neuroscience, 26(4), 792–809.
Aly, M., Wansard, M., Segovia, F., Yonelinas, A. P., & Bastin, C. (2014b). Cortical and subcortical contributions to state- and strength-based perceptual judgments. Neuropsychologia, 64, 145–156.
Foulsham, T., & Kingstone, A. (2013). Fixation-dependent memory for natural scenes: an experimental test of scanpath theory. Journal of Experimental Psychology: General, 142(1), 41.
Henderson, J. M., & Hollingworth, A. (1999). The role of fixation position in detecting scene changes. Psychological Science, 10(5), 438–443.
Henderson, J. M., & Choi, W. (2015). Neural correlates of fixation duration during real-world scene viewing: Evidence from fixation-related (fire) fMRI. Journal of Cognitive Neuroscience, 27(6), 1137–1145.
Henderson, J. M., & Hayes, T. R. (2017). Meaning-based guidance of attention in scenes as revealed by meaning maps. Nature Human Behaviour, 1(10), 743–747.
Henderson, J. M., & Hollingworth, A. (2003). Eye movements and visual memory: Detecting changes to saccade targets in scenes. Perception and Psychophysics, 65(1), 58–71.
Hollingworth, A., & Henderson, J. M. (2002). Accurate visual memory for previously attended objects in natural scenes. Journal of Experimental Psychology: Human Perception and Performance, 28, 113–136.
Hollingworth, A., Schrock, G., & Henderson, J. M. (2001). Change detection in the flicker paradigm: The role of fixation position within the scene. Memory & Cognition, 29(2), 296–304.
Holm, L., & Mantyla, T. (2007). Memory for scenes: Refixations reflect retrieval. Memory and Cognition, 35(7), 1664–1674.
Kafkas, A., & Montaldi, D. (2011). Recognition memory strength is predicted by pupillary responses at encoding while fixation patterns distinguish recollection from familiarity. The Quarterly Journal of Experimental Psychology, 64(10), 1971–1989.
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software, 82(13), 1–19.
Liu, Z. X., Shen, K., Olsen, R. K., & Ryan, J. D. (2017). Visual sampling predicts hippocampal activity. Journal of Neuroscience, 37(3), 599–609.
Liu, Z. X., Rosenbaum, S., Ryan, J. D., Rosenbaum, R. S., & Ryan, J. D. (2020). Restricting visual exploration directly impedes neural activity, functional connectivity, and memory. Cerebral Cortex Communications, 1(1), 1–42.
Ramey, M. M., & Zabelina, D. (2021). Creative people use memory differently: Divergent thinking modulates how episodic memory is combined with semantic knowledge. Psyarxiv. https://doi.org/10.31234/OSF.IO/FGKWQ
Ramey, M. M., Henderson, J. M., & Yonelinas, A. P. (2020a). The Spatial Distribution of Attention Predicts Familiarity Strength During Encoding and Retrieval. Journal of Experimental Psychology: General, 149(11), 2046–2062.
Ramey, M. M., Henderson, J. M., & Yonelinas, A. P. (2022). Episodic memory processes modulate how schema knowledge is used in spatial memory decisions. Cognition, 225, 105111.
Ramey, M. M., Yonelinas, A. P., & Henderson, J. M. (2019). Conscious and unconscious memory differentially impact attention: Eye movements, visual search, and recognition processes. Cognition, 185, 71–82.
Ramey, M. M., Yonelinas, A. P., & Henderson, J. M. (2020b). Why do we retrace our visual steps? Semantic and episodic memory in gaze reinstatement. Learning and Memory, 27(7), 275–283.
Rensink, R. A. (2000). Seeing, sensing, and scrutinizing. Vision Research, 40(10–12), 1469–1487.
Rosenthal, R., & Rosnow, R. L. (1991). Essentials of Behavioral Research: Methods and Data Analysis. In McGraw-Hill Series in Psychology.
Ryan, J. D., Althoff, R. R., Whitlow, S., & Cohen, N. J. (2000). Amnesia is a deficit in relational memory. Psychological Science, 11(6), 454–461.
Ryan, J. D., & Cohen, N. J. (2004). The nature of change detection and online representations of scenes. Journal of Experimental Psychology. Human Perception and Performance, 30(5), 988–1015.
Schwedes, C., & Wentura, D. (2019). The relevance of the first two eye fixations for recognition memory processes. Memory, 27(6), 792–806.
Smith, C. N., Hopkins, R. O., & Squire, L. R. (2006). Experience-dependent eye movements, awareness, and hippocampus-dependent memory. Journal of Neuroscience, 26(44), 11304–11312.
Smith, C. N., & Squire, L. R. (2008). Experience-Dependent Eye Movements Reflect Hippocampus-Dependent (Aware) Memory. Journal of Neuroscience, 28(48), 12825–12833.
Acknowledgements
We would like to thank Robin Goodrich for her contribution to the conceptualization and design of the experiment.
Funding
This work was supported by the National Eye Institute of the National Institutes of Health under Award Numbers R01EY025999 to Andrew Yonelinas and R01EY027792 to John Henderson. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author information
Authors and Affiliations
Contributions
Conceptualization, methodology, writing – Review and editing: J.M.H., A.P.Y., M.M.R. Funding acquisition: A.P.Y., J.M.H. Formal analysis, investigation, writing – original draft, visualization: M.M.R.
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no financial or proprietary interests in any material discussed in this article.
Ethics approval and consent
The methodology for this study was approved by the institutional review board of the University of California, Davis. Informed consent was obtained from all participants included in the study.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Model equations
The model specifications for the main analyses are reported below. Linear mixed-effects models with random intercepts of subject and image were used; random effects are italicized. Asterisks denote interactions. Main effects were also examined and reported individually.
Eq. A1: Resampling highly changed regions (perceiving vs. sensing)
Eq. A2: Alternate formulation of resampling highly changed regions using a logistic GLME model (perceiving vs. sensing)
Eq. A3: Dispersion
Rights and permissions
About this article
Cite this article
Ramey, M.M., Henderson, J.M. & Yonelinas, A.P. Eye movements dissociate between perceiving, sensing, and unconscious change detection in scenes. Psychon Bull Rev 29, 2122–2132 (2022). https://doi.org/10.3758/s13423-022-02122-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13423-022-02122-z