Tuning perception: Visual working memory biases the quality of visual awareness
Studies of consciousness reveal that it is possible to manipulate subjective awareness of a visual stimulus. For example, items held in visual working memory (VWM) that match target features increase the speed with which the target reaches visual awareness. To examine the effect of VWM on perception, previous studies have mainly used coarse measures of awareness, such as present/absent or forced-choice judgments. These methods can reveal whether or not an individual has seen an item, but they do not provide information about the quality with which the item was seen. Using continuous report methods it has been shown that the fidelity of a perceived item can be affected by whether or not that item is masked. In the present study, we used an object-substitution masking task to examine whether items held in VWM would influence the quality with which a masked target reached awareness, or whether the threshold for awareness was instead affected by stimuli held in memory. We observed that targets matching the contents of VWM were recalled with greater precision compared to items that did not match the contents of VWM. Importantly, this effect occurred without affecting the likelihood of the target being perceived. These results suggest that VWM plays a greater role in modulating the fidelity of perceived representations than in lowering the overall threshold of awareness.
KeywordsVisual working memory Visual perception Consciousness Object-substitution masking
Despite our experience of a rich and complete world, the amount of information that becomes consciously accessible to us is only a fraction of the information available to us at any given moment (Marois & Ivanoff, 2005). What information reaches the level of consciousness has therefore been a topic of much inquiry, particularly in the area of visual awareness (Crick & Koch, 1990; Lamme et al., 2000; Milner, 1995), as it has been demonstrated that awareness of a visual stimulus can be manipulated through various experimental paradigms, such as binocular rivalry, continuous flash suppression (CFS), and visual masking, among others (see Kim & Blake, 2005, for a review).
Previous studies of visual awareness have tended to use coarse measures of awareness, such as forced-choice judgments of shape or pattern (e.g., Olivers, Meijer, & Theeuwes, 2006; Pan, Cheng, & Luo, 2012). That is, most studies of visual perception have tended to focus on the question of sensitivity. It is possible, however, that items that reach the threshold for awareness may also vary in their fidelity or perceptual quality. For example, it has been shown that information stored in visual working memory (VWM) varies dramatically in its fidelity, in addition to (Zhang & Luck, 2008) or instead of (Bays & Husain, 2008) varying only in the likelihood that an item is encoded or maintained in memory.
To address the question of perceptual fidelity, Harrison, Rajsic, & Wilson (2015) examined how masking influences the fidelity of perceptual representations. They used object-substitution masking (OSM) (Enns & Di Lollo, 1997), in which the visibility of a target item is impaired by a sparse four-dot mask that has a delayed offset from the target (for a review, see Goodhew et al., 2013). Harrison and colleagues (2015) found that the precision of a visual representation was affected by the presence and timing of a mask. This suggests that, contrary to our naïve experience, our conscious experience of visual objects is not all-or-none, but rather comes with varying degrees of perceptual fidelity.
If visual awareness comes with varying degrees of fidelity, then it is possible that factors that alter the threshold of awareness could also affect the fidelity of those representations. For example, it has been shown that the contents of visual awareness can be biased towards task-relevant information. Gaillard and colleagues (2006) manipulated this threshold of awareness by consciously priming subsequently presented masked words, significantly increasing the rate of target detection. It has also been demonstrated that the threshold of awareness can be altered via the contents of visual working memory. Using CFS, Gayet, Paffen, and Van der Stigchel (2013) found that VWM contents matching the color of a suppressed stimulus allowed the memory-matching stimulus to reach awareness faster, indicating that visual awareness is prioritized for information matching the contents of VWM.
The results of Gayet et al. (2013) suggest that the contents of VWM could have two different (and potentially complementary) effects on subsequently perceived stimuli. On the one hand, the threshold for perceiving items that match the contents of VWM might be lowered, leading to increased sensitivity; alternatively, the perceived representation might be biased towards the contents of VWM, resulting in an increase in the quality or fidelity of perception. Consequently, in the current study, we examined to what extent information stored in VWM could bias whether a stimulus reached the threshold of conscious perception as opposed (or in addition) to whether it affected the fidelity of the visual representation. To examine this question, we used a combined VWM-OSM paradigm, and a mixture-model analysis of a continuous-report task.
A total of 35 participants completed the study and 29 participants were included in the final analysis [13 male, mean age (M age) = 20 years]. Sample size was determined a priori based on the assumption of a relatively small effect size compared to other studies using similar analytical approaches (Emrich & Ferber, 2012). The rejection criteria are defined below in the Data analysis section.
All participants self-reported normal or corrected-to-normal vision. Informed consent was provided before participation, as approved by the Research Ethics Board of Brock University. Participants received course credit or monetary remuneration for their participation.
The task was completed on a 41-inch NEC MultiSync LCD 2090UXi computer monitor (1024 × 768 pixels, 60 Hz refresh rate) in a private testing room. All stimuli were presented using Psychopy version 2 (Peirce, 2009). Viewing distance was set to approximately 57 cm.
Stimuli and procedure
Next, a search screen of six Landolt-Cs (radius of 0.95°) was presented for 30 ms. The stimuli were presented equally spaced around an imaginary circle with a radius of 5° from the fixation point. One of the Landolt-Cs was pseudo-randomly chosen to be surrounded by a four-dot mask (2.5° × 2.5°), which disappeared 80 ms following the stimuli offset (delayed mask offset). In the matching condition, the orientation of the target Landolt-C matched the orientation of the cued Landolt-C. In the non-matching condition, the orientation of the target Landolt-C matched the orientation of the non-cued Landolt-C.
Next, participants were probed to either report the orientation of the Landolt-C presented during the memory task, or the orientation of the target Landolt-C (perceptual task). Each of the conditions occurred on one-half of the trials and were intermixed pseudo-randomly. The VWM task was cued by a Landolt-C of the same size and central position of the larger to-be-memorized Landolt-C. This task ensured that participants were successfully holding the item in VWM on every trial. Participants responded to the perceptual task when they were presented with a Landolt-C the same size and non-central position of the target Landolt-C. Both of the response Landolt-C orientations were jittered on the response screen. To make their response, participants used the mouse to move the orientation of the Landolt-C on the screen to the orientation of either the to-be-memorized or target Landolt-C. Responses were recorded in degrees with the mouse click. Positive feedback was provided on-screen when participants were within 20° of the correct orientation (with the correct Landolt-C depending on the task).
Participants were given eight practice trials before beginning the experiment, with additional practice if necessary. All participants completed a total of 500 trials. These trials were divided evenly by task (VWM or OSM), and whether or not the item held in VWM matched the OSM target.
Analyses were completed in MATLAB version R2014a using the three-component swap model as developed by Bays et al. (2009) (http://www.paulbays.com/code/JV10/) using the MemToolBox 1.0 (Suchow, Brady, Fougnie, & Alvarez, 2013).
Response error was calculated on each trial as the difference between the target’s actual orientation and the participant’s response in degrees. Response error was then broken down into a mixture of three types of responses according to the methods of Bays et al. (2009): target responses, non-target responses, and guess responses. Target responses occur when the target orientation is correctly reported, non-target responses occur when the orientation of a distractor (non-target) is reported, and guess responses occur when a random orientation response is made. Target and non-target responses were drawn from a circular Gaussian (von Mises) distribution, and guess responses were drawn from a uniform distribution. The standard deviation (SD) of the circular von Mises distribution for non-guess responses was used to estimate the precision of responses. Note that SD is inversely related to precision. A fourth parameter (target responses) was defined as 1 – guess response – non-target error responses, and reflects the probability of correctly reporting the target orientation.
For both the perceptual and memory tasks, a total of six non-targets were included in the mixture model, five of which were defined as the orientations of the distractors that were presented during the OSM display. One additional non-target was defined for each task according to the memory-matching condition. This was done to ensure that participants were reporting the orientation of the ‘correct’ Landolt-C for each task and therefore served as a check for task compliance. In the matching condition of the perceptual task and both the matching and non-matching conditions of the VWM task, the additional non-target was defined as the orientation of the non-cued memory item. In the non-matching condition of the perceptual task, the additional non-target was defined as the cued memory item (i.e., not the target of the perceptual task).
A high non-target error rate in the perceptual task would suggest that the participant was instead reporting the orientation of the VWM item. Similarly, a high non-target error rate in the VWM task would suggest that the participant was reporting the orientation of the perceptual target. Five participants’ data were excluded for non-target error rates greater than three SDs from the mean in any condition in either the perceptual or memory task. Data from one additional participant were excluded due to guess rates that were greater than three SDs from the mean in both conditions of the perceptual (OSM) task.
As the memory task was not the main focus of the study, and served only as a check for task compliance, the results from the memory task can be found in the supporting information available on-line.
In contrast to the effect of memory-matching condition on the measures of perceptual threshold, we observed a significant reduction in the SD of the reported target in the memory-matching condition (M = 22.98, SD = 5.35) relative to the non-matching condition (M = 25.49, SD = 8.18), t(28) = –1.914, p = .033, d = –.355. This effect is further supported by the calculation of the Bayes Factor, which indicated that a smaller SD in the matching as compared to the non-matching condition was 1.876 times more likely than the null. Examining individual subjects’ data, 19 out of 29 participants were more precise in the matching than in the non-matching condition (see Fig. 2b).
Given the relatively small evidence (BF < 3) for a change in SD, as well as the potential that our initial rejection criteria could not remove those subjects who might be employing a biased response strategy, we examined the possibility that outliers were driving the observed effects by performing a second rejection step on the difference scores between memory-matching conditions for each parameter estimate. Participants with scores greater than 1.5 times the interquartile range outside of the first or third quartile of scores were excluded (Tukey, 1977), resulting in the exclusion of four additional participants (N = 25). Once again, SD was significantly different between the matching (M = 22.02, SD = 4.67) and non-matching (M = 24.47, SD = 6.46) conditions, t(24) = –2.70, p = .006, d = –0.539, BF10 = 7.79, with no significant differences observed between conditions for guess rate, non-target errors, or target rate, p values > 0.26, d values < 0.13, BFs01 > 2.7. Thus, removing outliers from the analysis actually increased the effect size of the observed difference in SD, consistent with the interpretation that the contents of VWM increase the fidelity of subsequently perceived stimuli, without affecting the threshold of awareness.
Is it possible that participants were simply reporting the orientation of the item held in VWM on the perceptual task, leading to the appearance of increased precision on the memory-matching trials? To test this question, we included the item held in memory in the calculation of non-target errors for non-matching trials (see Data analysis section). Critically, there was no increase in the proportion of non-target errors in the non-matching condition compared to the matching condition, t(28) = –.381, p = .353, d = –.071, BF 01 = 3.693. Thus, participants did not have a response bias towards the item held in memory that would account for the observed increase in precision.
In the current study, we sought to answer whether the contents of VWM affected the fidelity of perceived items, in addition to or instead of affecting the threshold of perceptual awareness. The results provide evidence that the contents of VWM solely affect the fidelity of subsequent memory-matching representations, without having an effect on the threshold of awareness.
How might the contents of VWM affect the fidelity of subsequently perceived stimuli? These findings may be explained partially by overlapping neural substrates in which perceptual and VWM representations are maintained. It has been found that neural activity associated with VWM maintenance occurs in part through sensory recruitment (Postle, 2006), such that patterns of activation in sensory areas (i.e., V1–V5) store stimulus-specific information (Ester, Serences, & Awh, 2009; Harrison & Tong, 2009; Serences et al., 2009; but see, Christophel, Hebart, & Haynes, 2012; Ester, Sprague, & Serences, 2015). Moreover, representations are similar enough that patterns of activation from one task (e.g., VWM) can be used to train and test patterns from another task (e.g., perception; Harrison & Tong, 2009). In particular, the fidelity of memory representations (Emrich et al., 2013; Ester, Anderson, Serences, & Awh, 2013), but not the likelihood of memory recall (Emrich et al., 2013), seems to be coded by early visual cortex, suggesting that VWM and perception share overlapping neural substrates in early visual cortex, and that this activity is associated with the fidelity of representations, more so than the likelihood of memory encoding/retrieval. Consequently, early visual cortex may serve as a locus for perceptual and VWM representations to interact, resulting in neural responses to percepts that are biased by those coding for VWM contents.
In accordance with these findings, there is evidence that top-down expectations influence visual cortex, such that our earliest visual representations are biased by higher-order information (Kok et al., 2013). Using functional magnetic resonance imaging (fMRI), Kok, Failing, and Lange (2014) found that expectation of a visual stimulus evoked a pattern of activity in V1 that was similar to the activity evoked by the stimulus itself. Prior expectations have also been found to specifically affect the resolution of visual representations in V1, even though these expectations reduce the amplitude of neural responses in these areas (Kok, Jehee, & de Lange, 2012). The authors suggested that the prior expectation of a stimulus provides a greater signal-to-noise ratio (SNR), allowing for a sharpening of encoded representations. If so, this evidence corroborates our finding that VWM—a higher-level process similar to expectation—influences the fidelity of early visual representations, and that this effect may be mediated by an increase in the SNR of early visual activity. This may also explain why a change in perceptual fidelity may lead to a reduced RT for detecting a stimulus (Gayet et al., 2013) without affecting the threshold of awareness: an increase in SNR resulting from a change in perceptual fidelity may affect response decisions (Scolari & Serences, 2010), thereby speeding reaction times (Mazurek et al., 2003).
Our observed change in perceptual fidelity is in conflict with Asplund and colleagues (2014), wherein it was suggested that attention does not affect the precision of visual representations in an attentional blink task, only the threshold of awareness. Unlike the current study, Asplund and colleagues (2014) used a mixture model analysis that does not take into account the influence of non-target errors (Zhang & Luck, 2008). Bays et al. (2009) have demonstrated that a significant number of responses within the uniform (guess) component can be accounted for by these non-target responses, which also affects the SD of the target responses. Consequently, it is possible that accounting for non-target errors in the Asplund et al. (2014) study would reveal a change in response precision rather than a change in the likelihood of reporting the target. Alternatively, it is possible that there may be two independent stages of processing that affect the threshold and precision of perceptual representations, respectively; an item may first pass the threshold of awareness and then be affected by higher-level processes (such as prior expectation or VWM) that could tune the fidelity of the initial representation.
In sum, our findings suggest that the contents of VWM alter the fidelity of perceptual representations at the earliest stages of visual awareness. This finding has significant implications for theories of visual awareness. For example, a number of theories of awareness emphasize the role of feedforward and re-entrant processing (i.e., Lamme & Roelfsema, 2000), which have been previously implicated in OSM (Di Lollo, Enns, & Rensink, 2000). These theories posit that recurrent interactions between and within multiple levels of the brain are needed for consciousness (e.g., Lamme & Roelfsema, 2000). Although this explains the differences between conscious and unconscious processes, it does not elaborate upon the effects of re-entrant processes on the quality of conscious representations. Consequently, future studies should investigate the effect of VWM contents on feedforward and re-entrant activity, and how these processes affect the fidelity of visual representations. Indeed, our results further emphasize the importance of examining perceptual quality, in addition to measures of sensitivity, in future studies of visual awareness.
This work was supported by a Natural Sciences and Engineering Research Council of Canada Discovery Grant (#435945) awarded to S.M.E. All research was approved by Brock University’s Research Ethics Board and participants provided written consent before completing the experiment.
Compliance with ethical standards
All authors contributed to the study concept and to the study design. Testing and data collection were performed by C.M.S. C.M.S. performed the data analysis and interpretation under the supervision of S.M.E. All authors interpreted the data and wrote the manuscript. All authors approved the final version of the manuscript for submission.
- Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321(5890), 851–854.Google Scholar
- Emrich, S. M., Riggall, A. C., LaRocque, J. J., & Postle, B. R. (2013). Distributed patterns of activity in sensory cortex reflect the precision of multiple items maintained in visual short-term memory. The Journal of Neuroscience, 33(15), 6516–6523. doi: 10.1523/JNEUROSCI.5732-12.2013 CrossRefPubMedPubMedCentralGoogle Scholar
- Harrison, G. W., Rajsic, J., & Wilson, D. E. (2015). Object-substitution masking degrades the quality of conscious object representations. Psychonomic Bulletin & Review, 1–7. doi: 10.3758/s13423-015-0875-7
- JASP Team (2016). JASP (Version 0.7.5.5). [Computer software].Google Scholar
- Tukey, J. W. (1977). Exploratory data analysis. Reading, Mass.: Addison-Wesley Pub. Co., c1977.Google Scholar