Looking sharp: Becoming a search template boosts precision and stability in visual working memory
Visual working memory (VWM) plays a central role in visual cognition, and current work suggests that there is a special state in VWM for items that are the goal of visual searches. However, whether the quality of memory for target templates differs from memory for other items in VWM is currently unknown. In this study, we measured the precision and stability of memory for search templates and accessory items to determine whether search templates receive representational priority in VWM. Memory for search templates exhibited increased precision and probability of recall, whereas accessory items were remembered less often. Additionally, while memory for Templates showed benefits when instances of the Template appeared in search, this benefit was not consistently observed for Accessory items when they appeared in search. Our results show that becoming a search template can substantially affect the quality of a representation in VWM.
KeywordsVisual search Template Visual attention Working memory
The source of voluntary visual attention—our ability to control what information we will and will not process—has long been debated in psychology (Awh, Belopolsky, & Theeuwes, 2012). One of the main tools to investigate voluntary attention has been visual search, where one attempts to determine whether a particular object is present among an array of objects. To do so, one must maintain a “template,” that is, a mental representation of the object that one is looking for, to know whether the desired object is one of the many objects visible. Visual working memory (VWM), a limited capacity store that maintains visual information in the service of ongoing cognitive operations (see Luck, 2008 for a review), has been proposed to be the cognitive basis of templates used in visual search (Desimone & Duncan, 1995). This has typically been tested by measuring whether items merely stored in memory lead to attentional capture towards memory-matching objects that appear in the context of a search (Soto, Hodsoll, Rotshtein, & Humphreys, 2008; Olivers, Meijer, & Theeuwes, 2006; Olivers, 2009). Whereas many studies have found such memory-driven capture, despite the fact that memory-matching objects are task-irrelevant (but see Woodman & Luck, 2007), this type of attentional capture seems to occur only when templates used for the search task have been practiced for several trials or more (Woodman, Luck, & Schall, 2007; Olivers, 2009; Carlisle, Arita, Pardo, & Woodman, 2011). Indeed, memory-driven capture also tends not to occur when multiple objects are held in VWM (van Moorselaar, Theeuwes, & Olivers, 2014). These findings are consistent with the proposal that novel search targets occupy a special state in VWM (Olivers, Peters, Houtkamp, & Roelfsema, 2011), such that one item can serve as a template, which can interact with ongoing perceptual processing, but other items held as “accessory” items that cannot interact.
Whereas there is converging evidence that search templates have a special state in VWM (Houtkamp & Roelfsema, 2009; Carlisle et al., 2011; Greene, Kennedy, & Soto, 2015; van Moorselaar et al., 2014), relatively little is known about the properties of these representations. For example, Hollingworth and Hwang (2013) showed that items that do not guide search need not be lower precision but simply deprioritized (van Moorselaar et al., 2014), but this leaves open the question of whether search templates have higher memory precision than accessory items. Evidence from neural data show that search templates, compared with accessory items, are associated with a sustained increase in the activity of relevant visual areas and a selective, transient increase in activation of fronto-parietal and visual areas when the search template, but not when accessory items, appear (Peters, Roelfsema, & Goebel, 2012). Although such neural differences suggest that the representation of search templates in VWM may be qualitatively different from accessory items, a direct measurement of the quality of memory for templates and accessory items is lacking.
What is not lacking is repeated demonstrations that observers are able to prioritize particular representations in VWM (see Souza & Oberauer, 2016, for a recent review). Such studies have relied on the retro-cuing technique, wherein a set of objects are encoded into VWM, and only afterwards is one designated to be the object that will be tested more often than not. When retro-cued, objects can be remembered more often (Murray, Nobre, Clark, Cravo, & Stokes, 2013) and sometimes more precisely (Gunseli, van Moorselaar, Meeter, & Olivers, 2015) than other items. This line of research shows that substantial differences can exist between items held in VWM, supporting the possibility that search templates may be remembered better than accessory items.
It is important to note that with retro-cuing there is an obvious benefit to shifting memory to the cued item, as memory for the cued items is tested more often than memory for uncued items. Retro-cue benefits are larger when retro-cues more often predict the tested item (Gunseli et al., 2015), consistent with the notion that participants will increasingly bias internal attention to cued items as the payoff increases, assuming participants intend to minimize their performance errors. This is not to say that shifts of attention are completely strategic; Berryhill, Richmond, Shay, and Olson (2012) have shown that retro-cuing effects persist when retro-cues do not predict the tested item, albeit after participants had gained experience with retro-cues that were completely valid. Similarly, Li and Saiki (2014) showed that a cue that loses its predictive validity later in a trial nonetheless produces a retro-cue effect, albeit when mixed with trials in which this cue is helpful on half of the overall trials. Our experiments differ in that the cues used to indicate which item should be searched for were always at chance in terms of predicting the tested item, therefore any differences between cued and uncued items (or, in the present terminology, template and accessory items) cannot be due to participants’ intention to minimize error in memory reports, but presumably instead to the need to represent templates with greater fidelity.
The question addressed in the current study is straightforward: does assigning template status to a memory, holding constant the testing probability of template and accessory items, nonetheless affect the quality of memory of the template item akin to that observed in retro-cuing? Indeed, if our results do show an enhancement of memory quality for templates compared to accessory items analogous to studies using retro-cues (Gunseli et al., 2015; Murray et al., 2013), then this would provide converging evidence for the notion that search templates require internal attention (Olivers et al., 2011). Such a finding would be consistent with Souza, Rerko, and Oberauer (2015), who found that memory error is lower when specific items are cued to be “refreshed” during the retention interval, lending support to the notion that attention can be shifted in memory even when it produces no clear performance gains. Relatedly, van Moorselaar et al. (2014) showed in their final experiment that memory-driven capture selectively occurred for the one item (out of two) in memory that participants expected to be tested on first, even when both were ultimately tested. Indeed, memory performance was better for the first item tested than the second. Both studies suggest that memory resources can be unevenly distributed in VWM even when these altered distributions might not be expected to reduce memory error. The goal of the present study was to assess directly the quality of memory for search templates compared to accessory items, with the hypothesis that assigning “search template” status to one item would shift resources in VWM towards the template, improving the quality of its memory. To do so, we conducted two experiments in which one of two items encoded into VWM was designated a search template, and subsequently measured the quality of memory for this search template, or an accessory item in VWM.
In Experiment 1, we compared the memory for templates and accessory objects after a visual search to memory for identical objects when no search occurred to determine their relative memory quality. If objects serving as search templates indeed enter a special representational state in VWM, they should show superior memory to accessory objects.
Eighteen undergraduate students (5 males) participated in this experiment as partial fulfillment of course credit for a first-year psychology course. All participants reported normal vision. Three participants were excluded due to excessive incorrect search responses, leading to fewer than 50 trials in one or more cells of the factorial design. Given that we intended to model memory performance using the Bays’ three-component model (Bays, Catalao, & Husain, 2009), we excluded these participants to preclude the possibility of poor model-fitting from small number of trials. Our goal was to collect approximately 15 participants whose data could be included as it is in the typical range of the number of participants collected in experiments on memory-driven capture (Olivers, 2009).
Materials and Procedure
After the offset of the memory items, a 500-ms blank display preceded the search instructions display. The instruction display, which lasted 500 ms, consisted of a left-facing arrow, a right-facing arrow, or an X drawn at fixation. Participants were instructed that if an arrow appeared, then they were to search for the object that had appeared on that side of fixation just previously in the upcoming display. This allowed us to designate one object in memory as the search template, and one object as the accessory item. If an X appeared, participants were told that they did not have to respond to the upcoming display and that it would offset on its own. This allowed us to measure the baseline memory for two items in VWM when no search occurred but with identical stimulus conditions during the retention interval. Each cue type (left arrow, right arrow, or an X) was equiprobable.
Next, the search display, which consisted of six peripheral shapes evenly spaced along an imaginary circle around fixation, appeared. The shapes were drawn approximately 6° from fixation and were the same size as the shapes presented in the memory array (approximately 7° in diameter). These shapes were colored using the six nonsampled color values from the array of values used to select the memory colors, except on trials where one shape matched the memory shapes, in which case five nonsampled colors were used for the nonmemory-matching shapes. Participants were instructed to use the computer mouse to report whether the search target was present or absent on search trials. For nonsearch trials, the search display offset after a random amount of time (drawn from a log-normal distribution with mean 0 and SD 0.5, with the constraint that samples could not exceed 4,000 ms, producing a mean time of 1,120 ms and standard deviation [SD] of 560 ms). Actual search RTs (when excluding only trials with >4,000 ms RTs for direct comparison) had a mean of 1,040 ms and an average SD of 557 ms.
To assess how visual repetition affects memory for search templates and accessory items, for two thirds of trials one of the shapes in the search display matched one of the objects in VWM (Fig. 1). For half of these trials, this shape was the search template (i.e., the search was target present), and for the other half, the shape was the accessory item (i.e., the search was target absent). As such, stimulus repetition effects could be measured independently of memory status (either template or accessory).
After the offset of the search array, and a 500-ms delay, the cued recall memory test occurred. Memory error was measured by presenting a gray shape at fixation accompanied by a peripheral color wheel. The shape matched one of the two shapes presented at the beginning of the trial, and participants were instructed to report the associated color for the probed shape by clicking a color on the color wheel. Participants again used the computer mouse to select the color that they believed belonged with the presented shape. The next trial began following this response.
Results and Discussion
For all analyses, trials were excluded when search RT fell below 100 ms or two standard deviations above a participant’s search RT. Mean correct search response time (RT) was affected by target presence, F(2, 28) = 4.26, p = 0.024, η2 p = 0.23, with search response times being shorter when the target was present (M = 919 ms, SE = 56 ms) but not differing between target absent trials when the accessory object appeared (M = 991 ms, SE = 61 ms) and when neither memory object appeared (M = 993 ms, SE = 61 ms), demonstrating that participants indeed searched for the instructed template. Accuracy was high, M = 94.8%, SE = 1.0%, but a slight speed-accuracy trade-off occurred, F(2, 28) = 7.91, p = 0.002, η2 p = 0.36, such that 3% more errors occurred on target present trials than target absent trials, which we attribute to the low prevalence of targets (33%; Wolfe et al., 2007).
On the trials where one item was a search template, precision of the template was better following searches where a template-matching item appeared in search compared to when neither the template nor the accessory item appeared in search (i.e., tested-present [TP] vs. neither present [NP], for search templates in Fig. 2), t(14) = 8.21, p < 0.001, but was not affected when the accessory item appeared (i.e., nontested present [NTP[vs. NP), t(14) = 1.61, p = 0.13. Accessory precision, on the other hand, benefitted from both the appearance of the accessory item in search (i.e., TP vs. NP for accessory item memory), t(14) = 2.52, p = 0.024, and, arguably also, the appearance of the template (i.e., NTP vs. NP), t(14) = 1.89, p = 0.08.
One potential reason for differences in memory between repeated exposure conditions is differences in overall search time, given that target present searches were faster than target absent searches. To test this possibility, we performed a median split on search display times for no search trials, comparing subsequent memory precision. Despite a large difference in display times, M short = 689 ms, SE = 3.7 ms, M long = 1349 ms, SE = 7.2 ms, t(14) = 92.73, p < 0.001, memory did not differ, M short = 0.022, SE = 0.0025, M long = 0.026, SE = 0.0029, t(14) = 0.93, p = 0.37. As such, differences in retention duration are unlikely to account for the present findings.
To better understand the nature of the changes in memory quality caused by searching, we applied the three-component model (Bays et al., 2009) to our data, which expresses memory performance as a mixture of three types of responses: correct responses (i.e., responses drawn from a distribution centered around the probed object’s color, with an estimated SD), swap responses (i.e., responses drawn from a distribution centred around the nonprobed object’s color, with the same SD as correct responses), and guess responses (i.e., responses drawn from a uniform distribution, where every color-response is equally likely). Data in each cell of our design, for each participant, were fitted with the model, and the resulting parameter estimates were analysed. The SD of correct responses for baseline VWM in the three repeated exposure conditions (neither present, nontested present, tested present) was 28° (SE = 2.5°), 29° (SE = 2.1°), and 18° (SE = 1.6°), respectively, and estimated p(Correct) for baseline VWM in the three repeated exposure conditions was 0.59 (SE = 0.06), 0.68 (SE = 0.06), and 0.67 (SE = 0.06), respectively. SD of correct responses was determined by memory type, F(2, 28) = 6.19, p = .006, η2 p = 0.31, repeated exposure, F(2, 28) = 4.42, p = 0.022, η2 p = 0.24, and their interaction, F(4, 56) = 6.39, p < 0.001, η2 p = 0.31 (Fig. 2b). However, the probability of a correct response was only affected by memory type, F(2, 28) = 31.37, p < 0.001, η2 p = 0.69, with no interaction with repeated exposure, F(4, 56) = 0.98, p = 0.43, η2 p = 0.07, but a marginal main effect of repeated exposure, F(2, 28) = 3.02, p = 0.065, η2 p = 0.18 (Fig. 2c). Critically, the SD of correct responses was lower for templates even when no memory-matching object appeared in search, t(14) = 2.09, p = 0.055. Repeated-exposure had opposite effects for accessory items and search templates; templates had lower SD on target present trials compared with none-present, target-absent trials, t(14) = 6.35, p < 0.001, and accessory-item present, target-absent trials, t(14) = 9.00, p < 0.001. Accessory items, however, showed no SD reduction when the accessory item appeared in search (i.e., TP vs. baseline for Accessory items), t(14) = 0.01, p = 0.99, η2 p = 0.001. Instead, their SD was lower when the template appeared in search (i.e., NTP vs. baseline), t(14) = 2.68, p = 0.02.
An analysis of the probability of swap errors (Fig. 2d) revealed a main effect of memory type, F(1, 14) = 7.13, p = 0.003, η2 p = 0.34, with a large difference in the probability of a swap errors between template-tested trials and accessory-tested trials, F(1, 14) = 8.86, p = 0.01, η2 p = 0.39. This shows that true “swaps” were not occurring, and participants were likely reporting the only color they knew (the template color) when the accessory item was tested. Because this would imply that the accessory color was unavailable, instead of truly swapped with the template color, then these excess swap responses on accessory-tested trials are better considered as guesses—trials in which the accessory color was unknown. Thus, we conclude that, in our task, both swap and guess responses reflected a loss of information about the tested item. The estimated p(Swap) for baseline VWM in each repeated exposure condition was 0.19 (SE = 0.04), 0.15 (SE = 0.03), 0.09 (SE = 0.01), respectively.
Finally, we analysed search performance as a function of subsequent memory quality. To equate the number of trials, we performed a median split on mean squared memory error for both template and accessory memories, thus comparing search when memory was “good” to when it was “bad.” Search was faster overall when memories were recalled with less error, F(1, 14) = 14.83, p = 0.002, η2 p = 0.51, M good = 938 ms, SE good = 54 ms, M bad = 998 ms, SE bad = 60 ms. In addition, searches were faster when the template was tested, which reflects a larger contribution of the memory quality of templates to search speed; trials with good template memory showed faster search than trials with good accessory memory, t(14) = 3.66, p = 0.003, but no such difference occurred when memory was bad, t(14) = 0.38, p = 0.71. No interactions were found between memory quality and search conditions (template present, accessory present, neither present), Fs < 0.98, ps > 0.38, corroborating the conclusion that, when a template is held in VWM, accessory items do not interfere with search (Woodman, Carlisle, & Reinhart, 2013; Hollingworth & Hwang, 2013).
Experiment 1 showed that objects represented in VWM that become search templates are remembered both more often and with greater precision. However, it is not clear whether the change in memory states occurs in anticipation of search or during the search itself. To resolve this ambiguity, we ran a second experiment where participants were again told which of two remembered items needed to be search for but included trials where no search occurred. If changes in memory states occur when a representation is selected for use as a search template, then we should observe differences between templates and accessory items even when no search is performed.
Twenty-four undergraduate students, enrolled in a first-year Psychology course at the University of Toronto, were recruited as participants in Experiment 2. All provided informed consent before participating, and none had participated in Experiment 1.
Materials and Procedure
Results and Discussion
Five of the 24 participants were excluded from data analysis for having poor search accuracy (<80% correct). Once again, trials with overly fast (<100 ms) or overly slow (>2 SD of overall search RT) were excluded in all analyses. For the remaining participants, correct mean search time surprisingly did not differ between no target, accessory present, and template present trials, F(2, 36) = 0.49, p = 0.62, η2 p = 0.03, M template present = 737 ms, SE template present = 23 ms; M accessory present = 749 ms, SE accessory present = 22 ms; M neither present = 753 ms, SE neither present = 25 ms. Search accuracy, however, did differ, F(1, 36) = 16.90, p < 0.001, η2 p = 0.48, such that accuracy was lower on template present trials, M = 83%, SE = 1.9%, than on accessory present trials, M = 93%, SE = 1.4%, and neither present trials, 94%, SE = 1.4%. While unusual, this reduction in accuracy for target present trials may have occurred because of the low prevalence of targets in our experiment, as in Experiment 1.
To determine the nature of these memory errors, we again analysed memory parameter estimates given by the three-component model (Bays et al., 2009). Memory SD was again better for templates than accessory items, F(1, 18) = 5.89, p = 0.03, η2 p = 0.25, and better after no-search (NS) trials and TP (target present) trials for both template and accessory items, F(3, 54) = 3.47, p = 0.022, η2 p = 0.16. However, the benefit in memory SD of seeing the tested object in search was greater for the template than for the accessory item (i.e., template vs. accessory, for TP trials), t(18) = 2.33, p = 0.032. Critically, even when no search occurred, template memory SD was lower than accessory memory SD (i.e., template vs. accessory for NP trials), t(18) = 3.70, p = 0.002, showing that template precision is increased in anticipation of search.
Estimated p(Correct) also was better for templates than accessory items, F(1, 18) = 18.82, p < 0.001, η2 p = 0.51, and was affected by search, F(3, 54) = 5.95, p = 0.001, η2 p = 0.25. As shown in Fig. 4, performing a visual search was more deleterious to accessory items than templates. Whether this is due to the increased retention intervals associated with search trials or the visual and cognitive interference that they likely produced is not clear. However, even when no search occurred, templates were more often remembered than accessory items, t(18) = 2.68, p = 0.015. Overall, these results show that changes in the representational status of objects in VWM occur in anticipation of, and not only as a consequence of, visual search.
We again analysed correct search RT as a function of memory quality, as in Experiment 1. As in Experiment 1, searches were faster when memory quality was higher, F(1, 18) = 8.82, p = 0.008, η2 p = 0.33, M good = 736 ms, SE good = 22 ms, M bad = 757 ms, SE bad = 21 ms. No other differences were observed, Fs < 1.65, ps < 0.21.
In the present experiments, we measured the quality of memory for objects in VWM that were (templates) and were not (accessory items) used to guide search. Overall, we found that Templates were recalled with greater precision and were less likely to be forgotten. Our inclusion of baseline conditions showed that search templates and accessory items compete for limited memory resources; when one object in VWM became a template, it caused the other item to be forgotten more often. Seeing an object held in memory during the context of visual search also improved its precision, although this did not always occur for accessory items. Experiment 2, however, showed that actually searching is not necessary for such a change to occur. Templates were remembered more often and with more precision even when no search occurred. These results paint a picture of how VWM representations are modulated by visual search and show that search templates are not just prioritized, but better represented (Olivers et al., 2011). However, they also go beyond this proposal in showing a representational cost for accessory items: Search templates’ colors were nearly always recalled at test, whereas accessory items’ colors were correctly recalled on approximately half of the memory tests; approximately 15% less often than our baseline VWM condition.
The present data allow us to distinguish between states in VWM based on memory quality; accessory items are fragile and have lower precision, whereas items being used for concurrent tasks (e.g., search templates) are robust and have relatively higher precision representations. Although it is too early to tell whether these findings primarily reflect task demands (e.g., the requirement to maintain color-shape bindings, the amount of color-precision required to distinguish targets from nontargets in search) or more fundamental differences in representation between states in VWM, we nonetheless provide initial evidence that the need to use a representation for search can affect its memory representation.
The present data fill a gap in previous investigations of the relationship between memory precision and search guidance. Whereas Hollingworth and Hwang (2013) showed that memories that do not guide attention are not necessarily less precise than memories being actively maintained, we show that memories being actively used for search are more precise and stable than accessory items. Additionally, Dowd, Kiyonaga, Beck, and Egner (2015) showed that the primary difference between instances in which memory-driven capture occurs and does not occur seems to lie in the probability that a memory is maintained, rather than the precision with which it is held. However, this result speaks to dynamics of whether a single item, which can vary in its task-relevance, affects search. In our task, two items in memory likely competed for representation, and thus the differences in precision may have resulted in a shift in representational resources towards the Template and away from the Accessory item due to its momentary task-relevance. An intriguing finding from Experiment 1 was that memory precision for the accessory item was, counter intuitively, impaired when that very item appeared in search. This finding may reflect the operation of distractor-suppression mechanisms that occur during search (Lamy, Tsal, & Egeth, 2003; Emrich, Al-Aidroos, Pratt, & Ferber, 2010); however, we are hesitant to draw strong conclusions from this finding given that it did not re-emerge in Experiment 2.
Our results support and extend the proposal of Olivers et al. (2011), who suggest that visual memories used to guide attention exist in a different state than other visual memories (see also Carlisle & Woodman, 2011). We have provided empirical evidence that the representational quality of Templates is superior to Accessory items. Our findings are likely related to the shifts in memory quality found using retro-cues (Murray et al., 2013; Gunseli et al., 2015) to the extent that guided visual search requires a form of internally focused attention within visual working memory. Indeed, the present design is similar to the “retro-cuing” paradigm used to study voluntary shifts of attention within VWM (Griffin & Nobre, 2003; Murray et al., 2013) but without the change in testing probability (see Zokaei, Manohar, Husain, & Feredoes, 2014; Zokaei, Ning, Manohar, Feredoes, & Husain, 2014; and van Moorselaar et al., 2014 for nonsearch alternatives). Despite this difference, the present effect and the retro-cue effect may reflect common mechanisms. In this case, the suggestion that templates occupy a special state in visual working memory can be seen as an application of Oberauer’s concept of the Focus of Attention (Oberauer, 2002); a single-item capacity state in working memory that maintains the representation currently being used for a mental task. In any case, critical future research needs to address underlying mechanisms behind these state differences, which could be due to differences in the temporal dynamics of task-relevant memories (Kiyonaga & Egner, 2014), differences in representational resources (e.g., active neural representation for Templates; Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012), or even both.
In summary, by measuring memory for VWM representations that guide search and those that do not, we have provided direct evidence for a privileged state in VWM for search templates over nonsearch items. This is not to say that voluntary attention necessarily requires VWM (Carlisle et al., 2011), but when it does, search templates enjoy a representational benefit in VWM. Becoming a selection template thus appears to shift VWM resources for the upcoming search task, demonstrating the role of internal attention in visual selection (Kiyonaga & Egner, 2013).
Funding provided by an NSERC Discovery grant (194537) to Jay Pratt, as well as an NSERC Doctoral Scholarship (PGS-D) to Jason Rajsic. The authors thank Dorothy Yu, Harendri Perera, and Kai Zhou for their assistance with this project.
- Bays, P. M., Catalao, R. F., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10).Google Scholar
- Hollingworth, A., & Hwang, S. (2013). The relation between visual working memory and attention: Retention of precise color information in the absence of effects on perceptual selection. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 368, 20130061.CrossRefPubMedPubMedCentralGoogle Scholar
- Kiyonaga, A., & Egner, T. (2014). Resource-sharing between internal maintenance and external selection modulates attentional capture by working memory content. Frontiers in Human Neuroscience, 8. doi: 10.3389/fnhum.2014.00670
- Kleiner, M., Brainard, D., Pelli, D., Ingling, A., Murray, R., & Broussard, C. (2007). What’s new in Psychtoolbox-3. Perception, 36(14), 1.Google Scholar
- Li, Q., & Saiki, J. (2014). The effects of sequential attention shifts within visual working memory. Frontiers in psychology, 5, 1–10.Google Scholar