Introduction

We often rely on visual information that is no longer seen yet lingers in visual short-term memory (VSTM). While external attention (to locations, features, or objects) sets the priorities for which information will be encoded into VSTM in the first place (reviewed in Carrasco, 2011), internal attention can select information currently maintained therein (reviewed in Chun, Golomb, & Turk-Browne, 2011). In the present study, we shed new light onto the relation between these core selective mechanisms of human perception and cognition, providing evidence for feature-based selection in VSTM and revealing its independence of visual priorities during sensory encoding.

Many studies have approached the relationship between attention and memory using cueing procedures. A cue to a location, for instance, presented shortly after the offset of a visual array, retains memory of stimuli that had occupied the cued location (Sperling, 1960). Indeed, spatial retro-cues can generate behavioral results equivalent to those of pre-cues (Griffin & Nobre, 2003; Landman, Spekreijse, & Lamme, 2003), even when presented seconds after the disappearance of the stimulus array (e.g., Astle, Summerfield, Griffin, & Nobre, 2012; Sligte, Scholte, & Lamme, 2008).

Non-spatial retro-cues may also result in benefits for memory performance (Heuer & Schubö, 2016; Lepsien & Nobre, 2007; Li & Saiki, 2015; Pertzov, Bays, Joseph, & Husain, 2013), suggesting that attention can influence VSTM maintenance across many flavors of attention. Most of these studies (Heuer & Schubö, 2016; Li & Saiki, 2015; Pertzov et al., 2013) have shown that colored retro-cues can improve memory for items of the same color. However, these studies invariably involved a spatial component, as the color cue identified isolated objects at particular locations. It remains unclear, therefore, whether selection in memory was based on spatial attention, drawn to the location of an object of a particular color (cf. Pertzov et al., 2013), or, alternatively, whether feature-based attention (FBA) – independently of space – affects memory maintenance. Moreover, few studies (if any) have addressed the interplay of external and internal attention.

We devised a novel double-cueing paradigm to fill these gaps. Observers reported from memory the orientation of one of two spatially interspersed groups of gratings (one white, one black). Using colored pre-cues (presented before stimulus onset) and retro-cues (presented after stimulus offset), predicting the group whose orientation would have to be reported, we manipulated observers’ FBA independently during stimulus encoding and maintenance, respectively. This strategy allowed us to address the relationship between FBA and VSTM when selection by location was explicitly ruled out.

Methods

Participants

Based on the sample size in previous studies on FBA and VSTM (e.g., Herrmann, Heeger, & Carrasco, 2012; van Moorselaar, Olivers, Theeuwes, Lamme, & Sligte, 2015), we recruited ten observers (aged 23–30 years; three females, two left-eye dominant, all right-handed) for participation in the experiment. All observers were naïve regarding the purpose of the study and participated in exchange for a fixed monetary payment of 7€ per session (1 h each), and a bonus of 7€ after completion of all six sessions. The Ethics Committee of the German Society for Psychology (DGPs) approved the study and observers provided written consent before participation. The study was conducted in accordance with the Declaration of Helsinki.

Experimental setup

Observers sat in a dimly lit room with their head positioned on a chin rest, at a distance of 57 cm from a gamma linearized 22-in Sony GDM-F520 screen (1,280 × 960 pixels, 100 Hz vertical refresh rate). An Eyelink 1000 Desktop Mount (SR Research, Ottawa, ON, Canada) recorded the position of the dominant eye at 1,000 Hz. For stimulus presentation, online gaze control, and response collection, we used MATLAB (Mathworks, Natick, MA, USA), including the Psychophysics toolbox (Brainard, 1997; Pelli, 1997), the Eyelink toolbox (Cornelissen, Peters, & Palmer, 2002) running on a Mac mini (Apple Inc., Cupertino, CA, USA). Manual reports were collected using a PowerMate 3.0 (Griffin Technology, Nashville, TN, USA).

Stimuli and procedure

During each trial (Fig. 1a) observers had to maintain fixation at a centrally presented red filled circle (diameter 0.2° of visual angle, dva). After 500 ms of fixation, a pre-cue appeared for 500 ms (a ring with a diameter of 1.0 dva, width of 0.12 dva, and color defined by experimental condition), either informing observers about the color of the stimulus that would most likely be probed in the later memory test (black or white; 67 % validity), or not providing any prior information (red; neutral cue). After an inter-stimulus-interval (ISI) of 500 ms, the memory array, consisting of two sets of differently colored and oriented gratings (one black, one white), was briefly flashed for 150 ms. Observers were asked to keep the orientations in memory and report the orientation of one set of gratings at the end of the trial. Following an ISI of 1,000 ms, a retro-cue appeared for 500 ms (same visual features as the pre-cue), telling observers in 75 % of the trials (valid retro-cue) which set of gratings would be probed in the memory test (100 % validity). In the remaining 25 % of trials, a neutral retro-cue was uninformative with respect to the memory test and observers could rely only on the information provided by the pre-cue. Finally, following an ISI of 1,000 ms, one set of colored gratings (the positions of which did not coincide with the positions occupied in the memory array; see below) was presented and observers were asked to rotate (using a volume knob) the orientation of the gratings (initial orientation drawn from a uniform distribution ranging from 0° to 180°) until they matched the orientation of the stimulus kept in memory. Finally, by pressing the knob, the response was saved and participants received visual feedback about their error (in the range of 0–90°). Throughout the trial, we controlled fixation behavior using an online eye-tracking routine that aborted a trial automatically as soon as the distance between eye position and fixation point was larger than 2 dva. Observers repeated these aborted trials at the end of a block.

Fig. 1
figure 1

(a) Trial sequence in the double-cueing paradigm. Observers fixated a red fixation point in the center of the screen. After 500 ms, a pre-cue was presented that was either informative (black or white; 67 % validity) or neutral (red). After an inter-stimulus-interval (ISI) of 500 ms, a memory array – consisting of two differently colored (black and white) and oriented sets of gratings – appeared for 150 ms. 1,000 ms after the disappearance of the memory array, a retro-cue (valid or neutral) appeared for 500 ms, followed by another retention interval of 1,000 ms. Finally, the probe array (either black or white; initial orientation drawn from a uniform distribution, ranging from 0° to 180°) appeared and observers were asked to adjust the orientation of gratings with a knob such that it matches the orientation of the same-colored group of gratings in the target array. By pressing the knob, the response was saved and participants received feedback about their error. Note that informative cues (valid and invalid) could be either black or white. In the example shown here, the black cue correctly predicts the target group and is thus valid; the white cue incorrectly predicts the non-target group (invalid). (b) Number of trials in each experimental condition. (c) Example distribution of orientation reports, including the fit of a mixture model. The model comprised random guesses and non-target reports in addition to target reports, each associated with a certain probability (γ, β, and 1–(γ+β), respectively). Target and non-target reports were assumed to have the same precision, captured by its inverse, the standard deviation (σ). Note that in the overall fit (top panel), the non-target reports are distributed across a range of angles, due to the variable relation between target and non-target orientations

Full size image

The memory array consisted of 32 (16 black and 16 white) gratings (spatial frequency = 3.8 cycles per dva) arranged on a 9 × 9 dva2 regular grid centered at fixation (inspired by the stimuli used in Herrmann et al., 2012). Each grating was the positive (white) or the negative (black) lobe of a sine wave grating, fading out in a Gaussian window (SD = 0.16°). The minimum distance between the center of the screen and the center of its closest neighboring grating was 2.7 dva. We randomly jittered (uniform distributions, ranging from 0 to 0.6 dva) the horizontal and vertical position of each grating in each interval and trial. This spatial layout and small size of the stimuli was used to require observers to allocate attention globally over the whole visual display, to encode the orientation of the full set of gratings – and not the orientation of a single one (cf. Herrmann et al., 2012). The orientations of the target set and the non-target set of gratings varied independently with a distance drawn from a uniform distribution in the range from 60° to 120°.

Each observer was tested in six experimental sessions (each session consisting of eight blocks with 40 trials per block) contributing a total of 1,920 trials to the final analysis (see Fig. 1b for trial numbers per condition), such that the least likely condition (invalid pre-cue and neutral retro-cue) had the minimum number of 96 reports.

Data analysis

Formal model comparison (see Supplemental Online Material) revealed that a mixture model (proposed by Bays, Catalao, & Husain, 2009) yielded the best description of individual report distributions across conditions. This model comprises target reports, non-target reports, and random guesses:

$$ p\left(\theta \right)=\left(1-\gamma -\beta \right){\varPhi}_{\mu, \kappa }+\beta {\varPhi}_{\delta, \kappa }+\gamma \frac{1}{2\pi }, $$

where θ is the difference between the reported value and the target value, Φ is a von Mises probability density function (i.e., the circular analogue of a Gaussian) with mean μ (target orientation) or δ (non-target orientation) and the spread parameter k (which relates to the standard deviation, \( \sigma =\sqrt{1/\upkappa} \)). γ and β are weight coefficients, corresponding to the proportion of random guesses and non-target reports, respectively. Fits of this model to the report of each observer in each condition provided the basis for the results reported in this manuscript. Note that for orientation reports the range of unique angles is 0°−180° (rather than 0°−360°); thus, we mapped orientation angles onto radians using the factor pi/90° prior to fitting.

Statistics

To assess the impact of pre-cueing and retro-cueing on memory performance, we conducted two-way repeated-measures analyses of variance (rmANOVA) on each dependent variable. As proportions (of non-target or random reports) are not usually normally distributed, we used arcsine-transformed data in all rmANOVAs on these dependent variables. To further assess the evidence for the observed effects (or their absence), we computed Bayes Factors (BF), complementing each rmANOVA. BFs express the posterior probability for a model given the data (fixed effect of factor + random effect of participants) relative to a model including only participants as a random factor. BFs were computed using the R package BayesFactor (Morey & Rouder, 2015).

We calculated 95 % confidence intervals (CI95%) using a bootstrap technique (Efron & Tibshirani, 1993). To generate each of the 1,000 bootstrap data sets, we first resampled the number of reports, N, that each participant provided in a given condition exactly N times (with replacement), fitted the model, and obtained the parameters of the best fit. Then, we averaged these bootstrapped parameters across the ten subjects, resulting in one bootstrap data set. We repeated this procedure 1,000 times and obtained 2.5 % and 97.5 % percentiles of a given dependent variable across the 1,000 bootstrap data sets. These 1,000 bootstrap data sets were also used to compute the CI95% of differences between two conditions by simply subtracting the parameter values of the first condition’s 1,000 bootstrap data sets from those of the second condition, and determining the 2.5 % and 97.5 % percentiles of the resulting distribution of differences.

Results

In our double-cueing paradigm, the pre-cue set observers’ priorities during the initial encoding of the stimulus. It would either direct observers’ attention correctly to the target orientation (valid condition), incorrectly to the non-target orientation (invalid condition), or to both orientations at the same time (neutral condition). If participants were able to change their priorities after the stimulus had disappeared, then a valid retro-cue should increase performance (relative to the neutral retro-cue) even if, initially, the wrong orientation had been attended. Indeed, the absence of an interaction between pre-cueing and retro-cueing would provide strong support for this hypothesis. We assessed this prediction on a number of dependent variables.

As a first step in our analysis, we calculated the mean angular error of observers’ orientation reports in each condition (Fig. 2a). To assess the effect of the pre-cue and the retro-cue on this global measure of memory performance, we conducted a two-way rmANOVA. We observed main effects for both types of cueing. Both during encoding (pre-cue: F(2, 18) = 13.98, p < 0.001; BF = 18182) and during maintenance (retro-cue: F(1, 9) = 45.47, p < 0.001; BF = 7.9), directing observers’ FBA significantly affected report errors, and these main effects were largely independent of one another (interaction: F(2, 18) = 0.14, p > 0.250; BFMain effects = 2677153 vs. BFMain effects + Interaction = 597676, BF-Ratio: 4.48). Here, the ratio of BFs (BF-Ratio) indicates that a model including the two main effects only is preferable by a factor of 4.48 to a model containing an additional interaction of the pre-cue and the retro-cue. As compared to performance in the neutral pre-cue conditions, reports were more accurate following valid pre-cues (Mean difference, Δ = 2.88°, CI95% 2.25 to 3.57) and less accurate following invalid pre-cues (Δ = –2.73°, CI95% –3.68 to –1.84). Moreover, the presentation of a valid retro-cue further decreased the error in orientation reports (Δ = –2.69° across all pre-cue conditions, CI95% –3.31 to –2.10).

Fig. 2
figure 2

Memory performance, averaged across observers, as a function of the pre-cue and retro-cue conditions. (a) Mean angular error of the orientation reports. (b) Standard deviation of target (and non-target) reports, σ. (c) Proportion of random guesses, γ. (d) Proportion of non-target reports, β. The error bars are bootstrapped CI95%

Full size image

Next, to identify potential sources of the observed errors and their differences across conditions, we modeled orientation report distributions with a three-parameter mixture model (Bays et al., 2009; see Methods and Fig. 1c). For each individual and each condition, we obtained estimates for the standard deviation of target reports (σ) around the true stimulus orientation, as well as for the proportion of non-target reports (δ) and the proportion of random guesses (γ).

The model estimates for the standard deviation σ (Fig. 2b) provide a measure of the precision of the memory representation (the inverse of the standard deviation, 1/σ). A rmANOVA revealed a significant effect of the pre-cue (F(2, 18) = 8.26, p = 0.003; BF = 27.39), but no effect of the retro-cue (valid vs. neutral: Δσ = 0.44°, CI95% –0.81 to 1.39; F(1, 9) = 0.6, p > 0.250; BF = 0.36) on σ. Importantly, there was no interaction between the pre-cue and retro-cue (F(2, 18) = 0.04, p > 0.250; BFMain effects = 10.9 vs. BFMain effects + Interaction = 2.3, BF-Ratio: 4.74). As compared to performance in the neutral pre-cue conditions, reports had a smaller σ (i.e., they were more precise) following valid pre-cues (Δσ = –1.12°, CI95% –0.48 to –1.80) and higher σ following invalid pre-cues (Δσ = 1.56°, CI95% 3.53 to 0.25). Thus, memory precision for orientation is limited by stimulus encoding, and does not improve (nor deteriorate) following a retrospective deployment of FBA to the memory representation.

Both pre-cue and retro-cue strongly affected the proportion of random guesses γ (Fig. 2c). As compared to neutral pre-cues, the proportion of random guesses was lower for valid (Δγ = –0.05, CI95% –0.02 to –0.08) and higher for invalid pre-cues (Δγ = 0.06, CI95% 0.02 to 0.10). The valid retro-cue further decreased the proportion of random guesses (Δγ = –0.06, CI95% –0.03 to –0.09), irrespective of pre-cue type. A rmANOVA corroborated these results, yielding a significant main effect of pre-cue (F(2, 18) = 4.83, p = 0.020; BF = 11.74) and retro-cue (F(1, 9) = 11.11, p = 0.009; BF = 2.22). The interaction between pre-cue and retro-cue again was not significant (F(2, 18) = 0.71, p > 0.250; BFMain effects = 43.7 vs. BFMain effects + Interaction = 12.1, BF-Ratio: 3.61). Thus, participants systematically rely on the information provided by both pre-cue and retro-cue in maintaining the memory of the orientations. More importantly, a switch in the priorities during maintenance (invalid pre-cue, valid retro-cue) leads to an improvement in performance that is as large as the retro-cueing effect in the valid or neutral pre-cue conditions.

Finally, the proportion of non-target reports β was consistently very small (δ ≈ 0.03) across all conditions (Fig. 2d). A rmANOVA revealed no main effects (pre-cue: F(2, 18) = 2.57, p = 0.10; BF = 0.9; retro-cue: F(1, 9) = 0.75 p > 0.250; BF = 0.298), and no significant interaction (F(2, 18) = 0.16, p > 0.250; BFMain effects = 0.3 vs. BFMain effects + Interaction = 0.072, BF-Ratio = 4.14).

In summary, valid pre-cues strongly reduced the mean angular error of observers’ reports by increasing the precision of target reports and decreasing the proportion of random guesses. Valid retro-cues further reduced mean angular errors by decreasing the proportion of random guesses. For none of our four dependent variables did we see evidence for an interaction between pre-cues and retro-cues. In fact, statistical models comprising only the two main effects were favored over models with an additional interaction term by a factor of about 4.

Discussion

We demonstrated that the deployment of FBA – much like spatial attention – is a powerful selection mechanism that not only increases observers’ sensitivity during sensory processing of visual information (reviewed in Carrasco, 2011), but also fortifies feature information held in VSTM, once the stimulus has disappeared from view. Whereas the initial encoding of a stimulus appears to set an upper limit for the precision of a memory representation, the deployment of FBA during maintenance effectively increases the likelihood that observers remember the target orientation (reduced guess rate). Our results suggest further that FBA prioritizes visual information during the maintenance of memoranda independently of the priorities the observer imposed during sensory encoding: reinforcing these initial priorities as well as changing priorities to a new feature in VSTM benefitted observers’ performance.

Whereas spatial retro-cues are known to strongly affect the content of visual memory (Astle et al., 2012; Griffin & Nobre, 2003; Landman et al., 2003; Makovski, Sussman, & Jiang, 2008; Murray, Nobre, Clark, Cravo, & Stokes, 2013; Rerko & Oberauer, 2013; Rerko, Souza, & Oberauer, 2014; Sligte et al., 2008), few studies have addressed the role of FBA during VSTM maintenance. In these studies, colored retro-cues identified one of several colored objects held in memory and participants reproduced their orientation (Heuer & Schubö, 2016; Li & Saiki, 2015; Pertzov et al., 2013). The observed advantage of valid retro-cues was comparable to that of spatial retro-cues in terms of its time course and the size of the effect (Pertzov et al., 2013), although it had a different spatial profile (Heuer & Schubö, 2016). Nevertheless, observers may have used the feature retro-cue to guide spatial attention to the remembered item’s location (cf. Pertzov et al., 2013). We meticulously designed our stimuli to pre-empt any spatial strategy to solve the task. Observers deployed attention globally to the orientation of one group of gratings, spatially interspersed with gratings of an uncued orientation, and displayed at sizable eccentricity. We thus isolated robust effects of FBA on the content of VSTM, despite a potentially privileged role of location in the maintenance of information in VSTM (Pertzov & Husain, 2014).

The literature on spatial attention in VSTM (see overview in Souza & Oberauer, 2016) may provide inspiration for the mechanisms underlying feature-based retro-cueing. One set of accounts suggests that retro-cues protect visual information from time-based decay or interference. Visual memory traces may become diffuse over time (Wimmer, Nykamp, Constantinidis, & Compte, 2014) or merge with neighbouring ones (Wei, Wang, & Wang, 2012). This view predicts a continuous decrease of the fidelity of memory representations and thus reduced precision in memory reports. A valid retro-cue could stop this diffusion, or reduce interference with distractor orientations (Pertzov et al., 2013). Alternatively, the entire representation may fade out (Wei et al., 2012), be suddenly lost (Zhang & Luck, 2009), or no longer available for recall, unless selected by retrospective attention (Murray et al., 2013). A proper evaluation of these scenarios calls for a manipulation of set-size or incentives to trade off quality and quantity of memory representations (Fougnie, Cormiea, Kanabar, & Alvarez, 2016). Moreover, if the retro-cue protected information from decay then memory performance should be the same in trials with valid retro-cues and trials with no retro-cue in which the report is required at the time the valid retro cue would normally appear (cf. Makovski et al., 2008; Pertzov et al., 2013).

Retro-cues may also protect memory representations from visual interference caused by the probe array (Makovski et al, 2008) or interference from a secondary task (Makovski & Pertzov, 2015). To account for the observed lack of interactions between pre-cues and retro-cues, this would imply that perceptual interference from the probe array affected all memory representations in a similar way, independent of the priority set by the pre-cue.

Finally, recent results show that repeated retro-cueing during the retention interval resulted in better memory performance for the repeatedly cued item as compared to single cues, suggesting that spatial retro-cues can strengthen memory representations (Rerko & Oberauer, 2013). Here, we observed a similar result: a valid retro-cues further added to the effect of the pre-cue. In contrast to the study on spatial retro-cueing, we presented cues before and after the memory array, suggesting that a combination of pre- and retro-cues can equally strengthen representations in a cumulative fashion.

In summary, we have demonstrated the visual system’s ability to flexibly re-set priorities for perception and memory. While future studies need to disentangle the mechanisms giving rise to these results, they highlight the role of FBA beyond initial sensory stages and add to the accumulating evidence for the versatility of fundamental selection processes in human vision and cognition.