Attention flexibly trades off across points in time
Sensory signals continuously enter the brain, raising the question of how perceptual systems handle this constant flow of input. Attention to an anticipated point in time can prioritize visual information at that time. However, how we voluntarily attend across time when there are successive task-relevant stimuli has been barely investigated. We developed a novel experimental protocol that allowed us to assess, for the first time, both the benefits and costs of voluntary temporal attention when perceiving a short sequence of two or three visual targets with predictable timing. We found that when humans directed attention to a cued point in time, their ability to perceive orientation was better at that time but also worse earlier and later. These perceptual tradeoffs across time are analogous to those found across space for spatial attention. We concluded that voluntary attention is limited, and selective, across time.
KeywordsTemporal attention Selective attention Voluntary attention Visual perception
Visual attention is spatially selective. Covertly attending to a specific location results in perceptual benefits at that location and costs at unattended ones, compared to when attention is evenly distributed across space (Giordano, McElree, & Carrasco, 2009; Montagna, Pestilli, & Carrasco, 2009; Pestilli & Carrasco, 2005; for reviews, see Carrasco, 2011; Kinchla, 1992). Is visual attention also temporally selective, resulting in perceptual tradeoffs between attended and unattended time points? The answer to this question is unknown, despite its importance to our understanding of dynamic vision.
Previous temporal cueing studies have found that perceptual accuracy and reaction time (RT) are better for attended compared to unattended times (accuracy: Correa, Lupiáñez, & Tudela, 2005; Davranche, Nazarian, Vidal, & Coull, 2011; Rohenkohl, Gould, Pessoa, & Nobre, 2014; Samaha, Bauer, Cimaroli, & Postle, 2015; RT: Coull & Nobre, 1998; Miniussi, Wilding, Coull, & Nobre, 1999; for a review, see Nobre & Rohenkohl, 2014). But none of these studies were able to assess perceptual tradeoffs between successive targets. The studies that measured accuracy presented only one target per trial and did not compare attended and unattended targets to a neutral baseline, so tradeoffs could not be evaluated. The studies that measured RT suffered from confounds: RT improvements at attended times can arise from motor preparation (Correa, 2010) or criterion changes (Carrasco & McElree, 2001), along with perceptual improvements.
In attentional blink studies, two targets (usually letters or numbers) are presented in a rapid sequence of nontarget items, and identification of the second target decreases when it appears 200–500 ms after the first (Dux & Marois, 2009; Raymond, Shapiro, & Arnell, 1992). Cueing observers to the timing of the second target mitigates this impairment without affecting the first target (Hilkenmeier & Scharlau, 2010; Martens & Johnson, 2005; Visser, Tang, Badcock, & Enns, 2014), suggesting that impairments follow but do not precede an attended item. However, identification accuracy for the first target is typically near ceiling, so cueing during an attentional blink task is not ideal to assess tradeoffs mediated by voluntary attention. We developed a new experimental protocol with high sensitivity to perceptual changes to establish the limits of voluntary temporal attention.
We compared four hypotheses. 1) Maximum sustained attention: If observers can sustain attention maximally across time without limits, they have no incentive to use the attentional precue and should always attend to both targets. This hypothesis predicts no difference across attention conditions. One-target cueing studies have generally not found cueing effects for the second of two possible target times, suggesting that observers may be able to sustain attention or rapidly reorient across time (Coull & Nobre, 1998; Davranche et al., 2011; Rohenkohl et al., 2014). 2) Sustained attention plus enhancement: If observers can sustain attention without limits, but also follow instructions to enhance attention at a particular point in time, they should show attentional benefits but not costs. Previous descriptions of voluntary temporal attention have focused on enhancement (Nobre, 2001). 3) Voluntary attentional blink: If attention decreases following a voluntarily attended target, there should be attentional costs for the second but not the first target. 4) Perceptual tradeoffs: If voluntary temporal attention governs perceptual tradeoffs between the targets, attentional benefits for one target should be mirrored by attentional costs for the other.
Ten observers participated in Experiment 1 (aged 19–43 years, 8 females). Author R.N.D. participated in all three experiments, and four other observers participated in both Experiments 1 and 3. The sample sizes in all experiments were similar to a previous temporal attention study that measured accuracy (Davranche et al., 2011) and to several studies on spatial attention (e.g., Anton-Erxleben, Herrmann, & Carrasco, 2013). All observers provided informed consent, and the University Committee on Activities involving Human Subjects at New York University approved the experimental protocols. All observers had normal or corrected-to-normal vision.
Observers discriminated the orientation of grating patches (Fig. 1a). On each trial, two targets (T1 and T2) were presented at the same spatial location, separated by an interval of 250 ms, an optimal interval for observing the attentional blink (Dux & Marois, 2009) (Fig. 1b). Each stimulus was tilted slightly clockwise (CW) or counterclockwise (CCW) from either the vertical or horizontal axis, with independent tilts and axes for T1 and T2. Both vertical and horizontal axes were used to reduce the number of trials on which the two stimuli were identical and discourage observers from adopting a strategy of judging whether the stimuli were the same or different to aid in discrimination. An auditory precue 1,000 ms before the first target instructed observers to attend to the first or the second target in selective trials (high tone: T1; low tone: T2) or to sustain attention across both targets in neutral trials (both tones simultaneously). Observers were asked to report the tilt of one of the two targets. An auditory response cue 500 ms after T2 indicated which of the two targets to report (high tone: T1; low tone: T2). The timing of auditory and visual events was the same on every trial. From trial to trial, the allocation of temporal attention varied (depending on the precue), and the response selection varied (depending on the response cue). Observers pressed one of two keys to indicate whether the tilt was CW or CCW relative to the main axis, with unlimited time to respond. For selective trials (80% of all trials), the response cue matched the precued target with a probability of 75% (valid trials) and the other target with a probability of 25% (invalid trials). For neutral trials (20% of all trials), the two targets were indicated by the response cue with equal probability. Observers received feedback at fixation (correct: green “+”; incorrect: red “−”) after each trial, as well as feedback about performance accuracy (percent correct) following each block of trials.
Observers completed two one-hour sessions on separate days. Each session consisted of 5 blocks of 64 trials with all combinations of cue type (valid: 60%, invalid: 20%, neutral: 20%), stimulus contrast (64%, 100%), probed target (T1, T2), target tilt (CW, CCW; independent for T1 and T2), and target axis (horizontal, vertical; independent for T1 and T2) in a randomly shuffled order, for a total of 640 trials across the two sessions. Observers completed 64 practice trials at the start of each session.
Observers also completed one session of training prior to the experiment to familiarize them with the task and determine their tilt thresholds. The training procedure was as follows. First, observers practiced single blocks (32 trials each) of the task with neutral precues only, a tilt of 5 degrees (which is typically easy to discriminate), and 80% contrast stimuli. They repeated this practice condition until they reached 75% accuracy within a block. Second, they performed 64 trials of the task with a 3-up-1-down staircase, all neutral precues, and 80% contrast stimuli to determine their individual tilt thresholds. These tilt values were used for the remainder of the experiment. Third, observers completed 64 trials of training with 80% contrast and all valid precues. Finally, observers completed 320 trials identical to an experimental session. One observer had difficulty discriminating the high and low tones, so tones of 1046.5 Hz (C6) and 440 Hz (A4) were used, and extra training was provided.
Online eye tracking was used to ensure central fixation throughout the experiment. Initiation of each trial was contingent on fixation, with a 750-ms minimum intertrial interval. Observers were required to maintain fixation, without blinking, from the onset of the precue until the onset of the response cue. If observers broke fixation during this period, the trial was stopped and repeated at the end of the run.
For each target and cue validity combination (T1 valid, T1 neutral, etc.), we obtained the mean accuracy (Experiments 1 and 2) or precision (Experiment 3) across observers and used these values to calculate the benefit and cost indices. The indices generalize across different performance measures, allowing comparisons across experiments, and normalize across targets, which may have different overall cueing effect magnitudes. To assess how benefits for a cued target were related to costs for the other target(s), we paired up the benefit and cost indices in each experiment according to the tradeoffs prediction: the benefit for target A was paired with the cost for target B when A was cued.
Statistical procedures are fully described in Supplementary Information.
Results and discussion
The results of Experiment 1 provided initial support for the perceptual tradeoffs hypothesis. Accuracy was highest for valid trials (mean across targets = 80%), intermediate for neutral trials (76%), and lowest for invalid trials (71%), reflecting both attentional benefits and costs compared to the neutral condition (Fig. 1c, d; ANOVA main effect of validity: F(2,18) = 17.64, p < 0.0001, η G 2 = 0.12). Performance was comparable for T1 and T2 (no main effect of target or interaction between target and validity, p > 0.1), and all pairwise comparisons among valid, neutral, and invalid trials were significant when combining across targets (valid vs. invalid: t(9) = 5.96, p = 0.0002, d = 1.89; valid vs. neutral: t(9) = 3.84, p = 0.004, d = 1.21; neutral vs. invalid: t(9) = 2.58, p = 0.03, d = 0.82). Accuracy also was higher on valid compared with invalid trials for each target individually (T1: t(9) = 7.84, p < 0.0001, d = 2.48; T2: t(9) = 3.9, p = 0.004, d = 1.23). T1 showed significant benefits (t(9) = 3.14, p = 0.012, d = 0.99) and costs (t(9) = 2.61, p = 0.028, d = 0.82). T2 benefits (t(9) = 1.48, p = 0.17, d = 0.47) and costs (t(9) = 1.54, p = 0.16, d = 0.48) were smaller and not significant. RT showed the same pattern as accuracy, with fastest responses for valid trials, intermediate for neutral, and slowest for invalid trials (Fig. 1e, f; Supplementary Information). These data also establish the temporal width of the voluntary attentional selection window to be shorter than 250 ms; if voluntary attention had less temporal precision, there would be no differences among cueing conditions.
The perceptual tradeoffs hypothesis predicts that cueing one target will benefit that target at the cost of the other target. These benefits and costs could be symmetrical in time (equal magnitude benefits and costs for both targets) or asymmetrical (e.g., strong benefits for T1 and strong costs for T2, but weaker benefits for T2 and costs for T1). To examine the pattern of benefits and costs and facilitate comparisons across experiments, we calculated for each target a benefit index and a cost index and paired them according to the cued target (see Methods). The similar magnitudes of benefits and costs for each cued target support the tradeoffs hypothesis (Fig. 1g).
Experiment 1 showed clear attentional costs for T1—a decrement in performance before an attended time—contrary to the voluntary attentional blink hypothesis. But the comparison between neutral and invalid was not statistically significant for T2, so whether attentional costs also occur after an attended time was not clear. We reasoned that precise temporal allocation of attention may be less critical for the last target, because there is extra time to process it. To test whether costs would occur after an attended time if the unattended item was not the last in the sequence, we conducted Experiment 2, with three targets (T1, T2, and T3).
Observers, stimuli, and procedure
Results and discussion
Voluntary temporal attention affected performance accuracy for all three targets, demonstrating its flexibility over sub-second timescales (Fig. 2b, c). Accuracy was again highest for valid (80%), intermediate for neutral (76%), and lowest for invalid trials (72%), with a main effect of cue validity (F(2,18) = 7.68, p = 0.0039, η G 2 = 0.12). RTs showed that the changes in accuracy were not due to speed-accuracy tradeoffs (Fig. 2d, e; Supplementary Information). Addressing the question of whether costs in accuracy can occur after an attended time, T2 showed attentional costs overall (t(9) = 2.94, p = 0.017, d = 0.93) and specifically on those trials in which T1 was cued (t(9) = 2.80, p = 0.021, d = 0.88). These costs were mirrored by attentional benefits for T1 (t(9) = 2.28, p = 0.049, d = 0.72). T3 had higher accuracy on valid than invalid trials (t(9) = 2.673, p = 0.026, d = 0.84), but did not show reliable benefits and costs. Benefits and costs were less symmetrical for each target compared to Experiment 1 (interaction between cue validity and target, F(2,18) = 2.81, p = 0.040, η G 2 = 0.033). Thus, there were no costs for T1 in this experiment, but there were marginal costs for T2 when T3 was cued (t(9) = 2.25, p = 0.051, d = 0.71), providing additional support for costs before an attended time. Pairing benefits with their associated costs shows that cueing T1 and T3 resulted in both benefits to those targets and costs to the other targets, but there was no evidence for benefits or costs when T2 was cued (Fig. 2f). The difference between valid and invalid performance accuracy for T2 was still about half that for T1 and similar to the T3 difference. The smaller effects of attentional cueing for T2 compared with T1 may have therefore resulted not from extra processing time but from T1 partially disrupting voluntary temporal attention to subsequent targets.
In what way does voluntary temporal attention change perception? One possibility is that voluntary temporal attention increases the quality of a visual representation, similar to rhythmic expectation (Cravo, Rohenkohl, Wyart, & Nobre, 2013; Rohenkohl, Cravo, Wyart, & Nobre, 2012). Another possibility is that attending to one target results in completely missing another target, similar to the attentional blink (Asplund, Fougnie, Zughni, Martin, & Marois, 2014; Sergent & Dehaene, 2004). The discrimination task, with its binary measure of accuracy, could not distinguish between these possibilities. In addition, a trivial explanation for performance tradeoffs would be that on invalid trials, observers mistakenly reported the tilt of the precued target instead of the one indicated by the response cue (as these tilts differed in half of the trials). This possibility must be ruled out to conclude that the tradeoffs are perceptual. To distinguish among these possibilities, we performed an experiment with a graded measure of performance accuracy on each trial.
Observers, stimuli, and procedure
Each session of the experiment consisted of eight blocks of 40 trials, for a total of 640 trials across two 1-hour sessions. Observers completed 16 practice trials at the start of each session. Observers who had not participated in the discrimination experiment (Experiment 1) completed one session of training before the experiment. The training consisted of practice with all neutral precues (blocks of 16 trials, until reaching criterion of 15% with <5° error), training with all valid precues (64 trials), and training on all precue validities (40 trials).
We modeled the distributions of estimation errors across trials to characterize: 1) the precision of the orientation representation, 2) the rate of all-or-none guessing, 3) the likelihood of target swapping (i.e., reporting the target not indicated by the response cue), and 4) a bias to report clockwise or counterclockwise (Bays, Catalao, & Husain, 2009; Suchow, Brady, Fougnie, & Alvarez, 2013; Zhang & Luck, 2008). Model comparison showed that only the first two parameters, precision and guess rate, were required to capture the data (Supplementary Information). For each experimental condition, we estimated the precision as the standard deviation of a circular normal distribution centered on zero error (inverse precision) and the guess rate as the height of a uniform distribution over orientation (Fig. 3b). Additional modeling details are available in Supplementary Information.
Results and discussion
Across targets, we found that temporal attention changed the precision of the visual representation (Fig. 3c, d; F(2,22) = 6.41, p = 0.006, η P 2 = 0.37) without affecting the guess rate (Fig. 3e, f; F(2,22) < 1). Precision was higher for T2 than T1 (F(1,11) = 57.75, p < 0.0001, η P 2 = 0.84), and only T1 showed reliable precision increases with attention (valid vs. invalid: Z = 2.20, p = 0.027, r = 0.64; valid vs. neutral: Z = 2.51, p = 0.009, r = 0.72). Cue validity affected precision across targets even when the swap parameter was included (F(2,22) = 5.75, p = 0.010, η P 2 = 0.34). Thus, changes in visual representations, rather than selection failures or order errors, were responsible for attention-related performance changes. Model comparison rejected the inclusion of a swapping parameter, so there was no evidence that observers mistakenly reported the orientation of the wrong target. The benefit/cost index for precision showed similar-magnitude benefits and costs when T1 was cued, but there was no evidence for benefits or costs when T2 was cued (Fig. 3g).
Performance across experiments
Altogether, these experiments support the perceptual tradeoffs hypothesis. Voluntarily directing attention to a point in time leads to perceptual improvements at that time but also impairments earlier and later. The presence of costs shows that voluntary temporal attention is selective—not only in the sense that it can select a particular moment in time, but in the sense that when it does so, perception at other moments suffers. Such selectivity is the hallmark of voluntary spatial attention (Carrasco, 2011; Giordano et al., 2009; Kinchla, 1992; Montagna et al., 2009; Pestilli & Carrasco, 2005), but whether voluntary attention also is selective in time had not been tested. Selectivity thus is common to both spatial and temporal voluntary attention.
The effects of voluntary temporal attention differ from those seen in the attentional blink. Voluntary temporal attention affects the precision of the visual representation rather than an all-or-none selection process, as has been reported for the blink (Asplund et al., 2014; Goodbourn et al., 2016; Sergent & Dehaene, 2004), and attentional costs can occur both before and after the attended target. Some experiments have shown that voluntary temporal attention and the attentional blink do not interact, whereas others have found that cued attention benefits the second target more when it is within the attentional blink period (Hilkenmeier & Scharlau, 2010; Martens & Johnson, 2005; Visser et al., 2014). It may be that the attentional blink reflects a form of involuntary temporal attentional reduction elicited by the target (Wyble, Potter, Bowman, & Nieuwenstein, 2011), which is dissociable from voluntary temporal attention, or that it reflects a separate limitation (Chun & Potter, 1995). Multiple processes are likely involved in performing the attentional blink task (e.g., target selection, distractor suppression, letter/number identification; Dux & Marois, 2009). In contrast, the perceptual tradeoffs we observed can be unambiguously linked to temporal attention, because attention was explicitly manipulated.
The variability in the specific cueing effects found across experiments may reflect different attentional demands and strategies for different displays and tasks. For example, with two targets (Experiment 1), there were roughly symmetrical benefits and costs for both targets; whereas with three targets (Experiment 2), there were benefits for T1 and T3 and corresponding costs for T2, but not vice versa. A possible explanation for this difference is that voluntary temporal attention is most effective when cued to the beginning or end of a sequence. In Experiment 3, precision was much higher for T2 than for T1, perhaps reflecting differences between estimation and discrimination tasks. This may have left little room to measure attentional cueing effects for T2. The correlation analysis shows that the tradeoffs hypothesis—the correspondence of benefits and costs—provides a parsimonious explanation for the pattern of data across varying experimental conditions. In the future, it will be interesting to study a wider variety of tasks and displays, including longer sequences.
The finding that observers cannot sustain attention over a sequence of just two target stimuli adds to previous findings of temporal processing limitations for longer, more rapid sequences, and sequences with irrelevant (nontarget) items (Broadbent & Broadbent, 1987; Duncan, Ward, & Shapiro, 1994; Nieuwenstein, Van der Burg, Theeuwes, Wyble, & Potter, 2009; Reeves & Sperling, 1986; Weichselgartner & Sperling, 1987). It suggests limitations in visual processing over short time intervals (<1 s), rather than a strategic disruption of attention when an irrelevant item appears (Olivers & Meeter, 2008), consistent with accounts of resource sharing over time (Shapiro, Schmitz, Martens, Hommel, & Schnitzler, 2006; Wieser & Keil, 2011). Our findings show that voluntary temporal attention is not a process that can be sustained at will in the presence of multiple targets, nor a process that simply boosts sensory information at a relevant time. Rather, it is a flexible, cognitive selection process that serves to manage limits in visual perception across time, perhaps rendering them less noticeable in daily life.
Funding was provided by National Institutes of Health National Eye Institute grants R01 EY019693 to D.J.H. and M.C., F32 EY025533 to R.N.D, and T32 EY007136 to NYU supporting R.N.D. We thank Katherine Chen and Jacob Parker for assistance with data collection and members of the Carrasco Lab for helpful discussions.
- Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10):7, 1–11. doi: 10.1167/9.10.7
- Chun, M. M., & Potter, M. C. (1995). A two-stage model for multiple target detection in rapid serial visual presentation. Journal of Experimental Psychology: Human Perception and Performance, 21(1), 109–127.Google Scholar
- Correa, A. (2010). Enhancing behavioural performance by viual temporal orienting. In A. C. Nobre & J. T. Coull (Eds.), Attention and Time. Oxford, UK: Oxford University Press.Google Scholar
- Coull, J. T., & Nobre, A. C. (1998). Where and when to pay attention: The neural systems for directing attention to spatial locations and to time intervals as revealed by both PET and fMRI. Journal of Neuroscience, 18(18), 7426–7435.Google Scholar
- Giordano, A. M., McElree, B., & Carrasco, M. (2009). On the automaticity and flexibility of covert attention: A speed-accuracy trade-off analysis. Journal of Vision, 9(3):30, 1–10. doi: 10.1167/9.3.30
- Kleiner, M., Brainard, D., & Pelli, D. (2007). What’s new in Psychtoolbox-3? Perception 36 ECVP Abstract SupplementGoogle Scholar
- Miniussi, C., Wilding, E. L., Coull, J. T., & Nobre, A. C. (1999). Orienting attention in time: Modulation of brain potentials. Brain, 122(Pt 8), 1507–1518.Google Scholar
- Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorial in Quantitative Methods for Psychology, 4(2), 61‐64. Google Scholar
- Nieuwenstein, M., Van der Burg, E., Theeuwes, J., Wyble, B., & Potter, M. (2009). Temporal constraints on conscious vision: On the ubiquitous nature of the attentional blink. Journal of Vision, 9(9):18, 1–14. doi: 10.1167/9.9.18
- Rohenkohl, G., Gould, I. C., Pessoa, J., & Nobre, A. C. (2014). Combining spatial and temporal expectations to improve visual perception. Journal of Vision, 14(4):8, 1–13. doi: 10.1167/14.4.8
- Suchow, J. W., Brady, T. F., Fougnie, D., & Alvarez, G. A. (2013). Modeling visual working memory with the MemToolbox. Journal of Vision, 13(10):9, 1–8. doi: 10.1167/13.10.9.doi
- Visser, T. A. W., Tang, M. F., Badcock, D. R., & Enns, J. T. (2014). Temporal cues and the attentional blink: A further examination of the role of expectancy in sequential object perception. Attention, Perception, & Psychophysics, 76(8), 2212–2220. doi: 10.3758/s13414-014-0710-7 CrossRefGoogle Scholar