Introduction

Visual search is an inherently specific task: A particular item is to be identified while other items, if present, are to be ignored. Consider searching for a specific flight’s departure time from a flight information screen at an airport. Search difficulty would be impacted by the number of flights displayed, with fewer displayed flights making for an easier search. Other visual information may also influence search efficiency, even if it is not relevant to the current task. For example, an adjacent display of flight information for arriving airplanes should be ignored. Airports also have ambient acoustic stimuli, including sounds that are meaningfully related but irrelevant to the visual search. For example, a loudspeaker may announce the flight numbers for departures and arrivals. Would such sounds impact the efficiency of searching a visual display, or are they easily ignored, because presented in a different sensory modality?

Crossmodal perceptual load effects (or lack thereof)

The search for flight information is analogous to perceptual-load manipulation experiments that have examined how visual search difficulty impacts the ability to disregard items clearly designated as to be ignored. The load theory of visual attention allocation suggests that the processing load of a search task determines the extent to which task-related stimuli designated as to be ignored (usually termed distractors) are processed (Lavie, 1995; Lavie & Tsal, 1994). According to load theory, perception has a limited capacity and processes all stimuli in an obligatory fashion until capacity runs out. Perceptual load is thought to vary with the perceptual demands of a search task, with load being determined by factors such as the similarity between target and nontarget items in the visual search array. According to the theory, if perceptual load is low and some attentional capacity goes unused during the search, leftover resources spill over to process visual stimuli outside of the designated search array. If perceptual load is high and all available attentional resource capacity is allocated to that task, no resources remain to process any additional stimuli. As a result, nontask visual stimuli can influence performance on the primary task in a low-load search, but will not in a high-load search situation. Recently, the load theory interpretation of these claims and the processes underlying such effects have been challenged (e.g., in dilution accounts; e.g., Tsal & Benoni, 2010; Wilson, Muroi, & MacLeod, 2011). However, the general pattern of results that load theory attempts to describe has been regularly replicated, and the theory itself is well-supported by both behavioral and neuroimaging studies involving visual stimuli (see Lavie, 2005, 2010, for reviews).

Although load theory has been tested extensively in the visual modality, relatively few studies have explored the crossmodal consequences of visual perceptual load. Only a handful of investigations have studied the impact of auditory information on visual searches in which visual perceptual load has been varied. The results of those studies do not give a consistent picture of the effects. Perhaps the first such crossmodal study was Tellinghuisen and Nowak (2003). In that study, following the visual search task utilized by Lavie and Cox (1997), participants searched a circular array of six centrally displayed letters for one of two target letters (X or N) presented along with five nontarget letters. In a high-load search, the nontargets were similar to the targets (H, Y, Z, K, and V), whereas in low-load searches, the nontargets were all the same (O) and differed from the targets. On each search trial, participants were also presented with to-be-ignored flanker stimuli that were either visual (a letter presented in a larger font to either the left or the right of the circular search array) or auditory (a letter pronounced via headphones, simultaneous with the onset of the circular search array). In some conditions, these to-be-ignored stimuli were task-related (i.e., distractors) and either compatible (e.g., the letter X when the target presented was an X) or incompatible (e.g., the letter N when the target presented was an X) with the target response.

Tellinghuisen and Nowak (2003) found that when these to-be-ignored or distractor stimuli were visual, the results were consistent with load theory. For low-load searches, response times (RTs) were slower on trials in which the distractor letter was incompatible with the target response than when that letter was compatible with the target response. For high-perceptual-load searches, RTs did not differ significantly, no matter whether the distractor letter was incompatible or compatible, indicating that the peripherally presented letter did not impact those visual searches.

The pattern obtained when letters were presented auditorily, however, was very different. Relative to response-compatible stimuli, response-incompatible auditory information resulted in longer RTs to identify the visual target in both the low-load and high-load conditions. Furthermore, the difference between RTs on trials with incompatible versus compatible auditory information was more than double in the high-load versus the low-load condition. This pattern suggests that the response compatibility of auditory information may be particularly difficult to ignore in a high visual perceptual load task. Recently, this pattern of results was replicated by Mahr and Wentura (2014) in a similar experiment that employed a crossmodal Stroop color identification task that varied perceptual load. In that study, participants searched for one of four particular visual color patches, arranged in a circular pattern similar to that used in Tellinghuisen and Nowak (2003). Perceptual load was manipulated by including nontarget patches of gray, for low-load searches, or nontarget patches of nontarget colors, for high-load searches. The to-be-ignored auditory stimuli consisted of spoken words that named color target words and could be compatible, incompatible, or neutral, relative to the required target response. Mahr and Wentura found that the effects of auditory stimuli (i.e., RT differences) were larger for high-load than for low-load searches.

In contrast, support for load theory has been found crossmodally by Lavie and colleagues (Macdonald & Lavie, 2011; Raveh & Lavie, 2015), showing that auditory stimuli unrelated to a visual task were less likely to be detected while participants performed a visual task when perceptual load was high than when it was low. In Macdonald and Lavie’s study, participants made judgments regarding the lines of a centrally presented stimulus in an otherwise blank field. High-load conditions involved making a line length discrimination between two slightly different lines in a crosshair pattern (vertical or horizontal). Low-load conditions consisted either of determining which of two noticeably different crosshair lines was longer or which of the two lines was blue. Load theory was tested crossmodally by determining whether perceptual load influenced the detection of a single, task-unrelated auditory stimulus (a pure tone of 100-ms or 150-ms duration) during a trial toward the end of a sequence of trials. Participants were less likely to report hearing the tone when it was presented in a high-load than in a low-load condition. Using a visual search task and load manipulation similar to those of Lavie and Cox (1997), Raveh and Lavie showed that in high-load search conditions, participants’ sensitivity (indexed by d’) was lower to detect a 100-ms tone presented simultaneously with a visual search task. This occurred even if the tone was presented on up to 50 % of trials and participants were asked to indicate whether a tone had been presented after each visual search task trial. Each of these study’s findings fits well with load theory, extending it to demonstrate that participants showed greater failure to perceive stimuli from another modality when visual perceptual load was high. Lavie and colleagues concluded that high perceptual load induces inattentional deafness, in that participants failed to notice the presence of a sound during a high-load task.

The methodology of Tellinghuisen and Nowak’s (2003) study differed, however, from those of the studies that demonstrated inattentional deafness (Macdonald & Lavie, 2011; Raveh & Lavie, 2015), and this might account for the larger effects of task-related auditory distractors during high-load searches in this paradigm. Although all stimuli had a simultaneous onset, Tellinghuisen and Nowak’s auditory stimuli were 300 ms in duration, and the visual search array appeared for only 100 ms. Thus, the auditory stimulus continued an additional 200 ms after the visual stimulus offset. Because targets are likely to be processed later in high-load than in low-load conditions (i.e., RTs are longer to high than to low loads), the temporal overlap of processing the visual target and the auditory stimulus might be greater in the high-load than in the low-load condition. Lavie (2005) reasoned that this difference might account for Tellinghuisen and Nowak’s findings that were counter to load theory.

In Experiment 1 of the present article, we addressed this methodological concern by replicating the conditions from Tellinghuisen and Nowak (2003) while varying the onset and offset between task-related auditory distractors and visual target search arrays. We included a condition in which the auditory distractor onset preceded the visual target search array onset, but both the distractor and the search array had a simultaneous offset. To anticipate the findings of that study, we showed that even in this condition, relative to compatible auditory distractors, incompatible distractors slowed RTs significantly more for high-load than for low-load searches. That pattern is contrary to perceptual load theory and replicates Tellinghuisen and Nowak.

Attention and processing task-related stimuli

The findings of Tellinghuisen and Nowak (2003) and Lavie and colleagues (Macdonald & Lavie, 2011; Raveh & Lavie, 2015) suggest opposing effects of visual perceptual load on auditory stimulus detection, with the former showing greater auditory processing in high-load searches, and the latter showing reduced auditory processing in high-load relative to low-load searches. Among the methodological differences between the tasks used by these studies, a critical one is the relationship between the target and auditory stimuli. Tellinghuisen and Nowak included auditory, to-be-ignored stimuli that were task-related, in that they named actual visual target possibilities that were compatible or incompatible with the target response. Lavie and colleagues’ auditory stimuli were pure tones, unrelated to the visual-task responses. In those studies, participants made a separate response to the auditory stimulus following visual target detection, effectively making Raveh and Lavie’s experiments (particularly) dual-task situations, whereas Tellinghuisen and Nowak’s was a single-task experiment.

In experiments testing load theory using only visual stimuli, the relationship between the targets and stimuli that are not part of the visual search array has been assumed to be a significant factor affecting how the stimuli impact search times. The relationship between the targets and the peripherally presented flankers is typically defined by whether the targets and flankers map onto the same (congruent) or opposite (incongruent) responses. For example, Lavie and Cox (1997) hypothesized that flanking letters that were not part of the target set (designated neutral distractors) would not influence RTs, but that flanking letters that were incompatible with the target response would. Neutral stimuli, being unrelated to any target stimuli, were hypothesized to have no substantive effect on measures of information processing, and Lavie and Cox’s results confirmed this hypothesis. Therefore, according to load theory, the relationship between targets and distractors may be critical for determining whether they will influence target processing.

Response-related flankers have been described as irrelevant to the search task because they are not informative regarding which target will appear, and thus did not predict the appropriate response (e.g., Lavie & Cox, 1997; Tellinghuisen & Nowak, 2003). However, such flankers may be prioritized as relevant by the attentional system if they are related to the response set for the task, by virtue of carrying at least some of the defining characteristics of the targets, such as belonging to the same visual category, being visually similar, having the same temporal characteristics, and/or appearing in a location nearby a target (Lleras, Buetti, & Mordkoff, 2013). Lleras et al. defined such stimuli as foils: a form of near-miss stimuli, very much like the target, but not fully meeting the target criteria. Flankers in visual search tasks would be foils when they are identical to one of the targets, except that they appear in the wrong location (i.e., in the periphery rather than as part of the designated target array). Lleras et al. contended that foils are automatically selected by the attention system at a very early stage if they have a strong perceptual or neural representation, and thus impact the RTs in visual search tasks (see also Buetti, Lleras, & Moore, 2014).

The auditory stimuli in Tellinghuisen and Nowak (2003; as well as in Mahr & Wentura, 2014) included those that mapped directly onto response-relevant attributes of the visual search array (i.e., audibly naming a target that could be in the visual array). However, by virtue of being presented in the to-be-ignored (auditory) modality, such stimuli were foils. By contrast, the auditory nontarget stimuli in Macdonald and Lavie (2011) and Raveh and Lavie (2015) would not be foils for participants’ responses to the visual task. Apart from being presented simultaneous with the onset of a visual target, the tones had no characteristics relevant to the visual task, and so in no way mapped onto the predefined responses for the task. In Macdonald and Lavie’s study, the auditory stimuli were unrelated to the overt task goals, because participants were not asked to monitor for a tone prior to performing the task. An efficient attention system should be expected to ignore nonfoils during a visual search task (Lleras et al., 2013).

The auditory stimuli in Raveh and Lavie (2015), by contrast, could be seen as relevant to responses, since participants were instructed to detect whether a tone was presented during a visual search task. However, this tone detection response was to occur following the response to the visual search task, leaving the participants in Raveh and Lavie effectively in a dual-task situation—required to detect and respond to two simultaneously presented, unrelated stimuli from different modalities. Dual-task performance across modalities has been shown to be difficult. Boot, Brockmole, and Simons (2005) demonstrated that the detection of one of two letters in a visual search condition was reduced when participants monitored a simultaneous auditory task involving numbers, but not when they were told to ignore the auditory stimuli. Furthermore, if the difficulty of a single task increases (high load), it could be expected that simultaneous performance of a second task (tone detection) should decrease, as greater cognitive control would be required to perform both tasks simultaneously. Santangelo, Olivetti Belardinelli, and Spence (2007), for example, demonstrated that under the highly focused attentional state required for monitoring a stream of centrally presented visual stimuli consisting of letters and numbers (a high-perceptual-load task), the performance of an orthogonal cuing task involving determining the spatial location of an auditory stimulus was diminished or eliminated. Similar effects have been found for the addition of a second attention-demanding task across other modalities (see Spence & Santangelo, 2009, for a review). Spence and Santangelo argued that auditory stimulus processing is not automatic when combined with performing a demanding visual task. If Raveh and Lavie’s experiment constituted a dual-task situation that to some degree included vigilance of auditory information as well as a visual task, their results may apply more to participants’ abilities to perform two tasks in different modalities than to the influence of to-be-ignored auditory information as a function of visual-task perceptual load.

Task rules may determine the task relevance of to-be-ignored auditory stimuli

The participants in visual search tasks are told what counts as a target. Such instructions establish a link between stimulus features and the task-appropriate responses required for a particular target or set of targets, thereby also determining what counts as a foil in the search. What makes such stimuli relevant, less relevant, or entirely irrelevant to task performance is the extent to which some of their defining characteristics match the target stimuli. Attention is “configurable,” in this sense, because stimuli that match the participants’ attentional-control settings within a task will produce involuntary shifts of attention to those stimuli (see contingent attentional-capture theory; e.g., Folk & Remington, 1998; Folk, Remington, & Johnston, 1992, 1993). Such control settings or goal-directed processes, based on task demands, play a determining role regarding which stimuli in a display capture attention (e.g., Gibson & Kelsey, 1998). Reisenauer and Dreisbach (2013, 2014) have argued that instructions cause automatic response activations. Cohen-Kdoshay and Meiran (2007, 2009) showed that task instructions resulted in flanker interference effects on even the very first trial of a target search task. Stimuli that do not include features relevant for task-appropriate responses would be unlikely to impact performance, since they would not correspond with a response activation applicable to the task being performed. Such rules define the links between the stimuli and responses. This was the case in Tellinghuisen and Nowak (2003), in which participants performed under instructions to categorize two possible target letter stimuli in a target search array, with one finger designated for responses when the letter X appeared, and another finger designated for responses when the letter N appeared. Therefore, when participants searched for X or N in a centrally presented visual array, those letters were task-relevant if they occurred anywhere in the visual scene, and as Tellinghuisen and Nowak (2003) demonstrated, even if they occurred in the auditory modality.

Reisenauer and Dreisbach (2013) directly tested how task performance rules and subsequent response activations influenced the processing of to-be-ignored picture distractors in a two-alternative forced choice visual-word categorization task. Participants responded to trials in which a single word was superimposed over a picture (e.g., the word flag superimposed on a picture of a spinning top), in which words were the targets, while the pictures were to be ignored. One group of participants was instructed to use a categorization rule based on a common feature of movement (abstract task rule condition), responding with one key if the word represented a moving object (e.g., top) and a different key for a nonmoving object (e.g., flag). Another group was instructed to categorize targets according to stimulus–response mappings (S–R mapping condition) in which words were arbitrarily assigned to one of two response keys (one set of unrelated words for one response, and a second set of unrelated words for a different response), rather than being assigned on the basis of a commonly held characteristic such as movement (vs. nonmovement). Therefore, the S–R mappings did not follow a general rule of categorization, but had to be learned by the participants in this group. All participants were presented with and responded to the same word and picture stimuli; the task rule and S–R conditions differed only in terms of the instructions given for categorizing the word targets, and in so doing, varying attentional-control settings. Across both conditions, responses were faster when the target word and picture belonged to the same category, and thus mapped to the same response. However, in the S–R mapping condition, these response congruency effects were found only for pictures that had an equivalent among the target words for a particular response. In the task rule condition, participants responded faster when the target word and picture belonged to the same natural category, and thus mapped to the same response (e.g., both the word and picture depicted moving objects), than when they did not (e.g., the word depicted a moving object, but the picture depicted a nonmoving object). In addition, such congruency effects were found even for picture stimuli that were not part of the target set (i.e., pictures of objects that never appeared as target words). Reisenauer and Dreisbach (2013) concluded that in the task rule condition, response interference occurred because the implementation of a relevant task rule increased processing of all task-related stimulus features, even if the features occurred in a to-be-ignored picture. The categorization rules set by the task instructions were the critical variable determining which word–picture combinations influenced the speed of processing.

In Experiments 2 and 3 of the present study, we explored the degree to which to-be-ignored auditory stimuli may be processed by manipulating categorization rules in a way similar to Reisenauer and Dreisbach’s (2013) formulation, and also as a function of perceptual load. We expected auditory stimuli to have a greater impact on processing during a visual search task when the auditory information mapped onto the response categories of the visual search task, particularly when perceptual load was high. This was tested using identical target search arrays and auditory stimuli, but varying the task instructions indicating the relationships between the targets and responses. That is, the stimuli were the same across conditions, but the rules for classifying targets differed. Such experiments should help explain the discrepancy between the findings of Lavie and colleagues (Macdonald & Lavie, 2011; Raveh & Lavie, 2015) and those of both Tellinghuisen and Nowak (2003) and Experiment 1 of this study. If the requirements of a task inherently activate rules for that task, as Dreisbach (2012; see also Reisenauer & Dreisbach, 2013, 2014) argued, we hypothesized that auditory stimuli that most directly map onto the categorization rules for responding to visual targets should have a greater influence on target processing than those that are less directly related to the categorization rules. In Tellinghuisen and Nowak’s study, the auditory stimuli were task-related, but participants were instructed that they were not task relevant; that is, participants were told that they should be ignored. As Reisenauer and Dreisbach (2013) showed, the currently active task rules, which are set by a participant acquiescing to follow the instructions of a task, will influence control settings and direct attention to any response-discriminative stimulus feature that fits that task rule, so such stimuli should act as foils (Lleras et al., 2013). We predicted that stimuli that map onto the goals of a visual search task should be processed such that incompatible stimuli will result in longer RTs than compatible stimuli, particularly under high-perceptual-load conditions, as had been found by Tellinghuisen and Nowak and our Experiment 1. We further predicted that stimulus compatibility should be processed less or not at all under different task rules in which stimuli did not map directly onto the visual search task goals.

Experiment 1

The influence of temporal overlap between visual target display and response-related, to-be-ignored auditory information was addressed by varying the onset of such auditory stimuli relative to that of visual search displays. As Lavie (2005; Macdonald & Lavie, 2011) has suggested, differences in the time courses of processing auditory stimuli presented in low-load versus high-load searches might be responsible for the effects found in Tellinghuisen and Nowak (2003). In Experiment 1, we varied the temporal relationship between the to-be-ignored auditory stimuli and visual search arrays in a task very similar to that of Tellinghuisen and Nowak (2003, Exp. 1). In this experiment, the auditory stimulus onset was either simultaneous with the visual search array (as had been the case in Tellinghuisen & Nowak, 2003) or began prior to the visual array, but the offsets of both the auditory stimulus and visual array were simultaneous. If processing time differences were responsible for Tellinghuisen and Nowak’s effects, auditory stimuli should not influence the RTs in high-load searches when auditory and visual stimuli had simultaneous offsets.

Method

Participants

We sought to test a minimum of 30 participants. Thirty-two students were recruited and participated in exchange for partial fulfillment of an introductory psychology course credit. Of these participants, one was excluded due to misunderstanding the task. The mean age of those included was 19.2 years (SD = 1.6 years), who comprised 23 females and nine males. All participants had normal or corrected-to-normal visual acuity and reported having normal hearing.

Stimuli and apparatus

The experimental stimuli were created and run by Super Lab Version 4.5 and presented via a PC with a 19-in. monitor in a normally lit room. The auditory stimuli were presented via headphones (Sony Model MDR-ZX 100). A viewing distance of 57 cm was maintained throughout the experiment by using a chinrest.

The visual stimulus displays were similar to those used in previous studies on perceptual load (e.g., Lavie & Cox, 1997; Tellinghuisen & Nowak, 2003). Example displays are depicted in Fig. 1. Visual displays comprised a target letter, either X or N, that appeared randomly and equal numbers of times in each of six positions arranged in a circular pattern with a radius of 2.3° from a central fixation point. The five remaining positions were occupied by combinations of other letters (O, H, K, V, Y, or Z), depending on the experimental condition (see the Design and procedure section below). All letters were white and uppercase (Helvetica font), presented against a black background, and subtended a visual angle of 0.6° vertically and 0.4° horizontally.

Fig. 1
figure 1

Schematic depictions of example low-load (a) and high-load (b) search conditions from Experiment 1

Full size image

In addition to the circular pattern of stimuli, auditory stimuli were presented via headphones and consisted of a spoken letter (“X,” “N,” or “L”) lasting 300 ms. The volume was approximately 70 dB, measured by a sound-level meter held about 1 cm away from the headphone speakers.

Design and procedure

This study was based on a fully within-subjects 2 × 3 × 2 design. The independent variables in this experiment were perceptual load (low or high), auditory stimulus compatibility (compatible, incompatible, or neutral), and onset of the stimulus (simultaneous onset or simultaneous offset with the visual display). All variables occurred equally across participants, and all conditions were randomized. The dependent variables were mean RTs (latency of response in milliseconds) and accuracy/error rate.

In the low-load condition, the letter O appeared in five positions of the visual display, and the target letter appeared in the sixth position. In the high-load condition, the letters H, K, V, Y, and Z appeared randomly in five positions, while the target letter occupied the sixth position. In all conditions, the target letter position was randomized.

All auditory stimuli were presented through headphones. Compatible auditory stimuli were the same letter as the target letter in a particular trial (e.g., X when the target was X). Incompatible auditory stimuli were the opposing target letter (X when the target was N, and N when the target was X). Neutral auditory stimuli were a letter unrelated to the response set, the letter L.

Auditory stimuli occurred in two different time sequences relative to the visual display. In the simultaneous-onset condition, the onset of the auditory stimulus coincided with the onset of the visual display, such that it was still heard after the visual display had disappeared. In the simultaneous-offset condition, the offset of the auditory stimulus coincided with the offset of the visual display, such that the auditory stimulus onsetted prior to the visual display and ended at the same time as the visual display.

Participants were tested individually under normal lighting conditions. Near visual acuity was measured with a Snellen 20-Foot Equivalent eye chart before the experiment began. Participants were seated at a table and instructed to keep their head in a chinrest for the duration of the experiment, and then were told to ignore the auditory distractor letters while searching for the target letter.

Each display began with a 100-ms beep (66 dB) followed by a fixation screen, appearing for 1 s. The fixation screen contained a plus sign in the center of the screen, which participants were told to focus on throughout the experiment. In each trial, the target display appeared for 125 ms. The 300-ms auditory distractor either preceded the visual display by 175 ms and ended at the same time as the display (simultaneous offset) or was presented at the same time as the visual display and was heard for 175 ms after the offset of the display (simultaneous onset). Participants rested their left index finger on the “X” key and their right index finger on the “N” key and were instructed to press “X” or “N” to indicate which target letter was presented. Speed and accuracy were both emphasized. If a participant failed to respond within 2 s following the onset of the target display, the computer presented the message “Too slow, please respond more quickly” for 2 s, after which the trial presentation resumed.

Participants first completed a practice block of 24 trials, consisting of trials randomly selected from the possible combinations of conditions used in the experiment. Following practice, participants completed three experimental blocks of 60 trials each (180 trials total), or 15 trials per condition. A short break was given between experimental blocks.

Results

Mean correct RTs and percentage error rates were computed for each participant as a function of perceptual load (low or high), auditory stimulus compatibility (compatible, incompatible, or neutral), and stimulus timing (simultaneous onset vs. offset). RTs below 125 ms or above 2 s, as well as trials with no responses, were counted as misses. Such trials were rare (less than 1.08 % of all trials; 0.22 % of low-load and 1.94 % of high-load trials). The percentage error rate was based on nonmiss trials.

Reaction time analysis

We performed a repeated measures analysis of variance (ANOVA) to assess differences in RTs for each of the independent variables. These data are shown in Table 1. Significant main effects were found for load, F(1, 30) = 303.10, MSE = 21,491.48, p < .001, η p 2 = .910, and compatibility, F(2, 60) = 40.21, MSE = 3,843.61, p < .001, η p 2 = .573. The main effects of load and compatibility were subsumed in a significant interaction between load and compatibility, F(2, 60) = 17.65, MSE = 3,604.83, p < .001, η p 2 = .370. This interaction suggests that the effects of compatibility varied as a function of load, and the nature of this interaction is explored below in the RT benefits, costs, and compatibility effect analyses. In addition, a significant main effect of the timing of auditory stimulus onset was found, F(1, 30) = 28.700, MSE = 3,421.69, p < .001, η p 2 = .489, with RTs being shorter when the auditory stimulus and visual display offsets were simultaneous (M = 732, SE = 17.76) than when the auditory stimulus and visual display onsets were simultaneous (M = 765, SE = 18.97). In the simultaneous-offset condition, the auditory stimulus began before the onset of the target stimulus; therefore, the auditory stimuli may have had a general alerting effect, resulting in shorter RTs than for the simultaneous-onset condition. This effect, however, did not differ depending on the compatibility of the stimulus or the difficulty of the target search. No other interactions approached significance (ps > .1), including the Load × Compatibility × Timing interaction, F(2, 60) = 0.91, MSE = 2,295.81, p = .914, η p 2 = .030. These analyses indicate that onset timing did not differentially influence auditory stimulus effects as a function of the other variables involved in this experiment.

Table 1 Mean response times (RTs, in milliseconds) and error rates (as percentages), with standard errors, as a function of auditory stimulus timing, compatibility, and visual-task perceptual load for Experiment 1
Full size table

RT benefits, costs, and compatibility effect analyses

In terms of the hypotheses tested in this experiment, the critical issue was whether the influence of auditory stimulus compatibility as a function of perceptual load differed with the temporal relationship between the auditory stimulus and visual search array presentations. The load by compatibility interaction found in the RT analysis above suggests that auditory stimuli presented during high-load searches had a greater influence on search times, and this possible outcome can best be evaluated by exploring the magnitude of the difference between RTs to trials with neutral auditory stimuli and trials with either compatible or incompatible auditory stimuli, as had been done previously by Tellinghuisen and Nowak (2003). Therefore, we computed three new dependent variables for each combination of load and timing for each participant. RT benefits were computed by subtracting the mean RT for compatible trials from the mean RT for neutral trials, yielding a measure of the RT benefit, relative to baseline, of being presented with auditory stimuli that correctly named the target. RT costs were computed by subtracting the mean RT for neutral trials from the mean RT for incompatible trials, yielding a measure of the RT cost, relative to baseline, of being presented with auditory stimuli that named the letter opposite the target. RT compatibility effects were computed by subtracting the mean RT for compatible trials from the mean RT for incompatible trials. This provided a measurement of the overall difference in RTs (by load and timing of stimulus onset) due to auditory stimulus compatibility.

A series of t tests were conducted to determine whether the RT benefits and costs were significantly greater than zero (indicating an influence of auditory stimulus compatibility relative to neutral) and to discover, among those effects that were significant, whether benefits and costs differed in their magnitudes. First, RT benefits and costs were analyzed with a series of one-sample t tests, comparing them to a set value of zero. The means involved and the outcomes of these tests are presented in Table 2. RT benefits were significantly greater than zero in all conditions. RT costs were significantly greater than zero for the two high-load conditions, but not for the two low-load conditions. Second, the magnitudes of the significant benefits versus costs across high-load conditions were compared with two paired-samples t tests. Benefits and costs were not found to differ significantly for either the high-load, simultaneous-onset or high-load, simultaneous-offset conditions, t(30) = 0.60, p = .551, and t(30) = 0.99, p = .328, respectively. This pattern indicates that robust benefits and costs were found across high-load conditions, whereas smaller benefits but no costs were found in low-load conditions, regardless of onset timing.

Table 2 Analysis of mean benefits and costs (RTs, in milliseconds), with standard errors, as a function of auditory stimulus timing and visual-task perceptual load, relative to a set value of 0, for Experiment 1
Full size table

Mean RT compatibility effects are found in Fig. 2. A series of one-sample t tests indicated that all RT compatibility effects were significantly different from zero, indicating that incompatible auditory stimuli yielded longer RTs than compatible auditory stimuli in all conditions. A paired-samples t test indicated that RT compatibility effects were larger in the high-load than in the low-load condition for both the simultaneous-onset, t(30) = 5.16, p < .001 and the simultaneous-offset, t(30) = 4.27, p < .001, conditions. The compatibility effects did not differ as a function of timing for high-load, t(30) = 1.91, p = .066, or low-load, t(30) = 1.73, p = .94, conditions.Footnote 1

Fig. 2
figure 2

Mean compatibility effects (RT) for the simultaneous-onset and simultaneous-offset conditions as a function of the two perceptual load levels in Experiment 1. Compatibility effect = mean RT for incompatible trials minus mean RT for compatible trials. Error bars indicate ±1 SEM. *Significantly different than zero

Full size image

Error rate analysis

We performed a repeated measures ANOVA to assess the difference in error rates for each of the independent variables. These data are shown in Table 1. We found significantly higher error rates in the high-load (M = 18.65 %, SE = 1.56) than in the low-oad (M = 4.62 %, SE = 0.52) condition, F(1, 30) = 107.41, MSE = 170.53, p < .001, η p 2 = .782. In addition, errors were significantly higher for incompatible (M = 12.79 %, SE = 1.14) and neutral (M = 12.64 %, SE = 1.11) auditory stimuli than for compatible auditory stimuli (M = 9.48 %, SE = 1.05), F(2, 60) = 7.93, MSE = 54.41, p < .001, η p 2 = .209. The main effects of load and compatibility were subsumed in a significant Load × Compatibility interaction, F(2, 60) = 5.50, MSE = 71.13, p = .006, η p 2 = .155. Error rates did not differ as a function of compatibility in the low-load condition, but were lower for compatible (M = 14.45 %, SE = 1.67) than for either incompatible (M = 20.89 %, SE = 1.85) or neutral (M = 20.62 %, SE = 2.04) auditory stimuli in the high-load condition. Error rates did not differ as a function of auditory stimulus timing, and importantly, no interactions involving timing were significant, including the Load × Compatibility × Timing interaction, F(2, 60) = 0.13, MSE = 74.27, p = .882, η p 2 = .004, indicating that participants’ error rates to auditory stimuli did not depend on whether the temporal overlap between the visual targets occurred through the first 125 ms or the last 125 ms of the auditory stimulus presentation.

Error rate compatibility effect analysis

Analyses were conducted on error rate compatibility effects, which were computed by subtracting the mean error rate for compatible trials from the mean error rate for incompatible trials. A series of one-sample t tests indicated that error rate compatibility effects were significantly different from zero for both the simultaneous-onset, t(30) = 3.71, p = .001 (M = 6.70 %, SE = 1.80), and simultaneous-offset, t(30) = 2.33, p = .027 (M = 6.18 %, SE = 2.6), high-load conditions, but not for the low-load conditions (both ps > .6, Ms < 1 %).

Discussion

Robust effects of auditory stimuli were found for both low-load and high-load searches, with these effects being significantly larger in high-load conditions. This pattern occurred in both simultaneous-onset and simultaneous-offset conditions and was not the result of speed–accuracy trade-offs, since error rates were elevated for trials on which RTs were also elevated. Although RTs were shorter in the stimultaneous-offset condition, in which the auditory stimulus onset preceded the onset of the target search arrays and may have had a general alerting effect, compatibility effects were similar to those in the simultaneous-offset conditions. Taken together, the pattern of results suggests that the potentially longer duration of auditory stimulus processing during high-load searches in the simultaneous-onset condition was not responsible for the larger compatibility effects obtained under high-load searches in the present study or in Tellinghuisen and Nowak (2003). Clearly, the auditory stimuli were perceived in high-load visual search tasks, showing that inattentional deafness (e.g., Macdonald & Lavie, 2011; Raveh & Lavie, 2015) does not occur with target-related auditory stimuli in this paradigm. The to-be-ignored auditory stimuli were processed, and this processing was modulated by visual perceptual load such that greater processing of those stimuli occurred under higher perceptual load.

Experiment 2

Having established that auditory stimuli can substantially influence visual target search processing in high-perceptual-load conditions when such stimuli are related to the task by mapping onto responses for the target search, we can now address questions regarding the conditions under which such influences can occur. Although Tellinghuisen and Nowak (2003) and Experiment 1 of the present study demonstrated that visual-target response-incompatible auditory stimuli significantly influenced search RTs, Lavie and colleagues (Macdonald & Lavie, 2011; Raveh & Lavie, 2015) showed that auditory stimuli presented during high-perceptual-load visual tasks were more likely to be ignored than auditory stimuli presented during low-load visual tasks. Experiment 2 explored a methodological difference between these sets of studies by manipulating the relationship between the auditory stimuli and visual targets.

Experiment 2 employed a visual search task similar to that of Experiment 1, to explore how the rules for categorizing targets by response, as manipulated by the task instructions, could influence the degree to which the to-be-ignored auditory stimuli influenced processing. Participants searched for a single letter or number in a circular search display while simultaneously being presented a to-be-ignored auditory stimulus, which was either the word “letter” or “number.” Auditory stimuli that named the superordinate category to which the targets belonged were used rather than stimuli that named the actual targets to allow for a manipulation of task instructions, and thus the relationship between the auditory stimuli and visual targets. In the category-rule condition, responses to number visual targets (4 or 7) were made with one hand, whereas responses to letter visual targets (X or N) were made with the other. Therefore, the targets mapped to a hand belonged to the same superordinate category, similar to the task rule condition of Reisenauer and Dreisbach (2013). In the arbitrary-rule condition, one letter and one number (4 and N) were assigned to the response by one hand, and a different letter and number (7 and X) were arbitrarily assigned to the response by the other hand. Therefore, the two targets mapped to each hand belonged to two different superordinate categories, similar to the S–R rule condition of Reisenauer and Dreisbach (2013).

We hypothesized that the variation in the relationship between the to-be-ignored auditory stimulus and visual search targets would be a critical factor in determining whether target processing was influenced by hearing the category to which a stimulus belonged. The category-rule condition should activate the larger categories of letters and numbers more effectively, because each response was directly mapped to a category type (one hand for the two letters, the other for the two numbers), and thus should act as an effective response-defining foil. In the arbitrary-rule condition, however, each response hand was designated for both one letter and one number. In that case, naming the category (letter vs. number) to which a target might belong would not map consistently and exclusively to one response rather than the other, so such auditory stimuli should be less effective foils and have less influence on target processing. Therefore, we predicted that the category-rule condition would result in significantly greater auditory stimulus influence on RTs than would the arbitrary-rule condition, despite the fact that participants would be presented with identical stimuli in both conditions. Following the results of Experiment 1 and Tellinghuisen and Nowak (2003), we predicted that this effect would be greatest in high-load searches.

Method

Participants

We sought to test a minimum of 30 participants in each of two conditions. Sixty-four participants were recruited in exchange for partial fulfillment of an introductory psychology course credit. Of these, four were excluded from the analysis for having accuracy rates below 60 % in some experimental conditions. The mean age of the final sample of 60 was 18.5 years (SD = 0.8 years), which comprised 40 females and 20 males. All participants had normal or corrected-to-normal visual acuity and reported having normal hearing.

Stimuli and apparatus

The apparatus was the same as in Experiment 1. The visual stimulus displays were identical to those in Experiment 1, although different individual elements were included. Example displays are depicted in Fig. 3. The visual displays comprised a target, either X, N, 4, or 7, that appeared randomly and equal numbers of times in each of six positions arranged in a circular pattern at a radius of 2.3° from a central fixation point. The five remaining positions were occupied by combinations of other items that, depending on the experimental condition, were either the letter O or symbols (#,], ^, √, and ∏) that were neither letters nor numbers (see the Design and procedure section). All items in the search arrays were presented against a black background and subtended a visual angle of approximately 0.6° vertically and 0.4° horizontally.

Fig. 3
figure 3

Schematic depiction of example low-load (a) and high-load (b) search conditions from Experiments 2 and 3

Full size image

In addition to the circular pattern of visual items, auditory to-be-ignored stimuli were presented via headphones. The auditory stimuli consisted of a spoken word (“letter” or “number”) or a burst of white noise. White noise was used in this experiment, rather than the letter L used in Experiment 1, so that there would be no relationship between this sound and any of the targets. All auditory stimuli were presented for 300 ms at approximately 70 dB, measured as in Experiment 1.

Design and procedure

This study was based on a mixed 2 × 3 × 2 design. The within-subjects variables in this experiment were perceptual load (low or high) and auditory stimulus compatibility (compatible, incompatible, or neutral), and the between-subjects variable was categorization rule (category rule vs. arbitrary rule). Combinations of perceptual load and auditory stimulus compatibility occurred equally across participants, and all conditions were randomized. Participants were randomly assigned to either the category-rule or the arbitrary-rule condition. The dependent variables were mean RTs (latency of response in milliseconds) and accuracy/error rate.

In the low-load condition, the letter O appeared in five positions of the visual display, and the target appeared in the sixth position. In the high-load condition, the symbols #,], ^, √, and ∏ appeared randomly in five positions, whereas the target occupied the sixth position. In all conditions, the target position was randomized.

Compatible auditory stimuli named the same category as the target in a particular trial (i.e., “letter” when the target was X or N; “number” when the target was 4 or 7). Incompatible auditory stimuli named the opposing target category (i.e., “letter” when the target was 4 or 7; “number” when the target was X or N). Neutral auditory stimuli consisted of white noise, and thus were unrelated to any response.

Participants in the category-rule condition were instructed to press the left-hand key if either N or X was presented in the visual array, and to press the right-hand key if either 4 or 7 was presented. Therefore, in the category-rule condition, a correct response was made using one key if a target was a letter and the other key if the target was a number. In this case, the compatible and incompatible auditory stimuli corresponded to only one response (i.e., “letter” was compatible with a correct left-hand response and incompatible with a correct right-hand response; “number” was compatible with a correct right-hand response and incompatible with a correct left-hand response). Participants in the arbitrary-rule condition were instructed to press their left-hand key if either a 4 or an N was presented in the visual array, and to press the right-hand key if either a 7 or an X was presented. Therefore, in the arbitrary-rule condition, each response key was related to both a letter and a number (e.g., “letter” could be compatible or incompatible with a response using either hand, depending on which target was presented).

Participants were tested in the same setting and at the same viewing distance as in Experiment 1. The sequence of each trial was the same as in the simultaneous-onset condition of Experiment 1. Participants rested their left index finger on one computer key and their right index finger on another key and were instructed to press the right or the left key to indicate which target was presented. Speed and accuracy were both emphasized.

Participants first completed a practice block of 24 trials, consisting of trials randomly selected from the possible combinations of conditions used in the experiment. Following practice, participants completed two experimental blocks of 144 trials each (288 trials total), or 48 trials per condition. Short breaks were given following practice and between the experimental blocks.

Results

Mean correct RTs and percentage error rates were computed for each participant as a function of perceptual load (low or high), auditory stimulus compatibility (compatible, incompatible, or neutral), and categorization rule (arbitrary rule or category rule). RTs below 125 ms or above 2 s, as well as trials with no responses, were counted as misses. Such trials were rare (less than 0.68 % of all trials; 0.38 % of low-load and 0.97 % of high-load trials). Percentage error rates were based on nonmiss trials.

Reaction time analysis

We performed a mixed-model ANOVA to assess differences in RTs for each of the independent variables. These data are shown in Table 3. Significant main effects were found for load, F(1, 58) = 218.50, MSE = 4,221.19, p < .001, η p 2 = .790, and compatibility, F(2, 116) = 9.46, MSE = 696.43, p < .001, η p 2 = .140. The main effects of load and compatibility were subsumed in a significant interaction between load and compatibility, F(2, 116) = 5.64, MSE = 782.91, p = .005, η p 2 = .089. The final significant effect in this analysis was the Compatibility × Rule interaction, F(2, 116) = 3.197, p = .045, η p 2 = .052. These interactions indicate that the effects of compatibility varied as a function of both load and categorization rule. The nature of these interactions is explored in the next section.

Table 3 Mean response times (RTs, in milliseconds) and error rates (as percentages), with standard errors, as a function of auditory stimulus type, compatibility, and visual-task perceptual load for Experiments 2 and 3
Full size table

RT benefits, costs, and compatibility effect analysis

As was the case for Experiment 1, the primary questions of interest in this study were whether auditory stimuli differ as a function of their compatibility with the responses, and in this experiment, with the categorization rule. Therefore, as in Experiment 1, we computed three new dependent variables for each combination of load and categorization rule for each participant: RT benefits, RT costs, and RT compatibility effects.

A series of t tests were conducted to determine whether RT benefits and costs were significantly greater than zero (indicating an influence of auditory stimulus compatibility relative to neutral), and to discover, among those effects that were significant, whether the benefits and costs differed in magnitude. First, RT benefits and costs were analyzed with a series of one-sample t tests, comparing them to a set value of zero. The means involved and the outcomes of these tests are presented in Table 4. Both RT benefits and costs were significantly greater than zero for all four category-rule conditions. Paired-samples t tests comparing the magnitudes of the benefits versus costs across the high-load conditions indicated that these means did not differ from each other, t(29) = 0.22, p = .830, but such a comparison across low-load conditions showed that costs were larger than benefits, t(29) = 3.328, p = .002. Note that the low-load benefits were actually negative, meaning that RTs were longer to compatible than to neutral auditory stimuli. In the arbitrary-rule conditions, no benefits or costs were significantly different from zero. These results indicate that auditory stimuli influenced processing as a function of compatibility in the category-rule conditions, but not in the arbitrary-rule conditions.

Table 4 Analysis of mean benefits and costs (RTs, in milliseconds), with standard errors, as a function of categorization rule and visual-task perceptual load, relative to a set value of 0, for Experiment 2
Full size table

Mean RT compatibility effects are found in Fig. 4. One-sample t tests indicated that RT compatibility effects were significantly greater than zero in the high-load category-rule condition, t(29) = 3.96, p < .001, but not in the low-load category-rule condition, t(29) = 1.57, p = .128. RT compatibility effects were not significantly greater than zero in the arbitrary-rule condition for either the high-load, t(29) = 1.48, p = .151, or low-load, t(29) = 1.04, p = .308, conditions.Footnote 2

Fig. 4
figure 4

Mean compatibility effects (RT) as a function of task rule (arbitrary rule vs. category rule) in Experiment 2 and as a function of auditory stimulus relationship (item-match vs. category-match) in Experiment 3. Compatibility effect = mean RT for incompatible trials minus mean RT for compatible trials. Error bars indicate ±1 SEM. *Significantly different than zero

Full size image

Error rate analysis

We performed a mixed-model ANOVA to assess the difference in error rates for each of the independent variables. These data are shown in Table 3. We found significantly higher error rates in the high-load (M = 14.77 %, SE = 0.93) than in the low-load (M = 6.61 %, SE = 0.62) condition, F(1, 58) = 170.54, MSE = 35.12, p < .001, η p 2 = .746. In addition, errors were significantly higher for incompatible (M = 11.61 %, SE = 0.86) than for compatible (M = 9.86 %, SE = 0.74) auditory stimuli, but neutral auditory stimuli (M = 10.61 %, SE = 0.77) did not differ from the other two types, F(2, 116) = 5.26, MSE = 17.68, p = .007, η p 2 = .083. These main effects for load and compatibility were subsumed in a significant Load × Compatibility interaction, F(2, 116) = 4.40, MSE = 16.15, p = .014, η p 2 = .71. Errors in the high-load condition were significantly higher for incompatible (M = 16.47 %, SE = 1.17) than for compatible (M = 13.17 %, SE = 0.93), but neither differed from neutral auditory stimuli (M = 14.68 %, SE = 1.06). Errors did not differ as a function of compatibility in the low-load condition. The final significant effect in this analysis was the Load × Rule interaction, F(1, 58) = 10.14, p = .002, η p 2 = .1.49. Error rates were higher in the arbitrary-rule, high-load condition (M = 17.18 %, SE = 1.31) than in the category-rule, high-load condition (M = 12.37 %, SE = 1.31), whereas errors were lower than the high-load conditions in both the arbitrary-rule and category-rule low-load conditions (M = 7.03 %, SE = 0.88, and M = 6.20 %, SE = 0.88, respectively).

Error rate compatibility effect analysis

Analyses were conducted on error rate compatibility effects, which were computed as in Experiment 1, for each combination of load and categorization rule. As was the case for the RT compatibility effect analysis, the only effect significantly greater than zero, as indicated by a one-sample t test, t(29) = 3.14, p = .004, was in the high-load, category-rule condition (M = 4.05, SE = 1.29). No other effects were significantly different from zero (all ps > .05).

Discussion

As predicted, a significant compatibility effect due to auditory stimuli was indicated in RT data in only the high-load, category-rule condition. Experiment 2 showed that for high-load visual target searches, when auditory stimuli were directly related to a superordinate category that carried response-discriminative information (category-rule condition), they influenced visual target processing to a greater extent than when a superordinate category name was related to the targets but was not response-discriminative (arbitrary-rule condition). This finding extends those of Tellinghuisen and Nowak (2003) that showed such an effect for foils that named the actual target items. No such effect was found in the arbitrary-rule condition, in which the category names did not exclusively coincide with one response or the other. This indicates that the relationship between the auditory stimulus and the visual target response is a critical determinant of whether to-be-ignored auditory stimuli influence processing. When auditory stimuli did not directly relate to a categorization rule, they did not significantly influence processing. The RT result pattern does not appear to be the result of speed–accuracy trade-offs, since error rates were high only in conditions in which RTs were also long. Finally, the patterns of results regarding costs and benefits differed between Experiments 1 and 2. Reasons for this discrepancy are explored in the General discussion.

Experiment 3

In Experiment 2, the magnitude of compatibility effects appeared to be considerably smaller in the high-load category-rule condition (32 ms) than when auditory stimuli named the actual target items in the high-load conditions of Tellinghuisen and Nowak (2003; approximately 80–120 ms), or in Experiment 1 of the present study (approximately 110 to 120 ms; see Fig. 2). In addition, significant RT compatibility effects were not found in the low-load, category-rule condition, although such an effect would be expected, on the basis of the results of Tellinghuisen and Nowak and this study’s Experiment 1. These differences in compatibility effect magnitudes may have been due to the degree of overlap between the representations of the auditory and visual information. In Experiment 1, as well as for Tellinghuisen and Nowak, the to-be-ignored stimuli named the actual identity of the targets, thus carrying very direct response-discriminative information.

Do auditory stimuli that name actual items in the target search array (item-match stimuli) have a greater effect on target processing than do auditory stimuli that name the category to which a target belongs on the basis of a categorization rule (category-match stimuli)? Following the logic of Lleras et al. (2013; see also Buetti et al., 2014), we hypothesized that stimuli that matched the item presented would be more effective foils than stimuli that matched the category to which the visual stimulus belonged, because they would be more closely related to the target stimuli, directly mapping onto the pronounced names of the targets. We predicted that participants would show larger compatibility effects when presented with to-be-ignored auditory stimuli that named an item of a target search set (item-match stimuli) than when such stimuli named a category to which a target item belonged (category-match stimuli). A within-subjects design was utilized in this study to allow for direct comparisons between item-match and category-match stimulus conditions.

Method

Participants

We sought to test a minimum of 30 participants. Thirty-four participants were recruited and participated in exchange for partial fulfillment of an introductory psychology course credit. Of these, three were excluded from the analysis for having accuracy rates below 60 % in some experimental conditions. The mean age of the final sample of 31 was 18.5 years (SD = 0.8 years), which comprised 23 females and eight males. All participants had normal or corrected-to-normal visual acuity and reported having normal hearing.

Stimuli and apparatus

The apparatus was identical to those of Experiments 1 and 2, and the stimuli were similar to those in Experiment 2 (see Fig. 3). The visual stimuli differed, in that the set of target stimuli consisted of only one letter (X) or one number (4). The auditory stimuli also differed, in that they either named the category to which a target might belong (“letter” or “number”) or named the target item itself (“X” or “4”). Only one letter and one number were used, making the mappings between auditory stimuli and the number of possible targets equal for both types of auditory stimuli. Unlike in Experiment 2, neutral auditory stimuli (white noise) were not included, because doing so would not allow for a fully within-subjects, balanced design, as the neutral condition would be the same for both the category and item conditions. In addition, including neutral conditions would have greatly increased the number of trials to which participants would have had to respond. All auditory stimuli were 300 ms in duration and were presented at approximately 70 dB.

Design and procedure

This study was based on a fully within-subjects 2 × 2 × 2 design. The independent variables in this experiment were perceptual load (low or high), auditory stimulus relationship (category-match or item-match), and auditory stimulus compatibility (compatible or incompatible). All variables occurred equally across participants, and all conditions were randomized. The dependent variables were mean RTs (latency of response in milliseconds) and accuracy/error rate.

The high- and low-perceptual-load conditions were the same as in Experiment 2. The category auditory stimuli were either the word “letter” or “number.” The item auditory stimuli named either the letter “X” or the number “4.” Compatible auditory stimuli named the same category or item to which the target belonged (i.e., “letter” or “X” if the target was X; “number” or “4” if the target was 4). Incompatible auditory stimuli named the opposing target category or item (i.e., “letter” or “X” if the target was 4; “number” or “4” if the target was X).

The sequence of each trial, the experimental setting, and viewing distance were the same as in Experiment 2. Participants rested their left index finger on one computer key and their right index finger on another key, and were instructed to press the right or the left key to indicate which target was presented. Participants were instructed to press their left-hand key if an X was presented in the visual array, and to press the right-hand key if a 4 was presented. Speed and accuracy were both emphasized.

Participants first completed a practice block of 24 trials, consisting of trials randomly selected from the possible combinations of conditions used in the experiment. Following practice, participants completed two experimental blocks of 144 trials each (288 trials total), or 36 trials per condition. Short breaks were given following practice and between the experimental blocks.

Results

Mean correct RTs and percentage error rates were computed for each participant as a function of perceptual load (low or high), auditory stimulus relationship (category-match or item-match), and auditory stimulus compatibility (compatible or incompatible). RTs below 125 ms or above 2 s, as well as trials with no responses, were counted as misses. Such trials were rare (less than 0.29 % of all trials; 0.29 % of low-load and 0.29 % of high-load trials). Percentage error rates were based on nonmiss trials.

Reaction time analysis

We performed a repeated measures ANOVA to assess differences in RTs for each of the independent variables. These data are shown in Table 3. Significant main effects were found for load, F(1, 30) = 128.62, MSE = 2,543.37, p < .001, η p 2 = .811, and auditory stimulus compatibility, F(1, 30) = 28.71, MSE = 664.65, p < .001, η p 2 = .489. These main effects were subsumed in a significant Load × Auditory Stimulus Compatibility interaction, F(1, 30) = 11.144, MSE = 724.77, p < .002, η p 2 = .271. We also observed a two-way interaction between auditory stimulus relationship and auditory stimulus compatibility, F(1, 30) = 7.425, MSE = 951.97 p < .011, η p 2 = .198. This interaction indicates that the effects of compatibility varied as a function of both load and auditory stimulus relationship, and is further explored below in the RT compatibility effect analysis. The Load × Auditory Stimulus Relationship × Auditory Stimulus Compatibility interaction was not significant, F(1, 30) = 0.452, MSE = 681.77, p = .507, η p 2 = .015.

RT compatibility effect analysis

As was the case for Experiments 1 and 2, the primary questions of interest in this study were whether the auditory stimulus effects differed as a function of their compatibility with the responses, and in this experiment, with the auditory stimulus relationship. Therefore, as in previous experiments, we computed RT compatibility effects. The means involved in this analysis are shown in Fig. 4. A series of one-sample t tests was conducted on the RT compatibility effects for each combination of load and auditory stimulus relationship. Compatibility effects were significantly greater than zero for item-matched auditory stimuli in both the low-load, t(30) = 2.63, p = .013, (M = 14, SE = 5.59), and high-load, t(30) = 4.82, p < .001 (M = 42, SE = 8.7), search conditions, and a paired-samples t test indicated that these means differed significantly, t(30) = 2.797, p = .009. Marginally significant nonzero effects for category-matched auditory stimuli occurred under high-load searches, t(30) = 2.02, p = .052 (M = 16, SE = 8.00), but not under low-load searches, t(30) = 0.54, p = .596 (M = –2, SE = 4.53). Paired-samples t tests indicated that the compatibility effects in high-load searches were greater for item-matched than for category-matched stimuli, t(30) = 2.109, p = .043, but the means for item-matched stimuli in low-load searches did not differ from those for category-matched stimuli in high-load searches, t(30) = 0.131, p = .896.Footnote 3

Error rate analysis

We performed a mixed-model ANOVA to assess the differences in error rates for each of the independent variables. The means involved in this analysis are shown in Table 3. We found significantly higher error rates in the high-load (M = 8.74 %, SE = 0.89) than in the low-load (M = 6.10 %, SE = 0.70) condition, F(1, 30) = 9.04, MSE = 47.97, p = .005, η p 2 = .232. Neither the main effect of auditory stimulus relationship nor that of auditory stimulus compatibility was significant. The Load × Auditory Stimulus Compatibility interaction was significant, F(1, 30) = 5.53, MSE = 10.07, p = .025, η p 2 = .156. Error rates were higher in the high-load, incompatible (M = 9.54 %, SE = 1.00) than in the high-load, compatible (M = 7.95 %, SE = 0.97) condition, but they did not differ as a function of compatibility in the low-load condition. No other interactions were significant.

Error rate compatibility effect analysis

Analyses were conducted on error rate compatibility effects, which were computed as in Experiments 1 and 2, for each combination of load and auditory stimulus relationship. None of these effects significantly differed from zero, as indexed by one-sample t tests (all ps > .05).

Discussion

Taken together, these results support our prediction that to-be-ignored auditory stimuli that more closely share characteristics of the targets would have a greater influence on RTs to identify the targets, particularly in high-load conditions. Auditory stimuli that were better foils, in the sense of more proximally naming a target, had a greater impact on RTs than those that only named the category to which a target belonged. The results of this experiment replicated the findings of Tellinghuisen and Nowak (2003) that auditory distractors yielded larger compatibility effects in high-load than in low-load search conditions when the auditory stimuli named items from the target set. This experiment also replicated the findings of Experiment 2 of the present study by showing that category-related auditory stimuli influenced target searches, but only in high-load search conditions. Finally, as in Experiments 1 and 2, the RT result pattern did not indicate any speed–accuracy trade-offs being present, since longer RTs were found in conditions in which error rates were higher.

The Experiment 3 compatibility effects were significant for item-match auditory stimuli in both low-load and high-load searches, albeit still not at the expected levels relative to past studies. It is important to note the methodological differences in these studies that may have impacted the compatibility effect magnitudes. Experiment 3 differed from Experiment 1 and from Tellinghuisen and Nowak (2003) in that there was more variation in the strength of matching between the target and the to-be-ignored stimuli. The category-matched auditory stimuli used in Experiment 3 had less impact on visual task performance, and may have effectively reduced the strength of the mapping for all auditory stimuli, because half of the stimuli were less task-relevant. In Experiment 1 and Tellinghuisen and Nowak’s study, two thirds of the auditory stimuli were item-matched (with the remaining one third being neutral), perhaps yielding a higher strength of the mapping between auditory and visual stimuli. In addition, the outcomes of Experiments 2 and 3 must also be compared with caution, due to the methodological differences between those experiments. To make the mappings between the auditory stimulus relationships and targets equal for both item-match and category-match stimuli in Experiment 3, we limited the number of potential targets related to each auditory stimulus: Only the target X mapped to the word “letter” or “X,” and only the target 4 mapped to the word “number” or “4.” If two letters and two numbers had been used, it would have necessitated two item-match auditory stimuli each for numbers and for letters, yet there would only be one category-match stimulus each for both numbers and both letters. The number items that counted as valid targets was thus smaller in Experiment 3 than in Experiment 2, since each response was mapped to only one target possibility rather than two, although identical nontarget items were used in both experiments. Future research might further explore the relationship between target set size and the number of possible auditory stimuli that map onto those targets.

General discussion

The present set of experiments clearly demonstrate that to-be-ignored auditory distractors are processed when they map onto visual stimulus characteristics during high-perceptual-load visual target searches. These findings are contrary to those of studies suggesting that high-perceptual-load searches lead to inattentional deafness (e.g., Macdonald & Lavie, 2011; Raveh & Lavie, 2015). Experiment 1 addressed a concern that in Tellinghuisen and Nowak’s (2003) study, the larger auditory stimulus compatibility effects in high-load searches occurred due to those stimuli being processed for a longer time than in low-load searches (Lavie, 2005). We observed large compatibility effects when the auditory stimulus onset occurred prior to and ended simultaneously with the visual target search display, so that the timeframe for auditory stimulus processing should have been the same for both levels of perceptual load. These effects were not significantly different from those conditions under which the auditory stimulus onset was simultaneous with the search display and ended after the search display offset, as had been the method in Tellinghuisen and Nowak. In Experiments 2 and 3, we explored a possible cause of this effect in high-load searches in a crossmodal paradigm by manipulating the relationship between visual target search items and to-be-ignored auditory stimuli. Experiment 2 showed that when auditory stimuli named a category assigned exclusively to one response, significant compatibility effects were found for high-load, but not for low-load, searches. Conversely, when auditory stimuli named the category for the targets in identical searches, but the categories were arbitrarily mixed across responses, no significant RT compatibility effects were found. Experiment 3 extended this finding by showing that in high-load search conditions, to-be-ignored auditory stimuli that named target items yielded significantly larger compatibility effects than did those that named the category of the targets. These results have implications for our understanding of how auditory stimuli impact visual processing. When auditory stimuli map onto how visual items are prioritized, they will influence processing. Alternatively, when auditory stimuli have a limited relationship to the task requirements for categorizing visual items, those stimuli appear to be largely ignored.

No inattentional deafness for foils

The experiments in this article demonstrate that under high-load search conditions, inattentional deafness did not occur for auditory stimuli related to target response categorization. By contrast, in Macdonald and Lavie (2011) and Raveh and Lavie (2015), inattentional deafness was apparent under high-load search conditions, as participants were less likely to report hearing an extraneous tone presented during a visual target classification task. In those studies, however, participants were directly instructed to respond to auditory stimuli. Whereas Macdonald and Lavie asked participants to report the presence of a single, unexpected auditory stimulus presented on a final trial, Raveh and Lavie’s study differed from a standard inattentional blindness/inattentional deafness paradigm in that participants were instructed to report about the auditory stimuli on each trial (perhaps a dual-task situation). In the present study, the task requirements were different, since participants were instructed to ignore all auditory stimuli. Therefore, the present findings do not contradict the results of either of those previous studies. An objection might be raised, however, to the inclusion of a beep to signal the beginning of a trial in the present studies. It is possible that this stimulus could have resulted in orienting toward auditory stimuli that may not have occurred in Lavie’s inattentional deafness paradigm, thus resulting in greater processing of all auditory stimuli. However, in Raveh and Lavie’s study, participants were directed to attend to auditory stimuli on each trial, yet failed to note those stimuli under high-load conditions. This finding, along with the fact that auditory stimulus processing varied as a function of compatibility in the present experiments, leads us to believe that the inclusion of the beep did not influence processing of the auditory stimuli presented with the target search arrays. Inclusion of the beep replicated the methodology used in Lavie’s previous research (e.g., Lavie & Cox, 1997) and by Tellinghuisen and Nowak (2003). We propose that auditory stimuli that are task-unrelated (i.e., neutral) likely do not impact processing, although the experiments in this article do not directly address this assumption. However, our studies do demonstrate that to-be-ignored auditory stimuli with response-defining attributes modulate visual search processing, particularly as a function of perceptual load. Whether inattentional deafness occurs seems to critically depend on the relationship between the auditory stimuli present and the requirements of the visual task.

Top-down attentional settings appear to be a key determinant of whether stimuli are processed as task-related, even if they are clearly not part of the target search set. When performing a visual search task, the attentional system prioritizes items that are among those the participant is seeking to identify, on the basis of the particular task’s demands, which are set by task instructions (see Buetti et al., 2014). Foils meet most of the criteria for targets, but differ on at least one key dimension, thus making them related to the task, but not valid targets. In the present studies, conditions were included so that auditory stimuli mapped onto visual target responses, thus making them foils (see Lleras et al., 2013). Consistently robust compatibility effects were found when this was the case in high-load searches. Task-irrelevant stimuli such as the task-unrelated tones used in Macdonald and Lavie (2011) may be ignored because tones are not within the set of stimuli to be detected a priori by attention. Because there would be no reason for an efficient attentional system to select items outside of the attentional set, such items go unnoticed.

The impact of to-be-ignored stimuli is typically framed as distraction effects, in light of the differential effects of such stimuli based on their compatibility with the target search task, such as when those stimuli are neutral versus compatible or incompatible (e.g., Lavie & Cox, 1997; Tellinghuisen & Nowak, 2003). Relative to the findings of Macdonald and Lavie (2011) and Raveh and Lavie (2015), it may be clearer to discuss the present phenomenon as attention being captured by selecting a foil (see Buetti et al., 2014; Lleras et al., 2013) or at least distraction effects due to foils. In this conceptualization, as Lleras et al. have stated, processing target-related, but to-be-ignored stimuli can be seen as a success of the attentional system. It “found” information that met most of the criteria set by top-down processes and in naturalistic settings, this would likely be helpful in finding an actual target. When that target-related information does not correspond with an actual target, further top-down processing would be necessary to reorient attention to a different item/response or to continue searching.

The impact of top-down settings was shown in Experiments 2 and 3 to extend to naming superordinate categories to which the stimuli belonged. To our knowledge, this has been the first study to show crossmodal capture of attention by naming superordinate categories in a visual search task. Wyble, Folk, and Potter (2013) showed that, when participants were instructed to identify an exemplar of a particular category presented in a rapid serial visual presentation (RSVP) stream, a peripherally presented image of a different exemplar from the target category negatively impacted the ability to identify the RSVP-stream-presented target. Wyble et al.’s findings extended the contingent-attentional-capture theory (Folk & Remington, 1998; Folk et al., 1992, 1993) to show that attention can be configured to prioritize the processing of category-related stimuli, resulting in rapid shifts of attentional resources to items that fit that concept. Our Experiment 2 and 3 results showed that in high-load searches, hearing the name of a category influenced processing of the category member targets that mapped to category-specific responses (e.g., letters for one response, numbers for another response). Experiment 2, adapting Reisenauer and Dreisbach’s (2013) method of manipulating task rules to vary the mappings between targets and responses, showed that when a superordinate category name applied to both items mapped to one response (category-rule condition), hearing the name of that category influenced processing. Such an influence was not apparent when the superordinate category names were not exclusive to one response or the other (arbitrary-rule condition). Experiment 3 demonstrated that the influence of an auditory stimulus that named a category was less than that of a stimulus that named the actual identity of the target. This finding may have been due to the item name serving as a more direct foil than did the category name.

Greater influence of crossmodal stimuli under high load

In replicating Tellinghuisen and Nowak (2003), and in light of Mahr and Wentura’s (2014) more recent study, the present findings call into question the applicability of load theory crossmodally. The findings from the visual modality that contradict load theory may parallel how task-related auditory distractors bring about large compatibility effects during high-perceptual-load visual searches. High perceptual load does not necessarily lead to ignoring nontask stimuli under certain conditions if those stimuli were given high priority by participants. Some nontarget visual stimuli have been shown to be prioritized for processing even while performing high-load tasks, and may draw on participant-specific backgrounds. Lavie, Ro, and Russell (2003) and He and Chen (2010) both showed that RTs to categorize centrally presented words in high-load searches were influenced by peripherally presented faces that were strongly related to the words. In addition, Ro, Friggel, and Lavie (2009), using a task similar to Lavie et al. (2003), demonstrated that prior experience can impact attention control settings when processing nonface visual stimuli. When expert musicians classified centrally presented words as naming one of two types of musical instruments presented in high-load searches, their RTs were influenced by peripherally presented pictures of musical instruments that varied in congruity with the target word. No such effect was found for nonmusicians.

Auditory stimuli in general may receive high processing priority. The auditory system has been depicted as an early-warning system (Dalton & Lavie, 2004), and auditory singletons have been shown to capture attention during nonspatial visual search (Dalton & Spence, 2007). Not all auditory information captures attention. Task sets play a role in determining the effects of auditory stimuli. Auditory attention capture can be modified by top-down expectations such that auditory stimuli that do not fit current attentional settings do not capture attention (Dalton & Lavie, 2007). Indeed, in the present study, task rules played a determining role in which stimuli captured attention. Dalton and Hughes (2014), in their review of research on the influence of auditory stimuli on processing information in a variety of paradigms, concluded that top-down influences clearly modulate the strength of attentional capture.

Attention in the auditory system has also been shown to function in some ways differently than in the visual system in tasks in which the perceptual load of auditory stimuli was manipulated. Gomes, Barrett, Duff, Barnhardt, and Ritter (2008) manipulated perceptual load by varying interstimulus intervals (ISIs), measuring both behavioral and electrophysiological responses. When ISIs were more brief (i.e., a higher perceptual load), processing of irrelevant stimuli (characterized by ERPs) was greater than in lower load conditions, a pattern contrary to load theory. Murphy, Fraenkel, and Dalton (2013) also demonstrated that load theory did not hold for auditory stimuli. Participants were to detect a target letter spoken within a stream of letters that varied in perceptual load, presented from a centrally positioned speaker. Congruency effects due to peripherally presented auditory distractor letters were found for both low-load and high-load conditions.

A high processing load for stimuli in one modality may be related to enhanced processing of another modality. Jacoby, Hall, and Mattingly (2012) found that when participants performed an auditory high-perceptual-load task, they were more likely to respond to and had greater steady-state evoked potentials to centrally presented, irrelevant visual stimuli than when performing an auditory low perceptual load task. Interestingly, in Jacoby et al.’s study, the visual stimuli in no way mapped onto the auditory task. The results of our experiments also suggest that the relationship between auditory stimuli is critical in determining if they will impact processing of a high visual perceptual load task. This may not be the case for auditory perceptual load tasks, however. Murphy et al. (2013) suggested that the auditory system differs in its capacity limitations, relative to those of the visual system. It is also possible that processing any salient visual stimulus is prioritized over auditory stimuli, as this would fit with the results of Jacoby et al. These possibilities merit further exploration.

Research from other tasks has indicated the potential impact of crossmodal information on attention processing. Several studies have shown that multisensory cues (e.g., both auditory and tactile) effectively direct attention even during high task loads (e.g., Ho, Santangelo, & Spence, 2009; Santangelo, Ho, & Spence, 2008; Santangelo & Spence, 2007). If attention particularly prioritizes auditory, task-related information during high-load visual tasks, perhaps designing auditory warning stimuli that map onto the identities of target search items might serve as effective alerts to important visual information. Evidence in support of this notion has been found in a task in which airplane pilots were to detect an auditory alarm under wind-shear situations in a flight simulator. Their ability to detect such warnings improved if pilots were trained on the sound of that alarm; others without such training demonstrated high rates of inattentional deafness (Dehais et al., 2014). When attention priorities were changed through instructions and training, crossmodal detection improved.

Regarding why high-perceptual-load tasks in one modality would lead to greater processing in another modality, Tellinghuisen and Nowak (2003) as well as Jacoby et al. (2012) hypothesized that processing higher-perceptual-load stimuli in one modality may reduce the capacity to inhibit the processing of stimuli from another modality. The present results also fit this hypothesis, with the important qualification that for this effect to occur as a result of visual perceptual load, stimuli from the auditory modality must map onto the response categories for the visual targets. In addition, in terms of anatomy, auditory and visual information have some degree of separate processing. Torralbo and Beck (2008) have suggested that some visual perceptual-load effects may occur because, in high-load searches, there is greater competition for perceptual representation of stimuli in the visual cortex. In such cases, to resolve which search item is a target, stronger top-down biasing is exerted toward processing stimuli in the spatial area in which targets might occur. This results in processing being constrained to the cortical area representing those stimuli, while not allowing for processing of stimuli represented in other visual cortical areas. In low-load searches, such biasing would be weaker, such that visual cortical resources could also be used to process flanking stimuli. This spatial processing bias may, however, be altered or even overridden by stimuli prioritized by attention, such as pictures of faces (e.g., Lavie et al., 2003) or highly meaningful objects (e.g., Ro et al., 2009). This spatial processing bias might also be altered by stimuli from another modality that are prioritized by attention. In the present study, auditory stimuli, by virtue of not being visual, would necessarily be represented outside of the visual cortex, allowing for these auditory stimuli to be processed well. Perhaps if the visual stimuli being processed utilize a great deal of visual processing capacity (i.e., high-load searches), the processing bias shifts and task-related stimuli from another modality are prioritized. The combination of (1) reduced capacity to inhibit the processing of auditory stimuli during a high-load visual task and (2) a shift to prioritize processing of auditory, task-related stimuli may account for our findings.

Auditory effects due to interference or facilitation?

Whether auditory stimuli act more to enhance or to interfere with processing visual stimuli remains yet to be fully determined. Some studies have shown that, when compared to neutral auditory stimuli that name either nontarget letters in a letter search task (Tellinghuisen & Nowak, 2003, Exp. 1) or nontarget color word names in a Stroop-like color-matching task (Mahr & Wentura, 2014, Exps. 1 and 2), target-compatible auditory stimuli yield faster RTs to name targets. In those studies, incompatible auditory stimuli appeared to have little impact on target processing. However, other studies that have used either silence (Tellinghuisen & Nowak, 2003, Exp. 2) or white noise (Tellinghuisen & Nowak, 2003, Exp. 3) as a neutral baseline condition found reduced RTs for target-compatible distractors, but longer RTs for target-incompatible distractors. This pattern suggests that participants’ performance was impacted by a variety of auditory stimuli, resulting in either facilitation or interference, depending on compatibility. In those experiments, nontarget set words or letters, rather than having no influence on performance, may have acted as foils to some degree. Such stimuli, by belonging to the same superordinate category as the targets (i.e., being words or letters), may have slowed responding. That is, the stimuli designated as neutral might have effectively been similar to the incompatible auditory stimuli. This would result in what appeared to be little to no interference by “wrong” (incompatible) relative to “neutral” auditory stimuli.

In the present study’s Experiment 1, with the neutral stimulus being the letter L, the facilitation for compatible auditory stimuli (RT benefits) and interference for incompatible auditory stimuli (RT costs) were both large, but in low-load conditions, only RT benefits were obtained. Interestingly, in the category-rule condition of Experiment 2, in which the auditory stimulus compatibility influenced processing (with white noise as neutral), significant RT costs were found regardless of load, but benefits were obtained only in the high-load condition, whereas in the low-load condition, neutral trials actually yielded the shortest RTs. These findings, along with Tellinghuisen and Nowak’s (2003) Experiments 2 and 3, do not suggest a clear pattern of facilitation over interference, and demonstrate the need for a better understanding of what a “neutral” stimulus is. Experiment 3 of the present study was not designed to address this question, so more research will be needed to systematically address the degree to which to-be-ignored auditory stimuli result in more facilitation or interference, relative to neutral stimuli, in this paradigm. For future studies, a variety of neutral stimuli might be used, where truly neutral stimuli would be completely unrelated to the visual stimuli, not foils (see Lleras et al., 2013).

Conclusion

Our results indicate that crossmodal attention can be flexibly allocated when the stimuli that receive prioritized processing depend on top-down factors such as the instructions regarding what is important to detect within a task. In addition, when a task-related stimulus is presented auditorily during a high-load visual target search, that information strongly impacts the processing of visual targets. This suggests that the auditory information is responded to quickly and is prioritized even if it is not part of the search task. If, however, that information is not task-related, it does not impact processing. Finally, the more closely that auditory stimuli map onto the target search set, the more strongly they will impact processing. Further research will be needed to more fully explore both the basic and applied conditions under which auditory stimuli may have unintended consequences on visual tasks, by virtue of their relationship to the visual task being performed.