Temporal organization of color and shape processing during visual search

Glavan, Joseph J.; Haggit, Jordan M.; Houpt, Joseph W.

doi:10.3758/s13414-019-01863-9

Temporal organization of color and shape processing during visual search

40 Years of Feature Integration: Special Issue in Memory of Anne Treisman
Published: 04 March 2020

Volume 82, pages 426–456, (2020)
Cite this article

Download PDF

Attention, Perception, & Psychophysics Aims and scope Submit manuscript

Temporal organization of color and shape processing during visual search

Download PDF

Joseph J. Glavan¹,
Jordan M. Haggit² &
Joseph W. Houpt¹

1744 Accesses
4 Citations
Explore all metrics

Abstract

The mechanisms guiding visual attention are of great interest within cognitive and perceptual psychology. Many researchers have proposed models of these mechanisms, which serve to both formalize their theories and to guide further empirical investigations. The assumption that a number of basic features are processed in parallel early in the attentional process is common among most models of visual attention and visual search. To date, much of the evidence for parallel processing has been limited to set-size manipulations. Unfortunately, set-size manipulations have been shown to be insufficient evidence for parallel processing. We applied Systems Factorial Technology, a general nonparametric framework, to test this assumption, specifically whether color and shape are processed in parallel or in serial, in three experiments representative of feature search, conjunctive search, and odd-one-out search, respectively. Our results provide strong evidence that color and shape information guides search through parallel processes. Furthermore, we found evidence for facilitation between color and shape when the target was known in advance but performance consistent with unlimited capacity, independent parallel processing in odd-one-out search. These results confirm core assumptions about color and shape feature processing instantiated in most models of visual search and provide more detailed clues about the manner in which color and shape information is combined to guide search.

A theoretical attempt to revive the serial/parallel-search dichotomy

Article 18 July 2019

Serial vs. parallel models of attention in visual search: accounting for benchmark RT-distributions

Article 03 December 2015

A target contrast signal theory of parallel processing in goal-directed search

Article 05 February 2020

Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Psychologists have long used the visual search paradigm in the laboratory to examine the underlying mechanisms of visual attention. In visual search tasks, a subject looks for a specific target object among a number of non-target objects (i.e., distractors). The guiding features of visual attention are inferred from the way target and distractor relationships impact accuracy or search time. Using this methodology, psychologists have put forth a number of basic visual features that are believed to be processed early, automatically, and in parallel across the visual scene (color, shape, size, luminance, etc.; see Wolfe & Horowitz, 2004, 2017). They are considered the primary drivers of attentional control in early vision and are instantiated in many prominent models of visual search (e.g., Wolfe 2007; Treisman & Gelade, 1980; Bundesen, 1990; Itti & Koch, 2001; Logan, 1996).

For example, in Feature Integration Theory (FIT; Treisman & Gelade, 1980) each feature dimension is processed in parallel by its own salience map. If the target cannot be detected by using these feature maps alone, then attention is deployed serially to each item in the scene. By invoking attention, the item’s features are bound together into a coherent object, which can then be compared to the target template. Wolfe’s (2007) Guided Search (GS) model shares many of the assumptions of Feature Integration Theory but allows for processing in the feature maps to influence where attention is directed in the binding stage (Wolfe et al., 1989). The activations of the individual feature maps are combined by weighted sum into a master salience map, and the location with the highest combined activation within this master salience map becomes the next most likely destination for attention. Items receiving very little activation in the master map are unlikely to and even may never be attended (Chun & Wolfe, 1996).

In contrast to serial two-stage models, the CODE theory of visual attention (CTVA) (Bundesen, 1990; Logan, 1996; Bundesen, 1998) proposes that all items and features within the focus of attention are processed in parallel, and attention can be concentrated onto a single item or expanded to include the entire display. CTVA models the identification of an item as an independent parallel race based on sensory evidence, attentional weight, and decision bias. Once an element wins its race, it is encoded into visual short-term memory (VSTM). The focus of attention can be moved around to different elements in VSTM, which themselves can also be perceptual groups. In this way, FIT, GS, and CTVA all allow for serial shifting of attention between items or groups of items in the display, but CTVA does not require an additional feature binding stage.

Although there are a variety of theories to explain visual search, many are based on three kinds of visual search experiments. Feature search, which we investigate in Experiment 1, is where the target can be discriminated from distractors by a single feature. This type of search is often thought to utilize parallel processing because visual attention tends to be directed towards the target in a spatially invariant manner and independent of the number of non-target objects in the scene. “Pop-out” search, where the target differs from the distractors to such a degree that its location becomes immediately apparent, is an extreme case of feature search. Conjunctive search, which we investigate in Experiment 2, is where the target must be discriminated from distractors by more than one feature. This type of search is often described as requiring serial processing of items (Treisman & Gelade, 1980) because attention must be deployed in a more deliberate manner, typically to multiple spatial locations in order to find the target. Search times are generally longer and dependent on the number and spatial layout of the distractors in the display. In contrast to the first two experiments, in which participants know the target identity in advance, in Experiment 3 we examine odd-one-out search, a special case of feature search where the target is defined solely by being different from the distractors (see Müller et al.,, 1995).

The aforementioned models make predictions about how the presence of multiple features guide attention; however, a significant source of evidence for their assumptions about the order in which individual features are processed has come under scrutiny (Wolfe, 1998; Townsend & Nozawa, 1995; Little et al., 2017; Kristjánsson, 2015; cf. Wolfe, 2016). A common method for determining whether processing occurs in serial or parallel is to examine mean RTs as a function of the number of items to process (set size; Sternberg, 1966). The search slope, resulting from regressing mean RT on set-size, is traditionally interpreted as indicating serial processing when it is positive such that search times increase with the number of items to process. Conversely, a flat function, where search times do not significantly increase with the number of items to process, is typically interpreted as indicating parallel processing.^{Footnote 1}Treisman and Gelade (1980) used this reasoning when they concluded that items are processed in parallel during feature search and in serial during conjunctive search. However, Townsend and others (Townsend, 1972; Townsend & Ashby, 1983; Townsend & Nozawa, 1995; Little et al., 2017) have demonstrated that this approach is inadequate because it confounds the temporal organization (e.g., serial or parallel) of the system with its workload capacity (i.e., how efficiently each item is processed as more are added). Because the throughput of the system may change with respect to the number of items to process, it is possible for some serial models to mimic parallel models and vice-versa. For example, while the traditional interpretations of search slopes hold so long as the processing rate of each item is independent of the number of items to process, if processing was to become less efficient with increased set-size, perhaps because of limits on attention, then a parallel model could produce positive search slopes. Likewise, if efficiency was to increase with set-size, perhaps because of redundancy gains, then a serial model could produce the flat search slopes traditionally associated with parallel models. In the next section, we will describe the Systems Factorial Technology (SFT; Townsend & Nozawa, 1995, Houpt et al.,, 2014) framework, which was designed to control this potential confound with workload.

The preceding research on visual search has largely considered whether search proceeds in parallel or serial at the item-level (i.e., whether objects A and B are compared to the target template simultaneously or first object A is considered before object B is considered). Alternatively, if we treat the items as a collection of features, we may instead ask how the feature dimensions are processed with respect to each other. For example, the stimuli in our study are defined in terms of their color and shape. In a truly parallel search, both feature dimensions would be used simultaneously to direct attention toward objects that are similar to the target. However, it may be the case that feature dimensions are processed in serial such that first a particular region of the display that contains color information similar to the target’s color is determined, and then attention is directed toward objects in that region that are of similar shape to the target. In our study, we leverage the power of SFT to determine the temporal organization of color and shape feature dimension processing.

While some models have assumed serial processing of feature dimensions (e.g., Treisman, 1988; Grossberg et al.,, 1994), a larger body of work has argued for parallel processing of feature dimensions. Many existing theories assume that features are processed in parallel at least at some stage of the visual search process. These theories are usually concerned with feature processing in general but often use color and shape as dimensions of interest. In FIT, the assumption of parallel feature processing is embodied by the simultaneous computation of separate feature maps. GS shares this assumption and even allows features to exert a parallel influence on the deployment of attention. In support of this, Findlay (1997) found evidence suggesting that the first saccade during both simple feature search and conjunction search is controlled by parallel processing of the stimuli features. Bundesen et al., (2003) and Kyllingsbæk and Bundesen (2007) found accuracy-based evidence in favor of parallel processing for color and shape, and Müller et al., (1995) argue that parallel processing is more plausible than serial processing of feature dimensions. Mordkoff and Yantis (1993) and Krummenacher et al., (2001) have also documented violations of Miller’s (1982) race model inequality for targets defined by redundant color-shape and color-orientation dimensions, respectively, suggesting coactive processing of these feature dimensions.

Based on the above research, we hypothesize that feature dimensions are processed in parallel during visual search. Although this is hardly a new proposal, we argue that the opportunity exists to provide more direct evidence given the potential for model mimicry when workload is not controlled (Townsend, 1972; Little et al., 2017). To this end, we conducted three visual search experiments where feature salience, rather than the number of distractors, is manipulated. Our analysis exploits a model-based statistical framework, detailed next, in order to infer the temporal organization of the visual search system without confounding workload.

Systems Factorial Technology

Systems Factorial Technology (SFT; Townsend and Nozawa, 1995) is a nonparametric, model-based, statistical framework for categorizing cognitive systems consisting of multiple information processing channels, which enables it to provide high-level constraints on more specific parametric models. It assumes that the cognitive system is an information processing black box that takes two or more channels as inputs and returns a single (usually binary) response after some time as output. In our study, the color and shape information in the display are the inputs, and the output is whether the target is present or absent. Inside the black box, processing of each channel terminates in a respective decision for that channel, and then the channel-specific decisions are combined via some logic gate to produce the response. It is often convenient to use an evidence accumulation metaphor to describe the channel processes, and “and” or “or” rules are commonly used to describe the logic gate. The bottom row of Fig. 1 provides example diagrams of this general model. Five models based on different combinations of processing are illustrated along with their corresponding SFT signatures. Details on the models and the signatures are given below.

SFT addresses four important qualitative properties of the general model: architecture, stopping rule, stochastic independence, and workload capacity. Architecture refers to the temporal organization of a system. Do competing processes take place simultaneously, as in a parallel system; or do they take place consecutively, as in a serial system? The stopping rule of a system, the point at which processing terminates and a response is made, is also important for classifying the temporal organization of an information processing system. Processing may terminate before all channels have finished, including cases when processing terminates as soon as any individual channel has finished (first-terminating, i.e., “OR” rule), or when all channels have finished (exhaustive or “AND” rule). Stochastic dependence refers to the statistical relationship among processing times of channels. Dependence can arise in many ways, perhaps the most relevant of which are facilitatory or inhibitory interactions among processes. Workload capacity refers to the change in throughput of the cognitive system as additional sources (channels) of information are added. It is the efficiency of processing all channels relative to each in isolation.

It is important to note that these four properties of the general model are orthogonal, and any relevant system can be characterized by its position in this four-dimensional space. Typically, we consider architecture and stopping rule to be binary^{Footnote 2} (either serial or parallel, and first-terminating or exhaustive, respectively), and the four possible combinations of each, assuming stochastic independence, are shown in the first four columns of Fig. 1. Stochastic independence may vary anywhere from fully inhibitory to completely independent to fully facilitatory, but we are usually only interested in the categorical distinction. Likewise, workload capacity can be treated as a continuous dimension, but we are usually more interested in whether processing both information channels together is less, equally, or more efficient than processing either alone. Townsend and Nozawa (1995) use the terms limited capacity, unlimited capacity, and super capacity, respectively, to describe these levels of processing efficiency. A special case of parallel processing, the coactive architecture, is depicted in the last column of Fig. 1. In such a system, the processing channels pool their information together into a single collective decision. Consequently, these systems often exhibit super capacity. Later in the paper when we evaluate our results with respect to SFT, we identify which of these five general classes of models best describes the data.

Three statistics identify basic distinctions amongst the properties of architecture, stopping rule, stochastic independence, and workload capacity. Two require selective influence of each information processing channel such that the rate of processing of one dimension (e.g., color) can be accelerated without affecting the rate of processing of the other dimension (e.g., shape). In perceptual experiments, this is commonly achieved through feature salience manipulations (Houpt et al., 2014). Importantly, because perceptual salience changes the rate of information accumulation without changing the number of items to process, such a manipulation does not confound workload (as in the search slope approach). The third statistic only requires that each source of information be able to be presented in the absence of the other source(s). The necessary levels of relative salience for all three statistics (H igh, L ow, and A bsent) can be combined into a single experimental design: the double factorial paradigm (DFP; Townsend & Nozawa, 1995; Houpt et al.,, 2014). The 3 × 3 DFP includes nine conditions; examples of these conditions in our study can be found in Table 1. Throughout the remainder of the paper, we use two-letter combinations of H, L, and A to index particular cells in the DFP design.

Table 1 Stimuli used in Experiment 1 and 2

Full size table

The Mean Interaction Contrast (MIC) allows one to determine architecture, as well as stopping rule to some degree, using similar logic to the interaction testing in an ANOVA (cf. Sternberg, 1969). Assuming that the salience manipulation of each process does not affect the other (i.e., selective influence^{Footnote 3}), the interaction contrast of mean RTs from each condition where both information sources are present will be zero for serial systems, positive for parallel, first-terminating (P-OR) systems, and negative for parallel, exhaustive (P-AND) systems (1).

$$ \text{MIC}(t) = [\overline{\text{RT}}_{\text{LL}}(t) - {\overline{\text{RT}}_{\text{LH}}}(t)] - [{\overline{\text{RT}}_{\text{HL}}}(t) - {\overline{\text{RT}}_{\text{HH}}}(t)] $$

(1)

To see this, consider that in an independent serial system the mean response time will be a sum of the information channels’ processing times; therefore, in Equation 1 the contrasts in brackets should be equal because they each reflect the difference between processing low salience and high salience along the second information channel. This general argument extends to situations in which a participant varies which information channel is used first, i.e., probability mixtures. For P-OR models, mean response time is a function of the minimum of the channel-specific processing times. Thus, any condition with an H in it should produce similar response times that are all faster than the LL condition. In Eq. 1, this causes the first contrast to be positive while the second contrast is approximately zero, yielding a positive MIC. For P-AND models, mean response time is a function of the maximum of the channel-specific processing times, which by similar reasoning causes the first contrast to be approximately zero, the second contrast to be positive, and the MIC to be negative.

The Survivor Interaction Contrast (SIC; Eq. 2) is a functional statistic that generalizes the MIC. In fact, ${\int \limits } \text {SIC}(t)~dt = \text {MIC}$. The SIC uses the full RT distributions as opposed to the mean RTs. Specifically, it computes an interaction contrast of the survivor functions (S(t), where S(t) = 1 −CDF(t)), or the cumulative probability that a response has not been made by time t, across each condition where both information sources are present. Assuming selective influence, the resulting SIC function uniquely identifies both the architecture and stopping rule of a system: serial, first-terminating (S-OR) systems exhibit flat (equal to zero for all time t) SIC functions; serial, exhaustive (S-AND) systems exhibit a first negative then positive S-shaped SIC function; parallel, first-terminating (P-OR) systems exhibit all positive SIC functions; and parallel, exhaustive (P-AND) systems exhibit all negative SIC functions. Coactive processes also exhibit a first-negative-then-positive SIC function, similar to S-AND; however, the SIC of S-AND models has equal negative and positive areas under the curve (i.e., MIC = 0), while the SIC of coactive models has more positive than negative area (i.e. MIC > 0; Townsend & Nozawa, 1995; Houpt et al.,, 2014). The top row of Fig. 1 illustrates the predicted SIC function for each class of models we consider in our study.

$$ \text{SIC}(t) = [{S_{\text{LL}}}(t) - \text{S}_{\text{LH}}(t)] - [\text{S}_{\text{HL}}(t) - \text{S}_{\text{HH}}(t)] $$

(2)

The capacity coefficient measures the joint influence of independence and workload capacity. Like the SIC, it is a functional statistic that uses full response time distributions instead of mean response times. The exact form is dependent upon whether a first-terminating (3) or exhaustive (4) stopping rule is assumed (see Houpt et al., (2014) for a complete derivation). The intuition behind the measure is to first estimate response time distributions from trials where only one source of information is present. For example, when the stimuli are all the same shape and the target only differs from the distractors by its color. Once CDFs have been estimated for each single-source condition, they are transformed into hazard functions^{Footnote 4} and combined according to the assumptions of an unlimited capacity, independent, parallel (UCIP) model. If the UCIP assumptions hold, then performance with all information sources (i.e., color and shape) present should be predicted by this baseline model (the denominator of Eq. 3 and the numerator of Eq. 4), and the resulting ratio of predicted to observed performance (3) and (4) should equal unity (C(t) = 1). Such a result indicates unlimited capacity. Similarly, C(t) < 1 indicates limited capacity, and C(t) > 1 indicates super capacity. When the UCIP model is rejected because C(t) ≠ 1, it may be because any of the three constituent assumptions have been violated. Although the MIC or SIC can be used to check the parallel assumption, it can be difficult to determine whether limited and super results are due to violations of the independence assumption or of the unlimited capacity assumption. Such an outcome likely requires further study, but a coactive SIC and MIC pattern would suggest the independence assumption has been violated (Eidels et al., 2011).

$$ \text{C}_{\text{OR}}(t) = \frac{H_{1{\dots} n}(t)}{{\sum}_{i=1}^{n} H_{i}(t)} $$

(3)

$$ \text{C}_{\text{AND}}(t) = \frac{{\sum}_{i=1}^{n} K_{i}(t)}{ K_{1 {\dots} n}(t)} $$

(4)

Prior usage of SFT to study the temporal organization of visual search is limited. Fifić et al., (2008) examined exhaustive searches of two strings of Cyrillic letters that varied in complexity. They found over-additive MICs and S-shaped SIC functions, suggesting facilitatory parallel processing. More recently, Yamani et al., (2013) and Huang et al., (2015) used SFT to demonstrate that redundant targets in a visual array may be processed in parallel or serial depending on the distance between them. However, these studies were concerned with the organization of processing at the item-level. Our study is therefore the first to use SFT to investigate the temporal organization of feature dimension processing during visual search.

Overview of experiments

We conducted three visual search experiments to be representative of feature search and conjunctive search when the target is known in advance, and odd-one-out search. To remove the potential for confounding workload, the number of stimuli was the same on all experimental trials. Unlike many SFT studies, instead of manipulating the features of the target to accomplish the salience manipulations of the DFP, we followed the example of Townsend and Fifić (2004) and Fifić et al., (2008) and varied the dissimilarity of the distractors to the target. In the first two experiments, we always instructed our participants to search for a conjoint target (a red circle), whereas in the third experiment participants only had to determine whether the stimuli were all the same or one was different. In the first and third experiments, distractors were homogenous (i.e., identical for a given trial), but in the second experiment, distractors were heterogeneous (i.e., all distractor types present on all trials).

In Experiment 1, although we instruct participants to search for a specific target (template-matching strategy), it is possible for them to use a singleton-search strategy where they attempt to find any feature that is different from the rest. In Experiment 2, we modify the design of Experiment 1 to eliminate the singleton-search strategy as a viable tactic: if multiple kinds of distractors are always present, then a difference between stimuli no longer signals a target. Furthermore, because some distractors share a feature with the target, participants must search for the conjunction of color and shape. In Experiment 3, we adapt the design of Experiment 1 to eliminate the template-matching strategy as an effective technique: if the target is not known beforehand, there is no template to guide attention.^{Footnote 5}

Based on previously cited work (Treisman and Gelade, 1980; Wolfe, 2007; Itti & Koch, 2001; Findlay, 1997; Bundesen, 1990; Bundesen et al., 2003; Kyllingsbæk & Bundesen, 2007; Müller et al., 1995), we expect to find evidence for parallel processing of stimulus features in all three experiments, but the stopping rule may vary by task because of differences in the instructions. Specifically, Experiment 1 may be accomplished by a first-terminating stopping rule (parallel-OR), consistent with a singleton-search strategy and indicated by an all positive SIC and a positive MIC, or an exhaustive stopping rule (parallel-AND), consistent with a template-matching strategy and indicated by an all negative SIC and a negative MIC. Additionally, a coactive architecture, suggesting facilitation between the feature processors, may be possible and would be indicated by an S-shaped SIC, a positive MIC, and a capacity coefficient greater than one. In Experiment 2, the task design forces exhaustive processing of color and shape, so we expect to find evidence in favor of either a parallel-AND model or a coactive model. In Experiment 3, there is no incentive to process both feature dimensions, so we expect to find evidence in favor of either a parallel-OR model or a coactive model. Despite this, it is still possible that participants excessively process both stimulus dimensions even when a single difference in features is sufficient to respond. In this case, we may find evidence in favor of a parallel-AND model. To summarize, we hypothesize to reject all serial models in each of the three experiments, and we additionally examine the process stopping-rule and workload capacity.

Experiment 1

Our first experiment investigated the architecture and stopping rule that participants use when searching for a red circle among color and shape distractors using the MIC and SIC. Specifically, the architecture refers to the distinction between subjects processing whether items in the display are red and whether items are circular simultaneously, or first one and then the other. The stopping rule refers to the distinction between observers using only one source of information (e.g., color) to respond, or if they use both. We used the capacity coefficient to jointly examine stochastic independence (e.g., does processing color affect the processing of shape and vice versa?) and workload capacity (e.g., are the stimulus features, color and shape, processed faster or slower together than in isolation?).

This first study examines single-feature visual search. While the search target was defined by the conjunction of two features, all the distractors on a given trial were identical. On trials in which a target (a red circle) was present, it differed from all the distractors in terms of color only (i.e., all distractors were circles), shape only (i.e., all distractors were red), or both features (e.g., purple diamonds, etc.).

Methods

Our experimental design was guided by a pilot study described in the Appendix. In our SFT analyses, we use null hypothesis tests to evaluate the SIC (Houpt & Townsend, 2010) and capacity coefficient (Houpt & Townsend, 2012) and a Bayesian test to evaluate the MIC (Houpt & Fifić, 2017b). Typically, SFT experiments employ many repeated trials in order to fully estimate the response time distributions used to calculate the SIC. To maximize the power of the SIC tests within a reasonable amount of experiment time per participant (1 week), we increased the number of sessions to four from the single session in the pilot study. To estimate the effect on the power for the SIC statistics, we used a stratified bootstrap procedure, in which we first chose a random subject, then randomly sampled from that participant’s response times within each SIC condition. We then estimated the D⁺ and D⁻ on the sample. Based on 1000 bootstrap samples following this procedure, the power increases from $\beta _{D^+}=0.988$ to $\beta _{D^+}>0.999$ and $\beta _{D^-}=0.336$ to $\beta _{D^-}=0.512$ for the target-present condition and from $\beta _{D^+}=0.967$ to $\beta _{D^+}>0.999$ and $\beta _{D^-}=0.403$ to $\beta _{D^-}=0.768$ for the target-absent condition.

We did not conduct a power analysis for the capacity coefficient because we did not evaluate workload capacity in our pilot study. That being said, we scaled the number of capacity samples we collected to the number of samples we collected for the SIC tests, the maximization of which, described in the previous paragraph, was our primary goal. However, the power of the workload capacity test is generally higher than for the SIC tests (cf. Houpt & Townsend, 2012).

Because we used a Bayesian test of the MIC (Houpt & Fifić, 2017b), which was developed after we conducted the pilot study in the Appendix, we appeal to simulations rather than power analysis for the adequacy of our design with regard to this measure. Houpt and Fifić (2017b) demonstrated that just 50 samples are sufficient for strong discrimination among classes of models with the Bayesian MIC test, and we analyzed 111 in Experiment 1.

Participants

The study was conducted at Wright State University and approved by its Institutional Review Board. All participants gave written informed consent before beginning the study and were compensated $8 per one-hour session. Participants who completed all four sessions were awarded an additional $2 per session bonus (i.e., $40 maximum compensation). We tried to keep all four sessions on consecutive weekdays, but one subject needed a span of 8 days to finish.

We recruited eighteen total participants, seventeen of whom completed every session. Data from two participants were discarded for exceptionally poor accuracy,^{Footnote 6} suggesting that they did not follow task instructions. Data from fifteen participants remained for analysis.

Materials

Experiments were developed using PsychoPy (Peirce, 2009) libraries for Python and presented on a Sony 20” Trinitron monitor (1280 × 1024 pixel resolution) positioned 36” from where participants were sitting. Nine different stimuli were used, the products of factorially combining three shades of red (determined by pilot testing to produce reasonable ordering of survivor functions) and three different shapes (circles, octagons, and diamonds; Table 1). For the sake of discussion, we label these three colors:^{Footnote 7} “red” (32.3 cd/m², CIE XY: 0.360/0.324), “pink” (33.3 cd/m², CIE XY: 0.350/0.309), and “purple” (35.3 cd/m², CIE XY: 0.321/0.278). All stimuli subtended 0.696 degrees of visual angle. The target was always a red circle. The background of the search field was white (62.9 cd/m², CIE XY: 0.272/0.298).

Procedure

In each session, participants completed three blocks of trials. The first two blocks — one color-only and one shape-only — were designed to measure response time distributions when only a single stimulus dimension was manipulated in order to calculate the baseline model for the capacity coefficient. The order of these single-feature blocks was randomized for each session. Each single-feature block consisted of 23 sets of four (2 distractor type: high/low salience × 2 target presence: present/absent) trials, randomized within-set. In the color-only block, distractors were limited to circles (i.e., target shape; HA and LA stimuli in Table 1). Thus participants were required to find a red circle amongst other circles. In the shape-only block, distractors were limited to red (i.e., target color) shapes (AH and AL stimuli in Table 1). Thus participants were required to find a red circle amongst other red shapes. The third block utilized all eight distractors found in Table 1, and consisted of 37 sets of 16 (8 distractor types × 2 target presence: present/absent) trials, again randomized within-set. This block allowed us to measure response time distributions when both stimulus dimensions were manipulated in order to calculate survivor interaction contrasts. In all three blocks, exactly 24 stimuli were presented on each trial, with an additional distractor replacing the target on target-absent trials, and the target was present on exactly 50% of trials. As previously stated, although distractor types varied between trials, within a given trial all distractors were of the same type. For example, the target-present, low-high (LH) salience trial condition consisted of 1 red circle (target) and 23 “pink” diamonds, and the target-absent, high-low (HL) salience trial condition contained 24 “purple” octagons.

Each trial, regardless of block, followed the same process. At the start of a new trial the words “Get ready…” were displayed in the center of the screen for 1 second, after which the search field would be presented. Stimuli locations were determined on each trial before presentation by randomly placing the target somewhere within the bounds of 100 pixels from the display’s edges. Distractor locations were then determined by choosing randomly generated locations within the search field bounds that did not place a distractor within 100 pixels of another stimulus. Stimuli were displayed for the full duration of the trial. Trials automatically advanced (i.e., were scratched) if no response was made within 20 seconds (this occurred on 0.04% of all trials across the three experiments). Trials simply advanced to the next trial after a negative (target-absent) response. If a positive (target-present) response was made, the stimuli would be immediately replaced by black outlines of triangles (chosen because they did not share color or shape information with any of the experimental stimuli), and the mouse, which was not normally visible on screen, would appear in the center of the screen. Such trials would advance only after the participant had clicked on a triangle. This portion of the trial was not timed; it was only used to determine whether a correct response had been made. Response times reported in the results section refer only to the interval between search field onset and time of yes/no response. Responses were scored as correct if and only if: 1) the target was present and the participant responded that the target was present followed by correctly designating the location of the target (by clicking on the appropriate triangle) or 2) the target was absent and the participant responded that the target was not present.

Participants received verbal instructions before the start of the first session as well as on-screen, text-based instructions before each block of every session. The experimenter described the stimuli the participant would encounter in each block, and while they were not explicitly told the ratio of target-present to target-absent trials (1:1), participants were advised that some trials may contain a target and some trials may not. Participants were instructed to determine the presence of the target (red circle) as quickly as possible while maintaining accuracy and to respond “yes” (target-present) by clicking the left mouse button and to respond “no” (target-absent) by clicking the right mouse button. Participants were also told that if they responded positively they would then be required to indicate where they found the target by clicking on the triangle that was located where the red circle had been prior to responding.

The text-based instructions provided before each block were accompanied by example images (Table 1) of the stimuli and followed by one practice trial for each unique trial condition that would be encountered in the upcoming block (i.e., four practice trials for the first two blocks and sixteen for the third block), in random order. Feedback was provided for these practice trials, including unlimited time to study their mistake: if the response was incorrect on a practice trial, the screen reverted to the original search field, and the target was highlighted if it was present.

After four sessions, each participant had completed a total of 3104 visual search trials, resulting in 92 unique observations of each trial condition in the single-feature blocks and 148 unique observations of each trial condition in the third search block.

Results

In order to limit variation in the data due to learning effects, we removed from future analysis the data from the first session,^{Footnote 8} leaving for each subject 69 unique observations of each trial condition in the single-feature blocks and 111 unique observations of each trial condition in the third search block.

All analyses were conducted separately for target-present and target-absent conditions. This partition is necessary because in order to respond accurately across all trial conditions, which are not known in advance by the subject, they must adequately search the entire display in the target-absent condition, but in the target-present condition they may respond as soon as the target is found. Thus separating each analysis controls for search comprehensiveness.

Accuracy

Subjects were able to reliably locate the target and reject target-absent trials (M = 94.30%, SD = 3.13%; M = 98.89%, SD = 0.81%; respectively), although it is clear that the pink circle and red octagon distractor types were most difficult. Figure 2 depicts group accuracy across all trial conditions.

Bayesian ANOVA was used to check the influence of our experimental manipulations (session: 2, 3, 4; color dissimilarity: high, low, none; shape dissimilarity: high, low, none). Because there was no experimental condition where the distractors did not differ from the target in color or shape, separate ANOVAs were conducted for trials where the distractors differed from the target in one feature dimension and for trials where the distractors differed from the target in both dimensions. Data have been collapsed across blocks. Bayes factors (BF) are always reported as evidence for the best-fitting model relative to an alternative (e.g., if reporting evidence against the third best-fitting model, the BF would be reported as $\frac {p(D|M_1)}{p(D|M_3)}$). Models with BF > 100 are not reported beyond the first. Unless otherwise noted, all ANOVA models include an individual subject main factor (see Morey et al.,, 2015; Jeffreys, 1998).

In the single feature difference trials, the most likely model included an interaction between the distractor dimension (color or shape) and salience level. In the target-present condition, this was the only plausible model (BF = 349.33 against the next most likely model, which added a main effect of session). In the target-absent condition, a second model, in which feature and dissimilarity did not interact, was worth mentioning (BF = 14.38). The Bayes factor over the third best model, which included the interaction of feature and dissimilarity and a main effect of session, was 544.77.

In the dual feature difference trials, our ANOVA results were less consistent, likely because of ceiling effects. In the target-present condition, the most likely model, which included main effects of color dissimilarity and shape dissimilarity, was indistinct from the model which also included their interaction (BF = 1.13). Further, there was not strong evidence against a main effect of shape dissimilarity only (BF = 6.79). Again, we observed evidence against the inclusion of session (BF = 117.26 against the fourth most likely model, which included main effects of color dissimilarity, shape dissimilarity, and session).

The target-absent ANOVA results suggest our manipulations did not strongly affect accuracy in the dual feature difference trials (BF = 3.59 for the most likely model, which included a main effect of color dissimilarity, against the next most likely model, which did not include any factor other than subject; BF = 15.29 against the third most likely model, which included main effects of color dissimilarity and shape dissimilarity; BF = 54.63 against the main effect of shape dissimilarity; BF = 227.42 against the interaction of color dissimilarity and shape dissimilarity). Again, this is likely a ceiling effect due to the high accuracy observed in this condition.

Response time

Figure 3 depicts mean RTs across all trial conditions. As with accuracy, Bayesian ANOVA was used to check the influence of our experimental manipulations. Data have also been collapsed across blocks and separated based on the number of distractor features that were different from the target.

In the single feature difference trials, there was strong evidence for a three-way interaction of feature, dissimilarity, and session in both target-present and target-absent conditions (BF = 1.09 × 10⁹ against the next most likely model in the target-present condition, which only included the composite two-way interactions; BF = 382.40 against the next most likely model in the target-absent condition, in which feature and session did not interact).

In the dual feature difference trials, the most likely model in the target-present condition included a main effect of session and the interaction of color dissimilarity and shape dissimilarity (BF = 91.44 against the second most likely model, which added the interaction of session and color dissimilarity; BF = 129.08 against the next most likely model, which only added the interaction of session and shape dissimilarity). In the target-absent condition, the most likely model included a three-way interaction of session, color dissimilarity, and shape dissimilarity (BF = 18.58 against the second most likely model, which included only the composite two-way interactions; BF = 952.63 against the next most likely model, which further lost the interaction of session and shape dissimilarity).

Architecture and stopping rule

Before the MIC or SIC can be interpreted, the assumption of selective influence must be checked. We use a series of Kolmogorov-Smirnov tests as in Houpt et al., (2014) and Heathcote et al., (2010). This assumption requires that the survivor functions be ordered at all t according to

$$\text{S}_{\text{LL}}(t)\le\text{S}_{\text{LH}}(t),\text{S}_{\text{HL}}(t)\le\text{S}_{\text{HH}}(t)$$

with strict dominance at some t.

In the target-present condition, 3 of the 15 subjects demonstrated strict selective influence. The remaining 12 subjects are retained for further analysis because they did not demonstrate significant violations of selective influence and their survivor functions were ordered appropriately upon visual inspection. We assume that our inability to detect strict dominance for these subjects is due more to the limitations of null hypothesis testing than to real violations of selective influence. In the target-absent condition, 11 of the 15 subjects demonstrated strict selective influence, while the remaining 4 subjects were classified as the ambiguous type described previously. Therefore, we interpret the MICs and SICs of all 15 subjects in both conditions.

In the target-present condition, Bayesian analysis suggested that the group’s MIC was positive ($\hat {p_+} = .91$). Estimation of individuals’ MICs agreed, and posterior probabilities ranged from .96 to 1.00. We found similar results in the target-absent condition such that the group MIC was likely positive ($\hat {p_+} = .91$). Individuals ranged from .97 to 1.00. All individual posterior probabilities are reported in the Supplemental Materials.

In the target-present condition, the SICs of all subjects had significant positive deviations (D⁺ ∈ [0.420, 0.826], p ∈ [7.10 × 10^− 17, 6.60 × 10^− 5], α = .33^{Footnote 9}) and nonsignificant negative deviations (D⁻∈ [0.000, 0.128], p ∈ [0.410, 1.000], α = .33). Individual SIC functions are shown in Fig. 4.

In the target-absent condition, the SICs of all subjects had significant positive deviations (D⁺ ∈ [0.599, 0.982], p ∈ [5.72 × 10^− 24, 3.44 × 10^− 9], α = .33). While the SIC of 8 of the 15 subjects had nonsignificant negative deviations (D⁻∈ [0.018, 0.127], p ∈ [0.411, 0.982], α = .33) as in the target-present condition, the remaining 7 subjects had significant negative deviations (D⁻∈ [0.148, 0.253], p ∈ [0.029, 0.298], α = .33). Individual SIC functions are shown in Fig. 5.

Workload capacity

We calculated capacity functions and corresponding z-scores for each subject in each of the four trial conditions (HH, HL, LH, LL) used to calculate MIC and SIC. Although it is possible that subjects could use a first-terminating (singleton search) strategy,^{Footnote 10} the task instructions were specifically to find a conjunctive target, so we assume exhaustive processing in the baseline (UCIP) model (C_AND(t); Equation 4). We discarded any z-score calculated from a condition with less than 90% accuracy in either the dual-source or single-source conditions, leaving 32 of 60 (15 subjects × 4 salience conditions) z-scores available for analysis in the target-present condition. All 60 z-scores passed the accuracy criterion in the target-absent condition. All subjects demonstrated significant super capacity in all conditions (individual functions are plotted in the Supplemental Materials). Distributions of Cz_AND across subjects are depicted in Fig. 6 for each experimental condition. Individual scores ranged from 4.156 to 9.181 in the target-present condition and from 2.571 to 10.211 in the target-absent condition (note that Cz_AND = 0 corresponds to the UCIP baseline).

We used Bayesian ANOVA to determine effects of color and shape target-distractor dissimilarity on capacity scores. In the target-present condition, there was no evidence for any effect of dissimilarity above and beyond the subject-only model after applying the accuracy criterion (BF = 1.010 for the main effect of shape dissimilarity against the subject-only model).^{Footnote 11} In the target-absent condition, there was moderate evidence in favor of a main effect of shape dissimilarity (BF = 2.424 × 10⁷ against the subject-only model) over main effects of color and shape (BF = 3.643) and over their interaction (BF = 7.785).

Discussion of Experiment 1

We found clear and consistent evidence against serial processing of shape and color and in favor of either first-terminating parallel or coactive processing. Every subject’s MIC was substantially positive in both target-present and target-absent conditions. Because the MIC was nonzero, we can reject all serial models of color and shape processing during feature search. Similarly, because the MIC was positive, we can reject parallel models of independent, exhaustive processing. The SIC results largely agree: each subject’s SIC demonstrated significant positive deviations from zero, ruling out serial models and independent, exhaustive, parallel models. While we failed to detect negative deviations from zero in the target-present condition, nearly half (7 of 15) of the SICs in the target-absent condition were significantly negative for some time, indicating coactive processing.

As stated earlier, we could not be certain from the task constraints whether subjects would use a template-matching or singleton-search strategy, corresponding to an exhaustive or first-terminating stopping rule, respectively, in the target-present condition. This ambiguity weakens the conclusions we can draw from the workload capacity analysis because the interpretation depends on which strategy one believes the subjects used. If an exhaustive stopping rule is assumed, then all subjects exhibited super capacity, and a coactive processing architecture is the only class of model that could explain our pattern of results. However if a first-terminating stopping rule is assumed, then the majority of capacity coefficients were consistent with UCIP processing. Nevertheless, a substantial number were better than UCIP (i.e., super capacity) even with this more conservative measure, suggesting the presence of facilitatory dependence or that additional resources are recruited when the distractors can differ from the target in both color and shape. A possible explanation for this would be top-down guidance made possible by subjects’ knowledge of the target’s identity before trial onset. We examine this issue with more control in the subsequent experiments.

In the target-absent condition, subjects must ensure both the target’s color and shape are absent from the display, so the above ambiguity is avoided. All subjects exhibited super capacity: performance on trials in which distractors differed from the target in both color and shape was significantly better than would be predicted by a UCIP model. Because the MIC and SIC results eliminated serial processing as a potential explanation, we know that this facilitation must be caused by either violations of the unlimited capacity assumption or the independence assumption. The exceptionally large capacity functions observed suggest the redundant information gain is strong. This is to be expected considering we found evidence for coactive processing, which is the extreme case where feature processing is fully dependent (Eidels et al., 2011). Furthermore, the degree of super capacity suggests that where we did not detect significant negative SIC deviations, color and shape features are still processed in some facilitatory parallel form, if not coactive.

Accuracy was very good in Experiment 1, demonstrating the relative ease of the task. Any deficits we observed corresponded with increased similarity between the target and distractors. Likewise, response times increased as expected with reduced trial salience. Further examining the ANOVAs conducted for RT in the conditions where the distractors differed from the target in both color and shape, we found a significant interaction of color and shape dissimilarity. This interaction is essentially a parametric MIC, corresponding to the results of our nonparametric MIC analysis.

Despite the uncertainty described earlier with regard to the stopping rule employed by subjects in the target-present condition, the success of the salience manipulations in the target-absent condition is evidence against a pure singleton-search strategy. If subjects were simply deciding whether a difference exists in the field of homogeneous distractors, then we wouldn’t expect to find significant differences in RT that systematically vary with respect to the distractors’ dissimilarity to the (absent) target.^{Footnote 12} However, because the target-distractor dissimilarity effect was present even when the target itself was not, we may conclude that some comparison process is taking place. If this is the case, then we may reasonably believe that the subjects followed our instructions to find a specific target rather than to engage in a singleton-search strategy, and we may therefore favor coactive processing as the more likely model in feature search. That being said, we cannot entirely rule out more complex, alternative models such as a blending of the two strategies.

Experiment 2

Our first experiment successfully characterized the display-level integration of color and shape information in the context of visual search with homogeneous distractors and a consistent, conjunctive target. However, because the design of Experiment 1 allowed for accurate task completion without forcing exhaustive processing of both stimulus features on every trial, the full extent to which color processing and shape processing interact remains uncertain. Experiment 2 sought to answer this question by altering the previous paradigm to instead use heterogeneous distractors. That is, we continued to manipulate the prevalent distractor on each trial as before, but we also presented every possible remaining distractor as well (i.e., all distractor types were present, but one type would be chosen to be more frequent than the others). Because all distractor types were present on every trial, participants were forced to process both color and shape information if they were to maintain a high level of accuracy throughout the experiment.