Asymmetric binding in serial memory for verbal and spatial information

Abstract

As the number of studies showing that items can be retained as bound representations in memory increases, researchers are beginning to investigate how the different features are bound together. In the present study, we examined the relative importances of the verbal and spatial features in serial memory for visual stimuli. Participants were asked to memorize the order of series of letters presented visually in different locations on the computer screen. The results showed that manipulating the phonological similarity of the letters affected recall of their spatial locations, but that increasing the complexity of the spatial pattern had no effect on recall of the letters. This finding was observed in both order reconstruction (Exps. 1 and 2) and probe serial recall (Exps. 3 and 4), suggesting that verbal–spatial binding in serial memory for visual information is asymmetric.

Natural objects encountered in our environment incorporate several features that combine to form unified percepts. Accumulating evidence has suggested that the different features of these objects can be retained in conjunction in memory (see, e.g., Prabhakaran, Narayanan, Zhao, & Gabrieli, 2000) rather than in separate stores (e.g., Wheeler & Treisman, 2002). Some models have been developed in order to account for binding in memory, such as by the incorporation of a new component for storing bindings in the working memory model (the episodic buffer; Baddeley, 2000; Baddeley, Allen, & Hitch, 2011; Baddeley & Hitch, 1974). One limit to the development of memory models, however, is that it is not clear how the different features of an object are integrated into a bound representation in memory. That memory representations seem to be bound conveys the idea that the features of an object are integrated in such a way that they are all equally useful for encoding and retrieving the object that they make up. The existing evidence, however, suggests that all features of an object are not equally important (see, e.g., Campo et al., 2010; Jiang, Olson, & Chun, 2000). The objective of the present study was to examine the relative importances of the verbal and spatial features of visual objects retained in serial memory.

Evidence for binding in memory

In a classic study by Prabhakaran et al. (2000), participants were first presented with a set of letters in different locations on an imaginary ellipse—the memory display. After presentation of the memory display, one letter was presented in one location—the probe. In the intact probe condition, the probe letter was presented in the same location as in the memory display. In the recombined probe condition, the probe location and the probe letter were each part of the memory display, but they were not presented together. Participants had to indicate whether the letter and the location were part of the memory display, irrespective of whether the probe was intact or recombined. The results showed that the participants were faster and more accurate to recognize the features of the intact probe, as compared to those of the recombined probe. Prabhakaran et al. suggested that the letters and locations were maintained as bound representations in memory, so that it was faster and easier to recognize the features of a probe that corresponded to the memory representation. Since then, increasing evidence has emerged of binding in memory (see, e.g., Alvarez & Thompson, 2009; Guérard, Tremblay, & Saint-Aubin, 2009a; Maybery et al., 2009; Treisman & Zhang, 2006; Ueno, Allen, Baddeley, Hitch, & Saito, 2011).

Among the possible features of an object, location information is one feature that has attracted much attention from researchers. A number of studies on binding have suggested that in memory for visual information, features such as color and shape are automatically bound to their spatial location (see, e.g., Jiang et al., 2000; Olson & Marshuetz, 2005; Poch et al., 2010; Treisman & Zhang, 2006). For instance, Poch et al. used a paradigm similar to that used by Prabhakaran et al. (2000): Participants were presented with an array of letters in different locations, followed by a probe item that was either intact—the letter in its original location—or recombined—a letter and a location from the memory display that had not been presented together. In one condition, participants were asked to indicate whether the probe letter had been part of the memory display, irrespective of its location. Poch et al. showed that recombined probes led to poorer accuracy than did intact probes. Therefore, letter recognition was impaired when the letter changed location from the memory display to the probe display, suggesting that memory for letters is at least partially dependent on memory for their locations. Moreover, this pattern was observed despite the fact that participants did not have to retain the locations, which suggests that spatial–verbal binding occurs even if spatial information is irrelevant to the task (see also Meegan & Honsberger, 2005).

Asymmetric binding in memory

If a representation including both location and identity information is automatically encoded, then one might expect that if either of these feature dimensions were probed, the other dimension would also be retrieved. However, this does not seem to be the case. Indeed, some researchers have suggested that features are integrated in such a way that the retrieval of one feature depends on the retrieval of another that can be retained independently. For instance, using a change detection task, Jiang et al. (2000) showed that memory for colors and shapes is impaired when the locations of the objects are changed from the memory display to the probe display (see also, e.g., Poch et al., 2010). Memory for the spatial configuration of the display, however, was not influenced by whether or not the colors of the objects remained the same from the memory to the probe display.

In another study, participants were asked to memorize a display of letters presented in different locations (Campo et al., 2010). At test, a probe was presented, either intact or recombined. In the verbal task, participants had to indicate whether the probe letter had been presented in the memory display, irrespective of its location. The results showed that as compared to an intact probe, a recombined probe impaired memory for letters, suggesting that letters are encoded along with their spatial locations. In the spatial task, participants had to indicate whether the probe location had been part of the memory display, irrespective of the letter at that location. Campo et al. showed that memory for locations was not modulated by whether the probe was intact or recombined, suggesting that locations can be retained independently of their associated features in memory.

Previous results, therefore, have suggested that binding in visual memory is asymmetric: Object features are bound to their locations, whereas locations—or the spatial configuration of the display as a whole (see Jiang et al., 2000)—can be retained independently (see Campo et al., 2010). This type of asymmetry has not been observed consistently in the literature, however. For instance, Maybery et al. (2009) showed a reverse pattern—that is, asymmetric binding with letters as the independent feature. In their study, sequences of letters were presented auditorily in different locations. After the list presentation, a probe was presented. The probe could be a letter in its original location—an intact probe—or a letter and a location from the memory set that had not been presented together—a recombined probe. In the verbal task, participants were required to indicate whether the letter had been part of the memory set and not to pay attention to the location. In the spatial task, participants were asked to indicate whether the probe location had been part of the memory set, irrespective of the letter. Maybery et al. found that spatial recall, but not verbal recall, was affected by presenting a recombined versus an intact probe. The latter result shows asymmetric binding in which letters can be recalled independently of their locations, whereas locations depend on retrieval of the letters. Maybery et al. suggested that such an asymmetry resulted from automatic binding. Interestingly, when participants were instructed to retain both the verbal and the spatial features of the items, binding was symmetric. Indeed, in one experiment, participants were cued to identify the verbal or the spatial feature only after the presentation of the list. Maybery et al. showed that when participants should retain both features until the time of recall, both spatial and verbal forms of recognition were impaired by recombined as compared to intact probes. They suggested that when participants must retain both features, they voluntarily bind the features together, which results in symmetric binding.

The finding that memory for locations is dependent on memory for the letters was also observed in a study using visual materials. Guérard et al. (2009a) presented series of letters sequentially in different locations on the computer screen. After presentation, participants were cued to recall either the letters in the same order in which they had been presented, or the locations where those letters had been presented, but in any order. The authors showed that the phonological similarity of the letters, a factor well known to modulate the retention of verbal information (see, e.g., Conrad, 1964), impaired recall of the spatial locations. Therefore, the study suggests that locations are not retained independently, since the identity of the letters influenced the capacity to recall their locations. From these results, however, it was not possible to know whether binding in serial memory for visual information is symmetric or asymmetric. Binding might have been symmetric, because of the requirements to retain both the verbal and spatial features at the same time. Another possibility, however, is that because of the intrinsic serial nature of speech, the verbal feature is the primary one when letters are presented sequentially, resulting in asymmetric binding.

The first objective of the present article was to examine how verbal and spatial features are bound in serial memory for visual information. The results of Guérard et al. (2009a) suggested that letters are not retained independently of their locations. This result may not generalize to serial memory, however, because the recall procedure differed between the verbal and spatial recall conditions in Guérard et al.’s (2009a) study: Participants were required to recall letters in their original order, but locations in any order. Therefore, in the first experiment of this series, we replicated the experiment of Guérard et al. (2009a) using order reconstruction for both spatial and verbal recall and manipulating the phonological similarity of the letters. Phonological similarity is a factor well known to affect the recall of letters (see, e.g., Conrad, 1964). If spatial locations are bound to letter identities in serial memory, the detrimental effect of phonological similarity should spread to the recall of spatial locations through the letter–location association.

In order to examine whether binding in serial memory for visual information is symmetric or asymmetric, in Experiment 2 we manipulated a factor known to impair spatial recall: the spatial complexity of the pattern of locations (see, e.g., Parmentier & Andrés, 2006; Parmentier, Elford, & Mayberry, 2005). Spatial complexity can be increased by making the path between successive to-be-remembered locations lengthier and by making those paths cross over. Robust evidence has shown that such manipulations of transitions have a detrimental impact on recall. Of course, spatial complexity can be deemed as being qualitatively different from phonological similarity. However, the main objective here was to use the most effective factors for modulating spatial recall, rather than to attempt to make the spatial and verbal manipulations equivalent.

If binding is asymmetric and letters can be retained independently of their locations, spatial complexity should not affect verbal recall. If letters are bound to their spatial locations in memory, though, verbal recall should be impaired by spatial complexity. In Experiments 3 and 4, we used a probe serial recall task in which participants recalled the verbal and spatial features of the item following a probe in the list. To anticipate, we observed asymmetric binding using both tasks: That is, phonological similarity affected the recall of locations, whereas the recall of letters was not affected by the spatial complexity of the pattern of locations.

A second objective of the present series of experiments was to examine the effect of task requirements on asymmetric binding. More precisely, we examined whether the number of features to be retained during the memory task influences the asymmetry. Indeed, in previous studies showing asymmetric binding, spatial information did not need to be retained (see, e.g., Campo et al., 2010; Jiang et al., 2000). In another study, Maybery et al. (2009) showed asymmetric binding with letters as the independent feature when participants had to retain one feature, but symmetric binding when participants had to retain both the verbal and spatial features. Whether binding is symmetric or asymmetric might, therefore, depend on the task’s requirement to explicitly retain both features of the objects. In order to test this idea, in Experiments 1 and 2 we included two groups. In one group, the participants were assumed to retain both the verbal and the spatial features, because they did not know in advance which feature dimension would be tested. In the other group, the participants were asked to retain a single feature, which was known to the participants before the to-be-remembered stimuli were presented.

Experiment 1

In Experiment 1, we replicated the study of Guérard et al. (2009a) using a reconstruction procedure in both verbal and spatial recall trials. Lists of similar and dissimilar letters were presented sequentially in different locations on the computer screen. After presentation of the list, participants tried to recall the verbal or the spatial features in the same order that these features had been presented. In the two-feature group, participants were informed after list presentation of whether they had to recall the letters or the locations. They therefore had to retain both the verbal and the spatial features. In the spatial recall trials, they had to click on the spatial locations of the items in the same order that they had been presented. In the verbal recall trials, they had to click on the letters to reproduce their initial order. In the single-feature group, participants memorized only the locations, and did not pay attention to the letters: Their task was always to recall the locations of the letters in the same order that they had been presented. If memory for spatial locations is dependent on memory for the letters presented in those locations, phonological similarity should modulate spatial recall performance, resulting in impaired spatial recall, as observed by Guérard et al. (2009a).

Method

Participants

A total of 60 undergraduate students from Université Laval volunteered to participate in this experiment. We tested 30 participants in each group.

Materials

The course of the experiment was controlled by a PC computer using E-Prime, with a resolution of 1,024 × 768 pixels. Sixty sequences of seven letters were constructed. Half of the sequences were made from the random ordering of the phonologically similar letters B, C, D, G, P, T, and V, and the other half were made from the random ordering of the dissimilar letters F, H, J, K, L, N, and R. The letters were presented in 35-point Arial Black font. The locations of the letters were determined quasirandomly, with the restrictions that (1) each location was separated from any other location in the same list by at least 4.5° and (2) all of the sequences contained between two and three crossings (i.e., the number of times that one path connecting two successive locations crossed another path; see Parmentier et al., 2005). The sequences in all conditions were equated in the numbers of crossings, the distances between all pairs of successive locations, and the total distances between all possible pairs of dots within a sequence.

Design and procedure

In each trial, a sequence of seven letters was presented at a rate of one letter per second. Each letter was visible on the screen for 750 ms, followed by a blank screen for 250 ms. After the list presentation, all items were re-presented simultaneously for recall. In the spatial recall condition, seven empty squares surrounded by a black frame of 1.2° × 1.2° were presented in the same locations where the letters had been presented. Participants were required to click on the squares in the same order that they had appeared on the screen. Once the squares were clicked, they became green. In the verbal recall condition, the letters reappeared horizontally in alphabetical order in the center of the screen. Participants were then required to click on the letters in the same order that they had been presented. The letters were each surrounded by a black frame once selected. The procedure is presented in Fig. 1.

Fig. 1

Illustration of the reconstruction procedure used in Experiments 1 and 2

In the two-feature group, the presentation of half of the sequences was followed by spatial recall, and the other half were followed by verbal recall. No information about the feature to recall was given prior to list presentation, so that participants had to retain both the spatial and verbal features of each item. The lists in the four conditions (similar/spatial recall, similar/verbal recall, dissimilar/spatial recall, and dissimilar/verbal recall) were presented randomly, in a different order for each participant. The experiment lasted 40 min. In the single-feature group, the 30 sequences in the spatial recall condition were used. In this group, participants were required only to retain the spatial locations of the items and not to pay attention to the letters that were presented. The lists in the two conditions (similar/spatial recall, dissimilar/spatial recall) were presented in a different random order for each participant. The experimental trials were preceded by two practice trials. The experiment lasted 20 min in all.

Results

The verbal recall trials in the two-feature group were first analyzed to establish the presence of the phonological similarity effect. The spatial recall trials were then compared between the two-feature and single-feature groups. A strict serial-recall criterion was used: A response was scored as correct when an item was recalled in the same serial position in which it had been presented.

Verbal recall

Recall performance was analyzed for the verbal trials of the two-feature group. Inspection of Fig. 2 (left panel) suggests that phonologically similar letters were more difficult to recall than dissimilar letters. A 2 (phonological similarity: similar, dissimilar) ×7 (serial position: 1 to 7) repeated measures analysis of variance (ANOVA) was performed on the proportions correct. In all analyses, the .05 level of significance was adopted, and the Greenhouse–Geisser correction was applied when the sphericity criterion was not met. The analysis confirmed that recall performance was higher for dissimilar- than for similar-sounding letters, F(1, 29) = 18.25, MSE = 0.07, p < .001, η ² _p = .39. The main effect of serial position was significant, F(6, 174) = 33.86, MSE = 0.02, p < .001, η ² _p = .54, consistent with the effects of primacy and recency observed in Fig. 2. The interaction between phonological similarity and serial position was also significant, F(6, 174) = 4.08, MSE = 0.02, p = .001, η ² _p = .12, suggesting that the phonological similarity effect was stronger for the last serial positions.

Fig. 2

Proportions of correct responses as a function of serial position and phonological similarity in the verbal recall condition (left) and the spatial recall condition of the two-feature group (middle) and the single-feature group (right) of Experiment 1. Errors bars represent 95 % confidence intervals

Spatial recall

Spatial recall performance for the two-feature group (middle panel) and the single-feature group (right panel) is also illustrated in Fig. 2. A 2 (number of features: two, one) ×2 (phonological similarity: similar, dissimilar) ×7 (serial position: 1 to 7) mixed ANOVA was carried out on the proportions correct in the two groups. The analysis showed that performance was higher in the single-feature than in the two-feature group, F(1, 58) = 25.06, MSE = 0.20, p < .001, η ² _p = .30. Recall performance also varied as a function of serial position, F(6, 348) = 91.29, MSE = 0.03, p < .001, η ² _p = .61. Although the main effect of phonological similarity was not significant, F(1, 58) = 1.50, MSE = 0.03, p = .23, η ² _p = .03, the interaction between phonological similarity and serial position was significant, F(6, 348) = 12.19, MSE = 0.01, p < .001, η ² _p = .17. Neither the interaction between phonological similarity and number of features nor the three-way interactions were significant (Fs < 1), suggesting that serial position modulated the effect of phonological similarity similarly in both groups. Paired-samples t tests showed that locations in the similar-letters condition were recalled better than locations in the dissimilar-letters condition at Serial Position 1, t(59) = –2.92, p < .007, and that locations with dissimilar letters were recalled better than locations with similar letters at Serial Positions 6, t(59) = 4.38, p < .007, and 7, t(59) = 5.23, p < .007. No other interactions were significant.

Discussion

The results of Experiment 1 replicated the phonological similarity effect: Similar-sounding letters were more difficult to recall than dissimilar-sounding letters (see, e.g., Conrad, 1964). The effect was not distributed uniformly across serial positions, however. Indeed, the effect seemed to be stronger for later serial positions, and to disappear at Serial Position 4. One possible explanation could be that participants used grouping strategies in the dissimilar condition, leading to an increase in accuracy at Serial Position 3 (see, e.g., Parmentier & Maybery, 2008).

We also replicated the results of Guérard et al. (2009a) using an order reconstruction procedure in both verbal and spatial recall conditions, by showing that the detrimental effect of phonological similarity affected the recall of spatial locations. This suggests that spatial locations are bound to the letters, and that recalling locations is impaired by detrimental effects associated with the retention of letters. As in the study of Guérard et al. (2009a), this effect was found whether or not participants had to pay attention to the verbal feature.

Interestingly, the detrimental effect of phonological similarity on spatial recall was restricted to the end of the list, and was even in the opposite direction for Serial Position 1. One possibility is that this interaction was due to output interference, according to which the recall of each item interfered with items not yet recalled in memory (see, e.g., Lewandowsky, Duncan, & Brown, 2004). For instance, spatial locations marked by similar letters might be more fragile to output interference than are locations marked by dissimilar letters (see Runquist & Horton, 1977), leading to an end-of-list advantage for spatial recall in the dissimilar condition.

Experiment 2

In Experiment 2, we replicated the procedure of Experiment 1, but in order to test symmetry, we manipulated spatial complexity, a factor known to impair memory for spatial locations. Spatial complexity was defined as the total length of, as well as the number of crossings between, the paths connecting successive dot locations. Both the number of crossings (see, e.g., Parmentier & Andrés, 2006; Parmentier et al., 2005) and path length (see, e.g., Guérard, Tremblay, & Saint-Aubin, 2009b; Parmentier et al., 2005) have been shown to substantially decrease order memory for spatial locations. In the two-feature group, participants tried to retain both the letters and their locations. In the single-feature group, participants were only required to retain the letters in their presentation order, and not to pay attention to the spatial locations. If binding in order memory for visual information is symmetric, spatial complexity should impair the recall of letters. If binding is asymmetric and letters can be retained independently from their spatial locations, spatial complexity should not impair memory for letters.

Method

Participants

A total of 40 undergraduate students from Université de Moncton volunteered to participate in this experiment, with 20 in each group.

Materials

Sixty sequences of seven letters were constructed. In each trial, the letters were selected randomly from the letters of the alphabet (except for the vowels and W). The sequences of locations were constructed from the quasirandom ordering of the same seven locations. We manipulated the total distance separating the seven successive locations (path length) and the number of times that one path connecting two successive locations crossed another path (number of crossings). In half of the trials, the sequences contained one crossing and the total distance between successive locations varied between 44.1° and 47.8° (low spatial complexity). In the other half of the trials, the sequences contained six crossings and the total distance between successive locations varied between 57.0° and 59.7° (high spatial complexity).

Design and procedure

The design and procedure were the same as in Experiment 1 (see Fig. 1), except that in the single-feature group, 30 trials with the verbal recall procedure were presented: Participants were instructed to pay attention to the letters and to ignore their spatial location.

Results

The spatial recall trials in the two-feature group were first analyzed to establish the presence of the effect of complexity on spatial recall. The verbal recall trials were then compared between the two-feature and single-feature groups.

Spatial recall

Inspection of Fig. 3 (left panel) suggests that spatial recall in the two-feature group was lower for high-complexity than for low-complexity sequences. A 2 (spatial complexity: low, high) ×7 (serial position: 1 to 7) repeated measures ANOVA was performed on the proportions correct. The analysis confirmed that recall performance was higher for low-complexity than for high-complexity sequences, F(1, 19) = 46.05, MSE = 0.03, p < .001, η ² _p = .71. The main effect of serial position was significant, F(6, 114) = 25.14, MSE = 0.04, p < .001, η ² _p = .57, and the interaction between spatial complexity and serial position was marginally significant, F(6, 114) = 2.09, MSE = 0.02, p = .06, η ² _p = .10.

Fig. 3

Proportions of correct responses as a function of serial position and spatial complexity in the spatial recall condition (left) and the verbal recall condition of the two-feature group (middle) and the single-feature group (right) of Experiment 2. Errors bars represent 95 % confidence intervals

Verbal recall

Figure 3 suggests that spatial complexity did not influence the recall of verbal information, irrespective of whether participants retained both features (middle panel) or only the verbal feature (right panel). A 2 (number of features: two, one) ×2 (spatial complexity: low, high) ×7 (serial position: 1 to 7) mixed ANOVA was carried out on the proportions of correct responses in the two groups. The analysis showed that performance was higher in the single-feature than in the two-feature group, F(1, 38) = 6.26, MSE = 0.23, p = .017, η ² _p = .14. Recall performance also varied as a function of serial position, F(6, 228) = 62.62, MSE = 0.04, p < .001, η ² _p = .62. The main effect of spatial complexity was not significant, F < 1, nor were any other interactions.

Discussion

Experiment 2 replicated previous studies in showing that spatial complexity impairs memory for spatial locations (see, e.g., Parmentier et al., 2005). The results are also clear in showing that spatial complexity had no effect on the recall of letters. Together, Experiments 1 and 2 provide evidence that binding in order memory for visual information is asymmetric: Locations seem to be bound to letters, so that any effect that is detrimental to memory for letters spreads to memory for locations (see, e.g., Guérard et al., 2009a). Verbal recall, however, appears to be immune to any effect modulating memory for spatial locations.

Interestingly, asymmetric binding was observed irrespective of whether participants had to pay attention to both the verbal and spatial features or only to a single feature. Maybery et al. (2009) suggested that asymmetric binding could be due to a primitive, automatic form of binding that occurs irrespective of the number of features to be retained. Symmetric binding, however, could occur when participants voluntarily try to remember the two features as bound representations. One possibility is, therefore, that in the two-feature groups of Experiments 1 and 2, participants did not voluntarily retain both features as integrated representations because of the task requirements. Indeed, the fact that they had to recall only one of the two features on each trial might have prompted them to retain two independent streams of features rather than one stream of bound objects. This strategy might be responsible for the asymmetry that we observed. There is reason to believe that concurrent verbal tasks impair spatial memory more than concurrent spatial tasks impair verbal memory (Morey & Mall, 2012), which could explain the asymmetries observed without assuming that binding occurred. In order to address this issue, a probe serial recall task was used in Experiments 3 and 4.

Experiment 3

In Experiments 3 and 4, we used a probe serial recall task (e.g., Sternberg, 1967) in which the two features of an item had to be recalled in each trial. In each trial, the presentation of a sequence of spatially distributed letters was followed by a probe consisting of a single feature—that is, either a letter or a spatial location. Participants were required to recall the feature following the probe in the sequence, and then to recall the feature associated with the recalled feature. For instance, if the probe was a spatial location, participants had to select first the location that followed in the sequence, and then the letter presented at the selected location. If the probe was a letter, participants recalled first the letter that followed in the sequence, and then the location where the selected letter had appeared. In this context, the participants should be more likely to try retaining the bindings between features, because of the requirement to recall the second feature through its association with the first-recalled feature. If symmetric binding occurs when participants voluntarily bind features together, we should observe phonological similarity affecting spatial recall as well as spatial complexity affecting verbal recall. Such a procedure would also allow further understanding of how the detrimental effect associated with one feature spreads to the other. For instance, given correct recall of a letter as the first feature, would spatial memory still suffer due to phonological similarity? In Experiment 3, we manipulated the phonological similarity of the letters.

Method

Participants

A group of 25 undergraduate students from Université de Moncton volunteered to participate in this experiment.

Materials

One hundred sequences of six letters were constructed. The sequences comprised six letters because a pilot study had shown a floor effect with sequences of seven letters. Half of the sequences were made from the random ordering of the similar letters B, C, D, G, P, and T. The other half of the sequences were made from the random ordering of the phonologically dissimilar letters F, H, J, K, L, and N. The locations of the letters in each trial were determined using the same restrictions as in Experiment 1.

Design and procedure

In each trial, a sequence of six letters was presented at a rate of one letter per second (750 onscreen/250 blank). After presentation, a probe containing a single feature was presented. In the verbal–spatial recall condition, a letter from the sequence was presented in red at the bottom left of the screen. All of the other letters from the sequence were presented in black, centered at the bottom of the screen in alphabetical order. Participants first had to click on the letter that had followed the red letter in the sequence. Upon clicking on the letter, all of the letters disappeared from the computer screen except for the selected letter, and six empty squares occupying 1.2° × 1.2° of visual angle appeared at the locations of the six letters from the sequence. Participants had to click on the square corresponding to the location of the selected letter. Once it was clicked, the square turned green for 500 ms. In the spatial–verbal recall condition, six squares surrounded by a black frame of 1.2° × 1.2° were presented in the same locations where the letters had been presented. All of the squares were empty except for the probe, which was filled with green. Participants had to click on the location that had followed the probe location in the sequence. Upon clicking on the location, all unselected locations, including the probe, disappeared from the computer screen, and the six letters from the sequence appeared at the bottom of the screen in alphabetical order. Participants had to click on the letter that had been presented in the selected location. Once selected, the letter was surrounded by a black frame and remained visible on the screen for 500 ms. Participants pressed the space bar to initiate the next trial. The procedure is illustrated in Fig. 4.

Fig. 4

Illustration of the probe recall task used in Experiments 3 and 4

Half of the sequences were followed by a verbal–spatial recall, and the other half by a spatial–verbal recall. In each conditions, 50 trials were presented, half of which contained similar letters, while the other half contained dissimilar letters. Each serial position (2–6) was probed in the same number of trials. The trials in all conditions (similar/verbal–spatial recall, similar/spatial–verbal recall, dissimilar/verbal–spatial recall, and dissimilar/spatial–verbal recall) were presented in a different random order for each participant. The experimental trials were preceded by four practice trials. The experiment lasted 25 min.

Results

We computed the proportions of verbal and spatial features recalled correctly as a function of their recalled order—that is, whether the feature was recalled first or second. We first analyzed the proportions of trials in which the verbal feature was correctly recalled, to see whether we would observe the typical phonological similarity effect. Then we analyzed the recall of the spatial feature, to examine whether the detrimental effect of phonological similarity would spread to spatial memory.

Verbal recall

Figure 5 (left panel) illustrates the proportions of correct recall when the verbal feature was recalled first—in the verbal–spatial recall condition—and when it was recalled second—in the spatial–verbal recall condition. Inspection of Fig. 5 suggests that phonologically similar letters were recalled more poorly than dissimilar letters, irrespective of their recalled order. A 2 (recalled order: first, second) ×2 (phonological similarity: similar, dissimilar) repeated measures ANOVA was performed on the proportions correct. The analysis confirmed that verbal recall was higher when the letter was recalled first than when it was recalled second, F(1, 24) = 18.59, MSE = 0.02, p < .001, η ² _p = .44, and higher for dissimilar- than for similar-sounding letters, F(1, 24) = 8.33, MSE = 0.01, p = .008, η ² _p = .26. The interaction between recalled order and phonological similarity was not significant, F < 1.

Fig. 5

Proportions of correct responses and of conditional responses (CR) as a function of recalled order and phonological similarity for verbal (left) and spatial (right) features. Errors bars represent 95 % confidence intervals

The analysis indicated that even when the verbal feature was recalled second—through its spatial location—phonological similarity affected verbal recall. However, this result is difficult to interpret, because errors during the recall of the second feature might be due to the fact that the first feature had not been recalled correctly. In order to clarify the effect of phonological similarity on binding, we measured conditional responses—that is, the proportions of correct responses for the second feature, given that the first feature has been recalled correctly. Conditional responses are presented in Fig. 5 (second CR). A repeated measures ANOVA performed on the conditional responses, with Phonological Similarity (similar, dissimilar) as the factor, showed that the difference between dissimilar- and similar-sounding letters was not significant, F(1, 24) = 3.07, MSE = 0.01, p = .093, η ² _p = .11.

To understand further the processes involved during retrieval of the bound representations, we computed the types of responses provided by the participants during recall of the second feature when the first feature had not been recalled correctly. Because some participants committed very few errors, the proportions of each type of response (not averaged across participants) were calculated. As is shown in Table 1 (verbal feature recalled second), when the spatial location was not recalled correctly, the letter associated with the erroneously recalled location was recalled 32 % of the time, and the letter that followed the probe location—that is, the correct response—was recalled 18 % of the time (chance level = 20 %). Therefore, when participants recalled the wrong location, they tended to recall the letter associated with the recalled location rather than the correct letter, and this tendency was more pronounced for dissimilar than for similar letters.

Table 1 Proportions of associated and correct second features recalled following an error in recalling the first feature

Spatial recall

The right panel of Fig. 5 shows the proportions of correct recall when the spatial feature was recalled first—in the spatial–verbal recall condition—and when it was recalled second—in the verbal–spatial recall condition. As is shown in Fig. 5, locations marked by phonologically similar letters seemed to be more difficult to recall than locations marked by dissimilar letters, irrespective of their recalled order. A 2 (recalled order: first, second) ×2 (phonological similarity: similar, dissimilar) repeated measures ANOVA confirmed that spatial recall performance was higher when the spatial feature was recalled first than when it was recalled second, F(1, 24) = 59.49, MSE = 0.01, p < .001, η ² _p = .71, and higher for dissimilar- than for similar-sounding letters, F(1, 24) = 8.94, MSE = 0.01, p = .006, η ² _p = .27. The interaction between recalled order and phonological similarity was not significant, F < 1. Although spatial recall was modulated by phonological similarity when the spatial feature was recalled second, an analysis of the conditional responses showed that the difference between dissimilar- and similar-sounding letters was not significant, F < 1 (see Fig. 5), suggesting that when the letter was recalled correctly, locations marked by similar and dissimilar letters were recalled equally well.

We also computed the types of responses when recall of the second feature was required and the first feature had been recalled incorrectly. When the letter had not been recalled correctly, the location associated with the erroneously recalled letter was recalled 36 % of the time, and the location that followed the probe letter—that is, the correct response—was recalled 20 % of the time. As is shown in Table 1 (spatial feature recalled second), phonological similarity seemed to reduce the tendency to recall the associated location following the erroneous recall of the letter.

Discussion

As in Experiment 1, phonological similarity was detrimental to the recall of letters. The effect of phonological similarity on verbal recall was observed irrespective of whether letters were recalled directly—as a first feature—or via their locations. Phonological similarity also modulated memory for spatial locations, suggesting that the recall of locations depends on the identity of the letters. Once the identity of the letter has been retrieved, however, memory for its location is not modulated by phonological similarity, as indicated by the analysis of the conditional responses on spatial recall. This suggests that errors during spatial recall occur mostly because of difficulty in retrieving the associated letter. Our results also indicated that when participants committed an error when recalling the first feature, they tended to recall the associated feature second, rather than the correct feature, suggesting that participants remember the bindings between features rather than two independent streams of features. The associated feature was recalled more often for dissimilar than for similar sequences, however, suggesting that phonological similarity disrupted the binding between the verbal and spatial features.

Experiment 4

Experiment 4 was designed as a further test of symmetry: As in Experiment 2, we manipulated path complexity. If letters can be recalled independently of their spatial location, path complexity should not impair verbal recall.

Method

Participants

A group of 14 undergraduate students from Université de Moncton volunteered to participate in this experiment.

Materials

In each trial, the six letters were selected randomly from the letters of the alphabet (except for the vowels and W). We manipulated the path length and the number of crossings between the successive locations of a sequence. Five hundred sequences of six locations were first generated, with the restriction that each location was separated from another location in the same list by at least 4.5°. From these, 100 sequences were selected for the experiment. On half of the trials, the sequences contained no or one crossing (M = 0.08, SD = 0.27), and the total distance between successive locations varied between 40.7° and 53.0° (low spatial complexity). On the other half of the trials, the sequences contained between three and six crossings (M = 3.90, SD = 0.81), and the total distance between successive locations varied between 50.7° and 66.2° (high spatial complexity).

Design and procedure

The design and procedure were the same as in Experiment 3.

Results

We first analyzed the proportions correct for spatial recall. Then we analyzed verbal recall to examine whether spatial complexity modulates verbal memory.

Spatial recall

Inspection of Fig. 6 (left panel) suggests that spatial complexity impaired spatial recall, irrespective of recalled order. A 2 (recalled order: first, second) ×2 (spatial complexity: low, high) repeated measures ANOVA confirmed that the proportion correct was higher when the spatial feature was recalled first than when it was recalled second, F(1, 13) = 31.27, MSE = 0.02, p < .001, η ² _p = .71, and higher for low-complexity than for high-complexity sequences, F(1, 13) = 52.13, MSE = 0.01, p < .001, η ² _p = .80. The interaction between recalled order and spatial complexity was not significant, F(1, 13) = 3.45, MSE = 0.03, p = .086, η ² _p = .21. The analysis of the conditional responses showed that when locations were recalled through letter identity, the difference between the high- and low-complexity sequences was not significant, F < 1 (see Fig. 6).

Fig. 6

Proportions of correct responses and of conditional responses (CR) as a function of recalled order and spatial complexity for spatial (left panel) and verbal (right panel) features. Errors bars represent 95 % confidence intervals

As is shown in Table 1 (spatial feature recalled second), the probabilities of recalling the location associated with the incorrectly recalled letter were relatively similar between the high- and low-complexity sequences, suggesting that spatial complexity did not affect the binding between the two features. Interestingly, however, the locations comprised in low-complexity sequences seemed to be easier to recall correctly following erroneous recall of the letter, as compared to high-complexity sequences.

Verbal recall

The verbal recall performance is illustrated in Fig. 6 (right panel). A 2 (recalled order: first, second) ×2 (spatial complexity: low, high) repeated measures ANOVA indicated that the proportion of correct responses was higher when the verbal feature was recalled first than when it was recalled second, F(1, 13) = 31.17, MSE = 0.02, p < .001, η ² _p = .71, and higher for low-complexity than for high-complexity sequences, F(1, 13) = 6.91, MSE = 0.01, p = .021, η ² _p = .35. Importantly, the interaction between recalled order and spatial complexity was significant, F(1, 13) = 8.26, MSE = 0.01, p = .013, η ² _p = .39: Paired-samples t tests showed that when the verbal feature was recalled second, low-complexity sequences were recalled better than high-complexity sequences, t(13) = –4.37, p < .001. Spatial complexity did not modulate recall performance when the verbal feature was recalled first, t(13) = 0.09, p = .931. It is, therefore, possible that spatial complexity modulated verbal recall when the spatial feature was recalled first only because the spatial feature was more difficult to recall in the high-complexity sequences. This was confirmed by an analysis of the conditional responses, which showed that when the spatial feature was correctly recalled, there was no significant difference between the verbal recall for high- and low-complexity sequences, F < 1 (see Fig. 6).

As is shown in Table 1 (verbal feature recalled second), path complexity did not seem to influence the tendency to recall the associated letter, following erroneous recall of the location. As was the case for spatial recall, however, the letters in low-complexity sequences were recalled better than those in high-complexity sequences, even following erroneous recall of the spatial location.

Discussion

As in Experiment 2, path complexity affected the recall of spatial locations (see, e.g., Parmentier et al., 2005) but had no effect on recall of the letters. To our knowledge, this is the first demonstration of a complexity effect with a probed recall procedure. This suggests that the effect of complexity is not due to the difficulty of retrieving and recalling the complex patterns, but is likely to result from encoding or rehearsal processes. Combined with the results of Experiment 3, these results suggest that even when participants must recall both features of an item in each trial, binding is asymmetric: Locations are bound to the identities of letters, so that manipulating the letters’ phonological similarity impairs spatial recall, whereas letters can be recalled independently of their locations, so that spatial complexity has no effect on verbal recall. Interestingly, when locations were retrieved from the recalled letter, the effect of path complexity on spatial recall was abolished. The additional data presented in Table 1 show that following erroneous recall of the first feature, path complexity did not influence the tendency to recall the associated feature, suggesting that bindings were not modulated by path complexity. The second feature was, nevertheless, recalled better in the low-complexity sequences. This might suggest that low-complexity sequences were generally easier to recall, which facilitated recall of the verbal feature. This facilitation, however, does not seem to be due to binding, but might be attributed to an overall facility to retain the two features.

General discussion

The objective of the present series of experiments was to examine how verbal and spatial features are integrated in serial memory for visual information. In Experiment 1, we replicated the findings of Guérard et al. (2009a) using an order reconstruction procedure: In line with their results, we found that the recall of spatial locations was influenced by the identity of the letters at those locations. Experiment 2 showed that letters could be recalled independently of their locations, since an effect modulating spatial recall had no effect on verbal recall. This asymmetry was found irrespective of whether participants had to retain both features of the stimuli or a single feature; that is, phonological similarity modulated spatial memory even when participants did not have to pay attention to the letters. Asymmetric binding was also replicated in Experiments 3 and 4 using a probe serial recall task in which participants were required to recall the verbal and spatial features of the item following a probe.

Asymmetric versus symmetric binding

Overall, the four experiments showed that an effect associated with the retention of verbal features modulates memory for spatial locations, but that an effect associated with the retention of spatial locations does not alter verbal memory. Such an asymmetry is contrary to previous studies on visual memory that have shown asymmetric binding in which spatial locations could be retained independently of object identity (see, e.g., Campo et al., 2010). Rather, our results are consistent with those of Maybery et al. (2009), who showed that in auditory memory for a single feature, spatial recognition, but not verbal recognition, is modulated by whether or not the probe letter and its location were presented together in the list.

Maybery et al. (2009) suggested that the discrepancy between studies in visual memory and their results with regard to the direction of the asymmetry was likely due to presentation modality. For instance, locations might be a more central feature in the visual modality, whereas the verbal identity would play a more important role in the auditory modality. The fact that we observed the same asymmetry using visual information suggests that modality might not be the critical factor for modulating the direction of the asymmetry. Rather, we suggest that one factor more likely to influence the direction of the asymmetry is whether items are presented simultaneously or sequentially. Indeed, Zimmer, Speiser, and Seidler (2003) suggested that different processes might be involved in retaining dynamic spatial information as compared to a static display. In line with this idea, all studies showing that spatial locations were automatically bound to object identity have used simultaneous presentation (see, e.g., Campo et al., 2010; Jiang et al., 2000; Meegan & Honsberger, 2005; Olson & Marshuetz, 2005; Poch et al., 2010; Treisman & Zhang, 2006). This could occur because participants encode the configuration of the display during simultaneous presentation. In such a case, location information might be more likely to be the primary feature (Jiang et al., 2000). On the contrary, the studies showing that letters could be retained independently have used sequential presentation (Guérard et al., 2009a; Maybery et al., 2009). In this case, the configuration might be more difficult to memorize, and items might be more likely to be rehearsed as lists, so that verbal features might be retained independently of their locations.

In order to explain how features are retained as bound representations in memory, Baddeley et al. (2011) recently proposed a new version of the working memory model. They suggested that features within bound objects are represented at both the feature level and the object level. In this revised version of the model, the episodic buffer was defined as a “purely passive system . . . that serves a crucial integrative role because of its capacity to bind information from a number of different dimensions into unitized episodes or chunks” (Baddeley et al., 2011, p. 1399). Therefore, the binding between spatial and verbal features would occur in the episodic buffer. The unified representations would be retained in the episodic buffer, and the spatial and verbal features could be retained separately in the visuospatial sketchpad and phonological loop.

Two aspects of our results appear to be problematic for this new version of the model. First, if locations are retrieved from the episodic buffer, they should be immune to any effect associated with memory for letters, such as the phonological similarity effect, since these effects occur at the level of the phonological loop—which is responsible for the retention of verbal information. Second, Baddeley et al.’s (2011) revised model predicts no asymmetry, because both letters and locations can be accessed independently. However, asymmetric binding has been observed in numerous studies (see Campo et al., 2010; Jiang et al., 2000; Maybery et al., 2009; Poch et al., 2010), including ours.

One hypothesis that could explain asymmetric binding is that both features of an object are stored together, but that one feature is primary, around which associated features are organized (see, e.g., Maybery et al., 2009). According to this idea, location information in these serial presentation paradigms would be appended to letter information. One location could therefore be accessed through the retrieval of the letter, whereas a letter could be retrieved without any reference to its spatial location. In line with this hypothesis, we observed that when participants first recalled a spatial location following a probe, their performance was influenced by phonological similarity. When the location was retrieved through the associated letter, however—that is, when it was recalled second—spatial memory was not modulated by phonological similarity (Exp. 3). This suggests that the effect of phonological similarity on spatial memory occurs because participants recall locations by first retrieving the associated letter, but that location information is then retrieved as easily from similar as from dissimilar letters. In Experiment 4, we also showed that path complexity affected spatial memory when the spatial feature was recalled first. When the location was recalled through its associated letter, however, spatial memory was not modulated by path complexity, suggesting that the spatial location tagged to a given letter is retained independently of the other locations.

Automatic versus voluntary binding

Some studies have suggested that binding can occur automatically as attention is paid to an item (see, e.g., Allen, Baddeley, & Hitch, 2006; Guérard et al., 2009a) and that spatial locations are bound to object identities, even if locations do not have to be retained (see, e.g., Jiang et al., 2000; Olson & Marshuetz, 2005; Poch et al., 2010; Treisman & Zhang, 2006). Maybery et al. (2009) suggested that such automatic binding would yield an asymmetry, whereas voluntary binding would yield a symmetric pattern. However, our results are inconsistent with this idea. In Experiments 1 and 2, we observed asymmetric binding irrespective of whether participants had to retain both features or a single feature. Moreover, in Experiments 3 and 4, in which participants had to retain bindings because the second feature could only be recalled through the first feature recalled, the same asymmetric pattern was found. This suggests that voluntary binding does not produce a symmetric effect, at least in serial memory for visual information.

The finding that the levels of performance differed between the two-feature and single-feature groups in Experiments 1 and 2 might suggest that the binding mechanisms differed between these conditions. For instance, binding in the two-feature group might have been more effortful than binding in the single-feature group, leading to a memory decrement. Alternatively, the decrease in performance observed in the two-feature groups might suggest that participants in this group retained two streams of features in addition to the automatic binding, which occurred in similar manners in the two groups. Importantly, the asymmetry was observed irrespective of whether participants had to retain a single feature or two features. Therefore, other factors than automaticity might modulate the relative importances of the two features, such as presentation format (i.e., sequential vs. simultaneous) or task requirements (i.e., item vs. order memory). For instance, the requirement to retain order information in the present experiments might have led to asymmetric binding, even when participants had to retain both features because of their reliance on verbal rehearsal.

Conclusions

The present study has shown asymmetric binding in serial memory for visual information when letters are the primary features. Our results diverge from those of studies using simultaneous presentations in the visual modality, in which the configuration of the display could be retained independently of the objects’ identities (see, e.g., Jiang et al., 2000). Although the asymmetry that we observed is consistent with that observed in the auditory modality (see, e.g., Maybery et al., 2009), it seems to occur during both voluntary and automatic binding. Serial memory might therefore tap into different processes that affect how the different features are integrated.