Empirical validation of the diffusion model for recognition memory and a comparison of parameter-estimation methods

The diffusion model is designed for fast binary choices (with mean reaction times faster than about 1,500 ms). It utilizes the information available from the participant’s responses in the best possible way. That is, it considers not only mean reaction times and accuracy, but also relative speed of false and correct responses, and the shape of reaction-time distributions (Ratcliff et al., 2004).

The main idea underlying the diffusion model is shown in Fig. 1. Confronted with a binary choice task like old–new recognition, a participant will start accumulating internal evidence for the decision. Depending on the relative amounts or quality of information favoring one of the options, the evidence will drift to one of two decision boundaries, and the process will terminate in a decision for one option when one of the boundaries is crossed. The drift towards a boundary is modelled as a diffusion process, which is the continuous generalization of a random walk. There are slight differences in parameter labels in the literature. We use the labels used by Voss and Voss (2007, 2008). However, they are easily translated to other labels.

Open image in new window

The drift rate v represents the quality of the information extracted from the stimuli (Ratcliff et al., 2004). The drift rate v depends on the degree of match between a memory probe and information stored in memory (Ratcliff, 1978; Ratcliff & Starns, 2009). It describes the information accumulation per time unit and is, therefore, the average gradient, that is, the mean rate of approach to one of the thresholds. Positive values indicate an approach to the upper threshold, whereas negative values indicate an approach to the lower threshold. The absolute value describes the speed of information accumulation. The higher the absolute value is, the faster the corresponding threshold is reached, and the less likely it is that the response opposite to the drift rate––which is often wrong––is chosen (Voss, Rothermund, & Voss, 2004). Every item in a recognition test has its own drift rate. The drift rate is assumed to be normally distributed with mean v and standard deviation s_v (Ratcliff et al., 2004).

The distance between the decision boundaries (threshold parameter a), on the other hand, defines how much information a participant needs before making a decision (Voss et al., 2004). The upper threshold a is the criterion for responding old in a recognition-memory test. Conventionally, the lower boundary is set to zero. Therefore, the value of the upper threshold a is a measure of the distance between the thresholds. Obviously, the decision boundaries also affect accuracy, because wider distances reduce the probability of the process diffusing across the “wrong” boundary by chance. Thus, v and a both contribute to performance, and a is believed to be affected by speed–accuracy trade-offs (Ratcliff, 1978). A higher threshold parameter a indicates that a person needs more information to make a decision. This leads to a higher rate of correct, but on average slower, responses (Ratcliff et al., 2004).

The starting point z of information accumulation describes possible asymmetries in the amount of information that is needed to exceed the response criteria for old versus new responses. If z equals a/2, there is no bias towards one response or the other. If z differs from a/2, the reaction times for old versus new responses will differ. The smaller the distance between starting point and threshold, the lower the reaction times will be. If z > a/2, less information is needed to exceed the upper threshold a. Thus, there is a bias towards the old response. If z < a/2, there is a bias towards the new response. It is assumed that the starting point varies between trials with a uniform distribution with mean z and range s_z (Ratcliff, 1978).

The model does not assume a linear process, but takes into account random influences that add to the constant influence of the drift rate. This explains why processes with the same drift rate can have different reaction times or even opposite responses (Ratcliff & Rouder, 1998). Random influences at time t are described by a normal distribution with mean 0. The variance increases with time. Increase in variance is represented by the diffusion constant s. s is a scaling parameter, fixed to any positive value.

In addition, there are other processes contributing to reaction time, such as motor processes and stimulus encoding. In the model, their total time is estimated as the response-time constant t₀. It contains the non-decisional proportion of the reaction time (Ratcliff, 1978). The total reaction time RT equals RT_decision + t₀. Like drift rate v and starting point z, this parameter differs between trials. t₀ is uniformly distributed with range s_t (Ratcliff et al. 2004). Ratcliff (2013) showed that, in most cases, these standard assumptions about the distributions of drift rate, starting point, and response-time constant lead to the same predictions as different distributional assumptions.

Since 1978, when the diffusion model was introduced, there have been some modifications in the use of the model. Ratcliff (1978) postulated that the process is self-terminating for matches, but exhaustive for non-matches. This implies that there are two different processes for the two boundaries. The upper boundary is reached when a match is found, and all other processes are then terminated. For the lower boundary, all processes must result in a non-match. According to Ratcliff (1978), recognition is best described by parallel processes. For each item in the search set, a comparison with the memory probe is running. The observed reaction time reflects only the maximum (for non-matches) or the minimum (for matches) of the diffusion processes (Ratcliff, 1978, 1988).

The model has also been used in paradigms other than recognition memory, where the assumption of a difference between the two boundaries is unnecessary because only one simultaneous comparison is assumed. In later descriptions of the diffusion model, there is no differentiation between descriptions of the diffusion model for recognition-memory experiments and descriptions of the diffusion model for other tasks. According to these descriptions, the response is initiated as soon as a boundary is reached (Ratcliff et al., 2004, 2007; Spaniol et al. 2006, 2008; White et al. 2009). Recently, the drift criterion has attracted some attention. The drift criterion can be seen as the zero point of the drift rate. It describes the amount of evidence above which evidence accumulates towards the upper threshold and below which evidence accumulates toward the lower threshold (Criss, 2010; Ratcliff, 1978, 1981, 1985, 1987; Ratcliff, Van Zandt, & McKoon, 1999). However, this parameter is not implemented in the available programs.

The aim of the parameter estimation is to find the optimal fit between theoretical and empirical reaction-time distributions and accuracy data. Therefore, formulas for the probability density functions (PDF) or the cumulative distribution function (CDF) for both thresholds are needed. For a detailed description and discussion of this topic, see Tuerlinckx, Maris, Ratcliff, and De Boeck (2001), Tuerlinckx (2004), and Ratcliff and Tuerlinckx (2002). To estimate the parameters, a criterion for the goodness-of-fit is needed. For a discussion of different criteria see Read and Cressie (1988), Ratcliff and Tuerlinckx (2002), and Voss et al. (2004). The parameter estimation of the diffusion model has no analytical solution. Therefore, to find the best fit, numerical integration procedures are implemented (Wagenmakers et al., 2007). Parameter estimation is quite complex and is a research topic of its own (Diederich & Busemeyer, 2003; Ratcliff & Tuerlinckx, 2002; Tuerlinckx, 2004; Wagenmakers et al., 2007; Vandekerckhove & Tuerlinckx, 2007). In recent years, some programs have been developed to make the diffusion model easy to use: EZ-diffusion model (Wagenmakers et al., 2007), DMAT (Vandekerckhove & Tuerlinckx, 2008), and fast-dm (Voss & Voss, 2007). Vandekerckhove et al. (2011) developed a hierarchical extension of the diffusion model.

Van Ravenzwaaij and Oberauer (2009) tested the parameter recovery of fast-dm, DMAT, and EZ with simulated data. They calculated correlations between the true values and the estimated parameter values. All methods were able to estimate the parameters with reasonable accuracy. Fast-dm seemed to be the least robust method for parameter estimation. This was due to an incapability of recovering individual differences for the dispersion parameters s_v and s_z, and a tendency to yield smaller differences between conditions, especially for the drift rate, with a small number of trials.

DMAT requires a large number of trials. In contrast, EZ and fast-dm provide useful estimates with about 80 trials per condition (Van Ravenzwaaij & Oberauer, 2009). In our experiments, we had a relatively small number of trials per condition because we wanted to mimic standard conditions of a recognition-memory experiment. For a small number of trials, Van Ravenzwaaij and Oberauer (2009) found that EZ was most robust. However, the parameter z is fixed in this model, and since we also wanted to validate this bias parameter, we used fast-dm and DMAT to estimate the parameters and cross-checked the results with EZ for the two experiments not targeting the bias parameter (Experiments 2 and 3).

The EZ-diffusion model (Wagenmakers et al., 2007) is an algorithm that was developed to make data analyses with the diffusion model as easy as possible. It transforms accuracy and the mean and variance of the reaction times of correct responses into drift rate v, threshold parameter a, and response-time constant t₀ via three equations. As an advantage, these equations do not require any parameter fitting and can be used even if the error rate is very small. To achieve this, the model makes some simplifications. That is, (1) it assumes there is no between-trial variability, and thus, s_v, s_z and s_t are set to zero. (2) The starting point is assumed to be unbiased, and thus, z/a is set to 0.5.

Fast-dm (Voss & Voss, 2007) uses the partial differential equation (PDE) method to compute the CDF (Voss & Voss, 2008) and the Kolmogorov–Smirnov test (KS test; Kolmogorov, 1941) to estimate the parameters and determine the model fit. The PDE method avoids infinite sums and has the advantage of evaluating all starting points at the same time, thus reducing computing time (Voss & Voss, 2008). The KS test uses the test statistic T as the optimization criterion, and parameters are chosen such that T is minimized. The reaction-time distributions of both thresholds are estimated together by giving the reaction times of the lower threshold a negative sign. The parameter space is searched via the simplex method (Nelder & Mead, 1965) to obtain the best model fit. Starting points for v, a, and t₀ are provided by the EZ model (Wagenmakers et al., 2007). Realistic values are chosen as starting points for the other parameters.

The diffusion model analysis toolbox (DMAT; Vandekerckhove & Tuerlinckx, 2008) is a Matlab toolbox with a graphical user interface. It uses design matrices to obtain parameter estimates. Chi-square and maximum-likelihood estimates are available for parameter estimation and goodness-of-fit tests.

As described above, fast-dm and DMAT use different test statistics. Each statistic has several advantages and disadvantages, and the authors of the programs motivated the choice of their statistics differently. Voss and Voss (2007) chose the KS test because it does not aggregate data and, thus, does not lose information. Additionally, it is not affected by outliers as much as the maximum-likelihood and the Chi-square statistic. The Chi-square statistic is more robust and faster than the maximum-likelihood method (Ratcliff & Tuerlinckx, 2002). Chi-square and maximum-likelihood methods are commonly used for parameter estimation.

In applications of the diffusion model reported by Ratcliff and colleagues, s was usually set to 0.1 (e.g., Ratcliff, 1978, 1988, 2002; Ratcliff & Rouder, 1998, 2000; Ratcliff et al., 2001, 2003, 2004, 2006a, b, 2007). DMAT and EZ set s = 0.1 by default. In applications of the model reported by Voss and colleagues, s was usually set to 1 (Voss et al., 2004; Spaniol et al. 2006, 2008). The fast-dm program (Voss & Voss, 2007) also uses a diffusion constant of 1. However, parameters that were obtained via computations based on other diffusion constants can simply be transformed by multiplying all parameters (except t₀) by the desired diffusion constant. We converted the fast-dm results to s = 0.1 to make the results more comparable.

When first publishing the diffusion model, Ratcliff applied it to several recognition-memory paradigms including the old–new paradigm used here. He showed that the drift rate accounted for primacy and recency effects (Ratcliff, 1978). Since 1978, the model has been applied in many studies of recognition memory, yielding insights into the underlying dynamics of the process (e.g., Ratcliff, 1978, 2006; Ratcliff et al., 2004; Spaniol et al., 2006; Thapar et al., 2003) and having far-reaching implications, for example falsifying the global slowing hypothesis of cognitive aging (Wagenmakers, 2009).

There are also neuroscientific studies that support the model’s fit to data. Ratcliff, Cherian, and Segraves (2003) examined macaques via the moving-dot paradigm. In this paradigm, there are several dots moving randomly. Among them, however, are some dots that move simultaneously. The task is to identify the dots that move simultaneously. Ratcliff et al. (2003) showed that the macaques’ behavior as well as their neuronal activity could be fitted by the diffusion model. The fit of behavioral data from the moving-dot paradigm (Julesz, 1971) was also shown for humans (Ratcliff & McKoon, 2008).

These studies supported the model because the model fitted the data well, and they were able to explain a range of phenomena. However, interpreting parameter estimates as measures of cognitive processes requires construct validity of the measurement model in the sense of Cronbach and Meehl (1955). That is, the measures must show convergent as well as discriminant validity. Convergent validity is assessed by a measure’s covariation with related constructs, whereas discriminant validity refers to the lack of covariation with unrelated constructs. Measures are “process-pure” to the extent they show both types of validity. A systematic experimental validation assessing both types of validity is lacking in the realm of recognition memory. Parameter estimates are mathematical abstractions, and a systematic empirical justification of their psychological interpretation is indispensable.

In the perceptual domain, a systematic experimental validation of the diffusion model was conducted by Voss et al. (2004), using a color discrimination task. In a first experiment, Voss et al. manipulated variables to affect the drift rate v, the threshold parameter a, and the response-time constant t₀. Their participants had to decide whether a dot stimulus was dominated by orange or by blue dots. There were four conditions, namely one standard condition and three other conditions that each targeted one specific model parameter. Task difficulty was increased to decrease the drift rate (difficult condition). An instruction to be very accurate was aimed at increasing the threshold parameter a exclusively (accuracy condition). Finally, by allowing participants to press the response keys with one finger only, the authors strove to increase the response-time constant t₀ (handicap condition). They found the predicted pattern. That is, higher task difficulty decreased drift rate, accuracy instructions led to a higher threshold parameter, and the handicap condition led to an increased response-time constant t₀. However, the authors also found unexpected results. In the accuracy condition, the t₀ parameter was higher than in the standard condition. In the handicap condition, the drift rate for blue dominated stimuli v_blue and the starting point z/a differed significantly from those in the standard condition. The increased t₀ parameter was easily explained because if participants have more time to respond they execute their responses more slowly. Differences in drift rate and starting point in the handicap condition, however, could not be explained that easily. However, all individual models revealed good model fit as assessed via the goodness-of-fit statistic T (see Voss et al., 2004, for a detailed description). In a second experiment, Voss et al. (2004) manipulated the starting point by promoting one response over the other. They found that the starting point was biased towards the promoted response. Overall, the models described the empirical data well. The authors concluded that the parameters of the diffusion model represent the process components of the perceptual task well. The study supported the convergent and partly the discriminant validity of the diffusion-model parameters in the perceptual domain.

Additional support for the model’s validity came from Ratcliff and Rouder (1998) for psychophysical tasks and from Wagenmakers, Ratcliff, Gomez, and McKoon (2008) for the lexical-decision task. They showed that accuracy instructions increased the threshold parameter a, and that easier stimuli have higher drift rates. Wagenmakers et al. (2008) showed that unequal presentation proportions affected not only the starting point but also the boundary separation.

The aim of the present study was to provide a similar test of the model’s validity in the recognition domain. In this article, we present three recognition-memory experiments each targeting one central model parameter. In Experiment 1, we manipulated the ratio of old to new items in the test (targeting bias parameter z). In Experiment 2, we manipulated the instructions for accuracy versus speed (targeting threshold parameter a). In Experiment 3, we used a manipulation to affect the quality of encoding (targeting drift-rate parameter v).

If each manipulation affects the predicted parameter in the expected direction without influencing other parameters, this would be strong support for the validity of the model. Therefore, we tested if experimental manipulations targeting the process components of the diffusion model affected the corresponding parameters (convergent validity) and only these (discriminant validity).

Ratcliff (1978) advised against between-subject designs because in such designs, differences in reaction times may be due to between-group differences in speed–accuracy criteria (threshold parameter a). However, some variables cannot be experimentally manipulated within participants, such as, for example, the age variable in studies of cognitive aging (e.g., Spaniol et al., 2006). Hence, it is useful to know if the model is valid for both types of design. We, therefore, tested model validity using within-subject designs (Experiment 3) as well as between-subject designs (Experiments 1 and 2).

We analyzed the data with three different methods: fast-dm (Voss & Voss 2007), EZ (Wagenmakers et al., 2007), and DMAT (Vandekerckhove & Tuerlinckx, 2008). Van Ravenzwaaij and Oberauer (2009) compared these methods with simulated data. For individual differences, they found that EZ did better than fast-dm and DMAT, and that there was no consistent difference between fast-dm and DMAT regarding the correlation with the true values that generated the data. Fast-dm and DMAT both had difficulties with the dispersion parameters which are not estimated by EZ. Regarding parameter means, EZ showed a small bias to underestimate drift rate and non-decision time and to overestimate the threshold parameter. However, it covered the mean structure of the data and showed mean parameter differences between conditions in the expected direction. DMAT showed the smallest bias, but underestimated response-time constant t₀, and overestimated drift rate and boundary separation. It covered group differences well. Fast-dm showed the largest bias and showed smaller group differences than there were in the simulated data sets. Van Ravenzwaaij and Oberauer concluded that all three methods show reasonable accuracy when they have sufficient data points. DMAT required a large number of data points, whereas EZ and fast-dm needed only 80 data points to produce reasonable estimates. EZ and DMAT proved better at detecting group differences. Thus, it is not easy to decide which toolbox to use. EZ seems to be very accurate but cannot detect differences in the bias parameter. DMAT is better than fast-dm at detecting group differences but needs more trials to yield reasonable estimates.

The aim of our study is similar to that by Voss et al. (2004) in that we experimentally evaluated the validity of the diffusion model. While Voss and colleagues validated the model in the perceptual domain, we evaluated its validity for recognition-memory experiments. There is no a priori reason to believe that perceptual evidence accumulation and retrieval from memory follow the same laws. Hence, an assessment of construct validity is necessary in both domains. Additionally, our work is similar to the work by van Ravenzwaaij and Oberauer (2009) in the sense that it compares different methods for estimating diffusion-model parameters. Unlike van Ravenzwaaij and Oberauer, we did not simulate data but we analyzed our data with all three toolboxes to perform a systematic comparison of the three methods with experimental data. Our experiments were typical recognition experiments and did, hence, not provide perfect conditions for data analysis with the diffusion model. For example, we used relatively few trials (resulting in relatively few error responses) compared with a lexical-decision task or a perceptual task. Hence, we examined the performance of the three methods in compromised fitting situations.

The first experiment tested the validity of the starting-point parameter z, using a standard response-bias manipulation, namely the manipulation of the ratio of old to new items in the test (e.g., Bröder & Schütz, 2009; Criss, 2010; Macmillan & Creelman, 2005; Starns, Ratcliff, & McKoon, 2012). Participants were informed about this ratio. Words were used as stimuli. We expected the ratio manipulation to affect the bias parameter z/a, exclusively. If there are more old words than new words in the test––and participants are aware of this––the starting point is expected to be biased towards the threshold for the old response. Accordingly, if there are more new words than old words in the test, the starting point is expected to be biased towards the threshold for the new response. If this manipulation specifically affects the bias parameter and not the other parameters, this would provide strong support for the diffusion model.

This response-bias manipulation was used by Rotello et al. (2006), for example. They found that participants adopted a lenient signal detection criterion when they were informed that the majority of the test items were old. The signal detection criterion resembles the bias parameter z/a of the diffusion model (Ratcliff & McKoon, 2008). Hit rate and false-alarm rate both increase as the proportion of old items increases (Criss, 2010; Rotello et al., 2006). Bröder and Schütz (2009) showed that this manipulation affected the bias parameters in SDT and a two high-threshold model in a similar fashion.

Results

Performance measures

Mean hit rates were 0.61 (SD = 0.14) in the new-bias condition and 0.71 (SD = 0.14) in the old-bias-condition, a significant difference, t(58) = −2.45, p = 0.02, d = 0.63. False-alarm rates were 0.17 (SD = 0.10) in the new-bias condition and 0.29 (SD = 0.15) in the old-bias condition, also a significant difference, t(58) = −3.65, p < 0.01, d = 0.94. The two groups did not differ in terms of SDT’s sensitivity parameter d′ (M (new-bias) = 1.35, SD (new-bias) = 0.54; M (old-bias) = 1.22, SD (old-bias) = 0.64) but differed significantly in the response criterion c (M (new-bias) = 0.38, SD (new-bias) = 0.41; M (old-bias) = 0.02, SD (old-bias) = 0.35), t(58) = 3.65, p < 0.01, d = 0.94. Mean reaction times showed no significant differences. They were 945 ms (SD = 0.14) in the new-bias condition and 978 ms (SD = 0.14) in the old-bias condition.

Parameter estimation and model fit

First, we performed parameter estimation and goodness-of-fit tests with the fast-dm program (Voss & Voss, 2007) and with DMAT (Vandekerckhove & Tuerlinckx, 2008). For each participant, we calculated one model with two different drift rates––one for old and one for new items. Each model was based on 210 trials (for the drift rates there were 140 and 70 trials, respectively). Following Voss et al. (2004), we excluded trials with reaction times below 300 ms and above 4,000 ms from analyses because Ratcliff and Tuerlinckx (2002) showed that outliers may have a strong effect on parameter estimation, and because after 4,000 ms, participants were reminded to answer faster. We excluded a total of 37 trials (<1 %). The upper threshold was associated with the old response; the lower threshold was associated with the new response and was set to 0. Thus, negative drift rates indicate an approach toward the new response, whereas positive drift rates indicate an approach toward the old response.

We estimated eight parameters per participant: the mean bias parameter z, the mean upper threshold a, the mean drift rate for old items v_old, the mean drift rate for new items v_new, the mean response-time constant t₀, the range of the bias parameter s_z, the range of the response-time constant s_t, and the standard deviation of the drift rates s_v. Like Voss et al. (2004), we present z/a instead of z because z/a is easier to interpret. A bias parameter of z/a = 0.5 represents an unbiased starting point. Values greater than 0.5 indicate a bias towards the old response; values lower that 0.5 indicate a bias towards the new response.

For fast-dm, the KS test showed a good fit for all individual models (p > 0.05). For DMAT, we used the Chi-square method with default bins to estimate parameters and to calculate the model fit. The Chi-square test showed good model fit for 57 models and bad model fit for the remaining three individual models. We only included models with sufficient model fit (i.e., p > 0.05). Since some participants made very few mistakes, we encountered several warnings with DMAT. However, we included the parameter estimates in the analysis when they had reasonable fit. As this experiment was designed to target the bias-parameter z/a, we did not analyze the data with the EZ method because in EZ, z/a is set to 0.5.

Parameter analyses with fast-dm

The significance level was set to 0.05 for all our tests. Drift rates were significantly steeper for new items than for old items in both conditions (new-bias: M (old) = 0.05, SD (old) = 0.08, M (new) = 0.14, SD (new) = 0.06, t(29) = −5.60, p < 0.01, d = 1.02; old-bias: M (old) = 0.04, SD (old) = 0. 05, M (new) = 0.14, SD (new) = 0.07, t(29) = −6.41, p < 0.01, d = 1.17). To test the influence of the manipulation, we conducted independent-samples t tests for each parameter. As predicted, the bias-parameter z/a was significantly higher in the old-bias condition than in the new-bias condition, M (old-bias) = 0.66, SD (old-bias) = 0.09, M (new-bias) = 0.49, SD (new-bias) = 0.10, t(58) = −7.11, p < 0.01, d = 1.84. As z/a = 0.5 represents an unbiased starting point, both conditions should differ significantly from this neutral point. The bias parameter for new items was not significantly different from 0.5, t(29) = −0.69, p = 0.49, d = 0.01, which suggests that contrary to prediction there was no bias in the starting point. In the old condition, the bias-parameter was significantly higher than 0.5, t(29) = −9.86, p < 0.01, d = 1.78, as predicted.

Contrary to predictions, the threshold parameter a also differed significantly between conditions, M (old-bias) = 0.14, SD (old-bias) = 0.02, M (new-bias) = 0.13, SD (new-bias) = 0.02, t(58) = −2.13, p = 0.04, d = 0.55. Participants in the old-bias condition showed a larger value of the threshold parameter than participants in the new-bias condition. Thus, the former were more conservative. The effect size (measured by Cohen’s d), however, was only about one-third of that of the bias parameter. No other comparison yielded significance (all p > 0.05). Averaged mean parameter estimates are shown in Fig. 2.

Open image in new window

Parameter analyses with DMAT

There was no significant difference between the absolute value of the drift rates for old and new items in either condition (all p > 0.05) Again, the bias-parameter z/a was significantly higher in the old-bias condition than in the new-bias condition, M (old-bias) = 0.60, SD (old-bias) = 0.16, M (new-bias) = 0.46, SD (new-bias) = 0.13, t(58) = −3.70, p < 0.01, d = 0.99. The bias parameter of the old-bias condition differed significantly from 0.5, p < 0.01, d = 0.65, but the bias parameter in the new-bias condition did not, p = 0.12, d = 0.31. No other comparison yielded significance (all p > 0.05). Averaged mean parameter estimates are shown in Fig. 3.

Open image in new window

In Experiment 2, we used the same materials and similar procedures as in Experiment 1. The aim of this experiment was to test the validity of the threshold parameter a. Participants received different kinds of feedback depending on their experimental condition. In the accuracy condition, participants received negative feedback if they made a mistake. In the speed condition, participants received negative feedback if they responded more slowly than within 1,000 ms. This manipulation was expected to lead to an adjustment of thresholds. Participants in the accuracy condition should adopt more conservative criteria and thus have a higher threshold parameter than participants in the speed condition. Ratcliff et al. (2004) used a similar manipulation as a within-subject manipulation, but they fixed the other model parameters between the conditions and compared the results for young and older adults. They showed that the model captured the effect of speed and accuracy instructions with only threshold parameter a changing.

Results

Performance measures

Mean hit rates were 0.61 (SD = 0.15) in the speed condition and 0.69 (SD = 0.14) in the accuracy condition, a significant difference, t(58) = −2.08, p = −0.04, d = 0.54. False-alarm rates were 0.29 (SD = 0.15) in the speed condition and 0.29 (SD = 0.13) in the accuracy condition (n.s.). The two groups neither differed in terms of SDT’s sensitivity parameter d′ [M (speed) = 0.93, SD (speed) = 0.62; M (accuracy) = 1.14, SD (accuracy) = 0.73] nor in the response criterion c [M (speed) = 0.16, SD (speed) = 0.34; M (accuracy) = 0.03, SD (accuracy) = 0.24]. Mean reaction times were 683 ms (SD = 0.08) in the speed condition and 1,312 ms (SD = 0.40) in the accuracy condition. Thus, in the speed condition, participants were significantly faster, t(58) = −8.45, p < 0.01, d = 2.18.

Parameter estimation and model fit

We used the same parameter estimation procedure as in Experiment 1. We excluded 57 trials (<1 %). With fast-dm, only one individual model had to be excluded, because the KS test indicated a significant difference between the empirical and the predicted distribution. A binomial test revealed that the probability of finding one or more significant tests by chance was p = 0.19. Thus, the results indicate that overall, the model fitted the data well.

With DMAT, the Chi-square test showed good model fit for 53 models and poor model fit for the remaining seven individual models. We only included models with sufficient fit (i.e., p > 0.05). Since some participants made very few mistakes, we encountered several warnings with DMAT.

Parameter analyses with fast-dm

Like in Experiment 1, drift rates for new items were significantly steeper than those for old items (speed: M (old) = 0.04, SD (old) = 0.83, M (new) = 0.17, SD (new) = 0.11, t(29) = −6.96, p < 0.01, d = 1.27; accuracy: M (old) = 0.05, SD (old) = 0.56, M (new) = 0.09, SD (new) = 0.76, t(29) = 3.10, p < 0.01, d = 0.57). Again, we conducted an independent-samples t test for each parameter. In both conditions, the bias parameter z/a was significantly biased towards the old response (speed: t(29) = 3.20, p < 0.01, d = 0.60; accuracy: t(29) = 4.25, p < 0.01, d = 0.78).

Consistent with the hypothesis, the threshold parameter a was significantly higher in the accuracy condition than in the speed condition, M (speed) = 0.09, SD (speed) = 0.02, M (accuracy) = 0.17, SD (accuracy) = 0.06, t(33.44) = −8.37, p < 0.01, d = 2.13. We also found significant differences in the drift rate for new items, M (speed) = −0.17, SD (speed) = 0.11, M (accuracy) = −0.09, SD (accuracy) = 0.08, t(58) = −3.30, p < 0.01, d = 0.96, and in the response-time constant t₀, M (speed) = 0.54, SD (speed) = 0.08, M (accuracy) = 0.67, SD (accuracy) = 0.12, t(58) = −5.10, p < 0.01, d = 1.19. The effect size (measured as Cohen’s d) was about twice as large for the threshold parameter a as for drift rate v_new and response-time constant t₀. However, all effect sizes represent large effects according to Cohen (1988). No other difference was significant (all p > 0.05). Averaged mean parameter estimates are shown in Fig. 4.

Open image in new window

Parameter analyses with DMAT

Drift rates for new items were significantly steeper than those for old items only in the speed condition (speed: M (old) = 0.04, SD (old) = 0.15, M (new) = 0.26, SD (new) = 0.21, t(22) = −3.85, p < 0.01, d = 0.80; accuracy: M (old) = 0.12, SD (old) = 0.22, M (new) = 0.20, SD (new) = 0.20, t(29) = −1.33, p = 0.19, d = 0.35). Again, consistent with the hypothesis, the threshold parameter a was significantly higher in the accuracy condition than in the speed condition, M (speed) = 0.09, SD (speed) = 0.02, M (accuracy) = 0.21, SD (accuracy) = 0.15, t(30.721) = −4.79, p < 0.01, d = 1.24. We also found significant differences in the response-time constant t₀, M (speed) = 0.55, SD (speed) = 0.09, M (accuracy) = 0.72, SD (accuracy) = 0.18, t(43.91) = −4.33, p < 0.01, d = 1.15. No other difference was significant (all p > 0.05). Averaged mean parameter estimates are shown in Fig. 5.

Open image in new window

Parameter analyses with EZ

For the analysis with the EZ-diffusion model, we computed only one drift rate for old and new items by coding the old and new responses as correct and incorrect responses. The analysis showed significant differences in drift rate, M (speed) = 0.08, SD (speed) = 0.05, M (accuracy) = 0.05, SD (accuracy) = 0.03, t(46.94) = 2.58, p = 0.01, d = 0.73, and in the threshold parameter, M (speed) = 0.08, SD (speed) = 0.01, M (accuracy) = 0.17, SD (accuracy) = 0.05, t(31.42) = −8.66, p < 0.01, d = 2.50. This concurs with the fast-dm results except that there was no difference in the response-time constant t₀. Averaged mean parameter estimates are shown in Fig. 6.

Open image in new window

The purpose of Experiment 3 was to test the validity of the drift-rate parameter v. To affect drift rate, we varied encoding by manipulating the frequency of item presentation. We presented half of the items once and half of the items twice. According to Ratcliff (1978), two presentations will lead to two memory traces that both compete in a diffusion race, and the faster process “wins”, leading to higher accuracy and higher drift rates. In contrast to Experiments 1 and 2, we used a within-subjects manipulation. Ratcliff et al. (2004) used a similar manipulation, but they fixed the other model parameters between the conditions and compared the results for young and older adults. They showed that the model was able to capture changes in response-time distributions and accuracy as a function of word frequency and number of repetitions with only drift rate changing. To achieve stable parameter estimates in the three conditions of our within-subjects design, longer learning lists were necessary. We used pictures in this study, which are easier to remember than words (Paivio, 1971), to avoid floor effects due to longer lists. Also, the use of pictures in this experiment allowed us to generalize findings to other materials. Participants received no feedback. Items that were presented twice should have a higher drift rate than items that were presented only once. This manipulation should not influence the other parameters.

Results

Performance measures

Mean hit rates differed significantly between items presented once and items presented twice, M (presented once) = 0.58, SD (presented once) = 0.18, M (presented twice) = 0.77, SD (presented twice) = 0.18, t(27) = −9.34, p < 0.01, d = 1.81. The false-alarm rate was 0.13 (SD = 0.08) for not presented items. Mean reaction times were 818 ms (SD = 0.11) for items that were not presented, 829 ms (SD = 0.11) for items presented once, and 781 ms (SD = 0.09) for items presented twice, a significant effect, F(2,54) = 8,86, p < 0.01, η_p² = 0.25.

Parameter estimation and model fit

Again, parameters were estimated with fast-dm, DMAT, and EZ. For each participant, we calculated three separate models, one for each condition. Thus, each model was based on 90 trials. We excluded a total of 24 trials (<1 %) according to the same criteria as in the other experiments. We estimated seven parameters: the mean bias parameter z, the mean upper threshold a, the mean drift rate v, mean response-time constant t₀ and the ranges of bias parameter s_z and response-time constant s_t as well as the standard deviation of the drift rates s_v. The KS test used in fast-dm showed a good fit (p > 0.05) for all models. The Chi-square test used in DMAT showed a good fit for all calculated models. However, for some participants, DMAT failed to estimate parameter values in some conditions due to too few error responses. Thus, we included only participants with given parameter estimates in all conditions, resulting in only 11 participants.¹

Parameter analyses with fast-dm

To test the influence of the manipulation we conducted repeated-measures ANOVAs for each parameter, with number of presentations as the independent variable. For the drift rates, we used the absolute values. As predicted, there were significant differences in the drift rates, M (not presented) = −0.21, SD (not presented) = 0.09, M (presented once) = 0.05, SD (presented once) = 0.08, M (presented twice) = 0.14, SD (presented twice) = 0.12, F(2,54) = 22.68, p < 0.01, η_p² = 0.46. As Helmert contrasts revealed, the difference between old versus new items was significant, F(1,27) = 19.99, p < 0.01. The difference between once- and twice-presented items was significant as well, F(1,27) = 30.52, p < 0.01.

Unexpectedly, the bias parameter also showed significant differences, M (not presented) = 0.41, SD (not presented) = 0.10, M (presented once) = 0.49, SD (presented once) = 0.14, M (presented twice) = 0.56, SD (presented twice) = 0.12, F(1.49,40.23) = 15.10, p < 0.01, η_p² = 0.36, dfs Greenhouse–Geisser corrected. The more often the items were presented (not, once, twice) the higher was the bias-parameter z/a. The response-time constant t₀ also showed significant differences, M (not presented) = 0.63, SD (not presented) = 0.07, M (presented once) = 0.60, SD (presented once) = 0.06, M (presented twice) = 0.59, SD (presented twice) = 0.07, F(2,54) = 6.18, p < 0.01, η_p² = 0.19. As expected, the threshold parameter a did not differ significantly between the conditions.

For Experiment 3, we also performed analyses in which we fixed all parameters between conditions, except one which was allowed to vary (either z, a, v, or t₀). This was recommended by Ratcliff (1978) and is possible for within-subject designs only. We expected that the model in which v was free to vary would show the best model fit. The models that allowed the threshold parameter a or the response-time constant t₀ to vary, did not fit the data. We found satisfactory model fit only for models that allowed either the drift rate v or the starting point z to vary. The model that allowed for drift rate variation had a model fit that was more than four times better than the model that allowed for variation of the starting point z. Averaged mean parameter estimates are shown in Fig. 7.

Open image in new window

Parameter analyses with DMAT

As predicted, there were significant differences in the drift rates, M (not presented) = −0.48, SD (not presented) = 0.41, M (presented once) = 0.17, SD (presented once) = 0.38, M (presented twice) = 0.69, SD (presented twice) = 0.69, F(1.74,17.44) = 4.49, p = 0.03, η_p² = 0.31, dfs Greenhouse–Geisser corrected. As Helmert contrasts revealed, the difference between old versus new items was not significant, F(1,10) = 0.07, p = 0.79. However, as predicted, the difference between once- and twice-presented items was significant, F(1,10) = 12.30, p < 0.01.

Unexpectedly, the bias parameter also showed significant differences, M (not presented) = 0.27, SD (not presented) = 0.27, M (presented once) = 0.57, SD (presented once) = 0.20, M (presented twice) = 0.61, SD (presented twice) = 0.18, F(2,20) = 6.99, p = 0.01, η_p² = 0.41 due to the difference between non-presented items and presented items, F(1,10) = 8.53, p = 0.02. There was no difference between items presented once and items presented twice, F(1,10) = 0.59, p = 0.47.

The threshold parameter a also showed significant differences, M (not presented) = 0.65, SD (not presented) = 0.72, M (presented once) = 0.12, SD (presented once) = 0.04, M (presented twice) = 0.12, SD (presented twice) = 0.03, F(1.01,10.05) = 6.02, p = 0.03, η_p² = 0.38, dfs Greenhouse–Geisser corrected. Again, this difference resulted due to the difference between non-presented items and presented items, F(1,10) = 6.04, p = 0.03. There was no difference between items presented once and items presented twice, F(1,10) = 0.59, p = 0.81. As expected, the response-time constant t₀ did not differ significantly between the conditions.

With DMAT, we also performed analyses in which we fixed all parameters between conditions, except one which was allowed to vary (either z, a, v, or t₀). The model that allowed for drift-rate variation had––with four exceptions––the best (and acceptable) model fit. Averaged mean parameter estimates are shown in Fig. 8.

Open image in new window

Parameter analyses with EZ

For two participants, the EZ-diffusion model could not be calculated in all conditions due to perfect accuracy. For the remaining 26 participants, the EZ-diffusion model showed significant differences in (absolute) drift rate only, M (not presented) = 0.19, SD (not presented) = 0.06, M (presented once) = 0.03, SD (presented once) = 0.08, M (presented twice) = 0.13, SD (presented twice) = 0.10, F(1.41,35.17) = 33.19, p < 0.01, η_p² = 0.57 (dfs Greenhouse–Geisser corrected). Helmert contrasts revealed that new and old items were significantly different, F(1,25) = 24.54, p < 0.01. Additionally, items presented once and items presented twice differed significantly, F(1,25) = 73.54, p < 0.01. Threshold parameter a and response-time constant t₀ did not show significant differences. Averaged mean parameter estimates are shown in Fig. 9.

Open image in new window

In three experiments, we explored the convergent and discriminant validity of the central diffusion-model parameters v, z, and a by manipulating variables that were each selected to affect a corresponding single process represented in the model. With respect to convergent validity, all tests were clearly positive. That is, in each instance, the experimental manipulation had a large effect on the target parameter in the expected direction.

With respect to discriminant validity, the results were less clear-cut. All manipulations had some side effects on other model parameters as well. In two cases, this was psychologically meaningful, namely the effect of the speed instruction on the response-time constant t₀ in Experiment 2 and the effect of the number of presentations on t₀ in Experiment 3. However, other effects were harder to reconcile psychologically, namely the effect of the number of presentations (Experiment 3) on the bias parameter z/a and on the threshold parameter a. Whereas the threshold effect may be explained within the diffusion model by invoking additional assumptions (see above), the bias effect is clearly unexplainable within the model.

For the effect of speed versus accuracy instructions on the drift rate only for new items (Experiment 2), a possible explanation entails Ratcliff’s (1978) original assumption of an exhaustive search for negative responses in recognition. It is possible that participants dispense with exhaustiveness under speed instructions and base their decision on a subset of parallel processes. This, in turn, leads to faster drift rates.

Comparing the different methods for parameter estimation, our conclusion is similar to those of van Ravenzwaaij and Oberauer (2009). Fast-dm showed smaller differences between conditions––especially for drift rates in Experiments 1 and 3. DMAT gave error messages when the number of trials was extremely small. EZ was very robust but cannot estimate all parameters.

Hence, if one assumes that the experimental manipulations used in our experiments selectively influenced the respective cognitive processes, one must conclude that the measures derived via parameter estimation––at least as estimated with the fast-dm method––are not entirely “process pure.” Analysis with the EZ-diffusion model showed similar results, except that it did not find differences in t₀ that were psychologically plausible. DMAT showed discriminant validity for Experiment 1 and Experiment 2, but failed to do so for Experiment 3. Parameters t₀, z, and v all represent cognitive processes that affect decision times. However, when they must be recovered from a noisy response-time distribution, their respective influences presumably cannot be clearly separated. This does not necessarily undermine the validity of the diffusion model in certain applications, though, given specific conditions discussed below. To put our findings into perspective, the reader is reminded that in every case, the effect sizes were considerably larger for the target parameters than for the side effects. Hence, the lion’s share of variation in the data could always be attributed to the correct parameter, and it may, thus, be warranted to conclude that the model has some discriminant validity, although it is rather weak. Voss et al. (2004) concluded that their findings supported the validity of the diffusion model. However, their results were more straightforward in that their manipulations only affected the hypothesized parameters (except for the response handicap condition which we did not test).

We recommend fixing parameters across conditions whenever this can be justified for strong theoretical or procedural reasons and to estimate freely only the parameters of interest in a study. If the parameter restrictions are not justified, this should reveal itself in a misfit of the model. To test this with our data, we calculated different restricted model versions in Experiment 3. These analyses showed that only the psychologically plausible restrictions provided good model fit, and models with implausible restrictions failed to fit the data.

The diffusion model is a very useful tool to disentangle processes in binary choice tasks such as recognition. It goes beyond SDT and threshold models in providing a dynamic process description. Furthermore, it allows us to disentangle memory performance further into objective (drift rate) and subjective (threshold and starting point) components of performance and to model speed–accuracy trade-offs (Wagenmakers, 2009). By incorporating response times, it uses a richer database than SDT or threshold models. The results of our experiments show that the model parameters can largely be used as valid measures of the proposed underlying processes with some caution regarding strong claims about discriminant validity. Whether this problem requires an extended theory, more robust estimation procedures, or sophisticated methods of cleaning response-time data, remains an issues for further research.