Investigating variability in morphological processing with Bayesian distributional models

We investigated the processing of morphologically complex words adopting an approach that goes beyond estimating average effects and allows testing predictions about variability in performance. We tested masked morphological priming effects with English derived (‘printer’) and inflected (‘printed’) forms priming their stems (‘print’) in non-native speakers, a population that is characterized by large variability. We modeled reaction times with a shifted-lognormal distribution using Bayesian distributional models, which allow assessing effects of experimental manipulations on both the mean of the response distribution (‘mu’) and its standard deviation (‘sigma’). Our results show similar effects on mean response times for inflected and derived primes, but a difference between the two on the sigma of the distribution, with inflectional priming increasing response time variability to a significantly larger extent than derivational priming. This is in line with previous research on non-native processing, which shows more variable results across studies for the processing of inflected forms than for derived forms. More generally, our study shows that treating variability in performance as a direct object of investigation can crucially inform models of language processing, by disentangling effects which would otherwise be indistinguishable. We therefore emphasize the importance of looking beyond average performance and testing predictions on other parameters of the distribution rather than just its central tendency. Supplementary Information The online version contains supplementary material available at 10.3758/s13423-022-02109-w.

Note. Prior distributions are in the notation Normal(mean, SD). These are expressed in different units due to the parametrizations used by the brms Rpackage: estimates on mu (mean) are in log-ms because a (shifted-)lognormal response distribution was assumed; estimates on sigma (SD) are modelled in the log-scale by default because they must be strictly positive (hence, in log of log-ms). Note that all sigma effects reported in the paper were back-transformed to log-ms, for easier interpretation and for consistency with the other estimates.

S2. Plot of estimated distributions in the different prime type conditions
Figure S1 below shows the estimated RT distributions in the three prime type conditions (unrelated, inflected, derived), with estimates obtained from the Bayesian distributional model reported in the main paper. RTs are shown in the modelled shifted log-millisecond scale; thus, they are normally distributed with estimated mu and sigma parameters. Plotting in a log scale allows bypassing the dependency between mean and standard deviation that exists for millisecond RTs, so that the effects of prime type can be visualised independently.
Note that, in a Bayesian framework, every quantity (i.e., every estimated parameter and every prediction) is associated with a full posterior distribution. Visualisation of distributions can thus become quite complex, as well as computationally intensive. The visualisation below is a simplification: it does not show the uncertainty associated with the predictions, but depicts only the idealised normal distributions that were estimated by the model, on the basis of point estimates (i.e., the mean of the posteriors) of both mu and sigma, for each of the three conditions.
As can be seen in Fig. S1, prime type had effects on both mu and sigma. The model estimated shorter mean RTs in the inflected and derived conditions than in the unrelated condition. With regards to sigma, the effects were small, but can be seen by inspecting the heights of the three distributions: responses in the derived condition were estimated as slightly more variable than in the unrelated condition ('shorter' distribution), and those in the inflected condition were more variable than both. Figure S1: Estimated distributions of RTs in the three prime type conditions (unrelated, inflected, derived). RTs are shown in the shifted log millisecond scale.

S3. Alternative analyses with Generalized Additive Mixed Models (GAMMs)
The present supplementary materials illustrate an additional way of analyzing reaction time (RT) distributions by making use of Generalized Additive Mixed Models (GAMMs); see Baayen et al. (2017) 1 . Similarly to the Bayesian distributional model reported in the main paper (Ciaccio & Veríssimo, 2022), GAMMs allow to test for experimental effects on both the mean of (transformed) RTs, i.e. the mu parameter of the distribution, and their standard deviation (SD), i.e. the sigma parameter. This can be achieved by specifying family = gaulss. This way, we fit a Gaussian model containing two formulae, the first specifying the predictors for estimating effects on mean RTs, and the second the predictors for effects on their SD.
We first load the data and prepare the dataset for the analyses.

# -Keep only experimental items (exclude fillers) # -Exclude incorrect responses and timeouts # -Set unrelated prime as baseline
varmorph <-droplevels(varmorph[varmorph$set=="experimental",]) varmorph <-droplevels(varmorph[varmorph$accuracy==1,]) varmorph$prime_type <-relevel(varmorph$prime_type, "unrelated") Before fitting our Gaussian model, we need to transform our RT data with an appropriate transformation. The transformation that best corresponds to the shifted log-normal distribution that we fitted in the main paper consists in log-transforming RTs after subtracting the 'shift', i.e. after shifting the entire distribution by an amount of milliseconds under which RTs are considered implausible. In our case, the 'amount of ms' we take is the shift estimated by the Bayesian distributional model reported in the main paper, i.e. 302.58 ms.
As we can see from the plots below, shifted log-transforming our data (left panel) makes them closer to a Gaussian distribution than a simple log-transformation (middle panel).
Admittedly, taking a shift estimated by another Bayesian distributional model may make the analysis unnecessarily complex. At least for this dataset, the inverse transformation seems to also work quite well (see right panel below). For those analyzing data directly with GAMMs, this may be a more practical option, though probably at the expense of interpretability of the estimates. In this document, we will stick to the shifted log transformation for consistency with the main paper.

Normal Q−Q Plot
Theoretical Quantiles

Sample Quantiles
In the following, we will start from a simple GAMM and make it successively more complex, leading to a final model with a very similar structure to the one reported in the main manuscript. We will additionally show that: (a) the final model testing effects on both the mu and the sigma parameters has a better fit than one only testing effects on mu; (b) the final model shows parallel results to the one reported in the manuscript regarding the main contrast we focused on, namely the difference between inflection and derivation with regards to the effect on sigma.
We first fit a baseline mixed-effects model (m0) testing only effects of Prime Type (inflected, derived, unrelated) on mu, i.e. on mean (shifted log) reaction times. The model contains by-subject and by-item random effects. It additionally includes the predictor (centered) Trial Number. Unlike in the main paper, because GAMMs allow for smooth predictors and the effect of trial number over the experiment is likely to be nonlinear, we specify Trial Number as a smooth term by using s(). Starting from this model, we will now fit more complex models and evaluate whether they provide a better fit than the simpler model based on their REML score (lower REML = better fit).

Normal Q−Q Plot
Theoretical Quantiles

Sample Quantiles
Printing the summary of the model, we can now evaluate its output. Note that, for the mu parameter, model estimates are provided in the same scale as the data to which we fitted the model (i.e., in the log scale). Instead, for the sigma parameter, they are in the log of log-ms. Concerning the effects of 'Prime Type' on mu, the model output shows similarly large priming effects (i.e., speed-ups in RTs) on mean RTs for inflected and derived primes (respectively, β = -0.1212462 and β =-0.1098143). Regarding effects on sigma, while both inflected and derived primes increase RT variability, this effect seems to be robust only for inflected primes, and it is larger for inflected primes than for derived primes (respectively, β =0.0810701 and β =0.0326677).