Motivational deficits in schizophrenia relate to abnormalities in cortical learning rate signals

Abstract

Individuals from across the psychosis spectrum display impairments in reinforcement learning. In some individuals, these deficits may result from aberrations in reward prediction error (RPE) signaling, conveyed by dopaminergic projections to the ventral striatum (VS). However, there is mounting evidence that VS RPE signals are relatively intact in medicated people with schizophrenia (PSZ). We hypothesized that, in PSZ, reinforcement learning deficits often are not related to RPE signaling per se but rather their impact on learning and behavior (i.e., learning rate modulation), due to dysfunction in anterior cingulate and dorsomedial prefrontal cortex (dmPFC). Twenty-six PSZ and 23 healthy volunteers completed a probabilistic reinforcement learning paradigm with occasional, sudden, shifts in contingencies. Using computational modeling, we found evidence of an impairment in trial-wise learning rate modulation (α) in PSZ before and after a reinforcement contingency shift, expressed most in PSZ with more severe motivational deficits. In a subsample of 22 PSZ and 22 healthy volunteers, we found little evidence for between-group differences in VS RPE and dmPFC learning rate signals, as measured with fMRI. However, a follow-up psychophysiological interaction analysis revealed decreased dmPFC-VS connectivity concurrent with learning rate modulation, most prominently in individuals with the most severe motivational deficits. These findings point to an impairment in learning rate modulation in PSZ, leading to a reduced ability to adjust task behavior in response to unexpected outcomes. At the level of the brain, learning rate modulation deficits may be associated with decreased involvement of the dmPFC within a greater RL network.

Introduction

Reinforcement learning (RL), the ability to learn from actions through reward and punishment, is an essential mechanism underlying optimal decision-making (Dayan & Berridge, 2014). A driving factor in learning from the environment is the reward prediction error (RPE) signal—the mismatch between outcome and expectation, thought to be signaled by midbrain dopaminergic neurons projecting to the ventral striatum (VS) (Schultz, Dayan, & Montague, 1997; Steinberg et al., 2013). There is now appreciable evidence suggesting that RL deficits exist in individuals from across the psychosis spectrum, ranging from adolescents and young adults at clinical high risk for psychotic illness to people with chronic, multi-episode schizophrenia (PSZ) (Barch et al., 2017; Waltz, Demro, et al., 2015b).

Although results have been mixed (Gold et al., 2012; Hartmann-Riemer et al., 2017; Reddy, Waltz, Green, Wynn, & Horan, 2016), there are a number of studies to date that have reported an association between RL deficits and the severity of (both clinical and subclinical) negative symptoms, suggesting to some extent that changes in adaptively responding to environmental stimuli may play an important role in the onset of motivational deficits (Barch et al., 2017; Strauss, Waltz, & Gold, 2014). In some, but not all, individuals with psychotic symptoms, RL deficits can be linked to aberrant VS RPE signals at the neural level (Maia & Frank, 2017; Murray et al., 2008) and, potentially, to increased presynaptic dopamine function at the neurochemical level (Boehme et al., 2015). Especially in individuals suffering from motivational deficits, RL impairments may result from a decrease in anticipatory pleasure (Engel, Fritzsche, & Lincoln, 2013; Frost & Strauss, 2016), which depends on VS function and has been shown to be affected in PSZ (Radua et al., 2015).

Perhaps surprisingly, there is some evidence that VS RPE signals in medicated PSZ are relatively intact (Dowd, Frank, Collins, Gold, & Barch, 2016; Gradin et al., 2011; Waltz et al., 2010); an observation most recently replicated in the largest sample of patients in this literature to date (Culbreth, Westbrook, Xu, Barch, & Waltz, 2016). In addition, work from our lab has revealed that RL deficits in medicated PSZ are associated with specific impairments in representations of expected value to drive choice, putatively linked to functioning of orbitofrontal cortex (OFC), while RPE-based learning, linked to basal ganglia and dopamine, is less affected in PSZ (Collins et al., 2014; Gold et al., 2012). This latter observation aligns well with reports of normalized reward-related signals in striatum following antipsychotic medication administration (Nielsen et al., 2012).

Thus, although previous work has reported altered striatal RPE signals in the psychosis spectrum (Murray et al., 2008; Schlagenhauf et al., 2014), there also are multiple reports of (generally medicated) PSZ with RL deficits but intact striatal RPE signals (Culbreth et al., 2016; Waltz et al., 2010). Based on these observations, we hypothesized that a mechanism determining the impact of RPEs on learning and behavior, rather than signaling the RPE per se, might contribute to RL deficits in PSZ. In computational models of RL (Sutton & Barto, 1998), a parameter called learning rate (α) acts as the multiplier of the RPE (denoted as δ) to determine how much each RPE signal is weighted in the updating of representations of stimulus or action value or stimulus-response association strength. Higher learning rates are not always adaptive. In stationary, but probabilistic, environments it is useful to decrease learning rates to avoid being overly sensitive to spurious outcomes. When contingencies are changing or uncertain, however, higher learning rates are desirable (Behrens, Woolrich, Walton, & Rushworth, 2007; Franklin & Frank, 2015). Evidence suggests that learning rate is dynamic, varying as a function of the volatility of the learning environment, and that the circuits that underlie the modulation of learning rate incorporate several frontal cortical regions, including dorsal anterior cingulate cortex (dACC) (Behrens et al., 2007) and dorsomedial prefrontal cortex (dmPFC) (Krugel, Biele, Mohr, Li, & Heekeren, 2009; McGuire, Nassar, Gold, & Kable, 2014), brain regions also implicated in learning from outcomes (Mars et al., 2005), expectation updating (Behrens et al., 2007; McGuire et al., 2014), and rapid, flexible decision-making (Krugel et al., 2009).

In medicated PSZ, we have previously reported abnormal outcome-related signals in frontal cortex (Waltz et al., 2010) and impaired expected value representation (Gold et al., 2012), thought to depend on intact orbitofrontal cortex function (Metereau & Dreher, 2015). In both studies, these impairments scaled with the severity of motivational deficits. Multiple other groups have reported abnormal prefrontal activity (or fronto-parietal connectivity) in association with belief updating in uncertain environments (Kaplan et al., 2016; Koch et al., 2010; Paulus, Frank, Brown, & Braff, 2003). While these findings suggest that abnormalities in learning rate modulation may contribute to RL deficits in PSZ, even when RPE signals are intact, no systematic investigation of dynamic learning rate modulation has been conducted in this population.

To directly test this hypothesis, we administered a probabilistic RL paradigm to medicated PSZ and healthy volunteers (HV), which involved choosing from three decks of cards with different reinforcement rates. Using computational models of decision making, we quantified subjects’ dynamic modulation of learning rate on trials surrounding shifts in reinforcement contingencies. Employing functional magnetic resonance imaging (fMRI), we also isolated RPE and learning rate modulation signals in the brain using model-based analyses, expecting intact VS but altered dmPFC signals in medicated PSZ relative to HV. Finally, if deficient learning rate modulation contributes to RL deficits in medicated PSZ, then this may be the consequence of decreased coupling between regions that signal and utilize RPEs. We therefore conducted a psycho-physiological interaction (PPI) analysis, investigating correlations between dmPFC and regions that are thought to signal RPEs, during stable and volatile phases of the task. For all these analyses, we expected to observe the greatest deficits in learning-rate-associated activity in PSZ with the most severe motivational deficits.

Methods

Participants

Twenty-seven participants meeting the diagnosis for schizophrenia or a schizoaffective disorder and 25 HV matched on age, gender, ethnicity, and parental education were recruited. All participants provided written, informed consent to protocols approved by the Institutional Review Board of the University of Maryland School of Medicine (Protocol HP-00051996). We recruited PSZ through clinics at the Maryland Psychiatric Research Center or other nearby community mental health centers. The presence of a diagnosis of schizophrenia or a schizoaffective disorder in PSZ, as well as the absence of a clinical disorder in HV, was confirmed using the SCID-I (First, Spitzer, Gibbon, & Williams, 1997). The absence of an Axis II personality disorder in HV was confirmed using the SIDP-R (Pfohl, Blum, Zimmerman, & Stangl, 1989). All PSZ were on a stable antipsychotic medication regimen (no changes in medication dose or type in the 4 weeks leading up to study participation; antipsychotic medication details and haloperidol equivalents can be found in Table S1). Major exclusion criteria included: pregnancy, current illegal drug use (verified using a urine screen), history of substance dependence (SCID-I), a neurological disorder, and/or a medical condition affecting study participation (such as chronic, uncontrolled hypertension or diabetes). Participants were asked to abstain from alcohol 24 hours before the study session, which was verified by a breathalyzer. In order to avoid any potential effects of nicotine withdrawal, smokers were allowed to smoke before the study session.

Clinical and cognitive assessment

To investigate a potential effect of motivational deficit severity on learning rate modulation, we used the Scale for the Assessment of Negative Symptoms (SANS). SANS data were collected by a trained and experienced clinical research associate. Brief Psychiatric Rating Scale (BPRS) ratings were collected in the same session and the positive symptom factor (suspiciousness, hallucinations, unusual thought content, grandiosity) was used as a measure of positive symptom severity (McMahon et al., 2002). The same clinical research associate assessed general cognitive ability (intelligence quotient; IQ) using the Wechsler Abbreviated Scale of Intelligence (WASI-II) (Wechsler, 2011) and performance on the MATRICS Consensus Cognitive Battery (Nuechterlein et al., 2008). Antipsychotic regimen doses were converted to haloperidol equivalents according to Andreasen et al. (2010).

Reinforcement Learning Paradigm

We used a probabilistic RL task adapted from a previous study by Krugel et al. (2009) (Figure S1 for graphical overview). Participants selected one out of three card decks, identified by colors (black, red, and blue) using the index finger of their right hand (2,000 ms). After a brief inter-stimulus interval (pseudo-randomized to 2,000, 4,000, or 6,000 ms) and a card flip animation (~200 ms), participants were informed about the outcome of their choice (1,000 ms), which could be a win (+100 points) or loss (−50 points). Choices were rewarded probabilistically, with a choice of the “optimal deck” leading to a 100-point gain on 90% of trials (and a loss of 50 points on 10% of trials). Choices of two nonoptimal decks led to 100-point gains on 50% and 10% of trials (and losses of 50 points on 50% and 90% of trials), respectively. Participants were instructed to try to identify the optimal deck (i.e., the one with the highest expected value) as quickly as possible; they also were informed that, occasionally, a new deck would become the optimal one. In fact, this occurred after subjects selected the optimal deck 8 times in a run of 9 trials (1 nonoptimal choice was allowed). The task consisted of 160 trials, subdivided into 4 runs of 40 trials (each run lasting ca. 7 minutes; total task duration just over 28 min). Deck locations only changed between runs, not within them. Prior to the task, participants underwent a training session where they received task instructions, completed a practice session, and were given the opportunity to ask questions. After the experiment, the total of amount of earned points was divided by 1,000 (total earnings for each group reported in Table S2).

Computational modeling

We modeled performance data using an algorithm optimized for sudden and dynamic shifts in reinforcement contingencies (Krugel et al., 2009; Sutton & Barto, 1998). Using the expected reward q_i(t) for each deck, the probability of choosing the deck p_t(t) was calculated according to the softmax function described in Eq. 1.

$$ {\mathrm{p}}_{\mathrm{i}}\left(\mathrm{t}\right)=\frac{\mathrm{e}\left[\upgamma \cdotp {\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right)\right]}{\sum_{\mathrm{j}=1}^{\mathrm{n}}\mathrm{e}\left[\upgamma \cdotp {\mathrm{q}}_{\mathrm{j}}\left(\mathrm{t}\right)\right]} $$

(1)

In accordance with previous work (Rescorla & Wagner, 1972; Sutton & Barto, 1998), the RPE δ_i(t) was operationalized as the difference between the expected reward q_i(t) and the actual reward r_i(t), as shown in Eq. 2.

$$ {\updelta}_{\mathrm{i}}\left(\mathrm{t}\right)={\mathrm{r}}_{\mathrm{i}}\left(\mathrm{t}\right)-{\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right) $$

(2)

Reward expectation updating following an RPE can then be formulated as Equation 3, where learning rate, α, acts as the multiplier of the RPE. In other words, α determines how strongly the RPE is used to update expectations.

$$ {\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right)={\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}-1\right)+\upalpha \cdotp {\updelta}_{\mathrm{i}}\left(\mathrm{t}-1\right) $$

(3)

Given the dynamic nature of the task, where reward expectation updating is especially likely to occur following a shift in reward contingencies, learning rates α(t), in addition to RPEs, δ(t), were estimated on a trial-by-trial basis, similar to Krugel et al. (2009). A useful heuristic is that to prevent overly updating to a single spurious RPE, a learner can keep track of the recent history of RPEs and increase the learning rate when errors in prediction are increasing across trials, and vice versa when errors are decreasing. Thus, we first computed the absolute RPE values (degree of unsigned surprise) and accumulated these (Eq. 4) to represent the recent average prediction error:

$$ \left|\delta (t)\right|=\kern0.5em \left|\delta \left(t-1\right)\cdotp \left(1-\alpha (1)\right)\right|+\delta (t)\cdotp \alpha (t) $$

(4)

where α(1) is a parameter that determines the initial learning rate on trial 1. Next, the slope m of the absolute smoothed RPEs |δ_i(t)| was calculated to obtain a metric for whether the averaged errors are increasing or decreasing, normalized to the average RPE (so that it is independent of the scale):

$$ \mathrm{m}\left(\mathrm{t}\right)=\kern0.5em \frac{\left|\updelta \left(\mathrm{t}\right)\right|-\mid \updelta \left(\mathrm{t}-1\right)\mid }{\left[\left(\left|\updelta \left(\mathrm{t}\right)\right|+\left|\updelta \left(\mathrm{t}-1\right)\right|\right)/2\right]} $$

(5)

Learning rate α(t) was fixed between 0 and 1, as shown in Eq. 6. Here, the β parameter estimates the effect of RPE slope m on learning rate α. Small β values are indicative of a dynamic learning rate, whereas with greater learning rates the dynamics are negligible [i.e., learning rates do not deviate much from α(1)] (Krugel et al., 2009).

$$ \mathrm{f}\left(\mathrm{m}\right)=\operatorname{sign}\left(\mathrm{m}\right)\cdotp \left[1-e{\left(-\left(\mathrm{m}/\upbeta \right)\right)}^2\right] $$

(6)

Finally, trial-by-trial estimates of α were calculated using function f, where α(t) increased and decreased as a function of a positive and negative slopes m, respectively (Eq. 7).

$$ {\displaystyle \begin{array}{c} If\ m>0;\alpha (t)=\alpha \left(t-1\right)+f\left[m(t)\right]\cdot \left[1-\alpha \left(t-1\right)\right]\\ {} If\ m<0;\alpha (t)=\alpha \left(t-1\right)+f\left[m(t)\right]\cdot \alpha \left(t-1\right)\end{array}} $$

(7)

Thus, if the slope m is positive (errors are increasing), the learning rate is increased toward 1, whereas if it is negative, the learning rate is driven toward 0. Note that this model reduces to a constant learning rate model when β values are large, i.e., f(m) ~0. Parameter initialization bounds for α (0.01>, <0.9), β (>0.01), and γ (>0.001) were fixed in accordance with previous work (Krugel et al., 2009). Supplemental Text 1 contains a detailed description of the model selection procedure, posterior predictions, and a demonstration of the effect of different β values on simulated performance. Model fit and individual parameters for the winning model are reported in Table S3.

FMRI acquisition and pre-processing

Whole-brain functional EPI images were acquired on a 3T Siemens Trio scanner (Erlangen, Germany) while participants completed the RL task. We acquired 852 T2*-weighted images with the following parameters: n slices = 81; TR = 2 s; TE = 30 ms; FA = 90°; voxel size = 1.5 mm³; FOV = 22 x 22 cm; matrix size = 128 x 128. Additionally, we acquired T1- weighted structural images (MPRAGE) with standard parameters (n slices = 192; TR = 8.6 s; TE = 4 ms; FA = 20; and voxel size = 1 mm³) for anatomical reference. To minimize head movement, foam padding was used.

Data were preprocessed and analyzed using the AFNI software package (Cox, 1996). Pre-processing steps consisted of co-registration of EPI and anatomical images, warping to Talairach space (using the 452 International Consortium for Brain Mapping template, which caused the images to be upsampled to 1.5-mm isotropic voxels), and smoothing with a 6-mm FWHM kernel. In keeping with other work (Shine et al., 2016), volumes with >0.5-mm displacement in any plane were excluded from analysis, and participants with >20% excluded volumes were excluded from analyses altogether. Three participants were excluded on this basis. Anatomical images were segmented using SPM 12’s segmentation algorithm (default settings) and were inspected for fit. In a manner similar to previous work (Hernaus, Casales Santa, Offermann, & Van Amelsvoort, 2017), an average grey matter image of the entire sample was constructed, which was later used to inclusively mask grey matter in group-level analyses.

FMRI data analysis

Regressors of interest for the model-based fMRI GLM were outcome onset times, amplitude-modulated (in separate GLMs) by trial-by-trial estimates of RPE [δ(t)] and learning rate [α(t)] (see Section 2.4). Eight regressors of no interest (presentation times of stimuli evoking nonresponses, “too slow” feedback, and the 6 demeaned motion parameters) were additionally added. All regressors of interest and the “too slow” feedback regressor of no interest were boxcar functions of 2s. We used a linear regression model with ARMA(1,1) modeling of serial correlation.

The strength of learning rate-dependent functional connectivity between dmPFC and other brain areas was investigated using a PPI analysis (O'Reilly, Woolrich, Behrens, Smith, & Johansen-Berg, 2012). We selected the 8 trials before and 8 trials following every reinforcement contingency shift for every participant and distinguished between “learning rate modulation trials” and “nonlearning rate modulation trials.” Trials 1-4 pre-shift and trials 1-4 post-shift were labeled as “learning rate modulation trials,” whereas trials 5-8 pre-shift and trials 5-8 post-shift were labeled as “nonlearning rate modulation trials.” Learning rate modulation trials were believed to represent a volatile environment, in which participants were unlikely to be certain that they were sampling from the optimal deck and therefore needed to utilize the RPE signal to rapidly update their expectations (with a large learning rate). Conversely, nonlearning rate trials were believed to represent a stable environment, in which participants were likely aware that they were sampling from the best deck and, hence, could update expectations more gradually updating (with a smaller learning rate). Note that an 8-trial interval was the minimum amount of trials before a sudden shift in reinforcement contingencies could occur. The GLM for the PPI analysis contained the psychological regressor (learning rate modulation trials = 1, nonlearning rate modulation trials = −1), the time-series extracted from the dmPFC ROI, the psychophysiological interaction, and eight regressors of no interest.

At the group level, one-sample t tests were conducted to identify significant activations in regions associated with regressors of interest in the entire sample, without respect to group; independent two-samples t tests were conducted to test for between-group differences. To estimate the minimum cluster size required to correct for multiple comparisons across the whole brain at a level of p(FWE-corrected) = 0.05, 10,000 Monte Carlo simulations of the probability distribution were run using the AFNI 3dClustSim command (-autocorrelation option), with a voxel-wise threshold of p < 0.001.

In addition to whole-brain analyses, unbiased ROIs were selected from the literature. As an area consistently associated with learning rate modulation (McGuire et al., 2014), we decided on a 1026-mm³ (304 functional voxels) region of interest in the dmPFC (Figure S3A). This dmPFC area (peak voxel x = 3, y = 8, z = 51) was identified in a conjunction analysis of surprise-, uncertainty-, and reward-driven learning, suggesting reliable involvement in learning rate modulation (McGuire et al., 2014). Additionally, this region showed a high degree of overlap with regions previously associated with outcome-driven action-selection (Mars et al., 2005). At a voxel level threshold of p < 0.005, activity in a large cluster encompassing the dmPFC (cluster size = 381 voxels, peak voxel coordinates x = 1, y = 13, z = 52) was observed in the entire sample, suggesting that dmPFC activity tracked trial-by-trial learning rate in our sample.

Given the well-established involvement of the VS in signaling RPEs, we additionally decided on a 1073-mm³ (318 voxels) region of nucleus accumbens, defined by the Desai Lab AFNI atlas (https://afni.nimh.nih.gov/pub/dist/doc/program_help/whereami.html; Figure S3B). Thus, our ROIs encompassed regions implicated in signalling RPEs, as well as regions thought to use RPEs to update expectations. For group-level analyses, mean beta values were extracted from the above-mentioned ROIs and converted to Z-scores. ROI activity thresholds were set to p < 0.005 (against 0) and group differences in activity were Bonferroni-corrected for the number of statistical ROI tests (n = 2).

Statistical analyses

Of the total sample of 52 participants, two HVs were excluded, because they never experienced a sudden shift in reinforcement contingencies; one PSZ produced no responses on approximately half of all trials (48.75%) and also was excluded. Thus, RL performance data from 26 PSZ and 23 HV were subjected to further analyses. Conform previous work (den Ouden et al., 2013; Waltz & Gold, 2007), these analyses included: number of reversals achieved, total amount of earnings, switching between decks on subsequent trials regardless of outcome, resampling a deck following a win (win-stay), switching decks following a loss (lose-shift), and the tendency to choose the previous-best deck following a reversal regardless of outcome (post-shift perseveration).

For fMRI analyses another five participants were excluded due to phase-instability artifacts associated with the acquisition protocol (n = 3), a technical error leading to an incomplete fMRI dataset (n = 1) and excessive motion (n = 1; see Section 2.5 for details). Thus, fMRI analyses were conducted for a sample of 22 HV and 22 PSZ.

In accordance with previous work from our group (Gold et al., 2012; Waltz, Brown, et al., 2015a), we used mean item scores from the SANS avolition subscale as measures of a motivational deficit severity, in the service of assessing potential relationships between motivational deficit severity and neural signals associated with learning rate modulation. In light of a bi-modal distribution for many of our SANS items, as well as previous work reporting the greatest performance impairments in individuals with high motivational deficits (Gold et al., 2012), we created high and low motivational deficit subgroups. The median score (in PSZ) on the SANS avolition subscale was 1.50, yielding a reasonable subgroup cutoff score (13 vs. 13 in the sample of all PSZ; 11 vs. 11 in the sample of all PSZ with usable fMRI data). We additionally investigated the effect of positive symptom severity. The median of the average BPRS positive symptom factor score was also 1.50, leading to a subgroup of 14 vs. 12 in the sample of all PSZ and 13 vs. 9 in the sample of all PSZ with usable fMRI data.

Results

Sample Demographics

Participant groups were matched on age, gender, race, and parental education level. There was a significant group difference in participant education, IQ score as measured by the WASI-II, and some subscales of the MATRICS Consensus Cognitive Battery performance (Table 1).

Table 1 Sample demographics

Reinforcement Learning performance

When comparing groups on all trials, HV and PSZ did not significantly differ in the number of stages achieved (Figure S4), total earnings, switching, win-stay, or lose-shift behavior (Table S2). However, there was evidence for subtle performance deficits in PSZ following reversals. On trials 2-8 following the first 5 stages (which 79.59% of all participants achieved), PSZ demonstrated increased post-shift perseveration. That is, following a sudden shift in contingencies, PSZ more often sampled from the deck that was optimal before the shift (t₄₇ = −2.63, p = 0.01; Fig. 1a; Figure S5 for trial-by-trial overview), which was especially pronounced on early trials (trials 2-5; t₄₇ = −2.26, p = 0.03; Fig. 1a). A between-group difference in post-shift perseveration was still present at trend level when comparing across all stages achieved, which included participants that performed particularly well (t₄₇ = −1.9, p = 0.06). Increased post-shift perseveration in PSZ may have lead to insufficient sampling of other decks following a reversal, reflected by a decrease in win-stay choices on early trials (t₄₇ = 2.19, p = 0.03; Fig. 1b). While lose-staying—that is, resampling a deck following a loss—on early trials after a reversal was numerically greater in PSZ (M = 10.26%, SD = 14.87) compared with HV (M = 5.46%, SD = 8.64), this did not reach significance (t₄₇ = 1.34, p = 0.19).

Fig. 1

Increased post-shift perseveration and decreased win-stay choices in PSZ. a Relative to HV, PSZ showed greater perseveration (i.e., selecting the previous-best card deck) following a sudden shift in reinforcement contingencies, especially in trials immediately following the shift. b Additionally, a decreased tendency to resample from the card deck that resulted in a win on the previous trial was observed immediately following the contingency shift. Note that post-shift perseveration was calculated across trials 2-8 post-shift; trial 2 being the first trial after participants have received feedback regarding a contingency shift. Win-stay choices were calculated for trial 3-8 post-shift; trial 3 being the earliest opportunity at which participants could have made a win-stay choice (following a shift to a new deck). Estimates were calculated on the basis of the first 5 stages. **p < 0.01, *p < 0.05, bars represent 95% confidence intervals

Computational modeling parameters

Other than changes in trial-by-trial estimates of learning rate reported in Section 3.4, no significant group differences were observed in free parameters, including learning rate on the first trial (Table 2).

Table 2 No group differences in free parameters

Learning rate modulation

Next, we extracted trial-by-trial learning rate estimates 8 trials before and after a sudden shift in reinforcement contingencies. A repeated measures ANOVA with group as between-subjects factor and trials 1-8 pre-shift as a within-subjects factor revealed a significant group difference in the slope of learning rate pre-shift (F_2,116 = 4.27, p = 0.01), as HV showed a greater pre-shift decline in learning rate than PSZ (Fig. 2a). A learning rate slope difference was also apparent when comparing the groups on all 16 time-points (F_6,257 = 2.74, p = 0.02; Fig. 2a), where learning rate in HV followed a significant (p < 0.001) quadratic trend. These results suggest that HV showed dynamic changes in learning rate modulation as a function of environmental stability, which was not present in PSZ.

Fig. 2

Learning rate modulation deficits increase with motivational deficit severity. a PSZ, relative to HV, demonstrated a decrease in learning rate modulation, especially in trials leading up to a contingency shift. Solid bars represent SEM. *p < 0.05. b Decreased learning rate modulation on trials leading up to a contingency shift was especially apparent for PSZ with the most severe motivational deficits. These PSZ showed little to no learning rate modulation across all trials. c Relative to HV, PSZ with greater motivational deficits additionally demonstrated smaller increases in learning rate following a contingency shift, defined as the difference between trial 8 pre-shift and trial 2-8 (“11-17”) post-shift

Interestingly, the group difference in pre-shift learning rate slope seemed to be driven by PSZ with the highest ratings for motivational deficits; a significant group-by-time interaction was observed when repeating the analysis with high and low avolition subgroups (F_5,114 = 2.73, p = 0.02; Fig. 2b). Comparing individual slopes (overall group main effect F_{2, 46} = 4.44, p = 0.02), we observed significant differences between HV and the high avolition subgroup (p_{Bonferroni-corrected} = 0.02) but not between HV and the low avolition subgroup (p_{Bonferroni-corrected} = 0.37). Pre-shift learning rate slope did not differ between low and high positive symptom subgroups (p_{Bonferroni-corrected} = 0.99).

We additionally observed an influence of avolition severity on post-shift learning rates. When we looked at the change in learning rate from trial 8 pre-shift, where participants have evidently identified the optimal deck, to mean learning rate on trials 10-17 post-shift, where participants need to locate the new optimal deck, we observed a main effect of participant group (F_2,46 = 4.58, p = 0.02; Fig. 2c), such that HV showed greater pre- to post-shift learning rate changes than high-avolition PSZ (p_{Bonferroni-corrected} = 0.03), while low-avolition PSZ showed similar, yet nonsignificant, deficits compared with HV (p_{Bonferroni-corrected} = 0.09). No differences in post-shift learning rates were observed between low and high positive symptom subgroups (p_{Bonferroni-corrected} = 0.99).

In summary, these results demonstrate that PSZ show deficits in the ability to update expectations dynamically through learning rate modulation, with this deficit being more pronounced in PSZ with more severe motivational deficits, but not in PSZ with greater positive symptoms.

Whole-brain analyses

As expected, trial-wise estimates of RPE were strongly associated with BOLD signal time-courses in VS (Fig. 3a; Table 3A) in the entire sample of participants with usable MRI data (n = 44). Trial-wise estimates of learning rate, on the other hand, were associated with BOLD signal time-courses in frontal and parietal cortex (Fig. 3b; Table 3B), in line with previous work demonstrating activity increases in the fronto-parietal network during decision-making tasks in volatile environments (Behrens et al., 2007; Koch et al., 2010; McGuire et al., 2014). At a voxel level threshold of p < 0.005, learning rate-related activity in dmPFC (cluster size: 381, peak voxel: x = 3, y = 8, z = 51) also was observed. Whole-brain analyses revealed no between-group differences in RPE- or LR-associated activity in any clusters large enough to survive correction for multiple comparisons across the whole brain.

Fig. 3

Whole brain and ROI fMRI analyses. Robust RPE and LR signals were observed in: (a) VS, (b) superior parietal lobule, and multiple other brain regions (Table 3). In subsequent ROI analyses, (c) no group differences in VS RPE signals were observed, while (d) a trend for a group difference was observed for dmPFC learning rate signals. a = trend-significant (p = 0.06), bars represent 95% confidence intervals. PE = prediction error, LR = learning rate

Table 3. Whole-brain one-sample t tests

Region of interest analyses

In the entire sample, VS RPE (t₄₃ = 5.75, p < 0.001) and dmPFC learning rate signals (t₄₃ = 3.04, p = 0.004) were significantly greater than zero, consistent with our whole-brain analysis results. Moreover, greater dmPFC LR signals correlated positively with the percentage of optimal choices in the entire sample (Pearson’s r = 0.36 p = 0.02 , n = 41, 3 participants >2.5 SDs removed from mean). In line with previous work, VS RPE signals did not differ between groups (t₄₂ = −1.61, p = 0.12; Fig. 3c). For model-based LR analyses, there was a trend for a group difference in dmPFC, with HV displaying a greater association between trial-wise estimates of learning rate and activity in dmPFC than PSZ (t₄₂ = 1.93, p = 0.06; Fig. 3d). Despite the observed trend in dmPFC, these analyses do not provide clear evidence for impairments in regions associated with expectation updating or signaling of the RPE.

Learning rate modulation-dependent functional connectivity

Next, we conducted a PPI analyses to investigate learning rate modulation-dependent functional connectivity between dmPFC and the rest of the brain. In the entire sample, functional connectivity between dmPFC and superior parietal lobule increased from nonlearning rate-modulation to learning rate-modulation trials (Table 2C; Fig. 4a). In a follow-up ROI analysis, we observed an overall group difference in dmPFC-VS functional connectivity (F_2,46 = 3.74, p = 0.03), which became highly significant after removing one HV outlier with a value smaller than 2.5*IQR (F_2,45 = 3.95, p = 0.008). Specifically, we observed that HV and the low avolition subgroup displayed increased dmPFC-VS functional from nonlearning to learning rate-modulation trials. DMPFC-VS functional connectivity decreased in the high-avolition group (HV vs. high-avolition subgroup p_{Bonferroni-corrected} = 0.01; low- vs. high-avolition subgroup p_{Bonferroni-corrected} = 0.02; Fig. 4b). DMPFC-VS functional connectivity did not differ between low and high positive symptom subgroups p_{Bonferroni-corrected} = 0.99). Taken together, these results suggest decreased coupling between regions that signal the RPE and regions that utilize the RPE to update predictions. In line with our modeling results, connectivity changes were expressed most in PSZ with high motivational deficits.

Fig. 4

Decreased learning rate modulation-related dmPFC-VS coupling in PSZ. a Whole-brain functional connectivity between dmPFC and inferior parietal lobule increased from nonlearning rate to learning rate trials in the entire sample. b In a follow-up ROI analysis in VS, functional connectivity increases were observed from nonlearning to learning-rate trials for HV and PSZ with more mild motivational deficits, while dmPFC-VS connectivity decreases were observed in PSZ with more severe motivational deficits. **p < 0.01, *p < 0.05, bars represent 95% confidence intervals. LR = learning rate

Correlations with clinical variables

Total haloperidol-equivalent antipsychotic levels did not correlate with performance measures or fMRI analyses reported above (all p < 0.46). In the entire sample of participants with MATRICS data (n = 43), correlations between overall MATRICS scores and total numbers of stages achieved trended toward significance (Spearman’s rho = 0.29, p = 0.06), but we observed no significant relationships between overall MATRICS scores and the slope of pre-reversal learning rate modulation (Spearman’s rho = 0.01, p = 0.98). In the entire sample (n = 49), age correlated negatively with stages achieved (Spearman’s rho = −0.41, p = 0.003) but not with the slope of pre-reversal learning rate modulation (Spearman’s rho = −0.18, p = 0.22). Adding MATRICS score, smoking status, or age as a covariate did not change the results.

Discussion

In this report, we present evidence of an impairment in learning rate modulation in PSZ, leading to a reduced ability to adjust task behavior in response to unexpected outcomes. We moreover observed abnormal dmPFC-VS functional connectivity during changing task environments, potentially suggesting that the absence of learning rate dynamics in PSZ may be associated with a decrease in the integrity of the RL network. At both the behavioral and neural levels of enquiry, we observed greater deficits in PSZ with high motivational deficits, once again linking aspects of impaired reinforcement learning to negative symptom severity.

Schizophrenia has long been associated with increased perseveration (Lysaker, Bell, Bryson, & Kaplan, 1998) and a decreased ability to adaptively respond to performance feedback (Cicero, Martin, Becker, & Kerns, 2014; Mahurin, Velligan, & Miller, 1998). This inability to adaptively modulate behavior has previously been linked to increased reliance on response history when planning future decisions (Paulus, Geyer, & Braff, 1999), as well as to degraded representations of the expected value of choices (Gold et al., 2012; Waltz & Gold, 2016). Here, we expand on this work and offer a mechanism by which expected value could become degraded. That is, underutilization of accurately propagated teaching signals may underlie choice perseveration, thereby hampering the formation of accurate and adaptive representations of expected value. Our observations of smaller decreases in learning rate modulation under relatively stable circumstances (pre-shift) points to a reduction in dynamic learning rate modulation, in PSZ, relative to HV. This idea is further supported by our observation of smaller increases in learning rate modulation under volatile circumstances (post-shift), in PSZ, relative to HV. Our subgroup-specific effects indicate that learning rate modulation deficits may be especially relevant in the endurance of motivational deficits, but not positive symptoms. Importantly, it should be noted that these analyses were conducted in subgroups of PSZ, and the specificity of symptom effects should therefore be interpreted with caution.

Consistent with the results of multiple previous studies (Culbreth et al., 2016; Dowd et al., 2016; Waltz et al., 2013; Waltz et al., 2010), we found that striatal RPE signals did not differ between HV and PSZ. Specifically, no significant group differences were observed in VS for a model-based analysis using trial-by-trial estimates of the RPE. As we have argued in the introduction, and in light of previous of work showing deficits in RPE coding (Murray et al., 2008; Reinen et al., 2016; Schlagenhauf et al., 2014), these results may suggest that RL impairments across the psychosis continuum are underlain by different mechanisms. Although we did not observe any associations between antipsychotic medication dose and performance, modeling or fMRI outcome measures in the current study, the possibility still exists that mechanisms of RL impairments may be dependent on illness phase (at-risk, first episode, or PSZ) or relate to the effects of antipsychotic medication on the reward system (Diederen et al., 2017; Nielsen et al., 2012). To summarize, our results related to VS RPE signals could add to the notion that the basic machinery that signals unexpected outcomes is unaffected in certain groups of PSZ. Direct comparisons of different patient groups or a systematic investigation of antipsychotic medication effects will be essential for developing a better understanding of the different neural mechanisms that may underlie RL impairments in the psychosis continuum.

Of note, we found subtle evidence of between-group differences in dmPFC activity related to learning rate modulation in our ROI analyses. In our model-based analysis, we observed a trend for HV to show greater learning rate modulation signals than PSZ, which should be interpreted with caution in light of the sample size and correction for multiple comparisons. Observations of reductions in error- and response conflict-related activity in dACC (Culbreth et al., 2016; Dowd et al., 2016), overlapping with dmPFC, and altered outcome-related signals in ACC and inferior frontal gyrus (Kerns et al., 2005; Polli et al., 2008) have been observed before in PSZ. Moreover, a study by Koch et al. (2010) revealed that frontal cortical activity changes in response to environmental volatility were absent in medicated PSZ. Taken together, there is growing evidence for disrupted frontal cortical signaling in association with expectation updating in PSZ. Here we show, for the first time, that some of these signals may relate specifically to impaired learning rate modulation. Given that learning rate signals have been shown to be dependent on catechol-o-methyl transferase (COMT) genotype (Krugel et al., 2009) and affected by catecholamine enhancement (Jepma et al., 2016), these deficits may arise from changes in brain catecholamine function. Finally, Collins et al. have previously shown in HV (2017) and PSZ (2014) that RL learning rate is critically affected by working memory, suggesting that some of our observed finding may be explained by working memory deficits. Future studies should investigate how dynamic learning rate modulation is affected by working memory demands, such as load and delay.

The current results additionally provide preliminary support for the idea that a reduced ability to adaptively update expectations in PSZ could be related to decreased coupling between regions that signal and utilize the RPE to update behavior; the strength of learning rate modulation-dependent dmPFC-VS connectivity was decreased in PSZ, and especially so in PSZ with more severe motivational deficits. This finding echoes those of Kaplan et al. (2016) who observed changes in ACC effective connectivity within a change-detection network. In that study, increased PFC-midbrain effective connectivity tracked with delusional severity, whereas we observed decreased dmPFC-VS connectivity as a function of motivational deficit severity. Thus, changes in frontal cortical connectivity may relate closely to RL deficits, with this mechanism potentially being differentially affected by certain symptom dimensions. The latter claim is supported by positive and negative symptom-specific alterations in cortico-striatal connectivity at rest (Sarpal et al., 2015; Wang et al., 2016; White et al., 2016).

To summarize, our current results suggest that aberrant learning rate modulation is a central feature of RL deficits in medicated PSZ, which can account for such deficits even in the presence of relatively unaffected RPE signaling. At the level of behavior, learning rate modulation impairments might manifest themselves as perseveration and/or a decreased tendency to sample the environment, thereby interfering with the formation of expected value. Functionally, we have provided initial support for the idea that learning rate modulation deficits might involve decreased embedding of the dmPFC within a greater RL network. Given its sensitivity to motivational deficits, learning rate modulation might serve as a mechanistic probe to evaluate the efficacy of future treatments aimed at alleviating negative symptoms of schizophrenia.

Some limitations to this study should be acknowledged. First, some of the reported fMRI results were only observed in ROIs or in whole-brain analyses at a lower voxel-wise threshold. Our ability to detect whole-brain group differences at a stringent whole-brain threshold may have been limited by our sample size and the use of patient subgroups (based on symptom severity). These results should therefore be considered preliminary, and replication is necessary. Moreover, while we found consistent evidence for learning rate modulation deficits at the neural (PPI) and behavioral (post-switch perseveration) in individuals with high negative symptoms, these measure were not directly correlated. This might suggest that our PPI phenotype may not underlie the observed behavioral deficits, although we would argue that this also strongly depends on our analysis approach. Specifically, our PPI analysis used all available experiment trials, while our modeling results pertained to highly specific trials surrounding contingency shifts.

Second, the ability to detect group differences in the neural signals of RL directly relates to how free parameters are calculated (Wilson & Niv, 2015). While model-based fMRI analyses of learning rate are robust to changes in parameter estimation it is not known exactly how this impacts estimates of RPE. Nevertheless, it is thought that poor model fitting should still be able to address questions related to the location of brain correlates of RL (and between-group differences therein).

Third, although previous studies have found PSZ to be impaired on reversal learning tasks (Schlagenhauf et al., 2014; Waltz & Gold, 2007), we only found subtle evidence for impairments. While our three-option task may be suitable to study learning rate dynamics, participants, on average, only achieved 6-7 stages, suggesting that on the majority of trials they were searching for the optimal deck. An increased number of decks to sample from may have made it more difficult for participants to achieve a reversal and, subsequently, reduced our ability to detect group differences in performance.

While the specific aim of this study was to disentangle RPE and LR signals in the brain, and how these may be altered in PSZ, there is a possibility that PSZ may show abnormal precision-weighted RPEs (the product of the LE and RPE), which has been shown to depend on dopamine function (Diederen et al., 2017). Future studies may wish to address how changes in learning rate dynamics in PSZ (with motivational deficits) may affect RPE-weighting.

Finally, this study was conducted in a sample of chronic, medicated PSZ. Although we did not find antipsychotic medication dose to be related to any of the reported outcome measures, direct comparison to a first-episode or prodromal cohort may be informative of the role of learning rate modulation deficits across the psychosis continuum.

Conclusions

RL deficits are common across the psychosis continuum. We show that aberrations in learning rate modulation in PSZ may drive RL deficits in some PSZ, even in the presence of accurately signaled RPEs. Our fMRI results additionally hint at the idea that learning rate modulation impairments relate to changes in dmPFC function and connectivity between dmPFC and other regions of the brain RL network. Furthermore, we observed greater abnormalities in learning rate modulation and associated neural signals in individuals with high motivational deficits. Abnormal learning rate modulation and associated brain function therefore might be an important mechanism involved in motivational deficits.