Short-term plasticity as cause–effect hypothesis testing in distal reward learning

Abstract

Asynchrony, overlaps, and delays in sensory–motor signals introduce ambiguity as to which stimuli, actions, and rewards are causally related. Only the repetition of reward episodes helps distinguish true cause–effect relationships from coincidental occurrences. In the model proposed here, a novel plasticity rule employs short- and long-term changes to evaluate hypotheses on cause–effect relationships. Transient weights represent hypotheses that are consolidated in long-term memory only when they consistently predict or cause future rewards. The main objective of the model is to preserve existing network topologies when learning with ambiguous information flows. Learning is also improved by biasing the exploration of the stimulus–response space toward actions that in the past occurred before rewards. The model indicates under which conditions beliefs can be consolidated in long-term memory, it suggests a solution to the plasticity–stability dilemma, and proposes an interpretation of the role of short-term plasticity.

Appendices

Appendix 1: Unlearning

Unlearning of the long-term components of the weights can be effectively implemented as symmetrical to learning, i.e., when the transient weights are very negative (lower than $-\varPsi$), the long-term component of a weight is decreased. This process represents the validation of the hypothesis that a certain stimulus–action pair is not associated with a reward anymore or that is possibly associated with punishment. In such a case, the neural weight that represents this stimulus–action pair is decreased and so is the probability of occurring. The conversion of negative transient weights to decrements of long-term weights, similarly to Eq.  (10), can be formally expressed as

$$\begin{aligned} \dot{w}^{lt}_{ji}(t) = -\rho \cdot H(-w^{st}_{ji}(t) - \varPsi ). \end{aligned}$$

(11)

No other changes are required to the algorithm described in the paper.

The case can be illustrated reproducing the preliminary test of Fig. 3, augmenting it with a phase characterized by a negative average modulation. Figure 10 shows that, when modulatory updates become negative on average (from reward 4,000 to reward 5,000), the transient weight detects it by becoming negative. The use of Eq. (11) then causes the long-term component to reduce its value, thereby reversing the previous learning.

Fig. 10

Unlearning dynamics. In this experiment, the model presented in the paper was augmented with Eq. (11), which decreases long-term weights if the transient weights are lower than $-\varPsi$. The stochastic modulatory update (top graph) is set to have a slightly negative average in the last phase (from reward 4,001 to 5,000). The negative average is detected by the short-term component that becomes negative. The long-term component decreases its value due to Eq. (11)

Preliminary experiments with unlearning on the complete neural model of this study show that the rate of negative modulation drops drastically as unlearning proceed. In other words, as the network experiences negative modulation, and consequently reduces the frequencies of punishing stimulus–action pairs, it also reduces the rate of unlearning because punishing episodes become sporadic. It appears that unlearning from negative experiences might be slower than that learning from positive experiences. Evidence from biology indicates that extinction does not remove completely the previous association (Bouton 2000, 2004), suggesting that more complex dynamics as those proposed here may regulate this process in animals.

Appendix 2: Implementation

All implementation details are also available as part of the open source Matlab code provided as support material. The code can be used to reproduce the results in this work, or modified to perform further experiments. The source code can be downloaded from http://andrea.soltoggio.net/HTP.

1.1 Network, inputs, outputs, and rewards

The network is a feed-forward single layer neural network with 300 inputs, 30 outputs, 9,000 weights, and sampling time of 0.1 s. Three hundred stimuli are delivered to the network by means of 300 input neurons. Thirty actions are performed by the network by means of 30 output neurons.

The flow of stimuli consists of a random sequence of stimuli each of duration between 1 and 2 s. The probability of 0, 1, 2 or 3 stimuli to be shown to the network simultaneously is described in Table 2.

The agent continuously performs actions chosen from a pool of 30 possibilities. Thirty output neurons may be interpreted as single neurons, or populations. When one action terminates, the output neuron with the highest activity initiates the next action. Once the response action is started, it lasts a variable time between 1 and 2 s. During this time, the neuron that initiated the action receives a feedback signal I of 0.5. The feedback current enables the output neuron responsible for one action to correlate correctly with the stimulus that is simultaneously active. A feedback signal is also used in Urbanczik and Senn (2009) to improve the reinforcement learning performance of a neural network.

The rewarding stimulus–action pairs are $(i,i)$ with $1 \le i \le 10$ during scenario 1, $(i,i-5)$ with $11 \le i \le 20$ in scenario 2, and $(i,i-20)$ with $21 \le i \le 30$ in scenario 3. When a rewarding stimulus–action pair is performed, a reward is delivered to the network with a random delay in the interval [1, 4] s. Given the delay of the reward, and the frequency of stimuli and actions, a number of stimulus–action pairs could be responsible for triggering the reward. The parameters are listed in Table 2. Table 3 lists the parameters of the neural model.

Table 2 Summary of parameters for the input, output and reward signals

Table 3 Summary of parameters of the neural model

1.2 Integration

The integration of Eqs. (3) and (2) with a sampling time $\Delta t$ of $100$ ms is implemented step-wise by

$$\begin{aligned} E_{ji}(t+\Delta t)&= E_{ji}(t) \cdot e^{\frac{-\Delta t}{\tau _{E}}} + \mathrm{RCHP}_{ji}(t)\end{aligned}$$

(12)

$$\begin{aligned} m(t+\Delta t)&= m(t) \cdot e^{\frac{-\Delta t}{\tau _{m}}} + \lambda \, r(t) + b. \end{aligned}$$

(13)

The same integration method is used for all leaky integrators used in this study. Given that $r(t)$ is a signal from the environment, it might be a one-step signal as in the present study, which is high for one step when reward is delivered, or any other function representing a reward: In a test of RCHP on the real robot iCub (Soltoggio et al. 2013a, b), $r(t)$ was determined by the human teacher by pressing skin sensors on the robots arms.

1.3 Rarely correlating Hebbian plasticity

Rarely correlating Hebbian plasticity (RCHP) (Soltoggio and Steil 2013) is a type of Hebbian plasticity that filters out the majority of correlations and produces nonzero values only for a small percentage of synapses. Rate-based neurons can use a Hebbian rule augmented with two thresholds to extract low percentages of correlations and decorrelations. RCHP expressed by Eq. (4) is simulated with the parameters in Table 4. The rate of correlations can be expressed by a global concentration $\omega _{c}$. This measure represents how much the activity of the network correlates, i.e., how much the network activity is deterministically driven by connections or is instead noise-driven. The instantaneous matrix of correlations $\mathrm{RCHP}^+$ (i.e. the first row in Eq. (4) computed for all synapses) can be low filtered as

$$\begin{aligned} \dot{\omega }_{c}(t) = -\frac{\omega _{c}(t)}{\tau _{c}} + \sum _{j = 1}^{300}\sum _{i = 1}^{30} \mathrm{RCHP}_{ji}^+(t), \end{aligned}$$

(14)

to estimate the level of correlations in the recent past, where $j$ is the index of input neurons, and $i$ the index of the output neurons. In the current settings, $\tau _{c}$ was chosen equal to 5 s. Alternatively, a similar measure of recent correlations $\omega _{c}(t)$ can be computed in discrete time over a sliding time window of 5 s summing all correlations $\mathrm{RCHP}^+(t)$

$$\begin{aligned} \omega _{c}(t) = \Delta t \frac{\sum _{0}^{t-5} \mathrm{RCHP}^+(t)}{5}. \end{aligned}$$

(15)

Similar equations to (14) and (15) are used to estimate decorrelations $\omega _{d}(t)$ from the detected decorrelations $\mathrm{RCHP}^-(t)$. The adaptive thresholds $\theta _{hi}$ and $\theta _{lo}$ in Eq. (4) are estimated as follows.

$$\begin{aligned} \theta _{hi}(t+\Delta t) = \left\{ \begin{array}{l@{\quad }l} \theta _{hi} + \eta \cdot \Delta t \,\, &{}\mathrm{if} \,\,\omega _{c}(t) > 2\mu \\ \theta _{hi} - \eta \cdot \Delta t \,\, &{}\mathrm{if} \,\,\omega _{c}(t) < \mu /2\\ \theta _{hi}(t)&{}\mathrm{otherwise} \end{array} \right. \end{aligned}$$

(16)

and

$$\begin{aligned} \theta _{lo}(t+\Delta t) = \left\{ \begin{array}{l@{\quad }l} \theta _{lo} - \eta \cdot \Delta t \,\, &{}\mathrm{if} \,\,\omega _{d}(t) > 2\mu \\ \theta _{lo} + \eta \cdot \Delta t \,\, &{}\mathrm{if} \,\,\omega _{d}(t) < \mu /2\\ \theta _{lo}(t)&{}\!\mathrm{otherwise}\\ \end{array} \right. \end{aligned}$$

(17)

with $\eta = 0.001$ and $\mu$, the target rate of rare correlations, set to 0.1 %/s. If correlations are lower than half of the target or are greater than twice the target, the thresholds are adapted to the new increased or reduced activity. This heuristic has the purpose of maintaining the thresholds relatively constant and perform adaptation only when correlations are too high or too low for a long period of time.

Table 4 Summary of parameters of the plasticity rules (RCHP and RCHP$^{+}$ plus HTP)