# Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

- 987 Downloads
- 9 Citations

## Abstract

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called *eligibility traces* are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.

## Keywords

Differential Hebbian learning Correlation-based learning Temporal difference learning Reinforcement learning## 1 1 Introduction

Control tasks for animals or machines (systems) require sensible actions that follow from the current state of the system. As consequence of an action the state of the system may change and a new action will be elicited, and so on. This procedure describes a sequence of states and actions, which follow each other in time. Such control mechanisms can either be hardwired into the system, but many times it is more appropriate to design a learning algorithm that tries to infer the next action from the previous sequence of states. Especially in animal (or robot-) control the complexity of the world may prevent hardwiring and learning is required instead to assure enough flexibility.

In general three mechanisms can be used for learning in such situations: (1) Unsupervised learning (finding statistical structure), (2) reinforcement learning (the learner receives a fairly unspecific reward-signal about the success of its actions) and (3) supervised learning (the learner receives a specific error signal). The distinction between these algorithmic classes is to some degree fuzzy and we will here deal with classes one and two only.

Specifically there are: reward-based reinforcement learning (Kaelbling et al. 1996; Sutton and Barto 1998 abbreviated with RL) as well as correlation based differential Hebbian learning (Kosco 1986; Klopf 1986; Porr and Wörgötter 2003), here abbreviated with CL (correlation-based learning). These algorithms differ fundamentally in their design and goal.

Most influential up to date is reinforcement learning and substantial efforts have been undertaken during the last 15 years to develop a highly successful theory of RL (Sutton 1988; Dayan 1992; Dayan and Seynowski 1994; Kaelbling et al. 1996; Sutton and Barto 1998). Mostly RL methods had been developed in conjunction with machine learning (e.g., Q-learning, Watkins 1989; Watkins and Dayan 1992) and only fewer attempts exist to design architectures which want to be more compatible with (biological) neural networks (Montague et al. 1995, 1996; Suri and Schultz 1998, 1999, 2001; Suri et al. 2001; Arleo and Gerstner 2000; Strösslin et al. 2005). When studying machine learning, it becomes clear that there is no easy way to adopt the state-action space structures used for (e.g.) Q-learning to a neural network. Most often Actor-Critic Architectures (Witten 1977; Sutton and Barto 1998) are used which emulate some aspects of the basal ganglia and the prefrontal cortex. Other biologically relevant approaches have implemented TD-learning in the context of place field guided navigation models in the hippocampus (Foster et al. 2000; Krichmar et al. 2005).

Correlation based differential Hebbian learning has been invented around 1985 (Kosco 1986; Klopf 1986, 1988) after so-called stimulus substitution models of classical (and instrumental) conditioning had been introduced (Sutton and Barto, 1981). At that time these models had not been used in any behaving closed loop system and the early CL-models had soon been superceded by the highly influential method of “temporal difference learning” (TD-learning; Sutton 1988) by which CL-methods essentially died out and became replaced by RL-approaches to which TD-learning belongs. Through TD (and related methods) it became possible to reliably learn the values of states and actions and to control the learning of an agent through reinforcement signals. Only much later CL was revived when discovering a method of how to embed CL into closed-loop control systems guaranteeing convergence of the learning process (Porr and Wörgötter 2003; Porr et al. 2003). By this CL became a possible alternative to RL. Furthermore, it has been found (Saudargiene et al. 2004; Roberts 1999) that differential Hebbian learning is related to spike-timing dependent plasticity (STDP; Gerstner et al. 1996; Markram et al. 1997).

Hence currently one finds the following situation: TD-learning is compatible to neuronal circuitry mostly at the network level, while differential hebbian learning is more directly compatible to STDP at single neurons. Only very recently several attempts have been presented in the literature to show that there is a more direct equivalence existing between these two worlds and two different proofs have been presented how RL can be emulated by a CL formalism (Florian 2007; Santiago et al. 2007; Pfister et al. 2006; Izhikevich 2007). Alas, both approaches are quite complex and the relation between RL and CL is not really straight forward.

Therefore, it may make sense to step back and first provide some simple circuits by which RL and CL can be emulated at a single unit in a time-continuous and causal way from where on it might become easier to address the question to what degree both approaches are similar. In addition, the growing number of different learning rules, especially for CL, makes it useful to provide a comparison, which may help the reader to better understand these rules when wanting to implement them.

Note, this paper deals with the different learning rules only in an open-loop context. Behavioral feedback is not treated. This would represent a second step and is beyond the scope of the current paper.

## 2 2 Basic approaches

*x*as the signal input and

*u*as the filtered input (

*u*=

*x**

*h*) with

*h*being an arbitrary filter function (we will define a particular filter function later). The * denotes a convolution. Furthermore,

*gu*will be the neuronal sum of the inputs weighted by the weights

*ω*and prime will denote the derivative with respect to time:

*υ*′ = d

*υ*/d

*t*.

*u*

_{1}at the input and \(\bar \upsilon (t)\) at the output, given by

*ã*and

*~b*. Mainly they discuss the case of

*~b*= 0 where \(\bar \upsilon (t) = \upsilon (t - 1)\), which turns their rule into (Fig. 2a):

**S&B**:

*r*, which affects the learning, but not the output of the system (Fig. 2b). Synaptic weights change according to

*δ*-error of TD-learning, which is the mismatch between predicted (expected) and actual reward. The parameter

*γ*is called the discount factor, which accounts for the fact that distant rewards should be valued less. For practical purposes it makes sense to also introduce a learning rate factor

*μ*<< 1 in this equation, which allows controlling the speed of learning. Hence we have:

**TD**:

*x*

_{0}or

*r*, respectively. Originally these models had been designed with the time-scale of classical or instrumental conditioning in mind, which can be seconds between CS and US. If dealing with biological systems, ultimately, however, one must match all learning models to the time scales of synaptic plasticity. When doing this, it does not make sense to separate the inputs to a neuron from their eligibility traces. Rather it seems more reasonable to assume that spikes get transformed at a membrane into several much slower processes, some of which determine the membrane potential, while others may determine plasticity. For simplicity here we might just assume

*one*slow process at the input to a neuron which equally affects the neuron's output

*and*its learning. When doing this we arrive at diagram Fig. 2c, which is called ISO-learning (Porr and Wörgötter 2003). The ISO-learning rule is given by

**ISO**:

*x*

_{1}and

*x*

_{0}are filtered before they enter the neuronal summation, the ISO-learning rule behaves much different from the S&B model.

*x*

_{0}also as a reward, we obtain Fig. 2d for rephrased TD-learning, where the learning rule takes the following shape:

**TD-r**:

- 1.
At the beginning of learning the output

*υ*is dominated by*x*_{0}. Hence, in this formalism, rephrased TD-learning shines up as a combination of Hebbian learning with the term*u*_{0}(*t*)*u*_{1}≈*υ*(*t*)*u*_{1}and differential-Hebbian learning with the term*υ*′(*t*)*u*_{1}. - 2.
Also we find that this rule is (apart from a minus sign) identical to the ISO-learning rule when we set

*x*_{0}= 0, a case which is of great importance in the discussion on the different convergence conditions below.

## 3 3 Stability analysis for *x* _{0}, *r* = 0

In the following simulations sequences of two delta-pulses have been repetitively presented to the different systems, where *x* _{1} is earlier than *x* _{0} (or than *r*, respectively) with a constant interval *T* = 30 steps between them. The interval between pulse pairs was 300. The step by step weight change *Δω* is calculated by integrating the respective learning rule: \(\Delta \omega = \smallint _0^\infty {{{\rm{d}}\omega (t)} \over {{\rm{d}}t}}{\rm{d}}t\). From this the development of the weights can be plotted. Learning rates *μ* have often been adjusted to yield similarly strong weight growth for the different systems when wanting to compare rules.

In addition we will show the different weight change curves plotting the weight change against the interval between inputs *T*. Note, strictly these curves reflect weight changes only for *t* → 0, hence when *w* _{1} is still approximate zero. For negative *T* the temporal order of the pulses is inverted.

*two*eligibility traces are being used. This case applies only if one has good knowledge about the temporal difference

*T*between the two incoming stimuli. Often, however, knowledge about

*T*is limited, then one needs to use a set of eligibility traces for spreading out the earlier stimulus across time to make sure that at least some of these signals can be related to the later occurring

*x*

_{0}or

*r*signal. This situation is discussed later (see filter bank approach in Fig. 8).

*E*we use in all cases a band-pass filter response defined by

*t*) being the Heaviside (step) function. Actual parameters are given in the figure legends.

We are especially interested in the stability of the learning rules. All rules learn by cross-correlating two signals with each other (*x* _{1} with *x* _{0} or with *r*), correlations of *x* _{1} with itself (auto-correlations) are normally unwanted. Hence it is of interest to subdivide the contributions of the learning rule into a cross- and an auto-correlation term by: \(\Delta w = \Delta {\omega^{ac}} + \Delta {\omega^{cc}}\), the latter term drives the weight change of *ω* _{1} during the occurrence of *x* _{0} (or *r*), whereas the autocorrelation term also changes the weight in the absence of the *x* _{0} signal. Hence, the pure auto-correlation contribution becomes visible when switching *x* _{0} (or *r*) off.

This is what we do in the following diagrams by setting *x* _{0} = 0 at time-step *t* = 6000 to show if the weight change for a given rule will then indeed stop. This is an important case in a closed-loop system which, for instance, has to avoid a reflex triggered by the *x* _{0} signal (Porr and Wörgötter 2003; Porr et al. 2003; Krichmar et al. 2005).

Let us now calculate the auto- and cross-correlation contributions for the rules introduced in the previous section.

*ω*′ << 1), exchanged

*u*

_{1}(

*t*) with

*h*(

*t*) and

*x*

_{1}(

*t*) and

*x*

_{0}(

*t*) with

*δ*(

*t*) and

*δ*(

*t*−

*T*), respectively. Additionally we used \(\smallint _0^\infty \delta '(t - {t_0})f(t) = - f'({t_0})\) (Boykina 2003).

**S&B**:

**TD**:

*h*associated with the

*x*

_{0}or rather

*r*input (which is set to

*ω*

_{0}

*δ*(

*T*) for comparison).

*ω*

_{1}will grow (left panels) until the later pulse is switched off. In the S&B model (Fig. 3a) one sees that the unfiltered input

*x*

_{0}and its derivative lead to strong, needle-like excursions of the weight growth for every step, which let the line in the diagram appear broad. These disappear as soon as

*x*

_{0}is switched off. In TD-learning (Fig. 3c) according to Sutton 1988, weights grow about ten times faster, which is due to the fact that the reward brings in only positive contributions. When we switch

*x*

_{0}(or

*r*) off, then we find in both cases that

*ω*

_{1}drops slightly. Note, this is not what ought to be done in a TD-rule. Switching off

*δ*would be the correct condition and obviously weights will be—by construction—stable then. Still, we want to look at the

*r*= 0 case, because it directly corresponds to the

*x*

_{0}= 0 condition of the CL-rules and shows the behavior of the pure auto-correlation contribution.

Weight drop of the S&B- and TD-rule is due to the fact that the remaining total influence on the weight growth, which now comes only from the auto-correlation term is negative. As mentioned above, for *x* _{0} = *r* = 0, both diagrams (Fig. 2a,c) are identical (see Eqs. Equ12, 13) and the remaining differences in weight drop come from the different values of *ω* _{1}, reached at switch-off time.

*t*→ ∞ as the filter function

*h*(

*t*) possesses only one maximum and eventually decays to zero.

Panels f and g of Fig. 3 show the relaxation behavior of *Δω* for a single input pulse pair. In Fig. 3f a small early auto-correlation component (AC) is followed by a big, cross-correlation dominated hump (AC + CC) as soon as *x* _{0} occurs. The curve relaxes to the final weight value only after some time depending on the filter characteristic of *h*. In Fig. 3g we magnify the auto-correlation component for a situation where we have switched *x* _{0} off (auto-correlation only!). The dashed curve shows that, following Eq. 14, the auto-correlation indeed drops to zero after a short time. This curve was numerically calculated with a ten-fold increased sampling rate as compared to the solid-line curve shown next to it. This represents the auto-correlation contribution, when using coarser sampling and here we see a potentially very strong source of error: The auto-correlation contribution does not vanish anymore. This is a pure numerical artifact of the integration procedure, but—as high sampling rates are often too costly (for example in real-time applications)—this artifact can strongly interfere with the convergence of ISO. Hence, we are facing two potential sources of error: (1) The tardy relaxation behavior of (essentially) the cross-correlation term (Fig. 3f). This error becomes relevant when pulse pairs follow each other in time too quickly. And (2) the non-negligible numerical error that renders the auto-correlation non zero even for long relaxation times. The other CL-rules discussed below have been invented to solve these problems.

Figure 3e shows the step-by-step behavior of ISO-learning. Weight growth is similar to that of the TD-rule. Here we find that, after switching off *x* _{0}, weights will drift upwards. This effect comes from the afore discussed numerical error and vanishes for very small integration step sizes *Δt* and large relaxation times *t* as shown by the dashed lines.

*u*

_{0}, which now also enters the learning rule. The contributions are as follows:

**TD-r**:

*h*(

*t*) is depicted as

*H*(

*t*). Otherwise, its behavior is very similar to that of the ISO-rule. As noted above, switching

*x*

_{0}off, transforms TD-Rephrased into ISO-learning.

When looking at the weight-change curves (Fig. 3b,d,h,k), we observe that the S&B model produces strong negative weights for values of *T* ≈ 0 (Fig. 3b). This effect had already been reported in their original papers (Sutton and Barto 1990).

TD-learning produces an asymmetrical weight change curve (Fig. 3d). This is a consequence of the missing E-trace in the reward pathway. If being early, the reward has already vanished before *x* _{1} occurs and the correlation result remains zero.

The rephrased TD-rule can produce mixed properties (Fig. 3k). As such it will early during learning, as long as *ω* ≈ 0, produce essentially plain heterosynaptic Hebbian learning. This is due to the fact that the term *u* _{0}, which enters the rule, has a much stronger influence than the derivative (as discussed above), which can be neglected. Only if we introduce an artificial attenuation factor *α*, by which we weaken the *u*0-input, we will receive a transition towards differential Hebbian learning, which will become identical to ISO-learning for *α* = 0. The inset shows magnifications of the curves. This also confirms that the plain, filtered input has a much stronger influence on the learning amplitude than the derivative.

### 3.1 3.1 Conditions of convergence

When do all these different algorithms converge? Trivially, weight growth at *ω* _{1} will stop as soon as *x* _{1} = 0 in all cases. Also we find that for the CL-rules weights will converge if *T* = 0. Hence these systems will be essentially stable if small positive values of *T* are followed by small negative ones (or vice versa). Other possible convergence conditions, however, are less obvious.

For the old S&B model at that time convergence properties had not been investigated. Clearly Fig. 2 shows that for *υ*′ = 0 this model will converge. But this condition is not useful as it requires the output to be constant. Hence, strictly for the old S&B model convergence cannot be guaranteed. On the other hand, ISO-learning is strictly convergent for *x* _{0} = 0 and weight *ω* _{1} will then stop to change. When switching *x* _{0} off (Fig. 3e), one can see that *ω* _{1} is (almost) stable for small learning rates (*μ* = 0.001). For a higher learning rate, weights drift upwards. This is the above mentioned consequence of the high numerical sensitivity of ISO learning to the integration step size.

*μ*and was proven by Sutton (Sutton 1988; Dayan 1992). Here, only the expected value of the output \({\left\langle {\upsilon (t)} \right\rangle _{t \ge {t_c}}}\) (where

*t*

_{ c }denotes the time convergence has been achieved) converges to an optimal value. In order to ensure convergence of the output

*υ*itself, the learning rate has to decrease over time

*μ*(

*t*) ∼

*f*(

*t*) with the condition \(\Sigma _{t = 0}^\infty f(t) = \infty \) and \(\Sigma _{t = 0}^\infty {f^2}(t) < \infty \) [e.g.

*f*(

*t*) = 1/

*t*); Dayan 1992]. Since we use time independent learning rates in this article, only the mean of the output should be constant at the end of learning. Thus both types of convergence do not need a vanishing delta error

*δ*, instead just its average over time has to be zero: \({\left\langle \delta \right\rangle _{t \ge {t_c}}} = 0\). The delta error

*δ*itself oscillates without any reward

*r*around zero. Simply due to the shape of the band-pass the weight is decreasing over time and does not stabilize as mentioned before. Consequently the convergence condition for TD-learning with a constant learning rate

*μ*is

Note, for this condition, the output *υ*, or rather its derivative, needs to take on a certain value as opposed to ISO-learning, where the input *x* _{0} needs to become zero. Hence, we have one algorithm (TD-learning) where convergence is guaranteed by output-control as opposed to one other algorithm (ISO-learning), which uses input-control to guarantee convergence. Looking back at Fig. 2b, it becomes clear that setting *r* = 0 does not enforce convergence. This had been only done for auto-correlation term evaluation (comparison to the ISO-approaches).

As discussed earlier, the rephrased TD-rule shown in Fig. 2d carries properties of both algorithmic classes, convergence can be achieved for 〈*δ*〉 = 0 but also with *x* _{0} = 0, because in this case the learning circuit is identical to the ISO-learning circuit, as already mentioned above.

*δ*〉 = 0 into an input condition. In order to attempt this, we need to use input terms only. Let us again use TD-Rephrased to show this and define

*x*

_{0}as the reward signal (Fig. 2d). Then we have \(\upsilon = {\omega _0}{u_0} + {\omega _1}{u_1}\). Using this in Eq. 18 we get as convergence condition for the synapse

*ω*

_{1}:

*ω*

_{1}, which we are supposed to stabilize, shines up on the right side. Hence, phrased in this way as an input condition, it cannot be fulfilled.

## 4 4 Features and problems of the basic architectures

- 1.
CL-rules are theoretically stable for

*x*_{0}= 0, but ISO learning is highly sensitive to numerical errors which can easily destroy convergence. - 2.
Input

*x*_{0}is connected to the output*υ*in the CL-rules. Thus, CL-rules will always produce a (motor) output which can be used to generate actions. - 3.
By construction, TD-learning is stable for 〈

*δ*〉 = 0. - 4.
As we are dealing with a single neuron with only two inputs one finds that, without additional assumptions, TD-learning cannot produce actions: The original TD-learning as depicted in Fig. 2b does not produce any output at all if starting with

*ω*_{1}= 0. Also, setting*ω*_{1}≠ 0 is nonsense from a conceptional point of view. This would assume that Pawlov's dog has already some knowledge about the meaning of the bell even before the first learning experience. This, however, can be a desired aspect of TD learning as it has led to the situation that TD-learning, when considered neuronally, has only been used to*judge*the quality of but not to actually*generate*actions. Actor and Critic generically remain separate in architectures that use TD-learning to implement RL (Barto 1995). Other approaches are using additional inputs to produce a (motor) output signal (Strösslin et al. 2005). While this works, we are here concerned with a more puristic view of only using two inputs unequivocally separating Actor and Critic.

## 5 5 Modified approaches

In order to address the above stated problems it is possible to modify the existing approaches.

First we ask, can we enforce stability in CL for *x* _{0} = 0 without this unwanted numerical instability? For this we need to design a learning rule for which the auto-correlation term truly vanishes.

*x*

_{0}. Hence we are only correlating inputs with each other, thus, the name of this rule: Input correlation learning (ICO, Porr and Wörgötter 2006). The learning rule is given by

**ICO**:

**ICO**:

*x*

_{0}= 0. The inset in Fig. 5a shows the relaxation behavior for a single pulse-pair. In comparison to ISO learning (inset in Fig 3c) the shallow initial rising phase in missing here as there is no auto-correlation contribution. For the same reason also the “hump” is a little bit smaller. This effect, however, is barely visible even when overlaying the curves. Incidentally, ICO is identical to ISO in the limit of

*μ*→ 0. The ICO-rule has proven to be very useful in difficult learning tasks (Porr and Wörgötter 2006). In fact this rule reliably works even with very high learning rates and will always converge if one manages to bring

*x*

_{0}down to zero. One should, however, notice that ICO-learning is a form of non-Hebbian (heterosynaptic) plasticity, which may be less realistic from a biological point of view.

The second question concerns the problem to design a TD-learner that will also produce some motor output when learning starts (hence, when *ω* _{1} is still zero).

The rephrased TD-rule in Fig. 2d will indeed do this. The output will be driven by *x* _{0} early during learning and this could be used to induce behavior. Remember, this rule has two ways to enforce stability: Either 〈*δ*〉 = 0 or *x* _{0} = 0, which makes this rule a strange chimera between TD and ISO, also rendering it sensitive to the numerical auto-correlation problems. The question, thus, is can we find a rule that needs to enforce 〈*δ*〉 = 0 only and still can produce some output at the start of learning? Figure 4b shows one novel possible such architecture called Acting-TD. Here we feed the *δ*-error into the *x* _{0} input line. Note, this system achieves stability *only* for 〈*δ*〉 = 0 and will always produce some output *υ*.

**TD-a**:

*υ*is recursively defined as

Figure 5c–f show how this system behaves in the open-loop condition. As expected Acting-TD is stable for 〈*δ*〉 = 0 (panel c). Pure Hebbian learning is implemented by this rule (panel e), even when attenuating the reward input with factor *α*. The feedback of *δ* into the input of the neuron, however, leads to a potential destabilization. To show this we have introduced the amplification factor *β* into the derivative of the output *υ*′. If this pathway enters the neuron with *β* larger than one (*β* > 1), destabilization occurs as shown in Figs. 5d. For values of *β* ≦ 1, the circuit, however, remains stable. Figure. 5f shows what happens when we invert the *x* _{0} pathway, by setting its synaptic weight *ω* _{0} to negative values. This leads to an unstable situation during weight growth and to a strong drop as soon as 〈*δ*〉 = 0.

Does this circuit produce reasonable behavioral output? At least we can make some statements about the shape of the output at the start and the end of learning. At the start we have *ω* _{1} = 0 and we receive \(\upsilon = {\omega _0}\bar r\), which makes sense. When the system has converged (〈*δ*〉 = 0) we get *υ* = *ω* _{1} *u* _{1}, which essentially amounts to a process of stimulus substitution as required from such models. The 〈*δ*〉 = 0 condition can also be written (though disregarding the averaging 〈〉) as \(0 = \bar r - {\omega _1}{{u'}_1}\). Hence we find that, if converged, \(r = - {\omega _1}{{u'}_1}\) or, more specifically, \({\left\langle r \right\rangle _{t \ge {t_c}}} = - {\left\langle {{\omega _1}{{u'}_1}} \right\rangle _{t \ge {t_c}}}\) should be fulfilled.

## 6 6 Symmetrical learning rules

So far we were able to address the problem of divergence, which was present in the basic CL-rules (Fig. 2) having achieved stable temporal sequence learning when employing the ICO-rule. Next we would like to ask if it is possible to implement learning (LTP) at one synapse and unlearning (LTD) at the other synapse *at the same time*. In principle this should be possible because one synapse experiences +*T* while the other experiences −*T* for any given input pair. Thus, causality is inverted for the two synapses and with the right design the one should grow, while the other would shrink.

Clearly TD-learning is designed in an asymmetrical way and cannot easily be symmetrized. This is different for the CL rules.

Figure. 6a and e show two isotropic setups. The one (Fig. 6a) for ISO-learning (which gave this rule its name) and the other for ICO-learning.

**ISO-sym**:

**ICO-sym**:

**ICO sym**:

For ISO-learning, naively one would expect that with the same starting weights (*ω* _{0} = *ω* _{1}) one should get exactly anti-symmetrical learning, because the positive influence of +*T* at one synapse should find its exact counterpart at the other synapse which experiences −*T*. At least the learning curve in Fig. 3c, seems to suggest this. However, this is *in general not* the case due to the fact that the overlap of the filter function is not symmetrical relative to the pulse pair. Hence, one does indeed find that one synapse grows while the other shrinks, but *not* in an anti-symmetrical way. This is shown in Fig. 6b, where we ran the analysis for very many time-steps. Both weights behave anti-cyclically and the observed oscillation grows as a consequence of the increasingly more influential auto-correlation term. Phase relationship between both synapses, however, remains the same (see inset in Fig. 6b). The result in Fig. 6b was obtained with an optimal choice of parameters *ω* _{0}, *ω* _{1} and *T* leading to an almost ideal anti-symmetrical learning, shown in Fig. 6c, which depicts the first learning steps. As mentioned, this situation is not generic. More often much more asymmetrical situations like in Fig. 6d are observed, which was obtained with the same setup only using a different value for *T*.

Symmetrical ICO-learning (Fig. 6e) produces a linear phase-relation, which is not shown here, but Fig. 6f shows instead that both weights develop in an accurate anti-symmetrical way. Weight development will exactly follow the weight change curve in Fig. 5b. There is however a problem. Symmetrical ICO-learning does not anymore have one shared control parameter for the weight change, which for symmetrical ISO-learning was the derivative of the output. For symmetrical ICO-learning, two totally independent control parameters exist (the derivatives of the inputs). This can possibly lead to problems when wanting to control behavior with such a symmetrized ICO-rule.

## 7 7 Stabilizing ISO-learning with a third factor

ICO-learning is very stable but, as mentioned above, it is a form of non-Hebbian learning, where the output does not influence the learning. This may be undesirable in certain cases. Therefore, we undertook the effort to stabilize our (Hebbian) ISO-learning rule (Porr and Wörgötter 2007). This can be achieved using a third factor, which we call the “relevance signal” *R* (Fig. 7a). For practical purposes most of the time we set it equal to *x* _{0}, but one should realize that—like the reward line in TD-learning—*R* is indeed an independent signal. The signal *R* is meant to arise when for the animal/agent a behaviorally relevant event occurs.

**ISO3**:

**ISO3**:

*T*

_{ R }, which regulates the timing of the third factor, and a very narrow filter signal with appropriate values of

*a*

_{ R }and

*b*

_{ R }(here we use

*a*= 0.6,

*b*= 0.66) to define the third factor. It is interesting that the auto-correlation term for ISO3 is in general unequal zero when using two filters pointing to a possible instability.

Figure 7b shows the signal structure. Let us assume that *u* _{1} reaches its maximum exactly at *T*. As *υ*′(*t*) = *u*′_{1}(*t*), *t* < *T*, we have \(\mathop {\lim }\limits_{t \to T\_} \upsilon (t)' = {{u'}_1}(T) = 0\). This is the situation depicted in panel (b). If we furthermore assume that the *R* signal is very short (e.g., using a delta-pulse for *R*) and that it also happens at *T*, then also the learning only takes place at this moment in time. Under these conditions, it is easy to see that we have totally eliminated the auto-correlation contribution. The outcome of panel c is obtained under these condition and ISO3 is stable (compare ISO and ISO3 in Fig. 7c). The inset shows again the relaxation behavior of ISO3 (step) for a single pulse-pair in comparison to ISO (curved), which demonstrates instantaneous relaxation of ISO3. Clearly this example is constructed as *T* is usually unknown such that \(\mathop {\lim }\limits_{t \to T\_} \upsilon (t)' = 0\) cannot be generally assured. Hence, it seems we have not gained anything so far by introducing ISO3. However, the situation changes when using a filter bank to spread signal *x* _{1} out in time (see also section on filter banks below). Then one can prove that the condition \(\mathop {\lim }\limits_{t \to T\_} \upsilon (t)' = 0\) will self-emerge as a consequence of the learning when using enough filters (Porr and Wörgötter 2007). Thus, when using a filter bank ISO3 becomes a very stable method, indeed. For ISO3 we receive the learning curve in Fig. 7d, which now only carries an LTP component. This behavior compares to TD-learning (see Fig. 3d).

## 8 8 Results when using a filter bank

The usefulness of all these rules as presented so far remains limited as most of the time the interval *T* between incoming inputs is not good enough known and might, in a behaving agent, even vary to quite some degree. Hence it is required to use a set of different eligibility traces *E* _{1} ^{1,…,N } to make sure that the earlier input is spread out over a long enough time such that the later input (*x* _{0} or *r*) can be correlated to it. Figure 8a depicts such a filter bank architecture for the ISO rule and panel b shows how the signals *u* _{1} ^{1,…,N } look like for a set of filters *h*.

Interestingly, convergence properties for the CL-rules are theoretically not affected when using a filter bank. It can be shown that for the ISO rule a set of filters *h* exist that fulfills a certain orthogonality criteria and ISO will then still converge for *x* _{0} = 0 (Porr and Wörgötter 2003; Porr et al. 2003). The problem is that this is only an existence proof and nothing is currently known of how to actually construct this filter bank. Hence, when wanting to use ISO one has to fall back onto heuristic assumptions for the filter bank. Generally, however, this leads to the situation that the error-sensitivity of ISO can become larger, rendering this rule instable. The conclusion is that, while it has been the first differential Hebbian learning rule to be used in closed loop behavioral control (Porr and Wörgötter 2003), ISO should only be used with great care anymore.

The properties of ICO and ISO3 are better. Both rules are stable and stability for *x* _{0} = 0 can be mathematically proved for both rules even when using filter banks (Porr and Wörgötter 2006, 2007). These rules have now been successfully tested in a variety of different applications (Porr and Wörgötter 2006; Kolodziejski et al. 2006, 2007; Manoonpong et al. 2007) and even chains of learning neurons can be constructed in a convergent way (Kulvicius et al. 2007).

Neuronal implementations of TD-learning generally solve the problem of unknown *T* by using a so called serial compound representation (Suri and Schultz 1998, 2001; Suri et al. 2001; Suri 2002). This is depicted in Fig. 8c. Essentially it represents a tapped delay line by which the earlier stimulus is spread out across several input lines *u* _{1} ^{1,…,N } and for which one can even graphically show that TD will converge (Fig. 8d, e). This approach has been discussed in greater detail in Wörgötter and Porr (2005). Such a rigid pulse protocol is not biologically realistic, though, and approaches exist which replace this by “smoother functions”, similar to the filter bank approach discussed above (Suri and Schultz 1998). These signals do not affect the *δ*-error, and convergence can still be assured for 〈*δ*〉 = 0. As the filter response enter the learning via *υ*′, it remains unclear of how to construct more realistic filter banks for TD-learning which in a behaving system will still allow approaching 〈*δ*〉 = 0 in a reliable way. So far there is no theory existing for this.

## 9 9 Discussion

Overview over all learning rules discussed in this paper

Class | Two Inputs | ||||
---|---|---|---|---|---|

Type | Name | Output: | Rule: d | Convergence | Comment |

CL | S&B | | u | divergent | stimulus substitution |

ISO | | | | symmetric, diff. Hebb | |

ICO | | | | diff. Heterosyn. | |

ISO3 | | \(u_1 \upsilon^\prime \bar R^\prime\) | | 3-factor diff. Hebb | |

RL | TD | | \(\delta_r u_1 = \bar r u_1 + u_1 \upsilon^\prime\) | 〈 | critic only, no actions |

TD-r | | \(\delta_{u0} u_1 = u_0 u_1 + u_1 \upsilon^\prime\) | 〈 | mixed Hebb + diff. Hebb | |

TD-a | \(w_0 \delta_r^\prime + w_1 u_1\) | \(\delta_r u_1 = \bar r u_1 + u_1 \upsilon^\prime\) | 〈 | recursive rule |

Class | Filter Bank | Summary | |||
---|---|---|---|---|---|

Type | Name | Output: | Convergence | General Comment | Use |

CL | S&B | not applicable | not applicable | only of historical relevance | − |

ISO | \(w_0 u_0 + \Sigma_i w_1^i u_1^i*\) | | unstable! As optimal | − | |

unknown | convergence cannot be guaranteed, input control | ||||

ICO | \(w_0 u_0 + \Sigma_i w_1^i u_1^i*\) | | robust, heterosynaptic, input control | + | |

ISO3 | \(w_0 u_0 + \Sigma_i w_1^i u_1^i*\) | | robust, input control | + | |

RL | TD | serial compound rep. | 〈 | robust, output control | + |

TD-r | not tested so far | 〈 | undesirable mix of Hebb & diff. Hebb | − | |

TD-a | not tested so far | 〈 | not tested so far | ? |

Concerning (a) we have found that all TD-rules are stable if one can assure *δ* = 0 or leastwise \({\left\langle \delta \right\rangle _{t \ge {t_c}}} = 0\). From the other rules the ICO and ISO3 rules are stable for *x* _{0} = 0. ISO is only theoretically stable but strongly affected by numerical artifacts.

Concerning (b) we can state that only the ICO-rule is truly symmetrical. Hence, with this rule it would be possible to implement learning at one input (the one that experiences +*T*) and unlearning at the other input (the one that experiences −*T*) *at the same time*. With ISO this is to some degree possible but growth and shrinkage are not the same, while all TD-rules are by construction not symmetrical.

Concerning (c) we observe that ICO and ISO3 will maintain their convergence properties for *x* _{0} = 0. TD-learning will converge when using a serial compound representation (see Fig. 8c). This sitation is very similar to a state-space tiling performed in machine learning version of TD. On the other hand, no clear theoretical results exist for TD, when using different, more neuron-compatible filters for the eligibility traces.

An additional consideration concerns the difference between input- and output-control. The CL-rules enforce convergence via *x* _{0} = 0, which represents an input condition. TD learning converges via output control (\({\left\langle \delta \right\rangle _{t \ge {t_c}}} = 0\)). This difference may lead also to differences for control applications as discussed elsewhere (Wörgötter and Porr 2005).

### 9.1 9.1 Biological relevance

To what degree are the above discussed models related to temporal sequence learning mechanisms in the brain. At first we notice that certainly all of them are at a much higher level of abstraction as compared to the biophysics of synapses. But, for example, we note that the learning curve of ISO or ICO learning resembles curves measured for spike timing-dependent plasticity (Markram et al. 1997; Magee and Johnston 1997; Bi and Poo 2001). Hence it is possible to model STDP with such a formalism (Saudargiene et al. 2004; Roberts 1999).

Furthermore, it has long been discussed that TD-learning could be related to dopaminergic responses in the brain. Especially the behavior of some cells in the substantia nigra and ventral tegmental area (VTA) suggest that they represent the *δ*-error of TD-learning. Models, which behave in a similar way have been made by Suri and co-workers (Montague et al. 1996; Suri and Schultz 1998, 1999, 2001; Suri et al. 2001) and we direct the reader to this literature for an in-depth discussion. The problem, with these models, however, is that it is difficult to find appropriate biophysical equivalents for the implementation of the TD-rule.

Concerning the CL-rules, there are different degrees of realism. For the ICO-rule we find that it represents a special case. This is due to the fact that ICO-learning implements plain heterosynaptic plasticity and this is found only at a few specialized synapses (Humeau et al. 2003; Tsukamoto et al. 2003). Heterosynaptic plasticity is usually associated with modulatory processes and not directly with Hebbian learning. This is different for the ISO-rule, which uses the derivative of the output to control learning and therefore represents conventional (differential) Hebbian learning.

The instability of the ISO-rule, however was the reason for us to design ISO3, which is a form of (differential) Hebbian learning using a three-factor learning rule (Miller et al. 1981). Such three-factor rules have recently also been discussed in conjunction with the Dopaminergic system of the brain (Schultz 1998). Also, since it is a Hebb-rule, it is better suited to be matched to our knowledge concerning LTP and LTD. Furthermore, we found, quite unexpectedly, that for weight stabilization ISO3 can use one interesting aspect of the behavior of dopamine cells in the substantia nigra and VTA (Schultz et al. 1997): These cells appear to learn anticipating a reward, whereby the temporal occurrence of their response shifts from (first) *t* _{ x0} to (later) *t* _{ x1}. When doing this with our relevance signal, we found that learning stops and that weights become essentially stable even without setting *x* _{0} = 0 (data not shown). Bringing the average TD-error
\({\left\langle \delta \right\rangle _{t \ge {t_c}}} = 0\) down to zero does require the dopamine responses to take a very specific shape whereas for stabilizing weights in ISO3 it is enough to get a somewhat sharp response at *x* _{1} while loosing the R-signal at *x* _{0}. This seems to be better in conjunction with the properties of dopaminergic responses which do not appear to fulfill high accuracy requirements.

Thus, an experimental question now arises: Do the dopamine cells in the substantia nigra and/or VTA represent the *δ*-error in TD-learning or do they reflect a relevance signal to be used as a third factor in the learning?

^{1}

## Footnotes

- 1.
A quasi-static approach is commonly assumed for such calculations (Dayan and Abbott 2003) and is justified as long as

*μ*is small. It is needed for neglecting the derivative of the weight*w*_{1}on the right hand side of the equation and for neglecting the variability of the homogeneous solution for the calculation of the inhomogeneous part.

## Notes

### Acknowledgment

The authors acknowledge the support of the European Commission, IP-Project “PACO-PLUS” (IST-FP6-IP-027657). We are grateful to T. Kulvicius for help with some figures.

### Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

## References

- Arleo A, Gerstner W (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol Cybern 83(3): 287–99CrossRefPubMedGoogle Scholar
- Barto A (1995) Adaptive critics and the basal ganglia. In: Houk JC, Davis JL, Beiser DG(eds) Models of information processing in the basal ganglia. MIT Press, Cambridge, pp 215–32Google Scholar
- Bi GQ, Poo M (2001) Synaptic modification by correlated activity: Hebb’s postulate revisited. Annu Rev Neurosci 24: 139–66CrossRefPubMedGoogle Scholar
- Boykina TB (2003) Derivatives of the dirac delta function by explicit construction of sequences. Am J Phys 71(5): 462–68CrossRefGoogle Scholar
- Dayan P (1992) The convergence of TD(λ). Mach Learn 8(3/4): 341–62CrossRefGoogle Scholar
- Dayan P, Abbott L (2003) Theoretical Neuroscience, Computational and mathematical modeling of neural systems. MIT Press, CambridgeGoogle Scholar
- Dayan P, Seynowski T (1994) TD(λ) converges with probability 1. Mach Learn 14: 295–01Google Scholar
- Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468–502CrossRefPubMedGoogle Scholar
- Foster DJ, Morris RGM, Dayan P (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10: 1–6CrossRefPubMedGoogle Scholar
- Gerstner W, Kempter R, van Hemmen JL, Wagner H (1996) A neuronal learning rule for sub-millisecond temporal coding. Nature 383: 76–8CrossRefPubMedGoogle Scholar
- Hull CL (1939) The problem of stimulus equivalence in behavior theory. Psychol Rev 46: 9–0CrossRefGoogle Scholar
- Hull CL (1943) Principles of behavior. Appleton Century Crofts, New YorkGoogle Scholar
- Humeau Y, Shaban H, Bissiere S, Luthi A (2003) Presynaptic induction of heterosynaptic associative plasticity in the mammalian brain. Nature 426(6968): 841–45CrossRefPubMedGoogle Scholar
- Izhikevich E (2007) Solving the distal reward problem through linkage of stdp and dopamine signaling. Cerebral Cortex 101093/cercor/bhl152Google Scholar
- Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4: 237–85Google Scholar
- Klopf AH (1972) Brain function and adaptive systems—a heterostatic theory. Technical report, Air Force Cambridge Research Laboratories Special Report No. 133, Defense Technical Information Center, Cameron Station, Alexandria, VA 22304Google Scholar
- Klopf AH (1982) The hedonistic neuron: a theory of memory, learning, and intelligence. Hemisphere, Washington DCGoogle Scholar
- Klopf AH (1986) A drive-reinforcement model of single neuron function. In: Denker JS (ed) Neural networks for computing: AIP Conference Proceedings. American Institute of Physics, New York, vol 151Google Scholar
- Klopf AH (1988) A neuronal model of classical conditioning. Psychobiology 16(2): 85–23Google Scholar
- Kolodziejski C, Porr B, Wörgötter F (2006) Fast, flexible and adaptive motor control achieved by pairing neuronal learning with recruitment. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS*2006, EdinburghGoogle Scholar
- Kolodziejski C, Porr B, Wörgötter F (2007) Anticipative adaptive muscle control: Forward modeling with self-induced disturbances and recruitment. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS*2007, TorontoGoogle Scholar
- Kosco B (1986) Differential Hebbian learning. In: Denker JS (eds) Neural networks for computing: AIP Conference proceedings, American Institute of Physics, New York, vol 151Google Scholar
- Krichmar JL, Seth AK, Nitz DA, Fleischer JG, Edelman GM (2005) Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics 3: 197–22CrossRefPubMedGoogle Scholar
- Kulvicius T, Porr B, Wörgötter F (2007) Chained learning architectures in a simple closed-loop behavioural context. Biol Cybern. doi: 10.1007/s00422-007-0176-y PubMedGoogle Scholar
- Magee JC, Johnston D (1997) A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science 275: 209–13CrossRefPubMedGoogle Scholar
- Manoonpong P, Geng T, Kulvicius T, Porr B, Wörgötter F (2007) Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput Biol 3(7): e134CrossRefPubMedGoogle Scholar
- Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213–15CrossRefPubMedGoogle Scholar
- Miller JD, Sanghera MK, German DC (1981) Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sci 29: 1255–263CrossRefPubMedGoogle Scholar
- Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive hebbian learning. Nature 377: 725–28CrossRefPubMedGoogle Scholar
- Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16(5): 1936–947PubMedGoogle Scholar
- Pfister JP, Toyoizumi T, Barber D, Gerstner W (2006) Optimal spike-timing dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18: 1309–339CrossRefGoogle Scholar
- Porr B, Wörgötter F (2003) Isotropic sequence order learning. Neural Comput 15: 831–64CrossRefPubMedGoogle Scholar
- Porr B, Wörgötter F (2006) Strongly improved stability and faster convergence of temporal sequence learning by utilising input correlations only. Neural Comput 18: 1380–412CrossRefPubMedGoogle Scholar
- Porr B, Wörgötter F (2007) Learning with ‘relevance’ Using a third factor to stabilise hebbian learning. Neural Comput (in press)Google Scholar
- Porr B, von Ferber C, Wörgötter F (2003) ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioral paradigm. Neural Comput 15: 865–84CrossRefPubMedGoogle Scholar
- Roberts P (1999) Computational consequences of temporally asymmetric learning rules: I. differential hebbian learning. J Comput Neurosci 7(3): 235–6CrossRefPubMedGoogle Scholar
- Santiago RA, Roberts PD, Lafferriere G (2007) Spike timing dependent plasticity implements reinforcement learning. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS*2007, TorontoGoogle Scholar
- Saudargiene A, Porr B, Wörgötter F (2004) How the shape of pre- and postsynaptic signals can influence STDP: a biophysical model. Neural Comp 16: 595–26CrossRefGoogle Scholar
- Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1–7PubMedGoogle Scholar
- Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: 1593–599CrossRefPubMedGoogle Scholar
- Singh SP, Sutton RS (1996) Reinforcement learning with replacing eligibility traces. Mach Learn 22: 123–58Google Scholar
- Strösslin T, Sheynikhovich D, Chavarriaga R, Gerstner W (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Netw 18(9): 1125–140CrossRefPubMedGoogle Scholar
- Suri RE (2002) TD models of reward predictive responses in Dopamine neurons. Neural Netw 15(4-6): 523–33CrossRefPubMedGoogle Scholar
- Suri RE, Schultz W (1998) Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res 121: 350–54CrossRefPubMedGoogle Scholar
- Suri RE, Schultz W (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neurosci 91(3): 871–90CrossRefGoogle Scholar
- Suri RE, Schultz W (2001) Temporal difference model reproduces anticipatory neural activity. Neural Comp 13(4): 841–2CrossRefGoogle Scholar
- Suri RE, Bargas J, Arbib MA (2001) Modeling functions of striatal dopamine modulation in learning and planning. Neurosci 103(1): 65–5CrossRefGoogle Scholar
- Sutton R, Barto A (1981) Towards a modern theory of adaptive networks: Expectation and prediction. Psychol Rev 88: 135–70CrossRefPubMedGoogle Scholar
- Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3: 9–4Google Scholar
- Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. In: Gabriel M, Moore J(eds) Learning and computational neuroscience: foundation of adaptive networks. MIT Press, CambridgeGoogle Scholar
- Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, 2002nd edn. Bradford Books, MIT Press, CambridgeGoogle Scholar
- Tsukamoto M, Yasui T, Yamada MK, Nishiyama N, Matsuki N, Ikegaya Y (2003) Mossy fibre synaptic NMDA receptors trigger non-Hebbian long-term potentiation at entorhino-CA3 synapses in the rat. J Physiol 546(3): 665–75CrossRefPubMedGoogle Scholar
- Watkins CJCH (1989) Learning from delayed rewards. PhD thesis, University of Cambridge, CambridgeGoogle Scholar
- Watkins CJCH, Dayan P (1992) Technical note: Q-Learning. Mach Learn 8: 279–92Google Scholar
- Witten IH (1977) An adaptive optimal controller for discrete-time Markov environments. Inf Control 34: 86–95CrossRefGoogle Scholar
- Wörgötter F, Porr B (2005) Temporal sequence learning for prediction and control - a review of different models and their relation to biological mechanisms. Neural Comput 17: 245–19CrossRefPubMedGoogle Scholar