Neural Network Simulations of a Possible Role of the Hippocampus in Pavlovian Conditioning

Abstract

In this paper, we simulate the effects of hippocampal lesions on Pavlovian conditioning with an existing neural network model. According to the model, the hippocampus sends a diffuse discrepancy signal that modulates efficacies of synapses from primary sensory to polysensory areas. We hypothesize that this signal lessens the detrimental effects of momentary nonreinforcement and weaker cues on such efficacies in Pavlovian conditioning. Hippocampal lesions are thus hypothesized to exacerbate both detrimental effects. To test this hypothesis against some relevant animal evidence, we ran two computer simulations using simple feedforward neural networks with two hidden layers, designed according to the model. Hippocampal lesions were simulated by removing the networks’ hippocampal units. Networks were trained in various conditions involving momentary nonreinforcement and reinforcement of differently salient cues. The results were reasonably consistent with animal evidence that hippocampal lesion is more disruptive of long-delay than short-trace conditioning (Simulation 1) and backward more than contiguous-trace conditioning (Simulation 2). Implications, limitations, and future directions are discussed.

Appendix

Activation Rule

The model’s two main equations are the activation rule and the learning rule. The activation rule is used to compute the momentary level of activation of a neural processing unit j at moment t. The learning rule is used to compute the change in the weight of a connection from afferent (pre-connection, presynaptic) unit i to target (post-connection, postsynaptic) unit j. The activation rule is defined as follows:

$$ {a}_{j,t}=\left\{\begin{array}{l}{a}_{S^{\ast },t},\kern0.5em \mathrm{if}\kern0.5em {a}_{S^{\ast },t}>0\kern0.5em \mathrm{and}\kern0.75em j\kern0.5em \mathrm{is}\kern0.5em {V}_D\kern0.5em \mathrm{or}\kern0.5em {R}^{\ast}\kern0.5em \left(\mathrm{unconditional}\ \mathrm{activation}\right);\kern0.5em \mathrm{otherwise},\\ {}L\left({exc}_{j,t}\right)+{\uptau}_jL\left({exc}_{j,t-1}\right)\left[1-L\left({exc}_{j,t}\right)\right]-L\left({inh}_{j,t}\right),\\ {}\kern1em \mathrm{if}\kern0.5em L\left({exc}_{j,t}\right)>L\left({inh}_{j,t}\right)\ \mathrm{and}\ L\left({exc}_{j,t}\right)\ge {\uptheta}_t\left(\mathrm{reactivation}\right)\\ {}{a}_{j,t-1}-{\upkappa}_j\ {a}_{j,t-1}\left(1-{a}_{j,t-1}\right)-L\left({inh}_{j,t}\right),\\ {}\kern1em \mathrm{if}\kern0.5em L\left({exc}_{j,t}\right)>L\left({inj}_{j,t}\right)\ \mathrm{and}\ L\left({exc}_{j,t}\right)<{\uptheta}_{j,t}\left(\mathrm{decay}\right)\\ {}0,\kern0.5em \mathrm{if}\kern0.5em L\left({exc}_{j,t}\right)\le L\left({inh}_{j,t}\right)\ \left(\mathrm{deactivation}\right)\end{array}\right. $$

(23.1)

where j is a neural processing (hidden, output, H, or V_D) unit, t is a moment in time, and

$$ L(x)=\frac{1}{1+{e}^{\frac{-x+\mu }{\sigma }}} $$

is the logistic function with constant mean μ = 0.5 and standard deviation σ = 0.1 (a spontaneous activation free parameter). In this function

$$ x=\sum \limits_{i=1}^m{a_{i,t}}^{+}{w_{i,j,t}}^{+} $$

for exc_j,t, and

$$ x=\sum \limits_{i=1}^n{a_{i,t}}^{-}{w_{i,j,t}}^{-} $$

for inh_j,t, where m denotes the total number of excitatory units connected to j and n the total number of inhibitory units connected to j. No inhibitory units were used in this study, so the amount of inhibition was 0.0 for all units and networks in all simulations. Whether the rule is in reactivation or decay mode at t depends on a Gaussian threshold (θ_j,t), a random number generated according to a Gaussian distribution with a mean of 0.2 and standard deviation of 0.15. θ_j,t is dynamical, as it is generated at every moment for every computational unit. The other two activation free parameters are temporal summation (τ_j = 0.1) and decay (κ _j = 0.1). The same free parameters that were used here have been used in most previous simulation research with the model.

On input activations and stimulus traces. A network’s inputs in this model are not activated via the activation rule but manually, by just setting their activations according to some training protocol that simulates a conditioning procedure of interest. The protocol includes simulations of sensory stimuli typically used as cues in conditioning studies (e.g., lights, tones, noises, etc.) and biologically significant stimuli typically used as USs or primary reinforcers (e.g., food, water, electrical shocks, etc.). In previous research with this model, we have assumed that these activations are real-time primary sensory effects of external stimuli. Hence, we do not intend input activations in this model to simulate “traces” qua input activations in the absence of external stimuli but real-time effects of ongoing stimuli. Thus, a greater-than-zero input activation means in this model that a stimulus is effectively present, roughly at that moment.

Thus far in research with this model, we have had no need to conceive input activations as “stimulus traces.” The notion of a stimulus trace in models of Pavlovian conditioning was introduced ad hoc to account for trace conditioning in a way that fits the conventional wisdom that the CS always acquires some associative strength, even if absent. In Simulation 1, however, we simulated trace conditioning (and in Simulation 2, backward conditioning) as only context conditioning. We know this to be extreme and against conventional wisdom, but it is consistent with the evidence (see Marlin, 1981).

Learning Rule

The learning rule is defined as follows:

$$ \Delta {w}_{i,j,t}=\left\{\begin{array}{l}{\upalpha}_j{a}_{j,t}{d}_t{p}_{i,t}{r}_{j,t},\mathrm{if}\ {d}_t\ge 0.001\\ {}-{\upbeta}_j{w}_{i,j,t-1}{a}_{i,t}{a}_{j,t},\mathrm{otherwise}\end{array}\right. $$

(23.2)

where α (rate of weight increment) and β (the rate of weight decrement) denote the two free parameters of the rule (α = 0.5 and β = 0.1 for all connections; the same parameters have been used in most previous simulation research with this model). Ideally, the relative values of various parameters would reflect independent, experimentally determined values. All initial weights were set to 0.1 (cf. Burgos 2007).

The other terms of the learning rule are:

• a_i,t: activation of afferent unit (i), either excitatory or inhibitory

• a_j,t: activation of target unit (j)

• d_t = d_H,t = |a_H,t − a_H,t−1| + d_D,t(1 − d_H,t−1), if j is an S″ or H unit (see Fig. 23.1 for the different kinds of units and how they are connected)

• d_t = d_D,t = a_D,t − a_D,t−1, if j is an M″, D, or M′ unit; if d_D,t < 0.0, then d_D,t = 0.0

  $$ {p}_{i,t}=\frac{a_{i,t}{w}_{i,j,t-1}}{N},\kern1em \mathrm{where}\kern0.5em N={exc}_{j,t}\kern0.5em \mathrm{or}\kern0.5em N={inh}_{j,t} $$

• depending on whether i is excitatory or inhibitory, respectively

$$ {r}_{j,t}=1-\sum \limits_{i=1}^n{w}_{i,j,t}. $$

The key factor is d_t, a signal that modulates changes of all weights in the same moment (i.e., it is a diffuse signal), inspired by evidence on the roles of hippocampal (e.g., CA1, simulated by H units in the H architecture at the top of Fig. 23.1) and dopaminergic (e.g., ventral tegmental area, simulated by the D unit in Fig. 23.1) areas in conditioning. d_t also is a discrepancy signal in that it is defined as a temporal difference between the actual activations of H and D units in successive pairs of moments. The learning rule includes two such modulating signals: d_H,t, which depends on the activations of the H units, and d_D,t, which depends on the activations of the D units. As shown above, d_H,t is amplified by d_D,t. However, in the ~H networks (bottom panel, Fig. 23.1), d_H,t = 0.0, for which d_t = d_D,t. Hence, d_t in the ~H networks tends to be weaker than in the H networks.

The p_i,t and r_j,t factors introduce a “rich get richer” sort of competition among connections for a limited amount of weight (1.0) on a common target unit. In the network architectures used in the simulations (see Fig. 23.1), this competition took place on units that received two connections (viz., all the S″ and M″ units, as well as the D unit). The p_i,t factor, like some other models, includes a Hebbian component where connection weights partly depend on the co-activations of the connected (afferent and target) units.

In general, connections tend to gain weight (to a greater or lesser degree, depending on several factors) when S* (see Fig. 23.1) is activated and lose weight when S* is not activated. Successive timesteps with a zero S* activation thus promote weight loss. The same learning rule was used to modify the connection weights across all times, connections, networks, units, and training protocols.

All activations and weights are updated at every moment t according to an asynchronous random procedure. In this procedure, a randomly ordered list of all units (or connections) is generated at t, and new activations (or weights) are computed in that order (according to Eq. 23.1 for activations or Eq. 23.2 for weights). The activations (or weights) from t − 1 are immediately replaced by the new activations at t. Hence, by chance, the activation of a unit at t could depend on the activations of its afferents (some or all) at t − 1. Therefore, the propagation of activations across the network, from input to hidden to output layers , is not strictly sequential and synchronous.