Conditioning and time representation in long short-term memory networks

Abstract

Dopaminergic models based on the temporal-difference learning algorithm usually do not differentiate trace from delay conditioning. Instead, they use a fixed temporal representation of elapsed time since conditioned stimulus onset. Recently, a new model was proposed in which timing is learned within a long short-term memory (LSTM) artificial neural network representing the cerebral cortex (Rivest et al. in J Comput Neurosci 28(1):107–130, 2010). In this paper, that model’s ability to reproduce and explain relevant data, as well as its ability to make interesting new predictions, are evaluated. The model reveals a strikingly different temporal representation between trace and delay conditioning since trace conditioning requires working memory to remember the past conditioned stimulus while delay conditioning does not. On the other hand, the model predicts no important difference in DA responses between those two conditions when trained on one conditioning paradigm and tested on the other. The model predicts that in trace conditioning, animal timing starts with the conditioned stimulus offset as opposed to its onset. In classical conditioning, it predicts that if the conditioned stimulus does not disappear after the reward, the animal may expect a second reward. Finally, the last simulation reveals that the buildup of activity of some units in the networks can adapt to new delays by adjusting their rate of integration. Most importantly, the paper shows that it is possible, with the proposed architecture, to acquire discharge patterns similar to those observed in dopaminergic neurons and in the cerebral cortex on those tasks simply by minimizing a predictive cost function.

Appendices

Appendix 1: Changes from (Rivest et al. 2010) and hyper-parameters search

The new simulations are done at 10 Hz instead of 5 Hz, do not use probe trials during training (only probe trials on frozen networks after training), and memory blocks have a single memory cell instead of two (see Sect. 4, Experiment 1). In the present study, some experiments tested networks with 1, 2, and 5 memory blocks while previous experiments always used 2 memory blocks $(M = 2)$. The TD hyper-parameters remained the same ($\alpha _\mathrm{{TD}} = 0.1,$ $= 0.9, \gamma ~= 0.98$) and are similar to those used by other authors (eg: Suri and Schultz 1999). The LSTM networks hyper-parameters were optimized for the new tasks and are now $(\alpha _\mathrm{{LSTM}} = 0.01, \beta = 1.0, \hbox {\char 106}_\mathrm{{LSTM}} = 0.0$).

We optimized the LSTM hyper-parameters $\alpha _\mathrm{{LSTM}} , \beta$, and (the last being the LSTM eligibility trace factor from the previous model) by performing an extensive grid search on all three tasks (Experiment 1) with different numbers of memory blocks $(M \in \{1,2,5\})$. Each hyper-parameter was varied within the range of powers $\{\mathrm{{sqrt}}(10^{i}) : i \in \{-4,\ldots ,2\}\}U\{0\}$. For each hyper-parameter setting, 30 networks were trained on each task as in Experiment 1. We looked at how many networks could learn the task in the allocated time and how fast they were learning it. The search revealed no clear improvement for nonzero LSTM eligibility traces factor in success rate or learning speed for the near optimal $(\alpha _\mathrm{{LSTM}}, \beta )$ pair. Therefore, we eliminated the LSTM eligibility trace learning mechanism from our previous paper and selected what seemed to be the best general values for all tasks and number of memory blocks confounded for the two other hyper-parameters $(\alpha _\mathrm{{LSTM}} = 0.01, \beta = 1.0)$.

Appendix 2: Model’s time interval learning capacity

To rapidly evaluate how large an interval the model could learn with the selected parameter, randomly initialized networks with a single memory block () were trained on 4-min training blocks of alternating trials and intertrials intervals as in Experiment 1. Under each conditioning paradigm, we started a pool of 20 randomly initialized networks for each CS–US onsets interval, starting with 1,000 ms and increasing by steps of 100 ms. For a given task, as soon as one of the networks had a successful block, we cleared up the pool and started a new pool with an interval 100 ms longer. A training block was considered successful when the LSTM network was able to properly predict its next input $({\vert }y_{\mathrm{{US}},t}- x_{\mathrm{{US}},t+1}{\vert } < 0.5)$ for 300 consecutive time steps (i.e., or about 5 consecutive trials). As in Experiment 1, each network was limited to 500 training blocks (5,000 for trace). Each network, independently of the task or interval length, was randomly initialized and therefore trained from scratch. We stopped increasing the CS–US onsets interval for a given task when no networks succeed for 5 different intervals. We have been able to successfully train a single memory block network for CS–US onsets intervals of 3.2 s (trace), 9.9 s (delay), and 5.1 s (extended).