Abstract
In this study a combination of both the Hebbian-based and reinforcement learning rule is presented. The concept permits the Hebbian rules to update the values of the synaptic parameters using both the value and the sign supplied by a reward value at any time instant. The latter is calculated as the distance between the output of the network and a reference signal. The network is a spiking neural network with spike-timing-dependent synapses. It is tested to learn the XOR computations on a temporally-coded basis. Results show that the network is able to capture the required dynamics and that the proposed framework can reveal indeed an integrated version of both Hebbian and reinforcement learning. This supports adopting the introduced approach for intuitive signal processing and computations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hopfield, J.J.: Brain, neural networks, and computation. Rev. Mod. Phys. 71(2), S431–S437 (1999)
Gerstner, W., Kistler, W.: Mathematical Formulations of Hebbian Learning. Biological Cybernetics 87(5-6), 404–415 (2002)
Hebb, D.O.: The Organization of Behavior. Wiley and Son, New York (1949)
Pennartz, C.: Reinforcement learning by hebbian synapses with adaptive thresholds. Neuroscience 81(2), 303–319 (1997)
Farries, M.A., Fairhall, A.L.: Reinforcement Learning With Modulated Spike Timing Dependent Synaptic Plasticity. J. Neurophysiol. 98(6), 3648–3665 (2007)
Urbanczik, R., Senn, W.: Reinforcement learning in populations of spiking neurons. Nature Neuroscience 12(3), 250–252 (2009)
Kempter, R., Gerstner, W., van Hemmen, J.: Hebbian learning and spiking neurons. Phys. Rev. E 59(4), 4498–4514 (1999)
Klemm, K., Bornholdt, S., Schuster, H.G.: Beyond hebb: Exclusive-or and biological learning. Physical Review Letters 84, 3013 (2000)
Carnell, A.: An analysis of the use of hebbian and anti-hebbian spike time dependent plasticity learning functions within the context of recurrent spiking neural networks. Neurocomput. 72(4-6), 685–692 (2009)
Florian, R.V.: Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation 19(6), 1468–1502 (2007)
Lee, K., Kwon, D.S.: Synaptic plasticity model of a spiking neural network for reinforcement learning. Neurocomputing 71(13-15), 3037–3043 (2008)
Bosman, R.J.C., van Leeuwen, W.A., Wemmenhove, B.: Combining hebbian and reinforcement learning in a minibrain model. Neural Networks 17(1), 29–36 (2004)
de Queiroz, M.S., de Berrdo, R.C., de Pdua Braga, A.: Reinforcement learning of a simple control task using the spike response model. Neurocomputing 70(1-3), 14–20 (2006)
Markram, H., Wang, Y., Tsodyks, M.: Differential signaling via the same axon of neocortical pyramidal neurons. Proc. of the Nat. Academy of Sciences of the USA 95(9), 5323–5328 (1998)
Song, S., Miller, K.D., Abbott, L.F.: Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience 3(9), 2259 (2000)
Seung, H.: Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40(6), 1063–1073 (2003)
Xie, X., Seung, H.S.: Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E 69(4), 69–79 (2004)
El-Laithy, K., Bogdan, M.: Synchrony state generation in artificial neural networks with stochastic synapses. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds.) ICANN 2009. LNCS, vol. 5768, pp. 181–190. Springer, Heidelberg (2009)
El-Laithy, K., Bogdan, M.: Predicting spike-timing of a thalamic neuron using a stochastic synaptic model. In: ESANN Proceedings, pp. 357–362 (2010)
Namarvar, H.H., Liaw, J.S., Berger, T.W.: A new dynamic synapse neural network for speech recognition. In: Proc. IEEE Int. Conf. Neural Networks, pp. 2985–2990 (2001)
Liaw, J., Berger, T.W.: Dynamic synapse: A new concept of neural representation and computation. Hippocampus 6, 591–600 (1996)
Morrison, A., Diesmann, M., Gerstner, W.: Phenomenological models of synaptic plasticity based on spike timing. Science 98, 459–478 (2008)
El-Laithy, K., Bogdan, M.: On the Role of Synaptic Dynamics in the Generation of Synchrony States (2010)
van Rossum, M.C.W.: A Novel Spike Distance. Neural Comp. 13(4), 751–763 (2001)
Kimura, D., Hayakawa, Y.: Reinforcement learning of recurrent neural network for temporal coding. Neurocomputing 71(16-18), 3379–3386 (2008)
Fiete, I.R., Seung, H.S.: Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physical Review Letters 97(4), 048104 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
El-Laithy, K., Bogdan, M. (2010). A Hebbian-Based Reinforcement Learning Framework for Spike-Timing-Dependent Synapses. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds) Artificial Neural Networks – ICANN 2010. ICANN 2010. Lecture Notes in Computer Science, vol 6353. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15822-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-15822-3_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15821-6
Online ISBN: 978-3-642-15822-3
eBook Packages: Computer ScienceComputer Science (R0)