Can’t Get You Out of My Head: A Connectionist Model of Cyclic Rehearsal

  • Herbert Jaeger
  • Douglas Eck
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4930)


Humans are able to perform a large variety of periodic activities in different modes, for instance cyclic rehearsal of phone numbers, humming a melody sniplet over and over again. These performances are, to a certain degree, robust against perturbations, and it often suffices to present a new pattern a few times only until it can be “picked up”. From an abstract mathematical perspective, this implies that the brain, as a dynamical system, (1) hosts a very large number of cyclic attractors, such that (2) if the system is driven by external input with a cyclic motif, it can entrain to a closely corresponding attractor in a very short time. This chapter proposes a simple recurrent neural network architecture which displays these dynamical phenomena. The model builds on echo state networks (ESNs), which have recently become popular in machine learning and computational neuroscience.


neural dynamics periodic attractors music processing Echo State Networks 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baddeley, A.: Working memory: looking back and looking forward. Nature Reviews: Neuroscience 4(10), 829–839 (2003)CrossRefGoogle Scholar
  2. 2.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (1994)CrossRefGoogle Scholar
  3. 3.
    Buonomano, D.V.: A learning rule for the emergence of stable dynamics and timing in recurrent networks. Journal of Neurophysiology 94, 2275–2283 (2005), CrossRefGoogle Scholar
  4. 4.
    Cariani, P.: Temporal codes, timing nets, and music perception. Journal of New Music Research 30(2), 107–135 (2001)CrossRefGoogle Scholar
  5. 5.
    Cooper, G., Meyer, L.B.: The Rhythmic Structure of Music. The Univ. of Chicago Press, Chicago (1960)Google Scholar
  6. 6.
    Daucé, E., Quoy, M., Doyon, B.: Resonant spatiotemporal learning in large random recurrent networks. Bological Cybernetics 87, 185–198 (2002)CrossRefzbMATHGoogle Scholar
  7. 7.
    Eck, D.: Finding downbeats with a relaxation oscillator. Psychological Research 66(1), 18–25 (2002), CrossRefMathSciNetGoogle Scholar
  8. 8.
    Eck, D.: Finding long-timescale musical structure with an autocorrelation phase matrix. Music Perception 24(2), 167–176 (2006)CrossRefGoogle Scholar
  9. 9.
    Eck, D.: Generating music sequences with an echo state network. In: NIPS 2006 Workshop on Echo State Networks and Liquid State Machines, Whistler, British Columbia (2006)Google Scholar
  10. 10.
    Eck, D., Schmidhuber, J.: Finding temporal structure in music: Blues improvisation with LSTM recurrent networks. In: Bourlard, H. (ed.) Neural Networks for Signal Processing XII, Proceedings of the 2002 IEEE Workshop, pp. 747–756. IEEE, New York (2002)CrossRefGoogle Scholar
  11. 11.
    Gouyon, F.: A computational approach to rhythm detection. Phd thesis, Dpt. of Technology of the University Pompeu Fabra, Barcelona (2005),
  12. 12.
    Hickok, G., Buchsbaum, B., Humphries, C., Muftuler, T.: Auditory-motor interaction revealed by fMRI: Speech, music, and working memory in area spt. Journal of Cognitive Neuroscience 15(5), 673–682 (2003)Google Scholar
  13. 13.
    Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  14. 14.
    Jaeger, H.: The ”echo state” approach to analysing and training recurrent neural networks. GMD Report 148, GMD - German National Research Institute for Computer Science (2001),
  15. 15.
    Jaeger, H.: Short term memory in echo state networks. GMD-Report 152, GMD - German National Research Institute for Computer Science (2002),
  16. 16.
    Jaeger, H.: Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the echo state network approach. GMD Report 159, Fraunhofer Institute AIS (2002),
  17. 17.
    Jaeger, H.: Generating exponentially many periodic attractors with linearly growing echo state networks. IUB Technical Report 3, International University Bremen (2006),
  18. 18.
    Jaeger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004),
  19. 19.
    Kaske, A., Maass, W.: A model for the interaction of oscillations and pattern generation with real-time computing in generic microcircuit models. Neural Networks 19(5), 600–609 (2006)CrossRefzbMATHGoogle Scholar
  20. 20.
    Krumhansl, C.: Cognitive Foundations of Muiscal Pitch. Oxford University Press, Oxford (1990)Google Scholar
  21. 21.
    Large, E., Palmer, C.: Perceiving temporal regularity in music. Cognitive Science 26, 1–37 (2002)CrossRefGoogle Scholar
  22. 22.
    Large, E., Kolen, J.: Resonance and the perception of musical meter. Connection Science 6(2/3), 177–208 (1994)CrossRefGoogle Scholar
  23. 23.
    Maass, W., Joshi, P., Sontag, E.: Computational aspects of feedback in neural circuits. PLOS Computational Biology 3(1), 1–20 (2007)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14(11), 2531–2560 (2002), CrossRefzbMATHGoogle Scholar
  25. 25.
    Mauk, M., Buonomano, D.: The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004)CrossRefGoogle Scholar
  26. 26.
    Mozer, M.C.: Neural network composition by prediction: Exploring the benefits of psychophysical constraints and multiscale processing. Cognitive Science 6, 247–280 (1994)Google Scholar
  27. 27.
    Palmer, C., Pfordresher, P.: Incremental planning in sequence production. Psychological Review 110, 683–712 (2003)CrossRefGoogle Scholar
  28. 28.
    Pasemann, F.: Characterization of periodic attractors in neural ring networks. Neural Networks 8(3), 421–429 (1995)CrossRefGoogle Scholar
  29. 29.
    Pongas, D., Billard, A., Schaal, S.: Rapid synchronization and accurate phase-locking of rhythmic motor primitives. In: Proc. of 2005 IEEE Conf. on Intelligent Robotics and Systems (IROS 2005), pp. 2911–2916 (2005)Google Scholar
  30. 30.
    Stevens, C., Wiles, J.: Representations of tonal music: A case study in the development of temporal relationship. In: Mozer, M., Smolensky, P., Touretsky, D., Elman, J., Weigend, A.S. (eds.) Proceedings of the 1993 Connectionist Models Summer School, pp. 228–235. Erlbaum, Hillsdale (1994)Google Scholar
  31. 31.
    Thaut, M., Kenyon, G., Schauer, M., McIntosh, G.: The connection between rhythmicity and brain function. IEEE Engineering in Medicine and Biology Magazine 18(2), 101–108 (1999)CrossRefGoogle Scholar
  32. 32.
    Todd, P.M.: A connectionist approach to algorithmic composition. Computer Music Journal 13(4), 27–43 (1989)CrossRefGoogle Scholar
  33. 33.
    van Gelder, T.: The dynamical hypothesis in cognitive science. Behavioural and Brain Sciences 21(5), 615–628 (1998)CrossRefGoogle Scholar
  34. 34.
    van Gelder, T., Port, R. (eds.): Mind as Motion: Explorations in the Dynamics of Cognition. Bradford/MIT Press (1995)Google Scholar
  35. 35.
    White, O.L., Lee, D.D., Sompolinsky, H.S.: Short-term memory in orthogonal neural networks. Phys. Rev. Lett. 92(14), 102–148 (2004)CrossRefGoogle Scholar
  36. 36.
    Widrow, B., Lehr, M.A.: Perceptrons, adalines, and backpropagation. In: Arbib, M. (ed.) The Handbook of Brain Theory and Neural Networks, pp. 719–724. MIT Press/Bradford Books (1995)Google Scholar
  37. 37.
    Williams, R.J., Zipser, D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. In: Chauvin, Y., Rumelhart, D.E. (eds.) Back-propagation: Theory, Architectures and Applications, pp. 433–486. Erlbaum, Hillsdale (1992)Google Scholar
  38. 38.
    Yang, W., Chong, N.Y.: Goal-directed imitation with self-adjusting adaptor based on a neural oscillator network. In: Proceedings, 12th International Conference on Advanced Robotics, ICAR 2005, pp. 404–410 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Herbert Jaeger
    • 1
  • Douglas Eck
    • 2
  1. 1.Jacobs University Bremen 
  2. 2.Department of Computer ScienceUniversity of MontrealCanada

Personalised recommendations