Encyclopedia of Computational Neuroscience

Living Edition
| Editors: Dieter Jaeger, Ranu Jung

Basal Ganglia: Songbird Models

  • Arthur LebloisEmail author
  • Ran Darshan
Living reference work entry
DOI: https://doi.org/10.1007/978-1-4614-7320-6_84-1

Keywords

Reinforcement Learning Inverse Model Auditory Feedback Efference Copy Hebbian Learning 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Definition

Songbirds produce complex vocalizations, a behavior that depends on the ability of juveniles to imitate the song of an adult. Song learning relies on a specialized basal ganglia-thalamocortical loop. Several computational models have examined the role of this circuit in song learning, shedding light on the neurobiological mechanisms underlying sensorimotor learning.

Detailed Description

Songbirds use learned vocalizations to communicate during courtship or aggressive behaviors. These vocalizations, called song, require fast coordination of laryngeal and respiratory muscles. Songbirds learn their song as juveniles through a long process comprising two sequential phases: the juvenile first listens to and memorizes one or more tutor songs and then uses auditory feedback to match its song to the memorized model through trial and error.

While song production is under the control of two cortical nuclei, HVC (used as a proper name) and the robust nucleus of the arcopallium (RA, Fig. 1a), song learning and plasticity strongly rely on the basal ganglia (BG, Scharff and Notthebohm 1991). Interestingly, songbirds display a specialized portion of their BG-thalamocortical circuitry devoted to song learning and plasticity (Fig. 1b; Brainard and Doupe 2002) and homologous to the mammalian motor BG-thalamocortical loop (Fig. 1c; Reiner et al. 1998). In particular, the song-related BG nucleus Area X receives dense dopaminergic innervation and contains neuron types homologous to both the mammalian striatal and pallidal neurons, forming similar circuits (Farries and Perkel 2002; Leblois et al. 2009). Songbirds and their specialized BG-thalamocortical loop devoted to a naturally learned complex sensorimotor task offer a unique opportunity to study the function of the BG in skill learning and execution (Doupe et al. 2005).
Fig. 1

The song-related BG-thalamocortical network in songbirds and its homologue circuit in mammals. In panels a, b, and c, the nature of synaptic transmission is indicated by the end of the arrow: black arrows for excitatory (glutamatergic) connections, black disks for inhibitory (GABAergic) connections, and gray diamonds for dopaminergic connections. (a) Schematic parasagittal representation of the song system. DLM, dorsolateral nucleus of the anterior thalamus; LMAN, lateral magnocellular nucleus of the anterior nidopallium; RA, robust nucleus of the arcopallium; VP, ventral pallidum; VTA, ventral tegmental area; nXIIts, supraspinal nucleus. (b) Diagram of the synaptic connectivity in the song-related BG-thalamocortical network of songbirds. (c) Diagram of the synaptic connectivity in the motor loop of the mammalian BG-thalamocortical network. GPe, globus pallidus pars externa; GPi, globus pallidus pars interna; SNc, substantia nigra pars compacta; STN, subthalamic nucleus; VA, ventral anterior nucleus of the thalamus; VL, ventral lateral nucleus of the thalamus

BG Functions in Songbirds: Experimental Background

The BG loop in songbirds is necessary for song learning but not for song production per se. BG neurons respond selectively to playbacks of the bird’s own song (Doupe and Solis 1997) and were initially thought to convey auditory feedback signals to be compared with a stored template of the tutor song. However, no neuronal correlate of such comparison can be found in the BG-thalamocortical loop (Leonardo 2004). Auditory feedback-related activity has however been reported in upstream cortical nuclei (HVC), where similar neuronal responses can be observed in response to syllable production or playback (“mirror neurons,” Prather et al. 2008). During song production, the BG-thalamocortical loop introduces motor variability allowing vocal exploration (Kao et al. 2005; Olveczky et al. 2005), as needed in a reinforcement learning (RL) framework, and can guide adaptive changes in song to minimize errors (Andalman and Fee 2009).

Reinforcement Learning in Songbirds

Most models of BG-dependent learning in songbirds are in a RL framework (but see Ganguli and Hahnloser 2013). In these models, a timing signal is assumed to be produced in HVC, in which output is sent to the BG to serve as a clock input during learning. Based on this clock input, the RL circuit learns to produce the correct motor gestures at each time step. In the RL framework, the agent learns to correlate random motor explorations with fluctuations in a reward signal in order to select motor patterns that lead to the highest reward. For example, changes in the song (the motor output) that increase the amount of expected reward should be implemented, whereas song changes leading to lower expected reward should be discarded. Implementation of such RL framework in the song-related neuronal circuitry must include four components: (1) an “actor” that produces the song; (2) a mechanism for exploration, implemented in a variability circuit (Fee and Goldberg 2011), a searcher (Doya and Sejnowski 1998), or an experimenter (Fiete et al. 2007); (3) a comparator circuit or “critic” (Doya and Sejnowski 1998) that computes the reward signal by evaluating the produced song with respect to the memorized template; (4) and a learning mechanism to modify motor output with time.

Models of Song Learning and the Involvement of the BG-Thalamocortical Loop

In a first RL model for song learning, Doya and Sejnowski (1998) proposed that nucleus LMAN could act as the “searcher,” perturbing each synaptic connection between HVC and RA independently, while the BG would evaluate each produced song using tutor-selective neurons (Fig. 2). This weight perturbation algorithm results in a plasticity in HVC-RA synapses after each epoch, i.e., at the end of song rendition.
Fig. 2

Architecture of the basic song RL models (Doya and Sejnowski 1998; Fiete et al. 2007). Dashed box represents the search/experimenter element. Gray box represents the critic. Plastic synapses are displayed in wide black arrows

Fiete et al. (2007) elaborated on the Doya and Sejnowski model in a biologically realistic network of spiking neurons. While the location of the critic in their model was only speculative, the other elements of the model were similar to Doya and Sejnowski (1998): plasticity took place in HVC-to-RA synapses and LMAN was again the search element, called the “experimenter.” Besides its realistic nature, their model differs from previous models by utilizing a new learning algorithm relying on node perturbations, where LMAN neurons perturb the activity of RA neurons instead of their synaptic efferences from HVC (Fig. 2), and an online reward signal during song. Using such a realistic learning algorithm, the circuit learns a simple song in a biologically plausible number of song renditions.

Recently, Fee and Goldberg (2011) have published a conceptual model that argues that the BG-thalamocortical loop is not a comparator circuit but is rather biasing the motor system (Fig. 3). Based on ideas taken from the classical implementation of RL in the mammalian motor BG loop (Wickens et al. 2003), they hypothesize that the song-related BG receive an evaluation signal from neuromodulators such as dopamine and use this signal to select the relevant motor gestures at the right time. As in mammals, the avian BG-thalamocortical circuit and the descending connections to motoneurons form a topographically organized circuit displaying segregated “motor channels.” In this model, LMAN again drives variability in song production and sends a copy of this variability signal to striatal neurons. Receiving a convergent timing signal (or “context”) from HVC and a global dopaminergic reward signal, the striatal neurons are considered as coincidence detectors and select the right motor variation in the right context in each motor channel. Synaptic plasticity at HVC-to-X synapses then allows the acquisition of correct motor biases. Although this model has not been computationally implemented yet, it is algorithmically similar to Fiete et al. (2007), with the learning occurring at HVC-to-X synapses instead of HVC-to-RA synapses.
Fig. 3

Architecture of the more complete song RL model including BG circuits (Fee and Goldberg 2011). Arrows and color code as in Fig. 2

Template Comparison, Efference Copy, or Inverse Model?

In the RL models discussed above, a global reward signal evaluating auditory feedback quality is delivered by the critic. Because of delays that are inherent in the transformation of neural activity into vocal gestures and in the auditory processing of produced sounds, the reward signal is necessarily delayed with respect to the neuronal activity underlying the rewarded gesture. In other words, the reward is likely temporally imprecise. To overcome this delay problem, an eligibility trace can be used (Doya and Sejnowski 1995; Fiete et al. 2007).

A different solution for the delay problem was given by Troyer and Doupe (2000, Fig. 4). While they still assume that the AFP evaluates the song, they propose that evaluation is based on an efference copy of the song’s motor program. In the first stage, a motor-to-sensory mapping is built by implementing Hebbian learning at synapses between HVC neurons projecting to RA (HVCRA), assumed to be motor, and HVC neurons projecting to Area X (HVCX), assumed to be auditory. The bird thereby learns to predict auditory consequences of its motor gestures. Such a mapping is usually termed a “forward model” or “efference copy.” After implementing the motor-to-sensory transformation (blue arrow in Fig. 4), the AFP neurons displaying tuned responses to tutor song evaluate the motor (efference) copy, whereas auditory feedback to HVCX is “gated on” by adaptation mechanisms to avoid interference. In this stage the AFP acts like a critic. As in Doya and Sejnowski, synapses within RA and from HVCRA to RA are modified to optimize song.
Fig. 4

Schematic description of the efference copy learning (a) and song learning (b) in the Troyer and Doupe (2000). Blue arrows represent the motor-auditory transformation in (a) and the “efference copy” of this transformation in (b). Other color code as in Fig. 2

The last model, proposed by Ganguli and Hahnloser (2013), consists of two brain areas, an auditory area and a motor area, represented in a very simplified framework. Their model relies on three already mentioned assumptions: production of highly variable vocalization during learning, gating mechanisms for specific synaptic inputs, and a synaptic mechanism for Hebbian learning. It is not based on a RL framework, and no evaluation of the song or comparison with the tutor’s song is needed. Instead, the bird learns an inverse model through the mapping of sensory (auditory) commands to the appropriate motor commands, relying solely on Hebbian learning. First, random vocalizations induce Hebbian learning between auditory and motor representation of the produced sounds (Fig. 5a), creating an inverse model. Later, the auditory feedback is gated off, while an alternative input to the auditory area activates sequentially the neuronal assemblies representing the tutor song (Fig. 5b). Thanks to the preceding learning of the inverse model in the auditory-motor connection, activation of the song representation in the auditory area drives the right motor pattern in the postsynaptic motor area. Thereby, the tutor song can be produced instantaneously.
Fig. 5

Architecture of the inverse model for song learning (Ganguli and Hahnloser 2013). Wide blue arrows represent the motor-auditory transformation. (a) Learning the inverse model (in wide black). (b) Song production. After learning the inverse model (dashed blue arrow), auditory feedback and the searcher element are gated, while the representation of syllables of the tutor song is sequentially activated in the auditory area

Conclusion

In summary, the models we have presented address the question of how sensorimotor learning leads to the faithful copy of a previously imprinted tutor song. The BG-thalamocortical loop plays the role of the actor-critic reinforcement circuit in these models, although learning of an inverse model, in the BG or elsewhere, may also participate to imitation. Whether these two types of learning coexist in the same neural circuits and how they may be combined to achieve song learning remains to be elucidated in future theoretical work.

References

  1. Andalman AS, Fee MS (2009) A basal ganglia-forebrain circuit in the songbird biases motor output to avoid vocal errors. Proc Natl Acad Sci USA 106:12518–12523Google Scholar
  2. Brainard MS, Doupe AJ (2002) What songbirds teach us about learning. Nature 417:351–358PubMedCrossRefGoogle Scholar
  3. Doupe AJ, Solis MM (1997) Song- and order-selective neurons develop in the songbird anterior forebrain during vocal learning. J Neurobiol 33:694–709PubMedCrossRefGoogle Scholar
  4. Doupe AJ, Perkel DJ, Reiner A, Stern EA (2005) Birdbrains could teach basal ganglia research a new song. Trends Neurosci 28:353–363PubMedCrossRefGoogle Scholar
  5. Doya K, Sejnowski T (1995) A novel reinforcement model of birdsong vocalization learning. Adv Neural Inf Process Syst 7:101–108.Google Scholar
  6. Doya K, Sejnowski TJ (1998) A computational model of birdsong learning by auditory experience and auditory feedback. In: Poon PWF, Brugge JF (eds) Central auditory processing and neural modeling. Plenum, New York, pp 77–88CrossRefGoogle Scholar
  7. Farries MA, Perkel DJ (2002) A telencephalic nucleus essential for song learning contains neurons with physiological characteristics of both striatum and globus pallidus. J Neurosci 22:3776–3787PubMedGoogle Scholar
  8. Fee MS, Goldberg JH (2011) A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience 198:152–170PubMedCentralPubMedCrossRefGoogle Scholar
  9. Fiete IR, Fee MS, Seung HS (2007) Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. J Neurophysiol 98:2038–2057PubMedCrossRefGoogle Scholar
  10. Ganguli S, Hahnloser RHR (2013) Bird song learning without reinforcement: the Hebbian self-organization of sensorimotor circuits. Soc Neurosci Abstr 105.12/YY4Google Scholar
  11. Kao MH, Doupe AJ, Brainard MS (2005) Contributions of an avian basal ganglia-forebrain circuit to real-time modulation of song. Nature 433:638–643PubMedCrossRefGoogle Scholar
  12. Leblois A, Bodor AL, Person AL, Perkel DJ (2009) Millisecond timescale disinhibition of thalamic neurons by relief of single spike basal ganglia GABAergic veto. J Neurosci 29:15420–15433PubMedCentralPubMedCrossRefGoogle Scholar
  13. Leonardo A (2004) Experimental test of the birdsong error-correction model. Proc Natl Acad Sci USA 101:16935–16940Google Scholar
  14. Olveczky BP, Andalman AS, Fee MS (2005) Vocal experimentation in the juvenile songbird requires a basal ganglia circuit. PLoS Biol 3:e153PubMedCentralPubMedCrossRefGoogle Scholar
  15. Prather JF, Peters S, Nowicki S, Mooney R (2008) Precise auditory-vocal mirroring in neurons for learned vocal communication. Nature 451:305–310PubMedCrossRefGoogle Scholar
  16. Reiner A, Medina L, Veenman CL (1998) Structural and functional evolution of the basal ganglia in vertebrates. Brain Res Brain Res Rev 28:235–285PubMedCrossRefGoogle Scholar
  17. Scharff C, Nottebohm F (1991) A comparative study of the behavioral deficits following lesions of various parts of the zebra finch song system: implications for vocal learning. J Neurosci 11:2896–2913PubMedGoogle Scholar
  18. Troyer TW, Doupe AJ (2000) An associational model of birdsong sensorimotor learning I. Efference copy and the learning of song syllables. J Neurophysiol 84:1204–1223PubMedGoogle Scholar
  19. Wickens JR, Reynolds JN, Hyland BI (2003) Neural mechanisms of reward-related motor learning. Curr Opin Neurobiol 13:685–690PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Centre de Neurophysique, Physiologie et pathologie (UMR CNRS 8119)Université Paris DescartesParisFrance
  2. 2.Interdisciplinary Center for Neural Computation (ICNC)The Hebrew University of JerusalemJerusalemIsrael