Encyclopedia of Computational Neuroscience

Living Edition
| Editors: Dieter Jaeger, Ranu Jung

Computational Models of Neuromodulation

  • Angela J. YuEmail author
Living reference work entry
DOI: https://doi.org/10.1007/978-1-4614-7320-6_625-1


Neuromodulatory systems serve a special meta-processing role in the brain. Due to their anatomical privileges of having massively extensive arborization patterns throughout the central and peripheral nervous systems and to their physiological capacity of finely controlling how other neurons communicate with each other and plasticize, they are ideally positioned to regulate the way information is acquired, processed, utilized, and stored in the brain. As such, neuromodulation has been a fertile ground for computational models for neural information processing, which have strived to explain not only how the major neuromodulators coordinate to enable normal sensory, motor, and cognitive functions but also how dysfunctions arise in psychiatric and neurological conditions when these neuromodulatory systems are impaired.

Detailed Description

Although much still remains unknown or opaque about neuromodulatory functions, a computationally sophisticated and coherent picture is...


Prediction Error Pupil Diameter Reward Prediction Error Neuromodulatory System Signal Prediction Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.


  1. Ammassari-Teule M, Maho C, Sara SJ (1991) Clonidine reverses spatial learning deficits and reinstates θ frequencies in rats with partial fornix section. Behav Brain Res 45:1–8PubMedCrossRefGoogle Scholar
  2. Aston-Jones G, Cohen JD (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28:403–450PubMedCrossRefGoogle Scholar
  3. Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20(7):1391–1397PubMedCrossRefGoogle Scholar
  4. Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS (2007) Learning the value of information in an uncertain world. Nature Neurosci 10(9):1214–1221PubMedCrossRefGoogle Scholar
  5. Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Rev 28:309–369PubMedCrossRefGoogle Scholar
  6. Boureau Y-L, Dayan P (2011) Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 36:74–97PubMedCentralPubMedCrossRefGoogle Scholar
  7. Cohen JD, Frank MJ (2009) Neurocomputational models of basal ganglia function in learning, memory and choice. Behav Brain Res 199:141–156PubMedCentralPubMedCrossRefGoogle Scholar
  8. Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N (2012) Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482:85–88PubMedCentralPubMedCrossRefGoogle Scholar
  9. Cools R (2011) Dopaminergic control of the striatum for high-level cognition. Curr Opin Neurobiol 21:402–407PubMedCrossRefGoogle Scholar
  10. Córdova, Yu, Chiba (2004) Soc Neurosci AbstrGoogle Scholar
  11. Coull JT, Frith CD, Dolan RJ, Frackowiak RS, Grasby PM (1997) The neural correlates of the noradrenergic modulation of human attention, arousal and learning. Eur J Neurosci 9(3):589–598PubMedCrossRefGoogle Scholar
  12. Daw ND, Kakade S, Dayan P (2002) Opponent interactions between serotonin and dopamine. Neural Netw 15:603–616PubMedCrossRefGoogle Scholar
  13. Dayan P (2012) Twenty-five lessons from computational neuromodulation. Neuron 76:240–256PubMedCrossRefGoogle Scholar
  14. Dayanik S, Yu AJ (2012) Reward-rate maximization in sequential identification under a stochastic deadline. SIAM J Control Optim 51(4):2922–2948CrossRefGoogle Scholar
  15. Deakin JFW (1983) Roles of brain serotonergic neurons in escape, avoidance and other behaviors. J Psychopharmacol 43:563–577Google Scholar
  16. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15(4–6):495–506PubMedCrossRefGoogle Scholar
  17. Dyon-Laurent C, Romand S, Biegon A, Sara SJ (1993) Functional reorganization of the noradrenergic system after partial fornix section: a behavioral and autoradiographic study. Exp Brain Res 96:203–211PubMedCrossRefGoogle Scholar
  18. Dyon-Laurent C, Hervé A, Sara SJ (1994) Noradrenergic hyperactivity in hippocampus after partial denervation: pharmacological, behavioral, and electrophysiological studies. Exp Brain Res 99:259–266PubMedCrossRefGoogle Scholar
  19. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433PubMedCrossRefGoogle Scholar
  20. Fotiou DF, Stergiou V, Tsiptsios D, Lithari C, Nakou M, Karlovasitou A (2009) Cholinergic deficiency in Alzheimer’s and Parkinson’s disease: evaluation with pupillometry. Int J Psychophysiol 73(2):143–149PubMedCrossRefGoogle Scholar
  21. Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17:51–72PubMedCrossRefGoogle Scholar
  22. Frank MJ, Seeberger LC, OReilly RC (2004) By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 306:1940–1943PubMedCrossRefGoogle Scholar
  23. Gerfen CR, Engber TM, Mahan LC, Susel Z, Chase TN, Monsma FJJ, Sibley DR (1990) D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science 250:1429–1432PubMedCrossRefGoogle Scholar
  24. Gilzenrat MS, Nieuwenhuis S, Jepma M, Cohen JD (2010) Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn Affect Behav Neurosci 10:252–269PubMedCentralPubMedCrossRefGoogle Scholar
  25. Hutchins JB, Corbett JJ (1997) The visual system. In: Haines DE (ed) Fundamental neuroscience. Churchill Livingstone, New York, pp 265–284Google Scholar
  26. Ikemoto S, Panksepp J (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Rev 31:6–41PubMedCrossRefGoogle Scholar
  27. Jepma M, Nieuwenhuis S (2011) Pupil diameter predicts changes in the exploration-exploitation trade-off: evidence for the adaptive gain theory. J Cogn Neurosci 23:1587–1596PubMedCrossRefGoogle Scholar
  28. Johnson J, Li W, Li J, Klopf A (2001) A computational model of learned avoidance behavior in a one-way avoidance experiment. Adapt Behav 9:91CrossRefGoogle Scholar
  29. Körding KP, Wolpert DM (2004) Bayesian integration in sensorimotor learning. Nature 427:244–247PubMedCrossRefGoogle Scholar
  30. Kravitz AV, Tye LD, Kreitzer AC (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci 15:816–818PubMedCentralPubMedCrossRefGoogle Scholar
  31. Little JT, Johnson DN, Minichiello M, Weingartner H, Sunderland T (1998) Combined nicotinic and muscarinic blockade in elderly normal volunteers: cognitive, behavioral, and physiological responses. Neuropsychopharmacology 19(1):60–69PubMedCrossRefGoogle Scholar
  32. Maia TV (2010) Two-factor theory, the actor-critic model, and conditioned avoidance. Learn Behav 38:50–67PubMedCrossRefGoogle Scholar
  33. McClure SM, Daw ND, Montague PR (2003) A computational substrate for incentive salience. Trends Neurosci 26:423–428PubMedCrossRefGoogle Scholar
  34. Miyazaki K, Miyazaki KW, Doya K (2011) Activation of dorsal Raphe serotonin neurons underlies waiting for delayed rewards. J Neurosci 31:469–479PubMedCrossRefGoogle Scholar
  35. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16:1936–1947PubMedGoogle Scholar
  36. Moutoussis M, Bentall RP, Williams J, Dayan P (2008) A temporal difference account of avoidance learning. Network 19:137–160PubMedCrossRefGoogle Scholar
  37. Murschall A, Hauber W (2006) Inactivation of the ventral tegmental area abolished the general excitatory influence of Pavlovian cues on instrumental performance. Learn Mem 13:123–126PubMedCrossRefGoogle Scholar
  38. Nakamura K, Hikosaka O (2006) Role of dopamine in the primate caudate nucleus in reward modulation of saccades. J Neurosci 26:5360–5369PubMedCrossRefGoogle Scholar
  39. Nassar MR, Rumsey KM, Wilson RC, Parikh K, Heasly B, Gold JI (2012) Rational regulation of learning dynamics by pupil-linked arousal systems. Nat Neurosci 15:1040–1046PubMedCentralPubMedCrossRefGoogle Scholar
  40. Niv Y, Daw ND, Joel D, Dayan P (2007) Tonic dopamine: opportunity cost and the control of response vigor. Psychopharmacology (Berl) 191:507–520CrossRefGoogle Scholar
  41. Sara SJ (1989) Noradrenergic-cholinergic interaction: its possible role in memory dysfunction associated with senile dementia. Arch Gerontol Geriatr Suppl 1:99–108PubMedGoogle Scholar
  42. Sara SJ, Dyon-Laurent C, Guibert B, Leviel V (1992) Noradrenergic hyperactivity after fornix section: role in cholinergic dependent memory performance. Exp Brain Res 89:125–132PubMedCrossRefGoogle Scholar
  43. Satoh T, Nakai S, Sat T, Kimura M (2003) Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci 23:9913–9923PubMedGoogle Scholar
  44. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593–1599PubMedCrossRefGoogle Scholar
  45. Schweimer JV, Ungless MA (2010) Phasic responses in dorsal raphe serotonin neurons to noxious stimuli. Neuroscience 171:1209–1215PubMedCrossRefGoogle Scholar
  46. Smith Y, Bevan MD, Shink E, Bolam JP (1998) Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 86:353–387PubMedCrossRefGoogle Scholar
  47. Surmeier DJ, Ding J, Day M, Wang Z, Shen W (2007) D1 and D2 dopamine receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30:228–235PubMedCrossRefGoogle Scholar
  48. Surmeier DJ, Shen W, Day M, Gertler T, Chan S, Tian X, Plotkin JL (2010) The role of dopamine in modulating the structure and function of striatal circuits. Prog Brain Res 183:149–167PubMedGoogle Scholar
  49. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3(1):9–44Google Scholar
  50. Talmi D, Seymour B, Dayan P, Dolan RJ (2008) Human Pavlovian-instrumental transfer. J Neurosci 28:360–368PubMedCentralPubMedCrossRefGoogle Scholar
  51. Thiel CM, Fink GR (2008) Effects of the cholinergic agonist nicotine on reorienting of visual spatial attention and top-down attentional control. Neuroscience 152(2):381–390PubMedCrossRefGoogle Scholar
  52. Ungless MA, Magill PJ, Bolam JP (2004) Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303:2040–2042PubMedCrossRefGoogle Scholar
  53. Yu AJ, Dayan P (2003) Expected and unexpected uncertainty: ACh and NE in the neocortex. In: Becker STS, Obermayer K (eds) Advances in neural information processing systems 15. MIT Press, Cambridge, MA, pp 157–164Google Scholar
  54. Yu AJ, Dayan P (2005a) Inference, attention, and decision in a Bayesian neural architecture. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems 17. MIT Press, Cambridge, MAGoogle Scholar
  55. Yu AJ, Dayan P (2005b) Uncertainty, neuromodulation, and attention. Neuron 46:681–692PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Cognitive ScienceUniversity of CaliforniaLa JollaUSA