Dual-System Learning Models and Drugs of Abuse

Abstract

Dual-system theories in psychology and neuroscience propose that a deliberative or goal-directed decision system is accompanied by a more automatic or habitual path to action. In computational terms, the latter is prominently associated with model-free reinforcement learning algorithms such as temporal-difference learning, and the former with model-based approaches. Due in part to the close association between drugs of abuse and dopamine, and also between dopamine, temporal-difference learning, and habitual behavior, addictive drugs are often thought to specifically target the habitual system.

However, although many drug-taking behaviors are well explained under such a theory, evidence suggests that drug-seeking behaviors must leverage a goal-directed controller as well. Indeed, one exhaustive theoretical account proposed that drugs may have numerous, distinct impacts on both systems as well as on other processes.

Here, we seek a more parsimonious account of these phenomena by asking whether the apparent profligacy of drugs’ effects might be explained by a single mechanism of action. In particular, we propose that the pattern of effects observed under drug abuse may reveal interactions between the two controllers, which have typically been modeled as separate and parallel. We sketch several different candidate characterizations and architectures by which model-free effects may impinge on a model-based system, including sharing of cached values through truncated tree search and bias of transition selection for prioritized value sweeping.

References

  1. Ainslie G (2001) Breakdown of will. Cambridge University Press, Cambridge Google Scholar
  2. Arkadir D, Morris G, Vaadia E, Bergman H (2004) Independent coding of movement direction and reward prediction by single pallidal neurons. J Neurosci 24(45):10047–10056 PubMedCrossRefGoogle Scholar
  3. Balleine BW, Daw ND, O’Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Glimcher PW, Camerer CF, Fehr E, Poldrack RA (eds) Neuroeconomics: decision making and the brain. Academic Press, London, pp 367–387 Google Scholar
  4. Balleine BW, Delgado MR, Hikosaka O (2007) The role of the dorsal striatum in reward and decision-making. J Neurosci 27(31):8161–8165 PubMedCrossRefGoogle Scholar
  5. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37(4–5):407–419 PubMedCrossRefGoogle Scholar
  6. Bechara A (2005) Decision making, impulse control and loss of willpower to resist drugs: a neurocognitive perspective. Nat Neurosci 8(11):1458–1463 PubMedCrossRefGoogle Scholar
  7. Berns GS, McClure SM, Pagnoni G, Montague PR (2001) Predictability modulates human brain response to reward. J Neurosci 21(8):2793–2798 PubMedGoogle Scholar
  8. Blodgett HC, McCutchan K (1947) Place versus response learning in the simple T-maze. J Exp Psychol 37(5):412–422 PubMedCrossRefGoogle Scholar
  9. Bonson KR, Grant SJ, Contoreggi CS, Links JM, Metcalfe J, Weyl HL et al. (2002) Neural systems and cue-induced cocaine craving. Neuropsychopharmacology 26(3):376–386 PubMedCrossRefGoogle Scholar
  10. Bromberg-Martin ES, Matsumoto M, Hong S, Hikosaka O (2010) A pallidus-habenula-dopamine pathway signals inferred stimulus values. J Neurophysiol 104(2):1068–1076 PubMedCrossRefGoogle Scholar
  11. Buckner RL (2010) The role of the hippocampus in prediction and imagination. Annu Rev Psychol 61:27–48, C1-8 PubMedCrossRefGoogle Scholar
  12. Buckner RL, Carroll DC (2007) Self-projection and the brain. Trends Cogn Sci 11(2):49–57 PubMedCrossRefGoogle Scholar
  13. Carr MF, Jadhav SP, Frank LM (2011) Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nat Neurosci 14(2):147–153 PubMedCrossRefGoogle Scholar
  14. Chib VS, Rangel A, Shimojo S, O’Doherty JP (2009) Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. J Neurosci 29(39):12315–12320 PubMedCrossRefGoogle Scholar
  15. Davidson TJ, Kloosterman F, Wilson MA (2009) Hippocampal replay of extended experience. Neuron 63(4):497–507 PubMedCrossRefGoogle Scholar
  16. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16(2):199–204 PubMedCrossRefGoogle Scholar
  17. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12):1704–1711 PubMedCrossRefGoogle Scholar
  18. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan R (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69(6):1204–1215 PubMedCrossRefGoogle Scholar
  19. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA (2000) Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84(6):3072–3077 PubMedGoogle Scholar
  20. Derdikman D, Moser M-B (2010) A dual role for hippocampal replay. Neuron 65(5):582–584 PubMedCrossRefGoogle Scholar
  21. Di Chiara G (1999) Drug addiction as dopamine-dependent associative learning disorder. Eur J Pharmacol 375(1–3):13–30 PubMedCrossRefGoogle Scholar
  22. Di Ciano P (2008) Facilitated acquisition but not persistence of responding for a cocaine-paired conditioned reinforcer following sensitization with cocaine. Neuropsychopharmacology 33(6):1426–1431 PubMedCrossRefGoogle Scholar
  23. Dickinson A (1985) Actions and habits: The development of behavioural autonomy. Philos Trans R Soc Lond B, Biol Sci 308:67–78 CrossRefGoogle Scholar
  24. Dickinson A, Balleine B (2002) The role of learning in the operation of motivational systems. In: Stevens’ handbook of experimental psychology. Wiley, New York Google Scholar
  25. Doya K (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw 12(7–8):961–974 PubMedCrossRefGoogle Scholar
  26. Dragoi G, Tonegawa S (2011) Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469(7330):397–401 PubMedCrossRefGoogle Scholar
  27. Everitt BJ, Robbins TW (2005) Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci 8(11):1481–1489 PubMedCrossRefGoogle Scholar
  28. Everitt BJ, Dickinson A, Robbins TW (2001) The neuropsychological basis of addictive behaviour. Brains Res Rev 36(2–3):129–138 CrossRefGoogle Scholar
  29. Faure A, Haberland U, Condé F, Massioui NE (2005) Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J Neurosci 25(11):2771–2780 PubMedCrossRefGoogle Scholar
  30. Foster DJ, Wilson MA (2006) Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440(7084):680–683 PubMedCrossRefGoogle Scholar
  31. Garavan H, Pankiewicz J, Bloom A, Cho JK, Sperry L, Ross TJ et al. (2000) Cue-induced cocaine craving: neuroanatomical specificity for drug users and drug stimuli. Am J Psychiatry 157(11):1789–1798 PubMedCrossRefGoogle Scholar
  32. Gläscher J, Daw ND, Dayan P, O’Doherty JP (2010) States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585–595 PubMedCrossRefGoogle Scholar
  33. Hampton AN, Bossaerts P, O’Doherty JP (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26(32):8360–8367 PubMedCrossRefGoogle Scholar
  34. Hampton AN, Bossaerts P, O’Doherty JP (2008) Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc Natl Acad Sci 105(18):6741–6746 PubMedCrossRefGoogle Scholar
  35. Hare TA, O’Doherty JP, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28(22):5623–5630 PubMedCrossRefGoogle Scholar
  36. Hasselmo ME (2008) Temporally structured replay of neural activity in a model of entorhinal cortex, hippocampus and postsubiculum. Eur J Neurosci 28(7):1301–1315 PubMedCrossRefGoogle Scholar
  37. Houk JC, Adams JL, Barto AG (1994) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk JC, Davis JL, Beiser DG (eds) Models of information processing in the basal ganglia. MIT Press, Cambridge, pp 249–270 Google Scholar
  38. Johnson A, Redish AD (2005) Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw 18(9):1163–1171 PubMedCrossRefGoogle Scholar
  39. Kable JW, Glimcher PW (2007) The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10(12):1625–1633 PubMedCrossRefGoogle Scholar
  40. Kahneman D, Frederick S (2002) Representativeness revisited: Attribute substitution in intuitive judgment. In: Gilovich T, Griffin DW, Kahneman D (eds) Heuristics and biases: the psychology of intuitive judgement. Cambridge University Press, New York, pp 49–81 Google Scholar
  41. Kalivas PW, Volkow ND (2005) The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry 162(8):1403–1413 PubMedCrossRefGoogle Scholar
  42. Killcross S, Coutureau E (2003) Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb Cortex 13(4):400–408 PubMedCrossRefGoogle Scholar
  43. Kim H, Sul JH, Huh N, Lee D, Jung MW (2009) Role of striatum in updating values of chosen actions. J Neurosci 29(47):14701–14712 PubMedCrossRefGoogle Scholar
  44. Koene RA, Hasselmo ME (2008) Reversed and forward buffering of behavioral spike sequences enables retrospective and prospective retrieval in hippocampal regions CA3 and CA1. Neural Netw 21(2–3):276–288 PubMedCrossRefGoogle Scholar
  45. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz CMA (2009) Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol 7(8):e1000173 PubMedCrossRefGoogle Scholar
  46. Loewenstein G, O’Donoghue T (2004) Animal spirits: Affective and deliberative processes in economic behavior (Working Papers Nos. 04–14). Cornell University, Center for Analytic Economics Google Scholar
  47. Lovibond PF (1983) Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. J Exp Psychol, Anim Behav Processes 9(3):225–247 CrossRefGoogle Scholar
  48. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38(2):339–346 PubMedCrossRefGoogle Scholar
  49. van der Meer MAA, Johnson A, Schmitzer-Torbert NC, Redish AD (2010) Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 67(1):25–32 PubMedCrossRefGoogle Scholar
  50. Meil W, See R (1996) Conditioned cued recovery of responding following prolonged withdrawal from self-administered cocaine in rats: an animal model of relapse. Behav Pharmacol 7(8):754–763 PubMedGoogle Scholar
  51. Moore AW, Atkeson CG (1993) Prioritized sweeping: Reinforcement learning with less data and less time. Mach Learn 13:103–130. (10.1007/BF00993104) Google Scholar
  52. Nordquist RE, Voorn P, de Mooij-van Malsen JG, Joosten RNJMA, Pennartz CMA, Vanderschuren LJMJ (2007) Augmented reinforcer value and accelerated habit formation after repeated amphetamine treatment. Eur Neuropsychopharmacol 17(8):532–540 PubMedCrossRefGoogle Scholar
  53. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38(2):329–337 PubMedCrossRefGoogle Scholar
  54. Olmstead MC, Lafond MV, Everitt BJ, Dickinson A (2001) Cocaine seeking by rats is a goal-directed action. Behav Neurosci 115(2):394–402 PubMedCrossRefGoogle Scholar
  55. Pan X, Sawa K, Sakagami M (2007) Model-based reward prediction in the primate prefrontal cortex. Neurosci Res 58(Suppl 1):229 CrossRefGoogle Scholar
  56. Panlilio LV, Thorndike EB, Schindler CW (2007) Blocking of conditioning to a cocaine-paired stimulus: testing the hypothesis that cocaine perpetually produces a signal of larger-than-expected reward. Pharmacol Biochem Behav 86(4):774–777 PubMedCrossRefGoogle Scholar
  57. Plassmann H, O’Doherty J, Rangel A (2007) Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J Neurosci 27(37):9984–9988 PubMedCrossRefGoogle Scholar
  58. Poldrack RA, Clark J, Paré-Blagoev EJ, Shohamy D, Creso Moyano J, Myers C et al. (2001) Interactive memory systems in the human brain. Nature 414(6863):546–550 PubMedCrossRefGoogle Scholar
  59. Rangel A, Camerer C, Montague P (2008) A framework for studying the neurobiology of value-based decision making. Nat Rev, Neurosci 9(7):545–556 CrossRefGoogle Scholar
  60. Redish AD (2004) Addiction as a computational process gone awry. Science 306(5703):1944–1947 PubMedCrossRefGoogle Scholar
  61. Redish AD, Johnson A (2007) A computational model of craving and obsession. Ann NY Acad Sci 1104(1):324–339 PubMedCrossRefGoogle Scholar
  62. Redish AD, Jensen S, Johnson A (2008) Addiction as vulnerabilities in the decision process. Behav Brain Sci 31(04):461–487 Google Scholar
  63. Rescorla RA (1994) Control of instrumental performance by Pavlovian and instrumental stimuli. J Exp Psychol, Anim Behav Processes 20(1):44–50 CrossRefGoogle Scholar
  64. Robinson TE, Berridge KC (2008) The incentive sensitization theory of addiction: some current issues. Philos Trans R Soc Lond B, Biol Sci 363(1507):3137–3146 CrossRefGoogle Scholar
  65. Root DH, Fabbricatore AT, Barker DJ, Ma S, Pawlak AP, West MO (2009) Evidence for habitual and goal-directed behavior following devaluation of cocaine: a multifaceted interpretation of relapse. PLoS ONE 4(9):e7170 PubMedCrossRefGoogle Scholar
  66. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310(5752):1337–1340 PubMedCrossRefGoogle Scholar
  67. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80(1):1–27 PubMedGoogle Scholar
  68. Schultz W (2011) Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs. Neuron 69(4):603–617 PubMedCrossRefGoogle Scholar
  69. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593–1599 PubMedCrossRefGoogle Scholar
  70. See RE (2005) Neural substrates of cocaine-cue associations that trigger relapse. Eur J Pharmacol 526(1–3):140–146 PubMedCrossRefGoogle Scholar
  71. Simon DA, Daw ND (2011) Neural correlates of forward planning in a spatial decision task in humans. J Neurosci 31(14):5526–5539 PubMedCrossRefGoogle Scholar
  72. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3(1):9–44 Google Scholar
  73. Sutton RS (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings of the seventh International Conference on Machine Learning. Morgan Kaufmann, San Mateo, pp 216–224 Google Scholar
  74. Sutton RS, Barto AG (1998) Reinforcement learning. MIT Press, Cambridge Google Scholar
  75. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7(8):887–893 PubMedCrossRefGoogle Scholar
  76. Tanaka SC, Samejima K, Okada G, Ueda K, Okamoto Y, Yamawaki S et al. (2006) Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw 19(8):1233–1241 PubMedCrossRefGoogle Scholar
  77. Thorndike EL (1898) Animal intelligence: An experimental study of the associative processes in animals. Psychol Rev Monogr Suppl 2(4):1–8 Google Scholar
  78. Tiffany ST (1990) A cognitive model of drug urges and drug-use behavior: Role of automatic and nonautomatic processes. Psychol Rev 97(2):147–168 PubMedCrossRefGoogle Scholar
  79. Tindell AJ, Smith KS, Berridge KC, Aldridge JW (2009) Dynamic computation of incentive salience: “wanting” what was never “liked”. J Neurosci 29(39):12220–12228 PubMedCrossRefGoogle Scholar
  80. Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55:189–208 PubMedCrossRefGoogle Scholar
  81. Tom SM, Fox CR, Trepel C, Poldrack RA (2007) The neural basis of loss aversion in decision-making under risk. Science 315(5811):515–518 PubMedCrossRefGoogle Scholar
  82. Vanderschuren LJMJ, Everitt BJ (2004) Drug seeking becomes compulsive after prolonged cocaine self-administration. Science 305(5686):1017–1019 PubMedCrossRefGoogle Scholar
  83. Verplanken B, Aarts H, van Knippenberg AD, Moonen A (1998) Habit versus planned behaviour: a field experiment. Br J Soc Psychol 37(1):111–128 PubMedCrossRefGoogle Scholar
  84. Volkow ND, Wang G-J, Telang F, Fowler JS, Logan J, Childress A-R et al. (2008) Dopamine increases in striatum do not elicit craving in cocaine abusers unless they are coupled with cocaine cues. NeuroImage 39(3):1266–1273 PubMedCrossRefGoogle Scholar
  85. Wood W, Neal DT (2007) A new look at habits and the habit-goal interface. Psychol Rev 114(4):843–863 PubMedCrossRefGoogle Scholar
  86. Wunderlich K, Rangel A, O’Doherty JP (2009) Neural computations underlying action-based decision making in the human brain. Proc Natl Acad Sci 106(40):17199–17204 PubMedCrossRefGoogle Scholar
  87. Yin HH, Knowlton BJ, Balleine BW (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19(1):181–189 PubMedCrossRefGoogle Scholar
  88. Yin HH, Ostlund SB, Knowlton BJ, Balleine BW (2005) The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci 22(2):513–523 PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of PsychologyNew York UniversityNew YorkUSA
  2. 2.Center for Neural Science and Department of PsychologyNew York UniversityNew YorkUSA

Personalised recommendations