Tonic dopamine: opportunity costs and the control of response vigor

Abstract

Rationale

Dopamine neurotransmission has long been known to exert a powerful influence over the vigor, strength, or rate of responding. However, there exists no clear understanding of the computational foundation for this effect; predominant accounts of dopamine’s computational function focus on a role for phasic dopamine in controlling the discrete selection between different actions and have nothing to say about response vigor or indeed the free-operant tasks in which it is typically measured.

Objectives

We seek to accommodate free-operant behavioral tasks within the realm of models of optimal control and thereby capture how dopaminergic and motivational manipulations affect response vigor.

Methods

We construct an average reward reinforcement learning model in which subjects choose both which action to perform and also the latency with which to perform it. Optimal control balances the costs of acting quickly against the benefits of getting reward earlier and thereby chooses a best response latency.

Results

In this framework, the long-run average rate of reward plays a key role as an opportunity cost and mediates motivational influences on rates and vigor of responding. We review evidence suggesting that the average reward rate is reported by tonic levels of dopamine putatively in the nucleus accumbens.

Conclusions

Our extension of reinforcement learning models to free-operant tasks unites psychologically and computationally inspired ideas about the role of tonic dopamine in striatum, explaining from a normative point of view why higher levels of dopamine might be associated with more vigorous responding.

Appendix

Here we describe in more mathematical detail the proposed RL model of free-operant response rates (Niv et al. 2005a) from which the results in this paper were derived.

Formally, the action selection problem faced by the rat can be characterized by a series of states, S∈S, in each of which the rat must choose an action and a latency (a,τ) which will entail a unit cost, C _u , and a vigor cost, C _v /τ, and result in a possible transition to a new state, $S\prime$, and a possible immediate reward with utility, U _r . The unit cost constant C _u and the vigor cost constant C _v can take different values depending on the identity of the currently chosen action a∈{LP, NP, “Other”} and on that of the previously performed action. The transitions between states and the probability of reward for each action are governed by the schedule of reinforcement. For instance, in a random-ratio 5 (RR5) schedule, every LP action has p=0.2 probability of inducing a transition from the state in which no food is available in the magazine to that in which food is available. An NP action in the “no-reward-available” state is never rewarded and, conversely, is rewarded with certainty (p _r =1) in the “food-available-in-magazine” state. As a simplification, for each reinforcement schedule, we define states that incorporate all the available information relevant to decision making, such as the identity of the previously chosen action, whether or not food is available in the magazine, the time that has elapsed since the last lever press (in random-interval schedules only), and the number of lever presses since the last reward (in fixed ratio schedules only). The animal’s behavior in the experiment is thus fully described by the successive actions and latencies chosen at the different states the animal encountered {(a _i, τ_i, S _i), i=1,2,3, ...}. The average reward rate $\overline{R}$ is simply the sum of all the rewards obtained minus all the costs incurred, all divided by the total amount of time.

Using this formulation, we can define the differential value of a state, denoted V(S), as the expected sum of future rewards minus costs encountered from this state and onward compared with the expected average reward rate. Defining the value as an expectation over a sum means that the value can be written recursively as the expected reward minus cost due to the current action, compared with the immediately forfeited average reward, plus the value of the next state (averaged over the possible next states). To find the optimal differential values of the different states, that is, the values $V^{*} {\left( S \right)}$ (and average value $\overline{R} ^{*}$) given the optimal action selection strategy, we can simultaneously solve the set of equations defining these values:

$$V^{*} {\left( S \right)} = {\mathop {\max }\limits_{a,\tau } }{\left\{ {p_{{\text{r}}} {\left( {a,\tau ,S} \right)}U_{{\text{r}}} - C_{{\text{u}}} {\left( {a,a_{{{\text{prev}}}} } \right)} - \frac{{C_{{\text{v}}} {\left( {a,a_{{{\text{prev}}}} } \right)}}}{\tau } - \overline{R} ^{{\text{*}}} \cdot \tau + {\sum\limits_{S\prime \in S} {p{\left( {\left. {S\prime } \right|a,\tau ,S} \right)}V^{{\text{*}}} {\left( {S\prime } \right)}} }} \right\}},$$

(1)

in which there is one equation for every state S∈ S, and p($S\prime$ |a,τ,S) is the schedule-defined probability to transition to state $S\prime$ given (a,τ) was performed at state S.

The theory of dynamic programming (Bertsekas and Tsitsiklis 1996) ensures that these equations have one solution for the optimal attainable average reward $\overline{R} ^{*}$, and the optimal differential state values $V^{*} {\left( S \right)}$ (which are defined up to an additive constant). This solution can be found using iterative dynamic programming methods such as “value iteration” (Bertsekas and Tsitsiklis 1996) or approximated through online sampling of the task dynamics and temporal-difference learning (Schwartz 1993; Mahadevan 1996; Sutton and Barto 1998). Here we used the former and report results using the true optimal differential values. We compare these model results to the steady-state behavior of well-trained animals as the optimal values correspond to values learned online throughout an extensive training period.

Given the optimal state values, the optimal differential value of an (a,τ) pair taken at state S, denoted $Q^{*} {\left( {a,\tau ,S} \right)}$, is simply:

$$Q^{*} {\left( {a,\tau ,S} \right)} = p_{r} {\left( {a,\tau ,S} \right)} \cdot U_{r} - C_{u} {\left( {a,a_{{{\text{prev}}}} } \right)} - \frac{{C_{{\text{v}}} {\left( {a,a_{{{\text{prev}}}} } \right)}}}{\tau } - \overline{R} ^{*} \cdot \tau + {\sum\limits_{S\prime \in S} {p{\left( {\left. {S\prime } \right|a,\tau ,S} \right)}V^{*} {\left( {S\prime } \right)}} }$$

(2)

The animal can select actions optimally (that is, such as to obtain the maximal possible average reward rate $\overline{R} ^{*}$) by comparing the differential values of the different (a,τ) pairs at the current state and choosing the action and latency that have the highest value. Alternatively, to allow more flexible behavior and occasional exploratory actions (Daw et al. 2006), response selection can be based on the so-called “soft-max” rule (or Boltzmann distribution) in which the probability of choosing an (a,τ) pair is proportional to its differential value. In this case, which is the one we used here, actions that are “almost optimal” are chosen almost as frequently as actions that are strictly optimal. Specifically, the probability of choosing (a,τ) in state S is:

$$p{\left( {a,\tau ,S} \right)} = \frac{{e^{{\beta Q^{*} {\left( {a,\tau ,S} \right)}}} }}{{{\sum\limits_{a\prime ,\tau \prime } {e^{{\beta Q^{*} {\left( {a\prime ,\tau \prime ,S} \right)}}} } }}},$$

(3)

where β is the inverse temperature controlling the steepness of the soft-max function (a value of zero corresponds to uniform selection of actions, whereas higher values correspond to a more maximizing strategy).

To simulate the (immediate) effects of depletion of tonic dopamine (Fig. 3b), Q values were recomputed from the optimal V values (using Eq. 2), but taking into account a lower average reward rate (specifically, $\overline{R} _{{{\text{depleted}}}} = 0.4\overline{R} ^{*}$). Actions were then chosen as usual, using the soft-max function of these new Q values, to generate behavior.

Finally, note that Eq. 2 is a function relating actions and latencies to values. Accordingly, one way to find the optimal latency is to differentiate Eq. 2 with respect to τ and find its maximum. For ratio schedules (in which the identity and value of the subsequent state $S\prime$ is not dependent on τ), this gives:

$$\tau ^{*} = {\sqrt {\frac{{C_{{\text{V}}} }}{{\overline{R} ^{*} }}} },$$

(4)

showing that the optimal latency $\tau ^{*}$ depends solely on the vigor cost constant and the average reward rate. This is true regardless of the action a chosen, which is why a change in the average reward has a similar effect on the latencies of all actions. In interval schedules, the situation is slightly more complex because the identity of the subsequent state is dependent on the latency, and this must be taken into account when taking the derivative. However, in this case as well, the optimal latency is inversely related to the average reward rate.