Analyzing dynamic decision-making models using Chapman-Kolmogorov equations

Abstract

Decision-making in dynamic environments typically requires adaptive evidence accumulation that weights new evidence more heavily than old observations. Recent experimental studies of dynamic decision tasks require subjects to make decisions for which the correct choice switches stochastically throughout a single trial. In such cases, an ideal observer’s belief is described by an evolution equation that is doubly stochastic, reflecting stochasticity in the both observations and environmental changes. In these contexts, we show that the probability density of the belief can be represented using differential Chapman-Kolmogorov equations, allowing efficient computation of ensemble statistics. This allows us to reliably compare normative models to near-normative approximations using, as model performance metrics, decision response accuracy and Kullback-Leibler divergence of the belief distributions. Such belief distributions could be obtained empirically from subjects by asking them to report their decision confidence. We also study how response accuracy is affected by additional internal noise, showing optimality requires longer integration timescales as more noise is added. Lastly, we demonstrate that our method can be applied to tasks in which evidence arrives in a discrete, pulsatile fashion, rather than continuously.

Appendices

Appendix A: Normative evidence-accumulation in dynamic environments

Here we derive the continuum limit of the Bayesian update equation for continuous evidence accumulation in a changing environment. Starting with the discrete time model, we define L_n,± = P(s(t_n) = s_±|ξ_1:n) as the probability of being in state s_± at time t_n assuming a sequence of observations ξ_1:n. The state s(t) changes between evenly spaced time points t_1:n (with Δt := t_n − t_n− 1) at a hazard rate h_Δt := h ⋅Δt := P(s(t_n) = s_∓|s(t_n− 1) = s_±). The likelihood function f_Δt,±(ξ) = P(ξ|s = s_±; Δt) is the conditional probability of observing sample ξ given state s_±, parameterized by Δt.

We begin by assuming an ideal observer who knows the environmental hazard rate h. Using Bayes’ rule and the law of total probability, we can relate L_n,± to the probability at the previous time step according to the weighted sum (Veliz-Cuba et al. 2016)

$$ L_{n, \pm} = f_{\Delta t, \pm}(\xi_{n}) \frac{\mathrm{P}(\xi_{1:n})}{\mathrm{P}(\xi_{1:n})} \left[ (1- h_{\Delta t}) L_{n-1, \pm} + h_{\Delta t} L_{n-1, \mp} \right], $$

(25)

where L_0,± = P(s(t₀) = s_±). Defining $y_{n} = \log \frac {L_{n, +}}{L_{n,-}}$, we can compute

$$ {\Delta} y_{n} : = y_{n} - y_{n-1} = \log \frac{f_{\Delta t, +}(\xi_{n})}{f_{\Delta t, -}(\xi_{n})} + \log \frac{1 - h_{\Delta t} + h_{\Delta t} \mathrm{e}^{-y_{n-1}}}{1 - h_{\Delta t} + h_{\Delta t} \mathrm{e}^{y_{n-1}}}. $$

(26)

In search of the continuum limit of this equation, we assume 0 < Δt ≪ 1, 0 < |Δy_n|≪ 1, and use the approximation $\log (1 + z) \approx z$ to obtain

$$ {\Delta} y_{n} \approx \log \frac{f_{\Delta t, +}(\xi_{n})}{f_{\Delta t, -} (\xi_{n})} - 2 h \cdot {\Delta} t \sinh(y_{n}). $$

(27)

Replacing the index n with the time t and applying the functional central limit theorem as in Billingsley (2008) and Bogacz et al. (2006), we can write Eq. (27) as

$$ {\Delta} y_{t} \approx {\Delta} t g_{\Delta t}(t) + \sqrt{\Delta t} \rho_{\Delta t}(t) \eta - 2 h \cdot {\Delta} t \sinh(y_{t}), $$

(28)

where η is a random variable with a standard normal distribution and

$$ \begin{array}{@{}rcl@{}} g_{\Delta t}(t) &&: = \frac{1}{\Delta t} \mathrm{E}_{\xi} \left[ \log \frac{f_{\Delta t, +}(\xi)}{f_{\Delta t, -} (\xi)} \Big| s(t) \right], \rho_{\Delta t}^{2} (t) \\&&: = \frac{1}{\Delta t} \text{Var}_{\xi} \left[ \log \frac{f_{\Delta t, +}(\xi)}{f_{\Delta t, -} (\xi)} \Big| s(t) \right]. \end{array} $$

(29)

The drift g_Δt and variance $\rho _{\Delta t}^{2}$ diverge unless f_Δt,±(ξ) are scaled appropriately in the Δt → 0 limit. A reasonable assumption that can be made to compute g_Δt and $\rho _{\Delta t}^{2}$ explicitly is to take observations ξ to follow normal distributions with mean and variance scaled by Δt (Bogacz et al. 2006; Veliz-Cuba et al. 2016)

$$ f_{\Delta t, \pm} (\xi) = \frac{1}{\sqrt{2 \pi {\Delta} t \sigma^{2}}} \mathrm{e}^{-(\xi \mp {\Delta} \mu)^{2}/(2 {\Delta} t \sigma^{2})}, $$

so we can compute the limits of Eq. (29) as

$$ \begin{array}{@{}rcl@{}} g(t) &=& \lim_{\Delta t \to 0} g_{\Delta t}(t) \in \pm \frac{2 \mu^{2}}{\sigma^{2}} = \pm g, \end{array} $$

(30a)

$$ \begin{array}{@{}rcl@{}} \rho^{2}(t) &=& \lim_{\Delta t \to 0} \rho_{\Delta t}^{2}(t) = \frac{4 \mu^{2}}{\sigma^{2}} = \rho^{2}, \end{array} $$

(30b)

where g(t) ∈{+g,−g} is a telegraph process with probability masses P(±g, t) evolving as $\dot {\mathrm {P}}(\pm g,t) = h \left [ \mathrm {P} (\mp g, t) - \mathrm {P} (\pm g,t) \right ]$ and ρ²(t) = ρ² remains constant. Therefore, the continuum limit (Δt → 0) of Eq. (28) is

$$ \mathrm{d} y(t) = g(t) \mathrm{d} t + \rho \mathrm{d} W_{t} - 2 h \sinh (y(t)) \mathrm{d} t, $$

(31)

where dW is a standard Wiener process. Equation (31) provide the normative model of evidence accumulation for an observer who knows the hazard rate h and wishes to infer the sign of g(t) at time t with maximal accuracy (Glaze et al. 2015; Veliz-Cuba et al. 2016).

However, we are also interested in near-normative models in which the observer assumes an incorrect hazard rate $\tilde {h} \neq h$. In such a case, the analysis proceeds as before, with the probabilistic inference process simply involving $\tilde {h}$ now rather than h, and the result is

$$ \mathrm{d} y(t) = g(t) \mathrm{d} t + \rho \mathrm{d} W_{t} - 2 \tilde{h} \sinh (y(t)) \mathrm{d} t. $$

(32)

Lastly, note that if indeed the original observations ξ are drawn from normal distributions, Eq. (30a), (30b) states g(t) ∈±g where g = 2μ²/σ² and ρ² = 2g. Rescaling time ht↦t, we can then express Eq. (32) in terms of the following rescaled equation

$$ \mathrm{d} y(t) = x(t) m \mathrm{d} t + \sqrt{2 m} \mathrm{d} W_{t} - 2 \frac{\tilde{h}}{h} \sinh(y) \mathrm{d} t, $$

where m = 2μ²/(hσ²) and x(t) ∈± 1 is a telegraph process with hazard rate 1, as shown in Eq. (2) of the main text.

Appendix B: Derivation of the Chapman-Kolmogorov equations

Here we outline the derivation of the CK equations given by Eqs. (4a), (4b) and (6) in the main text. Our goal is to write a PDE that describes the evolution of an ensemble of belief trajectories described by the SDE in Eq. (2) across all realizations of both sources of stochasticity – observation noise and state switching. We cannot simply write a Fokker-Planck equation to describe the desired probability distribution, as the process is doubly stochastic; the diffusion is described by a scaled Wiener process and the sign of the drift is controlled by a two-state Markov process. Therefore, following Droste and Lindner (2017), we condition our process on the state of x(t), and seek to obtain evolution equations for the conditional distributions p_±(y, t) := p(y, t|x(t) = ±).

We start by conditioning on x(t) = + 1 to find an equation for p₊(y, t). In the absence of state transitions in the background Markov process, the evolution equation for p₊(y, t) is now given by the Fokker-Planck equation

$$ \frac{\partial p_{+}}{\partial t}=-m\frac{\partial p_{+}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{+}\right]+m\frac{\partial^{2}p_{+}}{\partial y^{2}}. $$

On the other hand, in the absence of drift and diffusion, the evolution of p₊(y, t) is governed exclusively by the two-state Markov process; in this case, we can write the evolution equation in the form of a discrete master equation:

$$ \frac{dp_{+}}{dt}=p_{-}-p_{+}. $$

Note here that because of the rescaling of time in Eq. (2), the transition rates of the Markov process equal one. Given these two components, we can follow Gardiner (2004) to write the CK equation that describes the evolution of p₊(y, t) when drift, diffusion, and switching are all present as

$$ \frac{\partial p_{+}}{\partial t}=-m\frac{\partial p_{+}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{+}\right]+m\frac{\partial^{2}p_{+}}{\partial y^{2}}+\left[p_{-}-p_{+}\right], $$

which is Eqs. (4a); (4b) is obtained similarly by conditioning on x(t) = − 1.

To determine response accuracy, we are interested in the probability the observer responds with the correct state of the Markov process. Therefore, at an interrogation time t = T where x(T) = + 1, the observer is correct if y > 0. This probability is given by the integral ${\int \limits }_{0}^{\infty } p_{+}(y,T) dy$. Similarly, if x(T) = − 1, the observer is correct if y < 0, which happens with probability ${\int \limits }_{-\infty }^{0} p_{-}(y,T) dy$. Therefore, the total response accuracy at time T is given by summing these two integrals:

$$ \text{Acc}(T)={\int}_{0}^{\infty} p_{+}(y,T) dy+{\int}_{-\infty}^{0} p_{-}(y,T) dy, $$

which is Eq. (5).

Finally, in order to more efficiently simulate the evolution of p₊(y, t) and p₋(y, t), we notice that because the nonlinear leak is an odd function, a change of variables − y↦y in Eq. (4b) allows us to combine Eqs. (4a) and (4b) into a single PDE. Defining p_s(y, t) := p₊(y, t) + p₋(−y, t), we obtain the evolution equation

$$ \frac{\partial p_{s}}{\partial t}=-m\frac{\partial p_{s}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{s}\right]+m\frac{\partial^{2}}{\partial y^{2}}+\left[p_{s}(-y,t)-p_{s}(y,t)\right], $$

which is Eq. (6).

Appendix C: Finite difference methods for Chapman-Kolmogorov equations

Fig. 9

Comparison of CK equations with Monte Carlo sampling. a Calculation of accuracy of mistuned nonlinear leak model for m = 5. Monte Carlo simulations run with varied number of samples superimposed (legend). b Runtime (red) and L² error (blue) of Monte Carlo simulations as a function of sample size. Runtime of CK equations (black dashed) superimposed for comparison. L² error of Monte Carlo simulations calculated against results from CK equations. c Runtime (red) and $L^{\infty }$ error (blue) of finite difference simulations of the nonlinear model Eq. (6) under refinement of belief discretization, compared with the Δy = 10^− 4 case. Because our method is first-order accurate in time and second-order accurate in belief, the method is first-order accurate when Δy is decreased and Δt is held constant

We used a finite difference method to simulate the differential CK equations. The method is exemplified here for the normative CK equation from Eq. (6), but a similar approach was used for the linear, cubic, and pulsatile equations. For stability purposes, our method uses centered differences in y and backward-Euler in t. This gives the following finite difference approximations of the functions and their derivatives in Eq. (6):

$$ \begin{array}{@{}rcl@{}} \frac{\partial p_{s}(y,t)}{\partial t} &\approx& \frac{p_{s}(y,t+{\Delta} t)-p_{s}(y,t)}{\Delta t}, p_{s}(y,t) \approx p_{s}(y,t+{\Delta} t),\\ \frac{\partial p_{s}(y,t)}{\partial y} &\approx& \frac{p_{s}(y+{\Delta} y,t+{\Delta} t)-p_{s}(y-{\Delta} y,t+{\Delta} t)}{2{\Delta} y}, \\ \frac{\partial^{2}p_{s}(y,t)}{\partial y^{2}} & \approx& \frac{p_{s}(y\!-{\Delta} y,t\!+{\Delta} t)-2p_{s}(y,t+{\Delta} t)+p_{s}(y+{\Delta} y,t+{\Delta} t)}{({\Delta} y)^{2}}, \end{array} $$

where Δt and Δy are timestep and spacestep of the simulation, respectively. Substituting into Eq. (6) and solving for p_s(y, t) at each point on a mesh y for y gives the system of equations:

$$ \mathbf{A}p_{s}(\mathbf{y},t+{\Delta} t)=p(\mathbf{y},t), $$

(33)

where A is tridiagonal with elements along the primary off-diagonal. This system can be inverted at each timestep and used to calculate the updates p_s(y, t + Δt).

For the boundary conditions, we imposed no-flux conditions at the mesh boundaries ± b. For a standard drift-diffusion equation with drift A(y) and diffusion constant B(y), this condition takes the form

$$ J(\pm b,t)=A(\pm b)p(\pm b,t)-\frac{1}{2}\frac{\partial}{\partial y}\left[B(\pm b)^{2}p(\pm b,t)\right]=0. $$

(34)

Using the finite difference approximations

$$ \begin{array}{@{}rcl@{}} \frac{\partial p_{s}(y,t)}{\partial y} & \approx& \frac{3p_{s}(y,t + {\Delta} t) - 4p_{s}(y+{\Delta} y,t+{\Delta} t)+p_{s}(y+2{\Delta} y,t+\!{\Delta} t)}{2{\Delta} y}, \\ \frac{\partial p_{s}(y,t)}{\partial y} & \!\approx& \frac{\!-p_{s}(y-2{\Delta} y,t+{\Delta} t)+4p_{s}(y - {\Delta} y,t + {\Delta} t) - 3p_{s}(y,t+\!{\Delta} t)}{2{\Delta} y}, \end{array} $$

we can plug in ± b to the appropriate replacement and use Eq. (34) to find the appropriate boundary terms for the system in Eq. (33).

Figure 9 shows the results of Monte Carlo simulations compared against those from the CK equations; Monte Carlo simulations are less smooth (Fig. 9a), making optimality calculations less accurate. Furthermore, obtaining results that are close to those from the CK equations takes much longer to run (Fig. 9b), as we can obtain good accuracy (error ≈ 10^− 2) from the CK equations in less than a second (Fig. 9c).

Appendix D: Distinguishing linear discounting models using confidence

Fig. 10

Distinguishing linear discounting parameter λ using confidence reports. a Average number of trials needed to determine whether a subject uses the maximizing accuracy leak parameter, λ^Acc, or the minimizing KL divergence leak parameter, λ^KL, as a function of evidence strength m. Averages were computed over a thousand simulations for interrogation time T = 1, with each simulation run until confidence in the subject’s model reached 90%

Can we use the distributions obtained from the CK equations to distinguish which model a subject uses? To illustrate an example, we considered distinguishing the two linear discounting models given by Eq. (7) and asked how many trials an experimentalist would need to run in order to tell if a subject was using λ^Acc or λ^KL. We sampled from the distributions obtained from Eq. (8) and performed a likelihood ratio test to find the number of samples needed for the experimentalist to have 90% confidence in the model being used. Assuming a symmetric prior P(λ^Acc) = P(λ^KL) = 0.5, and defining $\mathcal { Y}_{j} = y_{j}(T)$ as the belief report at the end (t = T) of trial j, the likelihood ratio is computed using Bayes’ rule and independence after N trials as

$$ \frac{P(\lambda^{\text{Acc}}|\mathcal{Y}_{1:N})}{P(\lambda^{\text{KL}}|\mathcal{Y}_{1:N})} = \prod\limits_{j=1}^{N} \frac{P(\mathcal{Y}_{j} | \lambda^{\text{Acc}})}{P(\mathcal{Y}_{j} | \lambda^{\text{KL}})}, $$

and we counted the number of trials required to obtain ≥ 90% confidence in either model (so $P(\lambda |\mathcal {Y}_{1:N}) \geq 0.9$ for either λ ∈{λ^Acc, λ^KL}). The results of this simulation are given in Fig. 10, showing that as the strength m of evidence increases, the mean number of trials 〈N(m)〉 required to distinguish models decreases precipitously. This is to be expected based on the fact that the belief distributions become more separated as m increases (Fig. 3e).

Appendix E: Deriving differential CK equation with internal noise

Here we provide intuition for the form of the diffusion coefficient in Eq. (13) for the belief distribution of a normative observer strategy with additional internal noise of strength D. Starting with the SDE in Eq. (12), because $\sqrt {2m}\mathrm {d} W_{t}$ and $\sqrt {2D}\mathrm {d} X_{t}$ are increments of independent Wiener processes, we can define a new Wiener process $A\mathrm {d} Z_{t}=\sqrt {2m}\mathrm {d} W_{t}+\sqrt {2D}\mathrm {d} X_{t}$ that has the same statistics as the original summed Wiener processes (Gardiner 2004). To determine the appropriate effective diffusion constant A, we note that

$$ \text{Var}[A\mathrm{d} Z_{t}]=A^{2}\text{Var}[\mathrm{d} Z_{t}]=A^{2}t $$

and

$$ \text{Var}[\sqrt{2m}\mathrm{d} W_{t}+\sqrt{2D}\mathrm{d} X_{t}]=2m\text{Var}[\mathrm{d} W_{t}]+2D\text{Var}[\mathrm{d} X_{t}]=2mt+2Dt. $$

This requires $A=\sqrt {2(m+D)}$, and means Eq. (12) can be rewritten as

$$ \mathrm{d} y= x(t) \cdot m \mathrm{d} t +\sqrt{2(m+D)}\mathrm{d} Z_{t} -2\frac{\tilde{h}}{h}\sinh(y) \mathrm{d} t, $$

which following Gardiner (2004), has the differential CK equation given by Eq. (13).

Appendix F: Steady state solution of the bounded accumulator model

Steady state solutions of Eq. (16) are derived first by noting that ∂_tp^± = 0 implies

$$ 0 = \mp m \bar{p}_{\pm}^{\prime} (y) \pm m \bar{p}_{\pm}^{\prime\prime}(y) +\left[\bar{p}_{\mp} (y)-\bar{p}_{\pm}(y) \right], $$

(35)

with boundary conditions $\bar {p}_{\pm }(y) (\pm \beta ) = \bar {p}_{\pm }^{\prime }(\pm \beta )$. Equation (35) has solutions $\left (\begin {array}{c} \bar {p}_{+}(y)\\ \bar {p}_{-}(y) \end {array} \right ) = \left (\begin {array}{c} A \\ B \end {array} \right ) \mathrm {e}^{\alpha y}$, with characteristic equation $m^{2} \alpha ^{4}-\left (m^{2}+2m\right ) \alpha ^{2}=0$. The characteristic roots are α = 0,±q, where we define $q = \sqrt {1+\frac {2}{m}}$. For α = 0, we have A = B, whereas for α = ±q, the symmetry $\bar {p}_{+}(y)=\bar {p}_{-}(-y)$ implies B = (mq − (m + 1))A for α = +q and A = (mq − (m + 1))B for α = −q. Lastly, defining the sum $\bar {p}_{s}(y)=\bar {p}_{+}(y)+\bar {p}_{-}(-y)$, we obtain

$$ \bar{p}_{s}(y)=C_{1}+C_{2}\left[e^{qy}+\left( mq-(m+1)\right)e^{-qy}\right]. $$

The no flux boundary conditions $\bar {p}_{s}(\pm \beta )-\frac {\partial \bar {p}_{s}(\pm \beta )}{\partial y}=0$ along with the normalization requirement ${\int \limits }_{- \beta }^{\beta } \bar {p}_{s}(y) \mathrm {d} y = 1$ give explicit expressions for the constants

$$ C_{1}=\frac{1}{2\beta}-C_{2}\left( \frac{m(q-1)\sinh(q\beta)}{q\beta}\right) $$

and

$$ C_{2} = \left\{2\beta\left[(q-1)\left( e^{q\beta} + \frac{m\sinh(q\beta)}{q\beta}\right)-(q + 1)(mq-(m\!+1))e^{-q\beta}\right]\right\}^{-1}. $$

Appendix G: Steady state solution of the clicks-task bounded accumulator model

Considering Eq. (23), we look for stationary solutions of the form $\left (\begin {array}{llll} \bar {p}_{+}(n)\\ \bar {p}_{-}(n) \end {array}\right )=\left (\begin {array}{llll} C_{1}\\ C_{2} \end {array}\right ) \alpha ^{n}$, yielding the characteristic equation

$$ \begin{array}{@{}rcl@{}} &&\left[r_{+}\alpha^{n-1}+r_{-}\alpha^{n+1}-r_{s} \alpha^{n}\right]\left[r_{-}\alpha^{n-1}+r_{+} \alpha^{n+1} - r_{s} \alpha^{n}\right] \\&&\quad- \alpha^{2n}=0, \end{array} $$

(36)

where r_s = r₊ + r₋ + 1. Solving Eq. (36) gives α = 1 with eigenfunction C₁ = C₂ and two roots α = q_± of the quadratic $\alpha ^{2}- \alpha (r_{+}+r_{+}^{2}+r_{-}+r_{-}^{2})/(r_{+} r_{-}) +1=0$. Superimposing the eigenfunctions, redefining constants, and defining $\bar {p}_{s}(n)=\bar {p}_{+}(n)+\bar {p}_{-}(-n)$ gives the general solution

$$ \bar{p}_{s}(n)=C_{1}+C_{2}q_{+}^{n}+C_{3}q_{-}^{n}. $$

The constants C₁, C₂, and C₃ can be determined by normalization ${\sum }_{n=-\beta }^{\beta }\bar {p}_{s}(n)=1$ and the stationary boundary conditions

$$ \begin{array}{@{}rcl@{}} &&r_{+}\bar{p}_{s}(\beta-1)-r_{-}\bar{p}_{s}(\beta)+\bar{p}_{s}(-\beta)-\bar{p}_{s}(\beta)=0,\\ &&r_{-}\bar{p}_{s}(-\beta+1)-r_{+}\bar{p}_{s}(-\beta)+\bar{p}_{s}(\beta)-\bar{p}_{s}(-\beta)=0. \end{array} $$

Long term accuracy of the bounded accumulator is then determined by the weighted sum $\text {Acc}_{\infty }(\beta ) = \frac {1}{2} \bar {p}_{s}(0) + {\sum }_{n=1}^{\beta } \bar {p}_{s}(n)$.