Skip to main content
Log in

Analyzing dynamic decision-making models using Chapman-Kolmogorov equations

Journal of Computational Neuroscience Aims and scope Submit manuscript

Cite this article


Decision-making in dynamic environments typically requires adaptive evidence accumulation that weights new evidence more heavily than old observations. Recent experimental studies of dynamic decision tasks require subjects to make decisions for which the correct choice switches stochastically throughout a single trial. In such cases, an ideal observer’s belief is described by an evolution equation that is doubly stochastic, reflecting stochasticity in the both observations and environmental changes. In these contexts, we show that the probability density of the belief can be represented using differential Chapman-Kolmogorov equations, allowing efficient computation of ensemble statistics. This allows us to reliably compare normative models to near-normative approximations using, as model performance metrics, decision response accuracy and Kullback-Leibler divergence of the belief distributions. Such belief distributions could be obtained empirically from subjects by asking them to report their decision confidence. We also study how response accuracy is affected by additional internal noise, showing optimality requires longer integration timescales as more noise is added. Lastly, we demonstrate that our method can be applied to tasks in which evidence arrives in a discrete, pulsatile fashion, rather than continuously.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others


  1. The notation ot) means all other terms are of smaller order than Δt. More precisely, ot) represents a function ft) with the property \(\lim \limits _{\Delta t\downarrow 0}\frac {f({\Delta } t)}{\Delta t}=0\).


  • Bankó, É.M., Gál, V., Körtvélyes, J., Kovács, G., Vidnyánszky, Z. (2011). Dissociating the effect of noise on sensory processing and overall decision difficulty. Journal of Neuroscience, 31(7), 2663–2674.

    Article  Google Scholar 

  • Behrens, T.E., Woolrich, M.W., Walton, M.E., Rushworth, M.F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214.

    Article  CAS  Google Scholar 

  • Billingsley, P. (2008). Probability and measure. Wiley.

  • Bogacz, R., Brown, E., Moehlis, J., Holmes, P., Cohen, J.D. (2006). The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review, 113 (4), 700.

    Article  Google Scholar 

  • Brea, J., Urbanczik, R., Senn, W. (2014). A normative theory of forgetting: lessons from the fruit fly. PLoS Computational Biology, 10(6), e1003640.

    Article  Google Scholar 

  • Brody, C.D., Romo, R., Kepecs, A. (2003). Basic mechanisms for graded persistent activity: discrete attractors, continuous attractors, and dynamic representations. Current Opinion in Neurobiology, 13(2), 204–211.

    Article  CAS  Google Scholar 

  • Brunton, B.W., Botvinick, M.M., Brody, C.D. (2013). Rats and humans can optimally accumulate evidence for decision-making. Science, 340(6128), 95–98.

    Article  CAS  Google Scholar 

  • Busemeyer, J.R., & Townsend, J.T. (1992). Fundamental derivations from decision field theory. Mathematical Social Sciences, 23(3), 255–282.

    Article  Google Scholar 

  • Droste, F., & Lindner, B. (2014). Integrate-and-fire neurons driven by asymmetric dichotomous noise. Biological Cybernetics, 108(6), 825–843.

    Article  Google Scholar 

  • Droste, F., & Lindner, B. (2017). Exact results for power spectrum and susceptibility of a leaky integrate-and-fire neuron with two-state noise. Physical Review E, 95(1), 012411.

    Article  Google Scholar 

  • Drugowitsch, J. (2016). Fast and accurate monte carlo sampling of first-passage times from wiener diffusion models. Scientific Reports, 6, 20490.

    Article  CAS  Google Scholar 

  • Drugowitsch, J., Moreno-Bote, R., Churchland, A.K., Shadlen, M.N., Pouget, A. (2012). The cost of accumulating evidence in perceptual decision making. Journal of Neuroscience, 32(11), 3612–3628.

    Article  CAS  Google Scholar 

  • Eckhoff, P., Holmes, P., Law, C., Connolly, P., Gold, J. (2008). On diffusion processes with variable drift rates as models for decision making during learning. New Journal of Physics, 10(1), 015006.

    Article  Google Scholar 

  • Eissa, T.L., Barendregt, N.W., Gold, J.I., Josić, K, Kilpatrick, Z.P. (2019). Hierarchical inference interactions in dynamic environments. In Computational and Systems Neuroscience. Lisbon.

  • Erban, R., & Chapman, S.J. (2007). Reactive boundary conditions for stochastic simulations of reaction–diffusion processes. Physical Biology, 4(1), 16.

    Article  CAS  Google Scholar 

  • Faisal, A.A., Selen, L.P., Wolpert, D.M. (2008). Noise in the nervous system. Nature Reviews Neuroscience, 9(4), 292.

    Article  CAS  Google Scholar 

  • Friedman, J, Hastie, T, Tibshirani, R. (2001). The elements of statistical learning. chap 7: Model Assessment and Selection, Vol. 1. New York: Springer series in statistics.

  • Gardiner, C. (2004). Handbook of stochastic methods: for physics, chemistry & the natural sciences, (series in synergetics, vol. 13).

  • Geisler, W.S. (2003). Ideal observer analysis. The Visual Neurosciences, 10(7), 12–12.

    Google Scholar 

  • Glaze, C.M., Kable, J.W., Gold, J.I. (2015). Normative evidence accumulation in unpredictable environments. Elife, 4, e08825.

    Article  Google Scholar 

  • Glaze, C.M., Filipowicz, A.L., Kable, J.W., Balasubramanian, V., Gold, J.I. (2018). A bias–variance trade-off governs individual differences in on-line learning in an unpredictable environment. Nature Human Behaviour, 2(3), 213.

    Article  Google Scholar 

  • Gold, J.I., & Shadlen, M.N. (2007). The neural basis of decision making. Annual Review of Neuroscience, 30.

  • Hanson, F.B. (2007). Applied stochastic processes and control for Jump-diffusions: modeling, analysis, and computation, vol 13. SIAM.

  • Heath, R.A. (1992). A general nonstationary diffusion model for two-choice decision-making. Mathematical Social Sciences, 23(3), 283–309.

    Article  Google Scholar 

  • Horsthemke, W., & Lefever, R. (2006). Noise-induced transitions: theory and applications in physics, chemistry and biology. Springer Series in Synergetics. Berlin: Springer.

    Google Scholar 

  • Kiani, R., & Shadlen, M.N. (2009). Representation of confidence associated with a decision by neurons in the parietal cortex. Science, 324(5928), 759–764.

    Article  CAS  Google Scholar 

  • Moehlis, J., Brown, E., Bogacz, R., Holmes, P., Cohen, J.D. (2004). Optimizing reward rate in two alternative choice tasks: mathematical formalism. Center for the study of brain, mind and behavior (pp. 04–01). Princeton University.

  • Ossmy, O., Moran, R., Pfeffer, T., Tsetsos, K., Usher, M., Donner, T.H. (2013). The timescale of perceptual evidence integration can be adapted to the environment. Current Biology, 23(11), 981–986.

    Article  CAS  Google Scholar 

  • Piet, A.T., El Hady, A., Brody, C.D. (2018). Rats adopt the optimal timescale for evidence integration in a dynamic environment. Nature Communications, 9(1), 4265.

    Article  Google Scholar 

  • Piet, A., Hady, A.E., Boyd-Meredith, T., Brody, C. (2019). Neural dynamics during changes of mind. In Computational and Systems Neuroscience (p. 2019). Lisbon.

  • Radillo, A.E., Veliz-Cuba, A., Josić, K, Kilpatrick, Z.P. (2017). Evidence accumulation and change rate inference in dynamic environments. Neural Computation, 29(6), 1561–1610.

    Article  Google Scholar 

  • Radillo, A.E., Veliz-Cuba, A., Josić, K. (2019). Performance of normative and approximate evidence accumulation on the dynamic clicks task. Neurons, Behavior, Data analysis, and Theory. submitted.

  • Rahnev, D., & Denison, R.N. (2018). Suboptimality in perceptual decision making. Behavioral and Brain Sciences, 41, e223.

    Article  Google Scholar 

  • Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85(2), 59.

    Article  Google Scholar 

  • Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: theory and data for two-choice decision tasks. Neural Computation, 20(4), 873–922.

    Article  Google Scholar 

  • Salinas, E., & Sejnowski, T.J. (2002). Integrate-and-fire neurons driven by correlated stochastic input. Neural Computation, 14(9), 2111–2155.

    Article  Google Scholar 

  • Skellam, J.G. (1946). The frequency distribution of the difference between two poisson variates belonging to different populations. Journal of the Royal Statistical Society Series A (General), 109(Pt 3), 296–296.

    Article  CAS  Google Scholar 

  • Smith, P.L. (2010). From poisson shot noise to the integrated ornstein–uhlenbeck process: neurally principled models of information accumulation in decision-making and response time. Journal of Mathematical Psychology, 54 (2), 266–283.

    Article  Google Scholar 

  • Smith, P.L., & Ratcliff, R. (2004). Psychology and neurobiology of simple decisions. Trends in Neurosciences, 27(3), 161–168.

    Article  CAS  Google Scholar 

  • Urai, A.E., Braun, A., Donner, T.H. (2017). Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias. Nature Communications, 8, 14637.

    Article  Google Scholar 

  • Van Den Berg, R., Anandalingam, K., Zylberberg, A., Kiani, R., Shadlen, M.N., Wolpert, D.M. (2016). A common mechanism underlies changes of mind about decisions and confidence. Elife, 5, e12192.

    Article  Google Scholar 

  • Veliz-Cuba, A., Kilpatrick, Z.P., Josic, K. (2016). Stochastic models of evidence accumulation in changing environments. SIAM Review, 58(2), 264–289.

    Article  Google Scholar 

  • Wilson, R.C., Nassar, M.R., Gold, J.I. (2010). Bayesian online learning of the hazard rate in change-point problems. Neural Computation, 22(9), 2452–2476.

    Article  Google Scholar 

  • Yu, A.J., & Cohen, J.D. (2008). Sequential effects: superstition or rational behavior? Advances in Neural Information Processing Systems, 21, 1873–1880.

    PubMed  PubMed Central  Google Scholar 

  • Zhang, S., Lee, M.D., Vandekerckhove, J., Maris, G., Wagenmakers, E.J. (2014). Time-varying boundaries for diffusion models of decision making and response time. Frontiers in Psychology, 5, 1364.

    PubMed  PubMed Central  Google Scholar 

Download references


This work was supported by and NSF/NIH CRCNS grant (R01MH115557) and NSF (DMS-1517629). ZPK was also supported by NSF (DMS-1615737 & DMS-1853630). KJ was also supported by NSF (DBI-1707400). We thank Sam Isaacson and Jay Newby for feedback on the boundary value problem for the bounded accumulator model. We are also grateful to Adrian Radillo and Tahra Eissa for comments on a draft version of the manuscript.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Zachary P. Kilpatrick.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Action Editor: P. Dayan

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Code availability

See the MATLAB finite difference code used to perform the analysis and generate figures.

K. Josić and Z.P. Kilpatrick share equal authorship.


Appendix A: Normative evidence-accumulation in dynamic environments

Here we derive the continuum limit of the Bayesian update equation for continuous evidence accumulation in a changing environment. Starting with the discrete time model, we define Ln = P(s(tn) = s±|ξ1:n) as the probability of being in state s± at time tn assuming a sequence of observations ξ1:n. The state s(t) changes between evenly spaced time points t1:n (with Δt := tntn− 1) at a hazard rate hΔt := h ⋅Δt := P(s(tn) = s|s(tn− 1) = s±). The likelihood function fΔt(ξ) = P(ξ|s = s±; Δt) is the conditional probability of observing sample ξ given state s±, parameterized by Δt.

We begin by assuming an ideal observer who knows the environmental hazard rate h. Using Bayes’ rule and the law of total probability, we can relate Ln to the probability at the previous time step according to the weighted sum (Veliz-Cuba et al. 2016)

$$ L_{n, \pm} = f_{\Delta t, \pm}(\xi_{n}) \frac{\mathrm{P}(\xi_{1:n})}{\mathrm{P}(\xi_{1:n})} \left[ (1- h_{\Delta t}) L_{n-1, \pm} + h_{\Delta t} L_{n-1, \mp} \right], $$

where L0,± = P(s(t0) = s±). Defining \(y_{n} = \log \frac {L_{n, +}}{L_{n,-}}\), we can compute

$$ {\Delta} y_{n} : = y_{n} - y_{n-1} = \log \frac{f_{\Delta t, +}(\xi_{n})}{f_{\Delta t, -}(\xi_{n})} + \log \frac{1 - h_{\Delta t} + h_{\Delta t} \mathrm{e}^{-y_{n-1}}}{1 - h_{\Delta t} + h_{\Delta t} \mathrm{e}^{y_{n-1}}}. $$

In search of the continuum limit of this equation, we assume 0 < Δt ≪ 1, 0 < |Δyn|≪ 1, and use the approximation \(\log (1 + z) \approx z\) to obtain

$$ {\Delta} y_{n} \approx \log \frac{f_{\Delta t, +}(\xi_{n})}{f_{\Delta t, -} (\xi_{n})} - 2 h \cdot {\Delta} t \sinh(y_{n}). $$

Replacing the index n with the time t and applying the functional central limit theorem as in Billingsley (2008) and Bogacz et al. (2006), we can write Eq. (27) as

$$ {\Delta} y_{t} \approx {\Delta} t g_{\Delta t}(t) + \sqrt{\Delta t} \rho_{\Delta t}(t) \eta - 2 h \cdot {\Delta} t \sinh(y_{t}), $$

where η is a random variable with a standard normal distribution and

$$ \begin{array}{@{}rcl@{}} g_{\Delta t}(t) &&: = \frac{1}{\Delta t} \mathrm{E}_{\xi} \left[ \log \frac{f_{\Delta t, +}(\xi)}{f_{\Delta t, -} (\xi)} \Big| s(t) \right], \rho_{\Delta t}^{2} (t) \\&&: = \frac{1}{\Delta t} \text{Var}_{\xi} \left[ \log \frac{f_{\Delta t, +}(\xi)}{f_{\Delta t, -} (\xi)} \Big| s(t) \right]. \end{array} $$

The drift gΔt and variance \(\rho _{\Delta t}^{2}\) diverge unless fΔt(ξ) are scaled appropriately in the Δt → 0 limit. A reasonable assumption that can be made to compute gΔt and \(\rho _{\Delta t}^{2}\) explicitly is to take observations ξ to follow normal distributions with mean and variance scaled by Δt (Bogacz et al. 2006; Veliz-Cuba et al. 2016)

$$ f_{\Delta t, \pm} (\xi) = \frac{1}{\sqrt{2 \pi {\Delta} t \sigma^{2}}} \mathrm{e}^{-(\xi \mp {\Delta} \mu)^{2}/(2 {\Delta} t \sigma^{2})}, $$

so we can compute the limits of Eq. (29) as

$$ \begin{array}{@{}rcl@{}} g(t) &=& \lim_{\Delta t \to 0} g_{\Delta t}(t) \in \pm \frac{2 \mu^{2}}{\sigma^{2}} = \pm g, \end{array} $$
$$ \begin{array}{@{}rcl@{}} \rho^{2}(t) &=& \lim_{\Delta t \to 0} \rho_{\Delta t}^{2}(t) = \frac{4 \mu^{2}}{\sigma^{2}} = \rho^{2}, \end{array} $$

where g(t) ∈{+g,−g} is a telegraph process with probability masses P(±g, t) evolving as \(\dot {\mathrm {P}}(\pm g,t) = h \left [ \mathrm {P} (\mp g, t) - \mathrm {P} (\pm g,t) \right ] \) and ρ2(t) = ρ2 remains constant. Therefore, the continuum limit (Δt → 0) of Eq. (28) is

$$ \mathrm{d} y(t) = g(t) \mathrm{d} t + \rho \mathrm{d} W_{t} - 2 h \sinh (y(t)) \mathrm{d} t, $$

where dW is a standard Wiener process. Equation (31) provide the normative model of evidence accumulation for an observer who knows the hazard rate h and wishes to infer the sign of g(t) at time t with maximal accuracy (Glaze et al. 2015; Veliz-Cuba et al. 2016).

However, we are also interested in near-normative models in which the observer assumes an incorrect hazard rate \(\tilde {h} \neq h\). In such a case, the analysis proceeds as before, with the probabilistic inference process simply involving \(\tilde {h}\) now rather than h, and the result is

$$ \mathrm{d} y(t) = g(t) \mathrm{d} t + \rho \mathrm{d} W_{t} - 2 \tilde{h} \sinh (y(t)) \mathrm{d} t. $$

Lastly, note that if indeed the original observations ξ are drawn from normal distributions, Eq. (30a), (30b) states g(t) ∈±g where g = 2μ2/σ2 and ρ2 = 2g. Rescaling time htt, we can then express Eq. (32) in terms of the following rescaled equation

$$ \mathrm{d} y(t) = x(t) m \mathrm{d} t + \sqrt{2 m} \mathrm{d} W_{t} - 2 \frac{\tilde{h}}{h} \sinh(y) \mathrm{d} t, $$

where m = 2μ2/(hσ2) and x(t) ∈± 1 is a telegraph process with hazard rate 1, as shown in Eq. (2) of the main text.

Appendix B: Derivation of the Chapman-Kolmogorov equations

Here we outline the derivation of the CK equations given by Eqs. (4a), (4b) and (6) in the main text. Our goal is to write a PDE that describes the evolution of an ensemble of belief trajectories described by the SDE in Eq. (2) across all realizations of both sources of stochasticity – observation noise and state switching. We cannot simply write a Fokker-Planck equation to describe the desired probability distribution, as the process is doubly stochastic; the diffusion is described by a scaled Wiener process and the sign of the drift is controlled by a two-state Markov process. Therefore, following Droste and Lindner (2017), we condition our process on the state of x(t), and seek to obtain evolution equations for the conditional distributions p±(y, t) := p(y, t|x(t) = ±).

We start by conditioning on x(t) = + 1 to find an equation for p+(y, t). In the absence of state transitions in the background Markov process, the evolution equation for p+(y, t) is now given by the Fokker-Planck equation

$$ \frac{\partial p_{+}}{\partial t}=-m\frac{\partial p_{+}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{+}\right]+m\frac{\partial^{2}p_{+}}{\partial y^{2}}. $$

On the other hand, in the absence of drift and diffusion, the evolution of p+(y, t) is governed exclusively by the two-state Markov process; in this case, we can write the evolution equation in the form of a discrete master equation:

$$ \frac{dp_{+}}{dt}=p_{-}-p_{+}. $$

Note here that because of the rescaling of time in Eq. (2), the transition rates of the Markov process equal one. Given these two components, we can follow Gardiner (2004) to write the CK equation that describes the evolution of p+(y, t) when drift, diffusion, and switching are all present as

$$ \frac{\partial p_{+}}{\partial t}=-m\frac{\partial p_{+}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{+}\right]+m\frac{\partial^{2}p_{+}}{\partial y^{2}}+\left[p_{-}-p_{+}\right], $$

which is Eqs. (4a); (4b) is obtained similarly by conditioning on x(t) = − 1.

To determine response accuracy, we are interested in the probability the observer responds with the correct state of the Markov process. Therefore, at an interrogation time t = T where x(T) = + 1, the observer is correct if y > 0. This probability is given by the integral \({\int \limits }_{0}^{\infty } p_{+}(y,T) dy\). Similarly, if x(T) = − 1, the observer is correct if y < 0, which happens with probability \({\int \limits }_{-\infty }^{0} p_{-}(y,T) dy\). Therefore, the total response accuracy at time T is given by summing these two integrals:

$$ \text{Acc}(T)={\int}_{0}^{\infty} p_{+}(y,T) dy+{\int}_{-\infty}^{0} p_{-}(y,T) dy, $$

which is Eq. (5).

Finally, in order to more efficiently simulate the evolution of p+(y, t) and p(y, t), we notice that because the nonlinear leak is an odd function, a change of variables − yy in Eq. (4b) allows us to combine Eqs. (4a) and (4b) into a single PDE. Defining ps(y, t) := p+(y, t) + p(−y, t), we obtain the evolution equation

$$ \frac{\partial p_{s}}{\partial t}=-m\frac{\partial p_{s}}{\partial y}+2\frac{\tilde{h}}{h}\frac{\partial}{\partial y}\left[\sinh(y)p_{s}\right]+m\frac{\partial^{2}}{\partial y^{2}}+\left[p_{s}(-y,t)-p_{s}(y,t)\right], $$

which is Eq. (6).

Appendix C: Finite difference methods for Chapman-Kolmogorov equations

Fig. 9
figure 9

Comparison of CK equations with Monte Carlo sampling. a Calculation of accuracy of mistuned nonlinear leak model for m = 5. Monte Carlo simulations run with varied number of samples superimposed (legend). b Runtime (red) and L2 error (blue) of Monte Carlo simulations as a function of sample size. Runtime of CK equations (black dashed) superimposed for comparison. L2 error of Monte Carlo simulations calculated against results from CK equations. c Runtime (red) and \(L^{\infty }\) error (blue) of finite difference simulations of the nonlinear model Eq. (6) under refinement of belief discretization, compared with the Δy = 10− 4 case. Because our method is first-order accurate in time and second-order accurate in belief, the method is first-order accurate when Δy is decreased and Δt is held constant

We used a finite difference method to simulate the differential CK equations. The method is exemplified here for the normative CK equation from Eq. (6), but a similar approach was used for the linear, cubic, and pulsatile equations. For stability purposes, our method uses centered differences in y and backward-Euler in t. This gives the following finite difference approximations of the functions and their derivatives in Eq. (6):

$$ \begin{array}{@{}rcl@{}} \frac{\partial p_{s}(y,t)}{\partial t} &\approx& \frac{p_{s}(y,t+{\Delta} t)-p_{s}(y,t)}{\Delta t}, p_{s}(y,t) \approx p_{s}(y,t+{\Delta} t),\\ \frac{\partial p_{s}(y,t)}{\partial y} &\approx& \frac{p_{s}(y+{\Delta} y,t+{\Delta} t)-p_{s}(y-{\Delta} y,t+{\Delta} t)}{2{\Delta} y}, \\ \frac{\partial^{2}p_{s}(y,t)}{\partial y^{2}} & \approx& \frac{p_{s}(y\!-{\Delta} y,t\!+{\Delta} t)-2p_{s}(y,t+{\Delta} t)+p_{s}(y+{\Delta} y,t+{\Delta} t)}{({\Delta} y)^{2}}, \end{array} $$

where Δt and Δy are timestep and spacestep of the simulation, respectively. Substituting into Eq. (6) and solving for ps(y, t) at each point on a mesh y for y gives the system of equations:

$$ \mathbf{A}p_{s}(\mathbf{y},t+{\Delta} t)=p(\mathbf{y},t), $$

where A is tridiagonal with elements along the primary off-diagonal. This system can be inverted at each timestep and used to calculate the updates ps(y, t + Δt).

For the boundary conditions, we imposed no-flux conditions at the mesh boundaries ± b. For a standard drift-diffusion equation with drift A(y) and diffusion constant B(y), this condition takes the form

$$ J(\pm b,t)=A(\pm b)p(\pm b,t)-\frac{1}{2}\frac{\partial}{\partial y}\left[B(\pm b)^{2}p(\pm b,t)\right]=0. $$

Using the finite difference approximations

$$ \begin{array}{@{}rcl@{}} \frac{\partial p_{s}(y,t)}{\partial y} & \approx& \frac{3p_{s}(y,t + {\Delta} t) - 4p_{s}(y+{\Delta} y,t+{\Delta} t)+p_{s}(y+2{\Delta} y,t+\!{\Delta} t)}{2{\Delta} y}, \\ \frac{\partial p_{s}(y,t)}{\partial y} & \!\approx& \frac{\!-p_{s}(y-2{\Delta} y,t+{\Delta} t)+4p_{s}(y - {\Delta} y,t + {\Delta} t) - 3p_{s}(y,t+\!{\Delta} t)}{2{\Delta} y}, \end{array} $$

we can plug in ± b to the appropriate replacement and use Eq. (34) to find the appropriate boundary terms for the system in Eq. (33).

Figure 9 shows the results of Monte Carlo simulations compared against those from the CK equations; Monte Carlo simulations are less smooth (Fig. 9a), making optimality calculations less accurate. Furthermore, obtaining results that are close to those from the CK equations takes much longer to run (Fig. 9b), as we can obtain good accuracy (error ≈ 10− 2) from the CK equations in less than a second (Fig. 9c).

Appendix D: Distinguishing linear discounting models using confidence

Fig. 10
figure 10

Distinguishing linear discounting parameter λ using confidence reports. a Average number of trials needed to determine whether a subject uses the maximizing accuracy leak parameter, λAcc, or the minimizing KL divergence leak parameter, λKL, as a function of evidence strength m. Averages were computed over a thousand simulations for interrogation time T = 1, with each simulation run until confidence in the subject’s model reached 90%

Can we use the distributions obtained from the CK equations to distinguish which model a subject uses? To illustrate an example, we considered distinguishing the two linear discounting models given by Eq. (7) and asked how many trials an experimentalist would need to run in order to tell if a subject was using λAcc or λKL. We sampled from the distributions obtained from Eq. (8) and performed a likelihood ratio test to find the number of samples needed for the experimentalist to have 90% confidence in the model being used. Assuming a symmetric prior P(λAcc) = P(λKL) = 0.5, and defining \(\mathcal { Y}_{j} = y_{j}(T)\) as the belief report at the end (t = T) of trial j, the likelihood ratio is computed using Bayes’ rule and independence after N trials as

$$ \frac{P(\lambda^{\text{Acc}}|\mathcal{Y}_{1:N})}{P(\lambda^{\text{KL}}|\mathcal{Y}_{1:N})} = \prod\limits_{j=1}^{N} \frac{P(\mathcal{Y}_{j} | \lambda^{\text{Acc}})}{P(\mathcal{Y}_{j} | \lambda^{\text{KL}})}, $$

and we counted the number of trials required to obtain ≥ 90% confidence in either model (so \(P(\lambda |\mathcal {Y}_{1:N}) \geq 0.9\) for either λ ∈{λAcc, λKL}). The results of this simulation are given in Fig. 10, showing that as the strength m of evidence increases, the mean number of trials 〈N(m)〉 required to distinguish models decreases precipitously. This is to be expected based on the fact that the belief distributions become more separated as m increases (Fig. 3e).

Appendix E: Deriving differential CK equation with internal noise

Here we provide intuition for the form of the diffusion coefficient in Eq. (13) for the belief distribution of a normative observer strategy with additional internal noise of strength D. Starting with the SDE in Eq. (12), because \(\sqrt {2m}\mathrm {d} W_{t}\) and \(\sqrt {2D}\mathrm {d} X_{t}\) are increments of independent Wiener processes, we can define a new Wiener process \(A\mathrm {d} Z_{t}=\sqrt {2m}\mathrm {d} W_{t}+\sqrt {2D}\mathrm {d} X_{t}\) that has the same statistics as the original summed Wiener processes (Gardiner 2004). To determine the appropriate effective diffusion constant A, we note that

$$ \text{Var}[A\mathrm{d} Z_{t}]=A^{2}\text{Var}[\mathrm{d} Z_{t}]=A^{2}t $$


$$ \text{Var}[\sqrt{2m}\mathrm{d} W_{t}+\sqrt{2D}\mathrm{d} X_{t}]=2m\text{Var}[\mathrm{d} W_{t}]+2D\text{Var}[\mathrm{d} X_{t}]=2mt+2Dt. $$

This requires \(A=\sqrt {2(m+D)}\), and means Eq. (12) can be rewritten as

$$ \mathrm{d} y= x(t) \cdot m \mathrm{d} t +\sqrt{2(m+D)}\mathrm{d} Z_{t} -2\frac{\tilde{h}}{h}\sinh(y) \mathrm{d} t, $$

which following Gardiner (2004), has the differential CK equation given by Eq. (13).

Appendix F: Steady state solution of the bounded accumulator model

Steady state solutions of Eq. (16) are derived first by noting that tp± = 0 implies

$$ 0 = \mp m \bar{p}_{\pm}^{\prime} (y) \pm m \bar{p}_{\pm}^{\prime\prime}(y) +\left[\bar{p}_{\mp} (y)-\bar{p}_{\pm}(y) \right], $$

with boundary conditions \(\bar {p}_{\pm }(y) (\pm \beta ) = \bar {p}_{\pm }^{\prime }(\pm \beta )\). Equation (35) has solutions \(\left (\begin {array}{c} \bar {p}_{+}(y)\\ \bar {p}_{-}(y) \end {array} \right ) = \left (\begin {array}{c} A \\ B \end {array} \right ) \mathrm {e}^{\alpha y}\), with characteristic equation \(m^{2} \alpha ^{4}-\left (m^{2}+2m\right ) \alpha ^{2}=0\). The characteristic roots are α = 0,±q, where we define \(q = \sqrt {1+\frac {2}{m}}\). For α = 0, we have A = B, whereas for α = ±q, the symmetry \(\bar {p}_{+}(y)=\bar {p}_{-}(-y)\) implies B = (mq − (m + 1))A for α = +q and A = (mq − (m + 1))B for α = −q. Lastly, defining the sum \(\bar {p}_{s}(y)=\bar {p}_{+}(y)+\bar {p}_{-}(-y)\), we obtain

$$ \bar{p}_{s}(y)=C_{1}+C_{2}\left[e^{qy}+\left( mq-(m+1)\right)e^{-qy}\right]. $$

The no flux boundary conditions \(\bar {p}_{s}(\pm \beta )-\frac {\partial \bar {p}_{s}(\pm \beta )}{\partial y}=0\) along with the normalization requirement \({\int \limits }_{- \beta }^{\beta } \bar {p}_{s}(y) \mathrm {d} y = 1\) give explicit expressions for the constants

$$ C_{1}=\frac{1}{2\beta}-C_{2}\left( \frac{m(q-1)\sinh(q\beta)}{q\beta}\right) $$


$$ C_{2} = \left\{2\beta\left[(q-1)\left( e^{q\beta} + \frac{m\sinh(q\beta)}{q\beta}\right)-(q + 1)(mq-(m\!+1))e^{-q\beta}\right]\right\}^{-1}. $$

Appendix G: Steady state solution of the clicks-task bounded accumulator model

Considering Eq. (23), we look for stationary solutions of the form \(\left (\begin {array}{llll} \bar {p}_{+}(n)\\ \bar {p}_{-}(n) \end {array}\right )=\left (\begin {array}{llll} C_{1}\\ C_{2} \end {array}\right ) \alpha ^{n}\), yielding the characteristic equation

$$ \begin{array}{@{}rcl@{}} &&\left[r_{+}\alpha^{n-1}+r_{-}\alpha^{n+1}-r_{s} \alpha^{n}\right]\left[r_{-}\alpha^{n-1}+r_{+} \alpha^{n+1} - r_{s} \alpha^{n}\right] \\&&\quad- \alpha^{2n}=0, \end{array} $$

where rs = r+ + r + 1. Solving Eq. (36) gives α = 1 with eigenfunction C1 = C2 and two roots α = q± of the quadratic \(\alpha ^{2}- \alpha (r_{+}+r_{+}^{2}+r_{-}+r_{-}^{2})/(r_{+} r_{-}) +1=0\). Superimposing the eigenfunctions, redefining constants, and defining \(\bar {p}_{s}(n)=\bar {p}_{+}(n)+\bar {p}_{-}(-n)\) gives the general solution

$$ \bar{p}_{s}(n)=C_{1}+C_{2}q_{+}^{n}+C_{3}q_{-}^{n}. $$

The constants C1, C2, and C3 can be determined by normalization \({\sum }_{n=-\beta }^{\beta }\bar {p}_{s}(n)=1\) and the stationary boundary conditions

$$ \begin{array}{@{}rcl@{}} &&r_{+}\bar{p}_{s}(\beta-1)-r_{-}\bar{p}_{s}(\beta)+\bar{p}_{s}(-\beta)-\bar{p}_{s}(\beta)=0,\\ &&r_{-}\bar{p}_{s}(-\beta+1)-r_{+}\bar{p}_{s}(-\beta)+\bar{p}_{s}(\beta)-\bar{p}_{s}(-\beta)=0. \end{array} $$

Long term accuracy of the bounded accumulator is then determined by the weighted sum \(\text {Acc}_{\infty }(\beta ) = \frac {1}{2} \bar {p}_{s}(0) + {\sum }_{n=1}^{\beta } \bar {p}_{s}(n)\).

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barendregt, N.W., Josić, K. & Kilpatrick, Z.P. Analyzing dynamic decision-making models using Chapman-Kolmogorov equations. J Comput Neurosci 47, 205–222 (2019).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: