Policies and Priors

Abstract

This chapter considers addiction from a purely theoretical point of view. It tries to substantiate the idea that addictive behaviour is a natural consequence of abnormal perceptual learning. In short, addictive behaviours emerge when behaviour confounds its own acquisition. Specifically, we consider what would happen if behaviour interfered with the neurotransmitter systems responsible for optimising the conditional certainty or precision of inferences about causal structure in the world. We will pursue this within a rather abstract framework provided by free-energy formulations of action and perception. Although this treatment does not touch upon many of the neurobiological or psychosocial issues in addiction research, it provides a principled framework within which to understand exchanges with the environment and how they can be disturbed. Our focus will be on behaviour as active inference and the key role of prior expectations. These priors play the role of policies in reinforcement learning and place crucial constraints on perceptual inference and subsequent action. A dynamical treatment of these policies suggests a fundamental distinction between fixed-point policies that lead to a single attractive state and itinerant policies that support wandering behavioural orbits among sets of attractive states. Itinerant policies may provide a useful metaphor for many forms of behaviour and, in particular, addiction. Under these sorts of policies, neuromodulatory (e.g., dopaminergic) perturbations can lead to false inference and consequent learning, which produce addictive and preservative behaviour.

Appendices

Appendix A: Parameter Optimisation and Newton’s Method

There is a close connection between the updates implied by Eq. (9.9) and Newton’s method for optimisation. Consider the update under a local linearisation, assuming $\mathcal{L}_{\varphi}\approx\mathcal{F}_{\varphi}$

(A.1)

As time proceeds, the change in generalised mean becomes

(A.2)

The first line means the motion cancels itself and becomes zero, while the change in the conditional mean $\Delta\mu^{(\varphi )} = -\mathcal{L}_{\varphi \varphi} ^{ - 1}\mathcal{L}_{\varphi}$ becomes a classical Newton update. The conditional expectations of the parameters were updated after every simulated exposure using this scheme, as described in Friston (2008).

Appendix B: Simulating Action and Perception

The simulations in this paper involve integrating time-varying states in the environment and the agent. This is the solution to the following ordinary differential equation

(B.1)

To update these states we use a local linearisation; $\Delta u =(\exp(\Delta t\Im) - I)\Im(t)^{ - 1}\dot{u}$ over time steps of Δt, where $\Im = \partial\dot{u} / \partial u$ is evaluated at the current conditional expectation (Friston et al. 2010).