Deriving Time-Averaged Active Inference from Control Principles

Abstract

Active inference offers a principled account of behavior as minimizing average sensory surprise over time. Applications of active inference to control problems have heretofore tended to focus on finite-horizon or discounted-surprise problems, despite deriving from the infinite-horizon, average-surprise imperative of the free-energy principle. Here we derive an infinite-horizon, average-surprise formulation of active inference from optimal control principles. Our formulation returns to the roots of active inference in neuroanatomy and neurophysiology, formally reconnecting active inference to optimal feedback control. Our formulation provides a unified objective functional for sensorimotor control and allows for reference states to vary over time.

A Detailed Derivations

This appendix provides detailed derivations for equations used elsewhere, particularly where doing so would have distracted from the flow of the paper.

Proposition 1 (Variational free energy as divergence from an unnormalized joint distribution)

The variational free energy (Eq. 9) is defined as the Kullback-Leibler divergence of the recognition model $q_\phi$ from the unnormalized joint distribution of the generative model $p_\theta$

$$\begin{aligned} \mathcal {F}_{\theta , \phi }(t)&= D_{\text {KL}}\left( q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) \Vert p_\theta (\textbf{s}_{t} \mid \textbf{s}_{t-1}) \right) , \end{aligned}$$

and therefore equals a sum of the cross entropy between the recognition model and the sensory likelihood and the exclusive KL divergence from the recognition model to the generative model over the latent variables

$$\begin{aligned} \mathcal {F}_{\theta , \phi }(t) ={} & {} \mathbb {E}_{q_\phi } \left[ - \log p_{\theta }(o_t \mid a^{(1)}_t, s^{(1)}_t) \right] + \\{} & {} \qquad D_{\text {KL}}\left( q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) \Vert p_{\theta }(s^{(1:2)}_t, a^{(1:2)}_t \mid \textbf{s}_{t-1}) \right) . \end{aligned}$$

Proof

Taking a divergence between the (normalized) recognition model and the (unnormalized) joint generative model will yield

$$\begin{aligned} \mathcal {F}_{\theta , \phi }(t)&= D_{\text {KL}}\left( q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) \Vert p_\theta (\textbf{s}_{t} \mid \textbf{s}_{t-1}) \right) \\&= \mathbb {E}_{ q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) } \left[ -\log \frac{ p_\theta (\textbf{s}_{t} \mid \textbf{s}_{t-1}) }{ q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) } \right] \\&= \mathbb {E}_{ q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) } \left[ - \log \frac{ p_{\theta }(o_t \mid a^{(1)}_t, s^{(1)}_t) p_{\theta }(s^{(1:2)}_t, a^{(1:2)}_t \mid \textbf{s}_{t-1}) }{ q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) } \right] \\&= \mathbb {E}_{q_\phi } \left[ - \log p_{\theta }(o_t \mid a^{(1)}_t, s^{(1)}_t) \right] - \mathbb {E}_{ q_\phi } \left[ \log \frac{ p_{\theta }(s^{(1:2)}_t, a^{(1:2)}_t \mid \textbf{s}_{t-1}) }{ q_\phi (s^{(1:2)}_{t}, a^{(1:2)}_{t} \mid o_{t}, \textbf{s}_{t+1}, \textbf{s}_{t-1}) } \right] , \end{aligned}$$

as required.

Proposition 2 (KL divergence of the optimal feedback controller from the feedforward controller)

The exclusive Kullback-Leibler divergence of the optimal feedback controller $q^{*}$ from the feedforward generative model $p_\theta$ is

$$\begin{aligned} D_{\text {KL}}\left( q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) \Vert p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) \right){} & {} = -\mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] - \nonumber \\{} & {} \log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] . \end{aligned}$$

(25)

Proof

We begin by writing out the definition of a KL divergence

$$\begin{aligned} D_{\text {KL}}\left( q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) \Vert p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) \right)&= \mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ -\log \frac{ p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) }{ q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) } \right] . \end{aligned}$$

The definition of $q^{*}$ in terms of $p_\theta$ (Eq. 21) allows the inner ratio of densities to simplify to

This simplified ratio therefore has the logarithm

$$\begin{aligned} \log \frac{ p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) }{ q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) }&= \log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] + \tilde{H}^{*}(t+1; \textbf{s}_{0}) \end{aligned}$$

and the divergence becomes

$$\begin{aligned}{} & {} D_{\text {KL}}\left( q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) \Vert p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) \right) = \\{} & {} \qquad -\mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] - \log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] . \end{aligned}$$

Proposition 3 (Path-integral expression for the optimal differential surprise-to-go)

The optimal differential surprise-to-go function defined by the Bellman equation (Eq. 20)

$$\begin{aligned} \tilde{H}^{*}(t; \textbf{s}_{0})&= h(t; \textbf{s}_{0}) + \min _{q_\phi } \mathbb {E}_{\textbf{s}_{t+1}\sim q_\phi (\cdot \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] \end{aligned}$$

can be simplified by substituting in $q^{*}$ to obtain a path-integral expression

$$\begin{aligned} \tilde{H}^{*}(\textbf{s}_{0})&= -\log \mathbb {E}_{ p_\theta (\textbf{s}_{1:T} \mid \textbf{s}_{0}) } \left[ \exp {\left( \sum _{t=1}^{T} \left( J(\textbf{s}_{t}) + L(\textbf{s}_{t}) \right) - \bar{\mathcal {J}}(\textbf{s}_{0}) \right) } \right] , \\&= -\log \mathbb {E}_{ q_\phi (\textbf{s}_{1:T} \mid \textbf{s}_{0}) } \left[ \exp {\left( \sum _{t=1}^{T} \mathcal {J}_{\theta ,\phi }(t) - \bar{\mathcal {J}}(\textbf{s}_{0}) \right) } \right] . \end{aligned}$$

Proof

Substituting Eq. 21 into Eq. 20 yields

$$\begin{aligned} \tilde{H}^{*}(t; \textbf{s}_{0})&= \bar{\mathcal {J}}(\textbf{s}_{0}) - \mathcal {J}_{\theta ,\phi }(t) + \mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] , \end{aligned}$$

(26)

whose recursive term is $\mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right]$. The divergence term in $\mathcal {J}$ (Eq. 14) will cancel this term. By Proposition 2 the divergence equals

$$\begin{aligned}{} & {} D_{\text {KL}}\left( q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t}) \Vert p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t}) \right) = \\{} & {} \qquad -\mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] - \log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] . \end{aligned}$$

Substituting Eq. 25 into Eq. 14 will yield

$$\begin{aligned} -\mathcal {J}_{\theta ,\phi }(t) = \mathbb {E}_{q^{*}(\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \tilde{H}^{*}(t+1; \textbf{s}_{0}) \right] +{} & {} \log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] \\{} & {} \qquad \,\,\, + \mathbb {E}_{q_\phi } \left[ -J(\textbf{s}_{t}) \right] + \mathbb {E}_{q_\phi } \left[ -L(\textbf{s}_{t}) \right] , \end{aligned}$$

whose first term will cancel the recursive optimization when substituted into Eq. 26. The result will be a “smoothly minimizing” expression for the optimal differential surprise-to-go

$$\begin{aligned}{} & {} \tilde{H}^{*}(t; \textbf{s}_{0}) = \bar{\mathcal {J}}(\textbf{s}_{0}) - \left( J(\textbf{s}_{t}) + L(\textbf{s}_{t})\right) \\{} & {} \qquad \qquad \qquad \qquad \qquad \qquad -\log \mathbb {E}_{p_\theta (\textbf{s}_{t+1} \mid \textbf{s}_{t})} \left[ \exp {\left( -\tilde{H}^{*}(t+1; \textbf{s}_{0}) \right) } \right] , \end{aligned}$$

and after unfolding of the recursive expectation, a path-integral expression for the optimal differential surprise-to-go

$$\begin{aligned} \tilde{H}^{*}(\textbf{s}_{0})&= -\log \mathbb {E}_{ p_\theta (\textbf{s}_{1:T} \mid \textbf{s}_{0}) } \left[ \exp {\left( \sum _{t=1}^{T} \left( J(\textbf{s}_{t}) + L(\textbf{s}_{t}) \right) - \bar{\mathcal {J}}(\textbf{s}_{0}) \right) } \right] . \end{aligned}$$

Sampling a trajectory of states from a feedback controller $q_\phi$ instead of the feedforward planner $p_\theta$ will then result in a nonzero divergence term

$$\begin{aligned} \tilde{H}^{*}(\textbf{s}_{0})&= -\log \mathbb {E}_{ q_\phi (\textbf{s}_{1:T} \mid \textbf{s}_{0}) } \left[ \exp {\left( \sum _{t=1}^{T} \mathcal {J}_{\theta ,\phi }(t) - \bar{\mathcal {J}}(\textbf{s}_{0}) \right) } \right] . \end{aligned}$$