Optimal control as a graphical model inference problem
Authors
 First Online:
 Received:
 Accepted:
DOI: 10.1007/s1099401252787
 Cite this article as:
 Kappen, H.J., Gómez, V. & Opper, M. Mach Learn (2012) 87: 159. doi:10.1007/s1099401252787
Abstract
We reformulate a class of nonlinear stochastic optimal control problems introduced by Todorov (in Advances in Neural Information Processing Systems, vol. 19, pp. 1369–1376, 2007) as a KullbackLeibler (KL) minimization problem. As a result, the optimal control computation reduces to an inference computation and approximate inference methods can be applied to efficiently compute approximate optimal controls. We show how this KL control theory contains the path integral control method as a special case. We provide an example of a block stacking task and a multiagent cooperative game where we demonstrate how approximate inference can be successfully applied to instances that are too complex for exact computation. We discuss the relation of the KL control approach to other inference approaches to control.
Keywords
Optimal control Uncontrolled dynamics KullbackLeibler divergence Graphical model Approximate inference Cluster variation method Belief propagation1 Introduction
Stochastic optimal control theory deals with the problem to compute an optimal set of actions to attain some future goal. With each action and each state a cost is associated and the aim is to minimize the total future cost. Examples are found in many contexts such as motor control tasks for robotics, planning and scheduling tasks or managing a financial portfolio. The computation of the optimal control is typically very difficult due to the size of the state space and the stochastic nature of the problem.
The most common approach to compute the optimal control is through the Bellman equation. For the finite horizon discrete time case, this equation results from a dynamic programming argument that expresses the optimal costtogo (or value function) at time t in terms of the optimal costtogo at time t+1. For the infinite horizon case, the value function is independent of time and the Bellman equation becomes a recursive equation. In continuous time, the Bellman equation becomes a partial differential equation.
For high dimensional systems or for continuous systems the state space is huge and the above procedure cannot be directly applied. A common approach to make the computation tractable is a function approximation approach where the value function is parameterized in terms of a number of parameters (Bertsekas and Tsitsiklis 1996). Another promising approach is to exploit graphical structure that is present in the problem to make the computation more efficient (Boutilier et al. 1995; Koller and Parr 1999). However, this graphical structure is in general not inherited by the value function, and thus the graphical representation of the value function may not be appropriate.
In this paper, we introduce a class of stochastic optimal control problems where the control is expressed as a probability distribution p over future trajectories given the current state and where the control cost can be written as a KullbackLeibler (KL) divergence between p and some interaction terms. The optimal control is given by minimizing the KL divergence, which is equivalent to solving a probabilistic inference problem in a dynamic Bayesian network. The optimal control is given in terms of (marginals of) a probability distribution over future trajectories. The formulation of the control problem as an inference problem directly suggests exact inference methods such as the Junction Tree method (JT) (Lauritzen and Spiegelhalter 1988) or a number of wellknown approximation methods, such as the variational method (Jordan 1999), belief propagation (BP) (Murphy et al. 1999), the cluster variation method (CVM) or generalized belief propagation (GBP) (Yedidia et al. 2001) or Markov Chain Monte Carlo (MCMC) sampling methods. We refer to this class of problems as KL control problems.
The class of control problems considered in this paper is identical as in Todorov (2007, 2008, 2009), who shows that the Bellman equation can be written as a KL divergence of probability distributions between two adjacent time slices and that the Bellman equation computes backward messages in a chain as if it were an inference problem. The novel contribution of the present paper is to identify the control cost with a KL divergence instead of making this identification in the Bellman equation. The immediate consequence is that the optimal control problem is identical to a graphical model inference problem that can be approximated using standard methods.
We also show how KL control reduces to the previously proposed path integral control problem (Kappen 2005) when noise is Gaussian in the limit of continuous space and time. This class of control problem has been applied to multiagent problems using a graphical model formulation and junction tree inference in Wiegerinck et al. (2006, 2007) and approximate inference in van den Broek et al. (2008a, 2008b). In robotics, Theodorou et al. (2009, 2010a, 2010b) has shown the path integral method has great potential for application. They have compared the path integral method with some stateoftheart reinforcement learning methods, showing very significant improvements. In addition, they have successful implemented the path integral control method to a walking robot dog. The path integral approach has recently been applied to the control of character animation (da Silva et al. 2009).
2 Control as KL minimization
Let x=1,…,N be a finite set of states, x ^{ t } denotes the state at time t. Denote by p ^{ t }(x ^{ t+1}x ^{ t },u ^{ t }) the Markov transition probability at time t under control u ^{ t } from state x ^{ t } to state x ^{ t+1}. Let p(x ^{1:T }x ^{0},u ^{0:T−1}) denote the probability to observe the trajectory x ^{1:T } given initial state x ^{0} and control trajectory u ^{0:T−1}.
The optimal control is normally computed using the Bellman equation, which results from a dynamic programming argument (Bertsekas and Tsitsiklis 1996). Instead, we will consider the restricted class of control problems for which C in (1) can be written as a KL divergence. As a particular case, we consider that \(\hat{R}\) is the sum of a control dependent term and a state dependent term. We further assume the existence of a ‘free’ (uncontrolled) dynamics q ^{ t }(x ^{ t+1}x ^{ t }), which can be any first order Markov process that assigns zero probability to physically impossible state transitions.
The optimal cost, (6), is minus the log partition sum and is the expectation value of the exponentiated state costs \(\sum_{t=0}^{T}R(x^{t},t)\) under the uncontrolled dynamics q. This is a surprising result, because it means that we have a closed form solution for the optimal costtogo C(x ^{0},p) in terms of the known quantities q and R.
Equations (8) and (10) define a stochastic optimal control problem. The solution for the optimal costtogo for this class of control problems can be shown to be given as a socalled path integral, an integral over trajectories, which is the continuous time equivalent of the sum over trajectories in (6). Note, that the cost of control is quadratic in u, but of a particular form with the matrix ν ^{−1} in agreement with Kappen (2005). Thus, the KL control theory contains the path integral control method as a particular limit. As is shown in Kappen (2005), this class of problems admits a solution of the optimal costtogo as an integral over paths, which is similar to (6).
2.1 Graphical model inference

The uncontrolled dynamics factorizes over components$$q^t(x^{t+1}x^t)=\prod_{i=1}^n q_i^t(x^{t+1}_i x^t_i).$$

The interaction between components has a (sparse) graphical structure R(x,t)=∑_{ α } R _{ α }(x _{ α },t) with α a subset of the indices 1,…,n and x _{ α } the corresponding variables.
Thus, ψ in (4) has a graphical structure that we can exploit when computing the marginals in (7). For instance, one may use the junction tree (JT) method, which can be more efficient than simply using the backward messages. Alternatively, we can use any of a large number of approximate graphical model inference methods to compute the optimal control. In the following sections, we will illustrate this idea by applying several approximate inference algorithms in two different tasks.
3 Stacking blocks (KLblocksworld)
3.1 Numerical results
In the next section, we consider two particular problems. First, we are interested in finding a sequence of actions that, starting in a given initial state x ^{0}, reach a given goal state x ^{ T }, without state cost. Then we consider the case of entropy minimization, with no defined goal state and nonzero state cost.
3.1.1 Goal state and λ=0
To find the KL control we first condition the model both on the initial state and the final state variables by “clamping” all variables x ^{1} and x ^{ T }. The KL control solution is obtained by computing for t=1,…,T the marginal p(k ^{ t },l ^{ t }x ^{ t−1}). In this case, we can find the exact solution via the junction tree (JT) algorithm (Lauritzen and Spiegelhalter 1988; Mooij 2010). The k ^{ t },l ^{ t } is obtained by taking the MAP state of p(k ^{ t },l ^{ t }x ^{ t−1}) breaking ties at random, which results in a new state x _{ t }.
These probabilities p(k ^{ t },l ^{ t }x ^{ t−1}) are shown in Fig. 4b. Notice that the symmetry in the problem is captured in the optimal control, which assigns equal probability when moving the first block to left or right (Fig. 4b, c, t=1). Figure 4d shows the strategy resulting from the MAP estimate, which first unpacks the tower at position 1 leaving all four locations with one block at t=4, and then rebuilds it again at the goal position 3.
For larger instances, the JT method is not feasible because of too large tree widths. For instance, to stack 4 blocks on 6 locations within a horizon of 11, the junction tree has a maximal width of 12, requiring about 15 Gbytes of memory. We can nevertheless obtain approximate solutions using different approximate inference methods. In this work, we use the belief propagation algorithm (BP) and a generalization known as the Cluster Variation method (CVM). We briefly summarize the main idea of the CVM method in Appendix C. We use the minimal cluster size, that is, the outer clusters are equal to the interaction potentials ψ as shown in the graphical model Fig. 3.
3.1.2 No goal state and λ>0: entropy minimization
The computation time was approximately 1 hour per t iteration and memory use was approximately 27 Mb. This instance was too large to obtain exact results. We conclude that, although the CPU time is large, the CVM method is capable to yield an apparently accurate control solution for this large instance.
4 Multi Agent cooperative game (KLstaghunt)
In this section we consider a variant of the stag hunt game, a prototype game of social conflict between personal risk and mutual benefit (Skyrms 2004). The original twoplayer stag hunt game proceeds as follows: there are two hunters and each of them can choose between hunting hare or hunting stag, without knowing in advance the choice of the other hunter. The hunters can catch a hare on their own, giving them a small reward. The stag has a much larger reward, but it requires both hunters to cooperate in catching it.
Twoplayer stag hunt payoff matrix example: rows and columns indicate actions of one and the other player respectively. The payoff describes the reward for each hunter. For instance, if both go for the stag, they both get a reward of 3. If one hunter goes for the stag and the other for the hare, they get a reward of 0 and 1 respectively
Stag 
Hare  

Stag 
3,3 
0,1 
Hare 
1,0 
1,1 
We define the KLstaghunt game as a multiagent version of the original stag hunt game where M agents live in a grid of N locations and can move to adjacent locations on the grid. The grid also contains H hares and S stags at certain fixed locations. Two agents can cooperate and catch a stag together with a high payoff R _{ s }. Catching a stag with more than two agents is also possible, but it does not increase the payoff. The agents can also catch a hare individually, obtaining a lower payoff R _{ h }. The game is played for a finite time T and at each timestep all the agents perform an action. The optimal strategy is thus to coordinate pairs of agents to go for different stags.
For high values of λ (left plot), each hunter catches one of the hares. In this case, the cost function is dominated by KL term. For small enough values of λ (right plot), both hunters cooperate to catch the stag. In this case, the state cost, function R(x ^{ T }), governs the optimal control cost. Thus λ can be seen as a parameter that controls whether the optimal strategy is risk dominant or payoff dominant.
Note that computing the exact solution using this procedure becomes infeasible even for small number of agents, since the joint state space scales as N ^{ M }. In the next section, we show a more efficient representation using a factor graph for which approximate inference is tractable.
4.1 Graphical model for the KLstaghunt game
 1.
 2.Second, we add factors \(\psi_{s_{j}}(x_{i}^{T},d_{i,j})\) for each stag j defined on each state variable \(x_{i}^{T}\) and new introduced binary variables d _{ i,j }=0,1. These factors evaluate to one when variable d _{ i,j } takes the value of a Kronecker δ of the agent’s state \(x_{i}^{T}\) and the position of a stag s _{ j }, and zero otherwise:
 3.
 4.
The extended factor graph is tractable since it involves factors of no more than three variables of small cardinality. Note that this transformation can also be applied if additional state costs are incorporated at each timestep ψ _{ R }(x ^{ t })≠0,t=1,…,T. However, such a representation is not of practical interest, since it complicates the model unnecessarily.
Finally, note that the treewidth of the extended graph still grows fast with the number of agents M because variables d _{ i,j } and u _{ i,j } are coupled. Thus, exact inference using the JT algorithm is still possible on small instances only.
4.2 Approximate inference of the KLstaghunt problem
In this section we analyze large systems for which exact inference is not possible using the JT algorithm. The belief propagation (BP) algorithm is an alternative approximate algorithm that we can run on the previously described extended factor graph.
We use the following setup: for a fixed number of agents M, we set the number of stags H=2M and the number of hares \(S=\frac{M}{2}\). Their locations, as well as the initial states x ^{1} are chosen randomly and nonoverlapping. We then construct a factor graph with initial states “clamped” to x ^{1} and build instancedependent factors \(\psi_{s_{j}}\) and \(\psi_{h_{k}}\). We run BP using sequential updates of the messages. If BP converges in less than 500 iterations, the optimal trajectories of the agents are computed using the estimated marginals (factor beliefs) for ψ _{ q }(x ^{ t+1}x ^{ t }) after convergence. Starting from x ^{1}, we select the next state according to the most probable state from \(p_{BP}(x_{i}^{t+1}x_{i}^{t})\) until the end time. We break ties randomly. We analyze the system as a function of parameter λ for a several number of realizations.
To characterize the solutions, we compute the approximated expected cost as in (6), that is −logZ _{ BP }. We observe that for large systems that quantity changes abruptly at λ≈1. Qualitatively, the optimal control obtained on the boundary between riskdominant and payoffdominant strategies differs maximally between individual instances and strongly depends on the initial configuration. This suggests a phase transition phenomenon typical of complex physical systems, in this case separating the two kind of optimal behaviors, where λ plays the role of a “temperature” parameter.
5 Related work
The idea to treat a control problem as an inference problem has a long history. The best known example is the linear quadratic control problem, which is mathematically equivalent to an inference problem and can be solved as a Kalman smoothing problem (Stengel 1994). The key insight is that the value function that is iterated in the Bellman equation becomes the (log of the) backward message in the Kalman filter. The exponential relation was generalized in Kappen (2005) for the nonlinear continuous space and time (Gaussian case) and in Todorov (2007) for a class of discrete problems.
There is a line of research on how to compute optimal action sequences in influence diagrams using the idea of probabilistic inference (Cooper 1988; Tatman and Shachter 1990; Shachter and Peot 1992). Although this technique can be implemented efficiently using the junction tree approach for single decisions, the approach does not generalize in an efficient way to optimal decisions, in the expectedreward sense, in multistep tasks. The reason is that the order in which one marginalizes and optimizes strongly affects the efficiency of the computation. For a Markov decision process (MDP) there is an efficient solution in terms of the Bellman equation.^{1} For a general influence diagram, the marginalization approach as proposed in Cooper (1988), Tatman and Shachter (1990), Shachter and Peot (1992) will result in an intractable optimization problem over u ^{0:T−1} that cannot be solved efficiently (using dynamic programming), unless the influence diagram has an MDP structure.
The KL control theory shares similarities with work in reinforcement learning for policy updating. The notion of KL divergence appears naturally in the work of Bagnell and Schneider (2003) who proposes an information geometric approach to compute the natural policy gradient (for small step sizes). This idea is further developed into an ExpectationMaximization (EM) type algorithm (Dayan and Hinton 1997) in recent work (Peters et al. 2010; Kober and Peters 2011) using a relative entropy term. The KL divergence acts here as a regularization that weights the relative dependence of the new policy on the data observed and the old policy, respectively.
It is interesting to compare the notion of free energy in continuoustime dynamical systems with Gaussian noise considered in Friston et al. (2009) with the path integral formalism of Kappen (2005), which is a special case of KL control theory. Friston et al. (2009) advocate the optimization of free energy as a guiding principle to describe behavior of agents. The main difference between the KL control theory and Friston’s free energy principle is that in KL control theory, the KL divergence plays the role of an expected future cost and its optimization yields a (time dependent) optimal control trajectory, whereas Friston’s free energy computes a control that yields a timeindependent equilibrium distribution, corresponding to the minimal free energy. Friston’s free energy formulation is obtained as a special case of KL control theory when the dynamics and the reward/cost is timeindependent and the horizon time is infinite.

The TS approach is more general than the KL approach, in the sense that the reward considered in TS is an arbitrary function of state and action R(x,u), whereas the reward considered in KL is a sum of a state dependent term R(x) and a KL divergence.

The KL approach is significantly more efficient than the TS approach. In the TS approach, the backward messages are computed for a fixed policy π (Estep), from which an improved policy is computed (Mstep). This procedure is iterated until convergence. In the KL approach, the backward messages give the optimal control directly, with no further need for iteration.

In addition, the KL approach is more efficient than the TS approach for timedependent problems. Using the TS approach for timedependent problems makes the computation a factor T more timeconsuming than for the timeindependent case, since all mixture components must be computed. The complexity of the KL control approach does not depend on whether the problem is timedependent or not.

The TS and KL approach optimize with respect to a different quantity. The TS approach writes the state transition p(yx)=∑_{ u } p(yx,u)π(ux) and optimizes with respect to π. The KL approach optimizes the state transition probability p(yx) directly either as a table or in a parametrized way.
6 Discussion
In this paper, we have shown the equivalence of a class of stochastic optimal control problems to a graphical model inference problem. As a result, exact or approximate inference methods can directly be applied to the intractable stochastic control computation. The class of KL control problems contains interesting special cases such as the continuous nonlinear Gaussian stochastic control problems introduced in Kappen (2005), discrete planning tasks and multiagent games, as illustrated in this paper.
We notice, that there exist many stochastic control problems that are outside of this class. In the basic formulation of (1), one can construct control problems where the functional form of the controlled dynamics p ^{ t }(x ^{ t+1}x ^{ t },u ^{ t }) is given as well as the cost of control R(x ^{ t },u ^{ t },x ^{ t+1},t). In general, there may then not exist a q ^{ t }(x ^{ t+1}x ^{ t }) such that (2) holds.
In this paper, we have considered the model based case only. The extension to the model free case would require a sampling based procedure. See Bierkens and Kappen (2012) for initial work in this direction.
We have demonstrated the effectiveness of approximate inference methods to compute the approximate control in a block stacking task and a multiagent cooperative task.
For the KLblocksworld, we have shown that an entropy minimization task is more challenging than stacking blocks at a fixed location (goal state), because the control computation needs to find out where the optimal location is. Standard BP does not give any useful results if no goal state was specified, but apparently good optimal control solutions were obtained using generalized belief propagation (CVM). We found that the marginal computation using CVM is quite difficult compared to other problems that have been studied in the past (Albers et al. 2007), in the sense that relatively many inner loop iterations were required for convergence. One can improve the CVM accuracy, if needed, by considering larger clusters (Yedidia et al. 2005) as has been demonstrated in other contexts (Albers et al. 2006), at the cost of more computational complexity.
We have given evidence that the KL control formulation is particularly attractive for multiagent problems, where q naturally factorizes over agents and where interaction results from the fact that the reward depends on the state of more than one agent. A first step in this direction was already made in Wiegerinck et al. (2006), van den Broek et al. (2008a). In this case, we have considered the KLstaghunt game and shown that BP provides a good approximation and allows to analyze the behavior of large systems, where exact inference is not feasible.
We found that, if the game setting strongly penalizes large deviations from the baseline (random) policy, the coordinated solution is suboptimal. That means that the optimal solution distributes the agents among the different hares rather than bringing them jointly to the stags (riskdominant regime). On the contrary, if the agents are not constrained by deviating too much from the baseline policy to maximize 〈R〉, the coordinated solution becomes optimal (payoff dominant regime). We believe that this is an interesting result, since it provides a explanation of the emergence of cooperation in terms of an effective temperature parameter λ.
Here we mean by efficient, that the sum or min over a sequence of states or actions can be performed as a sequence of sums or mins over states.
When g is not a square matrix (when the number of controls is less than the dimension of x), g ^{−1} denotes the pseudoinverse of g. For any u, the pseudoinverse has the property that g ^{−1} gu=u.
Acknowledgements
We would like to thank anonymous reviewers for helping on improving the manuscript, Kees Albers for making available his sparse CVM code, Joris Mooij for making available the libDAI software and Stijn Tonk for useful discussions. The work was supported in part by the ICIS/BSIK consortium.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.