Generalized Thompson sampling for sequential decision-making and causal inference

Abstract

Purpose

Sampling an action according to the probability that the action is believed to be the optimal one is sometimes called Thompson sampling.

Methods

Although mostly applied to bandit problems, Thompson sampling can also be used to solve sequential adaptive control problems, when the optimal policy is known for each possible environment. The predictive distribution over actions can then be constructed by a Bayesian superposition of the policies weighted by their posterior probability of being optimal.

Results

Here we discuss two important features of this approach. First, we show in how far such generalized Thompson sampling can be regarded as an optimal strategy under limited information processing capabilities that constrain the sampling complexity of the decision-making process. Second, we show how such Thompson sampling can be extended to solve causal inference problems when interacting with an environment in a sequential fashion.

Conclusion

In summary, our results suggest that Thompson sampling might not merely be a useful heuristic, but a principled method to address problems of adaptive sequential decision-making and causal inference.

Background

In a research paper from 1933, Thompson studied the problem of finding out which one of two drugs was better when testing them on a patient population under the constraint that as few people as possible should be subjected to the inferior drug during the course of testing (Thompson [1933]). Given a current (Bayesian) probability estimate P of one treatment being better than the other, he suggested that it might be a good idea to adjust the proportions of future test subjects that take the two drugs to the respective probabilities P and 1-P. This way one would not run into the danger of permanently cutting off all future test subjects from a potentially superior treatment that so far seems inferior due to statistical fluctuations, while only temporarily risking exposure to a potentially inferior drug for a smaller proportion of the population. Randomizing actions based on the probability that this action is believed to be optimal when faced with an unknown environment is now sometimes called Thompson sampling.

Today, Thompson’s problem is generally thought of as a bandit problem that consists in determining which lever to pull at which point in time when facing a set of one-armed slot machines, each one having an unknown distribution over a reward variable (Russell and Norvig [1995]; Sutton and Barto [1998]). In the case of known prior probabilities and geometrically discounted future rewards, Gittins ([1979]) provides an optimal policy for the bandit problem that maximizes the expected future cumulative discounted reward. In contrast, Thompson sampling is usually considered as a heuristic approach to solve bandit problems (Wyatt [1997]; Granmo [2008, 2010]; Asmuth et al. [2009]; Graepel et al. [2010]; Scott [2010]; May and Leslie [2011]; Chapelle and Li [2011]; Agrawal and Goyal [2011]; Granmo and Glimsdal [2013]; May et al. [2012]; Kaufmann et al. [2012]; Russo and Roy [2013]; Korda et al. [2013]; Bubeck and Liu [2013]). However, the basic idea of Thompson sampling—that is, sampling actions from a mixture distribution of policies according to their probability of being optimal–can also be applied to solve more general problems in sequential adaptive control (Dearden et al. [1998]; Strens [2000]; Ortega and Braun [2010a, 2010b, 2012a]; Braun and Ortega [2010]; Osband and Russo [2013]; Cao and Ray [2012]; Tziortziotis et al. [2013a, 2013];x Dimitrakakis [2013]; Dimitrakakis and Tziortziotis [2013]; Mellor and Shapiro [2013]).

In this paper, our aim is to discuss some of the basic properties of Thompson sampling as a modeling approach. In particular, we want to argue that

• 1. Thompson sampling can be considered as an application of Bayes’ rule for acting where actions are treated as causally intervened random variables within the framework of statistical causality.

• 2. Thompson sampling can be considered as a form of optimal adaptive control under bounded rationality where limited information processing capabilities are modeled by entropic search costs.

• 3. Thompson sampling provides a natural strategy for causal induction when interacting with an environment with unknown causal structure.

Although the third section contains an algorithmic extension to previous work (Ortega and Braun [2010a]; [2010b]), it should be emphasized that the main contribution of the paper is not so much to present a novel algorithm, but to discuss basic properties of Thompson sampling, in particular how it relates to the information-theoretic bounded rationality model in (Ortega and Braun [2013]), how this boundedness can be interpreted in terms of sampling complexity, and how this method can be applied to solve problems of causal inference.

The paper is structured as follows. In Section “Problem statement” we clarify the problem statement and recapitulate the main result of (Ortega and Braun [2010b]). In Section “Decision-making with limited resources” we analyze the decision-making problem faced by agents that are unable to compute the single best policy. In Section “Causal induction” we investigate how this approach can be applied to adaptive agents that need to discover the causal structure of their environment. Finally, we discuss the significance of these results in Section “Discussion”.

Methods

Problem statement

In an adaptive control problem a decision-maker faces an environment Q _θ drawn from a set of potential environments $\mathcal{Q}=\left\{Q_{\theta }\right|\theta \in \Theta \}$. In general θ could be a continuous variable, but we restrict our exposition to the discrete case. Each environment Q _θ can be characterized by a set of conditional distributions Q(o _t |θ,a_≤t,o_<t) that indicate the probability of observing o _t given past observations o_<t=o₁…o_t-1 and past actions a_≤t=a₁…a _t . This class of environments is very general, and it encompasses multi-armed bandits, (partially observable) Markov decision processes and others –compare Chapter 3 (Legg [2008]). To allow for self-optimizing agents, the environment is typically assumed to be ergodic, so agents can recover from their mistakes –compare Section 3.5 (Legg [2008]). The decision-maker has perfectly fitting prediction models P(o _t |θ,a_≤t,o_<t)=Q(o _t |θ,a_≤t,o_<t), but is uncertain about θ. The uncertainty about θ can be represented by a prior distribution P(θ). The interaction proceeds as follows. First an environment θ is sampled from P(θ). The agent picks an action a₀ and receives an observation o₀, to which the agent responds with a₁ and receives observation o₁ etc. The agent’s policy can be described by a set of conditional distributions P(a _t |o_<t,a_<t).

Problem statement: decision-theory

In order to solve the problem within the framework of maximum expected utility theory, one requires

• a prior P(θ) over possible environments Q _θ

• a class of prediction models P(o _t |θ,a_≤t,o_<t)

• a utility function U(o_≤T,a_≤T).

Then one can reduce the problem of the unknown environment to a problem with known environment. Such a “known” environment can be created from the set of possible environments by marginalizing over the parameter of the possible environments, thus, obtaining the Bayesian mixture distribution, wherez

$$P\left(o_t\right|o_{<t},a_{\le t})=\sum _{\theta }P(\theta |o_{<t},a_{\le t})P\left(o_t\right|\theta ,o_{<t},a_{\le t}).$$

(1)

The adaptive control problem is then solved by finding the optimal policy $p\left(a_t\right|a_{<t},o_{<t})=\delta (a_t-a_t^{\ast })$, where

$$\begin{array}{ll}{a}_t^{\ast }& =\underset{a_t}{arg\; max}\sum _{o_t}P\left(o_t\right|o_{<t},a_{\le t}\left)\underset{a_{t+1}}{max}\sum _{o_{t+1}}P\right(o_{t+1}|o_{\le t},a_{\le t+1})\cdots \\ \cdots \underset{a_T}{max}\sum _{o_T}P\left(o_T\right|o_{<T},a_{\le T}\left)U\right(o_{\le T},a_{\le T})\end{array}$$

(2)

maximizes the expected utility under the mixture distribution (Hutter [2004]). Equations (1) and (2) define a Bayesian adaptive control problem (Martin [1967]; Duff [2002]). This problem formulation becomes quickly intractable, as the number of reachable information states grows exponentially in the time horizon (Duff [2002]).

Problem statement: probability theory & statistical causality

Ignoring the notion of utility for a moment and treating actions purely as (causally intervened) random variables (Pearl [2000]), one could think of another kind of adaptive control problem that is defined entirely in probabilistic and causal terms. This requires the following ingredients

• a prior P(θ) over possible environments Q _θ

• a class of prediction models P(o _t |θ,a_≤t,o_<t)

• a class of policy models P(a _t |θ,a_<t,o_<t)

such that for every possible environment indexed by θ there is a perfectly fitting predictor P(o _t |θ,a_≤t,o_<t)=Q _θ (o _t |a_≤t,o_<t) and a desirable custom-built^a policy P(a _t |θ,a_<t,o_<t). The problem statement is: What is the next action a _t given the uncertainty over θ and given the history of previous actions ${â}_{<t}$ and previous observations o_<t? As for any random variable, answering this question for the random variable a _t simply requires computing the predictive distribution conditioned on the past, that is

$$P\left(a_t\right|{â}_{<t},o_{<t})=\sum _{\theta }P(a_t|\theta ,{â}_{<t},o_{<t})P\left(\theta \right|{â}_{<t},o_{<t}).$$

(3)

As there can be only one action at any one time, single actions can be obtained as samples from $P\left(a_t\right|{â}_{<t},o_{<t})$. Importantly, sampling from $P\left(a_t\right|{â}_{<t},o_{<t})$ is equivalent to first sampling a random belief θ from the posterior $P\left(\theta \right|{â}_{<t},o_{<t})$ and then sampling an action from $P\left(a_t\right|\theta ,{â}_{<t},o_{<t})$. This componentwise sampling from a mixture distribution is known as hierarchical sampling and corresponds here to a generalized Thompson sampling procedure, where first a random belief is sampled and then the associated policy with respect to this belief is executed. If we assume now that each of the custom-built policies is optimal in their respective environments, we effectively select an action according to the probability that it is the optimal action, because we first sample the environment θ according to its posterior probability of being the true environment and then we perform the policy that is optimal in that environment. The question is how this problem formulation can be reconciled with a decision-theoretic problem statement that involves utilities. This is the topic of Section “Decision-making with limited resources”.

Statistical causality

While both actions and observations are treated as random variables in (3), there is an important difference between actions and observations. Observations are produced by the environment and can be used to update the agent’s state of knowledge about the environment. In contrast, actions are set by the agent itself and hence they do not provide information about the environment. This distinction becomes crucial when conditioning on the history of actions and observations. The theory that deals with the distinction between exogenous and endogenous information is statistical causality (Pearl [2000]; Glymour et al. [2000]).

What is a causal intervention?

A typical example for causal intervention is the manipulation of a barometer (Pearl [2000]). In a barometer the atmospheric pressure changes the height of the mercury: if it rises, we expect good weather; and if it drops rapidly, we expect rain. A simple Bayesian model captures this relation:

$$P\left(w\right|b)=\frac{P\left(b\right|w\left)P\right(w)}{P\left(b\right)},$$

where w and b are the weather and the Barometer variables respectively, P(w) is the prior probability of the weather (e.g. good or bad) and P(b|w) is the likelihood of the barometer change given the weather. The posterior P(w|b) allows us to infer the weather from the barometer reading.

Now, imagine you decide to change the level of the mercury yourself, say (using a bit of imagination) by means of a pressurizing device. Now, you set the value of the random variable—and intuition tells us that we cannot predict the weather anymore from the barometer reading. Apparently, our previous Bayesian model is useless now. This shouldn’t come as a surprise, as our intervention effectively changed the relation between the barometer and the weather.

Mathematically, we can model the causal relationship between different random variables by a particular factorization of the joint probability distribution into conditional probabilities reflecting generative mechanisms, where causes are in the conditional and effects in the argument. In our example the weather causes the barometer to rise and fall and not the other way around, that is in causal terms we have P(b,w)=P(b|w)P(w) and not P(b,w)=P(w|b)P(b), even though in purely probabilistic terms the two factorizations are of course equivalent. The causal factorization becomes important when modifying the causal relationships by intervention, as in our example. Manipulating the barometer setting directly severs the causal between weather and barometer. Consequently, we have to modify our joint probability distribution from P(b,w) to

$$P(\widehat{b},w)=\delta \left(b\right)P\left(w\right),$$

that is, where P(b|w), viewed as a generative mechanism, has been replaced by δ(b), thereby rendering the two random variables independent. The hat-notation (due to (Pearl [2000])) is just a shorthand referring to this particular transformation of the probability distribution. When we evaluate the posterior under the intervention $\widehat{b}$, we get

$$P\left(w\right|\widehat{b})=\frac{P(w,\widehat{b})}{\sum _{w}P(w,\widehat{b})}=\frac{1·P\left(w\right)}{1}=P(w).$$

In other words, we don’t gain knowledge about the weather—as expected. Notice that intervening the alternative factorization, $P(\widehat{b},w)=\delta \left(b\right)P\left(w\right|b)$, would give a different result that is inconsistent with our causal story: we have assumed that the mercury level of the barometer depends functionally on the weather, and not the other way around. The reason for this special treatment of actions is that when we set the value of a random variable ourselves, we change Nature’s probability law.

Causal interventions in Thompson sampling

To calculate the effect of an intervention, the causal model has to be known, that is the unique factorization of the joint distribution into conditional probabilities matching the causal dependencies over the random variables. In our case, the causal dependencies are straightforward: first, the environment secretly chooses a true parameter θ^∗∈Θ, and then the interactions a₁,o₁,a₂,o₂,… follow chronologically. This causal model allows us to study the effect of interventions in the problem statement given in (3). We start by re-expressing the posterior $P\left(\theta \right|{â}_{<t},o_{<t})$ as

$$\begin{array}{l}P\left(\theta \right|{â}_{<t},o_{<t})\\ \stackrel{\left(1.\right)}{=}\frac{P(\theta ,{â}_{<t},o_{<t})}{\sum _{{\theta }^{\prime }}P({\theta }^{\prime },{â}_{<t},o_{<t})}\\ \stackrel{\left(2.\right)}{=}\frac{P\left(\theta \right)\prod _{k=1}^{t}P\left({â}_k\right|\theta ,{â}_{<k},o_{<k}\left)P\right(o_k|\theta ,{â}_{\le k},o_{<k})}{\sum _{{\theta }^{\prime }}P\left({\theta }^{\prime }\right)\prod _{k=1}^{t}P\left({â}_k\right|{\theta }^{\prime },{â}_{<k},o_{<k}\left)P\right(o_k|{\theta }^{\prime },{â}_{\le k},o_{<k})}\\ \stackrel{\left(3.\right)}{=}\frac{P\left(\theta \right)\prod _{k=1}^{t}P\left({â}_k\right|\theta ,a_{<k},o_{<k}\left)P\right(o_k|\theta ,a_{\le k},o_{<k})}{\sum _{{\theta }^{\prime }}P\left({\theta }^{\prime }\right)\prod _{k=1}^{t}P\left({â}_k\right|{\theta }^{\prime },a_{<k},o_{<k}\left)P\right(o_k|{\theta }^{\prime },a_{\le k},o_{<k})}\\ \stackrel{\left(4.\right)}{=}\frac{P\left(\theta \right)\prod _{k=1}^{t}P\left(o_k\right|\theta ,a_{\le k},o_{<k})}{\sum _{{\theta }^{\prime }}P\left({\theta }^{\prime }\right)\prod _{k=1}^{t}P\left(o_k\right|{\theta }^{\prime },a_{\le k},o_{<k})},\end{array}$$

(4)

where we first expand the probabilities in terms of the joint distribution, second rewrite the joint distribution as the causal factorization, third remove the intervention tags from the intervened random variables that are in the probability conditions (Pearl’s second rule of do-calculus (Pearl [2000])), and fourth replace each conditional probability having an intervened variable in the argument by a delta function over its chosen value—compare Chapter 4.2 in (Pearl [2000]).

These equations show that beliefs are updated only using past observations, and that past actions provide no further evidence. Intuitively, the reason for this is that the agent can be surprised about his past observations and learn from them, but he cannot be surprised about his own actions chosen by himself in the past. Also, due to Pearl’s second law of do-calculus (Pearl [2000]) we have

$$P\left(a_t\right|\theta ,{â}_{<t},o_{<t})=P(a_t|\theta ,a_{<t},o_{<t}).$$

(5)

Using (3), (4) and (5), we arrive at the desired result, that is the predictive distribution for $a_t\in \mathcal{A}$ given by

$$P\left(a_t\right|{â}_{<t},o_{<t})=\sum _{\theta }P(a_t|\theta ,a_{<t},o_{<t})P\left(\theta \right|{â}_{<t},o_{<t}),$$

(6)

obtained only by applying probability theory and causal calculus.

Results

Decision-making with limited resources

The difference between the random belief model supposed by Thompson sampling and the standard maximum expected utility model for decision-making is highlighted by contrasting two simple decision scenarios depicted in Figure 1. The goal is to predict the outcome when throwing one of two possible biased coins. A rational decision maker places bets (shown inside speech bubbles) such that his subjective expected utility is maximized. These subjective beliefs are delimited within dotted boxes. A Thompson sampling agent first samples a random belief and then chooses the best prediction with respect to this belief.

Figure 1

Comparison between (a) a fully rational decision maker and (b) a decision maker with random beliefs. There are two biased coins. The probability of throwing the first coin is $\frac{1}{4}$, the probability of throwing the second coin is $\frac{3}{4}$. The first coin has a probability of $\frac{1}{4}$ for the outcome Head and the second coin has a probability of $\frac{3}{4}$ for Head. The decision-maker has to predict the most likely outcome. A correct guess is rewarded with $1, an incorrect guess yields $0.

The difference between the two becomes clear by inspecting the expected utility in each case: they are

$$\begin{array}{ll}\underset{P^{\prime }}{max}\sum _{\theta }P\left(\theta \right)\left\{\sum _o{P}^{\prime }\left(a\right|\theta \left)P\right(o|\theta ,a)U\left(o\right)\right\},& \text{(a)}\\ \text{and}& \sum _{\theta }P\left(\theta \right)\underset{P^{\prime }}{max}\left\{\sum _o{P}^{\prime }\left(a\right|\theta \left)P\right(o|\theta ,a)U\left(o\right)\right\}& \text{(b)}\end{array}$$

respectively, where the labels (a) and (b) correspond to the labels in Figure 1. Here it is clearly seen that the difference between the two lies in the order in which we apply the expectation (over the environment parameter) and the maximization operator. It should also be noted that the expected utility of (a) is an upper bound on the expected utility of (b). Yet, both cases can constitute optimal decisions depending on constraints. In (a), the decision-maker picks his action taking into account the uncertainty over the bias, while in (b), the decision-maker picks his action only after his beliefs over the coin bias are instantiated—that is, he is optimal w.r.t. his random beliefs. Here we consider how this optimality w.r.t. random beliefs can be considered as a form of optimal decision-making under information processing constraints.

Modeling bounded rational decision-making

Here we consider a particular information-theoretic model of bounded rational decision-making that formalizes limited information processing resources by a variational principle that trades off expected utility gains (or losses) and entropic information costs (Ortega [2011a]; Ortega and Braun [2011, 2012a, 2013]). Information processing costs are usually ignored in the study of perfectly rational decision-makers. Given a choice set $\mathcal{X}$ with choices $x\in \mathcal{X}$ and utilities U(x), a perfectly rational decision-maker would always choose the best option x^∗= arg maxx U(x)—presupposing there is a unique maximum. In general, a bounded rational decision-maker is unable to pick out the best option with certainty, and his choice can be described by a probability distribution P(x) reflecting uncertainty. Improving the choice strategy P(x) can be understood as a costly search process.

Let us assume the initial strategy of the decision-maker can be described by a probability distribution P₀(x). The search process for the optimum transforms this initial choice into a final choice P(x). In case of the perfectly rational decision-maker the final choice is $P\left(x\right)={\delta }_{x,x^{\ast }}$. In the general case of the bounded rational decision-maker the search is costly and he will not be able to afford such a stark reduction in uncertainty. Assuming that search costs are real-valued, additive and higher for rare events (Ortega and Braun [2010c]), it can be shown that the cost of the search is determined by the information distance D _{K L} between P₀ and P, that is $D_{\mathit{\text{KL}}}=\sum _{x}P\left(x\right)log\frac{P\left(x\right)}{P_0\left(x\right)}$. Both Bayesian search (Jaynes [1985]) and Koopman’s random search (Stone [1998]) are compatible with these assumptions, as well as energetic costs that would have to be paid by a Maxwellian demon for reducing uncertainty in statistical physical systems (Ortega and Braun [2013]). How this information-theoretic model of search costs relates to computational resources such as space and time complexity is still an open problem (Vitanyi [2005]).

Simple decisions

The decision process is modeled as a transformation of a prior choice probability P₀ into a posterior choice probability P by taking into account the utility gains (or losses) and the transformation costs arising from information processing, such that

$$P=arg\underset{\tilde{P}}{max}\left\{\sum _x\tilde{P}\left(x\right)U\left(x\right)-\frac{1}{\alpha }\sum _x\tilde{P}\left(x\right)log\frac{\tilde{P}\left(x\right)}{P_0\left(x\right)}\right\},$$

(7)

where the $x\in \mathcal{X}$ are the possible outcomes, P₀(x) are their prior probablities, U(x) are their utilities. The inverse temperature α≥0 can be regarded as a rationality parameter that translates the cost of information processing measured in units of information into units of utility. If the limits in information processing capabilities are given as a constraint D _{K L} (P||P₀)≤K with some positive constant K, then α is determined as a Lagrange multiplier. The maximizing distribution $\tilde{P}=P$ is the equilibrium distribution

$$P\left(x\right)=\frac{1}{Z_{\alpha }}{P}_0\left(x\right)e^{\mathrm{\alpha U}\left(x\right)},\text{where}{Z}_{\alpha }=\sum _x{P}_0\left(x\right)e^{\mathrm{\alpha U}\left(x\right)},$$

(8)

and represents the choice probabilities after deliberation—see Theorem 1.1.3 (Keller [1998]) and (Ortega and Braun [2013]) for a proof. The value V of the choice set $\mathcal{X}$ under choice probabilities P can be determined from the same variational principle

$$\begin{array}{ll}V\left[P\right]& =\underset{\tilde{P}}{max}\left\{\sum _x\tilde{P}\left(x\right)U\left(x\right)-\frac{1}{\alpha }\sum _x\tilde{P}\left(x\right)log\frac{\tilde{P}\left(x\right)}{P_0\left(x\right)}\right\}\hfill \\ =\frac{1}{\alpha }log\left(\sum _x{P}_0\left(x\right)e^{\mathrm{\alpha U}\left(x\right)}\right)=\frac{1}{\alpha }log{Z}_{\alpha }.\hfill \end{array}$$

(9)

For the two different limits of α, the value and the equilibrium distribution take the asymptotic forms

$$\begin{array}{lllllll}\alpha & \to +\infty & \frac{1}{\alpha }log{Z}_{\alpha }& =\underset{x}{max}U\left(x\right)& P\left(x\right)& ={\mathcal{U}}_{max}\left(x\right)& \text{(perfectly rational)}\\ \alpha & \to 0& \frac{1}{\alpha }log{Z}_{\alpha }& =\sum _x{P}_0\left(x\right)U\left(x\right)& P\left(x\right)& =P_0\left(x\right)& \text{(irrational)}\end{array}$$

where ${\mathcal{U}}_{max}$ is the uniform distribution over the maximizing subset ${\mathcal{X}}_{max}:=\{x\in \mathcal{X}:U(x)=\underset{x^{\prime }}{max}U(x^{\prime }\left)\right\}$. It can be seen that a perfectly rational agent with α→∞ is able to pick out the optimal action—which is a deterministic policy in the case of a single optimum—, whereas finitely rational agents have stochastic policies with non-zero probability of picking a sub-optimal action.

The model of bounded rational decision-making also lends itself to an interpretation in terms of sampling complexity. If we use a rejection sampling scheme to obtain samples from p(x) by first sampling from p₀(x), we can ask how many samples we will need on average from p₀ to obtain one sample from p. In this scheme, we produce a sample x∼p₀(x) and then decide whether to accept or reject the sample based on the criterion

$$u\le \frac{e^{\mathrm{\alpha U}\left(x\right)}}{e^{\mathrm{\alpha T}}},$$

(10)

where u is drawn from the uniform $\mathcal{U}[0;1]$ and T is the acceptance target value with T≥ maxx U(x). The equality holds for the most efficient sampler, but requires knowledge of the maximum. With this sampling scheme, the accepted samples will be distributed according to Equation (8). The average number of samples needed from p₀ to produce one sample of p is then

$$\overline{\mathrm{♯Samples}}=\frac{1}{\sum _x{p}_0\left(x\right)\frac{e^{\mathrm{\alpha U}\left(x\right)}}{e^{\mathrm{\alpha T}}}}=\frac{e^{\mathrm{\alpha T}}}{Z_{\alpha }}.$$

(11)

The important point about Equation (11) is that the average number of samples increases with increasing rationality parameter α. In fact, the average number of samples will grow exponentially for large α when T> maxx U(x), as

$$\frac{e^{\mathrm{\alpha T}}}{Z_{\alpha }}\underrightarrow{\stackrel{\alpha \to \infty }{}}\frac{e^{\alpha \left(T-U\left(x^{\ast }\right)\right)}}{P_0\left(x^{\ast }\right)},$$

where x^∗= arg maxU(x). It can also be straightforwardly seen that

$$\overline{\mathrm{♯Samples}}\ge e^{D_{\mathit{\text{KL}}}\left(p\left|\right|p_0\right)}=\frac{e^{\alpha \sum _{x}p\left(x\right)U\left(x\right)}}{Z_{\alpha }}$$

(12)

because $\sum _{x}p\left(x\right)U\left(x\right)\le T$, that is the exponential of the Kullback-Leibler divergence provides a lower bound on the average number of samples.

Decisions in the presence of latent variables

To model a Thompson sampling agent, we need at least a two-step decision with a variable x that has to be chosen by the agent, and a variable θ that is chosen by the environment. In the example described in Figure 1, the variable x is the agent’s prediction for the outcome of a coin toss, the variable θ indicates nature’s choice which one of the two coins is tossed. The agent’s prediction can take on the values x=H and x=T corresponding to the outcomes Head and Tail. The variable θ takes on the two values $\theta =\frac{1}{4}$ and $\theta =\frac{3}{4}$ corresponding to the biases of the two coins. The prior probability over θ is $p_0\left(\theta =\frac{1}{4}\right)=\frac{1}{4}$ and $p_0\left(\theta =\frac{3}{4}\right)=\frac{3}{4}$. The expected rewards for all combinations of x and θ are then $U\left(x=H,\theta =\frac{1}{4}\right)=\frac{1}{4}$, $U\left(x=T,\theta =\frac{1}{4}\right)=\frac{3}{4}$, $U\left(x=H,\theta =\frac{3}{4}\right)=\frac{3}{4}$ and $U\left(x=T,\theta =\frac{3}{4}\right)=\frac{1}{4}$.

In the case of two-step decisions, the variational problem can in general be formulated as a nested expression (Ortega and Braun [2011, 2012a, 2013])

$$\begin{array}{l}arg\underset{\tilde{p}(x,\theta )}{max}\sum _x\tilde{p}\left(x\right)\left[U\left(x\right)-\frac{1}{\alpha }log\frac{\tilde{p}\left(x\right)}{p_0\left(x\right)}+\sum _{\theta }\tilde{p}\left(\theta \right|x)\left[U(x,\theta )-\frac{1}{\beta }log\frac{\tilde{p}\left(\theta \right|x)}{p_0\left(\theta \right|x)}\right]\right].\end{array}$$

(13)

with the two different rationality parameters α and β for the two different variables x and θ. Limited information processing resources with respect to these variables can also be thought of as different degrees of control. For example, if α assumed a large value, the decision-maker could basically hand-pick a particular x, or if θ was determined by a coin toss that the agent cannot influence, we could model this by setting β to zero. The utility can in general depend on both action and observation variables. However, since the action by itself does not yield a reward in our case, we have U(x)≡0. Moreover, we see that in our case, nature’s probability of flipping either coin does not actually depend on the agent’s prediction, so we can replace the conditional probabilities p(θ|x) by p(θ). We have then an inner variational problem:

$$\begin{array}{l}arg\underset{\tilde{p}\left(\theta \right)}{max}\sum _{\theta }\tilde{p}\left(\theta \right)\left[-\frac{1}{\beta }log\frac{\tilde{p}\left(\theta \right)}{p_0\left(\theta \right)}+U(x,\theta )\right]\end{array}$$

(14)

with the solution

$$\begin{array}{l}p\left(\theta \right)=\frac{1}{Z_{\beta }\left(x\right)}{p}_0\left(\theta \right)exp\left(\mathrm{\beta U}(x,\theta )\right)\end{array}$$

(15)

and the normalization constant $Z_{\beta }\left(x\right)=\sum _{\theta }{p}_0\left(\theta \right)exp\left(\mathrm{\beta U}(x,\theta )\right)$ and an outer variational problem

$$\begin{array}{l}arg\underset{\tilde{p}\left(x\right)}{max}\sum _x\tilde{p}\left(x\right)\left[-\frac{1}{\alpha }log\frac{\tilde{p}\left(x\right)}{p_0\left(x\right)}+\frac{1}{\beta }log{Z}_{\beta }\left(x\right)\right]\end{array}$$

(16)

with the solution

$$\begin{array}{lcr}p\left(x\right)& =& \frac{1}{Z_{\mathrm{\alpha \beta }}}{p}_0\left(x\right)exp\left(\frac{\alpha }{\beta }log{Z}_{\beta }\left(x\right)\right)\end{array}$$

(17)

and the normalization constant $Z_{\mathrm{\alpha \beta }}=\sum _x{p}_0\left(x\right)exp\left(\frac{\alpha }{\beta }log{Z}_{\beta }\left(x\right)\right).$ From Equation (17) we can derive both the perfectly rational decision-maker and the Thompson sampling agent. To simplify, we assume in the following that the agent has no prior preference for x, that is $p_0\left(x\right)=\mathcal{U}\left(x\right)$.

The perfectly rational decision-maker is obtained in the limit α→∞ and β→0. If we first take the limit ${lim}_{\beta \to 0}\frac{1}{\beta }log{Z}_{\beta }\left(x\right)=\sum _{\theta }{p}_0\left(\theta \right)U(x,\theta )$, a decision-maker with rationality α chooses x with probability

$$p\left(x\right)=\frac{e^{\alpha \sum _{\theta }{p}_0\left(\theta \right)U(x,\theta )}}{Z_{\mathrm{\alpha \beta }}}.$$

(18)

The perfectly rational expected utility maximizer as depicted in Figure 1a is then obtained from Equation (18) by taking the limit α→∞.

In contrast, the Thompson sampling agent is obtained when β=α. In this case, the choice probability for x is given by

$$p\left(x\right)=\sum _{\theta }{p}_{0\left(\theta \right)}\frac{e^{\mathrm{\alpha U}(x,\theta )}}{Z_{\alpha }\left(x\right)}.$$

(19)

The resulting agent is a probabilistic superposition of agents that act optimally for any given θ as depicted in Figure 1b. It can be seen that in Equation (19) and in Equation (18) the order of the expectation operation and the (soft-)maximization operation are reversed.

Again we can interpret this formalism in terms of sampling complexity. Here we should accept a sample x∼p₀(x) if it fulfils the criterion

$$u\le \frac{e^{\alpha \frac{1}{\beta }log{Z}_{\beta }\left(x\right)}}{e^{\mathrm{\alpha T}}}={\left[\frac{Z_{\beta }\left(x\right)}{e^{\mathrm{\beta T}}}\right]}^{\frac{\alpha }{\beta }},$$

(20)

where $u\sim \mathcal{U}[0;1]$ and T≥ maxx maxθ U(x,θ). From Equation (11) we know that the ratio Z _β (x)/e^βT is the acceptance probability of θ∼p₀(θ). In order to accept one sample from x, we thus need to accept $\frac{\alpha }{\beta }$ consecutive samples of θ, with acceptance criterion

$$u\le \frac{e^{\mathrm{\beta U}(x,\theta )}}{e^{\mathrm{\beta T}}}$$

(21)

with $u\sim \mathcal{U}[0;1]$ and T as set above. Since α≫β we can assume α≈N β with $N\in \mathbb{N}$, and we can see easily that the perfectly rational agent will require infinitely many θ samples (α→∞ and β→0) to obtain one sample of x, whereas the Thompson sampling agent will only require a single sample (α=β). The Thompson sampling agent is therefore the agent that can solve the optimization problem of Equation (16) for a given α with the least amount of samples. This can also be seen from Equation (18), when doing the Monte Carlo approximation $\sum _{\theta }{p}_0\left(\theta \right)U(x,\theta )\approx \frac{1}{N}\sum _{i}U(x,{\theta }_i)$ by drawing N samples θ _i ∼p₀(θ _i ). For infinitely many samples, the average approximates the expectation, for a single sample we can rewrite Equation (18) into Equation (19). This sampling procedure also allows estimating the upper and lower bounds of the optimal utility (Tziortziotis et al. [2013]). Of course, the Thompson sampling agent will not achieve the same expected utility as the perfectly rational agent. But both agents can be considered optimal under particular information processing constraints.

Causal induction

A generalized Thompson-sampling agent can be thought of as a probabilistic superposition of models θ, where each model θ is characterized by a likelihood model P(o _t |θ,a_≤t,o_<t) and a policy model P(a _t |θ,a_<t,o_<t). In previous applications we assumed that all models θ have the same causal structure, i.e. considering multivariate random variables a _t and o _t , we assumed that the same variables a _t are intervened for all θ and the same causal model is used to predict the consequences of these interventions on the observational variables o _t . However, this need not be the case. In principle, different models θ could represent different causal structures and suggest intervention of different variables. Such a setup can be used for causal induction as illustrated in the following example.

Imagine we are working on a medical treatment that involves two gene sites X and Y, each of which can be active or inactive. We encode the 'on’ and 'off’ states of X as X=x and X=¬x and similarly Y=y and Y=¬y to denote the 'on’ and 'off’ states of Y. Assume we are unsure about the causal mechanism between the two variables, that is we are unsure whether the activity of X influences the activity of Y or the other way around. Formally, we are interested in the explanatory power of two competing causal hypotheses: either 'X causes Y’ (Θ=θ) or 'Y causes X’ (Θ=¬θ). Assume our goal is to have Y in an active state, but that it is much cheaper and easier to manipulate X instead of Y. This leaves us with the following policies. If X causes Y we prefer to manipulate X, because it is cheap and easy. If Y causes X we have no other choice, but to directly manipulate Y. When manipulating either gene, we can be 100% sure that the new state of the gene is set by us, but we only have a 50% chance that the state will be 'on’. Assume not manipulating either variable is not an option, because then both gene sites stay inactive. The question is how should we act if we do not know the causal dependency?

One of the main methods to deal with problems of causal inference is the framework of causal graphical models (Pearl [2000]). Given a graph that represents a causal structure, we can intervene this graph and ask questions about the probabilities of the variables in the graph. However, in causal induction we would like to discover the causal structure itself, that is we would like to do inference over a multitude of graphs representing different causal structures (Heckerman et al. [1999]). If one would like to represent the problem of causal discovery graphically, the main challenge is that the model Θ is a random variable that controls the causal structure itself. However, as argued in (Ortega [2011]), this difficulty can be overcome by using a probability tree to model the causal structure over the random events. Probability trees can encode alternative causal realizations, and in particular alternative causal hypotheses (Shafer [1996]). For instance, Figure 2a encodes the probabilities and functional dependencies among the random random variables of the previous problem.

Figure 2

Panel a: An exemplary probability tree to represent the agent’s prediction model about its environment. The probability of a realization (written under the leaves) is calculated by multiplying the probabilities starting from the root until a leave is reached. The immediate children of any note resolve the value of a single random variable. Note that the two hypotheses are statistically indistinguishable. Panel b: The probability tree resulting from (a) after intervening $\widehat{x}$.

In a probability tree, each (internal) node is a causal mechanism; hence a path from the root node to one of the leaves corresponds to a particular sequential realization of causal mechanisms. The logic underlying the structure of this tree is as follows:

• 1. Causal precedence: A node causally precedes its descendants. For instance, the root node corresponding to the sure event Ω causally precedes all other nodes.

• 2. Resolution of variables: Each node resolves the value of a random variable. For instance, given the node corresponding to Θ=θ and X=¬x, either Y=y will happen with probability $P\left(y\right|\theta ,\neg x)=\frac{1}{4}$ or Y=¬y with probability $P(\neg y|\theta ,\neg x)=\frac{3}{4}$.

• 3. Heterogeneous order: The resolution order of random variables can vary across different branches. For instance, X precedes Y under Θ=θ, but Y precedes X under Θ=¬θ. This is precisely how we model competing causal hypotheses.

While the probability tree represents the agent’s subjective model explaining the order in which the random values are resolved, it does not necessarily correspond to the temporal order in which the events are revealed to us. So for instance, under hypothesis Θ=θ, the value of the variable Y might be revealed before X, even though X causally precedes Y; and the causal hypothesis Θ, which precedes both X and Y, is never observed.

Consider a Thompson sampling agent that uses the beliefs outlined in Figure 2 that runs a single experiment. The agent does so by first manipulating X and observing Y:

• 1. Manipulating X: First, the agent instantiates his random beliefs by sampling the value of Θ from the prior:

  $$P(\Theta =\theta )=P(\Theta =\neg \theta )=\frac{1}{2}.$$

  Assume that the result is θ. Treating θ as if it was the true parameter, he proceeds to sample the action from P(X|θ) given by

  $$P(X=x|\theta )=P(X=\neg x\left|\theta \right)=\frac{1}{2},$$

  as indicated in the left branch of the probability tree. Assume that outcome is x, and this is the action that the agent executes. Because of this, the agent has to update its beliefs first by intervening the probability tree for $\widehat{x}$ and second by conditioning on x. The intervention $\widehat{x}$ is carried out by replacing all the nodes in the tree that resolve the value of X with new nodes assigning probability one to x and zero to ¬x. Figure 2b illustrates the result of this intervention. The posterior is then given by

  $$P\left(\theta \right|\widehat{x})=\frac{P\left(\widehat{x}\right|\theta \left)P\right(\theta )}{P\left(\widehat{x}\right|\theta \left)P\right(\theta )+P(\widehat{x}|\neg \theta )P(\neg \theta )}=\frac{1·\frac{1}{2}}{1·\frac{1}{2}+1·\frac{1}{2}}=\frac{1}{2}.$$

  In other words, the agent has switched on X, and has so far learned nothing from this fact.

• 2. Observing Y: Now, the agent observes the activity of Y, and assume that it is active, i.e. Y=y. Then, the posterior beliefs of the agent are given as

  $$\begin{array}{ll}P\left(\theta \right|\widehat{x},y)& =\frac{P\left(y\right|\theta ,\widehat{x}\left)P\right(\widehat{x}\left|\theta \right)P\left(\theta \right)}{P\left(y\right|\theta ,\widehat{x}\left)P\right(\widehat{x}\left|\theta \right)P\left(\theta \right)+P\left(\widehat{x}\right|\neg \theta ,y\left)P\right(y|\neg \theta )P(\neg \theta )}\\ =\frac{\frac{3}{4}·1·\frac{1}{2}}{\frac{3}{4}·1·\frac{1}{2}+1·\frac{1}{2}·\frac{1}{2}}=\frac{3}{5}.\end{array}$$

  Since $P\left(\theta \right)<P\left(\theta \right|\widehat{x},y)$, the agent has gathered evidence favoring the hypothesis “X causes Y”. This was only possible because the intervention introduced a statistical asymmetry among the two hypotheses that did not exist in the beginning. In comparison, if the action is not treated as an intervention, then the posterior is

  $$\begin{array}{ll}P\left(\theta \right|x,y)& =\frac{P\left(y\right|\theta ,x\left)P\right(x\left|\theta \right)P\left(\theta \right)}{P\left(y\right|\theta ,x\left)P\right(x\left|\theta \right)P\left(\theta \right)+P\left(x\right|\neg \theta ,y\left)P\right(y|\neg \theta )P(\neg \theta )}\\ =\frac{\frac{3}{4}·\frac{1}{2}·\frac{1}{2}}{\frac{3}{4}·\frac{1}{2}·\frac{1}{2}+\frac{3}{4}·\frac{1}{2}·\frac{1}{2}}=\frac{1}{2}=P\left(\theta \right),\end{array}$$

  that is, the agent doesn’t learn anything just from observing. This also highlights the importance of interventions (Box [1966]).

Naturally, multiple interventions and observations can be executed in consecution. In this case Thompson sampling is used in each time step to decide which policy model to use, which implies the decision which variables to intervene. Then, after the intervention, all variables are revealed simultaneously at every time step of the inference process. The update of the observational probabilities is done the same way as in the one step case, taking into account which variables were intervened. A simulation of the repeated Thompson sampling process for causal induction of our example system is shown in Figure 3. This very simple example contains the principles of causal induction using Thompson sampling. Of course, more complex causal structures require richer model classes as is customary in Bayesian modeling. But importantly, the essence of causal induction is already contained in our simple illustration.

Figure 3

Thompson sampling for causal induction. (Left) Posterior distribution P(θ|·) for 10 runs when the true system is given by Θ=θ. (Right) Posterior distribution P(θ|·) for 10 runs when the true system is given by Θ=¬θ. In both cases the agent is able to identify the causal structure of the environment with high confidence when P(θ|⋯) is close to one or zero respectively.

Discussion

The main contribution of the present paper is to show in how far generalized Thompson sampling can be regarded as an optimal solution method for adaptive decision-making in the presence of information-processing constraints and how this framework can be extended to solve problems of causal induction. We previously proposed Equation (3) as a Bayesian rule for acting in (Ortega and Braun [2010a, 2010b]) that optimally solves the adaptive coding problem for actions and observations. In practice, it is implemented by sampling an environment parameter ${\widehat{\theta }}_t$ for each time step from the posterior distribution $P\left(\theta \right|{â}_{<t},o_{<t})$, and then treating it as if it was the true parameter—that is, issuing the action a _t from $P\left(a_t\right|{\widehat{\theta }}_t,a_{<t},o_{<t})$. This action-sampling method where beliefs are randomly instantiated was first proposed as a heuristic in (Thompson [1933]) and is now known as Thompson sampling. Importantly, this method can be generalized and applied to solve general sequential adaptive decision-making problems.

So far Thompson sampling has been mainly applied to multi-armed bandit problems. Multi-armed bandits can be represented by a parameter θ that summarizes the statistical properties of the reward obtained for each lever. Reward distributions range from Bernoulli to Gaussian (with unknown mean and variance), and they can also depend on the particular context or state (Graepel et al. [2010]; May and Leslie [2011]; Granmo [2010]; Scott [2010]). In particular, the work of (May and Leslie [2011]) and the work of (Granmo [2010]) prove asymptotic convergence of Thompson sampling. The performance of bandit algorithms has also been studied in terms of the rate of growth of the regret (Lai and Robbins [1995]), and recent bandit algorithms have been shown to match this lower bound (Cappé et al. [2013]), including Thompson sampling algorithms for Bernoulli bandits (Kaufmann et al. [2012]). Also, the work of (Chapelle and Li [2011]) presents empirical results that show Thompson sampling is highly competitive, matching or outperforming popular methods such as UCB (Lai and Robbins [1995]; Auer et al. [2002]).

Another class of problems, where Thompson sampling has been applied in the past, are Markov decision processes (MDPs). MDPs can be represented by parameterizing the dynamics and reward distribution (model-based) (Strens [2000]) or by directly parameterizing the Q-table (model-free) (Dearden et al. [1998]; Ortega and Braun [2010a]). The first approach samples a full description of an MDP, solves it for the optimal policy, and then issues the optimal action. This is repeated in each time step. The second approach avoids the computational overhead of solving for the optimal policy in each time step by directly doing inference on the Q-tables. Actions are chosen by picking the one having the highest Q-value for the current state. The same ideas can also be applied to solve adaptive control problems with linear system equations, quadratic cost functions and Gaussian noise (Braun and Ortega [2010]).

Optimality

While maximum expected utility is formally appealing as a principle for the construction of adaptive agents, its strict application is in practice often problematic. This is mainly due to two reasons:

• 1. Computational complexity. The computations required to find the optimal solution (for instance, the computational complexity of solving the Bellman optimality equations) are prohibitive in general and scale exponentially with the length of the horizon. The problem is tractable only in very special cases under assumptions that reduce the effective size of the problem.

• 2. Causal precedence of policy choice. The choice of the policy has to be made before the interaction with the environment starts. That is, an agent has to have a unique optimal policy before it has even interacted once with the environment. An optimal policy constructed by the maximum expected utility principle is therefore a very risky bet, as a lot of resources have to be spent before any evidence exists that the underlying model or prior is adequate.

Because of these two reasons, it is practically often impossible to apply the maximum expected utility principle. This has led to the development of theories of bounded rational decision-making that take the information processing limitations of decision-makers into account. The modern study of bounded rationality was famously broached by Simon ([1956, 1972, 1984]) and has since been extensively investigated in psychology (Gigerenzer and Selten [2001]; Camerer [2003]), cognitive science (Howes et al. [2009]; Janssen et al. [2011]; Lewis et al.), economics (Aumann [1997]; Rubinstein [1998]; Kahneman [2003]), game theory (McKelvey and Palfrey [1995, 1998]; Wolpert [2004]), political science (Jones [2003]), industrial organization (Spiegler [2011]), computer science and artificial intelligence research (Lipman [1995]; Russell [1995]; Russell and Subramanian [1995]). Different conceptions of bounded rationality are divided as to whether bounded rational behavior is thought to be fundamentally non-optimizing or whether it can be expressed as a (constrained) optimization problem and as to whether it involves any kind of meta-reasoning (Klein [2001]). While the variational formulation in the free energy can also be thought of as a constrained optimization problem, this optimization is only implicit in an agent that runs an anytime algorithm to obtain samples that directly optimize the original (unconstrained) utility function. The average number of samples that can be afforded is determined by an inverse temperature parameter, such that the search for the optimum is aborted after some time, thereby generating some kind of satisficing solution. The free energy formulation of bounded rationality also allows reinterpreting a wider research program that has investigated relative entropy as a particular cost function for control (Kappen [2005]; Todorov [2006, 2009]; Theodorou et al. [2010]; Peters et al. [2010]; Braun and Ortega [2011]; Kappen et al. [2012]) and has inspired the formulation of optimal control problems as inference problems (Tishby and Polani [2011]; Kappen et al. [2012]; Rawlik et al. [2012]). In Section “Decision-making with limited resources” we have argued that Thompson sampling can be regarded as an instantiation of free energy optimizing bounded rationality requiring the minimal amount of samples of the latent variable θ in the decision-making process determining the next action. An agent that follows such a Thompson sampling strategy randomly samples beliefs θ and acts optimally with respect to these random beliefs. In contrast, a perfectly rational agent optimizes his utility over the entire belief tree.

Policy Uncertainty. Given a problem specification in terms of the predictive model and the utility function, we can think about policy uncertainty in terms of policy search methods. The task of a policy search method is to calculate a policy that approximates the optimal policy. More specifically, let π be a parameter in a set Π indexing the set of candidate policies P(a _t |π,a_1:t-1,o_1:t-1) indexed by θ∈Θ. Then, in the most general case, a policy search method returns a probability distribution P(π) over Π representing the uncertainty over the optimal policy parameters. If the algorithm solves the maximum expected utility problem, then the support of this distribution will exclusively cover the set of optimal policies Π^∗⊂Π. Otherwise there remains uncertainty over the optimal policy parameters. However, many policy search methods do not explicitly deal with the uncertainty over the policy parameters. Some methods only return a point estimate $\widehat{\pi }\in \Pi$. For instance, reinforcement learning algorithms (Sutton and Barto [1998]) start from a randomly initialized point estimate ${\widehat{\pi }}_0$ of the optimal policy and then generate refined point estimates ${\widehat{\pi }}_1,{\widehat{\pi }}_2,{\widehat{\pi }}_3,\dots$ in each time step t=1,2,3,… using the data provided by experience. In order to converge to the optimal policy, these algorithms have to deal with the exploration-exploitation trade-off. This means that the agents cannot just greedily act according to these point estimates; instead, they have to produce explorative actions as well, that is, actions that deviate from the current estimate of the optimal policy—for instance producing optimistic actions based on UCB (Lai and Robbins [1995]; Auer et al. [2002]).

Crucially, when sampling actions from the predictive distribution, the policy index π is identical to the index θ that identifies a particular environment with the likelihood model P(o _t |a_1:t-1,o_1:t-1). By turning the reinforcement learning problem thus into an inference problem, the exploration-exploitation trade-off becomes a bias-variance trade-off (Geman et al. [1992]) in policy space. This highlights the essence of the exploration-exploitation trade-off: any action issued by the agent has to respect the uncertainty over the policy parameter—otherwise they are biased. In particular, if the agent acts deterministically and greedily (i.e. it treats the estimate $\widehat{\pi }$ as if it were the true policy parameter) then it is overfitting the experience and introducing a bias; likewise, an agent that follows a stochastic policy introduces variance and will not produce the highest possible reward compared to the case when the optimal policy is known. An excessively stochastic agent therefore underfits its experience.

The operational distinction of having policy uncertainty has important algorithmic consequences. When there is policy uncertainty, the belief of the decision-maker is itself a random variable. This means that the very policy is undefined until the random variable is resolved. Hence, the computation of the optimal policy can be delayed and determined dynamically. It is precisely this fact that is (implicitly) exploited in popular reinforcement learning algorithms, and explicitly in the algorithms based on random beliefs. This is in stark contrast to the case when there is no policy uncertainty, where the policy is pre-computed and static. Another example where random beliefs play a crucial role is in games with incomplete information (Osborne and Rubinstein [1999]). Here, having incomplete information about the other player leads to a infinite hierarchy of meta-reasoning about the other player’s strategy. To avoid this difficulty, Harsanyi introduced Bayesian games (Harsanyi [1967]). In a Bayesian game, incomplete knowledge is modeled by randomly instantiating the player’s types, after which they choose their strategies optimally—thus eliminating the need for recurrent reasoning about the other players’ strategy. Similarly, a Thompson sampling agent randomly instantiates his belief at every point in time and acts optimally with respect to this belief. An important consequence of this is that agents have uncertainty about their policy.

Adaptive Coding. The adaptive control problem can also be construed as an adaptive coding problem both for actions and observations (Ortega and Braun [2010b, 2012b]). The question then is: How can we construct a system P defined by $P\left(o_t\right|{â}_{\le t},o_{<t})$ and $P\left(a_t\right|{â}_{<t},o_{<t})$ such that its behavior is as close as possible to the custom-made system $P\left(o_t\right|\theta ,{â}_{\le t},o_{<t})$ and $P\left(a_t\right|\theta ,{â}_{<t},o_{<t})$ under any realization of Q _θ ? Using the Kullback-Leibler divergence as a distance measure, we can formulate a variational problem in P r, where P r defines an input-output system trough a distribution over interaction sequences a₁o₁a₂o₂…, such that

$$P:=arg\underset{\mathbf{P}r}{min}\left\{{limsup}_{t\to \infty }\sum _{\theta }P\left(\theta \right)\sum _{\tau =1}^t\left(\underset{m}{\overset{a_{\tau }}{D}}\left(\mathbf{P}r\right)+\underset{m}{\overset{o_{\tau }}{D}}\left(\mathbf{P}r\right)\right)\right\}$$

with

$$\begin{array}{ll}{D}_{\theta }^{a_t}\left(\mathbf{Pr}\right)& =\sum _{{â}_{<t},o_{<t}}P({â}_{<t},o_{<t}|\theta \left)\sum _{a_t}P\right(a_t|\theta ,{â}_{<t},o_{<t})log\frac{P\left(a_t\right|\theta ,{â}_{<t},o_{<t})}{\mathbf{Pr}\left(a_t\right|{â}_{<t},o_{<t})}\\ D_{\theta }^{o_t}\left(\mathbf{Pr}\right)& =\sum _{a_{\le t},o_{<t}}P({â}_{\le t},o_{<t}|\theta \left)\sum _{o_t}P\right(o_t|\theta ,{â}_{\le t},o_{<t})log\frac{P\left(o_t\right|\theta ,{â}_{\le t},o_{<t})}{\mathbf{Pr}\left(o_t\right|{â}_{\le t},o_{<t})}.\end{array}$$

In the case of observations, this is a well-known variational principle for Bayesian inference, as it describes a predictor that requires, on average, the least amount of extra bits to capture informational surprise stemming from the behavior of the environment. In the case of actions, the same principle can be harnessed to describe resourceful generation of actions in a way that requires random bits with minimum length on average, when trying to match the optimal policy most suitable for the unknown environment (MacKay [2003]). When thinking about the adaptive control problem in this way, the aim of the adaptive agent is simply to avoid surprise. The fact that each custom-built policy $P\left(a_t\right|\theta ,{â}_{<t},o_{<t})$ can be thought of as maximizing a utility in environment Q _θ is not crucial, as this policy could also be given by a teacher’s demonstration in the absence of an explicitly stated utility function. The avoidance of surprise of adaptive systems has recently been discussed in the context of active inference and the free energy principle (Friston [2009, 2010]).

Causality

In Section “Causal induction”, we could demonstrate that generalized Thompson sampling can also be applied to the problem of causal induction, by designing policy and prediction models with different causal structures. This way generalized Thompson sampling can be used as a general method for causal induction that is Bayesian in nature. It is based on the idea of combining probability trees (Shafer [1996]) with interventions (Pearl [2000]) for predicting the behavior of a manipulated system with multiple causal hypotheses. Both the interventions and the constraints on the causal hypotheses introduce statistical asymmetries that permit the extraction of causal information. Unlike frameworks that aim to extract causal information from observational data alone (Shimizu et al. [2006]; Griffiths and Tenenbaum [2009]; Janzing and Schölkopf [2010]), the proposed method is designed for agents that interact with their environment and use these interactions to discover causal relationships.

To construct the Bayes-causal solution (3), we needed to treat actions as interventions. This raises the question about why this distinction was not made for deriving classical expected utility solutions. Since,

$$\begin{array}{ll}P\left(a_t\right|a_{<t},o_{<t})& =\sum _{\theta }P\left(a_t\right|\theta ,a_{<t},o_{<t}\left)P\right(\theta |{â}_{<t},o_{<t})\\ P\left(o_t\right|a_{\le t},o_{<t})& =\sum _{\theta }P\left(o_t\right|\theta ,a_{\le t},o_{<t}\left)P\right(\theta |{â}_{\le t},o_{<t}),\end{array}$$

determining the conditions boils down to analyzing when the equalities

$$\begin{array}{lcr}P\left(\theta \right|a_{<t},o_{<t})& =& P\left(\theta \right|{â}_{<t},o_{<t})\\ P\left(\theta \right|{â}_{\le t},o_{<t})& =& P\left(\theta \right|a_{\le t},o_{<t})\end{array}$$

hold. Replacing both sides yields,

$$\begin{array}{c}\frac{P\left(\theta \right)\prod _{k=1}^{t}P\left(a_k\right|\theta ,a_{<k},o_{<k}\left)P\right(o_k|\theta ,a_{\le k},o_{<k})}{\sum _{{\theta }^{\prime }}P\left({\theta }^{\prime }\right)\prod _{k=1}^{t}P\left(a_k\right|{\theta }^{\prime },a_{<k},o_{<k}\left)P\right(o_k|{\theta }^{\prime },a_{\le k},o_{<k})}\\ =\frac{P\left(\theta \right)\prod _{k=1}^{t}P\left(o_k\right|\theta ,a_{\le k},o_{<k})}{\sum _{{\theta }^{\prime }}P\left({\theta }^{\prime }\right)\prod _{k=1}^{t}P\left(o_k\right|{\theta }^{\prime },a_{\le k},o_{<k})}\end{array}$$

and we conclude that

$$P\left(a_k\right|\theta ,a_{<k},o_{<k})={\delta }_{{ā}_k}(a_k),$$

i.e. the actions have to be issued deterministically (but possibly history-dependent) from a unique policy. Intuitively speaking, this is because the operations of intervening and conditioning coincide when the random variables are deterministic.

Convergence

There are important cases where random belief approaches can fail. Indeed, it is easy to devise experiments where having policy uncertainty converges exponentially slower (or does not converge at all) than the Bayes adaptive optimal policy. Consider, for example, two k-order Markov chains with only one observable state when applying k times the same action, but we do not know which action it is. For two possible actions and a uniform prior over the two possible environments the distribution over possible worlds stays uniform as long as no reward has been observed. Choosing actions randomly according to this distribution would require 2^k actions to accidentally choose a sequence of the same action of length k. Thus, the Bayes adaptive optimal policy converges in time k, while the agent with policy uncertainty needs exponentially longer. A simple way to remedy this problem is, of course, to sample random beliefs only every k time steps (Strens [2000]). But this problem can be exacerbated in non-stationary environments. Take for instance, an increasing MDP with two actions and number of states $k=\lceil 10\sqrt{t}\rceil$, in which the optimal policy converges in 100 steps, while an agent with policy uncertainty would not converge at all in most realizations. Although (Ortega and Braun [2010b]) prove asymptotic convergence for general environments fulfilling a restrictive form of ergodicity condition, this condition needs to be weakened for the convergence proof to be applicable to most real problems. But it is clear that a form of ergodicity is required for an agent with policy uncertainty to be able to learn to act optimally. Intuitively, this means that an agent can only learn if the environment has temporally stable statistical properties.

Conclusion

In this paper we have argued that Thompson sampling is a bounded rational strategy in decision-making that can be considered optimal under given information processing constraints. Thompson sampling agents have uncertainty over their policy, which is a natural phenomenon that arises whenever there are not enough computational resources to apply the maximum expected utility principle to single out a unique optimal policy. Having policy uncertainty effectively weakens the two assumptions of the maximum expected utility principle: the optimal policy can be chosen and refined during interactions, and the computational complexity is lower. We have shown that treating this uncertainty in a Bayesian way with actions as random variables that obey causal calculus naturally leads to Thompson sampling and its Bayesian generalization. This generalized Thompson sampling can be straightforwardly applied to the problem of causal induction. Maintaining and updating Bayesian probabilities is an optimally efficient way to deal with uncertainty—be it with respect to the policy or the environment (Ortega and Braun [2010a]). As these random-belief approaches can be derived simply from probability theory and causal calculus we suggest that they cannot only be regarded as heuristic approximations to optimal decision-making, but as principled solution methods in their own right.

Endnote

^a Each custom-built policy P(a _t |θ,a_<t,o_<t) can be thought to maximize a utility function in its environment θ, but this is not essential—the policy could also just be given by a teacher’s demonstration as in imitation learning (Schaal [1999]).