Interactive POMDPs with finite-state models of other agents

Abstract

We consider an autonomous agent facing a stochastic, partially observable, multiagent environment. In order to compute an optimal plan, the agent must accurately predict the actions of the other agents, since they influence the state of the environment and ultimately the agent’s utility. To do so, we propose a special case of interactive partially observable Markov decision process, in which the agent does not explicitly model the other agents’ beliefs and preferences, and instead represents them as stochastic processes implemented by probabilistic deterministic finite state controllers (PDFCs). The agent maintains a probability distribution over the PDFC models of the other agents, and updates this belief using Bayesian inference. Since the number of nodes of these PDFCs is unknown and unbounded, the agent places a Bayesian nonparametric prior distribution over the infinitely dimensional set of PDFCs. This allows the size of the learned models to adapt to the complexity of the observed behavior. Deriving the posterior distribution is in this case too complex to be amenable to analytical computation; therefore, we provide a Markov chain Monte Carlo algorithm that approximates the posterior beliefs over the other agents’ PDFCs, given a sequence of (possibly imperfect) observations about their behavior. Experimental results show that the learned models converge behaviorally to the true ones. We consider two settings, one in which the agent first learns, then interacts with other agents, and one in which learning and planning are interleaved. We show that the agent’s performance increases as a result of learning in both situations. Moreover, we analyze the dynamics that ensue when two agents are simultaneously learning about each other while interacting, showing in an example environment that coordination emerges naturally from our approach. Furthermore, we demonstrate how an agent can exploit the learned models to perform indirect inference over the state of the environment via the modeled agent’s actions.

Appendices

Appendix 1: Derivation of the induced prior probability on the number of nodes

We want to obtain the probability of K, the number of nodes that gets “instantiated” when drawing from the prior in Eq. 12 as a function of the concentration parameter $\alpha$, the number of actions G and observations H. We can view the process of sampling a PDFC from the prior recursively, starting from one single node and drawing its outgoing transitions according to Eq. 13, some of which may point to new nodes; we then do the same with the second node, if any, and so on. By “instantiated” nodes, we refer to the nodes drawn as a result of this procedure. Since the prior is an exchangeable probability distribution, there is no loss of generality in interpreting a draw from $p(\tau |\alpha )$ sequentially as above.

Let us now derive the probability over the number of nodes K induced by this sequential drawing procedure. We observe that K is the index of the first node whose outgoing transitions $\tau _{K\cdot \cdot }$ all point to already existing nodes (including node K itself.) We will start from $K=1$, $K=2$, $K=3$, and then derive a general rule. Let us denote as $Y=GH$ the number of outgoing transitions from each node. In the following, we index the transitions in the order that they are sampled in our schema, so that transitions $1 \le y\le Y$ are from the first node, transitions $(Y+1)\le y\le 2Y$ are from the second node, and so on. From what we described above, we know that $K=1$ if and only if all of the first node’s outgoing transitions point to itself, i.e., no new node is generated besides the first, which is created with probability one ($\frac{\alpha }{\alpha }$). According to the CRP rule, the probability of this happening is:

$$\begin{aligned} p(K=1|\alpha )=\frac{\alpha }{\alpha }\frac{1}{(1+\alpha )}\frac{2}{(2+\alpha )}\ldots \frac{Y}{(Y+\alpha )} = \frac{\alpha Y!}{\alpha ^{(Y+1)}}, \end{aligned}$$

(31)

where $\alpha ^{(Y+1)}$ is the Pochhammer symbol indicating the rising factorial $\alpha ^{(Y+1)}=\alpha (\alpha +1)(\alpha +2)\ldots (\alpha +Y)$.

For $K=2$, it must be the case that at least one of the first node’s outgoing transitions points to the second node, and the second node’s transitions all point to the first or second node. The transition from the first to the second node with the lowest index, that is, the one that “generated” the second node when sampled, can be any of the first node’s Y outgoing transitions, therefore:

$$\begin{aligned} p(K=2|\alpha ) = \frac{\alpha }{\alpha ^{(2Y)}}\frac{(2Y)!}{Y!}\;\big (\alpha \cdot 2\cdot \ldots \cdot Y \;+\; 1\cdot \alpha \cdot \ldots \cdot Y \;+\; \ldots \;+\; 1\cdot 2\cdot \ldots \cdot \alpha \big ). \end{aligned}$$

(32)

The sum of products between round brackets is the combinatorial quantity whose computation is critical in the general case.

Let us now consider $K=3$: we know that there must be one transition from the first node, having index say $y\le Y$, that points to the second node (and contributes “one $\alpha$”) and one transition indexed $y<y'\le 2Y$ that goes to the third node. This transition may come from either the first or second node. The sum of products resulting from all such possible configurations of new transitions to the second and third node is needed to compute $p(K=3|\alpha )$. For a generic K, we have to consider all the “legal” configurations of the $(K-1)$ “$\alpha$’s” that occur in the nodes previously sampled. We formalize this concept by introducing some definitions.

Definition 1

A configuration for a PDFC with K nodes is a binary vector $w^K=(w^K_1,w^K_2,\ldots ,w^K_{(K-1)Y})$ of length $(K-1)Y$, containing exactly $(K-1)$ zeros. Intuitively, the position of the first zero in this sequence identifies the first transition that was sampled to point to the second node, the second zero indicates the transition that first points to the third node, and so on. We denote as $L_{k}$ the position of the $k^{{\text {th}}}$ zero in a configuration. By convention, $L(0)=0$.

Therefore, $L_{K-1}$ is the first transition that points to node K in a PDFC with K nodes. We know that this transition must be drawn after the first transition to node $(K-1)$ is drawn. This leads to the the definition of “legal” configuration.

Definition 2

A configuration $w^K$ is legal if, for all $0<k<K$, we have that $L_{k-1}< L_k < Y(K-1)$. We denote as $W^K$ the set of all legal configurations for a PDFC with K nodes.

Each legal configuration $w^K$ is associated to a quantity $z(w^K)$, that is the product of the positions of ones in the configuration, i.e. $z(w^K) = \prod _{i=1}^{Y(K-1)} i\cdot w^K_i$. The combinatorial quantity that we need for computing the probability of having K nodes, denoted as $\phi (K)$, is the sum of such quantities for all legal configurations, i.e.

$$\begin{aligned} \phi (K)=\sum _{w^K\in W^K} z(w^K). \end{aligned}$$

(33)

If $\phi (K)$ is known, then the probability of having K nodes is given by

$$\begin{aligned} p(K|\alpha ) = \frac{\alpha ^{K}}{\alpha ^{(KY+1)}} \frac{(KY)!}{((K-1)Y)!} \phi (K), \end{aligned}$$

(34)

where:

• $\alpha ^K$ are the numerators of the CRP terms corresponding to transition draws that resulted in the creation of new nodes, including the $\alpha$ in the first vacuous term $\frac{\alpha }{\alpha }$ that “creates” the first node;

• $\alpha ^{(KY+1)}=\alpha (\alpha +1)(\alpha +2)\ldots (\alpha +KY)$ is the rising factorial, resulting from the product of the denominators of the CRP conditional distributions;

• $\frac{(KY)!}{((K-1)Y)!} = \big ((K-1)Y\cdot ((K-1)Y+1)\cdot \ldots \cdot KY\big )$ are the numerators of CRP terms for transitions outgoing from the last node K, that did not result in the creation of any new node;

• $\phi (K)$ is the sum of products of legal configurations, described above.

1.1 Efficient computation

A brute-force computation of the $\phi$ terms in Eq. 15, according to the formula in Eq. 33, would have exponential complexity. In the following, we instead describe a way to compute $\phi (K)$ more efficiently. Let us introduce the quantity $\phi (K,l)$, that represents the sum of products $z(w^K)$ for legal configurations having the last zero in position l, i.e. $L_{K-1}=l$. Since the last zero in a legal configuration for a PDFC with K nodes can occur between positions $K-1$ and $Y(K-1)$, we have that $\phi (K)=\sum _{l=K-1}^{Y(K-1)}\phi (K,l)$. In order to make its manipulation easier, we decompose $\phi (K,l)$ into the sum of products of the configurations truncated at index l included, denoted as $\bar{\phi }(K,l)$, and the remaining product of the configuration (which does not contain any zero), i.e.:

$$\begin{aligned} \phi (K,l)=\bar{\phi }(K,l)\;(l+1)(l+2)\ldots ((K-1)Y). \end{aligned}$$

(35)

It follows that:

$$\begin{aligned} \phi (K)=\sum _{l=K-1}^{Y(K-1)}\bar{q}(K,l)\frac{((K-1)Y)!}{l!}. \end{aligned}$$

(36)

We can now derive a recursive relation for $\bar{\phi }(K,l)$ from $\bar{\phi }(K,l-1)$. When “moving” the position of the last $\alpha$ from $(l-1)$ to l, we have to multiply the previous $\bar{\phi }$ by $(l-1)$, since in the corresponding configuration the element $w^K_{l-1}$ switched from 0 to 1. Moreover, by shifting the position of the last $\alpha$ to l, we must acknowledge that there are now potentially more configurations that are legal for the first $(K-2)$ $\alpha$’s, that is to say $L_{K-2}$ can now take the value $l-1$. This is only true when $l-1$ is a legal value for $L_{K-2}$, i.e. when $(l-1)<(K-2)Y$. Putting all this together, we have:

$$\begin{aligned} \bar{\phi }(K,l) = (l-1)\;\bar{\phi }(K,l-1)+ {\left\{ \begin{array}{ll} \bar{\phi }(K-1,l-1) &{} {\text {if }} (l-1)<(K-2)Y \\ 0 &{} {\text {otherwise.}} \end{array}\right. } \end{aligned}$$

(37)

Computing the values of $\phi$ in this way has a complexity of $O(K^2)$, much lower than $O(2^K)$ that results from direct computation of Eq. 33. Moreover, these values can be pre-computed and stored, since they are not dependent on $\alpha$, and used when needed.

Appendix 2: Tiger problem specifications

Table 1 Specification of the “standard” multiagent Tiger problem

Table 2 Specification of the cooperative multi-agent Tiger Problem with observable actions

Table 3 Alternative reward models for the multiagent Tiger Problem

Table 4 Specification of the cooperative multi-agent Tiger Problem in the “Follow the Leader” scenario