Modeling and reinforcement learning in partially observable many-agent systems

Abstract

There is a prevalence of multiagent reinforcement learning (MARL) methods that engage in centralized training. These methods rely on all the agents sharing various types of information, such as their actions or gradients, with a centralized trainer or each other during the learning. Subsequently, the methods produce agent policies whose prescriptions and performance are contingent on other agents engaging in behavior assumed by the centralized training. But, in many contexts, such as mixed or adversarial settings, this assumption may not be feasible. In this article, we present a new line of methods that relaxes this assumption and engages in decentralized training resulting in the agent’s individual policy. The interactive advantage actor-critic (IA2C) maintains and updates beliefs over other agents’ candidate behaviors based on (noisy) observations, thus enabling learning at the agent’s own level. We also address MARL’s prohibitive curse of dimensionality due to the presence of many agents in the system. Under assumptions of action anonymity and population homogeneity, often exhibited in practice, large numbers of other agents can be modeled aggregately by the count vectors of their actions instead of individual agent models. More importantly, we may model the distribution of these vectors and its update using the Dirichlet-multinomial model, which offers an elegant way to scale IA2C to many-agent systems. We evaluate the performance of the fully decentralized IA2C along with other known baselines on a novel Organization domain, which we introduce, and on instances of two existing domains. Experimental comparisons with prominent and recent baselines show that IA2C is more sample efficient, more robust to noise, and can scale to learning in systems with up to a hundred agents.

Appendices

Appendix 1

1.1 The influence of history-dependent reward

In this subsection, we demonstrate the influence of history-dependent reward on policies in Org. Suppose that policy $\pi _0$ leads to the reward sequence $\{\beta r, \beta r, r, \beta r, \ldots \}$ for an agent, and policy $\pi _1$ leads to the reward sequence $\{\beta r, \beta r, \frac{1+\beta }{\alpha }r, \frac{1+\beta }{\alpha }r,\ldots \}$. For convenience, we set $d = \frac{1+\beta }{\alpha }$. For horizon $H = 4$, the total reward from performing policy $\pi _0$ is:

$$\begin{aligned}&\beta r + (\phi \beta r + \beta r) + (\phi ^2 \beta r + \phi \beta r + r) + (\phi ^3 \beta r + \phi ^2 \beta r + \phi r + \beta r)\\&\quad = \phi ^3\beta r + 2\phi ^2 \beta r + 2\phi \beta r+ \phi r + 3\beta r + r \end{aligned}$$

The total reward from performing policy $\pi _1$ is:

$$\begin{aligned}&\beta r + (\phi \beta r + \beta r) + (\phi ^2 \beta r + \phi \beta r + d r) + (\phi ^3 \beta r + \phi ^2 \beta r + \phi d r + d r)\\&\quad = \phi ^3\beta r + 2\phi ^2 \beta r + 2\phi \beta r + \phi dr + 2\beta r + 2d r \end{aligned}$$

Then the total rewards from these two policies with varying choices of $\beta$ and $\phi$ can be compared. A simple program is used to check if $\phi$ can affect which of the two policies is optimal for horizon $H \ge 4$ when d is set to $\frac{9}{4}$. The result shows that for every horizon from 4 to 100, the history-dependent parameter $\phi$ is always a deciding factor in finding the optimal policy. Figure 22 shows the total reward from policy $\pi _0$ and $\pi _1$ with varying $\beta$ and $\phi$ for horizon 4. Notice that $\phi$ influences when each surface has a higher total reward.

Fig. 22

The total reward from policy $\pi _0$ and $\pi _1$ with varying $\beta$ and $\phi$ for horizon of 4

Appendix 2

1.1 Proof of Proposition 1

Claim The probability of error (which occurs when an observed configuration $\omega _0'$ is different from the true configuration ${\mathcal {C}}^e$) with the $\epsilon$-noise model is a decreasing function of N if

$$\begin{aligned} N>\frac{|A_i|}{\log (1/1-\epsilon )}. \end{aligned}$$

Proof

According to the $\epsilon$-noise model, $P(a_j^o|a_k^e)$, where the subject agent observes action $a_j^o$ from another agent when the latter executed action $a_k^e$, is

$$\begin{aligned} P(a_j^o|a_k^e) = {\left\{ \begin{array}{ll}1-\epsilon &{} if\ a_j^o=a_k^e\\ \frac{\epsilon }{|A_i|-1} &{} otherwise\end{array}\right. } \end{aligned}$$

(27)

for some small $\epsilon$. The effect of such noise from the private observation of an individual agent’s action can be aggregated over N agents in terms of $\epsilon$ as follows. Suppose the observed configuration, $\omega _0'$, is ${\mathcal {C}}^o=(\#a_1^o,\#a_2^o,\ldots ,\#a_{|A_i|}^o)$, and the true configuration is ${\mathcal {C}}^e=(\#a_1^e,\#a_2^e,\ldots ,\#a_{|A_i|}^e)$. Then the probability of an error in the observation of a configuration is

$$\begin{aligned} P(error)&=\sum _{{\mathcal {C}}^e}\sum _{{\mathcal {C}}^o\ne {\mathcal {C}}^e}P({\mathcal {C}}^o\wedge {\mathcal {C}}^e) \end{aligned}$$

(28)

$$\begin{aligned}&=\sum _{{\mathcal {C}}^e}\sum _{{\mathcal {C}}^o\ne {\mathcal {C}}^e}P({\mathcal {C}}^o|{\mathcal {C}}^e)P({\mathcal {C}}^e) \end{aligned}$$

(29)

where

$$\begin{aligned} P({\mathcal {C}}^e) &=\prod _{i}\rho _i^{\#a_i^e},\ and \\ P({\mathcal {C}}^o|{\mathcal {C}}^e) &=\prod _{(j,k)\in A\times A} P(a_j^o|a_k^e)^{n_{jk}} \\ s.t.\ \&(\sum _jn_{jk}=\#a_k^e)\wedge (\sum _kn_{jk} = \#a_j^o) \end{aligned}$$

(30)

Let $m^{oe}_i=\min \{\#a_i^o,\#a_i^e\}$. Then $P({\mathcal {C}}^o|{\mathcal {C}}^e)$ can be maximized by setting the diagonal of the matrix $[n_{jk}]$ as $n_{ii}=m^{oe}_i$, and distributing the remaining weight $N-\sum _i m^{oe}_i$ to the off-diagonal positions while satisfying Eq. 30. This yields

$$\begin{aligned} P({\mathcal {C}}^o|{\mathcal {C}}^e) \le&(1-\epsilon )^{\sum _i m^{oe}_i}\left( \frac{\epsilon }{|A_i|-1}\right) ^{N-\sum _i m^{oe}_i} \end{aligned}$$

(31)

$$\begin{aligned} \le&(1-\epsilon )^{N-1}\left( \frac{\epsilon }{|A_i|-1}\right) \end{aligned}$$

(32)

in order to ensure that ${\mathcal {C}}^o\ne {\mathcal {C}}^e$. Furthermore, the number of solutions of Eq. 30 is $\le \prod _i(m^{oe}_i +1)=O(N^{|A_i|})$. Hence

$$\begin{aligned} P(error)\le N^{|A_i|}(1-\epsilon )^{N-1}\left( \frac{\epsilon }{|A_i|-1}\right) \end{aligned}$$

(33)

The above is a decreasing function of N when $N>\frac{|A_i|}{\log (1/1-\epsilon )}$. $\square$

1.2 Proof of Proposition 2

Claim Suppose the true configuration that $\mathcal{C}'$ estimates via Eq. 23 is $\mathcal{C}^*$. Then the estimation error $\Vert \mathcal{C}'-\mathcal{C}^*\Vert$ is upper bounded by

$$\begin{aligned} \Vert \mathcal{C}'-\mathcal{C}^*\Vert \le \left( \frac{1}{1-\frac{\epsilon |A_i|}{|A_i|-1}}\right) \Vert \omega _0'-\omega _0^*\Vert \end{aligned}$$

where $\omega _0^*$ is the LHS of Eq. 23 obtained by plugging $\mathcal{C}^*$ into its RHS, and $\Vert .\Vert$ represents L1 norm.

Proof

The linear system of Eq. 23 has a coefficient matrix B with $(1-\epsilon )$ in its diagonal and $\epsilon /|A_i|-1$ in the off-diagonal elements. Now, $\omega _0^*=B\mathcal{C}^*$ and $\omega _0'=B\mathcal{C}'$. A property of such a linear system is that

$$\begin{aligned} \frac{\Vert \mathcal{C}'-\mathcal{C}^*\Vert }{\Vert \mathcal{C}^*\Vert } \le \kappa (B)\frac{\Vert \omega _0'-\omega _0^*\Vert }{\Vert \omega _0^*\Vert }, \end{aligned}$$

(34)

where $\kappa (B)$ is the condition number of B. Note that the symmetric matrix B has one eigenvalue equal to 1 and $|A_i|-1$ eigenvalues equal to $1-\epsilon -\frac{\epsilon }{|A_i|-1}$. This yields a condition number of $\left( \frac{1}{1-\frac{\epsilon |A_i|}{|A_i|-1}}\right)$. The result is obtained by plugging the condition number into Eq. 34, and noting that $\Vert \mathcal{C}^*\Vert =\Vert \omega _0^*\Vert =N$. $\square$

When the number of actions is large and $\epsilon$ is small, the condition number is $\kappa (B)\approx (1+\epsilon )$. This shows that for small $\epsilon$, the rectified estimation is well-conditioned, i.e., the rectification error is not significantly larger than the observation error.