Learning structured communication for multi-agent reinforcement learning

Abstract

This work explores the large-scale multi-agent communication mechanism for multi-agent reinforcement learning (MARL). We summarize the general topology categories for communication structures, which are often manually specified in MARL literature. A novel framework termed Learning Structured Communication (LSC) is proposed by learning a flexible and efficient communication topology (hierarchical structure). It contains two modules: structured communication module and communication-based policy module. The structured communication module learns to form a hierarchical structure by maximizing the cumulative reward of the agents under the current communication-based policy. The communication-based policy module adopts hierarchical graph neural networks to generate messages, propagate information based on the learned communication structure, and select actions. In contrast to existing communication mechanisms, our method has a learnable and hierarchical communication structure. Experiments on large-scale battle scenarios show that the proposed LSC has high communication efficiency and global cooperation capability.

Appendix

1.1 CBRP Function and HCOMM function

We provide the CBRP and HCOMM function used in the LSC in Algorithm 2 and 3.

1.2 Communication structure effects On Q-learning

The performance of a multi-agent learning algorithm is often measured by the sum of utilities obtained by all the agents: $Q({\mathbf {o}}, a)=\sum _i^N Q^i({\mathbf {o}},{\mathbf {a}})$ where the $Q^i$ is the local utility function of agent i. The ${\mathbf {o}}, {\mathbf {a}}$ are the joint observation and joint action, respectively. It is hard to analyze the performance of multi-agent learning in partial observable settings. Thus we consider a simple case: all the agents can observe the state information, which means joint observation can be obtained. In contrast, joint action can only be accessed through communication. Communication-based MARL uses local and communicated actions to approximate the joint action:

$$\begin{aligned} Q({\mathbf {o}}, {\mathbf {a}}) \approx \sum _i^N Q^i({\mathbf {o}}, a_i, a_{C_i}) \le \sum _i^N \max _{{\hat{a}}_{C_i}} Q^i({\mathbf {o}}, a_i, {\hat{a}}_{C_i}), \end{aligned}$$

(5)

where the $C_i$ is the communication agents set of agent i. Denote $-i$ as $\{1, \dots , i-1\} \bigcup \{i+1, \dots , N\}$. The FC and STAR communication methods take $C_i=N(i)=-i$ while NBOR and TREE have $C_i = N(i)\subset -i$ where the N(i) is the neighbour agents set of agent i. For Hierarchical topology, the $C_i=N(i)\bigcup NH(i) \subset -i$ where the NH(i) is the communication reachable agents set of agent i through intra-group communication. For example, in Fig. 4 of our manuscript, the agent E can communicate to A through intra-group communication, then $\{A\} \subset NH(E)$. On the one hand, taking $C_i=-i$ often faces the dimensionality issue (i.e., as the number of agents increases, the complexity of learning the utility function exponentially increases [48]). On the other hand, inappropriate design of $C_i$ leads to a loss of cooperation. To quantify how much utility an agent will potentially lose, we define the potential loss in lacking communication. Before that, we first define the potential expected utility as follows:

Definition 1

The potential expected utility of agent i is the maximum expected utility of agent i when perfect coordinating with its neighbors by communication:

$$\begin{aligned} PV_i({{\textbf {o}}}, a_i, C_i)= \max _{a_{C_i}} Q^i({\mathbf {o}}, a_i, a_{C_i}), \end{aligned}$$

(6)

where $Q^i({\mathbf {o}}, a_i, a_{C_i}) =\sum _{{\mathbf {o}}} \sum _{a_{-i\backslash C_i}} P({\mathbf {o}})P(a_{-i\backslash C_i} | {\mathbf {o}}, a_i, a_{C_i}) Q({\mathbf {o}}, {\mathbf {a}})$.

The $P(a_{-i\backslash C_i} | {\mathbf {o}}, a_i, a_{C_i})$ and $P({\mathbf {o}})$ can be estimated based on the experienced data. The $\max$ operation is taken on the $a_{C_i}$. Therefore, this measure generally overestimates the expected utility that an agent can get if it communicates and coordinates with $C_i$.

Proposition 1

If $D\subset C \subset -i$, then $PV_i({\mathbf {o}}, a_i, D_i) \le PV_i({\mathbf {o}}, a_i, C_i)$

Thus the $C_i=-i$ gains the maximum potential expected utility. Then the potential utility loss if an agent only communicates with some agents when selecting its action.

Definition 2

The potential loss in lacking communication of agent i is the difference of the potential the expected utility of agent i when it coordinates with all agents and with neighbors $C_i$:

$$\begin{aligned} PL_i({\mathbf {o}}, C_i)= \max _{a_i} PV_i({\mathbf {o}}, a_i, -i) - \max _{a_i} PV_i({\mathbf {o}}, a_i, C_i) \end{aligned}$$

(7)

Similar definitions also appear in [48]. With Proposition 1, it can be inferred that for $D_i\subset C_i$, $PL_i({\mathbf {o}}, D_i) \le PL_i({\mathbf {o}}, C_i) \le PL_i({\mathbf {o}}, -i)$. Then if NBOR (or TREE) and the hierarchy structure have the same set of neighbor agents N(i), $N(i)\subset N(i)\bigcup NH(i)$. The hierarchical structure has a potential loss less than or equal to NBOR (or TREE). Thus, the hierarchical structure is a trade-off between cooperation loss (NBOR and TREE) and curse of dimension (FC and STAR).

Furthermore, PL also suggests the importance of choosing a suitable $C_i$. If $PL_i({\mathbf {o}}, C_i)$ is large, relaxing the communication set from $-i$ to $C_i$ has a large potential loss. This lowers the upper bound of Eq. 5 and makes the algorithm obtain a lower expected global utility. If $PL_i({\mathbf {o}}, a) = 0$, we can replace $Q^i({\mathbf {o}}, a_i, a_{-i})$ with $Q^i({\mathbf {o}}, a_i, a_{C_i})$ without loss of potential expected utility. The complexity of learning can be reduced with a smaller size of $C_i$. Thus, learning $C_i$ (communication structure learning) is critical for Q-Learning. It learns the communication structure to obtain a higher global utility under current policy. This helps both the communication structure and policy learning.