Reinforcement learning in a continuum of agents

Abstract

We present a decision-making framework for modeling the collective behavior of large groups of cooperatively interacting agents based on a continuum description of the agents’ joint state. The continuum model is derived from an agent-based system of locally coupled stochastic differential equations, taking into account that each agent in the group is only partially informed about the global system state. The usefulness of the proposed framework is twofold: (i) for multi-agent scenarios, it provides a computational approach to handling large-scale distributed decision-making problems and learning decentralized control policies. (ii) For single-agent systems, it offers an alternative approximation scheme for evaluating expectations of state distributions. We demonstrate our framework on a variant of the Kuramoto model using a variety of distributed control tasks, such as positioning and aggregation. As part of our experiments, we compare the effectiveness of the controllers learned by the continuum model and agent-based systems of different sizes, and we analyze how the degree of observability in the system affects the learning process.

Appendix: Derivation of the continuum equation

In the following, we show how the continuum equation (15) can be derived from the agent-based system of stochastic differential equations (1) as the number of agents in the system approaches infinity. Herein, we follow the derivation in Dean (1996), which we extend with the necessary control-related objects.

Our goal is to find an expression for the temporal evolution of the global agent density $\rho ^{(N)}(x,t)$ as $N\rightarrow \infty$. We start with an Itô expansion of the stochastic differential equation (5),

$$\begin{aligned} \begin{aligned} {\mathrm {d}}f\big (X_i(t)\big )&= \nabla f\big (X_i(t)\big ) \cdot h\Big (X_i(t), \pi _\theta \big (\xi (X_i(t),X(t))\big )\Big ){\mathrm {d}}t \\&\phantom {=}\ + \ D\nabla ^2f\big (X_i(t)\big )\,{\mathrm {d}}t + \nabla f\big (X_i(t)\big ) \cdot {\mathrm {d}}W_i(t) ,\end{aligned} \end{aligned}$$

(24)

where $f:\mathcal {X}\rightarrow \mathbb {R}$ is a twice-differentiable test function. Using the identity

$$\begin{aligned} f\big (X_i(t)\big ) = \int _{x\in \mathcal {X}} \rho _i(x,t)f(x) \, \mathrm {d}x ,\end{aligned}$$

(25)

which follows from the definition of the single-agent density in Eq. (14), we rewrite Eq. (24) as

$$\begin{aligned} {\mathrm {d}}f\big (X_i(t)\big )&= \int _{x\in \mathcal {X}} \bigg [ \nabla f(x) \cdot h\Big (x, \pi _\theta \big (\xi (x,X(t))\big )\Big ){\mathrm {d}}t \\&\phantom {=}\ + \ D\nabla ^2f(x)\,{\mathrm {d}}t + \nabla f(x) \cdot {\mathrm {d}}W_i(t)\bigg ] \rho _i(x,t) \, \mathrm {d}x .\end{aligned}$$

Next, we integrate this equation by parts, which yields

$$\begin{aligned} {\mathrm {d}}f\big (X_i(t)\big )&= \int _{x\in \mathcal {X}} \bigg \{-\nabla \cdot \bigg [\rho _i(x,t) h\Big (x, \pi _\theta \big (\xi (x,X(t))\big )\Big )\bigg ]{\mathrm {d}}t \\&\phantom {=}\ + D\nabla ^2\rho _i(x,t)\,{\mathrm {d}}t - \nabla \cdot \rho _i(x,t)\,{\mathrm {d}}W_i(t) \bigg \} f(x) \, \mathrm {d}x .\end{aligned}$$

On the other hand, identity (25) also implies that

$$\begin{aligned} {\mathrm {d}}f\big (X_i(t)\big ) = \int _{x\in \mathcal {X}} {\mathrm {d}}\rho _i(x,t)\,f(x) \, \mathrm {d}x .\end{aligned}$$

Comparing both equations, we conclude that

$$\begin{aligned} {\mathrm {d}}\rho _i(x,t)&= -\nabla \cdot \bigg [\rho _i(x,t) h\Big (x, \pi _\theta \big (\xi (x,X(t))\big )\Big )\bigg ]{\mathrm {d}}t \nonumber \\&\phantom {=}\ + D\nabla ^2\rho _i(x,t)\,{\mathrm {d}}t - \nabla \cdot \rho _i(x,t)\,{\mathrm {d}}W_i(t) .\end{aligned}$$

In order to obtain an expression for the global density, we sum up all agent-based increments, which gives

$$\begin{aligned} {\mathrm {d}}\rho ^{(N)}(x,t)&= \frac{1}{N} \sum _{i=1}^N {\mathrm {d}}\rho _i(x,t) \nonumber \\&= -\nabla \cdot \Big [\rho ^{(N)}(x,t) h\big (x, \bar{u}^{(N)}(x,t)\big )\Big ]{\mathrm {d}}t \nonumber \\&\phantom {=}\ + D\nabla ^2\rho ^{(N)}(x,t)\,{\mathrm {d}}t - \nabla \cdot \frac{1}{N} \sum _{i=1}^N \rho _i(x,t)\,{\mathrm {d}}W_i(t) , \end{aligned}$$

(26)

where we introduced the finite-size control field $\bar{u}^{(N)}(x,t)$,

$$\begin{aligned} \bar{u}^{(N)}(x,t) :=\pi _\theta \big (\overline{y}^{(N)}(x,t)\big ) , \end{aligned}$$

(27)

and the underlying observation field $\overline{y}^{(N)}(x,t)$,

$$\begin{aligned} \overline{y}^{(N)}(x,t)&:=\xi \big (x,X(t)\big ) = \frac{\int _\mathcal {X} \rho ^{(N)}(y,t)g(x,y)k(x,y) \, \mathrm {d}y}{\int _\mathcal {X} \rho ^{(N)}(y',t)k(x,y') \, \mathrm {d}y'} , \end{aligned}$$

(28)

as replacements for the agent-based control and observation signals, $\{u_i(t)\}$ and $\{Y_i(t)\}$, respectively. Note that the latter equation follows directly from Eq. (3) using the definition of the N-agent density in Eq. (13).

As shown by Dean (1996), the cumulative influence of the agent-dependent noise terms in Eq. (26) can be described by a statistically equivalent, agent-independent field of noise processes $\overline{W}(x,t)$ with correlation function

$$\begin{aligned} {\mathbb {E}}\left[ \overline{W}_m(x,t)\overline{W}_n\left( y,t'\right) \right] = 2D\delta _{m,n}\delta _{x,y}\min \left( t,t'\right) ,\end{aligned}$$

where $\overline{W}_m(x,t)$ denotes the $m\text {th}$ component of the field at position x and time t. Equation (26) then simplifies to

$$\begin{aligned} {\mathrm {d}}\rho ^{(N)}(x,t)&= -\nabla \cdot \Big [\rho ^{(N)}(x,t) h\big (x, \bar{u}^{(N)}(x,t)\big )\Big ]{\mathrm {d}}t\\&\phantom {=}\ + D\nabla ^2\rho ^{(N)}(x,t)\,{\mathrm {d}}t + \nabla \cdot \left[ \frac{1}{N}\sqrt{\rho ^{(N)}(x,t)}\,{\mathrm {d}}\overline{W}(x,t) \right] .\end{aligned}$$

In the limit $N\rightarrow \infty$, the stochastic component of this differential equation vanishes and we obtain our final convection–diffusion dynamics for the continuum density $\rho (x,t)$,

$$\begin{aligned} \frac{\partial \rho (x,t)}{\partial t} = -\nabla \cdot \Big [\rho (x,t) h\big (x, \bar{u}(x,t)\big )\Big ] + D\nabla ^2\rho (x,t) ,\end{aligned}$$

where the continuum control field $\bar{u}(x,t)$ and the underlying continuum observation field $\overline{y}(x,t)$ are defined as in Eqs. (27) and (28), respectively, but $\rho ^{(N)}(x,t)$ is replaced by $\rho (x,t)$.