Collective behavior of artificial intelligence population: transition from optimization to game

Abstract

Collective behavior in the resource allocation systems has attracted much attention, where the efficiency of the system is intimately depended on the self-organized processes of the multiple agents that composed the system. Nowadays, as artificial intelligence (AI) is adopted ubiquitously in decision making in various scenes, it becomes crucial and unavoidable to understand what would emerge in an multi-agent AI systems for resource allocation and how can we intervene the collective behavior there in the future, as we have experience of the possible unexpected outcomes that are induced by collective behavior. Here, we introduce the reinforcement learning (RL) algorithm into minority game (MG) dynamics, in which agents have learning ability based on one typical RL scheme, Q-learning. We investigate the dynamical behaviors of the system numerically and analytically for a different game setting, with combination of two different types of agents which mimic the diversified situations. It is found that through short-term training, the multi-agent AI system adopting Q-learning algorithm relaxes to the optimal solution of the game. Moreover, one striking phenomenon is the transition of interaction mechanism from self-organized optimization to game through tuning the fraction of RL agents $\eta _{q}$. The critical curve for transition between the two mechanisms in phase diagram is obtained analytically. The adaptability of the AI agents population against the time-variable environment is also discussed. To gain further understanding of these phenomena, a theoretical framework with mean-field approximation is also developed. Our findings from the simplified multi-agent AI system may give new enlightenment to how the reconciliation and optimization can be breed in the coming era of AI.

Appendix

The critical curve of interaction transition. There are two types of agents (RL agents and DD agents) as we have set in the model and the corresponding decision rules in a resource allocation system. Their density is $\eta _{q}$ and $\eta _{d}$, respectively, in system and satisfies $\eta _{q}+\eta _{d}=1$. There are only two kinds of resources $+$ and −, and the capacity of each resource which can accommodate agents is $C_r=1/2$. Moreover, the preference probability is p or $1-p$ for resource $+$ or − in a period T for DD agents. Therefore, the optimization resource allocation in the system needs to satisfy:

$$\begin{aligned} \left\{ \begin{array}{l} p\eta _d+\eta _{q}=C_r\\ \eta _d+\eta _{q}=1\\ \end{array} \right. \end{aligned}$$

(11)

Then, we can solve Eq. 11$p=\frac{C_r-1+\eta _{q}}{\eta _{d}}$. In the right area of the curve, there is not interaction mechanism between RL agents, but only exists the interaction between RL agent and DD agent, because the minority resource is determined by the density of DD agent. In the left area of the curve, these two interaction mechanisms determined the evolution behavior of the system simultaneously.

Fig. 9

Transient state behavior of $\rho _+^{q}$ for RL agents at $t = 2000$. a A small system size (cyan dot line) and a large system size (magenta dot), but the RL agent density $\eta _{q}$ is fixed. b Reports three exploration rates $\epsilon =0.04$ (bottle green), 0.03 (yellow dot) and 0.02 (magenta dot). The else parameters $p = 0.2, \eta _{q} = 0.02, \alpha =0.9, \gamma =0.9, T =2000$. (Color figure online)

Next, we calculate the target value of RL agents $\rho ^{*}$, namely the magenta line position as shown in Fig. 1. It is not difficult to understand that $\rho ^{*}$ is 0 or 1 in the right area of the curve in the p and $\eta _d$ parameter space, because the interaction between RL agents is eliminated, and they converge to the minority resource independently by oneself. Namely, the target resource is minority resource. In the left area of the curve, the target value of RL agents $\rho ^{*}$ satisfies as follows relationship $\rho ^{*}\eta _{q}=C_r-\eta _d$ in the p and $\eta _d$ parameter space, namely $\rho ^{*}=\frac{(C_r-\eta _dp)}{\eta _{q}}$. Therefore, the target value is $\min (\frac{(C_r-\eta _dp)}{\eta _{q}},1)$ for RL agents in entire p and $\eta _d$ parameter space. It gives the target that RL agent can learn.

We investigate evolution behavior of the deviation $\kappa =\sqrt{\langle (0.5-\rho _+^{q})^2\rangle }$ with the exploration rate $\epsilon$ in Section 3.2 in this paper. Then, in this section, we investigate the impact of system size and the exploration rate $\epsilon$ on the transient state when environment is changing. Figure 9a shows the transient behavior of $\rho ^{q}_+$ under different sizes of system, but the RL agents density $\eta _{q}$ is identical, a small system size (cyan dot line) and a big system size (magenta dot) at $t = 2000$. The parameters p and $\eta _{q}$ are still located on the right region of the critical curve in p and $\eta _{d}$ space. We find the two oscillatory transient curves of $\rho ^{q}_+$ are not different qualitatively, only become more smooth for the bigger system size with the same $\eta _{q}$. In fact, it is gradually revealing statistical effect of reinforcement learning agents when the number of RL agents increases for random exploration $\epsilon$. That is to say, the dynamics mechanism of the system is not changed. The oscillatory convergent character of $\rho ^{q}_+$ is not affected by the size of system.

Figure 9b shows the transient state behavior of $\rho ^{q}_+$ with three different exploration rates for reinforcement learning agents. The transient state time decreases prominently with the exploration rate rising slightly. The higher the exploratory rate for RL agents, the easier to discover the useful route; therefore, the convergence speed of $\rho ^{q}_+$ is faster. However, the larger $\epsilon$ is advantageous to adjust strategy for their payoff maximization when the environment occurs changing, but it is not useful to hold state in a stable environment for the large fluctuation from the exploration rate $\epsilon$.