Game-Theory-Based Consensus Learning of Double-Integrator Agents in the Presence of Worst-Case Adversaries

Abstract

This work proposes a game-theory-based technique for guaranteeing consensus in unreliable networks by satisfying local objectives. This multi-agent problem is addressed under a distributed framework, in which every agent has to find the best controller against a worst-case adversary so that agreement is reached among the agents in the networked team. The construction of such controllers requires the solution of a system of coupled partial differential equations, which is typically not feasible. The algorithm proposed uses instead three approximators for each agent: one to approximate the value function, one to approximate the control law, and a third one to approximate a worst-case adversary. The tuning laws for every controller and adversary are driven by their neighboring controllers and adversaries, respectively, and neither the controller nor the adversary knows each other’s policies. A Lyapunov stability proof ensures that all the signals remain bounded and consensus is asymptotically reached. Simulation results are provided to demonstrate the efficacy of the proposed approach.

Appendix

Proof of Lemma 3.1

The error in Eq. (23) after subtracting zero becomes

$$\begin{aligned} e_i&=\hat{W}_\mathrm {c_i}^T {\frac{\partial \phi _i}{\partial {s}_i}}\bigg (\begin{bmatrix}\mathbf {0}&\mathbf {I} \\ \mathbf {0}&\mathbf {0}\end{bmatrix}s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(\hat{u}_i+\hat{v}_i) -\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(\hat{u}_j+\hat{v}_j)\bigg )\nonumber \\&\quad -\,W_i^T{\frac{\partial \phi _i}{\partial {s}_i}}\bigg (\begin{bmatrix}\mathbf {0}&\mathbf {I} \\ \mathbf {0}&\mathbf {0}\end{bmatrix}s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_i^*+v_i^*) -\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_j^*+v_j^*)\bigg )\nonumber \\&\quad +\,\frac{1}{2} \bigg (\Vert \hat{u}_i\Vert ^2 +\sum _{j\in {\mathcal {N}}_i}\Vert \hat{u}_j\Vert ^2-\gamma _\mathrm {ii}^2\Vert v_i\Vert ^2-\sum _{j\in {\mathcal {N}}_i}\gamma _\mathrm {ij}^2\Vert \hat{v}_j\Vert ^2\bigg )\nonumber \\&\quad -\,\frac{1}{2} \bigg (\Vert u_i^*\Vert ^2 +\sum _{j\in {\mathcal {N}}_i}\Vert u_j^*\Vert ^2-\gamma _{ii}^2\Vert v_i^*\Vert ^2-\sum _{j\in {\mathcal {N}}_i}\gamma _{ij}^2\Vert v_j^*\Vert ^2\bigg )\nonumber \\&\quad -\,\frac{\partial \epsilon _\mathrm {c_i}}{\partial {s}_i}^T\bigg (\begin{bmatrix}\mathbf {0}&\mathbf {I} \\ \mathbf {0}&\mathbf {0}\end{bmatrix}s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_i^*+v_i^*) \\ {}&\quad -\,\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_j^*+v_j^*)\bigg ), \ \forall i \in N. \end{aligned}$$

Completing the squares we have

$$\begin{aligned} e_i&=-\,\tilde{W}_\mathrm {c_i}^T\omega _i-W_i^T{\frac{\partial \phi _i}{\partial {s}_i}}\bigg (d_i\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix} (\tilde{u}_i+\tilde{v}_i)-\sum _{j\in {\mathcal {N}}_i}\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(\tilde{u}_j+\tilde{v}_j)\bigg )\nonumber \\&\quad +\,\frac{1}{2}\Vert \tilde{u}_i\Vert ^2-\frac{1}{2}\gamma _\mathrm {ii}^2\Vert \tilde{v}_j\Vert ^2-\tilde{u}_i^T u_i^*+\gamma _\mathrm {ii}^2\tilde{v}_i^T v_i^*\nonumber \\&\quad +\,\frac{1}{2}\sum _{j\in {\mathcal {N}}_i}\bigg (\Vert \tilde{u}_j\Vert ^2-\gamma _\mathrm {ij}^2\Vert \tilde{v}_j\Vert ^2-2\tilde{u}_j^T u_j^*+2\gamma _\mathrm {ij}^2 \tilde{v}_j^T v_j^*\bigg )\nonumber \\&\quad -\,\frac{\partial \epsilon _\mathrm {c_i}}{\partial {s}_i}^T\bigg (\begin{bmatrix}\mathbf {0}&\mathbf {I} \\ \mathbf {0}&\mathbf {0}\end{bmatrix}s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_i^*+v_i^*)\nonumber \\&\quad -\,\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_j^*+v_j^*)\bigg ), \ \forall i \in N. \end{aligned}$$

(39)

The dynamics of the critic estimation error $\tilde{W}_{c_i}$ can be found by substituting (39) in (25) as

$$\begin{aligned} \dot{\tilde{W}}_\mathrm {c_i}&=-\,\alpha _i \bar{\omega }_{i}\bar{\omega }_{i}^T \tilde{W}_{c_i} \nonumber \\&\quad +\,\alpha _i\frac{ \omega _i}{(\omega _i^T\omega _i+1)^2}\bigg [-W_i^T{\frac{\partial \phi _i}{\partial {s}_i}}\bigg (d_i\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix} (\tilde{u}_i+\tilde{v}_i)\nonumber \\&\quad -\,\sum _{j\in {\mathcal {N}}_i}\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(\tilde{u}_j+\tilde{v}_j)\bigg ) +\,\frac{1}{2}\Vert \tilde{u}_i\Vert ^2-\frac{1}{2}\gamma _\mathrm {ii}^2\Vert \tilde{v}_j\Vert ^2-\tilde{u}_i^T u_i^*+\gamma _\mathrm {ii}^2\tilde{v}_i^T v_i^*\nonumber \\&\quad +\,\frac{1}{2}\sum _{j\in {\mathcal {N}}_i}\bigg (\Vert \tilde{u}_j\Vert ^2-2\gamma _\mathrm {ij}^2\Vert \tilde{v}_j\Vert ^2-2\tilde{u}_j^T u_j^*+2\gamma _\mathrm {ij}^2 \tilde{v}_j^T v_j^*\bigg )\nonumber \\&\quad -\,\frac{\partial \epsilon _\mathrm {c_i}}{\partial {s}_i}^T\bigg (\left[ \begin{array}{cc} \mathbf {0} &{} \mathbf {I} \\ \mathbf {0} &{} \mathbf {0} \end{array}\right] s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_i^*+v_i^*)\nonumber \\&\quad -\,\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_j^*+v_j^*)\bigg )\bigg ], \ \forall i \in {\mathcal {N}}, \end{aligned}$$

(40)

from which the result follows. $\square$

The following fact is a consequence of Assumption 3.1 and the properties of the projection operator ${\mathcal {P}}r[.]$.

Fact A

There exist constants $b_\mathrm {i1}, b_\mathrm {i2}, b_\mathrm {i3}, b_\mathrm {i4}, b_\mathrm {i5}\in \mathbb {R}_+$ for which the following bounds hold, for every agent $\forall i\in \mathcal {N}$ and time $\forall t\geqslant 0$:

$$\begin{aligned}&\left\| {\frac{\partial \phi _i}{\partial {s}_i}}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}{\frac{\partial \phi _i}{\partial {s}_i}}^T\right\| \leqslant b_\mathrm {i1},\ \ \Vert \tilde{W}_\mathrm {u_i}\Vert \leqslant b_\mathrm {i2}, \ \ \Vert \tilde{W}_\mathrm {v_i}\Vert \leqslant b_\mathrm {i3},\nonumber \\&\left\| -W_i^T{\frac{\partial \phi _i}{\partial {s}_i}}\bigg (d_i\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix} (\tilde{u}_i+\tilde{v}_i)-\sum _{j\in {\mathcal {N}}_i}\begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(\tilde{u}_j+\tilde{v}_j)\bigg ) \right. \nonumber \\&\quad \left. +\,\frac{1}{2}\Vert \tilde{u}_i\Vert ^2-\frac{1}{2}\gamma _\mathrm {ii}^2\Vert \tilde{v}_i\Vert ^2-\tilde{u}_i^T u_i^*+\gamma _\mathrm {ii}^2\tilde{v}_i^T v_i^*\right. \nonumber \\&\quad \left. +\,\frac{1}{2}\sum _{j\in {\mathcal {N}}_i}\bigg (\Vert \tilde{u}_j\Vert ^2-\gamma _\mathrm {ij}^2\Vert \tilde{v}_j\Vert ^2-2\tilde{u}_j^T u_j^*+2\gamma _\mathrm {ij}^2 \tilde{v}_j^T v_j^*\bigg ) -\frac{\partial \epsilon _\mathrm {c_i}}{\partial {s}_i}^T\right. \nonumber \\&\left. \bigg (\begin{bmatrix}\mathbf {0}&\mathbf {I} \\ \mathbf {0}&\mathbf {0}\end{bmatrix}s_i+ d_i \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_i^*+v_i^*) -\sum _{j\in {\mathcal {N}}_i} \begin{bmatrix}\mathbf {0} \\ \mathbf {I}\end{bmatrix}(u_j^*+v_j^*)\bigg )\right\| \leqslant b_\mathrm {i4} \end{aligned}$$

(41)

$$\begin{aligned}&\left\| {u_i^*}^T\tilde{u}_i-{v_i^*}^T\tilde{v}_i-\frac{{u_i^*}^T}{d_i}\sum _{j\in {\mathcal {N}}_i}\tilde{u}_j+\frac{\gamma _\mathrm {ii}^2}{d_i}{v_i^*}^T\sum _{j\in {\mathcal {N}}_i}\tilde{v}_j\right. \nonumber \\&\quad \left. -\,\frac{1}{2}\bigg (\Vert u_i^*\Vert ^2 +\sum _{j\in {\mathcal {N}}_i}\Vert u_j^*\Vert ^2 -\gamma _\mathrm {ii}^2\Vert v_i^*\Vert ^2-\sum _{j\in {\mathcal {N}}_i}\gamma _\mathrm {ij}^2\Vert v_j^*\Vert ^2\bigg )\right. \nonumber \\&\quad \left. -\,\tilde{W}_\mathrm {u_i}^T\bigg (d_i^2\frac{\partial \phi _i}{\partial {s}_i}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _i}{\partial {s}_i}^T W_i\frac{\bar{\omega }_i^T W_i}{{(\omega _i^T\omega _i+1)}} \right. \nonumber \\&\quad \left. -\,d_i^2\frac{\partial \phi _i}{\partial {s}_i}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _i}{\partial {s}_i}^T \tilde{W}_\mathrm {u_i}\frac{\bar{\omega }_i^T W_i}{{(\omega _i^T\omega _i+1)}} \right. \nonumber \\&\quad \left. +\,\sum _{j\in {\mathcal {N}}_i}d_j^2\frac{\partial \phi _j}{\partial {s}_j}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _j}{\partial {s}_j}^T W_j\frac{\bar{\omega }_i^T W_j}{{(\omega _i^T\omega _i+1)}} \right. \nonumber \\&\quad \left. -\,\sum _{j\in {\mathcal {N}}_i}d_j^2\frac{\partial \phi _j}{\partial {s}_j}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _j}{\partial {s}_j}^T\tilde{W}_\mathrm {u_j}\frac{\bar{\omega }_i^T W_j}{{(\omega _i^T\omega _i+1)}} \bigg )\right. \nonumber \\&\quad \left. +\,\tilde{W}_\mathrm {v_i}^T\bigg (\frac{d_i^2}{\gamma _\mathrm {ii}^2}\frac{\partial \phi _i}{\partial {s}_i}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _i}{\partial {s}_i}^T W_i\frac{\bar{\omega }_i^T W_i}{{(\omega _i^T\omega _i+1)}} \right. \nonumber \\&\quad \left. -\,\frac{d_i^2}{\gamma _\mathrm {ii}^2}\frac{\partial \phi _i}{\partial {s}_i}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _i}{\partial {s}_i}^T \tilde{W}_\mathrm {v_i}\frac{\bar{\omega }_i^T}{{(\omega _i^T\omega _i+1)}} W_i\right. \nonumber \\&\quad \left. +\,\sum _{j\in {\mathcal {N}}_i}\frac{d_j^2 \gamma _\mathrm {ij}^2}{\gamma _\mathrm {jj}^4}\frac{\partial \phi _j}{\partial {s}_j}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _j}{\partial {s}_j}^T W_j\frac{\bar{\omega }_i^T W_j}{{(\omega _i^T\omega _i+1)}} \right. \nonumber \\&\quad \left. -\,\sum _{j\in {\mathcal {N}}_i}\frac{d_j^2 \gamma _\mathrm {ij}^2}{\gamma _\mathrm {jj}^4}\frac{\partial \phi _j}{\partial {s}_j}\begin{bmatrix}\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {I}\end{bmatrix}\frac{\partial \phi _j}{\partial {s}_j}^T\tilde{W}_\mathrm {v_j}\frac{\bar{\omega }_i^T W_j}{{(\omega _i^T\omega _i+1)}} \bigg ) \right\| \leqslant b_\mathrm {i5}, \end{aligned}$$

(42)

where $\tilde{u}_i=-d_i \left[ \begin{array}{l} \mathbf {0} \\ \mathbf {I} \end{array} \right] ^T (\frac{\partial \phi _i}{\partial {s}_i}^T \tilde{W}_\mathrm {u_i}+\frac{\partial \epsilon _\mathrm {c_i} }{\partial {s}_i})$, $\tilde{v}_i=\frac{d_i}{\gamma _\mathrm {ii}^2} \left[ \begin{array}{l} \mathbf {0} \\ \mathbf {I} \end{array} \right] ^T (\frac{\partial \phi _i}{\partial {s}_i}^T \tilde{W}_\mathrm {v_i}+\frac{\partial \epsilon _\mathrm {c_i} }{\partial {s}_i})$, and $u_i^*,\ v_i^*,\ \hat{u}_i,\ \hat{v}_i$ are given by (19), (20), (21), and (22), respectively. Inequalities (41) and (42) result from the fact that all the quantities that appear in the left-hand side have known definable bounds. Also all the residual errors $\epsilon _\mathrm {c_i}$ that appear in ${u}_i^*$ [see (19)], ${v}_i^*$ [see (20)], and $\tilde{u}_i$ and $\tilde{v}_i$ (as shown above) can be reduced by increasing the number of basis functions. $\square$