Systemic design of distributed multi-UAV cooperative decision-making for multi-target tracking

Abstract

In this paper, we consider the cooperative decision-making problem for multi-target tracking in multi-unmanned aerial vehicle (UAV) systems. The multi-UAV decision-making problem is modeled in the framework of distributed multi-agent partially observable Markov decision processes (MPOMDPs). Specifically, the state of the targets is represented by the joint multi-target probability distribution (JMTPD), which is estimated by a distributed information fusion strategy. In the information fusion process, the most accurate estimation is selected to propagate through the whole network in finite time. We propose a max-consensus protocol to guarantee the consistency of the JMTPD. It is proven that the max-consensus can be achieved in the connected communication graph after a limited number of iterations. Based on the consistent JMTPD, the distributed partially observable Markov decision algorithm is used to make tracking decisions. The proposed method uses the Fisher information to bid for targets in a distributed auction. The bid is based upon the reward value of the individual UAV’s POMDPs, thereby removing the need to optimize the global reward in the MPOMDPs. Finally, the cooperative decision-making approach is deployed in a simulation of a multi-target tracking problem. We compare our proposed algorithm with the centralized method and the greedy approach. The simulation results show that the proposed distributed method has a similar performance to the centralized method, and outperforms the greedy approach.

Appendices

The properties of the trace of the covariance matrix

The estimations of multi-target states are produced by Kalman filter. The state of the filter is represented by two variables:

• $\hat{{\mathbf {s}}}^t(k)$, the posteriori state of the target at time k;

• $\hat{{\mathbf {P}}}^t(k)$, the posteriori error covariance matrix (a measure of the accuracy of the state estimation).

$$\begin{aligned} \hat{{\mathbf {P}}}^t(k) = {\text {var}} \big ({{\mathbf {s}}}^t(k)\big ). \end{aligned}$$

(31)

Then covariance matrices accurately reflect the covariance of estimations.

One generalization to a scalar-valued covariance for vector-valued random variables can be obtained by interpreting the deviation as the Euclidean distance:

$$\begin{aligned} {\text {var}}_s \big ({{\mathbf {s}}}^t(k)\big ) = \mathrm{E}\left[ {\Vert {{{\mathbf {s}}}^t(k)-\mathrm{E}[{{\mathbf {s}}}^t(k)]}\Vert _2^2} \right] . \end{aligned}$$

(32)

The expression can be rewritten as

$$\begin{aligned} \begin{aligned} {\text {var}}_s \big ({{\mathbf {s}}}^t(k)\big )&= \mathrm{E}\left[ {( {{{\mathbf {s}}}^t(k)-\hat{{\mathbf {s}}}^t(k)}) \cdot ({{{\mathbf {s}}}^t(k)-\hat{{\mathbf {s}}}^t(k)})} \right] \\&=\mathrm{E}\left[ \sum \nolimits _{i = 1}^n {( {{s}_i^t(k)-{\hat{s}}_i^t(k)})^2} \right] \\&= \sum \nolimits _{i = 1}^n{\mathrm{E}\left[ {( {{s}_i^t(k)-{\hat{s}}_i^t(k)})^2} \right] }\\&= \sum \nolimits _{i = 1}^n{{\text {var}}({s}_i^t(k))}\\&= \sum \nolimits _{i = 1}^n{{\hat{P}}_{ii}^t(k)}, \end{aligned} \end{aligned}$$

(33)

where ${s}_i^t(k)$ and ${\hat{s}}_i^t(k)$ are the ith element of the target state ${{\mathbf {s}}}^t(k)$ and its mean $\hat{{\mathbf {s}}}^t(k)$. The (i, j)th element of the covariance matrix $\hat{{\mathbf {P}}}^t(k)$ is ${{\hat{P}}_{ii}^t(k)}$. Finally, this can be simplified to

$$\begin{aligned} {\text {var}}_s \big ({{\mathbf {s}}}^t(k)\big ) = {\text {tr}}(\hat{{\mathbf {P}}}^t(k)), \end{aligned}$$

(34)

which is the trace of the covariance matrix.

In conclusion, the trace of the matrix is the Euclidean distance deviation of the state estimation, which can be a scalar-valued measure of the accuracy of the state estimation.

Proof of Theorem 1

The proof is based on the “Max-Plus Algebra”, defined in [43]. It is a powerful tool for the timed cyclic discrete-event systems and allows for a compact representation of weighted graphs.

The max-plus algebra consists of two binary operations, $\oplus$ and $\otimes$, on the set ${\mathbb {R}}_{\max } := {\mathbb {R}}\cup \{-\infty \}$. The operations are defined as follows:

$$\begin{aligned} \begin{aligned} a\oplus b&:=\max (a,b),\\ a\otimes b&:=a+b. \end{aligned} \end{aligned}$$

(35)

The neutral element of the max-plus addition $\oplus$ is $-\infty$, denoted by $\varepsilon$. The neutral element of multiplication $\otimes$ is 0, denoted by $\mathrm{e}$. The elements $\varepsilon$ and $\mathrm{e}$ are also referred to as the zero and one element of the max-plus algebra. Similar to conventional algebra, the associativity, commutativity, and distributivity of multiplication over addition also hold for the max-plus algebra. Both operations can be extended to matrices in a straightforward way. For $A,B\in {\mathbb {R}}_{\max }^{m\times n}$,

$$\begin{aligned} (A\oplus B)_{ij}:=a_{ij}\oplus b_{ij},~i=1,\ldots ,m,~j=1,\ldots ,n. \end{aligned}$$

(36)

For $A\in {\mathbb {R}}_{\max }^{m\times n}$, $B\in {\mathbb {R}}_{\max }^{n\times q}$,

$$\begin{aligned} (A\otimes B)_{ij}:=\mathop \oplus \limits _{k = 1}^n (a_{ik}\otimes b_{kj})=\mathop {\max }\limits _k(a_{ik}+b_{kj}) ,~i=1,\ldots ,m,~j=1,\ldots ,q. \end{aligned}$$

(37)

Multiplication of a matrix $A\in {\mathbb {R}}_{\max }^{m\times n}$ and a scalar $\alpha \in {\mathbb {R}}_{\max }$ is defined by

$$\begin{aligned} (\alpha \otimes B)_{ij}:=\alpha \otimes a_{ij}=\alpha +a_{ij} ,~i=1,\ldots ,m,~j=1,\ldots ,n. \end{aligned}$$

(38)

Note that, as in conventional algebra, the multiplication symbol $\otimes$ is often omitted.

In the sequel, we also need matrices of zero elements, denoted by N, and of one elements, denoted by E. The identity matrix I is a square matrix with

$$\begin{aligned} (I)_{ij}:= {\left\{ \begin{array}{ll} \mathrm e &{} \hbox {for }i=j;\\ \varepsilon &{} \hbox {else.} \end{array}\right. } \end{aligned}$$

(39)

For any matrix $A\in {\mathbb {R}}_{\max }^{n\times n}$, its precedence graph${\mathcal {G}}(A)$ is defined in the following way: it has n nodes, denoted by $1,\ldots ,n$, and (j, i) is an edge if and only if $a_{ij}\ne \varepsilon$. In this case $a_{ij}$ is the weight of edge (j, i). Then

• A path in ${\mathcal {G}}(A)$ is a sequence of $p > 1$ nodes, denoted by $\rho :=i_1,\ldots ,i_p$, such that $a_{i_{k+1}i_k}\ne \varepsilon , k=1,\ldots ,p-1$.

• $(A^k)_{ij}$ represents the maximal weight of all paths of length k from node j to node i, where

  $$\begin{aligned} A^k:=\underbrace{A\otimes A\otimes \ldots \otimes A}_{(k - 1)-\text{ times } \text{ multiplication }},k\ge 1 \end{aligned}$$

  (40)

  and $A^0=I$.

On this basis we define $A_i\in {\mathbb {R}}_{\max }^{N^u_i\times N^u_i}$ as a matrix, representing the communication topology of the connected undirected graph ${\mathcal {G}}^u_i$, where the ith UAV is located. There exists a path of length d from node m to node n if and only if $(A_i^d)_{mn}=\mathrm{{e}}$, where $m,n =1,2,\ldots ,N^u_i$. In addition, $E_{i}$ is defined as a particular class of matrix with one element: $(E_{i})_{mn}:=\mathrm{e}$ for $m=i$ or $n=i$. The other elements of $E_{i}$ can be at any value.

As mentioned above, the system is dynamic and the communication range is limited. Therefore, the communication graph of all UAVs may be split into multiple independent subgraphs and the topology is changing in real time. Theorem 1 essentially gives the minimum number of iterations to ensure that the ith UAV achieves the maximum in the subgraph ${\mathcal {G}}^u_i$. When each UAV in the subgraph has iterated for a corresponding number of times, the states in the entire subgraph achieve the max-consensus.

Now, it is ready to complete the proof.

Proof

A necessary and sufficient condition for max-consensus held in node i is that

$$\begin{aligned} A^l_i=E_{i},\exists l\in {\mathbb {N}}^0, \end{aligned}$$

(41)

First, the sufficiency. Given a undirected graph ${\mathcal {G}}^u_i$ composed of $N^u_i$ nodes, we define an initial vector of the perception confidence value $\gamma '^{t}_{m}(0):=[\gamma '^{t}_{1m}(0),\gamma '^{t}_{2m}(0),\ldots ,\gamma '^{t}_{N^u_im}(0)]^\mathrm{T},m=1,2,\ldots ,N^t$.

The Eq. (41) implies $\gamma '^{t}_{im}(l)=\big (E_i\otimes \gamma '^{t}_{m}(0)\big )_{i}$, where $(\cdot )_i$ is the ith element of the column vector. i.e.,

$$\begin{aligned} \begin{aligned} \gamma '^{t}_{im}(l)&=\mathop \otimes \limits _{j=1,\ldots ,N^u_i} (\gamma '^{t}_{jm}(0))\\&=\max \{\gamma '^{t}_{1m}(0),\ldots ,\gamma '^{t}_{N^u_im}(0)\}. \end{aligned} \end{aligned}$$

(42)

Applying the rules for multiplying matrices in the Max-Plus Algebra, we obtain

$$\begin{aligned} A_i^{d+l}=E_i,~\forall d\in {\mathbb {N}}^0, \end{aligned}$$

(43)

and hence

$$\begin{aligned} \gamma '^{t}_{im}(d+l)=\max \{\gamma '^{t}_{1m}(0),\ldots ,\gamma '^{t}_{N^u_im}(0)\}. \end{aligned}$$

(44)

Necessity is obvious.

If $A_i^{l}\ne E_i,~\forall l$; then $\forall l,~\exists k$ s.t. $(A_i^l)_{ik}=\varepsilon$, i.e., $\gamma '^{t}_{im}(l)$ dose not depend on $x_{km}(0)$. If $x_{km}(0)$ is the maximum element of $\{\gamma '^{t}_{1m}(0),\ldots ,\gamma '^{t}_{N^u_im}(0)\}$, the max-consensus will not hold.

In (41), $A^l_i=E_{i}$ implies that there exists a path of length l from the node i to any nodes in ${\mathcal {G}}^u_i$. Take l as the maximum of the length of all minimum path from i to each node on graph ${\mathcal {G}}^u_i$, which is the diameter of the shortest paths tree (SPT) of ${\mathcal {G}}^u_i$ rooted at node i [45].

Therefore, in order to establish the result of Theorem 1, it is essentially proved that if there exists an l, such that $A_i^l=E_i$ and $A_i^d \ne E_i$ for all $d<l$, then

$$\begin{aligned} l=D_i({\mathcal {G}}^u_i), \end{aligned}$$

(45)

where $D_i({\mathcal {G}}^u_i)$ is the diameter of SPT of ${\mathcal {G}}^u_i$ rooted at node i.

Recall the condition. $A^l=E_i$ and $A^d \ne E_i$ means that there is a shortest path from node i to each node in ${\mathcal {G}}_i$ whose length is less or equal to l. As the maximum length of these paths is the diameter of SPT of ${\mathcal {G}}_i$ rooted at node i, Eq. (45) is true. $\square$

The derivation of the FIM

The FIM [47] at time k is defined by

$$\begin{aligned} {\mathbf{G}}_{ij}(k) = E{\left[ \big ( \nabla _{{\mathbf{s}}_{i}(k)}\ln p ( {\mathbf{z}}^P_{ij}(k)\left| {\mathbf{s}}_i (k)\right. )\big )\big ( \nabla _{{\mathbf{s}}_{i}(k)}\ln p ( {\mathbf{z}}^P_{ij}(k)\left| {\mathbf{s}}_i (k)\right. )\big )^\mathrm{T} \right] }, \end{aligned}$$

(46)

where $p ( {\mathbf{z}}^P_{ij}(k)\left| {\mathbf{s}}_i (k)\right. )$ is the batch measurement likelihood. For the sake of simplicity, we omit the iterative step k in the following derivation. In this tracking scenario, as the measurement accuracy is determined by the position relationship between the UAV and the target, in which the covariance matrix is ${{\mathbf{C}}_{ij}}$ shown in (2), the FIM is only related to the position of the UAV and that of the target. Since the target is non-cooperative, it is able to change the FIM only by adjusting the position of the UAV. Therefore, it can be restated as:

$$\begin{aligned} {\mathbf{G}}_{ij} = E{\left[ \big ( \nabla _{{\mathbf{p}}^u_{i}}\ln p ( {\mathbf{z}}^P_{ij}\left| {\mathbf{p}}^u_{i},{\mathbf{p}}^t_{j}\right. )\big )\big ( \nabla _{{\mathbf{p}}^u_{i}}\ln p ( {\mathbf{z}}^P_{ij}\left| {\mathbf{p}}^u_{i},{\mathbf{p}}^t_{j}\right. )\big )^\mathrm{T} \right] }.\end{aligned}$$

(47)

The batch measurement likelihood in (47) is defined as (48):

$$\begin{aligned} \begin{aligned} p\left( {\mathbf{z}}^P_{ij}\left| {\mathbf{p}}^u_{i},{\mathbf{p}}^t_{j}\right. \right) = {\frac{1}{\sqrt{2\pi {\big | {{\mathbf{C}}_{ij}} \big |}}}\exp \Big ( - \frac{1}{2}{{\big ({{{\mathbf{z}}^P_{ij}} - \varvec{\ell } ({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}})} \big )}^\mathrm{T}}{{\mathbf{C}}_{ij}^{ - 1}}{\big ({{{\mathbf{z}}^P_{ij}} - \varvec{\ell } ({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}})} \big )} \Big )}. \end{aligned} \end{aligned}$$

(48)

In the above formula, $\varvec{\ell } ({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}})$ denotes the real value of range-bearing, which is given by

$$\begin{aligned} \begin{aligned} \varvec{\ell } ({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}})&= \left[ \begin{array}{c} d_{ij}({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}})\\ \theta _{ij}({{\mathbf{p}}^u_{i}},{{\mathbf{p}}^t_{j}}) \end{array} \right] \\&= \Bigg [ \begin{array}{c} \sqrt{(x^t_{j}-x^u_{i})^2 + {(y^t_{j}-y^u_{i})}^2+(h^{u}_{i})^2}\\ \arctan \left( \frac{y^t_{j}-y^u_{i}}{x^t_{j}-x^u_{i}} \right) \end{array} \Bigg ]. \end{aligned} \end{aligned}$$

(49)

The first order derivative of the log-density function is given by

$$\begin{aligned} \begin{aligned} {\frac{{\partial \ln p ( {\mathbf{z}}^P_{ij}\left| {\mathbf{p}}^u_{i},{\mathbf{p}}^t_{j}\right. )}}{{\partial {{\mathbf{p}}^u_{i}}}}} = - \frac{1}{2}{{\left[ {\frac{{\partial \ln \left| {\mathbf{C}}_{ij}\right| }}{{\partial {{\mathbf{p}}^u_{i}}}}} + \frac{\partial }{\partial {{\mathbf{p}}^u_{i}}}\Big ({{\big ({{{\mathbf{z}}^P_{ij}} - \varvec{\ell }} \big )}^\mathrm{T}}{{\mathbf{C}}_{ij}^{ - 1}}{\big ({{{\mathbf{z}}^P_{ij}} - \varvec{\ell }} \big )} \Big ) \right] }}. \end{aligned} \end{aligned}$$

(50)

Then the Fisher information matrix can be written as

$$\begin{aligned} \big [{\mathbf{G}}_{ij}\big ]_{mn}= {\left[ {\frac{{\partial \varvec{\ell }}}{{\partial {{\mathbf{p}}^u_{i}(m)}}}} \right] ^t}{{\mathbf{C}}_{ij}^{-1}}\left[ {\frac{{\partial \varvec{\ell }}}{{\partial {{\mathbf{p}}^u_{i}(n)}}}} \right] + \frac{1}{2}{Trace}\left( {{{\mathbf{C}}_{ij}^{-1}}\frac{{\partial {{\mathbf{C}}_{ij}}}}{{\partial {{\mathbf{p}}^u_{i}(m)}}}{{\mathbf{C}}_{ij}^{ - 1}}\frac{{\partial {\mathbf{C}}_{ij}}}{{\partial {{\mathbf{p}}^u_{i}(n)}}}} \right) . \end{aligned}$$

(51)

where ${\mathbf{G}}_{ij}$ is a square matrix of order 2; $m,n\in \{1,2\}$ represents the row number and column number of each element in ${\mathbf{G}}_{ij}$; ${\mathbf{p}}^u_{i}(1) = x^u_{i}$, and ${\mathbf{p}}^u_{i}(2)=y^u_{i}$. The specific form of each element is as following:

$$\begin{aligned}&\begin{aligned} \big [{\mathbf{G}}_{ij}\big ]_{11}&=\frac{\left( x^t_{j} - x^u_{i} \right) ^2}{d_{ij}^2\sigma _r^2}+\frac{\left( y^t_{j} - y^u_{i} \right) ^2}{r_{ij}^2\sigma _{\theta }^2}+\frac{2{k_d^2}{k_r^2} \exp \left( 2{k_r}(\frac{d_{ij}}{d^O_i}-1)\right) \left( x^t_{j} - x^u_{i} \right) ^2}{d_{ij}^2 (d^O_i)^2\sigma _r^2}\\&\quad +\frac{2{k_{\theta }^2}\left( x^t_{j} - x^u_{i} \right) ^2}{d_{ij}^2 (d^O_i)^2\sigma _{\theta }^2}, \end{aligned} \end{aligned}$$

(52)

$$\begin{aligned}&\begin{aligned} \big [{\mathbf{G}}_{ij}\big ]_{12}&=\big [{\mathbf{G}}_{ij}\big ]_{21}=\frac{\left( x^t_{j} - x^u_{i} \right) \left( y^t_{j} - y^u_{i} \right) }{d_{ij}^2\sigma _r^2} -\frac{\left( y^t_{j} - y^u_{i} \right) \left( y^t_{j} - y^u_{i} \right) }{r_{ij}^2\sigma _{\theta }^2}\\&\quad +\frac{2{k_d^2}{k_r^2} \exp \left( 2{k_r}(\frac{d_{ij}}{d^O_i}-1)\right) \left( x^t_{j} - x^u_{i} \right) \left( y^t_{j} - y^u_{i} \right) }{d_{ij}^2 (d^O_i)^2\sigma _r^2}\\&\quad +\frac{2{k_{\theta }^2}\left( x^t_{j} - x^u_{i} \right) \left( y^t_{j} - y^u_{i} \right) }{d_{ij}^2 (d^O_i)^2\sigma _{\theta }^2}, \end{aligned} \end{aligned}$$

(53)

$$\begin{aligned}&\begin{aligned} \big [{\mathbf{G}}_{ij}\big ]_{11}&=\frac{\left( y^t_{j} - y^u_{i} \right) ^2}{d_{ij}^2\sigma _r^2}+\frac{\left( x^t_{j} - x^u_{i} \right) ^2}{r_{ij}^2\sigma _{\theta }^2}+\frac{2{k_d^2}{k_r^2} \exp \left( 2{k_r}(\frac{d_{ij}}{d^O_i}-1)\right) \left( y^t_{j} - y^u_{i} \right) ^2}{d_{ij}^2 (d^O_i)^2\sigma _r^2}\\&\quad +\frac{2{k_{\theta }^2}\left( y^t_{j} - y^u_{i} \right) ^2}{d_{ij}^2 (d^O_i)^2\sigma _{\theta }^2}. \end{aligned} \end{aligned}$$

(54)