Maximal coverage problems with routing constraints using cross-entropy Monte Carlo tree search

Abstract

Spatial search, and environmental monitoring are key technologies in robotics. These problems can be reformulated as maximal coverage problems with routing constraints, which are NP-hard problems. The generalized cost-benefit algorithm (GCB) can solve these problems with theoretical guarantees. To achieve better performance, evolutionary algorithms (EA) boost its performance via more samples. However, it is hard to know the terminal conditions of EA to outperform GCB. To solve these problems with theoretical guarantees and terminal conditions, in this research, the cross-entropy based Monte Carlo Tree Search algorithm (CE-MCTS) is proposed. It consists of three parts: the EA for sampling the branches, the upper confidence bound policy for selections, and the estimation of distribution algorithm for simulations. The experiments demonstrate that the CE-MCTS outperforms benchmark approaches (e.g., GCB, EAMC) in spatial search problems.

Appendix A: Mathematical formulation of CEM

The mathematical formulation of CEM is as follows: First, the probability l representing f(x) is greater than or equal to the threshold $(\gamma )$. When l is small (e.g., $l\le 10^{-5}$), it is called a rare event. Mathematically, l can be expressed as

$$\begin{aligned} l = {\mathbb {P}}_u(f(\textbf{X})\ge \gamma )= {\mathbb {E}}_uI_{{f(\textbf{X})\ge \gamma }}, \end{aligned}$$

(15)

where I is an indicator function and x is sampled by the density $h(\cdot ;u)$ with the parameter u. Alternatively, l can be derived as

$$\begin{aligned} l = \int I_{{f(x)\ge \gamma }} \frac{h(x;u)}{g(x)}g(x) \,dx = {\mathbb {E}}_gI_{{f(\textbf{X})\ge \gamma }}\frac{h(\textbf{X};u)}{g(\textbf{X})}, \end{aligned}$$

(16)

where g is another density. Notice that, in Eq. 16, the expectation is taken by g. An unbiased estimator of l is

$$\begin{aligned} \hat{l} = \frac{1}{N} \sum _{i=1}^{N} I_{{f(x_i)\ge \gamma }}\frac{h(x_i;u)}{g(x_i)}. \end{aligned}$$

(17)

Second, density estimation of l is

$$\begin{aligned} g^*(x) := \frac{I_{{f(x)\ge \gamma }}h(x;u)}{l}. \end{aligned}$$

(18)

However, l is unknown. The direct way is to select g from the family of densities $h(\cdot ;v)$.

Third, a measurement between two density functions g and h is the $Kullback-Leibler$ distance (KLD) (so called the cross-entropy between g and h). The KLD is defined as:

$$\begin{aligned}&{\mathcal {D}}(g,h) \nonumber \\ {}&\quad = {\mathbb {E}}_g \ln \frac{g(\textbf{X})}{h(\textbf{X})} = \int g(x)\ln g(x) \,dx - \int g(x)\ln h(x) \,dx. \end{aligned}$$

(19)

Hence, minimizing KLD between $g^*$ in Eq. 18 and $h(\cdot ;v)$ is equal to maximize $\int g^*{(x)}\ln h(x;v) \,dx$. Replacing $g^*{(x)}$ with Eq. 18, it is equivalent to

$$\begin{aligned}&\max _{v} \int \frac{I_{{f(x)\ge \gamma }}h(x;u)}{l} \ln h(x;v) \,dx \nonumber \\ {}&\quad = \max _{v} {\mathbb {E}}_u I_{f(x)\ge \gamma } \ln h(\textbf{X};v). \end{aligned}$$

(20)

Finally, v can be obtained by solving Eq. 20.