Online Learning in Budget-Constrained Dynamic Colonel Blotto Games

Abstract

In this paper, we study the strategic allocation of limited resources using a Colonel Blotto game (CBG) under a dynamic setting and analyze the problem using an online learning approach. In this model, one of the players is a learner who has limited troops to allocate over a finite time horizon, and the other player is an adversary. In each round, the learner plays a one-shot Colonel Blotto game with the adversary and strategically determines the allocation of troops among battlefields based on past observations. The adversary chooses its allocation action randomly from some fixed distribution that is unknown to the learner. The learner’s objective is to minimize its regret, which is the difference between the cumulative reward of the best mixed strategy and the realized cumulative reward by following a learning algorithm while not violating the budget constraint. The learning in dynamic CBG is analyzed under the framework of combinatorial bandits and bandits with knapsacks. We first convert the budget-constrained dynamic CBG to a path planning problem on directed graph. We then devise an efficient algorithm that combines a special combinatorial bandit algorithm for path planning problem and a bandits with knapsack algorithm to cope with the budget constraint. The theoretical analysis shows that the learner’s regret is bounded by a term sublinear in time horizon and polynomial in other parameters. Finally, we justify our theoretical results by carrying out simulations for various scenarios.

Appendices

Appendix A: Existing Algorithms

For the sake of completeness, in this appendix, we provide a detailed description of the existing algorithms (Algorithms 4–6) that we use as subroutines in our main algorithm.

Algorithm 4

Weight-Pushing Algorithm, [42, Algorithm 2]

Algorithm 5

Co-occurrence Matrix Computation Algorithm, [42, Algorithm 3]

Algorithm 6

Hedge, [21, Fig. 1]

Appendix B: Concentration Inequalities

Lemma 3

[Bernstein’s inequality for martingales, Lemma A.8 in [13]] Let $Y_1, Y_2, \ldots$ be a martingale difference sequence (i.e., ${\mathbb {E}}[Y_t\vert Y_{t-1},\ldots ,Y_1]=0 \, \forall t \in {\mathbb {Z}}_+$). Suppose that $|Y_t |\le c$ and ${\mathbb {E}}[Y_t^2 \vert Y_{t-1}, \ldots , Y_1] \le v$ almost surely for all $t \in {\mathbb {Z}}_+$. For any $\delta > 0$,

$$\begin{aligned} \text {P }\left( \sum _{t=1}^T Y_t > \sqrt{2 Tv \ln (1/\delta )} + \frac{2}{3} c\ln (1/\delta ) \right) \le \delta . \end{aligned}$$

Lemma 4

[Azuma-Hoeffding’s inequality, Lemma A.7 in [13]] Let $Y_1, Y_2, \ldots$ be a martingale difference sequence and $\vert Y_t\vert \le c$ almost surely for all $t \in {\mathbb {Z}}_+$. For any $\delta > 0$,

$$\begin{aligned} \text {P }\left( \sum _{t=1}^T Y_t > \sqrt{2Tc^2 \ln (1/\delta )}\right) \le \delta . \end{aligned}$$

Appendix C: Proof of Theorem 1

The proof of Theorem 1 is built upon several results from combinatorial bandits [7, 12, 15]. In the following, we first state some useful lemmas from [12, 43], and then use them to prove Theorem 1.

Throughout the appendix, let $\varvec{r}_t(\varvec{u}) = \varvec{l}_t^{\intercal } \varvec{u}$ be the reward by playing arm $\varvec{u}$ in round t, and ${\hat{r}}_t(\varvec{u}) = \hat{\varvec{l}}_t^{\intercal } \varvec{u}$ be the estimated reward where $\hat{\varvec{l}}_t$ is the unbiased cost estimator computed in Algorithm 2.

Lemma 5

Let $\lambda ^*$ be the smallest nonzero eigenvalue of the co-occurrence matrix $M(\mu )=\mathbb {E}_{\varvec{u}\sim \mu }[\varvec{u}\varvec{u}^{\intercal }]$ for the exploration distribution $\mu$ in algorithm Edge. For all $\varvec{u} \in {\mathcal {S}} \subseteq \{0,1\}^E$ and all $t\in [T]$, the following relations hold:

• $\vert \vert \varvec{u}\vert \vert ^2 = \sum _{i=1}^{E} u_i^2 = n$

• $\vert {\hat{r}}_t(\varvec{u})\vert = \vert \hat{\varvec{l}}_t^{\intercal } \varvec{u}\vert \le \frac{n}{\gamma \lambda ^*}$

• $\varvec{u}^{\intercal } C_t^{-1} \varvec{u} \le \frac{n}{\gamma \lambda ^*}$

• $\sum _{\varvec{u} \in \mathcal {S}} p_t(\varvec{u}) \varvec{u}^{\intercal } C_t^{-1} \varvec{u} \le E$

• $\mathbb {E}_t[(\hat{\varvec{l}}_t^{\intercal } \varvec{u})^2] \le \varvec{u}^{\intercal } C_t^{-1} \varvec{u}$

where in the last expression $\mathbb {E}_t[\cdot ]{:}{=} \mathbb {E}[\cdot \vert \varvec{u}_{t-1},\ldots ,\varvec{u}_1]$.

Lemma 6

[12, Appendix A] By choosing $\eta = \frac{\gamma \lambda ^*}{n}$ such that $\eta \vert {\hat{r}}_t(\varvec{u})\vert \le 1$, for all $\varvec{u}^* \in \mathcal {S}$,

$$\begin{aligned} \sum _{t=1}^T {\hat{r}}_t(\varvec{u}^*) - \frac{1}{\eta }\ln S&\le \frac{1}{1-\gamma } \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} p_t(\varvec{u}) {\hat{r}}_t(\varvec{u}) \\&\quad + \frac{\eta }{1-\gamma } \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} p_t(\varvec{u}){\hat{r}}_t(\varvec{u})^2\\&\quad - \frac{\gamma }{1-\gamma } \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} {\hat{r}}_t(\varvec{u}) \mu (\varvec{u}). \end{aligned}$$

(C1)

Proof

This is a straightforward result of Eqs. (A.1)–(A.3) in [12] by flipping the sign of $\eta$. $\square$

Lemma 6 provides a baseline for bounding the regret. We will proceed to bound each summation in (C1). The following lemma, derived from Bernstein’s inequality (Lemma 3), provides a high-probability bound on the left side of (C1).

Lemma 7

With probability at least $1 - \delta$, for all $\varvec{u} \in \mathcal {S}$, it holds that

$$\begin{aligned} \sum _{t=1}^T r_t(\varvec{u}) - \sum _{t=1}^T {\hat{r}}_t(\varvec{u}) \le \sqrt{2T \left( \frac{n}{\gamma \lambda ^*}\right) \ln (S/\delta )} + \frac{2}{3}\left( \frac{n}{\gamma \lambda ^*}+1\right) \ln (S/\delta ). \end{aligned}$$

Proof

Fix $\varvec{u} \in \mathcal {S}$. Define $Y_t = r_t(\varvec{u}) - {\hat{r}}_t(\varvec{u})$. Then $\{Y_t\}_{t=1}^T$ is a martingale difference sequence. From Lemma 5, we know that $\vert Y_t\vert \le \frac{n}{\gamma \lambda ^*}+1$. Let $\mathbb {E}_t[Y_t^2] = \mathbb {E}[Y_t^2\vert Y_{t-1}, \ldots , Y_1]$. Then,

$$\begin{aligned} \mathbb {E}_t[Y_t^2] \le \mathbb {E}_t[(\hat{\varvec{l}}_t^{\intercal } \varvec{u})^2] \le \varvec{u}^{\intercal } C_t^{-1} \varvec{u} \le \frac{n}{\gamma \lambda ^*}. \end{aligned}$$

Using Bernstein’s inequality, with probability at least $1-\delta /S$,

$$\begin{aligned} \sum _{t=1}^T Y_t \le \sqrt{2T \left( \frac{n}{\gamma \lambda ^*}\right) \ln (S/\delta )} + \frac{2}{3}\left( \frac{n}{\gamma \lambda ^*}+1\right) \ln (S/\delta ) \end{aligned}$$

The lemma now follows by using the above inequality and taking a union bound over all $\varvec{u} \in \mathcal {S}$. $\square$

The following two lemmas obtained in [7] provide a high-probability bound on the first and second summands on the right side of (C1). The proofs of these lemmas, which are omitted here due to space limitation, use a direct application of Bernstein’s inequality and Azuma-Hoeffding’s inequality.

Lemma 8

[7, Lemma 6] With probability at least $1-\delta$,

$$\begin{aligned}{} & {} \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} p_t(\varvec{u}) {\hat{r}}_t(\varvec{u}) -\sum _{t=1}^T r_t(\varvec{u}_t) \le \left( \sqrt{E} + 1\right) \sqrt{2T\ln (1/\delta )}+\frac{4}{3}\ln (1/\delta )\left( \frac{n}{\gamma \lambda ^*}+1\right) . \end{aligned}$$

Lemma 9

[7, Lemma 8] With probability at least $1 - \delta$,

$$\begin{aligned} \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} p_t(\varvec{u}) {\hat{r}}_t(\varvec{u})^2 \le ET + \frac{n}{\gamma \lambda ^*}\sqrt{2T \ln (1/\delta )}. \end{aligned}$$

Now we are ready to complete the proof of Theorem 1. Using Lemma 7 and because $\sum _{t=1}^T r_t(\varvec{u}) \ge 0$ for every $\varvec{u}\in \mathcal {S}$, we can bound the last term on the right side of (C1) with probability at least $1-\delta$ as

$$\begin{aligned} - \gamma \sum _{t=1}^T \sum _{\varvec{u}\in \mathcal {S}} {\hat{r}}_t(\varvec{u}) \mu (\varvec{u}) \le \gamma \sqrt{2T\left( \frac{n}{\gamma \lambda ^*}\right) \ln (S/\delta )}+ \frac{2}{3}\gamma \left( \frac{n}{\gamma \lambda ^*}+1\right) \ln (S/\delta ) \end{aligned}$$

Using Lemmas 7 to 9 and the above inequality in Lemma 6, with probability at least $1 - 4\delta$, for all $\varvec{u} \in \mathcal {S}$ we have,

$$\begin{aligned}&\sum _{t=1}^T r_t(\varvec{u}) - \sum _{t=1}^T r_t(\varvec{u}_t) \le \sqrt{2T \left( \frac{n}{\gamma \lambda ^*}\right) \ln (S/\delta )} + 3\left( \frac{n}{\gamma \lambda ^*}+1\right) \ln (S/\delta ) \\&\quad + (\sqrt{E} + 1) \sqrt{2T \ln (1/\delta )} + \frac{\gamma \lambda ^*}{n} ET + \sqrt{2T \ln (S/\delta )} + \gamma T. \end{aligned}$$

Finally, if we set

$$\begin{aligned} \gamma = \frac{n}{\lambda ^*}\sqrt{\frac{\ln S}{\left( \frac{n}{E\lambda ^*}+1\right) ET^{2/3}}}, \end{aligned}$$

the following regret bound can be obtained:

$$\begin{aligned} \sum _{t=1}^T r_t(\varvec{u})&- \sum _{t=1}^T r_t(\varvec{u}_t) \le \sqrt{2 \left( \frac{n}{E\lambda ^*}+1\right) ^{1/2} T^{4/3} E^{1/2} (\ln (S/\delta ))^{1/2}} \\&+ 3 \sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E T^{2/3} \ln (S/\delta )} + 3 \ln (S/\delta ) \\&+ \sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E T^{4/3} \ln S}+ (\sqrt{E}+1)\sqrt{2T \ln (1/\delta )} + \sqrt{2T \ln (S/\delta )} \\&= O\left( T^{2/3}\sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E \ln (S/\delta )}\right) . \end{aligned}$$

Appendix D: Proof of Theorem 2

It has been shown in [21, 28] that the algorithm Hedge achieves the high-probability regret bound of

$$\begin{aligned} R_{\delta }(T) = O\left( \sqrt{T \ln (\vert A\vert /\delta )}\right) , \end{aligned}$$

where $\vert A\vert$ denotes the cardinality of action set, which in our setting is the number of resources. Since we only have two types of resources, namely the time and the troops, we have $\vert A\vert =2=O(1)$.

From Eq. (11), we have $\vert \mathcal {L}_t^{\text {troop}}\vert \le \max \{2, \vert 1-c \vert \} \le 1+c$ for $c \ge 1$. As a result, the actual reward $r_t(\varvec{u})$, estimated reward ${\hat{r}}_t(\varvec{u})$, and hence the regret bound of Theorem 1 are all scaled up by at most a constant factor $1+c$. Hence, in order to satisfy the assumption of Theorem 1 given in [12] that $\eta \vert {\hat{r}}_t(\varvec{u})\vert \le 1$, we set

$$\begin{aligned} \eta = \frac{\gamma \lambda ^*}{(1+c)n} = \frac{1}{1+c} \sqrt{\frac{\ln S}{\left( \frac{n}{E\lambda ^*}+1\right) ET^{2/3}}}. \end{aligned}$$

On the other hand, from Theorem 1, the high-probability regret bound of algorithm Lagrange-Edge is at most

$$\begin{aligned} R_{\delta }(T) = O\left( T^{2/3}\sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E \ln (S/\delta )}\right) . \end{aligned}$$

Now, using Lemma 1, and noting that $B_0$ is scaled to $\frac{B}{(cB/T)}=\frac{T}{c}$ such that $O\left( \frac{T}{B_0}\right) =O(1)$, we obtain that with probability at least $1-O(\delta T)$, it holds that

$$\begin{aligned} R(T)&\le O(1) \cdot \Bigg ( O\left( T^{2/3}\sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E \ln (ST/\delta )}\right) + O\left( \sqrt{T \ln (T/\delta )}\right) \Bigg ) \\&= O\left( T^{2/3}\sqrt{\left( \frac{n}{E\lambda ^*}+1\right) E \ln (ST/\delta )}\right) . \end{aligned}$$

Finally, we recall that $E = O(nm^2)$, $m = O(B/T)$, and $S = O\left( 2^{\min \{n-1,m\}}\right)$, such that $\ln S \le O(m) = O(B/T)$. Substituting these relations into the above inequality we get

$$\begin{aligned} R(T)&\le O\left( T^{2/3}\sqrt{\left( \frac{T^2}{B^2 \lambda ^*}+1\right) n \left( \frac{B}{T}\right) ^3 \ln (T/\delta )}\right) \\&= O \left( T^{1/6}\sqrt{\frac{nB}{\lambda ^*} \ln \left( T/\delta \right) }\right) . \end{aligned}$$