On Two Continuum Armed Bandit Problems in High Dimensions

Abstract

We consider the problem of continuum armed bandits where the arms are indexed by a compact subset of $\mathbb {R}^{d}$. For large d, it is well known that mere smoothness assumptions on the reward functions lead to regret bounds that suffer from the curse of dimensionality. A typical way to tackle this in the literature has been to make further assumptions on the structure of reward functions. In this work we assume the reward functions to be intrinsically of low dimension k ≪ d and consider two models: (i) The reward functions depend on only an unknown subset of k coordinate variables and, (ii) a generalization of (i) where the reward functions depend on an unknown k dimensional subspace of $\mathbb {R}^{d}$. By placing suitable assumptions on the smoothness of the rewards we derive randomized algorithms for both problems that achieve nearly optimal regret bounds in terms of the number of rounds n.

Appendix: Proofs of Results in Section 4

1.1 A.1 Proof of Lemma 3

Proof

We can bound R ₃ from above as follows.

$$\begin{array}{@{}rcl@{}} R_{3} &=& \sum\limits_{t=n_{1}+1}^{n}[\bar{r}(\mathbf x^{*}) - \bar{r}(\mathbf x^{**})] \end{array} $$

(44)

$$\begin{array}{@{}rcl@{}} &=& n_{2}[\bar{g}(\mathbf{A}\mathbf x^{*}) - \bar{g}(\mathbf{A}\mathbf x^{**})] \end{array} $$

(45)

$$\begin{array}{@{}rcl@{}} &=& n_{2}[\bar{g}(\mathbf{A}\mathbf x^{*}) - \bar{g}(\mathbf{A}\widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}}\mathbf x^{**})] \end{array} $$

(46)

$$\begin{array}{@{}rcl@{}} &\leq& n_{2}[\bar{g}(\mathbf{A}\mathbf x^{*}) - \bar{g}(\mathbf{A}\widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}}\mathbf x^{*})] \end{array} $$

(47)

$$\begin{array}{@{}rcl@{}} &\leq& n_{2} C_{2} \sqrt{k} \parallel{\mathbf{A}\mathbf x^{*} - \mathbf{A}\widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}}\mathbf x^{*}}\parallel \end{array} $$

(48)

$$\begin{array}{@{}rcl@{}} &\leq& n_{2} C_{2} \sqrt{k} (1+\nu) \parallel{\mathbf{A} - \mathbf{A}\widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}}}\parallel_{F} \end{array} $$

(49)

$$\begin{array}{@{}rcl@{}} &=& \frac{n_{2} C_{2} \sqrt{k} (1+\nu)}{\sqrt{2}} \parallel{\mathbf{A}^{T}\mathbf{A} - \widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}}}\parallel_{F}. \end{array} $$

(50)

In (46) we used the fact that $\mathbf x^{**} = \widehat {\mathbf {A}}^{T}\widehat {\mathbf {A}}\mathbf x^{**}$ since $\mathbf x^{**} \in \mathcal {P}$. In (47) we used the fact that $\bar {g}(\mathbf {A}\widehat {\mathbf {A}}^{T}\widehat {\mathbf {A}}\mathbf x^{**}) \geq \bar {g}(\mathbf {A}\widehat {\mathbf {A}}^{T}\widehat {\mathbf {A}}\mathbf {x}^{*})$ since $\widehat {\mathbf {A}}^{T}\widehat {\mathbf {A}}\mathbf {x}^{*} \in \mathcal {P}$ and $\mathbf {x}^{**} \in \mathcal {P}$ is an optimal strategy. Equation (48) follows from the mean value theorem along with the smoothness assumption made in (9). In (49) we used the simple inequality : ∥B x∥≤∥B∥ _F ∥x∥. Obtaining (50) from (49) is a straightforward exercise. □

1.2 A.2 Proof of Lemma 4

Proof

We first recall the following result by Candes et al. [14, Theorem 1], which we will use in our setting, for bounding the error of the matrix Dantzig selector.

Theorem 4

For any $\mathbf {X} \in \mathbb {R}^{d \times m_{\mathcal {X}}}$ such that rank( X) ≤k let $\widehat {\mathbf {X}}_{DS}$ be the solution of ( 32 ). If $\delta _{4k} < \delta < \sqrt {2}-1$ and ∥Φ ^∗ ( H+N)∥≤λ then we have with probability at least $1-2 e^{-m_{\Phi }q(\delta ) + 4k(d+m_{\mathcal X}+1)u(\delta )}$ that

$$\parallel{\mathbf{X} - \widehat{\mathbf{X}}_{DS}}\parallel_{F}^{2} \leq C_{0} k \lambda^{2} $$

where C ₀ depends only on the isometry constant δ _4k .

What remains to be found for our purposes is λ which is a bound on ∥Φ^∗(H+N)∥. Firstly note that ∥Φ^∗(H+N)∥≤∥Φ^∗(H)∥+∥Φ^∗(N)∥. From Tyagi et al. [41, Lemma 1,Corollary 1], we have that:

$$\parallel{{\Phi}^{*}(\mathbf H)}\parallel \leq \frac{C_{2} \epsilon d m_{\mathcal{X}} k^{2}}{2\sqrt{m_{\Phi}}}(1+\delta)^{1/2} $$

holds with probability at least $1-2 e^{-m_{\Phi }q(\delta ) + 4k(d+m_{\mathcal {X}}+1)u(\delta )}$ where δ is such that $\delta _{4k} < \delta < \sqrt {2}-1$. Next we note that $\mathbf {N} = [N_{1} N_{2} {\dots } N_{m_{\Phi }}]$ where

$$N_{i} = \underbrace{\frac{1}{\epsilon}\sum\limits_{j=1}^{m_{\mathcal{X}}} \eta_{j}}_{L_{1,i}} - \underbrace{\frac{1}{\epsilon}\sum\limits_{j=1}^{m_{\mathcal X}}\eta_{i,j}}_{L_{2,i}} $$

with $\mathbf {L_{1}} = [L_{1,1} {\dots } L_{1,m_{\Phi }}]$ and $\mathbf {L_{2}} = [L_{2,1} {\dots } L_{2,m_{\Phi }}]$ so that N=L ₁−L ₂. We then have that ∥Φ^∗(N)∥≤∥Φ^∗(L ₁)∥+∥Φ^∗(L ₂)∥. By using Lemma 1.1 of Candes et al. [14] and denoting $m=\max \left \{d,m_{\mathcal X}\right \}$ we first have that:

$$ \parallel{{\Phi}^{*}(\mathbf{L_{1}})}\parallel \leq \frac{2\gamma\sigma}{\epsilon} \sqrt{(1+\delta) m_{\Phi} m_{\mathcal{X}} m} $$

(51)

holds with probability at least 1 − 2e ^−cm where $c = \frac {\gamma ^{2}}{2} - 2\log 12$ and $\gamma \;>\;2\sqrt {\log 12}$. This can be verified using the proof technique of Candes et al. [14, Lemma 1.1]. Care has to be taken of the fact that the entries of L ₁ are correlated as they are identical copies of the same Gaussian random variable $\frac {1}{\epsilon }{\sum }_{j=1}^{m_{\mathcal X}} \eta _{j}$. Furthermore we also have that:

$$ \parallel{{\Phi}^{*}(\mathbf{L_{2}})}\parallel \leq \frac{2\gamma\sigma}{\epsilon} \sqrt{(1+\delta) m_{\mathcal{X}} m} $$

(52)

holds with probability at least 1−2e ^−cm with constants c,γ as defined earlier. This is again easily verifiable using the proof technique of Candes et al. [14, Lemma 1.1], as the entries of L ₂ are i.i.d Gaussian random variables. Combining (51) and (52) we then have that the following holds true with probability at least 1−4e ^−cm.

$$ \parallel{{\Phi}^{*}(\mathbf{L_{1}})}\parallel + \parallel{{\Phi}^{*}(\mathbf{L_{2}})}\parallel \leq \frac{4\gamma\sigma}{\epsilon} \sqrt{(1+\delta) m_{\mathcal{X}} m_{\Phi} m}. $$

(53)

Lastly, it is fairly easy to see that $\parallel {\widehat {\mathbf {X}}_{DS}^{(k)} - \mathbf {X}}\parallel _{F} \leq 2 \parallel {\widehat {\mathbf {X}}_{DS} - \mathbf {X}}\parallel _{F}$ where $\widehat {\mathbf {X}}_{DS}^{(k)}$ is the best rank k approximation to $\widehat {\mathbf {X}}_{DS}$ (see for example, the proof by Tyagi et al. [41, Corollary 1]). Combining the above observations we arrive at the stated error bound with probability at least $1-2 e^{-m_{\Phi }q(\delta ) + 4k(d+m_{\mathcal {X}}+1)u(\delta )} - 4 e^{-c m}$. □

1.3 A.3 Proof of Lemma 5

Proof

Let τ denote the bound on $\parallel {\widehat {\mathbf {X}}_{DS}^{(k)} - \mathbf {X}}\parallel _{F}$ as stated in Lemma 4. We now make use of a result by Tyagi et al. [41, Lemma 2]. This states that if $\tau < \frac {\sqrt {(1-\rho )m_{\mathcal {X}}\alpha k}}{\sqrt {k}+\sqrt {2}}$ holds, then it implies that

$$ \parallel{\widehat{\mathbf{A}}^{T}\widehat{\mathbf{A}} - \mathbf{A}^{T}\mathbf{A}}\parallel_{F} \leq \frac{2\tau}{\sqrt{(1-\rho)m_{\mathcal{X}}\alpha} - \tau} $$

(54)

holds true for any 0 < ρ < 1 with probability at least $1-k\exp \left (-\frac {m_{\mathcal {X}}\alpha \rho ^{2}}{2k {C_{2}^{2}}}\right )$. The proof makes use of Weyl’s inequality [45], Wedin’s perturbation bound [44] and a deviation bound for the extremal eigenvalues of the sum of random positive semidefinite matrices [39].

Therefore, upon using the value of τ we have that $\tau < f \frac {\sqrt {(1-\rho )m_{\mathcal {X}}\alpha k}}{\sqrt {k}+\sqrt {2}}$ holds for any 0<f<1 if:

$$\begin{array}{@{}rcl@{}} C_{0}^{1/2} k^{1/2} (1+\delta)^{1/2} \left(\frac{C_{2}\epsilon d m_{\mathcal{X}} k^{2}}{\sqrt{m_{\Phi}}} + \frac{8\gamma\sigma\sqrt{m_{\mathcal X} m_{\Phi} m}}{\epsilon}\right) &<& f \frac{\sqrt{(1-\rho)m_{\mathcal{X}}\alpha k}}{\sqrt{k}+\sqrt{2}} \end{array} $$

(55)

$$\begin{array}{@{}rcl@{}} \Leftrightarrow \overbrace{C_{2} d k^{2}}^{a_{1}} \epsilon \sqrt{\frac{m_{\mathcal{X}}}{m_{\Phi}}} + \frac{8\gamma\sigma\sqrt{m_{\Phi} m}}{\epsilon} &<& f \left(\overbrace{\frac{1}{C_{0}^{1/2} (1+\delta)^{1/2}} \frac{\sqrt{(1-\rho)\alpha}}{\sqrt{k}+\sqrt{2}}}^{b_{1}}\right) \end{array} $$

(56)

$$\begin{array}{@{}rcl@{}} \Leftrightarrow a_{1} \sqrt{\frac{m_{\mathcal{X}}}{m_{\Phi}}} \epsilon^{2} - f b_{1} \epsilon + 8\gamma\sigma\sqrt{m_{\Phi} m} &<& 0. \end{array} $$

(57)

From (57) we get the stated condition on 𝜖. Lastly upon using $\tau < \frac {f\sqrt {(1-\rho )m_{\mathcal X}\alpha k}}{\sqrt {k}+\sqrt {2}}$ in (54) we obtain the stated bound on $\parallel {\widehat {\mathbf {A}}^{T}\widehat {\mathbf {A}} - \mathbf {A}^{T}\mathbf {A}}\parallel _{F}$. □