Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling

Abstract

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked uniformly from a given dataset. Our method results in an O(d) improvement in complexity in comparison to LSTD, where d is the dimension of the data. We provide non-asymptotic bounds for our proposed method, both in high probability and in expectation, under the assumption that the matrix underlying the LSTD solution is positive definite. The latter assumption can be easily satisfied for the pathwise LSTD variant proposed by Lazaric (J Mach Learn Res 13:3041–3074, 2012). Moreover, we also establish that using our method in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. These rate results coupled with the low computational complexity of our method make it attractive for implementation in big data settings, where d is large. A similar low-complexity alternative for least squares regression is well-known as the stochastic gradient descent (SGD) algorithm. We provide finite-time bounds for SGD. We demonstrate the practicality of our method as an efficient alternative for pathwise LSTD empirically by combining it with the least squares policy iteration algorithm in a traffic signal control application. We also conduct another set of experiments that combines the SA-based low-complexity variant for least squares regression with the LinUCB algorithm for contextual bandits, using the large scale news recommendation dataset from Yahoo.

Appendix 1: Proof of Theorem 10.3

The proof of Theorem 10.3 relies on a general rate result built from Proposition 10.1

Proposition 13.1

Under (A1)–(A3) we have, for all $\epsilon \ge 0$ and $\forall n\ge 1$,

$$\begin{aligned}&{\mathbb {P}}( \left\| z_n \right\| _2- {\mathbb {E}}\left\| z_n \right\| _2\ge \epsilon ) \le \exp \left( - \dfrac{\epsilon ^2}{4h(n)^2\sum \limits _{m=1}^{n} L_m^2} \right) , \end{aligned}$$

where $L_i \triangleq \frac{\gamma _i}{n} \left( \sum _{l=i+1}^{n-1}\prod \limits _{j=i}^{l} \left( 1- \mu \gamma _{j+1}( 2 - \varPhi _{\max }^2\gamma _{j+1})) \right) ^{1/2}\right)$, and h(n) is as in Proposition 10.1.

Proof

This proof follows exactly the proof of Proposition 8.3, except that it uses the form of $L_i$ for non-averaged iterates as derived in Proposition 10.1 part (1), rather than as derived in Proposition 8.1 part (1). $\square$

We specialise this result with the choice of step size $\gamma _n \triangleq (c_0 c^{\alpha })/(n+c)^{\alpha }$. First, we prove the form of the $L_i$ constants for this choice of step size in the lemma below.

Lemma 13.1

Under conditions of Theorem 10.3 , we have

$$\begin{aligned} \sum _{i=1}^n L_i^2 \le \frac{1}{\mu ^2} \left\{ 2^\alpha + \left[ \left[ \frac{2\alpha }{ c_0\mu c^{\alpha }}\right] ^{\frac{1}{1-\alpha }} + \frac{2(1 - \alpha )(c_0\mu )^{\alpha }}{\alpha } \right] \right\} ^2\frac{1}{n}. \end{aligned}$$

Second, we bound the expected error by directly averaging the errors of the non-averaged iterates:

$$\begin{aligned} {\mathbb {E}}\left\| {\bar{\theta }}_{n} - {\hat{\theta }}_T\right\| _2\le \frac{1}{n}\sum _{k = 1}^n{\mathbb {E}}\left\| \theta _k - {\hat{\theta }}_T \right\| _2, \end{aligned}$$

(106)

and directly applying the bounds in expectation given in Proposition 8.1.

Lemma 13.2

Under conditions of Theorem 10.3, we have

$$\begin{aligned} {\mathbb {E}}\left\| {\bar{\theta }}_n - {\hat{\theta }}_T\right\| _2\le&C_0\left( C_1\left\| \theta _0 - \theta _T\right\| _2+ 2h(n)c^{\alpha }c_0 \left( 2 c_0 \mu c^{\alpha }\right) ^{\frac{\alpha }{(1-\alpha )}} \sqrt{e}\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{2(1-\alpha )}}\right) \frac{1}{n}\\&+ h(n) c^\alpha c_0 \left( 2c_0\mu c^\alpha \right) ^{\frac{\alpha }{2(1-\alpha )}} (n+c)^{-\frac{\alpha }{2}}, \end{aligned}$$

where $C_0$ and $C_1$ are as defined in Theorem 10.3.

Proof of Lemma 13.1

Recall from the statement of Theorem 10.3 that

$$\begin{aligned} 0< c_0 \varPhi _{\max }^2 < 1. \end{aligned}$$

(107)

Recall also from the formula in Proposition 13.1, that

$$\begin{aligned} L_i = \frac{\gamma _i}{n} \left( \sum _{l=i+1}^{n-1}\prod \limits _{j=i}^{l} \left( 1- \mu \gamma _{j+1}( 2 - \varPhi _{\max }^2\gamma _{j+1})) \right) ^{1/2}\right) . \end{aligned}$$

Notice that

$$\begin{aligned} \sum _{i = 1}^n L_i^2&= \sum _{i = 1}^n\left[ \frac{\gamma _i}{n} \left( \sum _{l=i+1}^{n-1}\prod \limits _{j=i}^{l} \left( 1- \mu \gamma _{j+1}( 2 - \varPhi _{\max }^2\gamma _{j+1})) \right) ^{1/2}\right) \right] ^2\\&\le \frac{1}{n^2}\sum _{i = 1}^n\left[ \gamma _i \left( \sum _{l = i+1}^{n-1} \exp \left( - \sum _{j=i}^l \mu \gamma _{j+1}(2 - \varPhi _{\max }^2\gamma _{j+1})) \right) \right) \right] ^2\\&< \frac{1}{n^2}\sum _{i = 1}^n {\underbrace{ \left[ c_0\left( \frac{c}{c+i}\right) ^\alpha \left( \sum _{l = i+1}^{n-1} \exp \left( - c_0\mu \sum _{j=i}^l \left( \frac{c}{c+j}\right) ^\alpha \right) \right) \right] }_{\triangleq (A)}}^2. \end{aligned}$$

To produce the final bound, we bound the summand (A) highlighted in line (91) by a constant, uniformly over all values of i and n, exactly as in the proof of Lemma 8.1. Thus, we have

$$\begin{aligned} \sum _{i=1}^n L_i^2 \le \frac{1}{\mu ^2} \left\{ 2^\alpha + \left[ \left[ \frac{2\alpha }{ c_0\mu c^{\alpha }}\right] ^{\frac{1}{1-\alpha }} + \frac{2(1 - \alpha )(c_0\mu )^{\alpha }}{\alpha } \right] \right\} ^2\frac{1}{n}. \end{aligned}$$

The rest of the proof follows that of Theorem 4.2. $\square$

Proof of Lemma 13.2

Recall that $\gamma _n \triangleq c_0\left( \frac{c}{(c+n)}\right) ^{\alpha }$. Recall that in Theorem 10.3 we have assumed that

$$\begin{aligned} 0< c_0 \varPhi _{\max }^2 < 1. \end{aligned}$$

(108)

Using (99), we have

$$\begin{aligned}&{\mathbb {E}}\left( \left\| \theta _n - {\hat{\theta }}_T\right\| _2\right) ^2\nonumber \\&\quad \le \left[ \prod _{k = 1}^n \left( 1 - \mu \gamma _k(2 - \gamma _k\varPhi _{\max }^2\right) \left\| z_0\right\| _2\right] ^2 + 4\sum _{k=1}^{n}\gamma _k^2 \left[ \prod _{j = k}^{n-1} (1 - \mu \gamma _j(2 - \gamma _j\varPhi _{\max }^2) \right] ^2 h(k)^2\nonumber \\&\quad \le \left[ \prod _{k = 1}^n \left( 1 - \frac{\mu c_0 c^{\alpha }}{(c+k)^{\alpha }}\right) \left\| z_0\right\| _2\right] ^2 + 4\sum _{k=1}^{n}\frac{c_0^2 c^{2\alpha }}{(c+k)^{2\alpha }} \left[ \prod _{j = k}^{n-1} \left( 1 - \frac{\mu c_0 c^{\alpha }}{(c+j)^{\alpha }}\right) \right] ^2 h(k)^2 \nonumber \\&\quad \le \left[ \exp \left( -\mu c_0 \sum _{k = 1}^n \frac{c^{\alpha }}{(c+k)^{\alpha }}\right) \left\| z_0\right\| _2\right] ^2 + 4h(n)^2\sum _{k=1}^{n}\frac{c_0^2 c^{2\alpha }}{(c+k)^{2\alpha }} \exp \left( -2\mu c_0\sum _{j = k}^{n-1} \frac{ c^{\alpha }}{(c+j)^{\alpha }}\right) . \end{aligned}$$

(109)

To obtain (109), we have applied (108). For the final inequality, we have exponentiated the logarithm of the products, and used the inequality $\ln (1+x) < x$ in several places.

Continuing the derivation, we have

$$\begin{aligned}&{\mathbb {E}}\left\| \theta _n - {\hat{\theta }}_T \right\| _2\end{aligned}$$

(110)

$$\begin{aligned}&\quad \le \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha } - c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - {\hat{\theta }}_T\right\| _2\nonumber \\&\quad \quad +\, 2h(n)\left( \sum _{k = 1}^{n}c_0^2\left( \frac{c}{k+c}\right) ^{2\alpha } \exp \left( -2c_0\mu c^\alpha ((n+c)^{1-\alpha } - (k+c)^{1-\alpha }\right) \right) ^{\frac{1}{2}} \end{aligned}$$

(111)

$$\begin{aligned}&\quad = \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha }\right) \nonumber \\&\quad \quad \times \, \Bigg [\exp \left( c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - {\hat{\theta }}_T\right\| _2\nonumber \\&\qquad +\, 2h(n) \left\{ \sum _{k = 1}^{n}c_0^2\left( \frac{c}{k+c}\right) ^{2\alpha } \exp \left( 2c_0\mu c^\alpha ((k+c)^{1-\alpha }\right) \right\} ^{\frac{1}{2}} \Bigg ]\nonumber \\&\quad \le \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha }\right) \nonumber \\&\quad \quad \times \, \Bigg [\exp \left( c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - {\hat{\theta }}_T\right\| _2\nonumber \\&\qquad +\, 2 h(n) \left\{ c^{2\alpha } c_0^2\int _{1}^{n+c}x^{-2\alpha }\exp \left( 2c_0\mu c^\alpha x^{1-\alpha }\right) dx \right\} ^{\frac{1}{2}} \Bigg ] \end{aligned}$$

(112)

$$\begin{aligned}&\quad \le \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha }\right) \nonumber \\&\quad \quad \times \, \Bigg [\exp \left( c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - {\hat{\theta }}_T\right\| _2\nonumber \\&\quad \quad +\, 2 h(n) \left\{ c^{2\alpha } c_0^2\left( 2c_0\mu c^\alpha \right) ^{\frac{2\alpha }{1-\alpha }} \times \int _{\left( 2c_0\mu c^\alpha \right) ^{1/(1-\alpha )}}^{(n+c)\left( 2c_0\mu c^\alpha \right) ^{1/(1-\alpha )}} y^{-2\alpha }\exp (y^{1-\alpha })dy \right\} ^{\frac{1}{2}} \Bigg ] \end{aligned}$$

(113)

As in the proof of Theorem 5.1, for arriving at (111), we have used Jensen’s inequality, and that $\sum _{j=k}^{n-1}(c+j)^{-\alpha }\ge \int _{j=k}^n(c+j)^{1-\alpha }dj=(c+n)^{1-\alpha } - (c+k)^{1-\alpha }$. To obtain (112), we have upper bounded the sum with an integral, the validity of which follows from the observation that $x\mapsto x^{-2\alpha }e^{x^{1-\alpha }}$ is convex for $x\ge 1$. Finally, for (113), we have applied the change of variables $y = (2c_0\mu c^\alpha )^{1/(1-\alpha )}x$.

Now, since $y^{-2\alpha } \le \frac{2}{1-\alpha } ((1-\alpha )y^{-2\alpha } - \alpha y^{-(1+\alpha )})$ when $y\ge \left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}$, we have

$$\begin{aligned}&\int _{\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}} ^{(n+c)\left( 2c_0\mu c^\alpha \right) ^{1/(1-\alpha )}} y^{-2\alpha }\exp (y^{1-\alpha })dy\\&\quad \le \frac{2}{1-\alpha } \int _{\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}} ^{(n+c)\left( 2c_0\mu c^\alpha \right) ^{1/(1-\alpha )}} ((1-\alpha )y^{-2\alpha } - \alpha y^{-(1+\alpha )}) \exp (y^{1-\alpha })dy\\&\quad \le \frac{2}{1-\alpha } \exp \left( 2c_0\mu c^\alpha (n+c)^{1-\alpha }\right) (n+c)^{-\alpha }\left( 2c_0\mu c^\alpha \right) ^{-\alpha /(1-\alpha )} \end{aligned}$$

and furthermore, since $y\mapsto y^{-2\alpha }\exp (y^{1-\alpha })$ is decreasing for $y\le \left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}$, we have

$$\begin{aligned} \int _{1}^{\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}} y^{-2\alpha }\exp (y^{1-\alpha })dy \le e \left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{1-\alpha }}. \end{aligned}$$

Plugging these into (113), we obtain

$$\begin{aligned}&{\mathbb {E}}\left\| \theta _n - {\hat{\theta }}_T \right\| _2\le \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha }\right) \\&\quad \times \, \left( \exp \left( c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - \theta _T\right\| _2+ 2h(n)c^{\alpha }c_0 \left( 2 c_0 \mu c^{\alpha }\right) ^{\frac{\alpha }{(1-\alpha )}} \sqrt{e}\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{2(1-\alpha )}}\right) \\&\quad +\, 2 h(n)c^\alpha c_0 \left( 2c_0\mu c^\alpha \right) ^{\frac{\alpha }{2(1-\alpha )}} (n+c)^{-\frac{\alpha }{2}}. \end{aligned}$$

Hence, we obtain

$$\begin{aligned}&{\mathbb {E}}\left\| {\bar{\theta }}_n - {\hat{\theta }}_T\right\| _2\le \left( \sum _{n=1}^{\infty } \exp \left( -c_0\mu c^\alpha (n+c)^{1-\alpha }\right) \right) \\&\quad \times \, \left( \exp \left( c_0\mu c^\alpha (1+c)^{1-\alpha }\right) \left\| \theta _0 - \theta _T\right\| _2+ 2h(n)c^{\alpha }c_0 \left( 2 c_0 \mu c^{\alpha }\right) ^{\frac{\alpha }{(1-\alpha )}} \sqrt{e}\left( \frac{2\alpha }{1-\alpha }\right) ^{\frac{1}{2(1-\alpha )}}\right) \frac{1}{n}\\&\quad +\, 2 h(n) c^\alpha c_0 \left( 2c_0\mu c^\alpha \right) ^{\frac{\alpha }{2(1-\alpha )}} (n+c)^{-\frac{\alpha }{2}}. \end{aligned}$$

$\square$