Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme

Abstract

We analyze the DQN reinforcement learning algorithm as a stochastic approximation scheme using the o.d.e. (for ‘ordinary differential equation’) approach and point out certain theoretical issues. We then propose a modified scheme called Full Gradient DQN (FG-DQN, for short) that has a sound theoretical basis and compare it with the original scheme on sample problems. We observe a better performance for FG-DQN.

Appendix: Elements of Non-smooth Analysis

The (Frechet) sub/super-differentials of a map $f: \mathcal {R}^d \mapsto \mathcal {R}$ are defined by

$$\begin{aligned} \partial ^-f(x):= & {} \left\{ z \in \mathcal {R}^d: \liminf _{y\rightarrow x}\frac{f(y) - f(x) - \langle z, y-x\rangle }{|x-y|} \ge 0\right\} , \\ \partial ^+f(x):= & {} \left\{ z \in \mathcal {R}^d: \limsup _{y\rightarrow x}\frac{f(y) - f(x) - \langle z, y-x\rangle }{|x-y|} \le 0\right\} , \end{aligned}$$

respectively. Assume f, g is Lipschitz. Some of the properties of $\partial ^{\pm }f$ are as follows.

• (P1) Both $\partial ^-f(x), \partial ^+f(x)$ are closed convex and are nonempty on dense sets.

• (P2) If f is differentiable at x, both equal the singleton $\{\nabla f(x)\}$. Conversely, if both are nonempty at x, f is differentiable at x and they equal $\{\nabla f(x)\}$.

• (P3) $\partial ^-f + \partial ^-g \subset \partial ^-(f + g), \ \partial ^+f + \partial ^+g \subset \partial ^+(f + g) .$

The first two are proved in [4], pp. 30-1. The third follows from the definition. Next consider a continuous function $f: \mathcal {R}^d\times B \mapsto \mathcal {R}$ where B is a compact metric space. Suppose $f( \cdot , y)$ is continuously differentiable uniformly w.r.t. y. Let $\nabla _xf(x,y)$ denote the gradient of $f(\cdot , y)$ at x. Let $g(x) := \max _yf(x,y), h(x) := \min _yf(x,y)$ with

$$ M(x) := \{\nabla _xf(x, y), y \in \text{ Argmax } f(x, \cdot )\} $$

and

$$ N(x) := \{\nabla _xf(x,y), y \in \text{ Argmin } f(x, \cdot )\}. $$

Then N(x), M(x) are compact nonempty subsets of B which are upper semi-continuous in x as set-valued maps. We then have the following general version of Danskin’s theorem [17]:

• (P4) $\partial ^-g(x) = \overline{\text{ co }}(M(x)), \partial ^+g(x) = y$ if $M(x) = \{y\}$, $= \phi$ otherwise, and g has a directional derivative in any direction z given by $\max _{y\in M(x)}\langle y, z\rangle$.

• (P5) $\partial ^+h(x) = \overline{\text{ co }}(N(x)), \partial ^-h(x) = y$ if $N(x) = \{y\}$, $= \phi$ otherwise, and h has a directional derivative in any direction z given by $\min _{y\in N(x)}\langle y, z\rangle$.

The latter is proved in [4], pp. 44-6, the former follows by a symmetric argument.