DCOB: Action space for reinforcement learning of high DoF robots

Abstract

Reinforcement learning (RL) for robot control is an important technology for future robots since it enables us to design a robot’s behavior using the reward function. However, RL for high degree-of-freedom robot control is still an open issue. This paper proposes a discrete action space DCOB which is generated from the basis functions (BFs) given to approximate a value function. The remarkable feature is that, by reducing the number of BFs to enable the robot to learn quickly the value function, the size of DCOB is also reduced, which improves the learning speed. In addition, a method WF-DCOB is proposed to enhance the performance, where wire-fitting is utilized to search for continuous actions around each discrete action of DCOB. We apply the proposed methods to motion learning tasks of a simulated humanoid robot and a real spider robot. The experimental results demonstrate outstanding performance.

Appendices

Appendix

Appendix A Wire-fitting

For a continuous state $\mathbf{x}\in \mathcal X$ and a continuous action $\mathbf{u}\in \mathcal U$, wire-fitting is defined as:

$$\begin{aligned} Q(\mathbf{x},\mathbf{u})&= \lim _{\epsilon \rightarrow 0^+} \frac{\sum _{i\in \mathcal W } (d_i+\epsilon )^{-1}q_i(\mathbf{x})}{\sum _{i\in \mathcal W } (d_i+\epsilon )^{-1}} , \end{aligned}$$

(35)

$$\begin{aligned} d_i&= \Vert \mathbf{u}-\mathbf{u}_i(\mathbf{x}) \Vert ^2 + C\bigl [\max _{i^{\prime }\in \mathcal W } (q_{i^{\prime }}(\mathbf{x})) - q_i(\mathbf{x})\bigr ]. \end{aligned}$$

(36)

Here, a pair of the functions $q_i(\mathbf{x}):\mathcal X \rightarrow \mathbb R$ and $\mathbf{u}_i(\mathbf{x}):\mathcal X \rightarrow \mathcal U$ ($i\in \mathcal W$) is called a control wire; wire-fitting is regarded as an interpolator of the set of control wires $\mathcal W$. $C$ is the smoothing factor of the interpolation; we choose $C=0.001$ in the experiments. Any function approximator is available for $q_i(\mathbf{x})$ and $\mathbf{u}_i(\mathbf{x})$. For any kind of the function approximators, one of $q_i(\mathbf{x})$, $i\in \mathcal W$ is equal to $\max _{\mathbf{u}}{Q(\mathbf{x},\mathbf{u})}$ and the corresponding $\mathbf{u}_i(\mathbf{x})$ is the greedy action at $\mathbf{x}$.

$$\begin{aligned}&\max _{\mathbf{u}}{Q(\mathbf{x},\mathbf{u})} = \max _{i\in \mathcal W } (q_i(\mathbf{x})), \end{aligned}$$

(37)

$$\begin{aligned}&\arg \,\max _{\mathbf{u}}{Q(\mathbf{x},\mathbf{u})} = \mathbf{u}_i(\mathbf{x})\Big |_{i=\arg \,\max _{i^{\prime }\in \mathcal W }(q_{i^{\prime }}(\mathbf{x}))}. \end{aligned}$$

(38)

Namely, the greedy action at state $\mathbf{x}$ is calculated only by evaluating $q_{i}(\mathbf{x})$ for $i\in \mathcal W$.

We use NGnet for $q_{i}(\mathbf{x})$ and a constant vector for $\mathbf{u}_{i}(\mathbf{x})$, that is, we let $q_i(\mathbf{x})= {\mathbf{\theta }}_i^\top {\mathbf{\phi }}(\mathbf{x})$ and $\mathbf{u}_i(\mathbf{x})= \mathbf{U}_i$, where ${\mathbf{\phi }}(\mathbf{x})$ is the output vector of the NGnet. The parameter vector ${\mathbf{\theta }}$ is defined as ${\mathbf{\theta }}^\top = ({\mathbf{\theta }}_1^\top , \mathbf{U}_1^\top , {\mathbf{\theta }}_2^\top , \mathbf{U}_2^\top , \dots {}, {\mathbf{\theta }}_{|\mathcal W |}^\top , \mathbf{U}_{|\mathcal W |}^\top )$, and the gradient $\mathbf{\nabla }_{\mathbf{\theta }} Q(\mathbf{x},\mathbf{u})$ can be calculated analytically.

Figure 18 shows an example of wire-fitting where both of $\mathbf{x}\in [-1,1]$ and $\mathbf{u}\in [-1,1]$ are a one-dimensional vector. There are two control wires (dashed lines) and three basis functions (dotted lines). The BFs (NGnet) are located at $\mathbf{x}=(-1),(0),(1)$ respectively, and the parameters of the wire-fitting are ${\mathbf{\theta }}_1=(0.0, 0.6, 0.0)^\top , \mathbf{U}_1=(-0.5), {\mathbf{\theta }}_2=(0.0, 0.3, 0.6)^\top , \mathbf{U}_2=(0.5)$. Each control wire is plotted as $(\mathbf{x}, \mathbf{u}_{1}(\mathbf{x}), q_{1}(\mathbf{x}))$ and $(\mathbf{x}, \mathbf{u}_{2}(\mathbf{x}), q_{2}(\mathbf{x}))$ respectively. Each $\times$-mark is put at $(\mathbf{x}, \mathbf{u}_{i^\star }(\mathbf{x}), q_{i^\star }(\mathbf{x}))\big |_{i^\star =\arg \,\max _{i}q_{i}(\mathbf{x})}$ which shows the greedy action at $\mathbf{x}$.

Fig. 18

Example of wire-fitting

Appendix B Calculations of BFTrans

1.1 Generating trajectory

The reference trajectory $\mathbf{q}^\mathrm{{D}}(t_n+t_a),\>{}{} t_a\in [0,T_{\mathrm{F}}]$ is designed so that the state changes from the starting state $\mathbf{x}_n=\mathbf{x}(t_n)$ to the target $\mathbf{q}^\mathrm{trg}$ in the time interval $T_{\mathrm{F}}$. We represent the trajectory with a cubic function,

$$\begin{aligned} {\mathbf{q}}^{\mathrm{D}}(t_n+t_a)= \mathbf{c}_0+ \mathbf{c}_1 t_a+ \mathbf{c}_2 t_a^2+ \mathbf{c}_3 t_a^3, \end{aligned}$$

(39)

where $\mathbf{c}_{0,\dots {},3}$ are the coefficient vectors. These coefficients are determined by the boundary conditions,

$$\begin{aligned}&\mathbf{q}^\mathrm{{D}}(t_n)=\mathbf{C}_{\mathrm{{P}}}(\mathbf{x}_n) ,\quad \mathbf{q}^\mathrm{{D}}(t_n+T_{\mathrm{F}})=\mathbf{q}^\mathrm{trg}, \nonumber \\&\dot{\mathbf{q}}^\mathrm{{D}}(t_n+T_{\mathrm{F}})=\mathbf{0},\quad {\ddot{\mathbf{q}}}^\mathrm{{D}}(t_n+T_{\mathrm{F}})=\mathbf{0}, \end{aligned}$$

(40)

where $\mathbf{0}$ denotes a zero vector.

1.2 Abbreviating trajectory

The abbreviation is performed as follows: (1) estimate $D_\mathrm{{N}}(\mathbf{x}_n)$ as the distance between two neighboring BFs around the start state $\mathbf{x}_n$, (2) calculate $T_{\mathrm{N}}$ from the ratio of $D_\mathrm{{N}}(\mathbf{x}_n)$ and the distance between $\mathbf{x}_n$ and $\mathbf{q}^\mathrm{trg}$.

To define $D_\mathrm{{N}}(\mathbf{x}_n)$, for each BF $k$, we first calculate $d_\mathrm{{N}}(k)$ as the distance between its center ${\mathbf{\mu }}_{k}$ and the center of the nearest BF from $k$. Then, we estimate $D_\mathrm{{N}}(\mathbf{x}_n)$ by interpolating $\{d_\mathrm{{N}}(k)|k\in \mathcal{K }\}$ with the output of the BFs at $\mathbf{x}_n$.

$d_\mathrm{{N}}(k)$ is calculated by

$$\begin{aligned} k_\mathrm{{N}}(k)&= \text{ arg } \text{ min }_{k^{\prime }\in \mathcal{K }, k^{\prime }\ne k} \Vert \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k^{\prime }}) - \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k})\Vert _\infty , \end{aligned}$$

(41)

$$\begin{aligned} d_\mathrm{{N}}(k)&= \max {}\bigl ( \Vert \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k_\mathrm{{N}}(k)}) - \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k})\Vert _\infty ,\>{}\text{ d }_{\min {}k} \bigr ), \end{aligned}$$

(42)

where $\text{ d }_{\min {}k}\in \mathbb R$ is a positive constant to adjust $d_\mathrm{{N}}(k)$ when $\Vert \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k_\mathrm{{N}}(k)}) - \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k})\Vert _\infty$ is too small. For NGnet, we define it as $\text{ d }_{\min {}k}= \sqrt{\lambda _{k}^\mathcal Q }$ where $\lambda _{k}^\mathcal Q$ is the maximum eigenvalue of the covariance matrix $\varvec{\varSigma }_k^\mathcal Q$ on the $\mathcal Q$ space⁹. Note that we can pre-compute $\{d_\mathrm{{N}}(k)|k\in \mathcal{K }\}$ for fixed BFs.

Using the output of BFs ${\mathbf{\phi }}(\mathbf{x}_n)$, $D_\mathrm{{N}}(\mathbf{x}_n)$ is estimated by

$$\begin{aligned} D_\mathrm{{N}}(\mathbf{x}_n) = (d_\mathrm{{N}}(1),\>{}d_\mathrm{{N}}(2),\ldots ,\>{}d_\mathrm{{N}}(|\mathcal{K }|))^\top {\mathbf{\phi }}(\mathbf{x}_n) \end{aligned}$$

(43)

Finally, $T_{\mathrm{N}}$ is defined by

$$\begin{aligned} T_{\mathrm{N}}(\mathbf{x}_n,\mathbf{u}_n) = \min \Bigl (1,\frac{\text{ F }_\mathrm{abbrv} D_\mathrm{{N}}(\mathbf{x}_n)}{\Vert \mathbf{q}^\mathrm{trg}-\mathbf{C}_{\mathrm{{P}}}(\mathbf{x}_n)\Vert _\infty }\Bigr ) T_{\mathrm{F}}. \end{aligned}$$

(44)

Appendix C Initialization and constraints of WF-DCOB

1.1 Initializing wire-fitting parameters

For a control wire $i\in \mathcal W$, we use $a_{i}^{\mathrm{dcob}}$ to denote the corresponding action in DCOB: $a_{i}^{\mathrm{dcob}} = (g_{i}^{\mathrm{dcob}}, k_{i}^{\mathrm{dcob}})$. Let $(\text{ g }^\mathrm{{S}}_i, \text{ g }^\mathrm{{E}}_i)$ denote the range of the interval factor which includes $g_{i}^{\mathrm{dcob}}$. For each control wire $i\in \mathcal W$, its parameter is defined as $\mathbf{U}_i=(g_i,\mathbf{q}^\mathrm{trg}_i)$ and is initialized by

$$\begin{aligned}&g_i \leftarrow \frac{\text{ g }^\mathrm{{S}}_i + \text{ g }^\mathrm{{E}}_i}{2}, \end{aligned}$$

(45a)

$$\begin{aligned}&\mathbf{q}^\mathrm{trg}_i \leftarrow \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k_{i}^{\mathrm{dcob}}}). \end{aligned}$$

(45b)

The other parameters of the control wires $\{{\mathbf{\theta }}_i | i\in \mathcal W \}$ are initialized by zero, since, in a learning-from-scratch case, we do not have prior knowledge of the action values.

1.2 Constraints on wire-fitting parameters

For $\mathbf{U}_i=(g_i,\mathbf{q}^\mathrm{trg}_i)$, the interval factor $g_i$ is constrained inside $(\text{ g }^\mathrm{{S}}_i, \text{ g }^\mathrm{{E}}_i)$, and the target point $\mathbf{q}^\mathrm{trg}_i$ is constrained inside a hypersphere of radius $d_\mathrm{{N}}(k_{i}^{\mathrm{dcob}})$ centered at $\mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k_{i}^{\mathrm{dcob}}})$. Here, $d_\mathrm{{N}}(k_{i}^{\mathrm{dcob}})$ denotes the distance to the nearest BF from $k_{i}^{\mathrm{dcob}}$ defined by Eq. 42. Specifically, the parameter $\mathbf{U}_i=(g_i,\mathbf{q}^\mathrm{trg}_i)$ of each control wire $i\in \mathcal W$ is constrained by

(46)

where

$$\begin{aligned} \mathbf{diff} \triangleq \mathbf{q}^\mathrm{trg}_i - \mathbf{C}_{\mathrm{{P}}}({\mathbf{\mu }}_{k_{i}^{\mathrm{dcob}}}). \end{aligned}$$

(47)

These constraints are applied after each update of an RL algorithm.