IOB: integrating optimization transfer and behavior transfer for multi-policy reuse

Abstract

Humans have the ability to reuse previously learned policies to solve new tasks quickly, and reinforcement learning (RL) agents can do the same by transferring knowledge from source policies to a related target task. Transfer RL methods can reshape the policy optimization objective (optimization transfer) or influence the behavior policy (behavior transfer) using source policies. However, selecting the appropriate source policy with limited samples to guide target policy learning has been a challenge. Previous methods introduce additional components, such as hierarchical policies or estimations of source policies’ value functions, which can lead to non-stationary policy optimization or heavy sampling costs, diminishing transfer effectiveness. To address this challenge, we propose a novel transfer RL method that selects the source policy without training extra components. Our method utilizes the Q function in the actor-critic framework to guide policy selection, choosing the source policy with the largest one-step improvement over the current target policy. We integrate optimization transfer and behavior transfer (IOB) by regularizing the learned policy to mimic the guidance policy and combining them as the behavior policy. This integration significantly enhances transfer effectiveness, surpasses state-of-the-art transfer RL baselines in benchmark tasks, and improves final performance and knowledge transferability in continual learning scenarios. Additionally, we show that our optimization transfer technique is guaranteed to improve target policy learning.

Appendices

Appendix

A.1 Proof of theorem 1

Proof

As $|\widetilde{Q}_{\pi _{tar}}(s,a)-{Q}_{\pi _{tar}}(s,a)|\le \mu \text {\ for\ all}\ s \in \mathcal {S}, a \in A$, we have that for all $s \in \mathcal {S}$, the difference between the true value function $V_{\pi _{tar}}$ and the approximated value function $\widetilde{V}_{\pi _{tar}}$ is bounded:

$$\begin{aligned}&V_{\pi _{tar}}(s)\\&=\mathbb {E}_{a \sim \pi _{tar}(\cdot |s)}\left[ Q_{\pi _{tar}}(s,a)-\alpha \log \pi _{tar}(a|s)\right] \\&\le \mathbb {E}_{a \sim \pi _{tar}(\cdot |s)}\left[ \widetilde{Q}_{\pi _{tar}}(s,a)-\alpha \log \pi _{tar}(a|s)+\mu \right] \\&= \widetilde{V}_{\pi _{tar}}(s)+ \mu . \end{aligned}$$

As $\pi _{tar}(\cdot |s)$ is contained in $\Pi ^s$, with $\widetilde{\pi }_{g}$ defined in Eq. (9), it is obvious that for all $s \in \mathcal {S}$,

$$\begin{aligned} \begin{aligned}&\mathbb {E}_{a \sim \widetilde{\pi }_{g}} (\cdot |s)\left[ \widetilde{Q}_{\pi _{tar}}(s,a)-\alpha \log \widetilde{\pi }_g(a|s)\right] \ge \\&\mathbb {E}_{a \sim \pi _{tar} (\cdot |s)}\left[ \widetilde{Q}_{\pi _{tar}}(s,a)-\alpha \log \pi _{tar}(a|s)\right] =\widetilde{V}_{\pi _{tar}}(s). \end{aligned} \end{aligned}$$

(1)

Then for all $s_i \in \mathcal {S}$,

$$\begin{aligned}&V_{\pi _{tar}}(s_i) \le \widetilde{V}_{\pi _{tar}}(s_i)+ \mu \\&\le \mathbb {E}_{a_i \sim \widetilde{\pi }_{g}(\cdot |s_i)}[\widetilde{Q}_{\pi _{tar}}(s_i,a_i)-\alpha \log \widetilde{\pi }_{g}(a_i|s_i)] + \mu \\&\le \mathbb {E}_{a_i \sim \widetilde{\pi }_{g}}(\cdot |s_i)[{Q}_{\pi _{tar}}(s_i,a_i)-\alpha \log \widetilde{\pi }_{g}(a_i|s_i)]+ 2\mu \\&=\mathbb {E}_{a_i \sim \widetilde{\pi }_{g}}(a|s_i)[r(s_i,a_i)-\alpha \log \widetilde{\pi }_{g}(a_i|s_i) +\gamma V_{\pi _{tar}}(s_{i+1})] \\&\ \ \ + 2\mu \\&\vdots \\&\le \mathbb {E}_{\widetilde{\pi }_{g}}[\sum _{\tau =0}^{\infty }\gamma ^{\tau }(r(s_{i+\tau }, a_{i+\tau })-\alpha \log \widetilde{\pi }_{g}(a_{i+\tau }|s_{i+\tau }))]\\&\ \ \ + 2 \sum _{\tau =0}^{\infty }\gamma ^{\tau }\mu \\&=V_{\widetilde{\pi }_{g}} (s_i)+\frac{2\mu }{1-\gamma }. \end{aligned}$$

A.2 Proof of theorem 2

Proof

According to the Pinsker’s inequality [49], $D_{KL}(\pi _{tar}^{l+1}(\cdot |s)||\widetilde{\pi }_{g}^l(\cdot |s))\ge \frac{1}{2\ln 2}||\pi _{tar}^{l+1}(\cdot |s)-\widetilde{\pi }_{g}^l(\cdot |s)||_1^2$, where $||\cdot ||_1$ is the L1 norm. So we have that for all $s \in \mathcal {S}$, $||\pi _{tar}^{l+1}(\cdot |s)-\widetilde{\pi }_{g}^l(\cdot |s)||_1 \le \sqrt{2\ln 2 \delta }$. According to the Performance Difference Lemma [50], we have that for all $s \in \mathcal {S}$:

$$\begin{aligned}&V_{\widetilde{\pi }_{g}^l}(s)-V_{\pi _{tar}^{l+1}}(s) \\&= \frac{1}{1-\gamma }\mathbb {E}_{s' \sim \rho _{s}^{\widetilde{\pi }_{g}^l}}(s') [\mathbb {E}_{a \sim \widetilde{\pi }_{g}^l(\cdot |s')}[Q_{\pi _{tar}^{l+1}}(s',a)-\alpha \log \widetilde{\pi }_{g}^l(a|s)]\\&\ \ \ -\mathbb {E}_{a \sim \widetilde{\pi }_{tar}^{l+1}(\cdot |s')}[Q_{\pi _{tar}^{l+1}}(s',a) -\alpha \log \widetilde{\pi }_{tar}^{l+1}(a|s)]] \\&\le \frac{1}{1-\gamma }\max \limits _{s' \in \mathcal {S}}[\mathbb {E}_{a \sim \widetilde{\pi }_{g}^l}(\cdot |s')[Q_{\pi _{tar}^{l+1}}(s',a)] \\&\ \ \ -\mathbb {E}_{a \sim \pi _{tar}^{l+1}(\cdot |s')}[Q_{\pi _{tar}^{l+1}}(s',a)]] \\&\ \ \ +\frac{\alpha }{1-\gamma }\max \limits _{s'' \in \mathcal {S}}\left| \mathcal {H} (\widetilde{\pi }_{g}^l(\cdot |s''))-\mathcal {H}(\pi _{tar}^l(\cdot |s''))\right| \\&= \frac{1}{1-\gamma }\max \limits _{s' \in \mathcal {S}}\int \left( \widetilde{\pi }_{g}^l(\cdot |s) -\pi _{tar}^{l+1}(a|s)\right) Q_{\pi _{tar}^{l+1}}(s',a)da \\&\ \ \ +\frac{\alpha }{1-\gamma }\widetilde{\mathcal {H}}_{max} \\&\le \frac{1}{1-\gamma } \max \limits _{s' \in \mathcal {S}}\int \left| \widetilde{\pi }_{g}^l(a|s) -\pi _{tar}^{l+1}(a|s)\right| \cdot \left| Q_{\pi _{tar}^{l+1}}(s',a)\right| da \\&\ \ \ +\frac{\alpha }{1-\gamma }\widetilde{\mathcal {H}}_{max} \\&\le \frac{1}{1-\gamma } \max \limits _{s' \in \mathcal {S}}\int \left| \widetilde{\pi }_{g}^l(a|s) -\pi _{tar}^{l+1}(a|s)\right| \cdot \frac{\widetilde{R}_{max}+\alpha \mathcal {H}_{max}^{l+1}}{1-\gamma }da \\&\ \ \ +\frac{\alpha }{1-\gamma }\widetilde{\mathcal {H}}_{max} \end{aligned}$$

$$\begin{aligned}&= \frac{\widetilde{R}_{max}+\alpha \mathcal {H}_{max}^{l+1}}{(1-\gamma )^2}\max \limits _{s' \in \mathcal {S}}||\widetilde{\pi }_{g}^l(\cdot |s)-\pi _{tar}^{l+1}(\cdot |s)||_1 \nonumber \\&\ \ \ +\frac{\alpha }{1-\gamma }\widetilde{\mathcal {H}}_{max} \nonumber \\&\le \frac{\sqrt{2\ln 2\delta }(\widetilde{R}_{max}+\alpha \mathcal {H}_{max}^{l+1}) +\alpha (1-\gamma )\widetilde{\mathcal {H}}_{max}}{(1-\gamma )^2}, \end{aligned}$$

(2)

where $\rho _{s}^{\widetilde{\pi }_{g}^l}(s')=(1-\gamma )\sum _{t=0}^{\infty }\gamma ^tp(s_t=s'|s_0=s,\widetilde{\pi }_{g}^l)$ is the normalized discounted state occupancy distribution. Note that

$$\begin{aligned}&|Q_{\pi _{tar}^{l+1}}(s,a)|\nonumber \\&=|\mathbb {E}_{\pi _{tar}^{l+1}}[\sum _{i=0}^{\infty }\gamma ^i(r(s_{\tau +i},a_{\tau +i}) \nonumber \\&\ \ \ -\alpha \log \pi _{tar}^{l+1}(\cdot |s))|s_\tau =s,a_\tau =a]| \nonumber \\&\le \mathbb {E}_{\pi }[\sum _{i=0}^{\infty }\gamma ^i(\widetilde{R}_{max}+\gamma \mathcal {H}_{max}^{l+1})] \end{aligned}$$

(3)

$$\begin{aligned}&=\frac{\widetilde{R}_{max}+\alpha \mathcal {H}_{max}^{l+1}}{1-\gamma }. \end{aligned}$$

(4)

Eventually, we have

$$\begin{aligned}&V_{\pi _{tar}^{l+1}}(s) \nonumber \\&\ge V_{\widetilde{\pi }_{g}^l}(s) - \frac{\sqrt{2\ln 2\delta }(\widetilde{R}_{max} +\alpha \mathcal {H}_{max}^{l+1})+\alpha (1-\gamma )\widetilde{\mathcal {H}}_{max}}{(1-\gamma )^2} \nonumber \\&\ge V_{\pi _{tar}^l}(s)-\frac{\sqrt{2\ln 2\delta }(\widetilde{R}_{max} +\alpha \mathcal {H}_{max}^{l+1})}{(1-\gamma )^2}- \frac{2\mu +\alpha \widetilde{\mathcal {H}}_{max}}{1-\gamma }. \end{aligned}$$

(5)