Improving Sample Efficiency in Evolutionary RL Using Off-Policy Ranking

Abstract

Evolution Strategy (ES) is a potent black-box optimization technique based on natural evolution. A key step in each ES iteration is the ranking of candidate solutions based on some fitness score. In the Reinforcement Learning (RL) context, this step entails evaluating several policies. Presently, this evaluation is done via on-policy approaches: each policy’s score is estimated by interacting several times with the environment using that policy. Such ideas lead to wasteful interactions since, once the ranking is done, only the data associated with the top-ranked policies are used for subsequent learning. To improve sample efficiency, we introduce a novel off-policy ranking approach using a local approximation for the fitness function. We demonstrate our idea for two leading ES methods: Augmented Random Search (ARS) and Trust Region Evolution Strategy (TRES). MuJoCo simulations show that, compared to the original methods, our off-policy variants have similar running times for reaching reward thresholds but need only around 70% as much data on average. In fact, in some tasks like HalfCheetah-v3 and Ant-v3, we need just 50% as much data. Notably, our method supports extensive parallelization, enabling our ES variants to be significantly faster than popular non-ES RL methods like TRPO, PPO, and SAC.

ESR was supported by the Prime Minister’s Research Fellowship (PMRF). SK was supported by the SERB Core Research Grant CRG/2021/008115. GT was supported in part by DST-SERB’s Core Research Grant CRG/2021/00833, in part by IISc Start-up grants SG/MHRD-19-0054 and SR/MHRD-19-0040, and in part by the “Pratiksha Trust Young Investigator” award.

Appendices

A Differences in the Original ARS from Our Off-Policy Variant

In this section, we mention the key differences between the original ARS and our off-policy variant OP-ARS. Table 4 indicates the exact steps that differ between the algorithms and their implications.

Table 4. Key differences between ARS and our off-policy variant OP-ARS. The step column refers to the step number in Algorithm 1.

B Hyperparameters

In this section, we would like to describe the set of hyperparameters used in our experiments. The ARS and TRES algorithms have a predefined set of hyperparameters, which have been fine-tuned in the corresponding papers. Hence, we use the same hyperparameters in our algorithms for most of the environments. Our off-policy variants OP-ARS and OP-TRES have two new hyperparameters: the number of trajectories to run using behavior policy $n_b$ and the bandwidth in kernel function h. We experiment with varying values of $n_b$ and h, as indicated in Table 5, as part of our hyperparameter exploration process. The most effective hyperparameters, denoted in bold, are employed to generate the outcomes presented in Table 1 and Fig. 1, including the images in the uppermost row of Fig. 3.

Table 5. Hyperparameter grid used in each environment

C LQR Experiments

In this section, we showcase the outcomes of our experiments conducted with the LQR environment. As elucidated in Sect. 4.3 of [13], there exist limitations inherent to MuJoCo robotic tasks. Particularly notable is the lack of knowledge concerning the optimal policies within these environments. This uncertainty extends to the comparison between the learned policy of their algorithm and the optimal policy. A viable approach involves applying the algorithms to straightforward, widely recognized environments with well-established optimal policies. In [13], the choice fell upon the Linear Quadratic Regulator (LQR) as the benchmarking environment due to its known dynamics. For a more in-depth understanding of this environment, additional insights can be found in [3, Appendix D.2].

We employ the same framework utilized by [13] to compare our approach with model-based Nominal Control, LSPI [9], and ARS [13]. As demonstrated in the work by [13], the Nominal method exhibits significantly greater sample efficiency than LSPI and ARS by orders of magnitude, underscoring the potential for enhancement. Our experiments corroborate this notion, revealing that our method surpasses ARS in terms of sample efficiency, as indicated in Fig. 4a. Furthermore, Fig. 4b underscores our algorithm’s superiority over ARS in terms of stability frequency.

Fig. 4.

Comparison of sample efficiency and stability of various algorithms on LQR.