Deep Deterministic Policy Gradient with Clustered Prioritized Sampling
As a famous deep reinforcement learning approach, deep deterministic policy gradient (DDPG) is able to deal with the problems in the domain of continuous control. To remove temporal correlations among the observed transitions, DDPG uses a sampling mechanism called experience reply which replays transitions at random from the replay buffer. Experience reply removes correlations among different transitions. However, random sampling does not consider the importance of transitions in replay buffer which leads to the longer training time and poor performance. In this paper, we propose a novel efficient sampling mechanism which we call deep deterministic policy gradient with clustered prioritized sampling (CPS-DDPG). CPS-DDPG clusters the transitions by K-means in order to reduce the complexity of the algorithm. In addition, CPS-DDPG samples transitions from different categories according to priorities so as to train targeted transitions. The key idea of CPS-DDPG is to set high priorities to the valuable categories and increase the priorities of the categories that have not been selected for long time appropriately in order to increase the diversity of the transitions. The experimental results show that the proposed model achieves better performance than the traditional deep reinforcement learning model in the continuous domain.
KeywordsReinforcement learning Deep reinforcement learning Deep deterministic policy gradient K-means Prioritized sampling
This work was supported by the National Natural Science Foundation of China (61303108, 61373094, 61772355); Jiangsu College Natural Science Research Key Program (17KJA520004); Program of the Provincial Key Laboratory for Computer Information Processing Technology (Soochow University) (KJS1524); China Scholarship Council Project (201606920013).
- 1.Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with deep reinforcement learning. In: Proceedings of Workshops at the 26th Neural Information Processing Systems, Lake Tahoe, USA (2013)Google Scholar
- 4.Schulman, J., Moritz, P., Levine, S., et al.: High-dimensional continuous control using generalized advantage estimation. In: Proceedings of the International Conference on Learning Representations (2015)Google Scholar
- 5.Watter, M., Springenberg, J.T., Riedmiller, M., et al.: Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in Neural Information Processing System, pp. 2728–2736 (2015)Google Scholar
- 6.Lillicrap, T.P., Hunt, J.J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. In: Proceedings of the International Conference on Learning Representations (2016)Google Scholar
- 7.Silver, D., Lever, G., Heess, N., et al.: Deterministic policy gradient algorithms. In: Proceedings of the International Conference on Machine Learning, pp. 387–395 (2014)Google Scholar
- 8.Heess, N., Wayne, G., Silver, D., et al.: Learning continuous control policies by stochastic value gradients. In: Advances in Neural Information Processing System, pp. 2926–2934 (2015)Google Scholar
- 9.Levine, S., Levine, S., Levine, S., et al.: Continuous deep Q-learning with model-based acceleration. In: Proceedings of the 33th International Conference on Machine Learning, pp. 2829–2838 (2016)Google Scholar
- 10.Levine, S., Abbeel, P.: Learning neural network policies with guided policy search under unknown dynamics. In: Advances in Neural Information Processing Systems, pp. 1071–1079 (2014)Google Scholar
- 15.Mnih, V., Badia, A.P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the International Conference on Machine Learning, New York, USA, pp. 1928–1937 (2016)Google Scholar