Advertisement

Deep Deterministic Policy Gradient with Clustered Prioritized Sampling

  • Wen Wu
  • Fei Zhu
  • YuChen Fu
  • Quan Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11302)

Abstract

As a famous deep reinforcement learning approach, deep deterministic policy gradient (DDPG) is able to deal with the problems in the domain of continuous control. To remove temporal correlations among the observed transitions, DDPG uses a sampling mechanism called experience reply which replays transitions at random from the replay buffer. Experience reply removes correlations among different transitions. However, random sampling does not consider the importance of transitions in replay buffer which leads to the longer training time and poor performance. In this paper, we propose a novel efficient sampling mechanism which we call deep deterministic policy gradient with clustered prioritized sampling (CPS-DDPG). CPS-DDPG clusters the transitions by K-means in order to reduce the complexity of the algorithm. In addition, CPS-DDPG samples transitions from different categories according to priorities so as to train targeted transitions. The key idea of CPS-DDPG is to set high priorities to the valuable categories and increase the priorities of the categories that have not been selected for long time appropriately in order to increase the diversity of the transitions. The experimental results show that the proposed model achieves better performance than the traditional deep reinforcement learning model in the continuous domain.

Keywords

Reinforcement learning Deep reinforcement learning Deep deterministic policy gradient K-means Prioritized sampling 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61303108, 61373094, 61772355); Jiangsu College Natural Science Research Key Program (17KJA520004); Program of the Provincial Key Laboratory for Computer Information Processing Technology (Soochow University) (KJS1524); China Scholarship Council Project (201606920013).

References

  1. 1.
    Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with deep reinforcement learning. In: Proceedings of Workshops at the 26th Neural Information Processing Systems, Lake Tahoe, USA (2013)Google Scholar
  2. 2.
    Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  3. 3.
    Silver, D., Huang, A., Maddison, C.J., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)CrossRefGoogle Scholar
  4. 4.
    Schulman, J., Moritz, P., Levine, S., et al.: High-dimensional continuous control using generalized advantage estimation. In: Proceedings of the International Conference on Learning Representations (2015)Google Scholar
  5. 5.
    Watter, M., Springenberg, J.T., Riedmiller, M., et al.: Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in Neural Information Processing System, pp. 2728–2736 (2015)Google Scholar
  6. 6.
    Lillicrap, T.P., Hunt, J.J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. In: Proceedings of the International Conference on Learning Representations (2016)Google Scholar
  7. 7.
    Silver, D., Lever, G., Heess, N., et al.: Deterministic policy gradient algorithms. In: Proceedings of the International Conference on Machine Learning, pp. 387–395 (2014)Google Scholar
  8. 8.
    Heess, N., Wayne, G., Silver, D., et al.: Learning continuous control policies by stochastic value gradients. In: Advances in Neural Information Processing System, pp. 2926–2934 (2015)Google Scholar
  9. 9.
    Levine, S., Levine, S., Levine, S., et al.: Continuous deep Q-learning with model-based acceleration. In: Proceedings of the 33th International Conference on Machine Learning, pp. 2829–2838 (2016)Google Scholar
  10. 10.
    Levine, S., Abbeel, P.: Learning neural network policies with guided policy search under unknown dynamics. In: Advances in Neural Information Processing Systems, pp. 1071–1079 (2014)Google Scholar
  11. 11.
    O’Neill, J., Pleydell-Bouverie, B., Dupret, D., et al.: Play it again: reactivation of waking experience and memory. Trends Neurosci. 33(5), 220–229 (2010)CrossRefGoogle Scholar
  12. 12.
    Wawrzyński, P., Tanwani, A.K.: Autonomous reinforcement learning with experience replay. Neural Netw. 41(5), 156 (2013)CrossRefGoogle Scholar
  13. 13.
    Kang, S.H., Sandberg, B., Yip, A.M.: A regularized k-means and multiphase scale segmentation. Inverse Probl. Imaging 5(2), 407–429 (2017)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)CrossRefGoogle Scholar
  15. 15.
    Mnih, V., Badia, A.P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the International Conference on Machine Learning, New York, USA, pp. 1928–1937 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Computer Science and TechnologySoochow UniversitySuzhouChina
  2. 2.Provincial Key Laboratory for Computer Information Processing TechnologySoochow UniversitySuzhouChina
  3. 3.School of Computer Science and EngineeringChangshu Institute of TechnologyChangshuChina

Personalised recommendations