Skip to main content
SpringerLink
Log in

Solution to reinforcement learning problems with artificial potential field

  • Published:
Journal of Central South University of Technology Aims and scope Submit manuscript

Abstract

A novel method was designed to solve reinforcement learning problems with artificial potential field. Firstly a reinforcement learning problem was transferred to a path planning problem by using artificial potential field(APF), which was a very appropriate method to model a reinforcement learning problem. Secondly, a new APF algorithm was proposed to overcome the local minimum problem in the potential field methods with a virtual water-flow concept. The performance of this new method was tested by a gridworld problem named as key and door maze. The experimental results show that within 45 trials, good and deterministic policies are found in almost all simulations. In comparison with WIERING’s HQ-learning system which needs 20 000 trials for stable solution, the proposed new method can obtain optimal and stable policy far more quickly than HQ-learning. Therefore, the new method is simple and effective to give an optimal solution to the reinforcement learning problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. KAELBLING L P, LITTMAN M L, MOORE A W. Reinforcement learning: A survey [J]. Journal of Artificial Intelligence Research, 1996, 4(1): 273–285.

    Google Scholar 

  2. SUTTON R S, BARTO A. Reinforcement learning: An introduction [M]. Cambridge: MIT Press, 1998.

    MATH  Google Scholar 

  3. BANERJEE B, STONE P. General game learning using knowledge transfer [C]// Proceedings of the 20th International Joint Conference on Artificial Intelligence. California: AAAI Press, 2007: 672–677.

    Google Scholar 

  4. ASADI M, HUBER M. Effective control knowledge transfer through learning skill and representation hierarchies [C]// Proceedings of the 20th International Joint Conference on Artificial Intelligence. California: AAAI Press, 2007: 2054–2059.

    Google Scholar 

  5. KONIDARIS G, BARTO A. Autonomous shaping: Knowledge transfer in reinforcement learning [C]// Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh: ACM Press, 2006: 489–496.

    Google Scholar 

  6. MEHTA N, NATARAJAN S, TADEPALLI P, FERN A. Transfer in variable-reward hierarchical reinforcement learning [C]// Workshop on Transfer Learning at Neural Information Processing Systems. Oregon: ACM Press, 2005: 20–23.

    Google Scholar 

  7. WILSON A, FERN A, RAY S, TADEPALLI P. Multi-Task reinforcement learning: A hierarchical Bayesian approach [C]// Proceedings of the 24th International Conference on Machine Learning. Oregon: ACM Press, 2007: 923–930.

    Google Scholar 

  8. GOEL S, HUBER M. Subgoal discovery for hierarchical reinforcement learning using learned policies [C]// Proceedings of the 16th International FLAIRS Conference. Florida: AAAI Press, 2003: 346–350.

    Google Scholar 

  9. TAYOR M E, STONE P. Behavior transfer for value-function-based reinforcement learning [C]// The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems. New York: ACM Press, 2005: 53–59.

    Google Scholar 

  10. HENGST B. Discovering hierarchy in reinforcement learning with HexQ [C]// Proceedings of the 19th International Conference on Machine Learning. San Francisco: Morgan Kaufmann, 2002: 243–250.

    Google Scholar 

  11. DIUK C, STREHL A L, LITTMAN M L. A hierarchical approach to efficient reinforcement learning in deterministic domains [C]// Proceedings of the 5th International Joint Conference on Autonomous Agents and Multiagent Systems. New York: ACM Press, 2006: 313–319.

    Google Scholar 

  12. ZHOU W, COGGINS R. A biologically inspired hierarchical reinforcement learning system [J]. Cybernetics and Systems, 2005, 36(1): 1–44.

    Article  Google Scholar 

  13. BARTO A, MAHADEVAN S. Recent advances in hierarchical reinforcement learning [J]. Discrete Event Dynamic Systems: Theory and Applications, 2003, 13(1): 41–77.

    Article  MathSciNet  Google Scholar 

  14. KEARNS M, KOLLER D. Efficient reinforcement learning in factored MDPs [C]// Proceedings of the 6th International Joint Conference on Artificial Intelligence. Stockholm: Morgan Kaufmann, 1999: 740–747.

    Google Scholar 

  15. WEN Zhi-qiang, CAI Zi-xing. Global path planning approach based on ant colony optimization algorithm [J]. Journal of Central South University of Technology, 2006, 13(6): 707–712.

    Article  Google Scholar 

  16. ZHU Xiao-cai, DONG Guo-hua, CAI Zi-xing. Robust simultaneous tracking and stabilization of wheeled mobile robots not satisfying nonholonomic constraint [J]. Journal of Central South University of Technology, 2007, 14(4): 537–545.

    Article  Google Scholar 

  17. ZOU Xiao-bing, CAI Zi-xing, SUN Guo-rong. Non-smooth environment modeling and global path planning for mobile robots [J]. Journal of Central South University of Technology, 2003, 10(3): 248–254.

    Article  Google Scholar 

  18. ANDREWS J R, HOGAN N. Impedance control as a framework for implementing obstacle avoidance in a manipulator [C]// Proceedings of Control of Manufacturing Process and Robotic System. New York: ASME Press, 1983: 243–251.

    Google Scholar 

  19. KHATIB O. Real-time obstacle avoidance for manipulators and mobile robots [J]. International Journal of Robotics Research, 1986, 5(1): 90–98.

    Article  Google Scholar 

  20. HUANG W H, FAJEN B R, FINK J R. Visual navigation and obstacle avoidance using a steering potential function [J]. Journal of Robotics and Autonomous Systems, 2006, 54(4): 288–299.

    Article  Google Scholar 

  21. PARK M G, LEE M C. Artificial potential field based path planning for mobile robots using a virtual obstacle concept [C]// Proceedings of IEEE/ASME International Conference on Advanced Intelligent Mechatronics. Victoria: IEEE Press, 2003: 735–740.

    Google Scholar 

  22. LIU C Q, KRISHNAN H, YONG L S. Virtual obstacle concept for local-minimum-recovery in potential-field based navigation [C]// Proceedings of the IEEE International Conference on Robotics & Automation. San Francisco: IEEE Press, 2000: 983–988.

    Google Scholar 

  23. BROCK O, KHATIB O. High-speed navigation using the global dynamic window approach [C]// Proceedings of the IEEE International Conference on Robotics and Automation. Detroit: IEEE Press, 1999: 341–346.

    Google Scholar 

  24. KONOLIGE K. A gradient method for real time robot control [C]// Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Victoria: IEEE Press, 2000: 639–646.

    Google Scholar 

  25. RIMON E, KODITSCHEK D. Exact robot navigation using artificial potential functions [J]. IEEE Transactions on Robotics and Automation, 1992, 8(5): 501–518.

    Article  Google Scholar 

  26. WIERING M, SCHMIDHUBER J. HQ-learning [J]. Adaptive Behavior, 1998, 6(2): 219–246.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Mental Health, Xiangya School of Medicine, Central South University, Changsha, 410011, China

    Li-juan Xie  (谢丽娟) & Guang-rong Xie  (谢光荣)

  2. School of Computer and Communication, Changsha University of Science and Technology, Changsha, 410076, China

    Li-juan Xie  (谢丽娟) & Huan-wen Chen  (陈焕文)

  3. Department of Computer Engineering, Hunan College of Information, Changsha, 410200, China

    Huan-wen Chen  (陈焕文)

  4. School of Computer Science, University of Birmingham, Birmingham, B15 2TT, UK

    Xiao-li Li  (李小俚)

Authors
  1. Li-juan Xie  (谢丽娟)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guang-rong Xie  (谢光荣)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huan-wen Chen  (陈焕文)
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao-li Li  (李小俚)
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huan-wen Chen  (陈焕文).

Additional information

Foundation item: Projects(30270496, 60075019, 60575012) supported by the National Natural Science Foundation of China

Rights and permissions

Reprints and permissions

About this article

Cite this article

Xie, Lj., Xie, Gr., Chen, Hw. et al. Solution to reinforcement learning problems with artificial potential field. J. Cent. South Univ. Technol. 15, 552–557 (2008). https://doi.org/10.1007/s11771-008-0104-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11771-008-0104-x

Key words

Search

Navigation