On the Analysis of Parameter Convergence for Temporal Difference Learning of an Exemplar Balance Problem
Bipedal walking/locomotion is a challenging control problem but also an interesting problem for studying learning algorithms. In 1981, Barto and Sutton developed a RL method based on TD which used the concept of learning from failure. Moreover, over the last few years the poor/slow convergence issues has gained more attention by researchers . In this paper, a closed form value function solution for an unstable plant and optimal polynomial basis for the value function are presented. The linear TD(0) algorithm is stated and it is shown that the finite horizon effect which is due to repeatedly simulating the system over a finite horizon introduces a near singularity/bias in the parameter estimation process. A method is proposed to overcome this problem. Finally, the simulation results for the exemplar problem are presented, and the parameter convergence is analyzed.
KeywordsFunction Solution Finite Horizon Convergence Issue Parameter Convergence Zigzag Pattern
- 1.Bertsekas, D.P.: Temporal Difference Methods for General Projected Equations. IEEE Trans. on Automat. Contr. (in press)Google Scholar