On the Analysis of Parameter Convergence for Temporal Difference Learning of an Exemplar Balance Problem

  • Martin Brown
  • Onder Tutsoy
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6856)


Bipedal walking/locomotion is a challenging control problem but also an interesting problem for studying learning algorithms. In 1981, Barto and Sutton developed a RL method based on TD which used the concept of learning from failure. Moreover, over the last few years the poor/slow convergence issues has gained more attention by researchers [1]. In this paper, a closed form value function solution for an unstable plant and optimal polynomial basis for the value function are presented. The linear TD(0) algorithm is stated and it is shown that the finite horizon effect which is due to repeatedly simulating the system over a finite horizon introduces a near singularity/bias in the parameter estimation process. A method is proposed to overcome this problem. Finally, the simulation results for the exemplar problem are presented, and the parameter convergence is analyzed.


Function Solution Finite Horizon Convergence Issue Parameter Convergence Zigzag Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bertsekas, D.P.: Temporal Difference Methods for General Projected Equations. IEEE Trans. on Automat. Contr. (in press)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Martin Brown
    • 1
  • Onder Tutsoy
    • 1
  1. 1.Control Systems Group, School of Electrical and Electronic EngineeringThe University of ManchesterManchesterUK

Personalised recommendations