Neural/fuzzy self learning Lyapunov control for non linear systems

Abstract

This work proposes Lyapunov theory based Fuzzy/Neural Reinforcement Learning (RL) controllers with guaranteed stability. We look at ways in which Lyapunov theory could be used to produce RL controllers wherein the control action is hybrid or Lyapunov constrained, resulting in self learning controllers that are optimal, effective and stable. Fuzzy systems and Neural networks have been used as generic function approximators to handle exponential rise in computational burden that arises when RL is extended to high dimensional/continuous state-action spaces. We propose two distinct approaches: (i) Hybridized fuzzy Lyapunov RL control by combining Fuzzy Q Learning methodology in a Lyapunov setting thereby guarantying stability, and (ii) Lyapunov constrained Neural RL control wherein the controller’s action set is constrained to satisfy Lyapunov stability condition. Incorporating Lyapunov theory based element in the action generation mechanism of an RL based controller guarantees stability. We implement our soft computing based Lyapunov RL on the two benchmark non linear problems: (a) inverted pendulum and (b) cart pole balancing. The results obtained and associated comparison with baseline Neural/Fuzzy Q-Learning based controllers bring out the advantage of our Lyapunov RL based scheme.

Appendix

From physics, we know that

$$\sum {\tau = I} \alpha$$

where ∑ τ gives sum of torques, I is moment of inertia and α is angular acceleration.

From the free body diagram (Fig. 16) is clear that there is motion only in the direction perpendicular to the rigid rod. We have:

$$a + mgL\sin \theta = mL^{2} \mathop \theta \limits^{ \bullet \bullet }$$

Fig. 16

Free body diagram for inverted pendulum

Considering m, g and L (where length of pendulum = L, mass of pendulum = m and acceleration due to gravity = g) to be unity as used in [9] we get:

$$a + \sin \theta = \mathop \theta \limits^{ \bullet \bullet }$$