Reinforcement Learning of Projections

Abstract

In this chapter, we derive three general reinforcement learning methods [234]. for projection problems under the framework of reinforcement learning. The three methods share the same structure in that the component weight vectors are represented by stochastic units drawn from the Gaussian distribution with mean m and variance β²I. We create adaptive update rules for these parameters of the Gaussian distribution so as to maximize the expected value of the long-term reward. We first derive a particular form of immediate reward reinforcement learning which can be applied to solve linear and non-linear projection problems. Then we suggest that the reinforcement learning neural network can be implemented for unsupervised projection problems. Lastly, based on temporal difference learning, we investigate two new algorithms that are based on Sarsa-learning and Q-learning for projection problems. We show that the last method has accurate convergence, even for non-linear projections.

Also, it is frequently important in projection methods to identify multiple components. Although we can find more than one component by deflationary methods such as the Gram-Schmidt method, these methods seem to be outwith the framework of reinforcement learning. Thus, we describe a general method that can find more components by re-defining the reward functions. To perform deflationary orthogonalization, we extend the definition of reward functions so that it includes one basic reward function and one extended reward function. Based on such an idea, two different ways are derived to identify orthogonal directions in linear projection problems and kernel methods.