Reinforcement Learning for Trading Systems and Portfolios: Immediate vs Future Rewards

Abstract

We propose to train trading systems and portfolios by optimizing financial objective functions via reinforcement learning. The performance functions that we consider as value functions are profit or wealth, the Sharpe ratio and our recently proposed differential Sharpe ratio for online learning. In Moody & Wu (1997), we presented empirical results in controlled experiments that demonstrated the efficacy of some of our methods for optimizing trading systems. Here we extend our previous work to the use of Q-Learning, a reinforcement learning technique that uses approximated future rewards to choose actions, and compare its performance to that of our previous systems which are trained to maximize immediate reward. We also provide new simulation results that demonstrate the presence of predictability in the monthly S&P 500 Stock Index for the 25 year period 1970 through 1994.