Compound Reinforcement Learning: Theory and an Application to Finance
This paper describes compound reinforcement learning (RL) that is an extended RL based on the compound return. Compound RL maximizes the logarithm of expected double-exponentially discounted compound return in return-based Markov decision processes (MDPs). The contributions of this paper are (1) Theoretical description of compound RL that is an extended RL framework for maximizing the compound return in a return-based MDP and (2) Experimental results in an illustrative example and an application to finance.
KeywordsReinforcement learning compound return value functions finance
Unable to display preview. Download preview PDF.
- 3.Campbell, J.Y., Lo, A.W., Graig MacKinlay, A.: The Econometrics of Financial Markets. Princeton University Press (1997)Google Scholar
- 4.CMA. Global sovereign credit risk report, 4th quarter 2010. Credit Market Analysis, Ltd. (CMA) (2011)Google Scholar
- 7.Heger, M.: Consideration of risk in reinforcement learning. In: Proc. of the Eleventh International Conference on Machine Learning, ICML 1994, pp. 105–111 (1994)Google Scholar
- 10.Poundstone, W.: Fortune’s Formula: The untold story of the scientific betting system that beat the casinos and wall street. Hill and Wang (2005)Google Scholar
- 11.Sato, M., Kobayashi, S.: Average-reward reinforcement learning for variance penalized Markov decision problems. In: Proc. of the Eighteenth International Conference on Machine Learning, ICML 2001, pp. 473–480 (2001)Google Scholar
- 12.Schwartz, A.: A reinforcement learning method for maximizing undiscounted rewards. In: Proc. of the Tenth International Conference on Machine Learning (ICML 1993), pp. 298–305 (1993)Google Scholar
- 13.Singh, S.P.: Reinforcement learning algorithms for average-payoff Markovian decision processes. In: Proc. of the Twelfth National Conference on Artificial Intelligence (AAAI 1994), vol. 1, pp. 700–705 (1994)Google Scholar
- 14.Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press (1998)Google Scholar
- 16.Vince, R.: Portfolio management formulas: mathematical trading methods for the futures, options, and stock markets. Wiley (1990)Google Scholar