Threshold Recurrent Reinforcement Learning Model for Automated Trading

  • Dietmar Maringer
  • Tikesh Ramtohul
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6025)


This paper presents the threshold recurrent reinforcement learning (TRRL) model and describes its application in a simple automated trading system. The TRRL is a regime-switching extension of the recurrent reinforcement learning (RRL) algorithm. The basic RRL model was proposed by Moody and Wu (1997) and used for uncovering trading strategies. We argue that the RRL is not sufficiently equipped to capture the non-linearities and structural breaks present in financial data, and propose the TRRL model as a more suitable algorithm for such environments. This paper gives a detailed description of the TRRL and compares its performance with that of the basic RRL model in a simple automated trading framework using daily data from four well-known European indices. We assume a frictionless setting and use volatility as an indicator variable for switching between regimes. We find that the TRRL produces better trading strategies in all the cases studied, and demonstrate that it is more apt at finding structure in non-linear financial time series than the standard RRL.


regime-switching automated trading reinforcement learning differential Sharpe ratio 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Moody, J., Wu, L.: Optimization of trading systems and portfolios. In: Proceedings of the IEEE/IAFE Computational Intelligence for Financial Engineering (CIFEr), pp. 300–307 (1997)Google Scholar
  2. 2.
    Moody, J., Wu, L., Liao, Y., Saffell, M.: Performance functions and reinforcement learning for trading systems and portfolios. Journal of Forecasting 17(56), 441–470 (1998)CrossRefGoogle Scholar
  3. 3.
    Moody, J., Saffell, M.: Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12(4), 875–889 (2001)CrossRefGoogle Scholar
  4. 4.
    Gold, C.: FX trading via recurrent reinforcement learning. In: IEEE International Conference on Computational Intelligence for Financial Engineering, pp. 363–370 (2003)Google Scholar
  5. 5.
    Dempster, M., Leemans, V.: An automated FX trading system using adaptive reinforcement learning. Expert Systems with Applications 30(3), 543–552 (2006); Intelligent Information Systems for Financial EngineeringGoogle Scholar
  6. 6.
    Bertoluzzo, F., Corazza, M.: Making financial trading by recurrent reinforcement learning. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007, Part II. LNCS (LNAI), vol. 4693, pp. 619–626. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  7. 7.
    Hamilton, J.: Regime-switching models. In: The New Palgrave Dictionary of Economics. Palgrave Macmillan, Basingstoke (2008)Google Scholar
  8. 8.
    Tong, H.: On a threshold model. In: Chen, C. (ed.) Pattern Recognition and Signal Processing, pp. 101–141. Sijthoff & Noordhoff (1978)Google Scholar
  9. 9.
    Hamilton, J.D.: A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica 57(2), 357–384 (1989)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    White, H.: Some asymptotic results for learning in single hidden-layer feedforward network models. Journal of the American Statistical Association, 1003–1013 (1989)Google Scholar
  11. 11.
    Teräsvirta, T.: Specification, estimation, and evaluation of smooth transition autoregressive models. Journal of the American Statistical Association, 208–218 (1994)Google Scholar
  12. 12.
    Maringer, D., Ramtohul, T.: Regime-switching recurrent reinforcement learning. Mimeo University of Basel (2009)Google Scholar
  13. 13.
    Jacobs, R., Jordan, M., Nowlan, S., Hinton, G.: Adaptive mixtures of local experts. Neural computation 3(1), 79–87 (1991)CrossRefGoogle Scholar
  14. 14.
    Sharpe, W.: Mutual fund performance. Journal of Business, 119–138 (1966)Google Scholar
  15. 15.
    Werbos, P.: Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10), 1550–1560 (1990)CrossRefGoogle Scholar
  16. 16.
    LeBaron, B.: Some relations between volatility and serial correlations in stock market returns. Journal of Business 65(2), 199–219 (1992)CrossRefGoogle Scholar
  17. 17.
    Sentana, E., Wadhwani, S.B.: Feedback traders and stock return autocorrelations: Evidence from a century of daily data. Economic Journal 102(411), 415–425 (1992)CrossRefGoogle Scholar
  18. 18.
    Koutmos, G.: Feedback trading and the autocorrelation pattern of stock returns: further empirical evidence. Journal of International Money and Finance 16(4), 625–636 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Dietmar Maringer
    • 1
  • Tikesh Ramtohul
    • 1
  1. 1.Universität BaselBaselSwitzerland

Personalised recommendations