Economical reinforcement learning for non stationary problems

  • Nathalie Chatenet
  • Hugues Bersini
Part II: Cortical Maps and Receptive Fields
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1327)


In a lot of reinforcement learning applications and solutions, time information is implicitly contained in state information. Although time sequential decomposition is inherent to dynamic programming, this aspect has been simply omitted in usual Q Learning applications. A non stationary environment, a non stationary reward/ punishment or a time dependent cost to minimize will naturally lead to non stationary optimal solutions in which time has to be explicitly accounted in the search for the optimal solution. Although largely neglected so far, non stationarity and the computational cost it is likely to rapidly induce, should instead becomes of concern to the reinforcement learning community. The particular nature of time entails a dedicated processing when attempting to develop economical and heuristic solutions. In this paper two of such possible heuristics are proposed, justified and illustrated on a simple application.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A.W. Moore. Variable Resolution Dynamic Programming: Efficiently Learning Action Map in Multivariate Real-valued State-spaces. In Proceedings of the Eighth International Workshop on Machine Learning, pages 333–337, 1991.Google Scholar
  2. 2.
    C. Watkins. Technical Note Qlearning. Machine Learning, 8:279–297, 1992.Google Scholar
  3. 3.
    D. Chapman and L. Kaebling. Input Generalization in Delayed Reinforcement Learning: An Algorithm and Performance Comparisons, 1991.Google Scholar
  4. 4.
    M. Dorigo and H. Bersini. A Comparison of Q-Learning and Classifier Systems. In Proceedings of SAB III, pages 248–255. MIT Press, 1994.Google Scholar
  5. 5.
    N. Chatenet. Adaptation of Turnpike Theorem in a Variant of Qlearning. Technical Report 96016, University of Bordeaux I, 1996.Google Scholar
  6. 6.
    R. Sutton. Integrated Modeling and Control Based on Reinforcement Learning and Dynamic Programming. In Advances in Neural Information Processing 3, pages 471–478, 1991.Google Scholar
  7. 7.
    S. Achmanov. Programmation Linéaire. MIR, 1984.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1997

Authors and Affiliations

  • Nathalie Chatenet
    • 1
  • Hugues Bersini
    • 2
  1. 1.Cemagref de Bordeaux (division ORH)Cestas CedexFrance
  2. 2.Université libre de Bruxelles - IRIDIA - CP194/6BruxellesBelgium

Personalised recommendations