1-Penalized Projected Bellman Residual

  • Matthieu Geist
  • Bruno Scherrer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7188)


We consider the task of feature selection for value function approximation in reinforcement learning. A promising approach consists in combining the Least-Squares Temporal Difference (LSTD) algorithm with ℓ1-regularization, which has proven to be effective in the supervised learning community. This has been done recently whit the LARS-TD algorithm, which replaces the projection operator of LSTD with an ℓ1-penalized projection and solves the corresponding fixed-point problem. However, this approach is not guaranteed to be correct in the general off-policy setting. We take a different route by adding an ℓ1-penalty term to the projected Bellman residual, which requires weaker assumptions while offering a comparable performance. However, this comes at the cost of a higher computational complexity if only a part of the regularization path is computed. Nevertheless, our approach ends up to a supervised learning problem, which let envision easy extensions to other penalties.


Feature Selection Reinforcement Learning Breaking Point Policy Iteration Hypothesis Space 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Antos, A., Szepesvári, C., Munos, R.: Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning 71(1), 89–129 (2008)CrossRefGoogle Scholar
  2. 2.
    Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena ScientificGoogle Scholar
  3. 3.
    Boyan, J.A.: Technical Update: Least-Squares Temporal Difference Learning. Machine Learning 49(2-3), 233–246 (1999)Google Scholar
  4. 4.
    Bradtke, S.J., Barto, A.G.: Linear Least-Squares algorithms for temporal difference learning. Machine Learning 22(1-3), 33–57 (1996)zbMATHCrossRefGoogle Scholar
  5. 5.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic Decomposition by Basis Pursuit. SIAM Journal on Scientific Computing 20, 33–61 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least Angle Regression. Annals of Statistics 32(2), 407–499 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Farahmand, A., Ghavamzadeh, M., Szepesvári, C., Mannor, S.: Regularized policy iteration. In: 22nd Annual Conference on Neural Information Processing Systems (NIPS 21), Vancouver, Canada (2008)Google Scholar
  8. 8.
    Ghavamzadeh, M., Lazaric, A., Munos, R., Hoffman, M.: Finite-Sample Analysis of Lasso-TD. In: International Conference on Machine Learning (2011)Google Scholar
  9. 9.
    Hoffman, M.W., Lazaric, A., Ghavamzadeh, M., Munos, R.: Regularized least squares temporal difference learning with nested ℓ2 and ℓ1 penalization. In: European Workshop on Reinforcement Learning (2011)Google Scholar
  10. 10.
    Johns, J., Mahadevan, S.: Constructing basis functions from directed graphs for value function approximation. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, pp. 385–392. ACM, New York (2007)CrossRefGoogle Scholar
  11. 11.
    Johns, J., Painter-Wakefield, C., Parr, R.: Linear Complementarity for Regularized Policy Evaluation and Improvement. In: Lafferty, J., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) NIPS 23, pp. 1009–1017 (2010)Google Scholar
  12. 12.
    Kolter, J.Z., Ng, A.Y.: Regularization and Feature Selection in Least-Squares Temporal Difference Learning. In: Proceedings of the 26th International Conference on Machine Learning (ICML 2009), Montreal, Canada (2009)Google Scholar
  13. 13.
    Loth, M., Davy, M., Preux, P.: Sparse Temporal Difference Learning using LASSO. In: IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, Hawaï, USA (2007)Google Scholar
  14. 14.
    Munos, R.: Error bounds for approximate policy iteration. In: International Conference on Machine Learning (2003)Google Scholar
  15. 15.
    Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., Littman, M.L.: An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 752–759. ACM, New York (2008)CrossRefGoogle Scholar
  16. 16.
    Petrik, M., Taylor, G., Parr, R., Zilberstein, S.: Feature Selection Using Regularization in Approximate Linear Programs for Markov Decision Processes. In: Proceedings of ICML (2010)Google Scholar
  17. 17.
    Rosset, S., Zhu, J.: Piecewise linear regularized solution paths. The Annals of Statistics 35(3), 1012–1030 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Scherrer, B.: Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view. In: 27th International Conference on Machine Learning - ICML 2010, Haïfa, Israël (2010)Google Scholar
  19. 19.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). The MIT Press (1998)Google Scholar
  20. 20.
    Sutton, R.S., Maei, H.R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., Wiewiora, E.: Fast gradient-descent methods for temporal-difference learning with linear function approximation. In: Proceedings of ICML, pp. 993–1000. ACM, New York (2009)Google Scholar
  21. 21.
    Szepesvári, C.: Algorithms for Reinforcement Learning. Morgan and Kaufmann (2010)Google Scholar
  22. 22.
    Taylor, G., Parr, R.: Kernelized value function approximation for reinforcement learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 1017–1024. ACM, New York (2009)Google Scholar
  23. 23.
    Thiery, C., Scherrer, B.: Building Controllers for Tetris. International Computer Games Association Journal 32, 3–11 (2009)Google Scholar
  24. 24.
    Tibshirani, R.: Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Zou, H.: The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101(476), 1418–1429 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Zou, H., Zhang, H.H.: On the adaptive elastic-net with a diverging number of parameters. The Annals of Statistics 37(4), 1733–1751 (2009)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Matthieu Geist
    • 1
  • Bruno Scherrer
    • 2
  1. 1.Supélec, IMS Research GroupMetzFrance
  2. 2.INRIA, MAIA Project-TeamNancyFrance

Personalised recommendations