Multilevel Preconditioners for Temporal-Difference Learning Methods Related to Recommendation Engines

  • Michael Thess
Part of the Lecture Notes in Applied and Computational Mechanics book series (LNACM, volume 66)


In many areas of retail and especially e-business recommendation engines are applied to increase the usability of the store or portal. Advanced recommendation engines use approaches from control theory for adaptive learning. At the forefront of these algorithms reinforcement learning is applied which however requires large transaction numbers to converge. To overcome this problem, we propose a hierarchical approach of reinforcement learning for recommendation engines by combining a multilevel preconditioner with the temporal-difference learning method, the most important algorithm class of reinforcement learning. The multilevel preconditioner works on a combined hierarchy of states and actions. We describe the preconditioner, prove its convergence and present results on real-life data.


Recommender System Bellman Equation Multilevel Method Transaction Type Eligibility Trace 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Balabanovic, M.: An Adaptive Web Page Recommendation Service. CACM (1997)Google Scholar
  2. 2.
    Bertsekas, D.P., Castanon, D.A.: Adaptive Aggregation Methods for Infinite Horizon Dynamic Programming. IEEE Trans. Automatic Control 34(6) (1989)Google Scholar
  3. 3.
    Bramble, J., Pasciak, J., Xu, J.: Parallel multilevel preconditioners. Math. Comp. 55, 1–12 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Brand, M.E.: Fast online svd revisions for lightweight recommender systems. In: SIAM International Conference on Data Mining, SDM (2003)Google Scholar
  5. 5.
    Burke, R.: Hybrid Recommender Systems: Survey and Experiments. User Modeling and User-Adapted Interaction 12(4) (2002)Google Scholar
  6. 6.
    Golovin, N., Rahm, E.: Reinforcement Learning Architecture for Web Recommendations. In: Proc. ITCC 2004. IEEE (2004)Google Scholar
  7. 7.
    Herlocker, J.L.: Evaluating Collaborative Filtering Recommender Systems. ACM Transactions on Information Systems 22(1) (2004)Google Scholar
  8. 8.
    Linden, G., Smith, B., York, J.: Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing (2003)Google Scholar
  9. 9.
    Munos, R.: A study of reinforcement learning in the continuous case by the means of viscosity solutions. Machine Learning 40 (2000)Google Scholar
  10. 10.
    Oswald, P.: Multilevel Finite Element Approximation. B.G. Teubner, Stuttgart (1994)zbMATHGoogle Scholar
  11. 11.
    Paprotny, A.: Praktikumsbericht zum Fachpraktikum bei der Firma prudsys AG. Report. TU Hamburg-Harburg (2009) (in German)Google Scholar
  12. 12.
    Paprotny, A.: Hierarchical methods for the solution of dynamic programming equations arising from optimal control problems related to recommendation. Diploma thesis, TU Hamburg-Harburg (2010)Google Scholar
  13. 13.
    Paprotny, A., Thess, M.: A stepwise approach to a self-learning recommendation engine. prudsys documentation, Chemnitz (2011)Google Scholar
  14. 14.
    Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice Hall, New Jersey (2002)Google Scholar
  15. 15.
    Rojanavasu, P., Phaitoon, S., Pinngern, O.: New Recommendation System Using Reinforcement Learning. In: Proceedings of the Fourth International Conference on eBusiness, Bangkok, Thailand, November 19-20 (2005)Google Scholar
  16. 16.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning. An Introduction. MIT Press, Cambridge (1998)Google Scholar
  17. 17.
    Shani, G., Heckerman, D., Brafman, R.I.: An MDP-based recommender system. Journal of Machine Learning Research 6 (2005)Google Scholar
  18. 18.
    Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of Recommendation Algorithms for E-Commerce. In: EC 2000, Minneapolis, Minnesota, October 17-20 (2000)Google Scholar
  19. 19.
    Ziv, O.: Algebraic Multigrid for Reinforcement Learning. Master thesis, Technion (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.prudsys AGChemnitzGermany

Personalised recommendations