Abstract
Reinforcement learning has gained wide popularity as a technique for simulation-driven approximate dynamic programming. A less known aspect is that the very reasons that make it effective in dynamic programming can also be leveraged for using it for distributed schemes for certain matrix computations involving non-negative matrices. In this spirit, we propose a reinforcement learning algorithm for PageRank computation that is fashioned after analogous schemes for approximate dynamic programming. The algorithm has the advantage of ease of distributed implementation and more importantly, of being model-free, i.e., not dependent on any specific assumptions about the transition probabilities in the random web-surfer model. We analyze its convergence and finite time behavior and present some supporting numerical experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Thorndike, E.L.: Animal intelligence: an experimental study of the associative processes in animals. Psychological Review, Monograph Supplement 2(8) (1998)
Bush, R.R., Mosteller, F.: A mathematical model of simple learning. Psychological Review 58, 313–323
Estes, K.W.: Towards a statistical theory of learning. Psychological Review 57, 94–107
Bertsekas, D.P.: Dynamic Programming and Optimal Control, 4th edn., vol. 2. Athena Scientific, Belmont (2007)
Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-dynamic Programming. Athena Scientific, Belmont (1996)
Gosavi, A.: Simulation-based Optimization, Parametric Optimization Techniques and Reinforcement Learning. Springer, New York (2003)
Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd edn. Wiley, New York (2011)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Szepesvari, C.: Algorithms for Reinforcement Learning. Morgan and Claypool Publishers (2010)
Sargent, T.J.: Bounded Rationality in Macroeconomics. Oxford Uni. Press, Oxford (1994)
Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. MIT Press, Cambridge (2005)
Robbins, H., Monro, J.: A stochastic approximation method. Annals of Math. Stat. 22, 400–407 (1951)
Borkar, V.S., Makhijani, R., Sundaresan, R.: How to gossip if you must (preprint, 2013), http://arxiv.org/abs/1309.7841
Borkar, V.: Reinforcement Learning - A Bridge between Numerical Methods and Markov Chain Monte Carlo. In: Sastry, N.S.N., Rajeev, B., Delampady, M., Rao, T.S.S.R.K. (eds.) Perspectives in Mathematical Sciences. World Scientific (2008)
Langville, A.N., Meyer, C.D.: Google’s PageRank and Beyond: The Science of Search Engine Rankings. Princeton Uni. Press, Princeton (2006)
Langville, A.N., Meyer, C.D.: Deeper inside PageRank. Internet Mathematics 1(3), 335–380 (2004)
Avrachenkov, K., Litvak, N., Nemirovsky, D., Osipova, N.: Monte Carlo methods in PageRank computation: when one iteration is sufficient. SIAM J. Numer. Anal. 45(2), 890–904 (2007)
Avrachenkov, K., Litvak, N., Nemirovsky, D., Smirnova, E., Sokol, M.: Quick detection of top-k personalized PageRank lists. In: Frieze, A., Horn, P., Prałat, P. (eds.) WAW 2011. LNCS, vol. 6732, pp. 50–61. Springer, Heidelberg (2011)
Polyak, B.T., Timonina, A.V.: PageRank: new regularizations and simulation models. In: Proc. of 11th IFAC World Congress, Milano, August 28-September, pp. 11202–11207 (2011)
Ishii, H.: Distributed randomized algorithms for PageRank computation. IEEE Trans. Auto. Control 55(9), 1987–2002 (2010)
Nazin, A.V., Polyak, B.T.: ‘The randomized algorithm for finding an eigenvector of the stochastic matrix with application to PageRank. Doklady Mathematics 79(3), 424–427 (2009)
Zhao, W., Chen, H-F. and Fang, H-T.: Convergence of distributed randomized PageRank algorithms. arXiv:1305.3178 [cs.SY] (2013)
Vigna, S.: Spectral ranking, http://arxiv.org/abs/0912.0238
Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Hindustan Publ. Agency, Cambridge Uni. Press, New Delhi, Cambridge (2008)
Ho, Y.-C.: An explanation of ordinal optimization: Soft computing for hard problems. Information Sciences 113(3-4), 169–192 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Borkar, V.S., Mathkar, A.S. (2014). Reinforcement Learning for Matrix Computations: PageRank as an Example. In: Natarajan, R. (eds) Distributed Computing and Internet Technology. ICDCIT 2014. Lecture Notes in Computer Science, vol 8337. Springer, Cham. https://doi.org/10.1007/978-3-319-04483-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-04483-5_2
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-04482-8
Online ISBN: 978-3-319-04483-5
eBook Packages: Computer ScienceComputer Science (R0)