PAC Bounds for Discounted MDPs

  • Tor Lattimore
  • Marcus Hutter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7568)


We study upper and lower bounds on the sample-complexity of learning near-optimal behaviour in finite-state discounted Markov Decision Processes (mdps). We prove a new bound for a modified version of Upper Confidence Reinforcement Learning (ucrl) with only cubic dependence on the horizon. The bound is unimprovable in all parameters except the size of the state/action space, where it depends linearly on the number of non-zero transition probabilities. The lower bound strengthens previous work by being both more general (it applies to all policies) and tighter. The upper and lower bounds match up to logarithmic factors provided the transition matrix is not too dense.


Reinforcement learning sample-complexity exploration exploitation PAC-MDP Markov decision processes 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [AJO10]
    Auer, P., Jaksch, T., Ortner, R.: Near-optimal regret bounds for reinforcement learning. J. Mach. Learn. Res. 99, 1563–1600 (2010)MathSciNetGoogle Scholar
  2. [AMK12]
    Azar, M., Munos, R., Kappen, B.: On the sample complexity of reinforcement learning with a generative model. In: Proceedings of the 29th International Conference on Machine Learning. ACM, New York (2012)Google Scholar
  3. [AO07]
    Auer, P., Ortner, R.: Logarithmic online regret bounds for undiscounted reinforcement learning. In: Advances in Neural Information Processing Systems 19, pp. 49–56. MIT Press (2007)Google Scholar
  4. [Aue11]
    Auer, P.: Upper confidence reinforcement learning. Unpublished, keynote at European Workshop of Reinforcement Learning (2011)Google Scholar
  5. [CL06]
    Chung, F., Lu, L.: Concentration inequalities and martingale inequalities a survey. Internet Mathematics 3, 1 (2006)MathSciNetCrossRefGoogle Scholar
  6. [Kak03]
    Kakade, S.: On The Sample Complexity of Reinforcement Learning. PhD thesis, University College London (2003)Google Scholar
  7. [LH12]
    Lattimore, T., Hutter, M.: PAC bounds for discounted MDPs. Technical report (2012),
  8. [LR85]
    Lai, T., Robbins, H.: Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6(1), 4–22 (1985)MathSciNetzbMATHCrossRefGoogle Scholar
  9. [MT04]
    Mannor, S., Tsitsiklis, J.: The sample complexity of exploration in the multi-armed bandit problem. J. Mach. Learn. Res. 5, 623–648 (2004)MathSciNetzbMATHGoogle Scholar
  10. [SL05]
    Strehl, A., Littman, M.: A theoretical analysis of model-based interval estimation. In: Proceedings of the 22nd International Conference on Machine Learning, ICML 2005, pp. 856–863 (2005)Google Scholar
  11. [SL08]
    Strehl, A., Littman, M.: An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences 74(8), 1309–1331 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  12. [SLL09]
    Strehl, A., Li, L., Littman, M.: Reinforcement learning in finite MDPs: PAC analysis. J. Mach. Learn. Res. 10, 2413–2444 (2009)MathSciNetzbMATHGoogle Scholar
  13. [SLW+06]
    Strehl, A., Li, L., Wiewiorac, E., Langford, J., Littman, M.: PAC model-free reinforcement learning. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 881–888. ACM, New York (2006)Google Scholar
  14. [Sob82]
    Sobel, M.: The variance of discounted Markov decision processes. Journal of Applied Probability 19(4), 794–802 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  15. [SS10]
    Szita, I., Szepesvári, C.: Model-based reinforcement learning with nearly tight exploration complexity bounds. In: Proceedings of the 27th International Conference on Machine Learning, pp. 1031–1038. ACM, New York (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Tor Lattimore
    • 1
  • Marcus Hutter
    • 1
  1. 1.Australian National UniversityAustralia

Personalised recommendations