Using Control Theory for Analysis of Reinforcement Learning and Optimal Policy Properties in Grid-World Problems

  • S. Mostapha Kalami Heris
  • Mohammad-Bagher Naghibi Sistani
  • Naser Pariz
Conference paper

DOI: 10.1007/978-3-642-04020-7_30

Part of the Lecture Notes in Computer Science book series (LNCS, volume 5755)
Cite this paper as:
Kalami Heris S.M., Naghibi Sistani MB., Pariz N. (2009) Using Control Theory for Analysis of Reinforcement Learning and Optimal Policy Properties in Grid-World Problems. In: Huang DS., Jo KH., Lee HH., Kang HJ., Bevilacqua V. (eds) Emerging Intelligent Computing Technology and Applications. With Aspects of Artificial Intelligence. ICIC 2009. Lecture Notes in Computer Science, vol 5755. Springer, Berlin, Heidelberg

Abstract

Markov Decision Process (MDP) has enormous applications in science, engineering, economics and management. Most of decision processes have Markov property and can be modeled as MDP. Reinforcement Learning (RL) is an approach to deal with MDPs. RL methods are based on Dynamic Programming (DP) algorithms, such as Policy Evaluation, Policy Iteration and Value Iteration. In this paper, policy evaluation algorithm is represented in the form of a discrete-time dynamical system. Hence, using Discrete-Time Control methods, behavior of agent and properties of various policies, can be analyzed. The general case of grid-world problems is addressed, and some important results are obtained for this type of problems as a theorem. For example, equivalent system of an optimal policy for a grid-world problem is dead-beat.

Keywords

Dynamic Programming Discrete-Time Control Systems Markov Decision Process Reinforcement Learning Stochastic Control 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • S. Mostapha Kalami Heris
    • 1
  • Mohammad-Bagher Naghibi Sistani
    • 2
  • Naser Pariz
    • 2
  1. 1.Control Engineering Department, Faculty of Electrical EngineeringK. N. Toosi University of TechnologyTehranIran
  2. 2.Electrical Engineering Department, Faculty of EngineeringFerdowsi University of MashhadMashhadIran

Personalised recommendations