Reinforcement Learning Algorithm for Partially Observable Environment Considering State Estimation Error Prediction

Abstract

Reinforcement learning is a machine learning technique that attempts to learn a policy based on a reward criterion through trial and error in the target environment. In this paper, we investigate a reinforcement learning algorithm for a partially observable environment without prior knowledge about the state transition and observation likelihood functions. When only noisy or incomplete measurement is available due to inexpensive sensors, we can use a state estimator in combination with a policy trained via reinforcement learning to recover the performance of the policy. In such cases, performance may degrade due to state estimation error. To solve this problem, we propose a planning method that considers the prediction of state estimation error. The proposed algorithm indirectly predicts the state estimation error and avoids to get into areas where large state estimation error may occur using prediction. We tested the proposed method in a simple inverted pendulum task, where the agent only receives noisy and drifted measurements. Empirical result demonstrates that performance of policy in a partially observable environment is improved by the proposed method.