Totally model-free actor-critic recurrent neural-network reinforcement learning in non-Markovian domains

Abstract

For solving a sequential decision-making problem in a non-Markovian domain, standard dynamic programming (DP) requires a complete mathematical model; hence, a totally model-based approach. By contrast, this paper describes a totally model-free approach by actor-critic reinforcement learning with recurrent neural networks. The recurrent connections (or context units) in neural networks act as an implicit form of internal state (i.e., history memory) for developing sensitivity to hidden non-Markovian dependencies, rendering the process Markovian implicitly and automatically in a totally model-free fashion. That is, the model-free recurrent-network agent neither learns transitional probabilities and associated rewards, nor by how much the state space should be enlarged so that the Markov property holds. For concreteness, we illustrate time-lagged path problems, in which our learning agent is expected to learn a best (history-dependent) policy that maximizes the total return, the sum of one-step transitional rewards plus special “bonus” values dependent on prior transitions or decisions. Since we can obtain an optimal solution by model-based DP, this is an excellent test on the learning agent for understanding its model-free learning behavior. Such actor-critic recurrent-network learning might constitute a mechanism which animal brains use when experientially acquiring skilled action. Given a concrete non-Markovian problem example, the goal of this paper is to show the conceptual merit of totally model-free learning with actor-critic recurrent networks, compared with classical DP (and other model-building procedures), rather than pursue a best recurrent-network learning strategy.

Appendices

Appendix 1: Stochastic dynamic programming (DP) solution procedures

We show the standard backward DP solution procedures for stochastic problems subject to the following two stochastic versions of the bonus rule introduced in Sect. 2.2:

• (1) Prior-transition dependent bonus rule, and

• (2) Prior-action dependent bonus rule.

Here, the process is stochastic in the sense that when the agent tries to move in a certain direction, either diagonally up or down, it does so with probability p but it goes in the other direction with probability $1-p$. In our numerical example in Fig. 8, we chose $p = 0.9$ and the bonus value 4.

As described in Sect. 2.3, the classical DP is a model-based approach, requiring explicit transition and reward models to allow a DP solution based on the principle of optimality. To solve our longest path problem with bonus rule, we must enlarge the state space to make the process Markovian, by choosing proper arguments for the optimal value function; this is in general a matter of art (rather than science) [see a variety of examples in Bellman and Dreyfus (1962), Dreyfus and Law (1977)]. These arguments must explicitly define the appropriate amount of information given the bonus rule.

Appendix 1.1: DP solution to the prior-transition dependent problem

For the prior-transition dependent case, the state space must contain the information of the current two-dimensional coordinates plus the last two consecutive “actual” transitions no matter what actions were taken. Our classical backward DP-formulation needs the following four steps under the assumption of the unit discount factor in our finite-horizon path problem.

• ① Definition of the optimal value function:

  $$\begin{aligned} \begin{array}{l} V(x,y,z_1,z_2) \mathop {=}\limits ^\mathrm{def}\; \text{ maximum } \text{ expected } \text{ reward-to-go, } \text{ starting } \text{ at } \text{ vertex }~(x,y)~\hbox {to end},\\ \text{ with }~transition~z_1~\hbox {one stage before, and}~z_2~\hbox {two stages before}. \end{array}\nonumber \\ \end{aligned}$$

  (16)

• ② Recurrence relation using one-stage reward $R_z(x,y)$ at the current vertex (x, y) when action z [$z=u$ (up) or d (down)] is taken:

  $$\begin{aligned} \begin{array}{l} V(x,y,u,d) = \text{ max } \left\{ \begin{array}{l} u: \begin{array}{l} p \left[ R_u(x,y) + V(x+1,y+1,u,u) \right] \\ +\,(1- p) \left[ R_d(x,y) + V(x+1,y-1,d,u) + \text{ bonus } \right] \\ \end{array}\\ d: \begin{array}{l} p \left[ R_d(x,y) + V(x+1,y-1,d,u) + \text{ bonus } \right] \\ +\,(1-p) \left[ R_u(x,y) + V(x+1,y+1,u,u) \right] \\ \end{array} \end{array} \right. \\ V(x,y,u,u) =\text{ max } \left\{ \begin{array}{l} u: \begin{array}{l} p \left[ R_u(x,y) + V(x+1,y+1,u,u) + \text{ bonus } \right] \\ +\,(1- p) \left[ R_d(x,y) + V(x+1,y-1,d,u) \right] \\ \end{array}\\ d: \begin{array}{l} p \left[ R_d(x,y) + V(x+1,y-1,d,u) \right] \\ +\,(1-p) \left[ R_u(x,y) + V(x+1,y+1,u,u) + \text{ bonus } \right] \\ \end{array} \end{array} \right. \end{array} \end{aligned}$$

  with obvious modifications for V(x, y, d, u) and V(x, y, d, d).

  • The best action z (u or d) selected for evaluating each $V(x,y,z_1,z_2)$ above is stored into the so-called optimal policy function as

    $$\begin{aligned} \pi (x,y, z_1, z_2)=z. \end{aligned}$$

• ③ Boundary condition at terminal stage $N (=4)$ in Fig. 8:

  $V(5,0,-,-)=-1$; $V(5,2,-,-)=3$; $V(5,4,-,-)=V(5,6,-,-)=0$; $V(5,8,-,-)=1$.

• ④ Answer is given by $V(A,-,-)=V(1,4,-,-)$ at the initial vertex A (1, 4).   $\square$

Using full backups (see Fig. 3) with $p=0.9$ and bonus value 4, the above DP procedure yields the optimal policy, which may be summarized with such eight possible transitional paths (or trajectories) as shown in Table 4 together with the simulation results obtained by model-free learning.

Appendix 1.2: DP solution to the prior-action dependent problem

Next, for solving the prior-action dependent case, the arguments of the optimal value function must explicitly define the current two-dimensional coordinates plus the last two consecutive actions taken whatever actual transitions followed. Here are the four steps of classical backward DP with the unit discount rate assumed:

• ① Definition of the optimal value function:

  $$\begin{aligned} \begin{array}{l} V(x,y,z_1,z_2) \mathop {=}\limits ^\mathrm{def}\; \text{ maximum } \text{ expected } \text{ reward-to-go, } \text{ starting } \text{ at } \text{ vertex }~(x,y)~\hbox {to end},\\ \text{ with } \text{ action }~z_1~\hbox {one stage before, and}~z_2~\hbox {two stages before}. \end{array}\nonumber \\ \end{aligned}$$

  (17)

• ② Recurrence relation using one-stage reward $R_z(x,y)$ at the current vertex (x, y) when action z [$z=u$ (up) or d (down)] is taken:

  $$\begin{aligned} \begin{array}{l} V(x,y,u,d) =\text{ max } \left\{ \begin{array}{l} u: \begin{array}{l} p \left[ R_u(x,y) + V(x+1,y+1,u,u) \right] \\ +\,(1- p) \left[ R_d(x,y) + V(x+1,y-1,u,u) \right] \\ \end{array}\\ d: \begin{array}{l} p \left[ R_d(x,y) + V(x+1,y-1,d,u) + \text{ bonus } \right] \\ +\,(1-p) \left[ R_u(x,y) + V(x+1,y+1,d,u) + \text{ bonus } \right] \\ \end{array} \end{array} \right. \\ V(x,y,u,u) =\text{ max } \left\{ \begin{array}{l} u: \begin{array}{l} p \left[ R_u(x,y) + V(x+1,y+1,u,u) + \text{ bonus } \right] \\ +\,(1- p) \left[ R_d(x,y) + V(x+1,y-1,u,u) + \text{ bonus } \right] \\ \end{array}\\ d: \begin{array}{l} p \left[ R_d(x,y) + V(x+1,y-1,d,u) \right] \\ +\,(1-p) \left[ R_u(x,y) + V(x+1,y+1,d,u) \right] \\ \end{array} \end{array} \right. \end{array} \end{aligned}$$

  with obvious modifications for V(x, y, d, u) and V(x, y, d, d).

  • The best action z (u or d) selected for evaluating each $V(x,y,z_1,z_2)$ above is stored into the so-called optimal policy function as

    $$\begin{aligned} \pi (x,y, z_1, z_2)=z. \end{aligned}$$

• ③ Boundary condition at terminal stage $N (=4)$ in Fig. 8:

  $V(5,0,-,-)=-1$; $V(5,2,-,-)=3$; $V(5,4,-,-)=V(5,6,-,-)=0$; $V(5,8,-,-)=1$.

• ④ Answer is given by $V(A,-,-)=V(1,4,-,-)$ at the initial vertex A (1, 4). $\square$