The value of (bounded) memory in a changing world

Abstract

This paper explores the value of memory in decision making in dynamic environments. We examine the decision problem faced by an agent with bounded memory who receives a sequence of signals from a partially observable Markov decision process. We characterize environments in which the optimal memory consists of only two states. In addition, we show that the marginal value of additional memory states need not be positive and may even be negative in the absence of free disposal.

Appendix

Proof of Theorem 2

Note that symmetry implies $\mu _{(1,L)}=\mu _{(2,H)}$ and $\mu _{(2,L)}=\mu _{(1,H)}$; therefore, the decision maker solves

$$\begin{aligned} \max _{\varphi _{1,2}^{H}\in [0,1]}\left\{ 2 \left( \gamma \mu _{(2,H)}+(1-\gamma )\mu _{(1,H)}\right) \right\} . \end{aligned}$$

We may write the steady-state condition in Eq. (3) for state $(2,H)$ as

$$\begin{aligned} \mu _{(2,H)}&= (\alpha \mu _{(1,L)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,2}^{H}+(1-\gamma )\varphi _{1,2}^{L}\right) \\&+\,(\alpha \mu _{(2,L)}+(1-\alpha )\mu _{(2,H)})\left( \gamma \varphi _{2,2}^{H}+(1-\gamma )\varphi _{2,2}^{L}\right) \\&= (\alpha \mu _{(2,H)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,2}^{H}+(1-\gamma )\varphi _{2,1}^{H}\right) \\&+\,(\alpha \mu _{(1,H)}+(1-\alpha )\mu _{(2,H)})\left( \gamma \varphi _{2,2}^{H}+(1-\gamma )\varphi _{1,1}^{H}\right) , \end{aligned}$$

where the second equality follows from symmetry. Recalling that $\varphi _{1,1}^{H}=1-\varphi _{1,2}^{H}$, and that monotonicity implies $\varphi _{2,1}^{H}=0$, this may be written as

$$\begin{aligned} \mu _{(2,H)}&= (\alpha \mu _{(2,H)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,2}^{H}\right) \\&+\,(\alpha \mu _{(1,H)}+(1-\alpha )\mu _{(2,H)})\left( \gamma \varphi _{2,2}^{H}+(1-\gamma )\left( 1-\varphi _{1,2}^{H}\right) \right) \\&= \mu _{(1,H)}\left( \alpha \gamma +\alpha (1-\gamma )\left( 1-\varphi _{1,2}^{H}\right) +(1-\alpha )\gamma \varphi _{1,2}^{H}\right) \\&+\,\mu _{(2,H)}\left( \alpha \gamma \varphi _{1,2}^{H}+(1-\alpha )\gamma +(1-\alpha )(1-\gamma )\left( 1-\varphi _{1,2}^{H}\right) \right) . \end{aligned}$$

Combining this expression with the observation from Eq. (2) that $\mu _{(1,H)}=\frac{1}{2}-\mu _{(2,H)}$, we can then solve for $\mu _{(2,H)}$. In particular, we must have

$$\begin{aligned} \mu _{(2,H)}=\frac{1}{2}\left( \frac{\alpha +(\gamma -\alpha ) \varphi _{1,2}^{H}}{2\alpha +(1-2\alpha )\varphi _{1,2}^{H}}\right) . \end{aligned}$$

With this in hand, we may write the decision maker’s payoff as

$$\begin{aligned} U_{2}\left( \varphi _{1,2}^{H}\right) =(1-\gamma )+(2\gamma -1) \frac{\alpha +(\gamma -\alpha )\varphi _{1,2}^{H}}{2\alpha +(1-2\alpha )\varphi _{1,2}^{H}}. \end{aligned}$$

(4)

Differentiating with respect to $\varphi _{1,2} ^{H}$ yields

$$\begin{aligned} U_{2}'\left( \varphi _{1,2}^{H}\right)&=(2\gamma -1)\frac{\left( 2\alpha +(1-2\alpha )\varphi _{1,2}^{H}\right) (\gamma -\alpha )-\left( \alpha +(\gamma -\alpha )\varphi _{1,2}^{H}\right) (1-2\alpha )}{\left( 2\alpha +(1-2\alpha )\varphi _{1,2}^{H}\right) ^{2}}\\&=\alpha \left( \frac{2\gamma -1}{2\alpha +(1-2\alpha )\varphi _{1,2}^{H}}\right) ^{2}. \end{aligned}$$

Since $\alpha \in (0,\frac{1}{2})$ and $\gamma \in (\frac{1}{2},1)$, this expression is strictly positive for all $\varphi _{1,2}^{H}\in [0,1]$; therefore, the maximum is achieved when $\varphi _{1,2}^{H}=1$, yielding a payoff of $U_{2}(1)=1-2\gamma (1-\gamma )$. $\square$

Proof of Theorem 3

Notice first that symmetry implies that $\mu _{(1,L)}=\mu _{(3,H)}, \mu _{(2,L)}=\mu _{(2,H)}$, and $\mu _{(3,L)}=\mu _{(1,H)}$. Thus, $\frac{\mu _{(2,H)}}{\mu _{(2,L)}+\mu _{(2,H)}}=\frac{1}{2}$, implying that the expected payoff, conditional on being in state 2, is $\frac{1}{2}\gamma +\frac{1}{2}(1-\gamma )=\frac{1}{2}$. Therefore, the agent solves

$$\begin{aligned}&\max _{\varphi _{1,2}^{H},\varphi _{1,3}^{H},\varphi _{2,3}^{H}} \left\{ 2\left( \gamma \mu _{(3,H)}+\frac{1}{2}\mu _{(2,H)} +(1-\gamma )\mu _{(1,H)}\right) \right\} \\&\quad \text {s.t. } 0\le \varphi _{1,2}^{H},\varphi _{1,3}^{H},\varphi _{2,3}^{H}\le 1\\&\quad \text {and } \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\le 1. \end{aligned}$$

We begin by writing the steady-state condition for states $(1,H)$ and $(3,H)$ from Eq. (3) as

$$\begin{aligned} \mu _{(1,H)}&=(\alpha \mu _{(1,L)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,1}^{H}+(1-\gamma )\varphi _{1,1}^{L}\right) \\&\quad +(\alpha \mu _{(2,L)}+(1-\alpha )\mu _{(2,H)})\left( \gamma \varphi _{2,1}^{H}+(1-\gamma )\varphi _{2,1}^{L}\right) \\&\quad +(\alpha \mu _{(3,L)}+(1-\alpha )\mu _{(3,H)})\left( \gamma \varphi _{3,1}^{H}+(1-\gamma )\varphi _{3,1}^{L}\right) \text { and}\\ \mu _{(3,H)}&=(\alpha \mu _{(1,L)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,3}^{H}+(1-\gamma )\varphi _{1,3}^{L}\right) \\&\quad +(\alpha \mu _{(2,L)}+(1-\alpha )\mu _{(2,H)})\left( \gamma \varphi _{2,3}^{H}+(1-\gamma )\varphi _{2,3}^{L}\right) \\&\quad +(\alpha \mu _{(3,L)}+(1-\alpha )\mu _{(3,H)})\left( \gamma \varphi _{3,3}^{H}+(1-\gamma )\varphi _{3,3}^{L}\right) . \end{aligned}$$

Imposing symmetry and monotonicity, these may be written as

$$\begin{aligned} \mu _{(1,H)}&=(\alpha \mu _{(3,H)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \left( 1-\varphi _{1,2}^{H}-\varphi _{1,3}^{H}\right) +(1-\gamma )\right) \\&\quad +\mu _{(2,H)}(1-\gamma )\varphi _{2,3}^{H}+(\alpha \mu _{(1,H)}+(1-\alpha )\mu _{(3,H)})\left( (1-\gamma )\varphi _{1,3}^{H}\right) \\&=\mu _{(1,H)}\left( 1-\alpha -(1-\alpha )\gamma \varphi _{1,2}^{H}-(\gamma -\alpha )\varphi _{1,3}^{H}\right) +\mu _{(2,H)}(1-\gamma )\varphi _{2,3}^{H}\\&\quad +\mu _{(3,H)}\left( \alpha -\alpha \gamma \varphi _{1,2}^{H}+(1-\alpha -\gamma )\varphi _{1,3}^{H}\right) \text { and}\\ \mu _{(3,H)}&=(\alpha \mu _{(3,H)}+(1-\alpha )\mu _{(1,H)})\left( \gamma \varphi _{1,3}^{H}\right) +\mu _{(2,H)}\gamma \varphi _{2,3}^{H}\\&\quad +(\alpha \mu _{(1,H)}+(1-\alpha )\mu _{(3,H)})\left( \gamma +(1-\gamma )\left( 1-\varphi _{1,2}^{H}-\varphi _{1,3}^{H}\right) \right) \\&=\mu _{(1,H)}\left( \alpha -\alpha (1-\gamma )\varphi _{1,2}^{H}+(\gamma -\alpha )\varphi _{1,3}^{H}\right) +\mu _{(2,H)}\gamma \varphi _{2,3}^{H}\\&\quad +\mu _{(3,H)}\left( 1-\alpha -(1-\alpha )(1-\gamma )\varphi _{1,2}^{H}-(1-\alpha -\gamma )\varphi _{1,3}^{H}\right) . \end{aligned}$$

Combining the two equations above with the observation in Eq. (2) that

$$\begin{aligned} \mu _{(2,H)}=\frac{1}{2}-\mu _{(1,H)}-\mu _{(3,H)}, \end{aligned}$$

we can solve for $\mu _{(1,H)}$ and $\mu _{(3,H)}$. In particular, we have

$$\begin{aligned}&\mu _{(1,H)}=\frac{1}{2}\\&\quad \times \frac{\left( \alpha +\left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) (1-\gamma -\alpha )-\varphi _{1,2}^{H}(1-2\alpha )\gamma (1-\gamma )\right) \varphi _{2,3}^{H}}{\alpha \left( \varphi _{1,2}^{H}+2\varphi _{2,3}^{H}\right) +(1-2\alpha ) \left( \left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) \varphi _{2,3}^{H}+\varphi _{1,2}^{H} \left( \varphi _{1,2}^{H}+2\left( \varphi _{1,3}^{H}-\varphi _{2,3}^{H}\right) \right) \gamma (1-\gamma )\right) },\\&\mu _{(3,H)}=\frac{1}{2}\\&\quad \times \frac{\left( \alpha +\left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) (\gamma -\alpha )-\varphi _{1,2}^{H}(1-2\alpha )\gamma (1-\gamma )\right) \varphi _{2,3}^{H}}{\alpha \left( \varphi _{1,2}^{H}+2\varphi _{2,3}^{H}\right) +(1-2\alpha ) \left( \left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) \varphi _{2,3}^{H}+\varphi _{1,2}^{H} \left( \varphi _{1,2}^{H}+2\left( \varphi _{1,3}^{H}-\varphi _{2,3}^{H}\right) \right) \gamma (1-\gamma )\right) }. \end{aligned}$$

Furthermore, note that the decision maker’s expected payoff is

$$\begin{aligned} 2\left( \gamma \mu _{(3,H)}+\frac{1}{2}\mu _{(2,H)}+(1-\gamma )\mu _{(1,H)}\right) =\frac{1}{2}+(2\gamma -1)(\mu _{(3,H)}-\mu _{(1,H)}), \end{aligned}$$

where we have again substituted for $\mu _{(2,H)}$ using Eq. (2). This implies that the decision maker maximizes

$$\begin{aligned}&U_{3}\left( \varphi _{1,2}^{H},\varphi _{1,3}^{H},\varphi _{2,3}^{H}\right) \nonumber \\&:=\frac{1}{2}+\frac{1}{2}\frac{(2\gamma -1)^{2}\left( \varphi _{1,2}^{H} +\varphi _{1,3}^{H}\right) \varphi _{2,3}^{H}}{\alpha \left( \varphi _{1,2}^{H}+2\varphi _{2,3}^{H}\right) +(1-2\alpha ) \left( \left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) \varphi _{2,3}^{H}+\varphi _{1,2}^{H} \left( \varphi _{1,2}^{H}+2\left( \varphi _{1,3}^{H}-\varphi _{2,3}^{H}\right) \right) \gamma (1-\gamma )\right) }.\nonumber \\ \end{aligned}$$

(5)

Note first that, if $\varphi _{1,2}^{H}=0$ (and, by symmetry, $\varphi _{3,2}^{L}=0$), then the “middle” memory state (state 2) is effectively redundant—the memory system only makes use of the two extremal states. Applying Theorem 2, the optimal memory, conditional on $\varphi _{1,2}^{H}=0$, must have $\varphi _{1,3}^{H}=1$. As in the optimal two-state memory from Theorem 2, this memory yields an expected payoff of

$$\begin{aligned} U_{3}\left( 0,1,\varphi _{2,3}^{H}\right) =1-2\gamma (1-\gamma ). \end{aligned}$$

Clearly, the value of $\varphi _{2,3}^{H}$ is irrelevant in this case. However, in order to ensure that there is only a single recurrent communicating class, we simply set $\varphi _{2,3}^{H}=1$ when $\varphi _{1,2}^{H}=0$.

Suppose instead that $\varphi _{1,2}^{H}>0$. Then differentiating the payoff in Eq. (5) with respect to $\varphi _{2,3}^{H}$ yields

$$\begin{aligned}&\frac{\partial U_{3}\left( \varphi _{1,2}^{H},\varphi _{1,3}^{H},\varphi _{2,3}^{H}\right) }{\partial \varphi _{2,3}^{H}}\\&\quad =\frac{\varphi _{1,2}^{H}\left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) (\alpha +(1-2\alpha ) \left( \varphi _{1,3}^{H}+2\varphi _{2,3}^{H}\right) \gamma (1-\gamma ))(2\gamma -1)^{2}}{2\left( \alpha \left( \varphi _{1,2}^{H}+2\varphi _{2,3}^{H}\right) +(1-2\alpha ) \left( \left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) \varphi _{2,3}^{H}+\varphi _{1,2}^{H}\left( \varphi _{1,2}^{H}+2 \left( \varphi _{1,3}^{H}-\varphi _{2,3}^{H}\right) \right) \gamma (1-\gamma )\right) \right) ^{2}}. \end{aligned}$$

Clearly, the denominator is positive. Moreover, $\varphi _{1,2}^{H}>0$ implies that the numerator is positive. Thus, it is without loss of generality to set $\varphi _{2,3}^{H}=1$ whenever $\varphi _{1,2}^{H}>0$.

With this in mind, we consider two cases. We first assume that $\varphi _{1,2}^{H}>0$ and $\varphi _{1,2}^{H}+\varphi _{1,3}^{H}=1$. In this case, the decision maker’s payoff is $U_{3}(\varphi _{1,2}^{H},1-\varphi _{1,2}^{H},1)$. Note, however, that

$$\begin{aligned} \frac{\partial ^{2} U_{3}(\varphi _{1,2}^{H},1-\varphi _{1,3}^{H},1)}{\partial \left( \varphi _{1,3}^{H}\right) ^{2}} =\frac{(2\gamma -1)^{2}\left( \kappa +\alpha ^{2}-3\alpha \kappa \varphi _{1,2}^{H}+\left( \kappa \varphi _{1,2}^{H}\right) ^{2}\right) }{\left( 1+\alpha \varphi _{1,2}^{H}-\kappa \left( \varphi _{1,2}^{H}\right) ^{2}\right) ^{3}}, \end{aligned}$$

where we define $\kappa :=(1-2\alpha )\gamma (1-\gamma )$. Since $\alpha \in (0,\frac{1}{2})$ and $\gamma \in (\frac{1}{2},1)$, we must have $\kappa \in (0,1)$. Thus, $\varphi _{1,2}^{H}\in [0,1]$ implies that

$$\begin{aligned} 1+\alpha \varphi _{1,2}^{H}-\kappa \left( \varphi _{1,2}^{H}\right) ^{2} > 1-\kappa >0. \end{aligned}$$

In addition, we can write

$$\begin{aligned} \kappa +\alpha ^{2}-3\alpha \kappa \varphi _{1,2}^{H}+\left( \kappa \varphi _{1,2}^{H}\right) ^{2} =\left( 2\kappa \varphi _{1,2}^{H}-\alpha \right) \left( \kappa \varphi _{1,2}^{H}-\alpha \right) +\kappa . \end{aligned}$$

Note that $(2\kappa \varphi _{1,2}^{H}-\alpha )(\kappa \varphi _{1,2}^{H}-\alpha )$ is negative if, and only if, $2\kappa \varphi _{1,2}^{H}-\alpha >0>\kappa \varphi _{1,2}^{H}-\alpha$. But $2\kappa \varphi _{1,2}^{H}-\alpha <2\kappa$ and $\kappa \varphi _{1,2}^{H}-\alpha >-\alpha$, implying that

$$\begin{aligned} \left( 2\kappa \varphi _{1,2}^{H}-\alpha \right) \left( \kappa \varphi _{1,2}^{H}-\alpha \right) +\kappa >(2\kappa )(-\alpha )+\kappa =(1-2\alpha )\kappa >0. \end{aligned}$$

Thus, $U_{3}(\varphi _{1,2}^{H},1-\varphi _{1,2}^{H},1)$ is a convex function of $\varphi _{1,2}^{H}$, and is therefore maximized either when $\varphi _{1,2}^{H}=0$ or $\varphi _{1,2}^{H}=1$. The decision maker’s expected payoff in each of these cases is

$$\begin{aligned} U_{3}(0,1,1)=1-2\gamma (1-\gamma ) \text { and } U_{3}(1,0,1)=\frac{2+\alpha -4\gamma (1-\gamma )-\kappa }{2(1+\alpha -\kappa )}. \end{aligned}$$

Then we have

$$\begin{aligned} U_{3}(1,0,1)-U_{3}(0,1,1)&=\frac{(2+\alpha -4\gamma (1-\gamma )-\kappa )-2(1+\alpha -\kappa )(1-2\gamma (1-\gamma ))}{2(1+\alpha -\kappa )}\\&=-\frac{(\alpha -\gamma +2\alpha \gamma +\gamma ^{2}-2\alpha \gamma ^{2})(2\gamma -1)^{2}}{2(1+\alpha -\kappa )}\\&=\frac{(\kappa -\alpha )(2\gamma -1)^{2}}{2(1+\alpha -\kappa )}. \end{aligned}$$

Recalling the definition of $\kappa$, we may then conclude that $U_{3}(1,0,1)>U_{3}(0,1,1)$ if, and only if,

$$\begin{aligned} \frac{\alpha }{1-2\alpha }<\gamma (1-\gamma ). \end{aligned}$$

Turning to our second case, suppose that $\varphi _{1,2}^{H}>0$ and $\varphi _{1,2}^{H}+\varphi _{1,3}^{H}<1$. In addition, assume that $\varphi _{1,3}^{H}>0$. Therefore, the first-order conditions for both $\varphi _{1,2}^{H}$ and $\varphi _{1,3}^{H}$ must hold; that is, we have

$$\begin{aligned} \frac{\partial U_{3}(\varphi _{1,2}^{H},\varphi _{1,3}^{H},1)}{\partial \varphi _{1,2}^{H}}&=\frac{(2\gamma -1)^{2}\left( \left( 2-\varphi _{1,3}^{H}\right) \left( \alpha +\kappa \varphi _{1,3}^{H}\right) -\kappa \left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}\right) ^{2}\right) }{2\left( \varphi _{1,2}^{H} +\varphi _{1,3}^{H}+\left( 2-\varphi _{1,2}^{H}-2\varphi _{1,3}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) \right) ^{2}}=0 \text { and}\\ \frac{\partial U_{3}(\varphi _{1,2}^{H},\varphi _{1,3}^{H},1)}{\partial \varphi _{1,3}^{H}}&=\frac{(2\gamma -1)^{2}\left( 2+\varphi _{1,2}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) }{2\left( \varphi _{1,2}^{H}+\varphi _{1,3}^{H}+\left( 2-\varphi _{1,2}^{H}-2\varphi _{1,3}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) \right) ^{2}}=0. \end{aligned}$$

Solving these two equations yields

$$\begin{aligned} \varphi _{1,2}^{H}=\frac{\alpha }{\kappa } \text { and } \varphi _{2,3}^{H}=1-\frac{\alpha }{2\kappa }. \end{aligned}$$

Note, however, that these two expressions sum to more than 1, a contradiction. Thus, we must have $\varphi _{1,3}^{H}=0$, and only the first of the FOCs above can hold. This implies that

$$\begin{aligned} \varphi _{1,2}^{H}=\sqrt{\frac{2\alpha }{\kappa }}. \end{aligned}$$

(Of course, this is less than 1 if, and only if, $\frac{2\alpha }{(1-2\alpha )}<\gamma (1-\gamma )$; otherwise, we are at the corner solution where $\varphi _{1,2}^{H}=1$.) Note, however, that

$$\begin{aligned} U_{3}\left( \varphi _{1,2}^{H},0,1\right) -U_{3}(0,1,1)&=\frac{2\varphi _{1,2}^{H}(1-2\gamma (1-\gamma ))+\left( 2-\varphi _{1,2}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) }{2\left( \varphi _{1,2}^{H}+\left( 2-\varphi _{1,2}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) \right) }\\&\,\quad \,-(1-2\gamma (1-\gamma ))\\&=\frac{\left( \varphi _{1,2}^{H}-2\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) (2\gamma -1) ^{2}}{2\left( \varphi _{1,2}^{H}+\left( 2-\varphi _{1,2}^{H}\right) \left( \alpha -\kappa \varphi _{1,2}^{H}\right) \right) }>0 \end{aligned}$$

if, and only if, $\alpha <\kappa \varphi _{1,2}^{H}$. Since $\varphi _{1,2}^{H}=\sqrt{2\alpha /\kappa }$, this implies that $U_{3}(\sqrt{2\alpha /\kappa },0,1)>U_{3}(0,1,1)$ if, and only if, $\frac{\alpha }{2(1-2\alpha )}<\gamma (1-\gamma )$. $\square$

Lemma 4

The expected payoff of the four-state memory system depicted in Fig. 4 is

$$\begin{aligned} U_{4}:= \frac{1-\alpha ^{2}-3\gamma +4\alpha \gamma +3\gamma ^{2}-4 \alpha \gamma ^{2}}{1+\alpha (1-2\alpha )-2(1-2\alpha )\gamma (1-\gamma )}. \end{aligned}$$

Proof

Note that we can write the Eq. (3) steady-state condition for states $(1,H), (2,H)$, and $(3,H)$, for the case of the four-state memory in Fig. 4, as

$$\begin{aligned} \mu _{(1,H)}&=\alpha (1-\gamma )\mu _{(1,L)}+(1-\alpha )(1-\gamma ) \mu _{(1,H)}+\alpha (1-\gamma )\mu _{(2,L)}\\&\quad +(1-\alpha )(1-\gamma )\mu _{(2,H)},\\ \mu _{(2,H)}&=\alpha \gamma \mu _{(1,L)}\!+\!(1\!-\!\alpha )\gamma \mu _{(1,H)} \!+\!\alpha (1-\gamma )\mu _{(3,L)}\!+\!(1-\alpha )(1-\gamma )\mu _{(3,H)}, \text { and}\\ \mu _{(3,H)}&=\alpha \gamma \mu _{(2,L)}+(1-\alpha )\gamma \mu _{(2,H)} +\alpha (1-\gamma )\mu _{(4,L)}+(1-\alpha )(1-\gamma )\mu _{(4,H)}, \end{aligned}$$

where we have made use of the fact that the memory transition rule is given by

$$\begin{aligned} \varphi _{m,m'}^{s}:= {\left\{ \begin{array}{ll} 1 &{}\text {if } (m,m',s)=(1,2,H),(2,3,H),(3,4,H),(4,4,H),\\ 1 &{}\text {if } (m,m',s)=(1,1,L),(2,1,L),(3,2,L),(4,3,L),\\ 0 &{}\text {otherwise.} \end{array}\right. } \end{aligned}$$

Symmetry also implies that $\mu _{(1,L)}=\mu _{(4,H)}, \mu _{(2,L)}=\mu _{(3,H)}, \mu _{(3,L)}=\mu _{(2,H)}$, and $\mu _{(4,L)}=\mu _{(1,H)}$; therefore, we may write

$$\begin{aligned} \mu _{(1,H)}&=(1-\alpha )(1-\gamma )\mu _{(1,H)}+(1-\alpha ) (1-\gamma )\mu _{(2,H)}+\alpha (1-\gamma )\mu _{(3,H)}\\&\quad \;+\alpha (1-\gamma )\mu _{(4,H)},\\ \mu _{(2,H)}&=(1-\alpha )\gamma \mu _{(1,H)}\!+\!\alpha (1\!-\!\gamma )\mu _{(2,H)}\!+\!(1\!-\!\alpha ) (1-\gamma )\mu _{(3,H)}\!+\!\alpha \gamma \mu _{(4,H)}, \text { and}\\ \mu _{(3,H)}&=\alpha (1-\gamma )\mu _{(1,H)}+(1-\alpha )\gamma \mu _{(2,H)}+ \alpha \gamma \mu _{(3,H)}+(1-\alpha )(1-\gamma )\mu _{(4,H)}. \end{aligned}$$

In addition, recall from Eq. (2) that $\mu _{(1,H)}+\mu _{(2,H)}+\mu _{(3,H)}+\mu _{(4,H)}=\frac{1}{2}$. Solving this system of four equations in four unknowns yields the stationary distribution of this memory system, which is given by

$$\begin{aligned} \mu _{(1,H)}&=\frac{(1-\gamma )\left( 1-\alpha -(2-\alpha -\gamma )(1-2\alpha ) \gamma \right) }{2\left( 1+\alpha (1-2\alpha )-2(1-2\alpha )\gamma (1-\gamma )\right) },\\ \mu _{(2,H)}&=\frac{\alpha (1-\alpha )+(1-\alpha -\gamma ) (1-2\alpha )\gamma (1-\gamma )}{2\left( 1+\alpha (1-2\alpha )-2(1-2\alpha ) \gamma (1-\gamma )\right) },\\ \mu _{(3,H)}&=\frac{\alpha (1-\alpha )+(\gamma -\alpha )(1-2\alpha )\gamma (1-\gamma )}{2\left( 1+\alpha (1-2\alpha )-2(1-2\alpha )\gamma (1-\gamma )\right) }, \text { and}\\ \mu _{(4,H)}&=\frac{\gamma \left( 2\alpha (1-\alpha )+(\gamma -\alpha ) (1-2\alpha )\gamma \right) }{2\left( 1+\alpha (1-2\alpha )-2(1-2\alpha ) \gamma (1-\gamma )\right) }. \end{aligned}$$

Therefore, the expected payoff of this memory system is

$$\begin{aligned} U_{4}&:= \gamma (\mu _{(1,L)}+\mu _{(2,L)}+\mu _{(3,H)}+\mu _{(4,H)})\\&+(1-\gamma )(\mu _{(1,H)}+\mu _{(2,H)}+\,\mu _{(3,L)}+\mu _{(4,L)})\\&= 2\gamma (\mu _{(3,H)}+\mu _{(4,H)})+2(1-\gamma )(\mu _{(1,H)}+\mu _{(2,H)})\\&= \frac{\alpha \gamma +\alpha \gamma ^{2}-2\alpha \gamma ^{3} -\alpha ^{2}\gamma +\gamma ^{3}}{1+\alpha (1-2\alpha )-2(1-2\alpha )\gamma (1-\gamma )}\\&+\,\frac{1-\alpha ^{2}-3\gamma +3\alpha \gamma +\alpha ^{2}\gamma +3\gamma ^{2} -5\alpha \gamma ^{2}-\gamma ^{3}+2\alpha \gamma ^{3}}{1+\alpha (1-2\alpha ) -2(1-2\alpha )\gamma (1-\gamma )}\\&= \frac{1-\alpha ^{2}-(3-4\alpha )\gamma (1-\gamma )}{1+\alpha (1-2\alpha )-2(1-2\alpha )\gamma (1-\gamma )}. \end{aligned}$$

$\square$