Markov perfect equilibria in a dynamic decision model with quasi-hyperbolic discounting

Abstract

We study a discrete-time non-stationary decision model in which the preferences of the decision maker change over time and are described by quasi-hyperbolic discounting. A time-consistent optimal solution in this model corresponds with a Markov perfect equilibrium in a stochastic game with uncountable state space played by countably many short-lived players. We show that Markov perfect equilibria may be constructed using a generalized policy iteration algorithm. This method is in part inspired by the fundamental works of Mertens and Parthasarathy (in: Raghavan, Ferguson, Parthasarathy, Vrieze (eds) Stochastic games and related topics, Kluwer Academic Publishers, Dordrecht, 1991; in: Neyman, Sorin (eds) Stochastic games and applications, Academic Publishers, Dordrecht, 2003) devoted to subgame perfect equilibria in standard n-person discounted stochastic games. If the one-period utilities and transition probabilities are independent of time, we obtain on new existence results on stationary Markov perfect equilibria in the models with unbounded from above utilities.

1 Introduction

The issue of dynamic inconsistency in sequential decision models with changing preferences in time was pointed out in the seminal paper of Strotz (1956). The time preference studies found that discount rates are much grater in the short run than in the long run. To model this phenomena researchers have adopted discount functions from the class of generalized hyperbolas, see Ainslie (1992) or Harris and Laibson (2001) and references therein. The discrete-time analog of quasi-hyperbolic discounting involves the functions: $1, \delta \beta ,\delta \beta ^2,\ldots ,$ where $\beta \in (0,1)$ is a long run discount factor and $\delta >0$ is a short run discount factor. Such discounting was first used by Phelps and Pollak (1968). Specifically, they noticed that finding a time-consistent solution may be obtained by looking for a Nash equilibrium in certain games played by countably many short-lived players assuming that each player can act only once. Recently, Montiel Olea and Strzalecki (2014) provided an axiomatic characterization of quasi-hyperbolic preferences.

The existence of time-consistent solutions in models with changing tastes is a difficult problem. The akin problem is also met in altruistic economic growth models, known as intergenerational games. Initial positive results expressed in terms of intergenerational games (with a relatively simple utility function for each generation) were given by Bernheim and Ray (1983) and Leininger (1986). Peleg and Yaari (1973), on the other hand, provided some counterexamples for models with slightly less restrictive assumptions. The existence of a stationary Markov perfect equilibrium in stochastic game with finite state and action spaces with quasi-hyperbolic discounting was stated first in Alj and Haurie (1983). Applications of quasi-hyperbolic discounting in ecological problems can be found in Haurie (2005) and Karp (2005) and the references cited therein. For additional examples (especially in macroeconomics) and a detailed discussion on the literature the reader is referred to Balbus et al. (2016) and Jaśkiewicz and Nowak (2018).

In this paper, we study a discrete-time consumption/savings model with changing in time preferences. To construct a time-consistent solution in this model, the decision maker is represented by a sequence of temporal players called “selves”. The state space is $S=[0,+\,\infty )$ and represents the set of stocks of a renewable resource. Current self (self t) decides how to split the available resource stock into consumption and investment for future selves. The outcome of the investment (saving) is determined by some transition probability function. We assume that the stage utility functions and transition probability functions may depend on time and self t measures his satisfaction using so-called short-run and long-run discount factors according to the formula used in quasi-hyperbolic discounting. The total utility of self t depends on his own consumption and consumptions of all future selves. The model is actually a non-cooperative stochastic game with uncountable state space and denumerable set of players. This model was studied successfully by Harris and Laibson (2001) within a stationary framework. They proved the existence of a stationary Markov perfect equilibrium and derived a hyperbolic Euler relation that is useful in characterizing equilibria. The transition probability function in their model is non-atomic and has a special form with additive noise. More general non-atomic transition functions were examined in Balbus et al. (2015a), where the state space was assumed to be a compact interval. Related results were obtained in Balbus and Nowak (2008), Balbus et al. (2014) and Balbus et al. (2015c), but with very specific transition functions being a convex combination of probability measures on the state space. The proof of an existence of a stationary Markov perfect equilibrium in Harris and Laibson (2001) and Balbus et al. (2015a) was based on a fixed point argument in a space of functions with locally bounded variation. The existence of an equilibrium in the class of Lipschitz continuous functions, on the other hand, was shown by Balbus et al. (2015c), who also discussed computational issues.

The existence of subgame perfect equilibria in a large class of standard discounted n-person stochastic games proved by Mertens and Parthasarathy (2003). Their proof is based on some set-valued recursive equations whose solutions are the sets of Nash equilibrium vector payoffs in one-shot games involving the stage payoffs and the transition probability. Mertens and Parthasarathy (2003) also proposed an algorithm extending the well-known “value iteration” in discounted dynamic programming (see, e.g., Blackwell 1965). A simplified version of the algorithm was given in Mertens and Parthasarathy (1991). Certain modification of their method was applied by Balbus and Woźny (2016) to characterize the sets of Markov perfect equilibria in supermodular stochastic games and the consumption/savings problem with quasi-hyperbolic discounting and with very specific transition functions. The existence of an equilibrium is proved in two steps. First, they show that a system of recursive equations in the value function space has a solution. Second, relatively strong concavity assumptions imposed on the transition operator allow to obtain a Lipschitz continuous equilibrium. Finally, it is worth mentioning that a large class of dynamic games for which the Markov perfect equilibria exist and have a natural interpretation was indicated by Maskin and Tirole (2001).

This paper proposes a new method for proving the existence of Markov perfect equilibria in dynamic consumption/savings model with quasi-hyperbolic discounting. Our approach is also inspired by the method of Mertens and Parthasarathy (2003), but it more resembles “policy iteration” algorithm in dynamic programming. Our assumptions on the primitive data are much weaker than in the previous works of Harris and Laibson (2001) and Balbus et al. (2015a). The state space and the utility functions may be unbounded. Our first model concerns the transition probability function that is weakly continuous in investments and has no atoms in the positive part of the state space $(0,\infty ).$ The weighted-norm approach, proposed by Wessels (1977) and applied to this model, allows us for consideration of a large class of unbounded (e.g., power) utility functions. The second model, studied in this paper, deals with a special additive type of the transition function that may involve positive atoms. It is worthy to point out that a Markov perfect equilibrium is obtained in this paper from the non-empty intersection of some nested family of compact subsets of the space of strategy profiles of all selves. Therefore, fixed point theorems are not applied in our approach. However, the equilibrium is non-stationary, even if the period utility functions and transition probabilities are independent of time.

This paper is organized as follows. In Sect. 2, we describe a general decision model and basic definitions. Section 3 contains our main results on Markov perfect equilibria in a model with non-atomic transitions, whereas the transitions involving atoms are studied in Sect. 4. Additional comments on our results and related literature are given in Sect. 5.

2 The model

Let ${\mathbb {R}}$ be the set of all real numbers and $\mathbb {N}$ be the set of all positive integers. Define $S:=[0,+\infty ),$$S_+:=(0,+\infty )$ and, for each $s\in S,$$A(s):=[0,s].$ The set S is referred to as the state space. It represents the set of “levels” for the renewable resource and A(s) is the set of available actions (possible consumption levels) in state $s\in S.$ In a dynamic choice model with quasi-hyperbolic preferences and the state space S we envision an individual decision maker as a sequence of autonomous temporal selves. These selves are indexed by period numbers $t\in T:=\mathbb {N}$ in which they make their choices. More precisely, for a given state $s_t\in S$ at the beginning of t-th period, self t chooses a consumption level $a_t\in A(s_t)$ and the remaining part $y_t:=s_t-a_t$ is invested for future selves. Self t’s satisfaction is measured by a period utility function$u_\tau : S\rightarrow S$ for all $\tau \ge t.$

Let $q_t$ be a transition probability from S to S. Then, the state $s_{t+1}$ is generated by $q_t(\cdot |y_t)$ depending on the investment $y_t\in A (s_t).$

Let $\varPhi$ be the set of all Borel measurable functions $\phi : S\rightarrow S$ such that $\phi (s)\in A(s)$ for each $s\in S.$ A Markov strategy for self t is a function $c_t\in \varPhi$. We put $i_t(s)=s-c_t(s),$$s\in S.$ This is an investment strategy (or saving) of self t for following selves. For any sequence $(c_n)\in \varPhi ^\infty :=\varPhi \times \varPhi \times \cdots$ of strategies of all selves and any $t\in T,$ we define $c^t:=(c_t,c_{t+1},\ldots ).$ For any state $s_t$ and any $c^t\in \varPhi ^\infty ,$ the transition probabilities $q_\tau (\cdot |i_\tau (s))$ induced by $q_\tau$ with $\tau \ge t$ generate due to the Ionescu-Tulcea theorem (see Proposition V.1.1 in Neveu (1965)) a unique probability measure $P^{c^t}_{s_t}$ on $S^\infty$ endowed with the product $\sigma$-algebra. Let $E^{c^t}_{s_t}$ denote the expectation operator corresponding to the measure $P^{c^t}_{s_t}.$ Assume that for each $\tau \in T,$ the function $u_\tau :S\rightarrow S$ is continuous. Note that it is non-negative.

The expected utility of self t is

$$\begin{aligned} U_t(c^t)(s_t):= E_{s_t}^{c^t}\left( u_t(c_t(s_t))+ \delta \beta \sum _{\tau =t+1}^\infty \beta ^{\tau -t-1}u_{\tau }(c_{\tau } (s_\tau ))\right) , \end{aligned}$$

(1)

where $\beta \in (0,1)$ is a long-run discount factor and $\delta \in (0,1]$ is a short-run discount factor. The idea of using utility functions of the form (1) goes back to Phelps and Pollak (1968). A detailed discussion with some applications of time preferences represented by utility functions of the type considered here can be found in Harris and Laibson (2001) and Montiel Olea and Strzalecki (2014). Clearly, the expression defined in (1) is non-negative. In the sequel, we give conditions under which it is finite.

For any $c^n=(c_n,c_{n+1},\ldots )\in \varPhi ^\infty ,$$n\ge 2$ and $s_n\in S,$ let

$$\begin{aligned} J_n(c^n)(s_n)= E_{s_{n}}^{c^n}\left( \sum _{\tau =n}^\infty \beta ^{\tau -n}u_\tau (c_\tau (s_\tau ))\right) . \end{aligned}$$

(2)

Then we have

$$\begin{aligned} U_t(c^t)(s_t)= u_t\left( c_t(s_t)\right) + \delta \beta \int _S J_{t+1}(c^{t+1})(s_{t+1}) q_{t}(ds_{t+1}|s_t-c_t(s_t)). \end{aligned}$$

Define

$$\begin{aligned} P_t(a,c^{t+1})(s):= u_t(a) + \delta \beta \int _S J_{t+1}\left( c^{t+1}\right) (s_{t+1}) q_{t}(ds_{t+1}|s-a), \ s\in S,\ a\in A(s). \end{aligned}$$

(3)

Definition 1

A Markov Perfect Equilibrium (${ MPE}$) is a sequence $\hat{c} = (\hat{c}_t)_{t\in T}\in \varPhi ^\infty$ such that for every $s\in S$ and $t\in T,$ we have

$$\begin{aligned} \sup _{a\in A(s)}P_t\left( a,\hat{c}^{t+1}\right) (s)=P_t\left( \hat{c}_t(s),\hat{c}^{t+1}\right) (s)=U_t(\hat{c}^{t})(s). \end{aligned}$$

(4)

Definition 2

A Stationary Markov Perfect Equilibrium (${ SMPE}$) is an ${ MPE}$$\hat{c} = (\hat{c}_t)_{t\in T} \in \varPhi ^\infty$ such that $\hat{c}_t = c_0$ for some $c_0\in \varPhi$ and for all $t\in T.$

In a stationary ${ MPE}$ every self uses the same consumption strategy.

One can think that every self is a short-lived player in a non-cooperative game and acts only once. The payoff function of self $t\in T$ is given by (1). Then an ${ MPE}$$\hat{c}=(\hat{c}_t)_{t\in T}\in \varPhi ^\infty$ is a Nash equilibrium in this game.

Let $c=(c_t)_{t\in T}\in \varPhi ^\infty$ and

$$\begin{aligned} \tilde{u}_{c_{t+n}}(s_{t+n}):= u_{t+n}(c_{t+n}(s_{t+n})), \quad n\in T, \ s_{t+n}\in S. \end{aligned}$$

For any Borel measurable function $v:S\rightarrow [0,\infty ]$, define

$$\begin{aligned} Q_{c_n}v(s_n):=\int _S v(s)q_n(ds|s_n -c_n(s_n)). \end{aligned}$$

(5)

Consider the composition $Q_{c_{t+1}}\cdots Q_{c_{t+n}}\tilde{u}_{c_{t+1+n}}(s_{t+1}).$ In particular, note that

$$\begin{aligned} Q_{c_{t+1}}\tilde{u}_{c_{t+2}}(s_{t+1})= \int _S u_{t+2}(c_{t+2}(s_{t+2}))q_{t+1}(ds_{t+2}|s_{t+1} -c_{t+1}(s_{t+1})) \end{aligned}$$

and

$$\begin{aligned}&Q_{c_{t+1}} Q_{c_{t+2}}\tilde{u}_{c_{t+3}}(s_{t+1})\nonumber \\&\quad =\int _S\int _S u_{t+3}(c_{t+3}(s_{t+3}))q_{t+2}(ds_{t+3}|s_{t+2}-c_{t+2}(s_{t+2}))q_{t+1}(ds_{t+2}|s_{t+1}-c_{t+1}(s_{t+1})).\nonumber \\ \end{aligned}$$

(6)

By the Ionescu-Tulcea theorem, it follows that

$$\begin{aligned} J_{t+1}(c^{t+1})(s_{t+1})= u_{t+1}(c_{t+1}(s_{t+1}))+\sum _{n=1}^\infty \beta ^n Q_{c_{t+1}}\cdots Q_{c_{t+n}}\tilde{u}_{c_{t+1+n}}(s_{t+1}). \end{aligned}$$

(7)

Clearly, (7) is well-defined, non-negative, but it can be infinite. Below we make assumptions implying that $J_{t+1}(c^{t+1})(s_{t+1})$ is finite for any $s_{t+1}\in S,$$c^{t+1}\in \varPhi ^\infty$ and $t\in T.$

• (W1) There exists a continuous increasing function $w:S\rightarrow [1,\infty )$ such that $0\le u_t (s)\le w( s)$ for all $s\in S$ and $t\in T.$

• (W2) There exists a constant $\alpha >0$ such that $\alpha \beta <1$ and

  $$\begin{aligned} \int _S w(s)q_t(ds|y)\le \alpha w(y)\quad \text{ for } \text{ all } \quad y\in S, \ t\in T. \end{aligned}$$

• (W3) The function $y\rightarrow \int _Sw(s)q_t(ds|y)$ is continuous on S for each $t\in T.$

By (W1) and (W2), we have

$$\begin{aligned} 0\le U_t(c^t)(s_t)\le \left( 1+\frac{\delta \alpha \beta }{1-\alpha \beta }\right) w(s_t) <\infty \end{aligned}$$

(8)

for all $s_t\in S$ and $c^t\in \varPhi ^\infty .$

Assumptions of the above type were used to study discounted Markov decision processes with unbounded stage utility function, see Wessels (1977), Hernández-Lerma and Lasserre (1996) or Jaśkiewicz and Nowak (2011).

3 Markov perfect equilibria in models with non-atomic transitions

Let $\Pr (S)$ be the set of all probability measures on the state space S. We recall that a sequence $(\mu _n)_{n\in \mathbb {N}}$ of probability measures on Sconverges weakly to some $\mu _0\in \Pr (S)$ ($\mu _n \Rightarrow \mu _0$ in short) if, for any bounded continuous function $f:S\rightarrow {\mathbb {R}},$ it holds that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _Sf(s)\mu _n(ds)= \int _Sf(s)\mu _0(ds). \end{aligned}$$

We now formulate our main assumptions in this section:

• (U) For each $t\in T,$ the function $u_t :S\rightarrow S$ is increasing, strictly concave and continuous at $s=0.$

• (Q) The transition probability $q_t$ is weakly continuous on S, that is, for each $y_0\in S$ and $y_m \rightarrow y_0,$ we have $q_t(\cdot |y_m) \Rightarrow q_t(\cdot |y_0)$ as $m\rightarrow \infty .$ Moreover, for each $y\in S_+,$ the probability measure $q_t(\cdot |y)$ is non-atomic and $q_t(\cdot |0)$ has no atoms in $S_+.$

We now define some special classes of strategies of the players. By F, we denote the set of all continuous from the left mappings $c:S\rightarrow S$ such that the function $i(s):=s-c(s)$ is non-decreasing and $0\le c(s)\le s$ for all $s\in S.$ Note that i is lower semicontinuous. Thus, $c\in F$ is upper semicontinuous.

We can now state our first two main results.

Theorem 1

Assume that (W1)–(W3), (U) and (Q) are satisfied. Then there exists an ${ MPE}$$\hat{c}=(\hat{c}_t)$ that belongs to the space $F^\infty :=F\times F\times \cdots .$

Theorem 2

Assume that (W1)–(W3), (U) and (Q) hold and the model is stationary, i.e., $q_t=q$ and $u_t=u$ are independent of $t\in T.$ Then there exists an SMPE$\hat{c}=(c_0,c_0,\ldots )$ with $c_0\in F.$

In the proof of Theorem 1 we define by backward induction a nested family of subsets of $F^\infty$ that has a non-empty intersection. A Markov perfect equilibrium is a sequence that belongs to this intersection. This method resembles a “set-valued dynamic programming” approach and is based on two important factors: the continuity of expected utilities with respect to some natural topology on the space $F^\infty$ and the continuity of the best response self t’s mapping defined on the space of sequences of future selves strategies. Observe that the assertion of Theorem 2 is stronger compared with Theorem 1, but the assumptions are also stronger (stationarity of the model). To obtain an SMPE in Theorem 2 we shall need to apply a fixed point theorem, which was not needed in the proof of Theorem 1.

Let X be the vector space of all continuous from the left functions $f:S\rightarrow {\mathbb {R}}$ such that $f(0)=0.$ It is also assumed that each $f\in X$ has bounded variation on every interval [0, m],  $m\in \mathbb {N}.$ We assume that X is endowed with the topology of weak convergence. Recall that a sequence $(f_n)_{n\in \mathbb {N}}$ converges weakly to $f \in X$ if and only if $f_n(s)\rightarrow f(s)$ as $n\rightarrow \infty$ at any continuity point $s\in S$ of f. Here we point out that $s=0$ is considered as a continuity point of $f\in X$ if $\lim _{s\rightarrow 0^+}f(s)=f(0).$ The weak convergence of $(f_n)_{n\in \mathbb {N}}$ to f is denoted by $f_n{\mathop {\rightarrow }\limits ^{\omega }} f$. Note that F is a metrizable topological subspace of X (see Appendix).

Let

$$\begin{aligned} I:=\{ i\in X: i(s)=s-c(s)\ \text{ where }\ s\in S,\ c\in F \}. \end{aligned}$$

Note that every $c\in F$ is upper semicontinuous and continuous from the left. The function $s\rightarrow s-c(s)$ belonging to the space I is non-decreasing and lower semicontinuous. Observe that $I\subset X.$ Moreover, $s=0$ is the continuity point of every function in F or I.

Lemma 1

The sets I and F are convex and sequentially compact in X.

Proof

It is obvious that I is convex. For any $f\in I$ and $m\in \mathbb { N}$, we define the function $f^m$ as follows: $f^m(s)=f(s)$ for all $s\in [0,m]$ and $f^m(s) = m$ for all $s> m.$ Then $f^m$ can be viewed as a continuous from the left “distribution function” of some non-negative countably additive measure $\rho _m$ such that $\rho _m(S)=m.$ Consider an arbitrary sequence $(f_n )_{n\in \mathbb {N}}$ of functions in I. We now apply the standard “diagonal method”. By Helly’s theorem (see Billingsley 1968), there exists a subsequence $(n_1(k))$ of (n) such that $(f^1_{n_{1}(k)})_{k\in \mathbb {N}}$ converges weakly (as $k\rightarrow \infty$) to some $f^1_o\in I.$ Next, there exists a subsequence $(n_2(k))$ of $(n_1(k))$ such that $(f^2_{n_{2}(k)})_{k\in \mathbb {N}}$ converges weakly to some $f^2_o\in I$ and $f^2_o(s)=f^1_o(s)$ for each $s\in [0,1].$ Proceeding along this way, we infer that for any $r\ge 2$, there exists a subsequence $(n_r(k))$ of $(n_{r-1}(k))$ such that $(f^r_{n_{r}(k)})_{k\in \mathbb {N}}$ converges weakly to some $f^r_o\in I$ and $f^r_o(s)=f^{r-1}_o(s)$ for each $s\in [0,r-1].$ Define $f_o(s):=f^m_o(s)$ if $s\in [0,m],$$m\in \mathbb {N}.$ Then $f_o\in I.$ Consider the “diagonal sequence” defined by $d(k):= n_k(k)$, $k\in \mathbb {N}.$ Then, $f_{d(k)}{\mathop {\rightarrow }\limits ^{\omega }} f_o$ as $k\rightarrow \infty .$ Thus, I is sequentially compact. Since F is obtained from I by a simple continuous transformation, the result also holds for F. $\square$

By Lemma 1 and Tychonoff’s theorem, we obtain the following auxiliary result.

Corollary 1

The space $F^\infty$ endowed with the product topology is sequentially compact.

Let $(g_m)_{m\in \mathbb {N}}$ be a sequence of Borel measurable real-valued functions on S. For each $s\in S,$ define

$$\begin{aligned} g_*(s) := \inf \{\liminf _{m\rightarrow \infty } g_m(s^m):\;s^m\rightarrow s\} \quad \text{ and }\quad g^*(s) := \sup \{\limsup _{m\rightarrow \infty } g_m(s^m):\;s^m\rightarrow s\}. \end{aligned}$$

Lemma 2

Assume that $\mu _m\Rightarrow \mu$ and $\int _Sw(s)\mu _m(ds) \rightarrow \int _Sw(s)\mu (ds)$ as $m\rightarrow \infty .$ Suppose that there exists $\lambda >0$ such that $0\le g_m(s)\le \lambda w(s)$ for all $s\in S$ and $m\in \mathbb {N}.$ Then

$$\begin{aligned}&\int _S g_*(s)\mu (ds) \le \liminf _{m\rightarrow \infty } \int _S g_m(s) \mu _m(ds)\quad \mathrm{and} \end{aligned}$$

(9)

$$\begin{aligned}&\int _S g^*(s)\mu (ds) \ge \limsup _{m\rightarrow \infty } \int _S g_m(s) \mu _m(ds). \end{aligned}$$

(10)

Proof

Inequality (9) follows from Lemma 3.2 in Serfozo (1982). We show how to obtain (10). Let $\varphi _m(s):=g_m(s)-\lambda w(s),$$s\in S,$$m\in \mathbb {N}.$ By Lemma 3.2 in Serfozo (1982), we have

$$\begin{aligned} \limsup _{m\rightarrow \infty }\int _S\varphi _m(s)\mu _m(ds)\le \int _S (g^*(s)-\lambda w(s))\mu (ds)= \int _S g^*(s)\mu (ds) -\lambda \int _S w(s))\mu (ds). \end{aligned}$$

On the other hand, we have

$$\begin{aligned} \limsup _{m\rightarrow \infty }\int _S\varphi _m(s)\mu _m(ds)= & {} \limsup _{m\rightarrow \infty } \int _S g_m(s)\mu _m(ds) - \lambda \lim _{m\rightarrow \infty }\int _S w(s)\mu _m(ds) \nonumber \\= & {} \limsup _{m\rightarrow \infty } \int _S g_m(s)\mu _m(ds) - \lambda \int _S w(s)\mu (ds). \end{aligned}$$

Hence, (10) follows. $\square$

Lemma 3

Let the assumptions of Lemma 2 be satisfied. Assume that $g:S\rightarrow {\mathbb {R}}$ is a Borel measurable function and $S^d$ is a denumerable subset of $S_+$ such that for any $s\in S{\setminus } S^d$ and $s^m\rightarrow s$ (as $m\rightarrow \infty$), we have $g_m(s^m) \rightarrow g (s).$ If, in addition $\mu (S^d)=0,$ then

$$\begin{aligned} \int _Sg_m(s)\mu _m(ds) \rightarrow \int _Sg(s)\mu (ds). \end{aligned}$$

(11)

Proof

We have, $g(s) =g_*(s) =g^*(s)$ for each $s\in S{\setminus } S^d.$ Since $\mu (S^d)=0,$ it follows that

$$\begin{aligned} \int _Sg(s)\mu (ds) = \int _Sg_*(s) \mu (ds)=\int _Sg^*(s)\mu (ds). \end{aligned}$$

This fact and Lemma 2 imply that

$$\begin{aligned} \limsup _{m\rightarrow \infty }\int _Sg_m(s) \mu _m(ds)\le \int _Sg(s)\mu (ds)\le \liminf _{m\rightarrow \infty }\int _S g_m(s)\mu _m(ds). \end{aligned}$$

Hence (11) follows. $\square$

Let $S_c$ denote the set of continuity points of $c\in X.$ If $c\in F,$ then $0\in S_c$ and $S{\setminus } S_c$ is a denumerable set. The proof of the following result is the same as that of Lemma 3.5 in Balbus et al. (2015a).

Lemma 4

Let $c^m {\mathop {\rightarrow }\limits ^{\omega }} c$ in F. If $s^m\in S$ for every $m\in \mathbb {N}$ and $s^m\rightarrow s\in S_c$ as $m\rightarrow \infty ,$ then $\lim _{m\rightarrow \infty }c^m(s^m)=c(s)$ and $\lim _{m\rightarrow \infty }i^m(s^m)=i(s).$

Consider $c^{\tau ,m}=(c^m_\tau ,c^m_{\tau +1},\ldots )$ and $c^{\tau }=(c_\tau ,c_{\tau +1},\ldots )$ that belong to $F^\infty$ endowed with the product topology. Then $c^{\tau ,m}\rightarrow c^{\tau }$ (as $m\rightarrow \infty$) if and only if, for any $n\ge \tau ,$$c_n^m {\mathop {\rightarrow }\limits ^{\omega }} c_n$ in F.

Lemma 5

Assume that (W1)–(W3), (U) and (Q) hold. If $c^{t+1,m} \rightarrow c^{t+1}$ in $F^\infty$, $s_{t+1}\in S_{c_{t+1}}$ and $s^m_{t+1}\rightarrow s_{t+1}$ as $m\rightarrow \infty$, then

$$\begin{aligned} \lim _{m\rightarrow \infty } J_{t+1}(c^{t+1,m})(s^m_{t+1})= J_{t+1}(c^{t+1})(s_{t+1}). \end{aligned}$$

(12)

Proof

Fix any $k\in \mathbb {N}$ and $s_{t+1+k}\in S_{c_{t+1+k}}.$ Assume that $s^m_{t+1+k}\rightarrow s_{t+1+k}$ as $m\rightarrow \infty .$ By Lemma 4 and assumption (U) we have $u_{t+1+k}(c^m_{t+1+k}(s^m_{t+1+k}))\rightarrow u_{t+1+k}(c_{t+1+k}(s_{t+1+k})).$ Let $s_{t +k}\in S_{c_{t +k}}.$ Consider any $s^m_{t +k}\rightarrow s_{t +k}$ as $m\rightarrow \infty$ and define

$$\begin{aligned}&g_m(\cdot )=u_{t+1+k}(c^m_{t+1+k}(\cdot )), \quad g(\cdot )=u_{t+1+k}(c_{t+1+k}(\cdot )),\\&\mu _m(\cdot )=q_{t+k}(\cdot |s^m_{t+k}-c^m_{t+k}(s^m_{t+k})),\quad \mu (\cdot )=q_{t+k}(\cdot |s_{t+k}-c_{t+k}(s^m_{t+k})). \end{aligned}$$

Then, by Lemma 3 (take $\lambda =1$ in Lemma 2), we obtain that $Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s^m_{t+k})\rightarrow Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s_{t+k}).$ Hence, we have $Q_{c^m_{t+1}}\tilde{u}_{c^m_{t+2}}(s^m_{t+1})\rightarrow Q_{c_{t+1}}\tilde{u}_{c_{t+2}}(s_{t+1}).$ Suppose that $k\ge 2.$ Consider any $s_{t +k-1}\in S_{c_{t +k-1}}$ and $s^m_{t +k-1}\rightarrow s_{t +k-1}$ as $m\rightarrow \infty .$ By Lemma 3 with $g_m(\cdot )=Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(\cdot )$ and $g(\cdot )=Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(\cdot )$ (take $\lambda =\alpha$ in Lemma 2), we conclude that

$$\begin{aligned} Q_{c^m_{t+k-1}}Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s^m_{t+k-1})\rightarrow Q_{c_{t+k-1}}Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s_{t+k-1}). \end{aligned}$$

Applying Lemma 3k times we finally obtain that

$$\begin{aligned} Q_{c^m_{t+1}}\cdots Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s^m_{t+1}) \rightarrow Q_{c_{t+1}}\cdots Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s_{t+1}), \text{ as } m\rightarrow \infty . \end{aligned}$$

(13)

Let $N\in \mathbb {N}$ and let

$$\begin{aligned} J^N_{t+1}(c^{t+1})(s_{t+1})= u_{t+1}(c_{t+1}(s_{t+1}))+\sum _{k=1}^N \beta ^k Q_{c_{t+1}}\cdots Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s_{t+1}). \end{aligned}$$

Similarly, we define $J^N_{t+1}(c^{t+1,m})(s^m_{t+1}).$ From (13) and Lemma 4, it follows that for any N, we have

$$\begin{aligned} \lim _{m\rightarrow \infty } J^N_{t+1}(c^{t+1,m})(s^m_{t+1})=J^N_{t+1}(c^{t+1})(s_{t+1}). \end{aligned}$$

(14)

Note that, since $s^m_{t+1}\rightarrow s_{t+1}$ as $m\rightarrow \infty$ and w is continuous, then there exists $b>0$ such that $2\le w(s^m_{t+1})+w(s_{t+1})\le b$ for all $m\in \mathbb {N}.$ Thus, we have

$$\begin{aligned}&|J_{t+1}(c^{t+1,m})(s^m_{t+1}) -J_{t+1}(c^{t+1})(s_{t+1})| \nonumber \\&\quad \le |J^N_{t+1}(c^{t+1,m})(s^m_{t+1}) -J^N_{t+1}(c^{t+1})(s_{t+1})| +\frac{\alpha ^{N+1}\beta ^{N+1}}{1-\alpha \beta }(w(s^m_{t+1})+w(s_{t+1})) \nonumber \\&\quad \le |J^N_{t+1}(c^{t+1,m})(s^m_{t+1}) -J^N_{t+1}(c^{t+1})(s_{t+1})| +\frac{b\alpha ^{N+1}\beta ^{N+1}}{1-\alpha \beta }. \end{aligned}$$

(15)

Using (14) and (15), one can easily show that (12) holds. $\square$

Observe that assumption (U) implies that every function $u_t$ is continuous on the space S. Now from Lemmas 3 and 5 with $g_m=g=J_{t+1}(c^{t+1})$ (note that $\lambda =\frac{\alpha \beta }{1-\alpha \beta }$ in Lemma 2, compare with (8)), we conclude the following auxiliary result.

Lemma 6

If (W1)–(W3), (U) and (Q) hold, then for any $c^{t+1 } \in F^\infty ,$ the function $a\rightarrow P_t(a,c^{t+1})$ is continuous on A(s).

Let $c^{t+1} \in F^\infty .$ Put

$$\begin{aligned} BR_t(c^{t+1})(s):= \text{ arg }\max _{a\in A(s)}P_t(a,c^{t+1})(s). \end{aligned}$$

(16)

The set $BR_t(c^{t+1})(s)$ can be regarded as the set of all best responses of self $t \in T$ in state s,  given that the following selves are going to use $c^{t+1}\in F^\infty .$ Under our assumptions, this set is non-empty and compact by Lemma 6. For any $s\in S$ and $t\in T$ let us define

$$\begin{aligned} br_t(c^{t+1})(s) :=\max BR_t(c^{t+1})(s). \end{aligned}$$

(17)

A simple adaptation of the arguments given in the proof of Theorem 6.3 in Topkis (1978) gives the following result.

Lemma 7

Let (W1)–(W3), (U) and (Q) be satisfied. Then

• (a) Let $\overline{BR}_t(c^{t+1})(s):= \{s-a: a\in BR_t(c^{t+1})(s)\}.$ The correspondence $s\rightarrow \overline{BR}_t(c^{t+1})(s)$ is compact-valued and strongly ascending, i.e., if $s_1<s_2$ and $b_i\in \overline{BR}_t(c^{t+1})(s_i)$ ($i=1,2$), then $b_1\le b_2.$

• (b) The function $br_t(c^{t+1})$ belongs to F,  i.e., $i_t(c^{t+1})(s):= s - br_t(c^{t+1})(s)\in A(s)$ for all $s\in S$ and $i_t(c^{t+1})$ is non-decreasing and continuous from the left.

• (c) If $\phi :S\rightarrow S$ is such that $\phi (s)\in BR_t(c^{t+1})(s)$ for all $s\in S$ and $s_0$ is a continuity point of $\phi$, then $BR_t(c^{t+1})(s_0)$ is a singleton.

• (d) The only function $\phi \in F$ such that $\phi (s)\in BR_t(c^{t+1})(s)$ for all $s\in S$ is $br_t(c^{t+1}).$

Proof

For parts (a)–(c) consult Lemmas 3.2 and 3.3 in Balbus et al. (2015a). If $\phi \in F,$ then the function i given by $i(s)=s-\phi (s)$ is non-decreasing and continuous from the left. If $s\in S_\phi ,$ then we have $\phi (s)=br_t(c^{t+1})(s)$ by (c). Assume that $s_o\in S{\setminus } S_\phi .$ Since i is continuous from the left, non-decreasing and the correspondence $s\rightarrow \overline{BR}_t(c^{t+1})(s)$ is strongly ascending, then $i(s_o)\le s_o-a$ for all $a\in BR_t(c^{t+1})(s_o).$ Hence, $\phi (s_o)=s_o-i(s_o) =br_t(c^{t+1})(s_o).$$\square$

Lemma 8

Assume that (W1)–(W3), (U) and (Q) hold. Then, the mapping $br_t: F^\infty \rightarrow F$ defined in (17) is continuous.

Proof

Let $c^{t+1,m} \rightarrow c^{t+1}$ as $m\rightarrow \infty .$ For notational convenience put $\sigma ^m:=c^{t+1,m},$$\sigma := c^{t+1}$ and $\psi ^m:= br_t(\sigma ^m).$ Let $\psi$ be any accumulation point of the sequence $(\psi ^m)$ in F being a compact space. We have to show that $\psi =br_t(\sigma ).$ Consider any $s\in S_\psi .$ Using Lemmas 3 and 5, one can show that $\psi (s)\in BR_t(\sigma )(s).$ Since $s\in S_\psi ,$ by Lemma 7(c), the set $BR_t(\sigma )(s)$ is a singleton and thus $\psi (s)=br_t(\sigma )(s).$ If $s_0\in S{\setminus } S_\psi ,$ then from the facts that $S_\psi$ is dense in S and $\psi$ is continuous from the left, it also follows that $\psi (s_0)\in BR_t(\sigma )(s_0).$ Since $\psi \in F,$ by Lemma 7(d), it follows that $\psi (s_0)=br_t(\sigma )(s_0).$$\square$

By backward induction we now define a family of sets used in the proof of Theorem 1. Let $\sigma =c^{t+1}\in F^\infty .$ Define $x_t(\sigma ):= br_t(\sigma )$ and $x_j(\sigma ):= br_j(x_{j+1}(\sigma ),\ldots ,x_t(\sigma ),\sigma )$ for $j=1,\ldots ,t-1.$ Let

$$\begin{aligned} G_t:=\left\{ (x_1(\sigma ),\ldots ,x_t(\sigma ),\sigma ): \sigma =c^{t+1}\in F^\infty \right\} . \end{aligned}$$

(18)

Clearly, $G_t\subset F^\infty .$ For clarity of the notation it could be helpful for the reader to consider the set $G_t$ for some specified value of t. Assume that $t=4.$ Then $\sigma =c^5= (c_5,c_6,\ldots ),$$x_4(\sigma ) =br_4(\sigma ),$$x_3(\sigma ) =br_3(br_4(\sigma ),\sigma ).$ Further, we have $x_2(\sigma )=br_2(br_3(br_4(\sigma ),\sigma ),br_4(\sigma ),\sigma )$ and

$$\begin{aligned} x_1(\sigma )=br_1(br_2(br_3(br_4(\sigma ),\sigma ),br_4(\sigma ),\sigma ), br_3(br_4(\sigma ),\sigma ),br_4(\sigma ),\sigma ). \end{aligned}$$

Note that $x_t(\sigma )$ is the best response of self t to $\sigma =c^{t+1}=(c_{t+1},c_{t+2},\ldots )$ assumed to be chosen by following selves, $x_{t-1}(\sigma )$ is the best response of self $t-1$ to the sequence $(br_t(c^{t+1}),c^{t+1})$ and so on.

Proof of Theorem 1

First note that for any $t\in T,$ we have $G_{t+1}\subset G_{t}.$ Since all best response mappings are continuous (Lemma 8) and the space $F^\infty$ is compact, every set $G_t$ is non-empty and compact. Therefore, the set $G:=\cap _{t\in T} G_t\not = \emptyset$ and G is compact. Choose any $\hat{c}=(\hat{c}_1,\hat{c}_2,\ldots )\in G.$ Then $\hat{c}\in G_t$ for every $t\in T.$ This implies immediately that, for every $t\in T,$ we have $\hat{c}_t=br_t(\hat{c}^{t+1}).$ Hence, $\hat{c}$ is an MPE. $\square$

Proof of Theorem 2

In the stationary case, we can restrict attention to constant sequences $(c,c,\ldots )\in F^\infty .$ Every such a sequence can be identified with $c\in F.$ Function (3) can be regarded as a function P(a, c)(s),  where $a\in A(s),$$c\in F.$ The best response mapping $br(c):= br_t(c^{t+1})$ (where t is arbitrary and $c^{t+1}=(c,c,\ldots )$ can be identified with $c\in F$) is by Lemma 8 continuous on the convex compact set F in the space X (Lemma 1). From the Schauder–Tychonoff fixed point theorem (see Aliprantis and Border 2006), it follows that there exists $c^*\in F$ such that $c^* =br(c^*)$. Clearly, the constant sequence $(c^*,c^*,\ldots )$ is an SMPE. $\square$

A natural example of transition probability that satisfies assumption (Q) is induced by the following equation

$$\begin{aligned} s_{t+1}= \bar{f} (y_t,\xi _t), \end{aligned}$$

where $y_t=s_t - a_t$ is the investment in state $s_t$, $(\xi _t)_{t\in T}$ is a sequence of i.i.d. random “shocks” having a probability distribution $\pi .$ The function $\bar{f}$ is continuous and for any Borel set D in S and investment $y\in S$

$$\begin{aligned} q _t(D|y):=q (D|y)= \int _S 1_D(\bar{f}(y,z)) \pi (dz),\quad t\in \mathbb {N}, \end{aligned}$$

where $1_D$ is the indicator function of the set D.

We now point out three special cases of our assumptions:

Example 1

The functions $\bar{f}(y_t,\xi _t)= \xi _tf_1(y_t) +(1-\xi _t)f_2(y_t)$, where $f_1, f_2:S \rightarrow S$ are continuous, increasing and such that $f_1(y)<f_2(y)$ for each $y\in S_+$ and $f_1(0)=f_2(0)=0.$ For instance, let $f_1(y)=y$ and $f_2(y)=y+\sqrt{y}.$ In addition, $\pi$ is a non-atomic probability measure on [0, 1]. For any $y>0,$$q(\cdot |y)$ is a non-atomic measure such that $q([f_1(y),f_2(y)]|y)=1.$ In particular, if $\pi$ has the uniform distribution on [0, 1], then $q(\cdot |y)$ has the uniform distribution on $[f_1(y),f_2(y)].$ Further assume that the utility function is independent of t,  e.g., $u(s)= s^\sigma$ with $\sigma \in (0,1).$ Then, setting $w(s)=(s+r)^\sigma ,$$r\ge 1,$ it follows that the assumptions (W1) and (U) are satisfied. Moreover, (W3) and (Q) also hold. Note that $q(\{0\}|0)=1.$ Now we prove (W2). Assuming that each $\xi _t$ has the uniform distribution on [0, 1], we obtain by Jensen’s inequality that

$$\begin{aligned} \frac{ \int _S w(s)q(ds|y)}{w(y)}= & {} \frac{\int _0^1 (z f_1(y) +(1-z)f_2(y)+r)^\sigma dz}{w(y)} \le \frac{(1/2f_1(y) +1/2 f_2(y)+r)^\sigma }{(y+r)^\sigma }\\= & {} \left( \frac{\frac{1}{2}y+\frac{1}{2}(y+\sqrt{y})+r}{y+r} \right) ^\sigma = \left( 1+ \frac{1}{2}\frac{ \sqrt{y} }{r+y}\right) ^\sigma \le \left( 1+ \frac{1}{4 \sqrt{ r} } \right) ^\sigma =:\eta ^\sigma (r). \end{aligned}$$

The function $r\rightarrow \eta (r)$ is decreasing and $\lim _{r\rightarrow \infty }\eta (r)=1.$ Hence, for any $\beta \in (0,1)$ we may choose sufficiently large $\hat{r}\ge 1$ such that $\beta \eta ^\sigma (\hat{r})<1.$ This shows that (W2) holds with $\alpha :=\eta ^\sigma (\hat{r}).$ The functions u and $f_1$ and $f_2$ may depend on t.

Example 2

The model with additive shocks: $\bar{f} (y_t,\xi _t)= f (y_t) +\xi _t,$ where $f :S\rightarrow S$ is a continuous increasing function. For instance, assume that $f(y)=\ln (1+y)$ for all $y\in S.$ The probability measure $\pi$ is non-atomic with support included in $[0, +\infty )$ and $\mathbb {E}\xi =\int _S z\pi (dz)<+\infty .$ Assume again that u is as in Example 1. Then, the function $w(s)=(s+r)^\sigma$ satisfies our assumptions. Indeed, observe that

$$\begin{aligned} \frac{ \int _S w(s)q(ds|y)}{w(y)}= & {} \frac{\int _S ( f(y) + \xi +r)^\sigma \pi (d\xi )}{w(y)} \le \frac{(f(y) +\mathbb {E}\xi +r)^\sigma }{(y+r)^\sigma }\\&\ =&\left( \frac{f(y)+r}{y+r} +\frac{\mathbb {E}\xi }{y+r} \right) ^\sigma \le \left( 1+\frac{\mathbb {E}\xi }{r}\right) ^\sigma =:\eta _0^\sigma (r). \end{aligned}$$

Hence, for any $\beta \in (0,1)$ we can choose sufficiently large $\tilde{r}\ge 1$ such that $\eta _0^\sigma (\tilde{r})\beta <1.$ This proves (W2) with $\alpha :=\eta _0^\sigma (\tilde{r}).$ Moreover, (W3) holds as well. In contrast to Example 1 note that $q(\{0\}|0)\not =1.$

Example 3

The model with multiplicative shocks: $\bar{f}(y_t,z_t)= f (y_t)\xi _t,$ where f is as in Example 2 and the probability measure $\pi$ is non-atomic with support included in $[0,+\,\infty ).$

Other functions w for which the transition probabilities of the type discussed above satisfy conditions (W1)–(W3) and (Q), (U) can be obtained by an adaptation of examples from Section 4 in Jaśkiewicz and Nowak (2011).

4 Markov perfect equilibria in models with transitions having atoms

The model considered in the previous section does not include deterministic transitions. In the deterministic case the basic continuity lemmas are doubtful in the class of discontinuous strategies $F^\infty .$ A discussion of this issue can be found in Balbus et al. (2015a). In this section, we study some models involving atoms in the set $S_+,$ but the transition probability has an additive form.

With any $c\in \varPhi$ we associate $i\in \varPhi$ given by $i(s):=s-c(s)$ and next define

$$\begin{aligned} F_L:= \{c\in \varPhi : \; c \text{ and } i \text{ are } \text{ non-decreasing }\}. \end{aligned}$$

(19)

It is easy to see that $F_L$ consists of Lipschitz functions with constant one.

We now assume that $q=q_t$ for each $t\in T$ and the following conditions hold.

• (A1) There exist probability measures $\nu _0, \nu _1,\ldots ,\nu _l$ on S and functions $h_0, h_1,\ldots ,h_l: S\rightarrow [0,1]$ such that

  $$\begin{aligned}q(\cdot |s-a )=\sum _{i=0}^l h_i(s-a)\nu _i (\cdot ),\end{aligned}$$

  where the functions $h_i$ are continuous and

  $$\begin{aligned}\sum _{i=0}^l h_i(s-a)=1\quad \text{ for } \text{ all } s\in S,\ a\in A(s).\end{aligned}$$

• (A2) Every function $h_i$ for $i=1,\ldots ,l$ is increasing and strictly concave.

• (A3) There exists a constant $M>0$ such that $\sup _{t\in T}\max _{i\in \{0,\ldots ,l\}}\int _S u_t(s)\nu _i(ds)\le M.$

• (A4) $\nu _i \succ \nu _0,$ i.e., $\nu _i$ stochastically dominates $\nu _0$ for each $i=1,\ldots ,l.$

It is known (see Topkis 1998) that (A4) holds if and only if for any non-decreasing function $v:S\rightarrow {\mathbb {R}}$ integrable with respect to every $\nu _i$, we have

$$\begin{aligned} \int _S v(s)\nu _i (ds) \ge \int _S v(s)\nu _0 (ds), \quad k=1,\ldots ,l. \end{aligned}$$

(20)

Some comments on the above assumptions are included in Remarks 1 and 2 at the end of this section.

We now state our main results in this section.

Theorem 3

Assume that (A1)–(A4) and (U) are satisfied. Then there exists an ${ MPE}$$\hat{c}=(\hat{c}_t)$ that belongs to the space $F_L^\infty :=F_L\times F_L\times \cdots .$

Theorem 4

Assume that (A1)–(A4) and (U) hold and the model is stationary, i.e., $u_t=u$ are independent of $t\in T.$ Then there exists an SMPE$\hat{c}=(c_0,c_0,\ldots )$ with $c_0\in F_L.$

Let $\mathcal{C}(S)$ be the space of all real-valued continuous functions on S. We assume that $\mathcal{C}(S)$ is endowed with the well-known topology of uniform convergence on compact sets. Recall that a sequence $(f_n)_{n\in \mathbb {N}}$ of functions in $\mathcal{C}(S)$ converges to some $f\in \mathcal{C}(S)$ if and only if for any compact interval $K\subset S$ we have that $\lim _{n\rightarrow \infty } \sup _{s\in K} |f_n(s)-f(s)|=0.$

Lemma 9

The set $F_L$ is convex and sequentially compact in $\mathcal{C}(S).$

Proof

Clearly, the set $F_L$ is convex. In order to show the sequential compactness we proceed similarly as in Lemma 1. However, in this proof, if $f\in F_L$ and $m\in \mathbb {N},$ then $f^m$ is the restriction of f to the interval [0, m]. Let $(f_n )_{n\in \mathbb {N}}$ be any sequence of functions in $F_L$ and let us apply the standard “diagonal method”. By the Arzelà–Ascoli theorem (see Billingsley 1968), there exists a subsequence $(n_1(k))$ of (n) such that $(f^1_{n_{1}(k)})_{k\in \mathbb {N}}$ converges uniformly (as $k\rightarrow \infty$) to some $f^1_o\in F_L$ on [0, 1]. Next, we chose a subsequence $(n_2(k))$ of $(n_1(k))$ such that $(f^2_{n_{2}(k)})_{k\in \mathbb {N}}$ converges uniformly to some $f^2_o\in F_L$ on [0, 2] and $f^2_o(s)=f^1_o(s)$ for each $s\in [0,1].$ Continuing this procedure we get a family of functions $\{f^r\}_{r\in \mathbb {N}}$ in $F_L$ such that $f^r_o(s)=f^{r-1}_o(s)$ for each $s\in [0,r-1]$ and $r\ge 2.$ The conclusion now follows from the same arguments used in the proof of Lemma 1. We define $f_o(s):= f^m_o(s)$ if $s\in [0,m],$$m\in \mathbb {N}.$ It is easy to see that the diagonal sequence converges to $f_o$ uniformly on every compact subset of S. $\square$

By Lemma 9 and Tychonoff’s theorem, we conclude the following fact.

Corollary 2

The space $F_L^\infty$ endowed with the product topology is sequentially compact.

Lemma 10

Assume that (A1), (A3) and (U) hold. If $c^{t+1,m} \rightarrow c^{t+1}$ in $F_L^\infty$, then

$$\begin{aligned} \lim _{m\rightarrow \infty } J_{t+1}(c^{t+1,m})(s)= J_{t+1}(c^{t+1})(s). \end{aligned}$$

(21)

Proof

Let $N\in \mathbb {N}$ and recall that

$$\begin{aligned}J^N_{t+1}(c^{t+1})(s)= u_{t+1}(c_{t+1}(s))+\sum _{k=1}^N \beta ^k Q_{c_{t+1}}\cdots Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s). \end{aligned}$$

Similarly, we define $J^N_{t+1}(c^{t+1,m})(s).$ We show that

$$\begin{aligned} \lim _{m\rightarrow \infty } J^N_{t+1}(c^{t+1,m})(s)= J^N_{t+1}(c^{t+1})(s) \end{aligned}$$

(22)

for all $N\in \mathbb {N}$ and $s\in S.$ Let us fix $k\in \{ 1,\ldots , N\}.$ We note that for every $s\in S$

$$\begin{aligned}\tilde{u}_{c^m_{t+1+k}}(s)\le u_{t+1+k}(s)\quad \text{ and } \quad u\left( c^m_{t+1+k}(s)\right) \rightarrow u\left( c_{t+1+k}(s)\right) \end{aligned}$$

by the continuity of $u_{t+k+1}.$ Hence, by the dominated convergence theorem, (A1) and (A3), it follows that

$$\begin{aligned} Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s)= & {} \sum _{i=0}^l\int _S u_{t+1+k}\left( c^m_{t+1+k}(s')\right) \nu _i(ds')h_i\left( s-c_{t+k}^m(s)\right) \nonumber \\&\quad \rightarrow \sum _{i=0}^l\int _S u_{t+1+k}\left( c_{t+1+k}(s')\right) \nu _i(ds')h_i(s-c_{t+1}(s))\nonumber \\= & {} Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s)\quad \text{ as } m\rightarrow \infty . \end{aligned}$$

(23)

Observe now that for all $s\in S,$ by (A3) and (A1) we have

$$\begin{aligned} 0\le Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s)<M \end{aligned}$$

and consider

$$\begin{aligned} Q_{c^m_{t+k-1}} Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s) = \sum _{i=0}^l\int _S Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s')\nu _i(ds') h_i(s-c_{t+k-1}^m(s)). \end{aligned}$$

Clearly, by (23), (A1) and the dominated convergence theorem, it follows

$$\begin{aligned} Q_{c^m_{t+k-1}} Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s)\rightarrow Q_{c_{t+k-1}} Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s)\quad \text{ as } m\rightarrow \infty \end{aligned}$$

for any $s\in S.$ Continuing this procedure we infer that

$$\begin{aligned} Q_{c^m_{t+1}}\cdots Q_{c^m_{t+k}}\tilde{u}_{c^m_{t+1+k}}(s) \rightarrow Q_{c_{t+1}}\cdots Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s) \quad \text{ as } m\rightarrow \infty \end{aligned}$$

(24)

for $s\in S.$ By assumption (U) for any $s\in S,$$u_{t+1}(c^m_{t+1}(s)) \rightarrow u_{t+1}(c_{t+1}(s)).$ This fact and (24) for $k=1,\ldots , N$ imply that (22) holds for every $N\in \mathbb {N}.$ Further, we have

$$\begin{aligned} |J_{t+1}(c^{t+1,m})(s) -J_{t+1}(c^{t+1})(s)| \le |J^N_{t+1}(c^{t+1,m})(s) -J^N_{t+1}(c^{t+1})(s)| +\frac{2\beta ^{N+1}M}{1-\beta }. \end{aligned}$$

(25)

Using (22) and (25), one can easily show that (21) holds. $\square$

From Lemma 10, we conclude the following result.

Lemma 11

If (A1), (A3) and (U) hold and $c^{t+1,m} \rightarrow c^{t+1}$ in $F_L^\infty$, then, for every $s\in S,$ we have

$$\begin{aligned} \sup _{a\in A(s)} P_t(a,c^{t+1,m})(s) \rightarrow \sup _{a\in A(s)} P_t(a,c^{t+1})(s) \quad \text{ as } m\rightarrow \infty . \end{aligned}$$

Proof

Observe that by (A1) we have

$$\begin{aligned}&\left| \sup _{a\in A(s)} P_t(a,c^{t+1,m})(s) - \sup _{a\in A(s)} P_t(a,c^{t+1})(s) \right| \nonumber \\&\quad \le \delta \beta \sum _{i=0}^l \left| \int _S J_{t+1}(c^{t+1,m})(s)\nu _i(ds)- \int _S J_{t+1}(c^{t+1})(s)\nu _i(ds)\right| . \end{aligned}$$

(26)

Since

$$\begin{aligned} J_{t+1} (c^{t+1,m})(s)\le u_{t+1}(s)+ \frac{M\beta }{1-\beta },\quad s\in S, \end{aligned}$$

the assertion follows from (26) and the dominated convergence theorem. $\square$

Lemma 12

Assume that (A1)–(A4) and (U) hold. Then for any $c^{t+1}$ in $F_L^\infty$ the function $s\rightarrow J_{t+1}(c^{t+1})(s)$ is non-decreasing.

Proof

First, we show that $s\rightarrow J^N_{t+1}(c^{t+1})(s)$ is non-decreasing. Clearly, $s\rightarrow u_{t+1+k}(c_{t+1+k}(s))$ is non-decreasing by (U). Moreover,

$$\begin{aligned} Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s)= & {} \sum _{i=1}^l h_i(s-c_{t+1}(s))\int _S u_{t+1+k}(c_{t+1+k}(s'))\nu _i(ds')\\&\quad + \left( 1-\sum _{i=1}^lh_i(s-c_{t+1}(s))\right) \int _S u_{t+1+k}(c_{t+1+k}(s'))\nu _0(ds')\\= & {} \int _S u_{t+1+k}(c_{t+1+k}(s'))\nu _0(ds')\\&\quad +\sum _{i=1}^lh_i(s-c_{t+1}(s)) \left( \int _S u_{t+1+k}(c_{t+1+k}(s'))\nu _i(ds')\right. \\&\quad \left. -\int _{S} u_{t+1+k}(c_{t+1+k}(s'))\nu _0(ds')\right) \end{aligned}$$

Hence, by (20), (A2) and the fact that $s\rightarrow s-c_{t+1}(s)$ is non-decreasing, it follows that $s\rightarrow Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s)$ is non-decreasing either. Continuing the procedure, we finally claim that

$$\begin{aligned} s\rightarrow Q_{c_{t+1}}\cdots Q_{c_{t+k}}\tilde{u}_{c_{t+1+k}}(s) \end{aligned}$$

is non-decreasing for any $k\in \{1,\ldots , N\}.$ Consequently, $s\rightarrow J^N_{t+1}(c^{t+1})(s)$ is non-decreasing. Since $\lim _{N\rightarrow \infty } J^N_{t+1}(c^{t+1})(s) = J_{t+1}(c^{t+1})(s),$$s\in S,$ the result follows. $\square$

The next result states that the best response set $BR_t(c^{t+s})(s)$ of self $t\in T$ is a singleton.

Lemma 13

Let (A1)–(A4) and (U) be satisfied. Assume that $c^{t+1}$ in $F_L^\infty .$ Then, $BR_t(c^{t+1})=\{br_t(c^{t+1})\}$ and $br_t(c^{t+1})\in F_L.$

Proof

Let $y:=s-a$ and observe first that the function $y\rightarrow \int _S J_{t+1}(c^{t+1})(s')q(ds'|y)$ is concave. Indeed, by (A1) we obtain

$$\begin{aligned}&\int _S J_{t+1}(c^{t+1})(s')q(ds'|y)= \int _S J_{t+1}(c^{t+1})(s')\nu _0(ds')\nonumber \\&\quad +\sum _{i=1}^lh_i(y) \left( \int _S J_{t+1}(c^{t+1})(s')\nu _i(ds')-\int _S J_{t+1}(c^{t+1})(s')\nu _0(ds')\right) . \end{aligned}$$

(27)

Due to Lemma 12 and (20) we know that for every $i=1,\ldots ,l$

$$\begin{aligned} \int _S J_{t+1}(c^{t+1})(s')\nu _i(ds')-\int _S J_{t+1}(c^{t+1})(s')\nu _0(ds')\ge 0. \end{aligned}$$

(28)

Thus, the above inequalities, assumption (A2) and (27) lead to the conclusion. Assume that at least one inequality (28) is strict. Then we can conclude that $y\rightarrow P_t(s-y,c^{t+1})(s)$ and $a\rightarrow P_t(a,c^{t+1})(s)$ are strictly concave on A(s) for every $s\in S_+.$ Therefore, the sets $BR_t(c^{t+1})(s)$ and $\overline{BR}_t(c^{t+1})(s)$ are singletons for every $s\in S.$ Moreover, we have $BR_t(c^{t+1})(s)=\{br_t(c^{t+1})(s)\}$ for all $s\in S$ and $\overline{BR}_t(c^{t+1})(s)=\{i_t(c^{t+1})(s)\}$ where $i_t(c^{t+1})(s)=s-br_t(c^{t+1})(s),$$s\in S.$ From the strict concavity of $u_t$ and Lemma 3.2 in Balbus et al. (2015a), it follows that the function $s\rightarrow i_t(c^{t+1})(s)$ is non-decreasing. From the strict concavity of $y\rightarrow \int _S J_{t+1}(c^{t+1})(s')q(ds'|y)$ and Lemma 3.2 in Balbus et al. (2015a), we conclude that the function $s\rightarrow br_t(c^{t+1})(s)$ is also non-decreasing. We obviously know that the functions $i_t(c^{t+1})$ and $br_t(c^{t+1})$ satisfy $0\le i_t(c^{t+1})(s)\le s$ and $0\le br_t(c^{t+1})(s)\le s,$$s\in S.$ Thus, $br_t(c^{t+1})\in F_L.$ If for every $i=1,\ldots ,l$, we have equality in (28), then $BR_t(c^{t+1})(s)=\{br_t(c^{t+1})(s)\}=\{s\}$ for all $s\in S.$$\square$

Lemma 14

Assume that (A1)–(A3) and (U) hold. Then, the mapping $br_t: F_L^\infty \rightarrow F_L$ is continuous.

Proof

Suppose that $c^{t+1,m} \rightarrow c^{t+1}$ as $m\rightarrow \infty .$ Put $\psi ^m:= br_t(c^{t+1,m}).$ Let $\psi$ be any accumulation point of the sequence $(\psi ^m)$ in $F_L$ being a compact space. We have to show that $\psi =br_t(c^{t+1}).$ However, this fact easily follows from the dominated convergence theorem, Lemmas 10, 11 and 13. $\square$

Similarly as in (18), we define the sets

$$\begin{aligned} \hat{G}_t:=\left\{ (x_1(\sigma ),\ldots ,x_t(\sigma ),\sigma ): \sigma =c^{t+1}\in F_L^\infty \right\} . \end{aligned}$$

Clearly, $\hat{G}_t\subset F_L^\infty .$

Proof of Theorem 3

We note that $\hat{G}_{t+1}\subset \hat{G}_{t}.$ Since all best response mappings are continuous by Lemma 14 and the space $F_L^\infty$ is compact, every set $\hat{G}_t$ is non-empty and compact. Therefore, the set $\hat{G}:=\cap _{t\in T} \hat{G}_t$ is non-empty and compact. Choose any $\hat{c}=(\hat{c}_1,\hat{c}_2,\ldots )\in \hat{G}.$ Then $\hat{c}\in \hat{G}_t$ for every $t\in T.$ This implies that $\hat{c}_t=br_t(\hat{c}^{t+1})$ for every $t\in T.$ Hence, $\hat{c}$ is an MPE. $\square$

Proof of Theorem 4

In the stationary case, we again restrict attention to constant sequences $(c,c,\ldots )\in F_L^\infty .$ Clearly, such a sequence can be identified with $c\in F.$ Function (3) can be regarded as a function P(a, c)(s),  where $a\in A(s),$$c\in F_L.$ By Lemma 14 the best response mapping $br(c):= br_t(c^{t+1})$ is continuous on the convex compact set $F_L$ in the space $\mathcal{C}(S)$ (Lemma 9). From the Schauder–Tychonoff fixed point theorem (see Aliprantis and Border 2006), it follows that there exists $c^*\in F_L$ such that $c^* =br(c^*)$. Hence, the constant sequence $(c^*,c^*,\ldots )$ is an SMPE. $\square$

Remark 1

In assumptions (A1)–(A4) the functions $h_i$ and the measures $\nu _i,$ ($i=0,\ldots ,l$) may depend on $t\in T.$ Then, Lemmas 10–14 and Theorems 3, 4 remain valid. However, for the sake of clarity of notation we skip this dependence.

Remark 2

The special transition structure in this section and condition (U) imply that both functions $y\rightarrow P_t(s-y,c^{t+1})(s)$ and $a\rightarrow P_t(a,c^{t+1})(s)$ are strictly concave on A(s) for every $s\in S_+$ and $c^{t+1}\in F_L^\infty .$ This plays a crucial role in getting the best reply $br_t(c^{t+1})\in F_L.$ If the transition probability has some atoms in $S_+$ and does not have the special additive form, then there is a problem with the continuity of $a\rightarrow P_t(a,c^{t+1})(s)$ for some $c^{t+1} \in F^\infty$ and the best response $br_t(c^{t+1})$ to $c^{t+1}$ may not exist.

5 Comments

In this section we give some remarks on the relation of our results with the literature.

Remark 3

The decision model studied in this paper can be viewed as a game between generations. Self t can be considered as generation t having the total utility depending on all its descendants. Further comments on this issue can be found in Balbus et al. (2015a). In the intergenerational game setting, it is desirable to assume that the period utility functions $u_t$ and transition functions $q_t$ depend on generation $t\in T.$ This natural requirement motivates us to consider the non-stationary models. Theorems 1 and 3 are new results in the area of decision processes with quasi-hyperbolic discounting (or intergenerational games). It is interesting to note that their proofs are not based on any fixed point argument. The idea is to consider the intersection G of the sets $G_t$ and is partly inspired by the fundamental work of Mertens and Parthasarathy (2003) on subgame perfect equilibria in n-person discounted stochastic games with simultaneous moves of the players. Balbus and Woźny (2016) also deal with a similar stationary model but with a compact state space and a relatively narrow class of transition functions.

Remark 4

Theorems 2 and 4 establish the existence of SMPE in stationary models and have some predecessors in the literature. A version of Theorem 2 with compact state space follows from Theorem 3.1 in Balbus et al. (2015a). In this case the function u is bounded. Here, on the other hand, we deal with unbounded state space and unbounded period utility functions. Such a setting covers more applications in economics, e.g., models with power or logarithmic utility functions. A related result to Theorem 1 is stated in Harris and Laibson (2001). Nonetheless, some of their assumptions are much stronger. They assume that $S=[0,+\infty )$ but the transition probability is induced by a difference equation with additive non-atomic noise with values in $[y_1,y_2]\subset S_+.$ The period utility function u may be unbounded from below. However, it has a bounded relative risk-aversion coefficient. The class of strategies F (applied in Sect. 2) was used by Bernheim and Ray (1983) to study equilibria in altruistic growth models and by Majumdar and Sundaram (1991) to study Nash equilibria in symmetric stochastic games of resource extraction. Our paper owes much to their contributions.

Remark 5

The additivity assumptions similar to (A1)–(A3) were used in the study of models of intergenerational stochastic games or decision processes with quasi-hyperbolic discounting. In Balbus and Nowak (2008) a class of stochastic games between generations is considered where each generation consists of finitely many players. The proof of the main results is based on different ideas than in this work. Jaśkiewicz and Nowak (2014) also study models with additive non-atomic transitions but with risk-sensitive preferences. The most relevant work on models satisfying assumptions of type (A1)–(A3) is the paper of Balbus et al. (2014) where the transition probability depends on some unknown parameter chosen by a malevolent nature. Therefore, the notion of a robust Markov perfect equilibrium is introduced. Balbus et al. (2014) assume that the state space is a compact interval and the utility function u is bounded from above. The additivity condition on the transition probability function is also made in Balbus et al. (2015c), but with an additional restrictive assumption that $\nu _0$ is the Dirac measure concentrated at zero. Thus, $s=0$ is an absorbing state. Theorem 4 is not a corollary to their results. It should be noted, however, that they study an n-dimensional state space model.

Remark 6

The existence of an MPE in models with quasi-hyperbolic discounting and deterministic transitions is open and seems to be difficult. Examples 5.1 and 5.2 in Balbus et al. (2015a) suggest that Lemma 6 may fail to hold in the class $F^\infty$ of strategies. On the other hand, working with the class $F_L^\infty$ as a strategy set leads to very restrictive assumptions.

Remark 7

Our proofs in both models (in atomic and non-atomic transitions) heavily rely on the assumption that the set S is one-dimensional. For example, the definition of $br_t(c^{t+1})$ given in (17) is based on the fact that $BR_t(c^{t+1})(s)\subset {\mathbb {R}}.$ Furthermore, the monotonicity of best reply functions with respect to state variable is very difficult to obtain in the model with multidimensional state space. We conjecture that a solution to dynamic decision models with quasi-hyperbolic preferences and more than one resource requires some new ideas and methods.

Appendix

Let $S_m=[0,m]$ for $m\in \mathbb {N}$ and $\Pr (S_m)$ be the space of probability measures on $S_m.$ It is well-known that there is a metric, say $d_m,$ on $\Pr (S_m)$ that induces the weak topology on $\Pr (S_m),$ see Billingsley (1968) or Parthasarathy (1967). For any $f\in I$ and $m\in \mathbb {N}$, we define $f^m$ as in the proof of Lemma l1. Then $\frac{f^m}{m}$ is a distribution function of a countably additive measure, say $p(f^m)\in \Pr (S_m).$ It is obvious that $f_n$ converges weakly to f in I (as $n\rightarrow \infty$) if and only if, for each $m\in \mathbb {N},$$f_n^m$ converges weakly to $f^m.$ Furthermore, the weak convergence of $f_n^m$ to $f^m$ (as $n\rightarrow \infty$) is equivalent to the weak convergence of $p(f_n^m)$ to $p(f^m)$ in $\Pr (S_m),$ see Billingsley (1968).

For any functions $f_1, f_2 \in I$ and $m\in \mathbb {N},$ we can define $\rho ^m(f_1,f_2):= d_m(p(f_1^m),p(f_2^m)).$ Note that $\rho ^m$ is a semimetric on the space I. Now we can define the metric $\rho$ on I as

$$\begin{aligned} \rho (f_1,f_2):= \sum _{m=1}^\infty \frac{1}{2^m}\frac{\rho ^m(f_1,f_2)}{ 1+ \rho ^m(f_1,f_2)}. \end{aligned}$$

Let $f_n$ converge weakly to f in I as $n\rightarrow \infty .$ This is equivalent to saying that $\rho (f_n,f)\rightarrow 0$ as $n\rightarrow \infty .$

Let $\hat{f}_1, \hat{f}_2 \in F.$ Define $f_i(s)=s-\hat{f}_i(s),$$i=1,2,$$s\in S.$ Then $f_i\in I$ and, we can define the metric on F as $\hat{\rho }(\hat{f}_1,\hat{f}_2):= \rho (f_1,f_2).$ Clearly, convergence with respect to the metric $\hat{\rho }$ on F is equivalent to the convergence in the weak topology on F.