# Markov perfect equilibria in a dynamic decision model with quasi-hyperbolic discounting

- 302 Downloads

## Abstract

We study a discrete-time non-stationary decision model in which the preferences of the decision maker change over time and are described by quasi-hyperbolic discounting. A time-consistent optimal solution in this model corresponds with a Markov perfect equilibrium in a stochastic game with uncountable state space played by countably many short-lived players. We show that Markov perfect equilibria may be constructed using a generalized policy iteration algorithm. This method is in part inspired by the fundamental works of Mertens and Parthasarathy (in: Raghavan, Ferguson, Parthasarathy, Vrieze (eds) Stochastic games and related topics, Kluwer Academic Publishers, Dordrecht, 1991; in: Neyman, Sorin (eds) Stochastic games and applications, Academic Publishers, Dordrecht, 2003) devoted to subgame perfect equilibria in standard *n*-person discounted stochastic games. If the one-period utilities and transition probabilities are independent of time, we obtain on new existence results on stationary Markov perfect equilibria in the models with unbounded from above utilities.

## Keywords

Stochastic game Uncountable state space Quasi-hyperbolic discounting Non-atomic transition probability Markov perfect equilibrium## Mathematics Subject Classification

91A13 91A25 91B51 91B62## 1 Introduction

The issue of dynamic inconsistency in sequential decision models with changing preferences in time was pointed out in the seminal paper of Strotz (1956). The time preference studies found that discount rates are much grater in the short run than in the long run. To model this phenomena researchers have adopted discount functions from the class of generalized hyperbolas, see Ainslie (1992) or Harris and Laibson (2001) and references therein. The discrete-time analog of quasi-hyperbolic discounting involves the functions: \(1, \delta \beta ,\delta \beta ^2,\ldots ,\) where \(\beta \in (0,1)\) is a long run discount factor and \(\delta >0\) is a short run discount factor. Such discounting was first used by Phelps and Pollak (1968). Specifically, they noticed that finding a time-consistent solution may be obtained by looking for a Nash equilibrium in certain games played by countably many short-lived players assuming that each player can act only once. Recently, Montiel Olea and Strzalecki (2014) provided an axiomatic characterization of quasi-hyperbolic preferences.

The existence of time-consistent solutions in models with changing tastes is a difficult problem. The akin problem is also met in altruistic economic growth models, known as intergenerational games. Initial positive results expressed in terms of intergenerational games (with a relatively simple utility function for each generation) were given by Bernheim and Ray (1983) and Leininger (1986). Peleg and Yaari (1973), on the other hand, provided some counterexamples for models with slightly less restrictive assumptions. The existence of a stationary Markov perfect equilibrium in stochastic game with finite state and action spaces with quasi-hyperbolic discounting was stated first in Alj and Haurie (1983). Applications of quasi-hyperbolic discounting in ecological problems can be found in Haurie (2005) and Karp (2005) and the references cited therein. For additional examples (especially in macroeconomics) and a detailed discussion on the literature the reader is referred to Balbus et al. (2016) and Jaśkiewicz and Nowak (2018).

In this paper, we study a discrete-time consumption/savings model with changing in time preferences. To construct a time-consistent solution in this model, the decision maker is represented by a sequence of temporal players called “selves”. The state space is \(S=[0,+\,\infty )\) and represents the set of stocks of a renewable resource. Current self (self *t*) decides how to split the available resource stock into consumption and investment for future selves. The outcome of the investment (saving) is determined by some transition probability function. We assume that the stage utility functions and transition probability functions may depend on time and self *t* measures his satisfaction using so-called short-run and long-run discount factors according to the formula used in quasi-hyperbolic discounting. The total utility of self *t* depends on his own consumption and consumptions of all future selves. The model is actually a non-cooperative stochastic game with uncountable state space and denumerable set of players. This model was studied successfully by Harris and Laibson (2001) within a stationary framework. They proved the existence of a stationary Markov perfect equilibrium and derived a hyperbolic Euler relation that is useful in characterizing equilibria. The transition probability function in their model is non-atomic and has a special form with additive noise. More general non-atomic transition functions were examined in Balbus et al. (2015a), where the state space was assumed to be a compact interval. Related results were obtained in Balbus and Nowak (2008), Balbus et al. (2014) and Balbus et al. (2015c), but with very specific transition functions being a convex combination of probability measures on the state space. The proof of an existence of a stationary Markov perfect equilibrium in Harris and Laibson (2001) and Balbus et al. (2015a) was based on a fixed point argument in a space of functions with locally bounded variation. The existence of an equilibrium in the class of Lipschitz continuous functions, on the other hand, was shown by Balbus et al. (2015c), who also discussed computational issues.

The existence of subgame perfect equilibria in a large class of standard discounted *n*-person stochastic games proved by Mertens and Parthasarathy (2003). Their proof is based on some set-valued recursive equations whose solutions are the sets of Nash equilibrium vector payoffs in one-shot games involving the stage payoffs and the transition probability. Mertens and Parthasarathy (2003) also proposed an algorithm extending the well-known “value iteration” in discounted dynamic programming (see, e.g., Blackwell 1965). A simplified version of the algorithm was given in Mertens and Parthasarathy (1991). Certain modification of their method was applied by Balbus and Woźny (2016) to characterize the sets of Markov perfect equilibria in supermodular stochastic games and the consumption/savings problem with quasi-hyperbolic discounting and with very specific transition functions. The existence of an equilibrium is proved in two steps. First, they show that a system of recursive equations in the value function space has a solution. Second, relatively strong concavity assumptions imposed on the transition operator allow to obtain a Lipschitz continuous equilibrium. Finally, it is worth mentioning that a large class of dynamic games for which the Markov perfect equilibria exist and have a natural interpretation was indicated by Maskin and Tirole (2001).

This paper proposes a new method for proving the existence of Markov perfect equilibria in dynamic consumption/savings model with quasi-hyperbolic discounting. Our approach is also inspired by the method of Mertens and Parthasarathy (2003), but it more resembles “policy iteration” algorithm in dynamic programming. Our assumptions on the primitive data are much weaker than in the previous works of Harris and Laibson (2001) and Balbus et al. (2015a). The state space and the utility functions may be unbounded. Our first model concerns the transition probability function that is weakly continuous in investments and has no atoms in the positive part of the state space \((0,\infty ).\) The weighted-norm approach, proposed by Wessels (1977) and applied to this model, allows us for consideration of a large class of unbounded (e.g., power) utility functions. The second model, studied in this paper, deals with a special additive type of the transition function that may involve positive atoms. It is worthy to point out that a Markov perfect equilibrium is obtained in this paper from the non-empty intersection of some nested family of compact subsets of the space of strategy profiles of all selves. Therefore, fixed point theorems are not applied in our approach. However, the equilibrium is non-stationary, even if the period utility functions and transition probabilities are independent of time.

This paper is organized as follows. In Sect. 2, we describe a general decision model and basic definitions. Section 3 contains our main results on Markov perfect equilibria in a model with non-atomic transitions, whereas the transitions involving atoms are studied in Sect. 4. Additional comments on our results and related literature are given in Sect. 5.

## 2 The model

Let \({\mathbb {R}}\) be the set of all real numbers and \(\mathbb {N}\) be the set of all positive integers. Define \(S:=[0,+\infty ),\) \(S_+:=(0,+\infty )\) and, for each \(s\in S,\) \( A(s):=[0,s].\) The set *S* is referred to as the *state space*. It represents the *set of “levels” for the renewable resource* and *A*(*s*) is the set of *available actions* (possible consumption levels) in state \(s\in S.\) In a *dynamic choice model* with *quasi-hyperbolic preferences* and the state space *S* we envision an individual decision maker as a sequence of autonomous temporal *selves*. These selves are indexed by period numbers \(t\in T:=\mathbb {N}\) in which they make their choices. More precisely, for a given state \(s_t\in S\) at the beginning of *t*-th period, self *t* chooses a consumption level \(a_t\in A(s_t)\) and the remaining part \(y_t:=s_t-a_t\) is invested for future *selves.* Self *t*’s satisfaction is measured by a *period utility function* \(u_\tau : S\rightarrow S\) for all \(\tau \ge t.\)

Let \(q_t\) be a transition probability from *S* to *S*. Then, the state \(s_{t+1}\) is generated by \(q_t(\cdot |y_t)\) depending on the investment \(y_t\in A (s_t).\)

Let \(\varPhi \) be the set of all Borel measurable functions \(\phi : S\rightarrow S\) such that \(\phi (s)\in A(s)\) for each \(s\in S.\) A *Markov strategy* for self *t* is a function \(c_t\in \varPhi \). We put \(i_t(s)=s-c_t(s),\) \(s\in S.\) This is an investment strategy (or saving) of self *t* for following selves. For any sequence \((c_n)\in \varPhi ^\infty :=\varPhi \times \varPhi \times \cdots \) of strategies of all selves and any \(t\in T,\) we define \(c^t:=(c_t,c_{t+1},\ldots ).\) For any state \(s_t\) and any \(c^t\in \varPhi ^\infty ,\) the transition probabilities \(q_\tau (\cdot |i_\tau (s))\) induced by \(q_\tau \) with \(\tau \ge t\) generate due to the Ionescu-Tulcea theorem (see Proposition V.1.1 in Neveu (1965)) a unique probability measure \(P^{c^t}_{s_t}\) on \(S^\infty \) endowed with the product \(\sigma \)-algebra. Let \(E^{c^t}_{s_t}\) denote the expectation operator corresponding to the measure \(P^{c^t}_{s_t}.\) Assume that for each \(\tau \in T,\) the function \(u_\tau :S\rightarrow S\) is continuous. Note that it is non-negative.

*expected utility*of self

*t*is

*long-run discount factor*and \(\delta \in (0,1]\) is a

*short-run discount factor*. The idea of using utility functions of the form (1) goes back to Phelps and Pollak (1968). A detailed discussion with some applications of time preferences represented by utility functions of the type considered here can be found in Harris and Laibson (2001) and Montiel Olea and Strzalecki (2014). Clearly, the expression defined in (1) is non-negative. In the sequel, we give conditions under which it is finite.

### Definition 1

*Markov Perfect Equilibrium*(\({ MPE}\)) is a sequence \(\hat{c} = (\hat{c}_t)_{t\in T}\in \varPhi ^\infty \) such that for every \(s\in S\) and \(t\in T,\) we have

### Definition 2

A *Stationary Markov Perfect Equilibrium* (\({ SMPE}\)) is an \({ MPE}\) \(\hat{c} = (\hat{c}_t)_{t\in T} \in \varPhi ^\infty \) such that \(\hat{c}_t = c_0\) for some \(c_0\in \varPhi \) and for all \(t\in T.\)

In a stationary \({ MPE}\) every self uses the same consumption strategy.

One can think that every self is a short-lived player in a non-cooperative game and acts only once. The payoff function of self \(t\in T\) is given by (1). Then an \({ MPE}\) \(\hat{c}=(\hat{c}_t)_{t\in T}\in \varPhi ^\infty \) is a Nash equilibrium in this game.

- (W1)
There exists a continuous increasing function \(w:S\rightarrow [1,\infty )\) such that \(0\le u_t (s)\le w( s)\) for all \(s\in S\) and \(t\in T.\)

- (W2)There exists a constant \(\alpha >0\) such that \( \alpha \beta <1 \) and$$\begin{aligned} \int _S w(s)q_t(ds|y)\le \alpha w(y)\quad \text{ for } \text{ all } \quad y\in S, \ t\in T. \end{aligned}$$
- (W3)
The function \(y\rightarrow \int _Sw(s)q_t(ds|y)\) is continuous on

*S*for each \(t\in T.\)

Assumptions of the above type were used to study discounted Markov decision processes with unbounded stage utility function, see Wessels (1977), Hernández-Lerma and Lasserre (1996) or Jaśkiewicz and Nowak (2011).

## 3 Markov perfect equilibria in models with non-atomic transitions

*S*. We recall that a sequence \((\mu _n)_{n\in \mathbb {N}}\) of probability measures on

*S*

*converges weakly*to some \(\mu _0\in \Pr (S)\) (\(\mu _n \Rightarrow \mu _0\) in short) if, for any bounded continuous function \(f:S\rightarrow {\mathbb {R}},\) it holds that

- (U)
For each \(t\in T,\) the function \(u_t :S\rightarrow S\) is increasing, strictly concave and continuous at \(s=0.\)

- (Q)
The transition probability \(q_t\) is weakly continuous on

*S*, that is, for each \(y_0\in S\) and \(y_m \rightarrow y_0,\) we have \(q_t(\cdot |y_m) \Rightarrow q_t(\cdot |y_0)\) as \(m\rightarrow \infty .\) Moreover, for each \(y\in S_+,\) the probability measure \(q_t(\cdot |y)\) is non-atomic and \(q_t(\cdot |0) \) has no atoms in \(S_+.\)

*F*, we denote the set of all continuous from the left mappings \(c:S\rightarrow S\) such that the function \( i(s):=s-c(s)\) is non-decreasing and \(0\le c(s)\le s\) for all \(s\in S.\) Note that

*i*is lower semicontinuous. Thus, \(c\in F \) is upper semicontinuous.

We can now state our first two main results.

### Theorem 1

Assume that (W1)–(W3), (U) and (Q) are satisfied. Then there exists an \({ MPE}\) \(\hat{c}=(\hat{c}_t)\) that belongs to the space \(F^\infty :=F\times F\times \cdots .\)

### Theorem 2

Assume that (W1)–(W3), (U) and (Q) hold and the model is stationary, i.e., \(q_t=q\) and \(u_t=u\) are independent of \(t\in T.\) Then there exists an *SMPE* \(\hat{c}=(c_0,c_0,\ldots )\) with \(c_0\in F.\)

In the proof of Theorem 1 we define by backward induction a nested family of subsets of \(F^\infty \) that has a non-empty intersection. A Markov perfect equilibrium is a sequence that belongs to this intersection. This method resembles a “set-valued dynamic programming” approach and is based on two important factors: the continuity of expected utilities with respect to some natural topology on the space \(F^\infty \) and the continuity of the best response self t’s mapping defined on the space of sequences of future selves strategies. Observe that the assertion of Theorem 2 is stronger compared with Theorem 1, but the assumptions are also stronger (stationarity of the model). To obtain an *SMPE* in Theorem 2 we shall need to apply a fixed point theorem, which was not needed in the proof of Theorem 1.

Let *X* be the vector space of all continuous from the left functions \(f:S\rightarrow {\mathbb {R}}\) such that \(f(0)=0.\) It is also assumed that each \(f\in X\) has bounded variation on every interval [0, *m*], \(m\in \mathbb {N}.\) We assume that *X* is endowed with the *topology of weak convergence*. Recall that a sequence \((f_n)_{n\in \mathbb {N}}\) converges weakly to \(f \in X\) if and only if \(f_n(s)\rightarrow f(s)\) as \(n\rightarrow \infty \) at any continuity point \(s\in S\) of *f*. Here we point out that \(s=0\) is considered as a continuity point of \(f\in X\) if \(\lim _{s\rightarrow 0^+}f(s)=f(0).\) The weak convergence of \((f_n)_{n\in \mathbb {N}}\) to *f* is denoted by \(f_n{\mathop {\rightarrow }\limits ^{\omega }} f\). Note that *F* is a metrizable topological subspace of *X* (see Appendix).

*I*is non-decreasing and lower semicontinuous. Observe that \(I\subset X.\) Moreover, \(s=0\) is the continuity point of every function in

*F*or

*I*.

### Lemma 1

The sets *I* and *F* are convex and sequentially compact in *X*.

### Proof

It is obvious that *I* is convex. For any \(f\in I\) and \(m\in \mathbb { N}\), we define the function \(f^m\) as follows: \(f^m(s)=f(s)\) for all \(s\in [0,m]\) and \(f^m(s) = m\) for all \(s> m.\) Then \(f^m\) can be viewed as a continuous from the left “distribution function” of some non-negative countably additive measure \(\rho _m\) such that \(\rho _m(S)=m.\) Consider an arbitrary sequence \((f_n )_{n\in \mathbb {N}}\) of functions in *I*. We now apply the standard “diagonal method”. By Helly’s theorem (see Billingsley 1968), there exists a subsequence \((n_1(k))\) of (*n*) such that \((f^1_{n_{1}(k)})_{k\in \mathbb {N}}\) converges weakly (as \(k\rightarrow \infty \)) to some \(f^1_o\in I.\) Next, there exists a subsequence \((n_2(k))\) of \((n_1(k))\) such that \((f^2_{n_{2}(k)})_{k\in \mathbb {N}}\) converges weakly to some \(f^2_o\in I\) and \(f^2_o(s)=f^1_o(s)\) for each \(s\in [0,1].\) Proceeding along this way, we infer that for any \(r\ge 2\), there exists a subsequence \((n_r(k))\) of \((n_{r-1}(k))\) such that \((f^r_{n_{r}(k)})_{k\in \mathbb {N}}\) converges weakly to some \(f^r_o\in I\) and \(f^r_o(s)=f^{r-1}_o(s)\) for each \(s\in [0,r-1].\) Define \(f_o(s):=f^m_o(s)\) if \(s\in [0,m],\) \(m\in \mathbb {N}.\) Then \(f_o\in I.\) Consider the “diagonal sequence” defined by \(d(k):= n_k(k)\), \(k\in \mathbb {N}.\) Then, \(f_{d(k)}{\mathop {\rightarrow }\limits ^{\omega }} f_o\) as \(k\rightarrow \infty .\) Thus, *I* is sequentially compact. Since *F* is obtained from *I* by a simple continuous transformation, the result also holds for *F*. \(\square \)

By Lemma 1 and Tychonoff’s theorem, we obtain the following auxiliary result.

### Corollary 1

The space \(F^\infty \) endowed with the product topology is sequentially compact.

*S*. For each \(s\in S,\) define

### Lemma 2

### Proof

### Lemma 3

### Proof

Let \(S_c\) denote the set of continuity points of \(c\in X.\) If \(c\in F,\) then \(0\in S_c\) and \(S{\setminus } S_c\) is a denumerable set. The proof of the following result is the same as that of Lemma 3.5 in Balbus et al. (2015a).

### Lemma 4

Let \(c^m {\mathop {\rightarrow }\limits ^{\omega }} c\) in *F*. If \(s^m\in S\) for every \(m\in \mathbb {N}\) and \(s^m\rightarrow s\in S_c\) as \(m\rightarrow \infty ,\) then \(\lim _{m\rightarrow \infty }c^m(s^m)=c(s)\) and \(\lim _{m\rightarrow \infty }i^m(s^m)=i(s).\)

Consider \(c^{\tau ,m}=(c^m_\tau ,c^m_{\tau +1},\ldots )\) and \(c^{\tau }=(c_\tau ,c_{\tau +1},\ldots )\) that belong to \(F^\infty \) endowed with the product topology. Then \(c^{\tau ,m}\rightarrow c^{\tau }\) (as \(m\rightarrow \infty \)) if and only if, for any \(n\ge \tau ,\) \(c_n^m {\mathop {\rightarrow }\limits ^{\omega }} c_n\) in *F*.

### Lemma 5

### Proof

*k*times we finally obtain that

*N*, we have

*w*is continuous, then there exists \(b>0\) such that \(2\le w(s^m_{t+1})+w(s_{t+1})\le b\) for all \(m\in \mathbb {N}.\) Thus, we have

Observe that assumption (U) implies that every function \(u_t\) is continuous on the space *S*. Now from Lemmas 3 and 5 with \(g_m=g=J_{t+1}(c^{t+1})\) (note that \(\lambda =\frac{\alpha \beta }{1-\alpha \beta }\) in Lemma 2, compare with (8)), we conclude the following auxiliary result.

### Lemma 6

If (W1)–(W3), (U) and (Q) hold, then for any \(c^{t+1 } \in F^\infty ,\) the function \(a\rightarrow P_t(a,c^{t+1})\) is continuous on *A*(*s*).

*s*, given that the following selves are going to use \(c^{t+1}\in F^\infty .\) Under our assumptions, this set is non-empty and compact by Lemma 6. For any \( s\in S\) and \(t\in T\) let us define

### Lemma 7

- (a)
Let \(\overline{BR}_t(c^{t+1})(s):= \{s-a: a\in BR_t(c^{t+1})(s)\}.\) The correspondence \(s\rightarrow \overline{BR}_t(c^{t+1})(s)\) is compact-valued and strongly ascending, i.e., if \(s_1<s_2\) and \(b_i\in \overline{BR}_t(c^{t+1})(s_i)\) (\(i=1,2\)), then \(b_1\le b_2.\)

- (b)
The function \(br_t(c^{t+1})\) belongs to

*F*, i.e., \(i_t(c^{t+1})(s):= s - br_t(c^{t+1})(s)\in A(s) \) for all \(s\in S\) and \(i_t(c^{t+1})\) is non-decreasing and continuous from the left. - (c)
If \(\phi :S\rightarrow S\) is such that \(\phi (s)\in BR_t(c^{t+1})(s)\) for all \(s\in S\) and \(s_0\) is a continuity point of \(\phi \), then \(BR_t(c^{t+1})(s_0)\) is a singleton.

- (d)
The only function \(\phi \in F\) such that \(\phi (s)\in BR_t(c^{t+1})(s)\) for all \(s\in S\) is \(br_t(c^{t+1}).\)

### Proof

For parts (a)–(c) consult Lemmas 3.2 and 3.3 in Balbus et al. (2015a). If \(\phi \in F,\) then the function *i* given by \(i(s)=s-\phi (s)\) is non-decreasing and continuous from the left. If \(s\in S_\phi ,\) then we have \(\phi (s)=br_t(c^{t+1})(s)\) by (c). Assume that \(s_o\in S{\setminus } S_\phi .\) Since *i* is continuous from the left, non-decreasing and the correspondence \(s\rightarrow \overline{BR}_t(c^{t+1})(s)\) is strongly ascending, then \(i(s_o)\le s_o-a\) for all \(a\in BR_t(c^{t+1})(s_o).\) Hence, \(\phi (s_o)=s_o-i(s_o) =br_t(c^{t+1})(s_o).\) \(\square \)

### Lemma 8

Assume that (W1)–(W3), (U) and (Q) hold. Then, the mapping \(br_t: F^\infty \rightarrow F\) defined in (17) is continuous.

### Proof

Let \(c^{t+1,m} \rightarrow c^{t+1}\) as \(m\rightarrow \infty .\) For notational convenience put \(\sigma ^m:=c^{t+1,m},\) \(\sigma := c^{t+1}\) and \(\psi ^m:= br_t(\sigma ^m).\) Let \(\psi \) be any accumulation point of the sequence \((\psi ^m) \) in *F* being a compact space. We have to show that \(\psi =br_t(\sigma ).\) Consider any \(s\in S_\psi .\) Using Lemmas 3 and 5, one can show that \(\psi (s)\in BR_t(\sigma )(s).\) Since \(s\in S_\psi ,\) by Lemma 7(c), the set \(BR_t(\sigma )(s)\) is a singleton and thus \(\psi (s)=br_t(\sigma )(s).\) If \(s_0\in S{\setminus } S_\psi ,\) then from the facts that \(S_\psi \) is dense in *S* and \(\psi \) is continuous from the left, it also follows that \(\psi (s_0)\in BR_t(\sigma )(s_0).\) Since \(\psi \in F,\) by Lemma 7(d), it follows that \(\psi (s_0)=br_t(\sigma )(s_0).\) \(\square \)

*t*. Assume that \(t=4.\) Then \(\sigma =c^5= (c_5,c_6,\ldots ),\) \(x_4(\sigma ) =br_4(\sigma ), \) \(x_3(\sigma ) =br_3(br_4(\sigma ),\sigma ). \) Further, we have \(x_2(\sigma )=br_2(br_3(br_4(\sigma ),\sigma ),br_4(\sigma ),\sigma ) \) and

*t*to \(\sigma =c^{t+1}=(c_{t+1},c_{t+2},\ldots )\) assumed to be chosen by following selves, \(x_{t-1}(\sigma )\) is the best response of self \(t-1\) to the sequence \((br_t(c^{t+1}),c^{t+1})\) and so on.

### Proof of Theorem 1

First note that for any \(t\in T,\) we have \(G_{t+1}\subset G_{t}.\) Since all best response mappings are continuous (Lemma 8) and the space \(F^\infty \) is compact, every set \(G_t\) is non-empty and compact. Therefore, the set \(G:=\cap _{t\in T} G_t\not = \emptyset \) and *G* is compact. Choose any \(\hat{c}=(\hat{c}_1,\hat{c}_2,\ldots )\in G.\) Then \(\hat{c}\in G_t\) for every \(t\in T.\) This implies immediately that, for every \(t\in T,\) we have \(\hat{c}_t=br_t(\hat{c}^{t+1}).\) Hence, \(\hat{c}\) is an *MPE*. \(\square \)

### Proof of Theorem 2

In the stationary case, we can restrict attention to constant sequences \((c,c,\ldots )\in F^\infty .\) Every such a sequence can be identified with \(c\in F.\) Function (3) can be regarded as a function *P*(*a*, *c*)(*s*), where \(a\in A(s),\) \(c\in F.\) The best response mapping \(br(c):= br_t(c^{t+1})\) (where *t* is arbitrary and \(c^{t+1}=(c,c,\ldots )\) can be identified with \(c\in F\)) is by Lemma 8 continuous on the convex compact set *F* in the space *X* (Lemma 1). From the Schauder–Tychonoff fixed point theorem (see Aliprantis and Border 2006), it follows that there exists \(c^*\in F\) such that \(c^* =br(c^*)\). Clearly, the constant sequence \((c^*,c^*,\ldots )\) is an *SMPE*. \(\square \)

*D*in

*S*and investment \(y\in S\)

*D*.

We now point out three special cases of our assumptions:

### Example 1

*t*, e.g., \(u(s)= s^\sigma \) with \(\sigma \in (0,1).\) Then, setting \(w(s)=(s+r)^\sigma ,\) \(r\ge 1,\) it follows that the assumptions (W1) and (U) are satisfied. Moreover, (W3) and (Q) also hold. Note that \(q(\{0\}|0)=1.\) Now we prove (W2). Assuming that each \(\xi _t\) has the uniform distribution on [0, 1], we obtain by Jensen’s inequality that

*u*and \(f_1\) and \(f_2\) may depend on

*t*.

### Example 2

*additive shocks*: \(\bar{f} (y_t,\xi _t)= f (y_t) +\xi _t,\) where \(f :S\rightarrow S\) is a continuous increasing function. For instance, assume that \(f(y)=\ln (1+y)\) for all \(y\in S.\) The probability measure \(\pi \) is non-atomic with support included in \([0, +\infty )\) and \(\mathbb {E}\xi =\int _S z\pi (dz)<+\infty .\) Assume again that

*u*is as in Example 1. Then, the function \(w(s)=(s+r)^\sigma \) satisfies our assumptions. Indeed, observe that

### Example 3

The model with *multiplicative shocks*: \(\bar{f}(y_t,z_t)= f (y_t)\xi _t,\) where *f* is as in Example 2 and the probability measure \(\pi \) is non-atomic with support included in \([0,+\,\infty ).\)

Other functions *w* for which the transition probabilities of the type discussed above satisfy conditions (W1)–(W3) and (Q), (U) can be obtained by an adaptation of examples from Section 4 in Jaśkiewicz and Nowak (2011).

## 4 Markov perfect equilibria in models with transitions having atoms

The model considered in the previous section does not include deterministic transitions. In the deterministic case the basic continuity lemmas are doubtful in the class of discontinuous strategies \(F^\infty .\) A discussion of this issue can be found in Balbus et al. (2015a). In this section, we study some models involving atoms in the set \(S_+,\) but the transition probability has an additive form.

- (A1)There exist probability measures \(\nu _0, \nu _1,\ldots ,\nu _l\) on
*S*and functions \(h_0, h_1,\ldots ,h_l: S\rightarrow [0,1]\) such thatwhere the functions \(h_i\) are continuous and$$\begin{aligned}q(\cdot |s-a )=\sum _{i=0}^l h_i(s-a)\nu _i (\cdot ),\end{aligned}$$$$\begin{aligned}\sum _{i=0}^l h_i(s-a)=1\quad \text{ for } \text{ all } s\in S,\ a\in A(s).\end{aligned}$$ - (A2)
Every function \(h_i\) for \(i=1,\ldots ,l\) is increasing and strictly concave.

- (A3)
There exists a constant \(M>0\) such that \(\sup _{t\in T}\max _{i\in \{0,\ldots ,l\}}\int _S u_t(s)\nu _i(ds)\le M.\)

- (A4)
\(\nu _i \succ \nu _0, \) i.e., \(\nu _i\) stochastically dominates \(\nu _0\) for each \(i=1,\ldots ,l.\)

We now state our main results in this section.

### Theorem 3

Assume that (A1)–(A4) and (U) are satisfied. Then there exists an \({ MPE}\) \(\hat{c}=(\hat{c}_t)\) that belongs to the space \(F_L^\infty :=F_L\times F_L\times \cdots .\)

### Theorem 4

Assume that (A1)–(A4) and (U) hold and the model is stationary, i.e., \(u_t=u\) are independent of \(t\in T.\) Then there exists an *SMPE* \(\hat{c}=(c_0,c_0,\ldots )\) with \(c_0\in F_L.\)

Let \(\mathcal{C}(S)\) be the space of all real-valued continuous functions on *S*. We assume that \(\mathcal{C}(S)\) is endowed with the well-known topology of uniform convergence on compact sets. Recall that a sequence \((f_n)_{n\in \mathbb {N}}\) of functions in \(\mathcal{C}(S)\) converges to some \(f\in \mathcal{C}(S)\) if and only if for any compact interval \(K\subset S\) we have that \(\lim _{n\rightarrow \infty } \sup _{s\in K} |f_n(s)-f(s)|=0.\)

### Lemma 9

The set \(F_L\) is convex and sequentially compact in \(\mathcal{C}(S).\)

### Proof

Clearly, the set \(F_L\) is convex. In order to show the sequential compactness we proceed similarly as in Lemma 1. However, in this proof, if \(f\in F_L \) and \(m\in \mathbb {N},\) then \(f^m \) is the restriction of *f* to the interval [0, *m*]. Let \((f_n )_{n\in \mathbb {N}}\) be any sequence of functions in \(F_L\) and let us apply the standard “diagonal method”. By the Arzelà–Ascoli theorem (see Billingsley 1968), there exists a subsequence \((n_1(k))\) of (*n*) such that \((f^1_{n_{1}(k)})_{k\in \mathbb {N}}\) converges uniformly (as \(k\rightarrow \infty \)) to some \(f^1_o\in F_L\) on [0, 1]. Next, we chose a subsequence \((n_2(k))\) of \((n_1(k))\) such that \((f^2_{n_{2}(k)})_{k\in \mathbb {N}}\) converges uniformly to some \(f^2_o\in F_L\) on [0, 2] and \(f^2_o(s)=f^1_o(s)\) for each \(s\in [0,1].\) Continuing this procedure we get a family of functions \(\{f^r\}_{r\in \mathbb {N}}\) in \(F_L\) such that \(f^r_o(s)=f^{r-1}_o(s)\) for each \(s\in [0,r-1]\) and \(r\ge 2.\) The conclusion now follows from the same arguments used in the proof of Lemma 1. We define \(f_o(s):= f^m_o(s)\) if \(s\in [0,m],\) \(m\in \mathbb {N}.\) It is easy to see that the diagonal sequence converges to \(f_o\) uniformly on every compact subset of *S*. \(\square \)

By Lemma 9 and Tychonoff’s theorem, we conclude the following fact.

### Corollary 2

The space \(F_L^\infty \) endowed with the product topology is sequentially compact.

### Lemma 10

### Proof

From Lemma 10, we conclude the following result.

### Lemma 11

### Proof

### Lemma 12

Assume that (A1)–(A4) and (U) hold. Then for any \( c^{t+1}\) in \(F_L^\infty \) the function \(s\rightarrow J_{t+1}(c^{t+1})(s)\) is non-decreasing.

### Proof

The next result states that the best response set \(BR_t(c^{t+s})(s)\) of self \(t\in T\) is a singleton.

### Lemma 13

Let (A1)–(A4) and (U) be satisfied. Assume that \( c^{t+1}\) in \(F_L^\infty .\) Then, \(BR_t(c^{t+1})=\{br_t(c^{t+1})\}\) and \(br_t(c^{t+1})\in F_L.\)

### Proof

*A*(

*s*) for every \(s\in S_+.\) Therefore, the sets \(BR_t(c^{t+1})(s)\) and \(\overline{BR}_t(c^{t+1})(s)\) are singletons for every \(s\in S.\) Moreover, we have \(BR_t(c^{t+1})(s)=\{br_t(c^{t+1})(s)\}\) for all \(s\in S\) and \(\overline{BR}_t(c^{t+1})(s)=\{i_t(c^{t+1})(s)\}\) where \(i_t(c^{t+1})(s)=s-br_t(c^{t+1})(s),\) \(s\in S.\) From the strict concavity of \(u_t\) and Lemma 3.2 in Balbus et al. (2015a), it follows that the function \(s\rightarrow i_t(c^{t+1})(s)\) is non-decreasing. From the strict concavity of \(y\rightarrow \int _S J_{t+1}(c^{t+1})(s')q(ds'|y)\) and Lemma 3.2 in Balbus et al. (2015a), we conclude that the function \(s\rightarrow br_t(c^{t+1})(s)\) is also non-decreasing. We obviously know that the functions \(i_t(c^{t+1})\) and \(br_t(c^{t+1})\) satisfy \( 0\le i_t(c^{t+1})(s)\le s \) and \(0\le br_t(c^{t+1})(s)\le s,\) \(s\in S.\) Thus, \(br_t(c^{t+1})\in F_L.\) If for every \(i=1,\ldots ,l\), we have equality in (28), then \(BR_t(c^{t+1})(s)=\{br_t(c^{t+1})(s)\}=\{s\}\) for all \(s\in S.\) \(\square \)

### Lemma 14

Assume that (A1)–(A3) and (U) hold. Then, the mapping \(br_t: F_L^\infty \rightarrow F_L\) is continuous.

### Proof

Suppose that \(c^{t+1,m} \rightarrow c^{t+1}\) as \(m\rightarrow \infty .\) Put \(\psi ^m:= br_t(c^{t+1,m}).\) Let \(\psi \) be any accumulation point of the sequence \((\psi ^m)\) in \(F_L\) being a compact space. We have to show that \(\psi =br_t(c^{t+1}).\) However, this fact easily follows from the dominated convergence theorem, Lemmas 10, 11 and 13. \(\square \)

### Proof of Theorem 3

We note that \(\hat{G}_{t+1}\subset \hat{G}_{t}.\) Since all best response mappings are continuous by Lemma 14 and the space \(F_L^\infty \) is compact, every set \(\hat{G}_t\) is non-empty and compact. Therefore, the set \(\hat{G}:=\cap _{t\in T} \hat{G}_t\) is non-empty and compact. Choose any \(\hat{c}=(\hat{c}_1,\hat{c}_2,\ldots )\in \hat{G}.\) Then \(\hat{c}\in \hat{G}_t\) for every \(t\in T.\) This implies that \(\hat{c}_t=br_t(\hat{c}^{t+1})\) for every \(t\in T.\) Hence, \(\hat{c}\) is an *MPE*. \(\square \)

### Proof of Theorem 4

In the stationary case, we again restrict attention to constant sequences \((c,c,\ldots )\in F_L^\infty .\) Clearly, such a sequence can be identified with \(c\in F.\) Function (3) can be regarded as a function *P*(*a*, *c*)(*s*), where \(a\in A(s),\) \(c\in F_L.\) By Lemma 14 the best response mapping \(br(c):= br_t(c^{t+1})\) is continuous on the convex compact set \(F_L\) in the space \(\mathcal{C}(S)\) (Lemma 9). From the Schauder–Tychonoff fixed point theorem (see Aliprantis and Border 2006), it follows that there exists \(c^*\in F_L\) such that \(c^* =br(c^*)\). Hence, the constant sequence \((c^*,c^*,\ldots )\) is an *SMPE*. \(\square \)

### Remark 1

In assumptions (A1)–(A4) the functions \(h_i\) and the measures \(\nu _i,\) (\(i=0,\ldots ,l\)) may depend on \(t\in T.\) Then, Lemmas 10–14 and Theorems 3, 4 remain valid. However, for the sake of clarity of notation we skip this dependence.

### Remark 2

The special transition structure in this section and condition (U) imply that both functions \(y\rightarrow P_t(s-y,c^{t+1})(s)\) and \(a\rightarrow P_t(a,c^{t+1})(s)\) are strictly concave on *A*(*s*) for every \(s\in S_+\) and \(c^{t+1}\in F_L^\infty .\) This plays a crucial role in getting the best reply \(br_t(c^{t+1})\in F_L.\) If the transition probability has some atoms in \(S_+\) and does not have the special additive form, then there is a problem with the continuity of \(a\rightarrow P_t(a,c^{t+1})(s)\) for some \(c^{t+1} \in F^\infty \) and the best response \(br_t(c^{t+1})\) to \(c^{t+1}\) may not exist.

## 5 Comments

In this section we give some remarks on the relation of our results with the literature.

### Remark 3

The decision model studied in this paper can be viewed as a game between generations. Self *t* can be considered as generation *t* having the total utility depending on all its descendants. Further comments on this issue can be found in Balbus et al. (2015a). In the intergenerational game setting, it is desirable to assume that the period utility functions \(u_t\) and transition functions \(q_t\) depend on generation \(t\in T.\) This natural requirement motivates us to consider the non-stationary models. Theorems 1 and 3 are new results in the area of decision processes with quasi-hyperbolic discounting (or intergenerational games). It is interesting to note that their proofs are not based on any fixed point argument. The idea is to consider the intersection *G* of the sets \(G_t\) and is partly inspired by the fundamental work of Mertens and Parthasarathy (2003) on subgame perfect equilibria in *n*-person discounted stochastic games with simultaneous moves of the players. Balbus and Woźny (2016) also deal with a similar stationary model but with a compact state space and a relatively narrow class of transition functions.

### Remark 4

Theorems 2 and 4 establish the existence of *SMPE* in stationary models and have some predecessors in the literature. A version of Theorem 2 with compact state space follows from Theorem 3.1 in Balbus et al. (2015a). In this case the function *u* is bounded. Here, on the other hand, we deal with unbounded state space and unbounded period utility functions. Such a setting covers more applications in economics, e.g., models with power or logarithmic utility functions. A related result to Theorem 1 is stated in Harris and Laibson (2001). Nonetheless, some of their assumptions are much stronger. They assume that \(S=[0,+\infty )\) but the transition probability is induced by a difference equation with additive non-atomic noise with values in \([y_1,y_2]\subset S_+.\) The period utility function *u* may be unbounded from below. However, it has a bounded relative risk-aversion coefficient. The class of strategies *F* (applied in Sect. 2) was used by Bernheim and Ray (1983) to study equilibria in altruistic growth models and by Majumdar and Sundaram (1991) to study Nash equilibria in symmetric stochastic games of resource extraction. Our paper owes much to their contributions.

### Remark 5

The additivity assumptions similar to (A1)–(A3) were used in the study of models of intergenerational stochastic games or decision processes with quasi-hyperbolic discounting. In Balbus and Nowak (2008) a class of stochastic games between generations is considered where each generation consists of finitely many players. The proof of the main results is based on different ideas than in this work. Jaśkiewicz and Nowak (2014) also study models with additive non-atomic transitions but with risk-sensitive preferences. The most relevant work on models satisfying assumptions of type (A1)–(A3) is the paper of Balbus et al. (2014) where the transition probability depends on some unknown parameter chosen by a malevolent nature. Therefore, the notion of a robust Markov perfect equilibrium is introduced. Balbus et al. (2014) assume that the state space is a compact interval and the utility function *u* is bounded from above. The additivity condition on the transition probability function is also made in Balbus et al. (2015c), but with an additional restrictive assumption that \(\nu _0\) is the Dirac measure concentrated at zero. Thus, \(s=0\) is an absorbing state. Theorem 4 is not a corollary to their results. It should be noted, however, that they study an *n*-dimensional state space model.

### Remark 6

The existence of an MPE in models with quasi-hyperbolic discounting and deterministic transitions is open and seems to be difficult. Examples 5.1 and 5.2 in Balbus et al. (2015a) suggest that Lemma 6 may fail to hold in the class \(F^\infty \) of strategies. On the other hand, working with the class \(F_L^\infty \) as a strategy set leads to very restrictive assumptions.

### Remark 7

Our proofs in both models (in atomic and non-atomic transitions) heavily rely on the assumption that the set *S* is one-dimensional. For example, the definition of \(br_t(c^{t+1})\) given in (17) is based on the fact that \(BR_t(c^{t+1})(s)\subset {\mathbb {R}}.\) Furthermore, the monotonicity of best reply functions with respect to state variable is very difficult to obtain in the model with multidimensional state space. We conjecture that a solution to dynamic decision models with quasi-hyperbolic preferences and more than one resource requires some new ideas and methods.

## Notes

### Acknowledgements

We thank two anonymous referees for helpful comments. The authors acknowledge the financial support from the National Science Centre, Poland: Grants 2016/23/B/HS4/02398 (Ł. Balbus) and 2016/23/B/ST1/00425 (A. Jaśkiewicz and A. S. Nowak).

## References

- Ainslie, G. (1992).
*Picoeconomics*. Cambridge: Cambridge University Press.Google Scholar - Aliprantis, C., & Border, K. (2006).
*Infinite dimensional analysis: A hitchhiker’s guide*. New York: Springer.Google Scholar - Alj, A., & Haurie, A. (1983). Dynamic equilibria in multigenerational stochastic games.
*IEEE Transactions on Automatic Control*,*28*, 193–203.CrossRefGoogle Scholar - Balbus, Ł., Jaśkiewicz, A., & Nowak, A. S. (2014). Robust Markov perfect equilibria in a dynamic choice model with quasi-hyperbolic discounting. In J. Haunschmied, V. M. Veliov, & S. Wrzaczek (Eds.),
*Dynamic games in economics Dynamic modeling and econometrics in economics and finance*(Vol. 16, pp. 1–22). Berlin: Springer.Google Scholar - Balbus, Ł., Jaśkiewicz, A., & Nowak, A. S. (2015a). Existence of stationary Markov perfect equilibria in stochastic altruistic growth economies.
*Journal of Optimization Theory and Applications*,*165*, 295–315.CrossRefGoogle Scholar - Balbus, Ł., & Nowak, A. S. (2008). Existence of perfect equilibria in a class of multigenerational stochastic games of capital accumulation.
*Automatica*,*44*, 1471–1479.CrossRefGoogle Scholar - Balbus, Ł., Reffett, K., & Woźny, Ł. (2015c). Computing time-consistent Markov policies for quasi-hyperbolic consumers under uncertainty.
*International Journal of Game Theory*,*44*, 83–112.CrossRefGoogle Scholar - Balbus, Ł., Reffett, K., & Woźny, Ł. (2018). Dynamic games in macroeconomics. In T. Başar & G. Zaccour (Eds.),
*Handbook of dynamic game theory*. Basel: Birkhäuser. https://doi.org/10.1007/978-3-319-27335-8_18-1.Google Scholar - Balbus, Ł., & Woźny, Ł. (2016). A strategic dynamic programming method for studying short-memory equilibria of stochastic games with uncountable number of states.
*Dynamic Games and Applications*,*6*, 187–208.CrossRefGoogle Scholar - Bernheim, D., & Ray, D. (1983).
*Altruistic growth economies I. Existence of bequest equilibria.*Technical Report no. 419. Institute for Mathematical Studies in the Social Sciences, Stanford University.Google Scholar - Billingsley, P. (1968).
*Convergence of probability measures*. New York: Wiley.Google Scholar - Blackwell, D. (1965). Discounted dynamic programming.
*Annals of Mathematical Statistics*,*36*, 226–235.CrossRefGoogle Scholar - Harris, C., & Laibson, D. (2001). Dynamic choices of hyperbolic consumers.
*Econometrica*,*69*, 935–957.CrossRefGoogle Scholar - Haurie, A. (2005). A multigenerational game model to analyze sustainable development.
*Annals of Operations Research*,*137*, 369–386.CrossRefGoogle Scholar - Hernández-Lerma, O., & Lasserre, J. B. (1996).
*Discrete-time Markov control processes: Basic optimality criteria*. New York: Springer.CrossRefGoogle Scholar - Jaśkiewicz, A., & Nowak, A. S. (2011). Stochastic games with unbounded payoffs: Applications to robust control in economics.
*Dynamic Games and Applications*,*1*, 253–279.CrossRefGoogle Scholar - Jaśkiewicz, A., & Nowak, A. S. (2014). Stationary Markov perfect equilibria in risk-sensitive stochastic overlapping generations models.
*Journal of Economic Theory*,*151*, 411–447.CrossRefGoogle Scholar - Jaśkiewicz, A., & Nowak, A. S. (2018). Non-zero-sum stochastic games. In T. Başar & G. Zaccour (Eds.),
*Handbook of dynamic game theory*. Basel: Birkhäuser. https://doi.org/10.1007/978-3-319-27335-8_33-1.Google Scholar - Karp, L. (2005). Global warming and hyperbolic discounting.
*Journal of Public Economics*,*89*, 261–282.CrossRefGoogle Scholar - Leininger, W. (1986). The existence of perfect equilibria in model of growth with altruism between generations.
*Review of Economic Studies*,*53*, 349–368.CrossRefGoogle Scholar - Majumdar, M. K., & Sundaram, R. K. (1991). Symmetric stochastic games of resource extraction. The existence of non-randomized stationary equilibrium. In T. E. S. Raghavan, T. S. Ferguson, T. Parthasarathy, & O. J. Vrieze (Eds.),
*Stochastic games and related topics*(pp. 175–190). Dordrecht: Kluwer Academic Publishers.CrossRefGoogle Scholar - Maskin, E., & Tirole, J. (2001). Markov perfect equilibrium: I. Observable actions.
*Journal of Economic Theory*,*100*, 191–219.CrossRefGoogle Scholar - Mertens, J. F., & Parthasarathy, T. (1991). Nonzero-sum stochastic games. In T. E. S. Raghavan, T. S. Ferguson, T. Parthasarathy, & O. J. Vrieze (Eds.),
*Stochastic games and related topics*(pp. 145–148). Dordrecht: Kluwer Academic Publishers.CrossRefGoogle Scholar - Mertens, J. F., & Parthasarathy, T. (2003). Equilibria for discounted stochastic games. In A. Neyman & S. Sorin (Eds.),
*Stochastic games and applications*(pp. 131–217). Dordrecht: Academic Publishers.CrossRefGoogle Scholar - Montiel Olea, J. L., & Strzalecki, T. (2014). Axiomatization and measurement of quasi-hyperbolic discounting.
*Quarterly Journal of Economics*,*129*, 1449–1499.CrossRefGoogle Scholar - Neveu, J. (1965).
*Mathematical foundations of the calculus of probability*. San Francisco: Holden-Day.Google Scholar - Parthasarathy, K. R. (1967).
*Probability measures on metric spaces*. New York: Academic.CrossRefGoogle Scholar - Peleg, B., & Yaari, M. E. (1973). On the existence of a consistent course of action when tastes are changing.
*Review of Economic Studies*,*40*, 391–401.CrossRefGoogle Scholar - Phelps, E., & Pollak, R. (1968). On second best national savings, and game equilibrium growth.
*Review of Economic Studies*,*35*, 195–199.CrossRefGoogle Scholar - Serfozo, R. (1982). Convergence of Lebesgue integrals with varying measures.
*Sankhya: The Indian Journal of Statistic (Ser A)*,*44*, 380–402.Google Scholar - Strotz, R. H. (1956). Myopia and inconsistency in dynamic utility maximization.
*Review of Economic Studies*,*23*, 165–180.CrossRefGoogle Scholar - Topkis, D. (1978). Minimizing a submodular function on a lattice.
*Operations Research*,*26*, 305–321.CrossRefGoogle Scholar - Topkis, D. (1998).
*Supermodularity and complementarity*. Princeton: Princeton University Press.Google Scholar - Wessels, J. (1977). Markov programming by successive approximations with respect to weighted supremum norms.
*Journal of Mathematical Analysis and Applications*,*58*, 326–335.CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.