Markov perfect Nash equilibria in models with a single capital stock

Abstract

Many economic problems can be formulated as dynamic games in which strategically interacting agents choose actions that determine the current and future levels of a single capital stock. We study necessary as well as sufficient conditions that allow us to characterise Markov perfect Nash equilibria for these games. These conditions can be translated into an auxiliary system of ordinary differential equations that helps us to explore stability, continuity and differentiability of these equilibria. The techniques are used to derive detailed properties of Markov perfect Nash equilibria for several games including voluntary investment in a public capital stock, the inter-temporal consumption of a reproductive asset and the pollution of a shallow lake.

Appendices

Appendix 1: Evolution near non-Lipschitz points

For continuous one-dimensional vector fields $F:X\rightarrow {\mathbb R}$, where $X$ is a closed interval of ${\mathbb R}$, Peano’s theorem (Peano 1890) guarantees the existence of a positive constant $T>0$, possibly infinite, and a differentiable function $x:[0,T]\rightarrow {\mathbb R}$ satisfying

$$\begin{aligned} \dot{x} = F(x) \end{aligned}$$

(39)

for all $t\in [0,T]$, and such that $x(T)\in \partial X$.

At points $\hat{x}$ where the right-hand side $F$ of an ordinary differential equation has an isolated discontinuity, Peano’s theorem does not apply. For our purposes, it is sufficient to have the existence of continuous functions $x(t)$ that satisfy (39) for all $t\in [0,\infty )\backslash N$, where $N$ is a discrete set, that is, a set without limit points. For the purpose of this appendix, we shall call these piecewise solutions in analogy to piecewise differentiable functions. Piecewise solutions are a special case of Carathéodory solutions, which are absolutely continuous functions $x(t)$, satisfying (39) almost everywhere on $[0,\infty )$ (cf. Hájek 1979).

The theorem of this section gives a condition for one-dimensional vector fields with isolated jump discontinuities to have piecewise solutions.

Theorem 8

Let $U\subset {\mathbb R}$ be an open interval including $\hat{x}$, and let $F$ restricted to $U\backslash \{\hat{x}\}$ be continuous, non-zero and such that the right and left limits $F_R$ and $F_L$ of $F(x)$ exist as $x$ tends to $\hat{x}$ from the right and from the left, respectively. Assume that

$$\begin{aligned} \text {if} \quad F_L \ge 0 \ge F_R, \quad \text {then}\quad F(\hat{x}) = 0. \end{aligned}$$

Then for all $x_0\in U$, there exists a piecewise solution of (39) that satisfies $x(0)=x_0$ and that is defined for all $t$ such that $x(t)\in U$.

Proof

In the proof, ‘trajectory’ will indicate a solution $x$ of the differential equation, whose existence is guaranteed by Peano’s theorem, that is, as long as $x(t)\ne \hat{x}$. A statement about a trajectory $x$ that holds ‘for all $t$’ will always mean ‘for all $t$ such that $x(t)\in U$’.

There are a number of different situations. Firstly, if $F_L=F_R$, then $F$ is continuous on $U$, and Peano’s theorem yields the existence of a solution to the differential equation for all $t$.

Secondly, $F_L$ and $F_R$ may be both nonnegative or both non-positive. Assume for definiteness that both are nonnegative. Then, a trajectory starting at $x_0>\hat{x}$ does not decrease, never reaches $\hat{x}$ and yields a piecewise solution for all $t$, while a trajectory $x_1$ starting at $x_0\le \hat{x}$ reaches $\hat{x}$ at some time $T\ge 0$; if $T=\infty$, then $x_1$ is already a piecewise solution. Assume therefore that $T$ is finite. Introduce

$$\begin{aligned} G(x) = \left\{ \begin{array}{l@{\quad }l} F(x) &{} x > \hat{x},\\ F_R &{} x \le \hat{x}. \end{array}\right. \end{aligned}$$

(40)

Let then $x_2$ be a solution of $\dot{x} = G(x)$ with initial condition $x_2(T)=\hat{x}$, which exists as $G$ is continuous. The trajectory that is equal to $x_1(t)$ for $0\le t < T$ and $x_2(t)$ for $t\ge T$ is a piecewise solution.

Thirdly, there is the possibility that $F_L>0>F_R$. By assumption, we then have $F(\hat{x}) = 0$. A trajectory $x_1$ starting at $x_0<\hat{x}$ will satisfy $\lim _{t\uparrow T} x_1(T) = \hat{x}$ for some finite time $T$. Concatenation with the constant trajectory $x_2(t)=\hat{x}$ for $t\ge T$ again yields a piecewise solution. The case $x_0>\hat{x}$ is handled in the same manner.

Lastly, there is the situation that $F_L<0<F_R$. As above, a trajectory with initial value $x_0>\hat{x}$ is increasing, hence defined for all $t$ and yields a piecewise solution; likewise, trajectories starting at $x_0<\hat{x}$ are decreasing and are also defined for all $t\ge 0$. If finally $x_0=\hat{x}$, let $G$ be defined as in (40), and let $x$ be a solution of $\dot{x} = G(x)$ with $x(0)=x_0$. As $G(x)>0$ for all $x$, $x(t)$ is increasing in $t$ and satisfies therefore $G(x(t))=F(x(t))$ for all $t>0$. Hence, it is a piecewise solution as well. $\square$

Appendix 2: Proof of the sufficiency theorem

In this section, the Proof of theorem 1 is given. Before starting, we make a general remark on superdifferentials of viscosity solutions $V:X\rightarrow {\mathbb R}$ to the Hamilton–Jacobi equation

$$\begin{aligned} \rho V = G(x,V'), \end{aligned}$$

where $G:X\times {\mathbb R}\rightarrow {\mathbb R}$.

Theorem 9

Let $G=G(x,p)$ be a continuous function that is strictly convex in $p$, let $\hat{x}$ be a point in $X$, let $U$ be an open neighbourhood of $\hat{x}$ in $X$, and let $V$ be a viscosity solution of the Hamilton–Jacobi equation (11) that is continuously differentiable on $U\backslash \{\hat{x}\}$. Then necessarily

$$\begin{aligned} \lim _{x\uparrow \hat{x}} V'(x) \le \lim _{x\downarrow \hat{x}} V'(x) \end{aligned}$$

and

$$\begin{aligned} \lim _{x\uparrow \hat{x}} G_p(x,V'(x)) \le \lim _{x\downarrow \hat{x}} G_p(x,V'(x)) . \end{aligned}$$

Corollary 2

Let $G$ and $V$ be as in Theorem 9. Consider the state evolution equation

$$\begin{aligned} \dot{x} = F(x) = G_p(x,V'(x)), \end{aligned}$$

(41)

defined for all $x$ where $V'$ is differentiable in $x$. If at a point $\hat{x}$ the left and right limits $F_L$ and $F_R$ of $F$ exist, then $F_L\le F_R$.

Remark 1

It follows from theorem 8 that under the conditions of Theorem 9, the state evolution Eq. (41) has a piecewise solution.

Remark 2

Theorem 9 applies for instance to the global Markov perfect Nash equilibrium of the shallow lake model, discussed in Sect. 4.2.

Proof

(Of theorem 9). Let

$$\begin{aligned} p_L = \lim _{x\uparrow \hat{x}} V'(x),\quad p_R = \lim _{x\downarrow \hat{x}} V'(x), \end{aligned}$$

and assume by contradiction that

$$\begin{aligned} p_L > p_R. \end{aligned}$$

Note that since $V$ is $C^1$ on $U\backslash \{\hat{x}\}$, we have

$$\begin{aligned} \rho V(\hat{x}) = G(\hat{x}, p_L) = G(\hat{x}, p_R). \end{aligned}$$

Moreover, since $p_L>p_R$

$$\begin{aligned}{}[p_R,p_L] \subset D_+V(\hat{x}). \end{aligned}$$

Since $G(\hat{x},p)$ is strictly convex in $p$, it follows that for $\bar{p} \in (p_R,p_L)$, we have

$$\begin{aligned} G(\hat{x},\bar{p}) < \rho V(\hat{x}) \end{aligned}$$

On the other hand, since $V$ is a viscosity solution and $\bar{p} \in D_+V(\hat{x})$, it follows that

$$\begin{aligned} G(\hat{x},\bar{p}) \ge \rho V(\hat{x}). \end{aligned}$$

This is impossible, hence $p_L\le p_R$. $\square$

We now give the Proof of theorem 1.

Proof

We have to show the following. If the strategy vector of the players other than player $i$ equals $\mathbf{u}^*_{-i}$, then $u^*_i$ is a best response of player $i$; in other words, $u_i(x)=u^*_i(x)$ solves the optimisation problem of player $i$.

The proof consists of two parts, and it rests on the verification that $V_i(x)$ is the value function of the optimisation problem of player $i$. Let $u_i(x)$ be any admissible strategy, set

$$\begin{aligned} {\bar{\mathbf{u}}}(x) = (u_i(x),\mathbf{u}^*_{-i}(x)), \end{aligned}$$

and let $\bar{x}$ be any piecewise solution of

$$\begin{aligned} \dot{\bar{x}} = f(\bar{x},{\bar{\mathbf{u}}}(\bar{x})), \qquad \bar{x}(0) = x_0, \end{aligned}$$

(42)

whose existence is guaranteed by Theorem 8. The first part of the proof shows that then

$$\begin{aligned} \int \limits _0^\infty L_i(\bar{x},{\bar{\mathbf{u}}}(x))\mathrm {e}^{-\rho t} \mathrm {d}t \le V_i(x_0); \end{aligned}$$

(43)

that is, $V_i(x_0)$ is an upper bound for the pay-off of player $i$.

Then, for $u_i=u_i^*(x)$, and $x=x^*$ being any piecewise solution of

$$\begin{aligned} \dot{x} = f(x,\mathbf{u}^*(x)),\quad x(0)=x_0, \end{aligned}$$

(44)

the second part of the proof demonstrates the opposite inequality

$$\begin{aligned} \int \limits _0^\infty L_i(x^*,\mathbf{u}^*(x^*))\mathrm {e}^{-\rho t} \mathrm {d}t \ge V_i(x_0). \end{aligned}$$

(45)

Together, these inequalities show that $u_i^*$ is a best response of player $i$.

Part one. As ${\bar{\mathbf{u}}}$ is piecewise differentiable, the set $C$ of points where $f(x, {\bar{\mathbf{u}}}(x))$ fails to be differentiable is a set of isolated points.

Let $D$ be the set of states at which $\mathbf{V}$ fails to be differentiable. By assumption this is a set of isolated points as well. Take $\varepsilon \in (0,1)$ arbitrarily. Because of condition (15), there is a constant $T>1/\varepsilon >0$ such that

$$\begin{aligned} \left| V_i(\bar{x}(T))\mathrm {e}^{-\rho T}\right| < \varepsilon . \end{aligned}$$

(46)

Let $\Sigma \subset [0,T]$ be such that

$$\begin{aligned} \bar{x}(t) \in C\cup D \end{aligned}$$

if and only if $t\in \Sigma$. As $\bar{x}$ is a piecewise solution, there are time points

$$\begin{aligned} 0 \le t_1 < t_2 < \cdots < t_k\le T, \end{aligned}$$

such that the set $\Sigma$ is the union of the finite set $\Sigma _1=\{t_1,\ldots ,t_{k-1}\}$ and the interval $\Sigma _2=[t_k,T]$, where it is understood that $\Sigma _1$ may be empty and $\Sigma _2$ may have zero length. Note that

$$\begin{aligned} f(\bar{x}(t),\bar{u}(\bar{x}(t)))=0 \end{aligned}$$

if $t\in \Sigma _2$.

Also set

$$\begin{aligned} x_j = \bar{x}(t_j), \quad j=1,\ldots , k. \end{aligned}$$

From (46) it follows that

$$\begin{aligned}&- V_i(x_0) \le \varepsilon + V_i(\bar{x}(T))\mathrm {e}^{-\rho T} - V_i(x_0)\nonumber \\&\quad = \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j} \frac{\mathrm {d}}{\mathrm {d}t}\left( V_i(\bar{x}) \mathrm {e}^{-\rho t}\right) \mathrm {d}t + \int \limits _{t_k}^T \frac{\mathrm {d}}{\mathrm {d}t}\left( V_i(x_k) \mathrm {e}^{-\rho t}\right) \mathrm {d}t. \end{aligned}$$

(47)

As $V_i(x)$ is differentiable in the open intervals $(x_{j-1},x_j)$ as a function of $x$ and $\bar{x}$ is differentiable in $(t_{j-1},t_j)$ as a function of $t$, the differentiations can be performed to yield

$$\begin{aligned} - V_i(x_0)&\le \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j}\left( V_i'(\bar{x})f\bigl (\bar{x},{\bar{\mathbf{u}}}(\bar{x})\bigr ) - \rho V_i(\bar{x})\right) \mathrm {e}^{-\rho t} \mathrm {d}t\end{aligned}$$

(48)

$$\begin{aligned}&\qquad + \int \limits _{t_k}^T \left( p_i f\bigl (x_k,{\bar{\mathbf{u}}}(x_k)\bigr ) - \rho V_i(x_k)\right) \mathrm {e}^{-\rho t} \mathrm {d}t. \end{aligned}$$

(49)

Here, the constant $p_i\in D_-V_i(x_k)$ is an arbitrary subderivative of $V_i$ at $x_k$; as $x_k$ is a steady state, we have that the inserted term

$$\begin{aligned} p_i f(x_k,{\bar{\mathbf{u}}}(x_k)) = 0. \end{aligned}$$

We compute

$$\begin{aligned}&\int \limits _0^{T} L_i(\bar{x},{\bar{\mathbf{u}}})\mathrm {e}^{-\rho t} \mathrm {d}t - V_i(x_0)\\&\le \varepsilon + \sum _{j=1}^k\int \limits _{t_{j-1}}^{t_j} L_i(\bar{x},{\bar{\mathbf{u}}}(\bar{x})) + V_i'(\bar{x}) f(\bar{x},{\bar{\mathbf{u}}}) - \rho V_i(\bar{x}) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\qquad + \int \limits _{t_k}^T L_i(x_k,{\bar{\mathbf{u}}}(x_k)) + p_i f(x_k,{\bar{\mathbf{u}}}(x_k)) - \rho V_i(x_k) \mathrm {e}^{-\rho t} \mathrm {d}t\\&= \varepsilon + \sum _{j=1}^k\int \limits _{t_{j-1}}^{t_j} P_i\bigl (\bar{x},V_i'(\bar{x}),u_i(\bar{x});\mathbf{u}^*_{-i}(\bar{x}) \bigr ) - \rho V_i(\bar{x}) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\qquad + \int \limits _{t_k}^T P_i(x_k,p_i,u_i(x_k); \mathbf{u}_{-i}^*(x_k))-\rho V_i(x_k) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\le \varepsilon + \sum _{j=1}^k\int \limits _{t_{j-1}}^{t_j} H_i\bigl (\bar{x},V_i'(\bar{x});\mathbf{u}^*_{-i}(\bar{x})\bigr ) - \rho V_i(\bar{x}) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\qquad + \int \limits _{t_k}^T H_i(x_k,p_i; \mathbf{u}_{-i}^*(x_k))-\rho V_i(x_k) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\le \varepsilon + \sum _{j=1}^k\int \limits _{t_{j-1}}^{t_j} G_i\bigl (\bar{x},\mathbf{V'}(\bar{x})\bigr ) - \rho V_i(\bar{x}) \mathrm {e}^{-\rho t} \mathrm {d}t = \varepsilon . \end{aligned}$$

Note that for the second inequality, we used that $H_i(x,p_i) = \max _{u_i} P_i(x,p_i,u_i)$. In the last inequality, we used that $V_i$ is a viscosity supersolution of the Hamilton–Jacobi Eq. (14). Letting $\varepsilon \rightarrow 0$ now yields inequality (43).

Part two. It remains to show the opposite inequality (45) if $u_i(x) = u^*_i(x)$ for all $x$, and $x=x^*$ solving (44). Let $C$ denote the isolated set of states where $f(x,\mathbf{u}^*(x))$ fails to be differentiable, and let the set $D$ be defined as above. Repeat the construction of $T$, the $t_i$, and the sets $\Sigma _1$ and $\Sigma _2$, but now with $\bar{x}$ replaced by $x^*$.

With an analogous reasoning as used to derive (48), we can show that

$$\begin{aligned} - V_i(x_0)&\ge - \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j}\left( V_i'(x^*) f\bigl (x^*,\mathbf{u}^*(x^*)\bigr ) - \rho V_i(x^*)\right) \mathrm {e}^{-\rho t} \mathrm {d}t\nonumber \\&\qquad + \int \limits _{t_k}^T \left( p_i f\bigl (x_k,\mathbf{u}^*(x_k)\bigr ) - \rho V_i(x_k)\right) \mathrm {e}^{-\rho t} \mathrm {d}t, \end{aligned}$$

(50)

where $p_i\in D_-V_i(x_k)$ is any subderivative of $V_i$ at $x_k$.

Again, if the interval $\Sigma _2$ is nontrivial, the point $x_k$ is a steady state of equation (42), with ${\bar{\mathbf{u}}}$ replaced by $\mathbf{u}^*$. Introduce

$$\begin{aligned} F(x)=f(x,\mathbf{u}^*(x)). \end{aligned}$$

By assumption, the strategy vector $\mathbf{u}^*$, and consequently the function $F$, is either left or right continuous at $x_k$ – say it is left continuous, the other case being similar. Setting

$$\begin{aligned} p_{iL} = \lim _{x\uparrow x_k} V'_i(x), \end{aligned}$$

it follows by continuity that

$$\begin{aligned} P_i(x^*,V_i'(x^*),u_i;\mathbf{u}^*_{-i}(x^*))\rightarrow P_i(x^*,p_{iL},u_i;\mathbf{u}^*_{-i}(x^*)) \quad \text {as} \quad x\uparrow x_k, \end{aligned}$$

and hence that

$$\begin{aligned} u_i^*(x_k) = \text {arg max}_{u_i} P_i(x_k,p_{iL},u_i; \mathbf{u}^*_{-i}(x_k)). \end{aligned}$$

(51)

Inequality (50) implies that

$$\begin{aligned}&\int \limits _0^T L_i(x^*,\mathbf{u}^*(x^*)) \mathrm {e}^{-\rho t} \mathrm {d}t - V_i(x_0)\\&\ge - \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j}\left( L_i(x^*,\mathbf{u}^*(x^*)) + V_i'(x^*) f\bigl (x^*,\mathbf{u}^*(x^*)\bigr ) - \rho V_i(x^*) \right) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\,\, + \int \limits _{t_k}^T \left( L_i(x_k,\mathbf{u}^*(x_k)) + p_{iL} f\bigl (x_k,\mathbf{u}^*(x_k)\bigr ) - \rho V_i(x_k)\right) \mathrm {e}^{-\rho t} \mathrm {d}t\\&= - \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j}\left( P_i(x^*,V_i'(x^*),u_i^*(x^*);\mathbf{u}^*_{-i}(x^*)) - \rho V_i(x^*)\right) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\,\, + \int \limits _{t_k}^T \left( P_i(x_k,p_{iL},u_i^*(x^*);\mathbf{u}^*_{-i}(x_k)) - \rho V_i(x_k)\right) \mathrm {e}^{-\rho t} \mathrm {d}t, \end{aligned}$$

by definition of $P_i$, and

$$\begin{aligned}&= - \varepsilon + \sum _{j=1}^k \int \limits _{t_{j-1}}^{t_j}\left( H_i(x^*,V_i'(x^*);\mathbf{u}^*_{-i}(x^*)) - \rho V_i(x^*)\right) \mathrm {e}^{-\rho t} \mathrm {d}t\\&\,\, + \int \limits _{t_k}^T \left( H_i(x_k,p_{iL};\mathbf{u}^*_{-i}(x_k)) - \rho V_i(x_k)\right) \mathrm {e}^{-\rho t} \mathrm {d}t, \end{aligned}$$

by equation (51).

All the terms of the sum vanish, since

$$\begin{aligned} H_i(x,V_i'(x);\mathbf{u}^*_{-i}(x)) = G_i(x,\mathbf{V}'(x)) = \rho V_i(x) \end{aligned}$$

whenever $x\in (x_{j-1},x_j)$. The final integral vanishes as well, as by continuity

$$\begin{aligned} \rho V_i(x)=G_i(x,\mathbf{V}'(x))\rightarrow H_i(x_k,p_{iL}; \mathbf{u}^*_{-i}(x_k)) \quad \text {as} \quad x\uparrow x_k. \end{aligned}$$

Continuity of $V_i$ then implies that

$$\begin{aligned} \rho V_i(x_k) = H_i(x_k,p_{iL}; \mathbf{u}^*_{-i}(x_k)). \end{aligned}$$

Taking the limit $\varepsilon \rightarrow 0$ demonstrates inequality (45). $\square$