Elementary results on solutions to the bellman equation of dynamic programming: existence, uniqueness, and convergence

Abstract

We establish some elementary results on solutions to the Bellman equation without introducing any topological assumption. Under a small number of conditions, we show that the Bellman equation has a unique solution in a certain set, that this solution is the value function, and that the value function can be computed by value iteration with an appropriate initial condition. In addition, we show that the value function can be computed by the same procedure under alternative conditions. We apply our results to two optimal growth models: one with a discontinuous production function and the other with “roughly increasing” returns.

Appendix: Proofs

1.1 Proof of Theorem 2.1

The proof consists of three lemmas and a concluding argument. The proof of the first lemma slightly generalizes an argument of Stokey and Lucas (1989, Theorem 4.3). The second lemma essentially shows that $B^{T} v$ with $T \in {{\mathbb {N}}}$ and $v \in V$ is the value function of the $T$-period problem with the value of the terminal stock $x_{T}$ given by $v(x_{T})$. This result extends the classical idea of Bertsekas and Shreve (1978, Section 3.2) to our setting. The last lemma is less trivial than the first two. The concluding argument applies the Knaster–Tarski fixed point theorem and combines the first and last lemmas.

Lemma 6.1

Let $\overline{v} \in V$ satisfy (2.17). Let $v \in V$ be a fixed point of $B$ with $v \le \overline{v}$. Then $v \le v^{*}$.

Proof

Let $x_{0} \in X$. If $v(x_{0}) = -\infty$, then $v(x_{0}) \le v^{*}(x_{0})$. Consider the case $v(x_{0}) > -\infty .$ Let $\epsilon > 0$. Let $\{\epsilon _{t}\}_{t=0}^{\infty } \subset (0,\infty )$ be such that $\sum _{t=0}^{\infty } \beta ^{t} \epsilon _{t} \le \epsilon .$ Since $v = B v$, for any $t \in {{\mathbb {Z}}}_{+}$ and $x_{t} \in X$, there exists $x_{t+1} \in \Gamma (x_{t})$ such that

$$\begin{aligned} v(x_{t}) \le u(x_{t}, x_{t+1}) + \beta v(x_{t+1}) + \epsilon _{t}. \end{aligned}$$

(6.1)

We pick $x_{1} \in \Gamma (x_{0}), x_{2} \in \Gamma (x_{1}), \ldots$ so that (6.1) holds for all $t \in {{\mathbb {Z}}}_{+}$. Then $\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})$. By repeated application of (6.1), we have

$$\begin{aligned} v(x_{0})&\le u(x_{0}, x_{1}) + \beta v(x_{1}) + \epsilon _{0} \end{aligned}$$

(6.2)

$$\begin{aligned}&\le u(x_{0}, x_{1}) + \beta [u(x_{1}, x_{2}) + \beta v(x_{2}) + \epsilon _{1}] + \epsilon _{0} \end{aligned}$$

(6.3)

$$\begin{aligned}&\quad \vdots \end{aligned}$$

(6.4)

$$\begin{aligned}&\le \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{T} v(x_{T}) + \epsilon , \quad {\forall }T \in {{\mathbb {N}}}. \end{aligned}$$

(6.5)

Since $v(x_{0}) > -\infty$, we have $\beta ^{T} v(x_{T}) > -\infty$ for all $T \in {{\mathbb {N}}}$. It follows that

$$\begin{aligned} v(x_{0}) - \epsilon - \beta ^{T} v(x_{T}) \le \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}). \end{aligned}$$

(6.6)

Applying ${\underline{\lim }}_{T \uparrow \infty }$ to both sides and recalling (2.5) and (2.6), we have

$$\begin{aligned} v(x_{0}) - \epsilon - {\overline{\lim }}_{T \uparrow \infty } \beta ^{T} v(x_{T}) \le {\underline{\lim }}_{T \uparrow \infty } \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) \le S(\{x_{t}\}_{t=0}^{\infty }) \le v^{*}(x_{0}). \end{aligned}$$

(6.7)

By (2.17), we have $v(x_{0}) - \epsilon \le v^{*}(x_{0})$. Since this is true for any $\epsilon > 0$, we have $v(x_{0}) \le v^{*}(x_{0})$. Since $x_{0}$ is arbitrary, we obtain $v \le v^{*}$. $\square$

For any $v \in V$, define $v_{1} = B v$; for each $n \in {{\mathbb {N}}}$, provided that $v_{n} \in V$, define $v_{n+1} = B v_{n}$. The following remark follows from (2.11).

Remark 6.1

Let $v, w \in V$ satisfy $v \le w$ and $B w \le w$. Then for all $n \in {{\mathbb {N}}}$, we have $v_{n} \le w$ and thus $v_{n} \in V$.

Lemma 6.2

Let $\overline{v} \in V$ satisfy (2.15). Let $v \in V$ satisfy $v \le \overline{v}$. Then for any $T \in {{\mathbb {N}}}$, we have $v_{T} \in V$ and

$$\begin{aligned} {\forall }x_{0} \in X, \quad v_{T}(x_{0}) = \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})} \left\{ \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{T} v(x_{T}) \right\} . \end{aligned}$$

(6.8)

Proof

Note from (2.15) and Remark 6.1 with $w = \overline{v}$ that $v_{n} \in V$ for all $n \in {{\mathbb {N}}}$. For any $x_{0} \in X$, we have

$$\begin{aligned} v_{1}(x_{0})&= \sup _{x_{1} \in \Gamma (x_{0})} \{u(x_{0}, x_{1}) + \beta v(x_{1})\} \end{aligned}$$

(6.9)

$$\begin{aligned}&= \sup _{x_{1} \in \Gamma (x_{0})} \sup _{\{x_{t}\}_{t=2}^{\infty } \in \Pi (x_{1})} \{u(x_{0}, x_{1}) + \beta v(x_{1})\} \end{aligned}$$

(6.10)

$$\begin{aligned}&= \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})} \{u(x_{0}, x_{1}) + \beta v(x_{1})\}, \end{aligned}$$

(6.11)

where (6.10) holds because $\{u(x_{0}, x_{1}) + \beta v(x_{1})\}$ is independent of $\{x_{t}\}_{t=2}^{\infty }$,¹⁷ and (6.11) follows by combining the two suprema (see Kamihigashi 2008, Lemma 1). It follows that (6.8) holds for $T = 1$.

Now assume (6.8) for $T = n \in {{\mathbb {N}}}$. For any $x_{0} \in X$, we have

$$\begin{aligned} v_{n+1}(x_{0})&= \sup _{x_{1} \in \Gamma (x_{0})} \{u(x_{0}, x_{1}) + \beta v_{n}(x_{1})\} \end{aligned}$$

(6.12)

$$\begin{aligned}&= \sup _{x_{1} \in \Gamma (x_{0})} \Biggl \{u(x_{0}, x_{1}) \nonumber \\&\quad \quad + \beta \sup _{\{x_{i+1}\}_{i=1}^{\infty } \in \Pi (x_{1})} \Bigl \{ \sum _{i=0}^{n-1} \beta ^{i} u(x_{i+1}, x_{i+2}) + \beta ^{n} v(x_{n+1}) \Bigr \} \Biggr \} \end{aligned}$$

(6.13)

$$\begin{aligned}&= \sup _{x_{1} \in \Gamma (x_{0})} \sup _{\{x_{i+1}\}_{i=1}^{\infty } \in \Pi (x_{1})} \left\{ \sum _{t=0}^{n} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{n+1} v(x_{n+1}) \right\} \end{aligned}$$

(6.14)

$$\begin{aligned}&= \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})} \left\{ \sum _{t=0}^{n} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{n+1} v(x_{n+1}) \right\} , \end{aligned}$$

(6.15)

where (6.13) uses (6.8) for $T=n$, (6.14) holds because $u(x_{0}, x_{1})$ is independent of $\{x_{i+1}\}_{i=1}^{\infty }$, and (6.15) follows by combining the two suprema (see Kamihigashi 2008, Lemma 1). It follows that (6.8) holds for $T = n+1$. By induction, (6.8) holds for all $T \in {{\mathbb {N}}}$. $\square$

Lemma 6.3

Let $\underline{v}, \overline{v} \in V$ satisfy (2.13)–(2.16). Then $\underline{v}^{*} \equiv \lim _{T \uparrow \infty } \underline{v}_{T} \ge v^{*}$.¹⁸

Proof

Note from (2.13)–(2.15), (2.11), and Remark 6.1 that $\{\underline{v}_{T}\}_{T=1}^{\infty }$ is an increasing sequence in $V$. Thus for any $x_{0} \in X$, we have

$$\begin{aligned} \underline{v}^{*}(x_{0})&= \sup _{T \in {{\mathbb {N}}}} \underline{v}_{T}(x_{0}) \end{aligned}$$

(6.16)

$$\begin{aligned}&= \sup _{T \in {{\mathbb {N}}}} \, \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})} \left\{ \sum _{t=0}^{T -1} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{T} \underline{v}(x_{T}) \right\} \end{aligned}$$

(6.17)

$$\begin{aligned}&= \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})} \, \sup _{T \in {{\mathbb {N}}}} \left\{ \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{T} \underline{v}(x_{T}) \right\} \end{aligned}$$

(6.18)

$$\begin{aligned}&\ge \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi ^{0}(x_{0})} L_{T \uparrow \infty } \left\{ \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) + \beta ^{T} \underline{v}(x_{T}) \right\} \end{aligned}$$

(6.19)

$$\begin{aligned}&\ge \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi ^{0}(x_{0})} \left\{ L_{T \uparrow \infty } \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) + {\underline{\lim }}_{T \uparrow \infty } \beta ^{T} \underline{v}(x_{T}) \right\} \end{aligned}$$

(6.20)

$$\begin{aligned}&\ge \sup _{\{x_{t}\}_{t=1}^{\infty } \in \Pi ^{0}(x_{0})} L_{T \uparrow \infty } \sum _{t=0}^{T-1} \beta ^{t} u(x_{t}, x_{t+1}) = v^{*}(x_{0}), \end{aligned}$$

(6.21)

where (6.17) uses Lemma 6.2, (6.18) follows by interchanging the two suprema (see Kamihigashi 2008, Lemma 1), (6.19) holds because $\Pi ^{0}(x_{0}) \subset \Pi (x_{0})$ (recall (2.8)) and $L_{T \uparrow \infty } a_{T} \le \sup _{T \in {{\mathbb {N}}}} a_{T}$ for any sequence $\{a_{T}\}$ in $[-\infty , \infty )$, (6.20) follows from the properties of ${\underline{\lim }}$ and ${\overline{\lim }}$,¹⁹ and the inequality in (6.21) uses (2.16). It follows that $\underline{v}^{*} \ge v^{*}$. $\square$

To complete the proof of Theorem 2.1, suppose that there exist $\underline{v}, \overline{v} \in V$ satisfying (2.13)–(2.17). The order interval $[\underline{v}, \overline{v}]$ is partially ordered by $\le$ (recall (2.10)). Given any $F \subset [\underline{v}, \overline{v}]$, we have $\sup F \in [\underline{v}, \overline{v}]$ because

$$\begin{aligned} {\forall }x \in X, \qquad (\sup F)(x) = \sup \{f(x) : f \in F\} \in [\underline{v}(x), \overline{v}(x)]. \end{aligned}$$

(6.22)

Since $B$ is a monotone operator, and since $B([\underline{v}, \overline{v}]) \subset [\underline{v}, \overline{v}]$ by (2.13)–(2.15) and (2.11), it follows that $B$ has a fixed point $v$ in $[\underline{v}, \overline{v}]$ by the Knaster–Tarski fixed point theorem (e.g., Aliprantis and Border 2006, p. 16). Since $\underline{v} \le v = B v$, we have $\underline{v}_{n} \le v$ for all $n \in {{\mathbb {N}}}$ by Remark 6.1; thus $\underline{v}^{*} \le v$.²⁰ Since $v \le v^{*}$ by Lemma 6.1 and $v^{*} \le \underline{v}^{*}$ by Lemma 6.3, it follows that $v \le v^{*} \le \underline{v}^{*} \le v$. Hence $v = \underline{v}^{*} = v^{*}$. Therefore, $v^{*}$ is a unique fixed point of $B$ in $[\underline{v}, \overline{v}]$; this establishes conclusions (a) and (b). Finally, conclusion (c) holds because $\underline{v}^{*} = v^{*}$.

1.2 Proof of Proposition 2.1

Let $\overline{v} \in V$ satisfy (2.17) and (2.18). Then by Lemma 6.1, any fixed point $v$ of $B$ with $v \le \overline{v}$ satisfies $v \le v^{*}$. Since $v^{*} \le \overline{v}$ and $v^{*}$ is a fixed point of $B$ by Lemma 2.1, it follows that $v^{*}$ is the largest fixed point of $B$ in $[-\infty , \overline{v}]$.

1.3 Proof of Therem 2.2

Let $\underline{v} \in V$ satisfy (2.14), (2.16), and (2.19). Since $B v^{*} = v^{*}$ by Lemma 2.1, (2.13)–(2.15) hold with $\overline{v} = v^{*}$. Hence, by Lemma 6.3, $\underline{v}^{*} \ge v^{*}$. Since $\underline{v} \le v^{*}$, we also have $\underline{v}^{*} \le v^{*}$. Thus $\underline{v}^{*} = v^{*}$. We have verified the second conclusion. If $B$ has a fixed point $v \in [\underline{v}, \infty ]$, then $v^{*} = \underline{v}^{*} \le v$; thus the first conclusion also holds.