Abstract
We establish some elementary results on solutions to the Bellman equation without introducing any topological assumption. Under a small number of conditions, we show that the Bellman equation has a unique solution in a certain set, that this solution is the value function, and that the value function can be computed by value iteration with an appropriate initial condition. In addition, we show that the value function can be computed by the same procedure under alternative conditions. We apply our results to two optimal growth models: one with a discontinuous production function and the other with “roughly increasing” returns.
Similar content being viewed by others
Notes
Recall that the Theorem of the Maximum (e.g., Stokey and Lucas 1989, Theorem 3.6) shows that parametric continuity of the objective function only implies upper hemicontinuity of the solution correspondence, which means that the correspondence is in general not continuous. See Dutta and Mitra (1989a); Dutta and Mitra (1989b) for related discussions.
See Kamihigashi and Furusawa (2010, Figure 5) for a simple example.
We follow the convention that \(\sup \emptyset = -\infty \).
Given \(r \in \overline{{{\mathbb {R}}}}\), the notation \(r\) is understood as the extended real number \(r\) or the function identically equal to \(r\) depending on the context.
In this paper, “increasing” means “nondecreasing,” “decreasing” means “nonincreasing,” “positive” means “nonnegative,” and “negative” means “nonpositive.”
This means that any fixed point \(v\) of \(B\) in \([-\infty , \overline{v}]\) satisfies \(v \le v^{*}\). A similar remark applies to the first conclusion of Theorem 2.2.
See Strauch (1966, p. 880) for a related example of an undiscounted stochastic model.
By using a different state variable, it is possible to construct a continuous return function, but the feasibility correspondence cannot be made continuous.
To derive these values of \(\underline{\gamma }\) and \(\underline{\eta }\), for example, suppose that \(\underline{v}\) takes the form given by (4.6), and solve the maximization problem \(\max _{y \in \underline{\Gamma }(x)} \{\ln (\underline{\theta } x - y) + \beta [\underline{\gamma } \ln y + \underline{\eta }]\}\). Substitute the solution into the objective function and find the values of \(\underline{\gamma }\) and \(\underline{\eta }\) such that the maximized objective function always equals \(\underline{\gamma } \ln x + \underline{\eta }\).
The latter approach is used by Stokey and Lucas (1989, Section 4.4).
Note from Le Van and Morhaim (2002) that continuity of \(v^{*}\) is not necessarily immediate. One can verify the continuity of \(v^{*}\) here by following Le Van and Morhaim (2002, Theorem 3(iii)), even though their assumptions rule out increasing returns. One can also verify it more directly using the interiority of optimal paths. However, given that \(v^{*}\) is known to be upper semicontinuous, the following argument seems to be more efficient.
Let \(i \in {{\mathbb {N}}}\). Then at state \((i,i)\), we have \(v^{*}((i,i)) = -\beta ^{-i}/(1-\beta )\). Note that \(v^{*}((i,i-k)) = \beta ^{k} v^{*}((i,i))\) for \(k = 1,\ldots ,i\); thus \(v^{*}((i,j)) = -\beta ^{i-j} v^{*}((i,i)) = -\beta ^{-j}/(1-\beta )\). It remains to compute \(v((0,0))\). If \(x_{t} = (0,0)\) for all \(t \in {{\mathbb {Z}}}_{+}\), then \(S(\{x_{t}\}_{t=0}^{\infty }) = -\alpha /(1-\beta )\). If \(x_{1} = (i,0)\) with \(i > 0\), then \(S(\{x_{t}\}_{t=0}^{\infty }) = \beta v^{*}((i,0)) = -\beta /(1-\beta ) < -\alpha /(1-\beta )\). Hence, it is never optimal to leave state \((0,0)\), so that \(v^{*}((0,0)) = -\alpha /(1-\beta )\).
To see this, define \(\overline{v}_{0} = \overline{v} = 0\). Then \(\overline{v}_{0}((0,0)) = 0\). Let \(n \in {{\mathbb {Z}}}_{+}\). With \(\overline{v}_{n}\) given by (4.13), we have \(\overline{v}_{n+1}((0,0)) = \beta \sup _{i \in X} \overline{v}_{n}((i,0)) = 0\) since \(\overline{v}_{n}((i, 0)) = 0\) for all \(i \ge n\). By induction, \(\overline{v}_{n}((0,0)) = 0\) for all \(n \in {{\mathbb {N}}}\).
This step uses the assumption that \(\Gamma \) is nonempty-valued.
Here \(\underline{v}_{T} = B^{T} \underline{v}\) for all \(T \in {{\mathbb {N}}}\) and \(\underline{v}^{*}(x) = \lim _{T \uparrow \infty } \underline{v}_{T}(x)\) for all \(x \in X\).
We have \({\underline{\lim }}(a_{t} + b_{t}) \ge {\underline{\lim }}a_{t} + {\underline{\lim }}b_{t}\) and \({\overline{\lim }}(a_{t} + b_{t}) \ge {\overline{\lim }}a_{t} + {\underline{\lim }}b_{t}\) for any sequences \(\{a_{t}\}\) and \(\{b_{t}\}\) in \([-\infty , \infty )\) whenever both sides are well-defined (e.g., Michel 1990, p. 706).
See footnote 18 for the definitions of \(\underline{v}_{n}\) and \(\underline{v}^{*}\).
References
Aliprantis, C.D., Border, K.C.: Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd edn. Springer, Berlin (2006)
Algan, Y., Challe, E., Ragot, X.: Incomplete markets and the output-inflation tradeoff. Econ. Theory 46, 55–84 (2011)
Alvarez, F., Stokey, N.L.: Dynamic programming with homogenous functions. J. Econ. Theory 82, 167–189 (1998)
Azariadis, C., Drazen, A.: Threshold externalities in economic development. Q. J. Econ. 105, 501–526 (1990)
Amir, R., Mirman, L.J., Perkins, W.R.: One-sector nonclassical optimal growth: optimality conditions and comparative dynamics. Int. Econ. Rev. 32, 625–644 (1991)
Bertsekas, D.P., Shreve, S.E.: Stochastic Optimal Control: The Discrete Time Case. Academic Press, New York (1978)
Bloch, F., Houy, N.: Optimal assignment of durable objects to successive agents. Econ. Theory 51, 13–33 (2012)
Durán, J.: On dynamic programming with unbounded returns. Econ. Theory 15, 339–352 (2000)
Dutta, P.K., Mitra, T.: On continuity of the utility function in intertemporal allocation models: an example. Int. Econ. Rev. 30, 527–536 (1989a)
Dutta, P.K., Mitra, T.: Maximum theorems for convex structures with an application to the theory of optimal intertemporal allocation. J. Math. Econ. 18, 77–86 (1989b)
Dutta, P.K., Radner, R.: Capital growth in a global warming model: will China and India sign a climate treaty? Econ. Theory 49, 411–443 (2012)
Goenka, A., Lin, L.: Infectious diseases and endogenous fluctuations. Econ. Theory 50, 125–149 (2012)
Herrera, H., Martinelli, C.: Oligarchy, democracy, and state capacity. Econ. Theory 52, 165–186 (2013)
Kamihigashi, T.: On the principle of optimality for nonstationary deterministic dynamic programming. Int. J. Econ. Theory 4, 519–525 (2008)
Kamihigashi, T.: An order-theoretic approach to dynamic programming: an exposition. Econ. Theory Bull. (2013). doi:10.1007/s40505-013-0022-4
Kamihigashi, T., Furusawa, T.: Global dynamics in repeated games with additively separable payoffs. Rev. Econ. Dyn. 13, 899–918 (2010)
Kamihigashi, T., Roy, S.: Dynamic optimization with a nonsmooth, nonconvex technology: the case of a linear objective function. Econ. Theory 29, 325–340 (2006)
Kamihigashi, T., Roy, S.: A nonsmooth, nonconvex model of optimal growth. J. Econ. Theory 132, 435–460 (2007)
Karp, L., Zhang, J.: Taxes versus quantities for a stock pollutant with endogenous abatement costs and asymmetric information. Econ. Theory 49, 371–409 (2012)
Le Van, C., Morhaim, L.: Optimal growth models with bounded or unbounded returns: a unifying approach. J. Econ. Theory 105, 158–187 (2002)
Llanes, G., Trento, S.: Paten policy, patent pools, and the accumulation of claims in sequential innovation. Econ. Theory 50, 703–725 (2012)
Martins-da-Rocha, V.F., Vailakis, Y.: Existence and uniqueness of a fixed point for local contractions. Econometrica 78, 1127–1141 (2010)
Matkowski, J., Nowak, A.S.: On discounted dynamic programming with unbounded returns. Econ. Theory 46, 455–474 (2011)
Michel, P.: Some clarifications on the transversality condition. Econometrica 58, 705–723 (1990)
Muradian, R.: Ecological thresholds: a survey. Ecol. Econ. 38, 7–24 (2001)
Nævdal, R.: Optimal regulation of natural resources in the presence of irreversible threshold effects. Natur. Resour. Model. 16, 305–333 (2003)
Reis, C.: Taxation without commitment. Econ. Theory 52, 565–588 (2013)
Rinćon-Zapatero, J.P., Rodríguez-Palmero, C.: Existence and uniqueness of solutions to the Bellman equation in the unbounded case. Econometrica 71, 1519–1555 (2003)
Rinćon-Zapatero, J.P., Rodríguez-Palmero, C.: Recursive utility with unbounded aggregators. Econ. Theory 33, 381–391 (2007)
Rinćon-Zapatero, J.P., Rodríguez-Palmero, C.: Corrigendum to “Existence and uniqueness of solutions to the Bellman equation in the unbounded case” Econometrica, Vol. 71, No. 5 (September, 2003), 1519–1555. Econometrica 77, 317–318 (2009)
Romer, P.: Increasing returns and long-run growth. J. Polit. Econ. 94, 1002–1037 (1986)
Roy, S., Zilcha, I.: Stochastic growth with short-run prediction of shocks. Econ. Theory 51, 539–580 (2012)
Schaar, M.V.D., Xu, J., Zame, W.: Efficient online exchange via fiat money. Econ. Theory 54, 211–248 (2013)
Stokey, N., Lucas, R.E., Jr.: Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge, MA (1989)
Strauch, R.E.: Negative dynamic programming. Ann. Math. Stat. 37, 871–890 (1966)
Author information
Authors and Affiliations
Corresponding author
Additional information
I would like to thank participants at the 11th SAET Conferance in Faro, 2011, the Workshop in Honor of Cuong Le Van in Exeter, 2011, a seminar at the Paris School of Economics in 2012, and the 21st European Workshop on General Equilibrium Theory in Exeter, 2012—especially, Larry Blume, Myrna Wooders, Felix Kübler, Juan Pablo Rincón-Zapatero, V. Filipe Martins-da-Rocha, Yiannis Vailakis, Cuong Le Van, Kevin Reffett—for helpful comments and discussions. This paper has benefited from comments and suggestions by three anonymous referees and an associate editor. Financial support from the Japan Society for the Promotion of Science is gratefully acknowledged.
Appendix: Proofs
Appendix: Proofs
1.1 Proof of Theorem 2.1
The proof consists of three lemmas and a concluding argument. The proof of the first lemma slightly generalizes an argument of Stokey and Lucas (1989, Theorem 4.3). The second lemma essentially shows that \(B^{T} v\) with \(T \in {{\mathbb {N}}}\) and \(v \in V\) is the value function of the \(T\)-period problem with the value of the terminal stock \(x_{T}\) given by \(v(x_{T})\). This result extends the classical idea of Bertsekas and Shreve (1978, Section 3.2) to our setting. The last lemma is less trivial than the first two. The concluding argument applies the Knaster–Tarski fixed point theorem and combines the first and last lemmas.
Lemma 6.1
Let \(\overline{v} \in V\) satisfy (2.17). Let \(v \in V\) be a fixed point of \(B\) with \(v \le \overline{v}\). Then \(v \le v^{*}\).
Proof
Let \(x_{0} \in X\). If \(v(x_{0}) = -\infty \), then \(v(x_{0}) \le v^{*}(x_{0})\). Consider the case \(v(x_{0}) > -\infty .\) Let \(\epsilon > 0\). Let \(\{\epsilon _{t}\}_{t=0}^{\infty } \subset (0,\infty )\) be such that \(\sum _{t=0}^{\infty } \beta ^{t} \epsilon _{t} \le \epsilon .\) Since \(v = B v\), for any \(t \in {{\mathbb {Z}}}_{+}\) and \(x_{t} \in X\), there exists \(x_{t+1} \in \Gamma (x_{t})\) such that
We pick \(x_{1} \in \Gamma (x_{0}), x_{2} \in \Gamma (x_{1}), \ldots \) so that (6.1) holds for all \(t \in {{\mathbb {Z}}}_{+}\). Then \(\{x_{t}\}_{t=1}^{\infty } \in \Pi (x_{0})\). By repeated application of (6.1), we have
Since \(v(x_{0}) > -\infty \), we have \(\beta ^{T} v(x_{T}) > -\infty \) for all \(T \in {{\mathbb {N}}}\). It follows that
Applying \({\underline{\lim }}_{T \uparrow \infty }\) to both sides and recalling (2.5) and (2.6), we have
By (2.17), we have \(v(x_{0}) - \epsilon \le v^{*}(x_{0})\). Since this is true for any \(\epsilon > 0\), we have \(v(x_{0}) \le v^{*}(x_{0})\). Since \(x_{0}\) is arbitrary, we obtain \(v \le v^{*}\). \(\square \)
For any \(v \in V\), define \(v_{1} = B v\); for each \(n \in {{\mathbb {N}}}\), provided that \(v_{n} \in V\), define \(v_{n+1} = B v_{n}\). The following remark follows from (2.11).
Remark 6.1
Let \(v, w \in V\) satisfy \(v \le w\) and \(B w \le w\). Then for all \(n \in {{\mathbb {N}}}\), we have \(v_{n} \le w\) and thus \(v_{n} \in V\).
Lemma 6.2
Let \(\overline{v} \in V\) satisfy (2.15). Let \(v \in V\) satisfy \(v \le \overline{v}\). Then for any \(T \in {{\mathbb {N}}}\), we have \(v_{T} \in V\) and
Proof
Note from (2.15) and Remark 6.1 with \(w = \overline{v}\) that \(v_{n} \in V\) for all \(n \in {{\mathbb {N}}}\). For any \(x_{0} \in X\), we have
where (6.10) holds because \(\{u(x_{0}, x_{1}) + \beta v(x_{1})\}\) is independent of \(\{x_{t}\}_{t=2}^{\infty }\),Footnote 17 and (6.11) follows by combining the two suprema (see Kamihigashi 2008, Lemma 1). It follows that (6.8) holds for \(T = 1\).
Now assume (6.8) for \(T = n \in {{\mathbb {N}}}\). For any \(x_{0} \in X\), we have
where (6.13) uses (6.8) for \(T=n\), (6.14) holds because \(u(x_{0}, x_{1})\) is independent of \(\{x_{i+1}\}_{i=1}^{\infty }\), and (6.15) follows by combining the two suprema (see Kamihigashi 2008, Lemma 1). It follows that (6.8) holds for \(T = n+1\). By induction, (6.8) holds for all \(T \in {{\mathbb {N}}}\). \(\square \)
Lemma 6.3
Let \(\underline{v}, \overline{v} \in V\) satisfy (2.13)–(2.16). Then \(\underline{v}^{*} \equiv \lim _{T \uparrow \infty } \underline{v}_{T} \ge v^{*}\).Footnote 18
Proof
Note from (2.13)–(2.15), (2.11), and Remark 6.1 that \(\{\underline{v}_{T}\}_{T=1}^{\infty }\) is an increasing sequence in \(V\). Thus for any \(x_{0} \in X\), we have
where (6.17) uses Lemma 6.2, (6.18) follows by interchanging the two suprema (see Kamihigashi 2008, Lemma 1), (6.19) holds because \(\Pi ^{0}(x_{0}) \subset \Pi (x_{0})\) (recall (2.8)) and \(L_{T \uparrow \infty } a_{T} \le \sup _{T \in {{\mathbb {N}}}} a_{T}\) for any sequence \(\{a_{T}\}\) in \([-\infty , \infty )\), (6.20) follows from the properties of \({\underline{\lim }}\) and \({\overline{\lim }}\),Footnote 19 and the inequality in (6.21) uses (2.16). It follows that \(\underline{v}^{*} \ge v^{*}\). \(\square \)
To complete the proof of Theorem 2.1, suppose that there exist \(\underline{v}, \overline{v} \in V\) satisfying (2.13)–(2.17). The order interval \([\underline{v}, \overline{v}]\) is partially ordered by \(\le \) (recall (2.10)). Given any \(F \subset [\underline{v}, \overline{v}]\), we have \(\sup F \in [\underline{v}, \overline{v}]\) because
Since \(B\) is a monotone operator, and since \(B([\underline{v}, \overline{v}]) \subset [\underline{v}, \overline{v}]\) by (2.13)–(2.15) and (2.11), it follows that \(B\) has a fixed point \(v\) in \([\underline{v}, \overline{v}]\) by the Knaster–Tarski fixed point theorem (e.g., Aliprantis and Border 2006, p. 16). Since \(\underline{v} \le v = B v\), we have \(\underline{v}_{n} \le v\) for all \(n \in {{\mathbb {N}}}\) by Remark 6.1; thus \(\underline{v}^{*} \le v\).Footnote 20 Since \(v \le v^{*}\) by Lemma 6.1 and \(v^{*} \le \underline{v}^{*}\) by Lemma 6.3, it follows that \(v \le v^{*} \le \underline{v}^{*} \le v\). Hence \(v = \underline{v}^{*} = v^{*}\). Therefore, \(v^{*}\) is a unique fixed point of \(B\) in \([\underline{v}, \overline{v}]\); this establishes conclusions (a) and (b). Finally, conclusion (c) holds because \(\underline{v}^{*} = v^{*}\).
1.2 Proof of Proposition 2.1
Let \(\overline{v} \in V\) satisfy (2.17) and (2.18). Then by Lemma 6.1, any fixed point \(v\) of \(B\) with \(v \le \overline{v}\) satisfies \(v \le v^{*}\). Since \(v^{*} \le \overline{v}\) and \(v^{*}\) is a fixed point of \(B\) by Lemma 2.1, it follows that \(v^{*}\) is the largest fixed point of \(B\) in \([-\infty , \overline{v}]\).
1.3 Proof of Therem 2.2
Let \(\underline{v} \in V\) satisfy (2.14), (2.16), and (2.19). Since \(B v^{*} = v^{*}\) by Lemma 2.1, (2.13)–(2.15) hold with \(\overline{v} = v^{*}\). Hence, by Lemma 6.3, \(\underline{v}^{*} \ge v^{*}\). Since \(\underline{v} \le v^{*}\), we also have \(\underline{v}^{*} \le v^{*}\). Thus \(\underline{v}^{*} = v^{*}\). We have verified the second conclusion. If \(B\) has a fixed point \(v \in [\underline{v}, \infty ]\), then \(v^{*} = \underline{v}^{*} \le v\); thus the first conclusion also holds.
Rights and permissions
About this article
Cite this article
Kamihigashi, T. Elementary results on solutions to the bellman equation of dynamic programming: existence, uniqueness, and convergence. Econ Theory 56, 251–273 (2014). https://doi.org/10.1007/s00199-013-0789-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00199-013-0789-4