Solving Stochastic Dynamic Programming Problems: A Mixed Complementarity Approach

Abstract

We present a mixed complementarity problem (MCP) formulation of continuous state dynamic programming problems (DP-MCP). We write the solution to projection methods in value function iteration (VFI) as a joint set of optimality conditions that characterize maximization of the Bellman equation; and approximation of the value function. The MCP approach replaces the iterative component of projection based VFI with a one-shot solution to a square system of complementary conditions. We provide three numerical examples to illustrate our approach.

Appendices

Appendix

1.1 A. Value Function Iteration Using Collocation

Value function iteration is already a workhorse of numerous studies and textbooks aimed to make DP more accessible to numerical economic models (Howitt et al. 2002a; Aruoba and Fernández-Villaverde 2014; Aruoba et al. 2006; Manuelli and Sargent 2009; Sargent and Stachurski 2015). In this section of the “Appendix”, we describe the conventional collocation based VFI algorithm, in which the least-squares norm is used to obtain coefficients for the value function approximant.

The Bellman equation in a deterministic infinite-horizon setting takes the following form:

$$\begin{aligned} \begin{aligned} V(x) = \,\max _{a \in A} \,[ C(x, a) + \beta V(x{'}) ] \quad s.t. \,x{'} = h(x,a) \end{aligned} \end{aligned}$$

(7)

where x is the vector of state variables, A is the action space and C, the immediate contribution function. $\beta \in (0,1)$ is the discount factor. As a standard numerical algorithm for finding $V^{*}$, VFI is motivated by the contraction properties of the Bellman equation. VFI updates the value function via the Bellman operator, given the current estimate of the value function ($V^{n}(x)$); i.e.

$$\begin{aligned} V^{n+1}(x) = \,\max _{a \in A} \,[ C(x, a) + \beta V^{n}(x{'}) ] \quad s.t. \,x{'} = h(x,a) \end{aligned}$$

By the contraction mapping theorem, the solution to the iterative scheme converges to the true value function for any initial guess $V^{0}(\cdot )$ (Judd 1998).

Algorithm. Value Function Iteration Using Collocation for Infinite Horizon Problems

• 1. Set m collocation points in state space and a functional form for $V(x\,;\alpha )$;

  for $\forall i \le m$, choose approximation nodes $x_i \in X$;

  fix a tolerance parameter $\epsilon$;

  denote $V^{n}(x)$ to be value estimate output for iteration count n.

• 2. Initialize estimate of value function $V^{0}(x)$.

• 3. For $n \ge 1$: obtain parameters $\alpha ^{n-1}$s.t. $V(x_i\,;\alpha ^{n-1}) = V^{n-1}(x_i)$

  solve  $\min _{\alpha ^{n-1}} \,\bigg [\sum \limits _i\big ( V^{n-1}(x_i) - V(x_i\,;\alpha ^{n-1})\big )^{2} \bigg ]$

• 4. For $\forall i$, compute:

  $V^{n}(x_i) = \,\max _{a_i \in A} \,\bigg [ C(x_i, a_i) + \beta V(x_i{'}\,;\alpha ^{n-1})\bigg ] \,$ s.t. $x_i'=h(x_i,a_i)$.

• 5. If $\Vert V^{n} - V^{n-1}\Vert < \epsilon (1-\beta )/2\beta$, stop;

  else set $n = n+1$ and go to step 3.

The algorithm seeks an approximation to the value function such that the sum of the maximized contribution and the discounted carry-over value, based on the approximant, maximizes the total value function. The value iteration procedure solves for two objectives. The function approximation objective in step 3 computes coefficients of the value function approximant, by minimizing the sum of square deviations between the maximized Bellman value at each collocation point, and the function approximant. The next objective, (step 4) is to maximize the Bellman value via the control variable, given the coefficients of the approximated value function.

Conventional VFI is stable; yet without the use of techniques that help convergence (e.g. implementing Howard’s improvement, exploiting concavity or monotonicity of value function), the procedure is typically known to be slow. The slowness is particularly salient in economic growth models with a discount factor close to unity. Again, without the use of frontier numerical methods, the use of VFI is limited by the curse of dimensionality. Finding the equilibrium quickly becomes a daunting computational task as we increase the dimensions of the state space.

B. Gauss–Hermite Approximation Data

See Table 5

Table 5 Gauss–Hermite approximation data

C. Illustration of Smolyak Sparse Grid Method

We illustrate the method in a two dimensional state space. Consider the extrema of the Chebyshev polynomial function as grid points $\bigg \{-1, \frac{-1}{\sqrt{2}}, 0,\frac{1}{\sqrt{2}},1 \bigg \}$. It is not essential to use these points for the Smolyak method. Other unidimensional grid points can be used instead, per the collocation points used in this example. Using the common tensor product would produce 25 grid points $\big \{(-1,-1), (-1,\frac{-1}{\sqrt{2}}), \ldots \big \}$.

The Smolyak grid samples grid points in the following way. We first construct a sequence of nested sets, $S_i$, that exhaust the unidimensional grid previously defined; i.e.

$$\begin{aligned} i= & {} 1 \rightarrow S_1 = \big \{ 0\big \} \\ i= & {} 2 \rightarrow S_2 = \big \{ -1,0,1\big \}\\ i= & {} 3 \rightarrow S_3 =\bigg \{-1, \frac{-1}{\sqrt{2}}, 0,\frac{1}{\sqrt{2}},1 \bigg \} \end{aligned}$$

From all possible two-dimensional tensor products using elements of sets, $S_i$, the Smolyak method chooses grid points such that:

$$\begin{aligned} i_1 + i_2 \le d + \mu \end{aligned}$$

where $i_1$ and $i_2$ are the set numbers corresponding to each dimension; d is the number of dimensions of the state space; and $\mu$ is the approximation level parameter. With $\mu = 2$, the Smolyak sparse grid consists of 13 points.

The Smolyak polynomials that accompany the sparse grid is constructed in a similar way. Instead of unidimensional grid points, the sequence of sets now correspond to unidimensional Chebyshev basis functions. i.e.

$$\begin{aligned} i&= 1 \rightarrow S_1 = \big \{ 1\big \} \\ i&= 2 \rightarrow S_2 = \big \{ 1,x,2x^{2} - 1\big \}\\ i&= 3 \rightarrow S_3 =\bigg \{1,x,2x^{2} - 1,4x^{3}-3x,8x^{4}-8x^{2}+1\bigg \} . \end{aligned}$$

The GAMS code for the 4-sector model automates the construction of the Smolyak grid and polynomial, given the dimension and approximation level of the problem. Smolyak’s method was introduced to dynamic economic modeling in Krueger and Kubler (2004), and is currently used as a popular non-product approach to avoid the curse of dimensionality in numerical DP modeling (Fernández-Villaverde et al. 2015; Lemoine and Rudik 2017).

Appendix: GAMS Code

1.1 A. Stochastic Neoclassical Growth Model Data File: data.gms

1.2 B. Stochastic Neoclassical Growth Model: Hand-Coded MCP Formulation

1.3 C. Stochastic Neoclassical Growth Model: Automated EMP Formulation