Dynamic programming with Hermite approximation

Abstract

Numerical dynamic programming algorithms typically use Lagrange data to approximate value functions over continuous states. Hermite data can be easily obtained from solving the Bellman equation and used to approximate the value functions. We illustrate the use of Hermite data with one-, three-, and six-dimensional examples. We find that Hermite approximation improves the accuracy in value function iteration (VFI) by one to three digits using little extra computing time. Moreover, VFI with Hermite approximation is significantly faster than VFI with Lagrange approximation for the same accuracy, and this advantage increases with the dimension of the continuous states.

Appendices

Appendix 1: One-dimensional Chebyshev polynomial approximation

One-dimensional Chebyshev basis polynomials on $[-1,1]$ are defined as $\mathcal {T}_{j}(x)=\cos (j\cos ^{-1}(x)),$ while general Chebyshev polynomials on $[x_{\min },x_{\max }]$ are defined as $\mathcal {T}_{j}\left( Z(x)\right)$ where

$$\begin{aligned} Z(x)=(2x-x_{\min }-x_{\max })/(x_{\max }-x_{\min }) \end{aligned}$$

for $j=0,1,2,\ldots$. These polynomials are orthogonal under the weighted inner product: $\left<f,g\right>=\int _{x_{\min }}^{x_{\max }}f(x)g(x)w(x)dx$ with the weighting function $w(x)=\left( 1-\left( Z(x)\right) ^{2}\right) ^{-1/2}$. The degree $n$ Chebyshev polynomial approximation for $V(x)$ on $[x_{\min },x_{\max }]$ is

$$\begin{aligned} \hat{V}(x;\mathbf{b})=\sum _{j=0}^{n}b_{j}\mathcal {T}_{j}\left( Z(x)\right) , \end{aligned}$$

(15)

where $\mathbf{b}=\left\{ b_{j}\right\}$ are the Chebyshev coefficients.

If we choose the Chebyshev nodes on $[x_{\min },x_{\max }]$: $x_{i}=(z_{i}+1)(x_{\max }-x_{\min })/2+x_{\min }$ with $z_{i}=-\cos \left( (2i-1)\pi /(2m)\right)$ for $i=1,\ldots ,m$, and Lagrange data $\{(x_{i},v_{i}):$ $i=1,\ldots ,m\}$ are given (where $v_{i}=V(x_{i})$), then the coefficients $\left\{ b_{j}\right\}$ in (15) can be easily computed by the following formula:

$$\begin{aligned} b_{0}&= \frac{1}{m}\sum _{i=1}^{m}v_{i},\nonumber \\ b_{j}&= \frac{2}{m}\sum _{i=1}^{m}v_{i}\mathcal {T}_{j}(z_{i}) \qquad j=1,\ldots ,n. \end{aligned}$$

(16)

The method is called the Chebyshev regression algorithm in Judd (1998).

When the number of the Chebyshev nodes is equal to the number of the Chebyshev coefficients (i.e., $m=n+1$), the approximation (15) with the coefficients given by (16) becomes the Chebyshev polynomial interpolation (which is a Lagrange interpolation), as $\hat{V}(x_{i};\mathbf{b})=v_{i}$, for $i=1,\ldots ,m$.

It is often more stable to use the expanded Chebyshev polynomial interpolation (Cai et al. 2010), as the above standard Chebyshev polynomial interpolation gives poor approximation in the neighborhood of the end points of the approximation interval. That is, we use the following formula to approximate $V(x)$:

$$\begin{aligned} \hat{V}(x;\mathbf{b})=\sum _{j=0}^{n}b_{j}\mathcal {T}_{j}\left( \frac{2x-\widetilde{x}_{\min }-\widetilde{x}_{\max }}{\widetilde{x}_{\max }-\widetilde{x}_{\min }}\right) , \end{aligned}$$

(17)

where $\widetilde{x}_{\min }=x_{\min }-\delta$ and $\widetilde{x}_{\max }=x_{\max }+\delta$ with $\delta =(z_{1}+1)(x_{\min }-x_{\max })/(2z_{1})$. Moreover, if we choose the expanded Chebyshev nodes on $[x_{\min },x_{\max }]$: $x_{i}=(z_{i}+1)(\tilde{x}_{\max }-\tilde{x}_{\min })/2+\tilde{x}_{\min }$, then the coefficients $\left\{ b_{j}\right\}$ can also be calculated easily by the expanded Chebyshev regression algorithm (Cai et al. 2010), which is similar to (16).

Appendix 2: Multidimensional complete Chebyshev polynomial approximation

In a $d$-dimensional approximation problem, let the approximation domain of the value function be

$$\begin{aligned} \left\{ \mathbf {x}=(x_{1},\ldots ,x_{d}):\, x_{\min ,j}\le x_{j}\le x_{\max ,j},\, j=1,\ldots d\right\} , \end{aligned}$$

for some real numbers $x_{\min ,j}$ and $x_{\max ,j}$ with $x_{\max ,j}>x_{\min ,j}$ for $j=1,\ldots ,d$. Let $\mathbf {x}_{\min }=(x_{\min ,1},\ldots ,x_{\min ,d})$ and $\mathbf {x}_{\max }=(x_{\max ,1},\ldots ,x_{\max ,d})$. Then we denote $[\mathbf {x}_{\min },\mathbf {x}_{\max }]$ as the approximation domain. Let $\alpha =(\alpha _{1},\ldots ,\alpha _{d})$ be a vector of nonnegative integers. Let $\mathcal {T}_{\alpha }(\mathbf {z})$ denote the product $\prod _{1\le j\le d}\mathcal {T}_{\alpha _{j}}(z_{j})$ for $\mathbf {z}=(z_{1},\ldots ,z_{d})\in [-1,1]^{d}$. Let

$$\begin{aligned} \mathbf {Z}(\mathbf {x})=\left( \frac{2x_{1}-x_{\min ,1}-x_{\max ,1}}{x_{\max ,1}-x_{\min ,1}},\ldots ,\frac{2x_{d}-x_{\min ,d}-x_{\max ,d}}{x_{\max ,d}-x_{\min ,d}}\right) \end{aligned}$$

for any $\mathbf {x}=(x_{1},\ldots ,x_{d})\in [\mathbf {x}_{\min },\mathbf {x}_{\max }]$.

Using these notations, the degree $n$ complete Chebyshev approximation for $V(\mathbf {x})$ is

$$\begin{aligned} \hat{V}_{n}(\mathbf {x};\mathbf {b})=\sum _{0\le |\alpha |\le n}b_{\alpha }\mathcal {T}_{\alpha }\left( \mathbf {Z}(\mathbf {x})\right) , \end{aligned}$$

(18)

where $|\alpha |=\sum _{j=1}^{d}\alpha _{j}$. The number of terms with $0\le |\alpha |=\sum _{j=1}^{d}\alpha _{i}\le n$ is $\left( {\begin{array}{c}n+d\\ d\end{array}}\right)$ for the degree $n$ complete Chebyshev approximation in ${\mathbb {R}}^{d}$.

Appendix 3: Multivariate numerical integration

From the formula (11), we know that $V_{t+1}(R_{f}B+\mathbf {R}^{\top }\mathbf {S})$ in the objective function of (9) can be rewritten as $G(\varsigma )$ for some function $G$ where $\varsigma$ is the $D$-dimensional vector of the corresponding standard normal random variables with a correlation matrix $\Sigma$. Since $\Sigma$ must be a positive semi-definite matrix from the covariance property, we can have its Cholesky factorization, $\Sigma =LL^{\top }$, where $L$ is a $D\times D$ lower triangular matrix. Therefore,

$$\begin{aligned}&{\mathbb {E}}\left\{ V_{t+1}(R_{f}B+\mathbf {R}^{\top }\mathbf {S})\right\} ={\mathbb {E}}\{G(\varsigma )\}\\&\quad = \left( (2\pi )^{D}{\det }(\Sigma )\right) ^{-1/2}\int _{{\mathbb {R}}^{D}}G(\mathbf {q})e^{-\mathbf {q}^{\top }\Sigma ^{-1}\mathbf {q}/2}d\mathbf {q}\\&\quad = \left( (2\pi )^{D}{\det }(L)^{2}\right) ^{-1/2}\int _{{\mathbb {R}}^{D}}G\left( \sqrt{2}L\mathbf {q}\right) e^{-\mathbf {q}^{\top }\mathbf {q}}\cdot 2^{D/2}{\det }(L)d\mathbf {q}\\&\quad \doteq \pi ^{-{\frac{D}{2}}}\sum _{i_{1}=1}^{K}\cdots \sum _{i_{D}=1}^{K}w_{i_{1}}\cdots w_{i_{D}}G\left( \left( \sum _{j=1}^{D}L_{D,j}q_{i_{j}}\right) \sqrt{2}L_{1,1}q_{i_{1}},\right. \\&\quad \left. \sqrt{2}(L_{2,1}q_{i_{1}}+L_{2,2}q_{i_{2}}),\cdots ,\sqrt{2}\left( \sum _{j=1}^{D}L_{D,j}q_{i_{j}}\right) \right) , \end{aligned}$$

where $\{w_{i}:\,1\le i\le K\}$ and $\{q_{i}:\,1\le i\le K\}$ are the Gauss-Hermite quadrature weights and nodes over $(-\infty ,\infty )$, $L_{i,j}$ is the $(i,j)$-element of $L$, and $\det (\cdot )$ means the matrix determinant operator.

Appendix 4: Tree method

Assume that $\theta _{t}$ is a Markov chain with possible states, $\vartheta _{1},\ldots ,\vartheta _{M}$, and the probability of going from state $\vartheta _{i}$ to state $\vartheta _{j}$ in one step is

$$\begin{aligned} {\mathbb {P}}(\theta _{t+1}=\vartheta _{j}\,|\,\theta _{t}=\vartheta _{i})=p_{i,j}. \end{aligned}$$

Thus, from time 0 to time $t$, there are $M^{t}$ paths of $\theta _{t}$ for a given initial state, for $0\le t\le T$. These paths comprise a tree structure. We can then compute the exact solutions of the dynamic stochastic problems with the tree structure by applying an optimization solver to solve a large-scale optimal control problem directly, called the tree method here.

In the tree method for the stochastic optimal growth problem (14), the goal is to find the optimal decisions $\left\{ \left( \mathbf {I}_{t,i},\mathbf {c}_{t,i},\varvec{\ell }_{t,i}\right) :\,1\le i\le M^{t},\,0\le t<T\right\}$ to maximize the expected total utility:

$$\begin{aligned} \max _{\mathbf {I},\mathbf {c},\varvec{\ell }}&\&\sum _{t=0}^{T-1}\beta ^{t}\overset{M^{t}}{\underset{i=1}{\sum }}(P_{t,i}u(\mathbf {c}_{t,i},\varvec{\ell }_{t,i}))+\nonumber \\&\beta ^{T}\overset{M^{T-1}}{\underset{i=1}{\sum }}P_{T-1,i}\sum _{j=1}^{M}p_{\mathrm{mod}(i-1,M)+1,j}V_{T}(\mathbf {k}_{T,i},\vartheta _{j}), \end{aligned}$$

(19)

subject to the corresponding constraints of (14) for all paths, where $\hbox {mod}(i-1,M)$ is the remainder of division of $(i-1)$ by $M$, $P_{t,i}$ is the probability of the $i$-th path from time 0 to time $t$ with the following recursive formula:

$$\begin{aligned} P_{t+1,(i-1)M+j}=P_{t,i}\cdot p_{\mathrm{mod}(i-1,M)+1,j}, \end{aligned}$$

for $j=1,\ldots ,M$, $i=1,\ldots ,M^{t}$, $t=1,\ldots ,T-2$, where $P_{0,1}=1$ and $P_{1,j}={\mathbb {P}}(\theta _{1}=\vartheta _{j}\,|\,\theta _{0})$ for a given $\theta _{0}$.

This approach is practical for only small values of $M$ and $T$. In our examples, we set $M=2$ and $T=5$, implying that there are only 32 possible paths, and nonlinear optimization solvers will produce results with as much precision as possible in a double precision environment. We will treat those solutions as the “true” solution which can be used for the error analysis of numerical DP algorithms.