A bias-corrected Least-Squares Monte Carlo for solving multi-period utility models

Abstract

The Least-Squares Monte Carlo (LSMC) method has gained popularity in recent years due to its ability to handle multi-dimensional stochastic control problems, including problems with state variables affected by control. However, when applied to the stochastic control problems in the multi-period expected utility models, such as finding optimal decisions in life-cycle expected utility models, the regression fit tends to contain errors which accumulate over time and typically blow up the numerical solution. In this paper we propose to transform the value function of the problems to improve the regression fit, and then using either the smearing estimate or smearing estimate with controlled heteroskedasticity to avoid the re-transformation bias in the estimates of the conditional expectations calculated in the LSMC algorithm. We also present and utilise recent improvements in the LSMC algorithms such as control randomisation with policy iteration to avoid accumulation of regression errors over time. Presented numerical examples demonstrate that transformation method leads to an accurate solution. In addition, in the forward simulation stage of the control randomisation algorithm, we propose a re-sampling of the state and control variables in their full domain at each time t and then simulating corresponding state variable at $t+1$, to improve the exploration of the state space that also appears to be critical to obtain a stable and accurate solution for the expected utility models.

Appendices

Appendix 1. Duan’s smearing estimate

The results in this section are based on [16]. Denote the non-transformed observations $Y_i, i=1,\ldots ,n$ and the transformed observations $\eta _i, i=1,\ldots ,n$ such that $\eta _i = g(Y_i)$, $Y_i = h(\eta _i)$, i.e. $h := g^{-1}$. Assume g (the transformation) and h (the re-transformation) are known monotonic and continuously differentiable functions, such as a CRRA utility function $h(x)=x^\gamma /\gamma$, $\ \gamma <0$. Consider the linear regression carried out on the transformed observations

$$\begin{aligned} \eta _i = \varvec{\beta }' \varvec{X}_i + \epsilon _i, \quad \epsilon _i \overset{i.i.d}{\sim } F(\cdot ), \quad {\mathbb {E}}[\epsilon _i]=0, \quad \mathrm {var}[\epsilon _i] = \sigma ^2, \end{aligned}$$

(29)

where $\varvec{\beta }$ is the vector of coefficients, $\varvec{X}_i$ is the vector of covariates and $\epsilon _i$ are the independent and identically distributed residuals from some zero mean distribution $F(\cdot )$ with finite variance. The error terms do not need to have a known distribution, although they are expected to have zero mean and constant variance. Now, if the re-transformation is applied to the prediction of the transformed variables we would get an incorrect estimate due to Jensen’s inequality, because ${\mathbb {E}}[Y] \le h({\mathbb {E}}[\varvec{\beta }' \varvec{X} + \epsilon ])$ if h is a concave function such as a utility function.

Smearing estimate attempts to approximate the non-transformed expectation

$$\begin{aligned} {\mathbb {E}}[Y] = {\mathbb {E}} \left[ h(\varvec{\beta }' \varvec{X} + \epsilon ) \right] = \int h \left( \varvec{\beta }' \varvec{X} + \epsilon \right) dF(\epsilon ) \end{aligned}$$

(30)

after estimating the regression coefficients $\widehat{\varvec{\beta }}$ using the empirical distribution function of the residuals ${\widehat{\epsilon }}_i = \eta _i-\widehat{\varvec{\beta }}'\varvec{X}_i$:

$$\begin{aligned} {\widehat{F}}_n(e) = \frac{1}{n} \sum ^n_{i=1} {\mathbb {I}} \left\{ {\widehat{\epsilon }}_i \le e \right\} , \end{aligned}$$

(31)

where ${\mathbb {I}} \{ \cdot \}$ is the indicator symbol that equals 1 if the statement in brackets $\{\cdot \}$ is true and 0 otherwise. The estimated expectation of Y can then be found as

$$\begin{aligned} \mathbb {{\widehat{E}}}[Y] = \int h \left( \widehat{\varvec{\beta }}' \varvec{X} + \epsilon \right) d{\widehat{F}}_n(\epsilon ) = \frac{1}{n} \sum ^n_{i=1} h \left( \widehat{\varvec{\beta }}' \varvec{X} + {\widehat{\epsilon }}_i \right) . \end{aligned}$$

(32)

To illustrate, suppose we consider regression $\ln Y_i = \varvec{\beta }' \varvec{X}_i + \epsilon _i$ and we want to estimate ${\mathbb {E}}[Y^\gamma /\gamma ]$, then the smearing estimate is

$$\begin{aligned} \frac{1}{n} \sum ^n_{i=1} \frac{\left( e^{\widehat{\varvec{\beta }}' \varvec{X} + {\widehat{\epsilon }}_i} \right) ^\gamma }{\gamma } = \frac{\left( e^{\widehat{\varvec{\beta }}' \varvec{X}} \right) ^\gamma }{n \gamma } \sum ^n_{i=1} e^{{\widehat{\epsilon }}_i \gamma }. \end{aligned}$$

(33)

The smearing estimate works well for non-normal errors and can accommodate for heteroskedasticity, provided it is not related to a covariate.

Appendix 2. Controlled Heteroskedasticity

Consider a simple model with heteroskedasticity, such as $Y = \varvec{\beta }' \varvec{X} + \epsilon$ where $\varvec{X}$ is a vector of covariates, $\varvec{\beta }$ is a vector of regression coefficients, ${\mathbb {E}}[\epsilon ]=0$ and $\mathrm {var}[\epsilon ]=\sigma ^2 c(\varvec{X})$. There are various ways to estimate function $c(\varvec{X})$ that is causing heteroskedasticity. In particular, we adopt a popular method from Harvey [22] (also see Baser [5], Greene [21, chapter 8]). Assume $c(\varvec{X})=e^{\varvec{{\mathcal {L}}' X}}$ to avoid negative values, where $\varvec{{\mathcal {L}}} = {\mathcal {L}}_0, {\mathcal {L}}_1,\ldots , {\mathcal {L}}_K$ is another vector of regression coefficients. Thus

$$\begin{aligned} \epsilon ^2 = \sigma ^2 c(\varvec{X}) v = \sigma ^2e^{\varvec{{\mathcal {L}}}' \varvec{X}}v^2, \quad {\mathbb {E}}[v]=0,\;{\mathbb {E}}[v^2] = 1 \end{aligned}$$

(34)

and we can write

$$\begin{aligned} \ln (\epsilon ^2) = a + {\mathcal {L}}_1 X_1 + \cdots + {\mathcal {L}}_K X_K +\ln v^2, \end{aligned}$$

(35)

where $a = \ln (\sigma ^2) + {\mathcal {L}}_0$. The parameter estimates are found by two-stage procedure. First, we find the ordinary least squares estimate $\widehat{\varvec{\beta }}$ and calculate the observed residuals ${\widehat{\epsilon }}=Y-\widehat{\varvec{\beta }}'\varvec{X}$. Then we perform the ordinary linear regression (35) where unobserved $\epsilon$ are replaced with ${\widehat{\epsilon }}$ to estimate the variance function ${\widehat{\sigma }}^2 \exp (\widehat{\varvec{{\mathcal {L}}}}' \varvec{X})$. Finally, using estimated variance, $\varvec{\beta }$ is approximated by the weighted least squares method. The process can be iterated to improve the estimates.

Other methods to estimate c(X) include random effect representation [23], kernel estimates [29] or via link functions [34].

Appendix 3. Solution to multiperiod utility model

In this section we derive the analytical solution for optimal drawdown and risky asset allocation in the multiperiod utility model considered in Sect. 5.2. The objective is to find

$$\begin{aligned} V_{0}(x) = \underset{\alpha ,\delta }{\sup } \, {\mathbb {E}} \left[ \frac{\beta ^N X_N^\gamma }{\gamma }+ \sum _{t={0}}^{N-1} \beta ^t \frac{(\alpha _t X_t)^\gamma }{\gamma } \Bigm | X_{0} = x; \alpha ,\delta \right] ,\;\gamma <0 \end{aligned}$$

(36)

and corresponding optimal values of controls $\alpha =(\alpha _0,\ldots ,\alpha _{N-1})$ and $\delta =(\delta _0,\ldots ,\delta _{N-1})$ with $\alpha _t\in (0,1)$ and $\delta _t\in {\mathbb {R}}$, $t=0,\ldots ,N-1$, when

$$\begin{aligned} X_{t+1}=X_t(1-\alpha _t) e^{\delta _t Z_{t+1}+(1-\delta _t) r},\;Z_{t+1}\overset{iid}{\sim } {\mathcal {N}}\left( \mu ,\sigma ^2\right) ,\;t=0,\ldots ,N-1. \end{aligned}$$

This problem can be solved with the backward in time recursion of the Bellman equation

$$\begin{aligned} V_{t}(X_t)=\sup _{\alpha _t,\delta _t}\left\{ \frac{(\alpha _t X_t)^\gamma }{\gamma }+\beta {\mathbb {E}}\left[ V_{t+1}(X_{t+1})|X_t;\alpha _t,\delta _t\right] \right\} ,\;\;t=N-1,\ldots ,0 \end{aligned}$$

(37)

starting with the value function at the terminal time $t=N$, $V_N(X_N) = {X_N^\gamma }/{\gamma }$. The optimal values of control are calculated as

$$\begin{aligned} (\alpha ^*_t,\delta ^*_t)=\arg \sup _{\alpha _t,\delta _t}\left\{ \frac{(\alpha _t X_t)^\gamma }{\gamma }+\beta {\mathbb {E}}[V_{t+1}(X_{t+1})|X_t;\alpha _t,\delta _t]\right\} ,\;\;t=N-1,\ldots ,0. \end{aligned}$$

(38)

Denote the stochastic component in the transition function as $\xi _{t+1}(\delta _t) = e^{\delta _t Z_{t+1}+(1-\delta _t) r}$ and define

$$\begin{aligned} {\overline{V}}_t(X_t;\alpha _t,\delta _t)=\frac{(\alpha _t X_t)^\gamma }{\gamma }+\beta {\mathbb {E}}\left[ V_{t+1}(X_{t+1})|X_t;\alpha _t,\delta _t\right] , \end{aligned}$$

(39)

so that $V_t(x)=\sup _{\alpha _t,\delta _t} {\overline{V}}_t(x;\alpha _t,\delta _t)$.

At time $t=N-1$,

$$\begin{aligned} {\overline{V}}_{N-1}(X_{N-1};\alpha _{N-1},\delta _{N-1}) =\frac{(\alpha _{N-1} X_{N-1})^\gamma }{\gamma } + \frac{\left( (1-\alpha _{N-1}) X_{N-1})^\gamma \beta {\mathbb {E}} \left[ (\xi _{N}(\delta _{N-1})\right) ^ \gamma \right] }{\gamma }. \end{aligned}$$

(40)

To find the optimal values of $\alpha _{N-1}$ and $\delta _{N-1}$, the first-order conditions for a regular interior maximum are

$$\begin{aligned}&\frac{\partial {\overline{V}}_{N-1}}{\partial \alpha _{N-1}} = X_{N-1}^\gamma \alpha _{N-1}^{\gamma -1} - X_{N-1}^\gamma (1-\alpha _{N-1})^{\gamma -1} \beta {\mathbb {E}} \left[ (\xi _{N}(\delta _{N-1}))^ \gamma \right] =0,\\&\frac{\partial {\overline{V}}_{N-1}}{\partial \delta _{N-1}} = (1-\alpha _{N-1})^\gamma X_{N-1}^{\gamma }\beta {\mathbb {E}} \left[ (Z_N-r)(\xi _{N}(\delta _{N-1}))^ \gamma \right] =0. \end{aligned}$$

Using standard closed-form integrals, the second condition gives

$$\begin{aligned} \delta ^*_{N-1} = \frac{r-\mu }{\gamma \sigma ^2}, \end{aligned}$$

(41)

substituting which into the first condition gives

$$\begin{aligned} \alpha ^*_{N-1} = (1 +\beta {\mathbb {E}} \left[ (\xi _{N}(\delta ^*_{N-1}))^ \gamma \right] ^{\frac{1}{1 - \gamma }})^{-1}. \end{aligned}$$

(42)

It is simple calculus to show that at $\alpha _{N-1}=\alpha _{N-1}^*$ and $\delta _{N-1}=\delta _{N-1}^*$,

$$\begin{aligned} \frac{\partial ^2 {\overline{V}}_{N-1}}{\partial \alpha _{N-1}\partial \alpha _{N-1}}<0,\;\frac{\partial ^2 {\overline{V}}_{N-1}}{\partial \delta _{N-1}\partial \delta _{N-1}}<0,\; \frac{\partial ^2 {\overline{V}}_{N-1}}{\partial \alpha _{N-1}\partial \delta _{N-1}}=0, \end{aligned}$$

i.e. sufficient conditions for a regular interior maximum are satisfied.

Finally, using $\alpha ^*_{N-1}$ and $\delta ^*_{N-1}$ we find the value function

$$\begin{aligned} \begin{aligned} V_{N-1}(X_{N-1})&= \frac{(\alpha ^*_{N-1} X_{N-1})^\gamma }{\gamma } + \frac{\left( (1-\alpha ^*_{N-1}) X_{N-1}\right) ^\gamma \beta {\mathbb {E}} \left[ (\xi _{N}(\delta ^*_{N-1}))^ \gamma \right] }{\gamma }\\&= \frac{(X_{N-1})^\gamma }{\gamma } (\alpha ^*_{N-1})^{\gamma -1},\\ \end{aligned} \end{aligned}$$

(43)

which will be used in the next iteration.

Similarly, at time $t=N-2$ we obtain

$$\begin{aligned} \begin{aligned}&V_{N-2}(x) = \frac{x^\gamma }{\gamma } (\alpha ^{*}_{N-2})^{\gamma -1},\;t=N-1,\ldots ,0,\\&\alpha ^{*}_{N-2} =\left( 1+\left( \beta {\mathbb {E}}\left[ (\xi _{N-1}(\delta _{N-2}^*))^{\gamma }\right] (\alpha ^{*}_{N-1})^{\gamma -1}\right) ^{\frac{1}{1-\gamma }}\right) ^{-1},\;\delta ^{*}_{N-2} = \frac{r-\mu }{\gamma \sigma ^2}, \end{aligned} \end{aligned}$$

(44)

Now, it is easy to see the solution for any t follows the pattern

$$\begin{aligned} \begin{aligned}&V_{t}(x) = \frac{x^\gamma }{\gamma } (\alpha ^*_t)^{\gamma -1},\;t=N-1,\ldots ,0,\\&\alpha ^*_t =\left( 1+\left( \beta {\mathbb {E}}\left[ (\xi _{t+1}(\delta _t^*))^{\gamma }\right] (\alpha ^{*}_{t+1})^{\gamma -1}\right) ^{\frac{1}{1-\gamma }}\right) ^{-1},\;\delta ^*_{t} = \frac{r-\mu }{\gamma \sigma ^2}, \end{aligned} \end{aligned}$$

(45)

where we set $\alpha ^*_N=1$. This claim is easily proved by induction principle (i.e. assuming that claim holds for $t+1$ one can show that it holds for t).

Note that in the above equations, we can use closed-form formula

$$\begin{aligned} {\mathbb {E}}\left[ (\xi _{t+1}(\delta _t))^{\gamma }\right] =e^{\gamma \delta _t\mu +\gamma ^2\delta _t^2\sigma ^2/2+(1-\delta _t)\gamma r}. \end{aligned}$$

Also, note that the closed form solution for consumption only problem, considered in Sect. 5.1, can be obtained by setting $\delta _t=1$ in the above formulas in this section.