Numerical Approximation of a System of Hamilton–Jacobi–Bellman Equations Arising in Innovation Dynamics

Abstract

We consider a system of fully nonlinear partial differential equations that corresponds to the Hamilton–Jacobi–Bellman equations for the value functions of an optimal innovation investment problem of a monopoly firm facing bankruptcy risk. We compare several algorithms for the numerical solution of the considered problem: the collocation method, the finite difference method, WENO method and the adaptive finite element method. We discuss implementation issues for the considered schemes and perform numerical studies for different model parameters to assess their performance.

1 Introduction

We study the system of Hamilton–Jacobi–Bellman (HJB) equations

$$\begin{aligned} rV_1(x)&= \sup _{I} {\mathcal {L}}_1 (V_1, V_2,x, I) + f_1(x)&\quad x\in \mathbb {R}, \end{aligned}$$

(1.1)

$$\begin{aligned} rV_2(x,y)&={\mathcal {L}}_2 (V_2,x,y)+ f_2(x)&\quad x \in \mathbb {R},\;y\ge 0, \end{aligned}$$

(1.2)

where $r>0$ is a constant (the so-called discount rate) and the operators ${\mathcal {L}}_1$, ${\mathcal {L}}_2$ are defined as

$$\begin{aligned} {\mathcal {L}}_1 (V_1, V_2,x,I):=&\, b_1(x,I)\partial _xV_1(x) - p_0(x)V_1(x) +p_2(I)(V_2(x,0)-V_1(x)), \end{aligned}$$

(1.3)

$$\begin{aligned} {\mathcal {L}}_2 (V_2,x,y):=&\, \tfrac{1}{2}\sigma ^2(y)\partial _{yy}^2V_2(x,y) + b_2(x,y)\partial _xV_2(x,y) +\mu (y)\partial _yV_2(x,y)\nonumber \\&- p_0(x)V_2(x,y) . \end{aligned}$$

(1.4)

The terms $f_1$ and $f_2$ in (1.1)–(1.2) as well as $b_1$, $b_2$, $p_0$, $p_2$, $\sigma$, and $\mu$ in (1.3)–(1.4) are real-valued continuous functions specified in Table 2 below.

The HJB system (1.1)–(1.2), which can be classified as a fully-nonlinear degenerate second-order partial differential equation (PDE) in non-divergence form, was derived in [9] to study optimal innovation investment strategies of a monopoly firm under financial constraints, i.e., that is facing a bankruptcy risk. The solutions $V_1$ and $V_2$ to (1.1)–(1.2) correspond to the value functions of different investment modes of the monopoly firm considered on an infinite time horizon and the control variable I in (1.1) is the innovation investment (i.e. the amount of innovation activity the firm performs). Given the liquidity $x\equiv x(t)$ and the innovation investment $I\equiv I(t)$ at time $t\ge 0$, the monopoly firm may enter three different modes of production $m=0,1,2$ with certain probability rates that are characterized by a Markov process $m:=\{m_t \in \{0, 1, 2\}:t\ge 0 \}$ with $m_0 = 1$, and for every $t\ge 0$

$$\begin{aligned} \mathbb {P}[m_{t + \mathrm {d}t} = j| m_t = i] = {\left\{ \begin{array}{ll} p_0(x(t))\mathrm {d}t, &{} (i,j) = (1,0) \text{ or } (i,j) = (2,0), \\ p_2({ I(t)})\mathrm {d}t, &{} (i,j) = (1,2), \\ 0, &{} \text{ for } \text{ any } \text{ other } (i,j),\,i\ne j, \end{array}\right. } \end{aligned}$$

(1.5)

where $p_0\ge 0$ is the bankruptcy rate and $p_2\ge 0$ is the innovation rate. For every mode $i\in \{0,1,2\}$, it holds that $\sum _{j = 0}^{2}\mathbb {P}[m_{t + \mathrm {d}t} = j| m_t=i] = 1$.

The possible scenarios given by the above Markov process can be summarized as follows. The mode $m=1$ corresponds to the pre-innovation mode where the firm is selling an old product on an established market and is developing a new product with a suitable amount of innovation investment I. The latter decreases firm’s liquidity, but allows it to enter the new market and to move from selling only the old product ($m=1$) to selling both the old and the new product on the market ($m=2$) with a probability rate $p_2\equiv p_2({ I})$. Finally, depending on its liquidity the firm may go bankrupt with a probability rate $p_0\equiv p_0(x)$ and exit the market, that is the firm leaves the respective mode $m=1$ or $m=2$ and enters the mode $m=0$, where the value $V_0=0$ is constant. The objective of the firm is to maximize the accumulated dividend $f_{m}$, $m=1,2$ over time, which yields the system (1.1)–(1.2) for the modes $m=1$, $m=2$, respectively. For details on the derivation of the HJB Eqs. (1.1–(1.2) we refer to [9].

In general, the solutions to (1.1)–(1.2) do not exist in the classical sense but can be defined in the sense of viscosity solutions, cf., [14, 28, 33]. The existence of the unique viscosity solution of a general HJB equation written in the form

$$\begin{aligned} F(\mathrm {D}^2 V, \nabla V, V,x) = 0\qquad x\in \mathbb {R}^d, \end{aligned}$$

(1.6)

is guaranteed provided that the functional F is semicontinuous and satisfies a monotonicity condition, i.e., it holds for $A,A'\in \mathbb {R}^{d\times d}$, $A\ne A'$, $p\in \mathbb {R}^d$, $x\in \mathbb {R}^d$, $d\ge 1$ that

$$\begin{aligned} F(A,p,q, x)\le F(A',p,q', x), \end{aligned}$$

(1.7)

whenever $A-A'$ is negative semidefinite and $q\ge q'$, see [18] and [33, Chapter 5].

Equation (1.2) corresponds to (1.6) with $d=2$ and F given by

$$\begin{aligned} F(A,p,q,(x,y)) = \tfrac{1}{2} \begin{pmatrix} 0 &{} 0 \\ 0 &{} \sigma ^2(y) \end{pmatrix} : A + (b_2(x, y), \mu (y))\cdot p - (p_0(x) + r)q + f_2(x)\,, \end{aligned}$$

where for two $d\times d$ matrices $A:=\{a_{i,j}\}_{i,j}$, $B:=\{b_{i,j}\}_{i,j}$ we denote $A : B :=\sum _{i,j}^d a_{i,j}b_{i,j}$. The above F is monotone since $\sigma ^2\ge 0$ and $( p_0 + r)>0$. The continuity of F is a consequence of the continuity of the functions $\sigma$, $\mu$, and $f_m, b_m$, $m=1,2$. Hence, we deduce that (1.2) admits a unique viscosity solution.

Equation (1.1) corresponds to (1.6) with $d=1$ and F given by

$$\begin{aligned} F(0,p,q,x) = \sup _{I} \Big \{b_1(x, I)p - (p_0(x) + p_2(I) + r)q + p_2(I)V_2(x, 0)\Big \} + f_1(x)\,, \end{aligned}$$

which is monotone since $r>0$ and $p_0,p_2\ge 0$. Formally, one can equivalently formulate the Hamiltonian using $I^{opt} := \displaystyle \arg \max _I \left\{ b_1(x, I)p - (p_0(x) + p_2(I) + r)q + p_2(I)V_2(x, 0)\right\} = \frac{\gamma _I}{\xi }\frac{V_2-q}{p}$ as

$$\begin{aligned} F(0,p,q,x) = \Big \{b_1(x, I^{opt})p - (p_0(x) + p_2(I^{opt}) + r)q + p_2(I^{opt})V_2(x, 0)\Big \} + f_1(x)\,. \end{aligned}$$

For the considered model parameters (cf., Table 2) F is continuous, except for $p=0$; the singular case $p=0$ can be dealt with by imposing an additional condition $I(x) \le I^{max}$, $x\in \mathbb {R}$ for some $0<I^{max}< \infty$.

Apart from theoretical considerations, the monotonicity condition (1.7) is also required for the convergence of monotone and consistent numerical schemes, cf., [1] and the recent overview paper [27]. The finite difference schemes are among the most established schemes for the approximation of HJB equations. Monotone finite difference schemes can be constructed by appropriately choosing the finite difference stencil, for instance by the upwind finite difference approximation [32, 34] or by adopting a semi-Lagrangian approach [7, 10, 11]. Much fewer contributions are available on monotone finite element (FEM) based discretization methods including [15, 17, 29], where in particular [17] is the first to show convergence of FEM discretizations of HJB equations of the type (1.1). A popular choice in applications is the collocation method [9, 35] because of its capability to solve HJB equations in higher dimensions, cf., [6]. As far as we are aware, very few results exist on adaptive numerical approximation of HJB equations. We mention [35] which employs an adaptive collocation approach; and [15, 21] utilize adaptive finite element methods to linear second-order problems that satisfy the so-called Cordes condition see [27, Definition 2.46]. We note that the Cordes condition is not satisfied for the model (1.1)–(1.2). A widely used class of stable higher order discretization schemes for the time-dependent HJB equations, so-called WENO (weighted essentially non-oscillatory) schemes, have been introduced in [19, 23]. The WENO schemes have been used to solve HJB equations in conjunction with the semi-Lagrangian approach, see for instance [3, 8, 12]. An alternative approach to achieve higher-order accuracy in space is to use the discontinuous Galerkin approximation, see for instance [22] and the references therein. Finally, we also mention that the considered HJB model bears some similarities with the problems of utility maximization under regime switching, that leads to a system of coupled HJB equations. The individual equations in the HJB system can be decoupled via a fixed-point type iteration; for further details see [25] (as well as the references therein), where a monotone finite difference method for the solution of the HJB system is analyzed using the Barles–Souganidis framework. In contrast to the regime switching problem the HJB system (1.1) is decoupled but the respective equations are posed on different state spaces. Furthermore, whereas the generator of the regime switching process in [25] is constant over time, in our setting the switching rate between modes is a function of the state and of the control, in general.

In the present work we compare different discretization methods for the approximation of the HJB Eqs. (1.1)–(1.2). We note that the Eq. (1.2) can be solved independently of (1.1). Hence we first determine the numerical approximation of the value function $V_2$ in mode $m=2$. This numerical approximation is then used to compute the approximation of the value function $V_1$ in mode $m=1$. We consider a stabilized finite element method (FEM) for the approximation of the linear degenerate Eq. (1.2). Furthermore, we propose two adaptive finite element discretization strategies for the solution of the (nonlinear) HJB Eq. (1.1); the first approach employs a stabilized finite element approximation and the second one is based on an adaptive least-squares finite element method (LSFEM). The nonlinear algebraic system that results from the discretization of (1.1) is solved iteratively using the well-known policy iteration algorithm, see [16, Chapter 4] and [5]. We compare the proposed finite element methods with alternative discretization approaches: the Chebyshev collocation method and the finite difference method. We discuss relevant implementation details of the considered numerical schemes. In particular, the convergence of the policy iteration for the discrete approximation of (1.1) proves to be problematic in certain scenarios; we discuss suitable modification of the considered numerical algorithms which improve the convergence behavior of the policy iteration.

In addition to the above schemes that approximate the problem (1.1) directly, we propose a numerical approach where (1.1) is interpreted as a steady state of the corresponding time-dependent problem. The time-dependent problem is solved using the policy iteration where the subproblems are discretized in time using approach that combines a total-variation diminishing time discretization with a WENO spatial-discretization of the time dependent problem. The spatial discretisation is based on the upwind WENO5 scheme proposed in [23, 19]. The resulting finite-dimensional system of ordinary differential equations is approximated by a strong stability preserving time-discretization scheme, cf. [30], which is used to compute the steady state of the discrete problem.

The paper is organized as follows. In Sect. 2, we give details on the notation and the considered parameters in (1.1)–(1.2), and discuss analytical properties which are useful to deduce boundary conditions that are required in the considered numerical approximations. In Sect. 3, we discuss the discretization of (1.1)–(1.2) by the collocation method, Sect. 4 is dedicated to the finite difference approximation. The WENO scheme is introduced in Sect. 5. In Sect. 6, we present the stabilized FEM method and the LSFEM method as well as the corresponding adaptive mesh refinement algorithms. We discuss the performance of the considered numerical schemes in Sect. 7.

2 Preliminaries

2.1 Model Parameters

The functions in (1.1)–(1.4) are defined in Table 2 and the values of the constants $\nu _1$, $\nu _2$, ${\widetilde{\alpha }}_n$, $\bar{\alpha }_n$, $\eta$, $\delta$, $\sigma$, r, $\gamma _I$, $\xi$ in these expressions are fixed throughout the paper, see Table 1. We compute the numerical solution for different values of the parameters $\alpha _o$, $\gamma _B$; we consider the following three Scenarios: Scenario 1 with ($\alpha _o = 0.8$, $\gamma _B = 0.05$), Scenario 2 with ($\alpha _o = 0.8$, $\gamma _B = 0.005$), and Scenario 3 with ($\alpha _o = 1.0$, $\gamma _B = 0.05$); the parameters in the considered Scenarios can be interpreted as follows: $\alpha _o$ is the established market size and $\gamma _B$ is the bankruptcy parameter. Furthermore, the state variable $y \equiv y(t)$ is interpreted as the evolving size of the market demand for the new product.

Table 1 Parameters of the monopoly model

Table 2 Coefficients of the HJBs

2.2 Boundary Conditions

All numerical schemes considered in this paper can only be applied on bounded domains. Hence, we restrict the computations on sufficiently large finite domains and deduce suitable boundary conditions from the considerations below. Regarding (1.5), the probability of bankruptcy, i.e., $V_1= V_0$ or $V_2 = V_0$ is increasing when $x\rightarrow -\infty$, thus at the left end of the domain, we impose a zero Dirichlet boundary condition to $V_1$ while at the right end, the solution can be characterized by the following result from [9, Proposition 1].

Proposition 2.1

Assume that $r>\nu _1$. For every $x,y\ge 0$ the value in mode $m=2$ is given by

$$\begin{aligned} V_2(x,y) = x + c + \frac{\delta {\widetilde{\alpha }}_n+ (\bar{\alpha }_n-{\alpha _o}\eta )}{2(r + \delta )(1 - \eta ^2)}y + \frac{1}{4(1 - \eta ^2)(r + 2\delta -\sigma ^2)}y^2, \end{aligned}$$

(2.1)

with $c :=\dfrac{\delta ^2{\widetilde{\alpha }}_n^2 + \delta {\widetilde{\alpha }}_n(\bar{\alpha }_n - \alpha _o\eta )(r+2\delta -\sigma ^2)}{2r(r + \delta )(1 -\eta ^2)(r+2\delta -\sigma ^2)} + \dfrac{\bar{\alpha }_n^2 + \alpha _o^2 - 2\eta \alpha _o\bar{\alpha }_n}{4r(1-\eta ^2)}.$

Furthermore, it holds for every $x\ge {\tilde{x}}:=\max \bigg \{\dfrac{2\xi |I^{\mathrm {nc}}|^2-\alpha _o^2}{4(r-\nu _1)},0\bigg \}$ with $I^{\mathrm {nc}}: =\sqrt{\frac{r^2}{\gamma _I^2} + \frac{2rc}{\xi } - \frac{\alpha _o^2}{2\xi }} - \frac{r}{\gamma _I}$

• (i) The optimal control is constant over time, that is $I = I^{\mathrm {nc}}$;

• (ii) The value in mode $m=1$ is given by

  $$\begin{aligned} V_1(x) = x + c - \frac{\xi }{\gamma _I}I^{\mathrm {nc}}. \end{aligned}$$

  (2.2)

3 Discretization by Collocation Method

In this section, we describe the Chebyshev collocation method to compute an approximate solution for (1.1) and (1.2). The method approximates $V_1(x)$ and $V_2(x, y)$ by finite series of Chebyshev polynomials with unknown coefficients. The finite series are then substituted into (1.1) and (1.2) and the coefficients are determined by requiring that the differential equations are satisfied at some finite number of collocations nodes. The collocation method can only be applied on a finite domain and thus, for convenience, the scaling functions (3.1)–(3.2) are used.

3.1 Rescaling Transformation

A modification of the differential Eqs. (1.1)–(1.2) on an infinite domain results into equivalent differential equations on finite domains. In the manner of [9], we introduce the function

$$\begin{aligned} z:\mathbb {R}\rightarrow (0,1),\quad x\mapsto z(x)=\frac{1}{1 + \exp (- x/2)}, \end{aligned}$$

(3.1)

which transforms $\mathbb {R}$ into (0, 1). The inverse transformation of (3.1) reads

$$\begin{aligned} x:(0,1)\rightarrow \mathbb {R},\quad z\mapsto x(z) = 2\ln \left( \frac{z}{1-z}\right) . \end{aligned}$$

(3.2)

Thanks to the transformation (3.2), we can work on the bounded interval (0, 1). However, the transformation (3.2) is still undefined at $z = 0$ and $z = 1$. Thus, we make an approximation of (0, 1) by cutting off the left and right ends of the interval and work in $[{\underline{z}},\bar{z}]$. Analogously, we perform the computations for $y\in [{\underline{y}},\bar{y}]$ instead of $y\ge 0$.

We set $U(z):=V_1(x(z))$ and $V(z,y):=V_2(x(z),y)$. The inverse transformation (3.2) implies that $\partial _xV_1 = \tfrac{1}{2} z(1-z)\partial _zU$ and as a result the differential equations in (1.1)–(1.2) are transformed into

$$\begin{aligned} r U(z)&= \sup _{I}L_1\big (U,V, z, I\big ) + f_1(x(z))&\quad z\in [{\underline{z}}, \bar{z}]; \end{aligned}$$

(3.3)

$$\begin{aligned} r V(z,y)&= L_2\big (V,z,y\big ) + f_2(x(z))&\quad z\in [{\underline{z}}, \bar{z}],\; y \in [{\underline{y}}, \bar{y}]; \end{aligned}$$

(3.4)

where

$$\begin{aligned} L_1 (U, V,z,I) :=&\tfrac{1}{2}z(1-z)b_1(x(z),I)\partial _zU(z) - p_0(x(z))U(z)\nonumber \\&+p_2(I)(V(z,0)-U(z)), \end{aligned}$$

(3.5)

$$\begin{aligned} L_2 (V,z,y):=&\tfrac{1}{2}\sigma ^2(y)\partial _{yy}^2V_2(z,y) \nonumber \\&+ \tfrac{1}{2} z(1-z)b_2(x(z),y)\partial _zV(z,y) +\mu (y)\partial _yV(z,y) \nonumber \\&- p_0(x(z))V(z,y). \end{aligned}$$

(3.6)

3.2 Discretization of the Value in Post–Innovation Mode, $m=2$

In a given state space $[{\underline{z}}, \bar{z}]\times [{\underline{y}}, \bar{y}]$, we first construct a grid of collocation nodes $\mathcal {N}:=\mathcal {N}_z\times \mathcal {N}_y$, where $\mathcal {N}_z :=\{z^i\}_{i=1,\ldots ,n_z}$ and $\mathcal {N}_y :=\{y^j\}_{j=1,\ldots ,n_y}$, $z^i$ and $y^j$ are defined as

$$\begin{aligned} z^{i}&= \frac{\bar{z}+ {\underline{z}}}{2} + \frac{\bar{z}- {\underline{z}}}{2} \cos {\left( \frac{\left( n_z - i + 0.5 \right) \pi }{n_z} \right) }, \end{aligned}$$

(3.7)

$$\begin{aligned} y^j&= \frac{\bar{y}+ {\underline{y}}}{2} + \frac{\bar{y}- {\underline{y}}}{2} \cos {\left( \frac{\left( n_y - j + 0.5 \right) \pi }{n_y} \right) }. \end{aligned}$$

(3.8)

We construct a set of basis functions $\{\phi _{i,j}(z,y)\}_{ \{i=1,\ldots ,n_z\}\times \{j=1,\ldots ,n_y\}}$ corresponding to our Chebyshev grid such that

$$\begin{aligned} \phi _{i,j}(z, y) = T_{i-1} \left( \frac{2 \left( z - {\underline{z}}\right) }{\bar{z}- {\underline{z}}}-1 \right) T_{j-1}\left( \frac{2 \left( y - {\underline{y}}\right) }{\bar{y}- {\underline{y}}}-1 \right) , \end{aligned}$$

(3.9)

where $T_k:t \mapsto T_k(t)$ is the Chebyshev polynomial of degree k defined for any $t\in [0,1]$. For all $z\in [0.5, 1)$ and $y\ge 0$, the value function in mode $m=2$ is given by (2.1). Our calculation is carried out only in the space $[{\underline{z}}, 0.5]\times [0, \bar{y}]$. In order to make sure the calculated value function is continuous at $z=0.5$, that corresponds to $x = 0$, we specify further that

$$\begin{aligned} z^{i} = {\left\{ \begin{array}{ll} \dfrac{0.5 + {\underline{z}}}{2} + \dfrac{0.5 - {\underline{z}}}{2} \cos {\left( \dfrac{\left( n_z - i + 0.5 \right) \pi }{n_z} \right) } &{} 1 \le i \le n_z-1,\\ 0.5 &{} i=n_z. \end{array}\right. } \end{aligned}$$

(3.10)

The expansion of $V_2$ as a Chebyshev series is given by

$$\begin{aligned} {{\widehat{V}}}_2(z,y) :=\sum _{i=1}^{n_{z}} \sum _{j=1}^{n_y} v_{i,j} \phi _{i,j}(z,y) ={\bar{v}}^\top \phi (z,y),~z\in [{\underline{z}}, 0.5], \end{aligned}$$

(3.11)

where ${\bar{v}}:=\{{\bar{v}}_k:k=1,\ldots ,n_zn_y\}_k$ and $\phi :=\{\phi _k:k=1,\ldots ,n_zn_y\}$ such that ${{\bar{v}}}_k :=v_{i,j}$ and ${\phi }_k(z,y) :=\phi _{i,j}(z, y)$ with $k=(i-1)n_z + j$ for $i \in \{1,\ldots ,n_z \}$ and $j \in \{1,\ldots ,n_y\}$. The solution V(z, y) of (3.4) is substituted by the expression (3.11). Thus the approximation of the value function V is equivalent to determine the components ${\bar{v}}_k$ of the vector ${{\bar{v}}}$ such that (3.4) holds on the collocation nodes $\{z^i, y^j\}$ with $i,j\in \{1,\ldots ,n_z-1\}\times \{1,\ldots ,n_y \}$. The approximation of $V_2$ is therefore defined by

$$\begin{aligned} \bar{V}_2(z,y) :={\left\{ \begin{array}{ll} {{\widehat{V}}}_2(z, y), &{}z\in [{\underline{z}},0.5], \\ V_2(x(z), y), &{}z\in [0.5, \bar{z}], \end{array}\right. } \end{aligned}$$

(3.12)

where $x:z\mapsto x(z)$ is given by (3.2) and $V_2:(x,y)\mapsto V_2(x,y)$ is given by the formula (2.1). Furthermore, we impose ${{\widehat{V}}}_2(0.5, y^j) = V_2(x(0.5), y^j)$ to enforce the continuity of (3.12) at $z = 0.5$.

In total there are $n_z n_y$ number of nodes, implying $n_z n_y$ number of equations. Furthermore, for $i \in \{1,\ldots ,n_z-1\}$ and $j\in \{1,\ldots ,n_y\}$ we introduce four $(n_z-1) n_y \times n_z n_y$ matrices $\mathbf {B}$, $\mathbf {B}^y$, $\mathbf {B}^{yy}$, and $\mathbf {B}^z$ with entries

$$\begin{aligned} \mathbf {B}_{s,k} = \phi _k(z^i,y^j),~~ \mathbf {B}^y_{s,k}= \partial _y\phi _k(z^i,y^j), ~~ \mathbf {B}^{yy}_{s,k}= \partial _{yy}^2\phi _k(z^i,y^j),~~ \mathbf {B}^z_{s,k}= \partial _z\phi _k(z^i,y^j),\nonumber \\ \end{aligned}$$

(3.13)

where $s=(i-1)n_z + j$ denotes node s. These four matrices capture the values of all base functions and their partial derivatives at the nodes in $\mathcal {N}$ that are not on the boundary of $z^{n_z}=0.5$. For each node $\{z^{i},y^j\}$ with $i\in \{1,\ldots ,n_z-1\}$ and $j\in \{1,\ldots ,n_y\}$, we define the five column vectors ${{\bar{\sigma }}}:=({{\bar{\sigma }}}_s)_s$, ${{\bar{\mu }}}:=({{\bar{\mu }}}_s)_s$, ${\bar{b}}_2:=({\bar{b}}_{2,s})_s$, ${\bar{p}}_0:=({\bar{p}}_{0,s})_s$, and ${\bar{f}}_2:=({\bar{f}}_{2,s})_s$ as follows:

$$\begin{aligned} {{\bar{\mu }}}_s&= \mu (y^j), ~~ {{\bar{\sigma }}}_s = {\tfrac{1}{2}\sigma (y^j)^2},~~ {\bar{b}}_{2,s} = \tfrac{1}{2} z^i(1-z^i)b_2(x(z^i), y^j),~{\bar{f}}_{2,s} = f_2(x(z^i)),~\\ {\bar{p}}_{0,s}&= p_0(x(z^i)), \end{aligned}$$

and $s=(i-1)n_z + j\in \{1,\ldots , (n_z-1)n_y\}$. For the s nodes, the resulting linear system of equations reads

$$\begin{aligned} r \mathbf {B}{\bar{v}}= {{\bar{\sigma }}}^\top \mathbf {B}^{yy} {\bar{v}}+ {{\bar{\mu }}}^\top \mathbf {B}^y{\bar{v}}+ {\bar{b}}_2^\top \mathbf {B}^z{\bar{v}}- {\bar{p}}_0^\top \mathbf {B}{\bar{v}}+ {\bar{f}}_2. \end{aligned}$$

For the other $n_y$ nodes, it reads

$$\begin{aligned} {\bar{v}}\cdot {\phi }(0.5,y^j)= V_2(0,y^j),~~~j \in \{1,\ldots , n_y\}. \end{aligned}$$

3.3 Discretization of the Value in Pre–Innovation Mode, $m=1$

In contrast to mode $m=2$, the value function $V_1$ in mode $m=1$ is a one-dimensional problem defined on $\mathbb {R}$. The optimal control $I^*={\mathop {{{\,\mathrm{arg\,max}\,}}}\limits _{{\widetilde{I}}}}\,{\mathcal {L}}_1 (V_1, V_2,x, {\widetilde{I}})$ can be expressed explicitly as

$$\begin{aligned} I^*(x)=\frac{\gamma _I(V_2(x,0) - V_1(x))}{\xi \partial _xV_1(x) }. \end{aligned}$$

(3.14)

Similarly as for the mode $m=2$, we employ the transformation (3.2). After the transformation, the optimal control in (3.14) can be rewritten as

$$\begin{aligned} {{\tilde{I}}}^*(z) = I^*(x(z)) = \frac{2\gamma _I(V_2(x(z),0) - V_1(x(z)))}{ \xi z (1-z) \partial _zV_1(x(z))}, \end{aligned}$$

(3.15)

for all z where $\partial _zV_1(x(z))$ exists. We proceed with the same collocation method as in mode $m=2$, but on the one dimensional state space of $z \in [{\underline{z}},\bar{z}]$. However, in mode $m=1$ for some values of the model paramaters there potentially exists a kink in $V_1(x(z))$ at $z=0.5$ and consequently the control function ${{\tilde{I}}}^*(z)$ is discontinuous at $z=0.5$ ($x=0$). In [9, Proposition 2 and Corollary 3] an analytical characterization of scenarios where such a jump in the optimal control function occurs is provided. In particular, it is shown that the following three cases can occur.

• (i) For $\gamma _I \in [{\underline{\gamma }}_I, \bar{\gamma }_I]$ with

  $$\begin{aligned} {\underline{\gamma }}_I = \frac{r \alpha _o \sqrt{2 \xi }}{2 r c - \alpha _o^2}, \quad \bar{\gamma }_I = \frac{\left( r + \gamma _B c \right) \alpha _o \sqrt{2 \xi } }{ 2 r c - \alpha _o^2 } \end{aligned}$$

  the state $x = 0$ is a (stable) fixed point under the dynamics with optimal investment in mode $m_1$, i.e. $b_1(0,I^*(0))=0$ and both intervals $(-\infty ,0]$ and $[0,\infty )$ are invariant under the dynamics $\dot{x} = b_1(x,I^*(x))$. For $\gamma _I \in ({\underline{\gamma }}_I, \bar{\gamma }_I)$ the optimal control is discontinuous at $x=0$ with $\lim _{x \rightarrow 0-} I^*(x)< I^*(0) < \lim _{x \rightarrow 0+} I^*(x)$.

• (ii) If $\gamma _I < {\underline{\gamma }}_I$ then $b_1(0,I^*(0)) > 0$. Hence, the interval $[0,\infty )$ is invariant under the optimized state dynamics, but $(-\infty ,0]$ is not.

• (iii) If $\gamma _I > \bar{\gamma }_I$ then $b_1(0,I^*(0)) < 0$. Hence, the interval $(-\infty ,0]$ is invariant under the optimized state dynamics, but $[0,\infty )$ is not.

In the case (i) the discontinuity of the control at $x=0$ is guaranteed, whereas in cases (ii) and (iii) the theoretical results do not allow for a clear conclusion with respect to the continuity of optimal investment at $x = 0$. Due to the polynomial approximation of the value function in our collocation approach, by definition the approximate value function is smooth on the entire considered state space and a value function with a kink cannot be generated. Hence, based on the considerations above, we apply the collocation scheme separately on the intervals $(-\infty ,0]$ and $[0,+\infty )$. We define the approximation of $V_1$ as follows

(3.16)

where ${{\widehat{V}}}_1$ and are respectively the approximation of $V_1$ by Chebyshev expansion in $[{\underline{z}},0.5]$ and $[0.5,\bar{z}]$, that is for $x\le 0$ and $x\ge 0$. Depending on which of the three listed cases applies boundary conditions for one or both of the collocation schemes are inserted. In particular, in the case (i) the value at $z=0.5$ is calculated as

$$\begin{aligned} V_1(x(0.5)) = V_1(0) = \int _{0}^{\infty } e^{-(r + \gamma _I {\tilde{I}}^*(0.5)) t } \gamma _I {{\tilde{I}}}^*(0.5) V_2(0,0) dt = \frac{\gamma _I \alpha _o V_2(0, 0)}{r \sqrt{2 \xi } + \gamma _I \alpha _o} ~,\nonumber \\ \end{aligned}$$

(3.17)

and in both collocations, similar to the procedure in mode $m_2$, one node is set at $z=0.5$ and the condition ${{\widehat{V}}}_1(x(0.5)) = V_1(x(0.5))$ respectively is included. In case (ii) the collocation is first applied to $[0.5, \bar{z}]$ without any boundary condition, i.e. all $n_z$ nodes in the interior of the interval and all conditions determined by the HJB equation at the node. This yields . In a second step the collocation is applied to $[{\underline{z}},0.5]$ with the boundary condition . Analogous for case (iii).

We illustrate the application of the collocation method for scenario (iii), the other scenarios are analogous. In the case (iii), the interval $[{\underline{z}},0.5]$ is invariant under the dynamics and the expansion of $V_1$ as a Chebyshev series on this interval is given by

$$\begin{aligned} {{\widehat{V}}}_1(z) :=\sum _{i=1}^{n_{z}} u_{i} \phi _{i, 1}(z, 0) = { {{\bar{u}}^\top } {\phi }(z, 0)},~z\in [{\underline{z}},0.5], \end{aligned}$$

(3.18)

where ${\bar{u}}:=\{u_i:i=1,\ldots ,n_z\}$ is a column vector of coefficients. The value function U(z) in (3.3) is then substituted by the expression (3.18) and the approximation is found by determining the components $u_i$ of the vector ${{\bar{u}}}$ such that (3.3) holds on the collocation nodes $\{z^i:i=1,\ldots ,n_z\}$. To solve the resulting nonlinear problem we use a policy iteration procedure, see for instance [2, 16]. In particular, we consider a sequence of vectors ${{\bar{u}}^{(n-1)}}$, with $n\in \mathbb {N}$ as the iteration counter. By (3.15), we derive the discrete optimal control $\bar{I}^{(n-1)}:=\{I^{(n-1)}_i:i = 1,\ldots ,n_z \}$ at the stage $n \in \mathbb {N}$ of the iteration by

$$\begin{aligned} \bar{I}^{(n-1)}= \frac{\gamma _I\big (\mathbb {B}{\bar{v}}_{\cdot ,1}- \mathbb {B}{{\bar{u}}^{(n-1)}}\big ) }{\xi {{\bar{\lambda }}}^\top \mathbb {B}^z {\bar{u}}^{(n-1)}}, \end{aligned}$$

(3.19)

where the column vectors ${\bar{v}}_{\cdot ,1}:=\{v_{i,1}:i = 1,\ldots ,n_z \}$ and ${{\bar{\lambda }}}:=\{\tfrac{1}{2} z^i(1-z^i):z^i\in \mathcal {N}_z, i = 1,\ldots ,n_z\}$ and the matrices $\mathbb {B}$ and $\mathbb {B}^z$ are such that

$$\begin{aligned} \mathbb {B}_{i,k} = \phi _k(z^i,0), ~~~ \mathbb {B}^z_{i,k}=\partial _z\phi _k(z^i,0), ~~i \in \{1,\ldots , n_z\}. \end{aligned}$$

Overall, the numerical algorithm that couples the Chebyshev collocation method and the policy iteration can be summarized as follows:

Algorithm 3.1

(Collocation method) Start with an initial guess for ${\bar{u}}^{(0)}$ (see below) and calculate the control $\bar{I}^{(0)}$ according to (3.19). Then, for every $n \in \mathbb {N}$, we proceed along the following two steps iteration:

Policy evaluation: Find ${\bar{u}}^{(n)} = \{u^{(n)}_i:i=1,\ldots ,n_z\}$ such that

$$\begin{aligned} r \mathbb {B}{{\bar{u}}^{(n)}} = {\bar{b}}_1^\top (\bar{I}^{(n-1)})\mathbb {B}^z{\bar{u}}^{(n)}-({\bar{p}}_0+{\bar{p}}_2(\bar{I}^{(n-1)}))^\top \mathbb {B}{{\bar{u}}^{(n)}}+ {\bar{p}}_2^\top (\bar{I}^{(n-1)}) \mathbb {B}{\bar{v}}_{\cdot ,1}+ {\bar{f}}_1. \end{aligned}$$

Policy improvement: Update the optimal control by (cf. (3.19))

$$\begin{aligned} \bar{I}^{(n)}= \frac{\gamma _I\big (\mathbb {B}{\bar{v}}_{\cdot ,1}- \mathbb {B}{{\bar{u}}^{(n)}}\big ) }{\xi {{\bar{\lambda }}}^\top \mathbb {B}^z {\bar{u}}^{(n)}}. \end{aligned}$$

Stopping criterion: Repeat the loop until

$$\begin{aligned} \frac{\big \vert r \mathbb {B}{{\bar{u}}^{(n)}} - {\bar{b}}_1^\top (I^{(n)})\mathbb {B}^z{\bar{u}}^{(n)}+({\bar{p}}_0+{\bar{p}}_2(I^{(n)}))^\top \mathbb {B}{{\bar{u}}^{(n)}}- {\bar{p}}_2^\top (I^{(n)}) \mathbb {B}{\bar{v}}_{\cdot ,1}- {\bar{f}}_1 \big |_{\infty }}{\big |\mathbb {B}{{\bar{u}}^{(n)}}\big |_{\infty }}\le \epsilon . \end{aligned}$$

In order to determine the initial guess for ${\bar{u}}^{(0)}$ we employ a continuation approach. For $\gamma _I = 0$ the value function is linear and can be determined in closed form (see [9]). The corresponding coefficient vector ${\bar{u}}$ is found by interpolation of this function at the nodes $\{z^i:i=1,\ldots ,n_z\}$. The parameter $\gamma _I$ is then stepwise increased till its target value is reached, where in each step the coefficient vector found in the previous step is used as the initial guess ${\bar{u}}^{(0)}$. The numerical calculation of with $z\in (0.5,\bar{z}]$ is similar to the calculation of ${\widehat{V}}_1(z)$ except we further enforce so that the continuity of (3.16) is preserved at $z = 0.5$.

3.4 Numerical Results

Fig. 1

Numerical solutions for the collocation method. Top: Effect of the domain on V1 (left) and I (right) for Scenario 2 for lower bounds ${\underline{x}} = -20, -15, -10$ with boundary condition $V_1({\underline{x}}) = 0$; the dashed green lines correspond to the solution computed without state space transformation for ${\underline{x}} = -20$. Bottom: The numerical solutions $V_1$ (left) and I (right) computed on $(-10,10)$ for Scenario 1 (red), 2 (dashed green) and 3 (blue) and on $(-20,20)$ for Scenario 2 (solid green)

To illustrate the results obtained using the collocation method, we show at the bottom part of Fig. 1 the value and feedback functions obtained for the Scenarios 1–3. The number of collocation nodes for these calculations are $n_z = 40$ for the collocation on the domain $z \in [0.5, {\bar{z}}]$ (i.e. $x\ge 0$) and $n_z = 25$ on the domain $z \in [{\underline{z}}, 0.5]$ (i.e. $x \le 0$).

Note that in light of the considered parameter values we have $\gamma _I \in ({\underline{\gamma }}_I, {{\bar{\gamma }}}_I)$ in Scenario 1, whereas $\gamma _I > {{\bar{\gamma }}}_I$ in Scenario 2 and $\gamma _I < {\underline{\gamma }}_I$ in Scenario 3. Hence, in Scenario 1 $x = 0$ is a stable fixed point of the state dynamics under optimal investment and we impose boundary conditions , where $V_1(0)$ is given by (3.17), for both the collocations run on $z \in [{\underline{z}}, 0.5]$ and $z \in [0.5, {\bar{z}}]$. In the other two scenarios no boundary conditions were imposed on the domain $z \in [{\underline{z}}, 0.5]$ (Scenario 2) respectively on the domain $z \in [0.5, {\bar{z}}]$ (Scenario 3) and, following our description above, the obtained values of $\widehat{V}_1(0.5)$ respectively were used as the boundary conditions on the second domain. For Scenario 2 we have calculated the value and control functions not only for our standard value ${\underline{z}} = z(-10)$, but also for ${\underline{z}} = z(-20)$. The reason for also considering a smaller lower bound of the domain is that ${{\widehat{V}}}_1(z(-10))$ still has a substantial positive value, whereas it is close to zero at $z(-20)$. Comparing the dashed green lines (${\underline{z}} = z(-10)$) and the solid green lines (${\underline{z}} = z(-20)$) however shows that the value functions calculated under these two domain specifications are indistinguishable, and also the optimal control determined under ${\underline{z}} = z(-10)$ is very close to that obtained for ${\underline{z}} = z(-20)$. Although hard to discern from the figure in the right lower panel, we like to point out that not only for Scenario 1, where according to our theoretical results we know that a jump in the control occurs at $x = 0$, but also for Scenario 3 the calculated optimal investment function exhibits a jump at $x=0$.

Considering still the lower panels of Fig. 1, we observe that the optimal investment function in Scenario 2 with ${\underline{z}} = z(-20)$ crosses the line $I=0$ at approximately $x = -14$. This indicates that for the calculated value functions ${{\widehat{V}}}_2(z,0) - {{\widehat{V}}}_1(z) < 0$ holds for z close to the lower bound ${\underline{z}}$. Since the value after successful innovation cannot be lower than that in mode $m_1$ before the innovation, this is a numerical artifact suggesting that the value of ${{\widehat{V}}}_1$ is overestimated for z close to ${\underline{z}}$. To address this issue, we show in the upper part of Fig. 1 results for Scenario 2 on different domains where the boundary condition ${{\widehat{V}}}_1({\underline{z}}) = 0$ is imposed; for comparison the results computed with the finite difference method are shown in Fig. 3. For ${\underline{z}} = z(-20)$ imposing the boundary condition yields an optimal investment function which for values of $z(-10) \le z \le 0.5$ closely resembles the one calculated without a boundary condition. Furthermore, the domain on which investment is positive is substantially larger if the boundary condition is imposed. If the boundary condition is however imposed at ${\underline{z}} = z(-15)$ or ${\underline{z}} = z(-10)$ the generated investment functions are more volatile and in particular for ${\underline{z}} = z(-10)$ yield unsatisfactory results. This can be attributed to the fact that the distribution of the collocation points is getting coarser away from $x(z=0.5)=0$ due to the transformation (3.1), (3.2) which results in low accuracy of the scheme for large domains. Nevertheless, without the transformation the policy iteration does not converge, unless a good initial guess for the policy iteration is used (see next paragraph below). Furthermore, we observe in Fig. 1 that the results get worse if a homogeneous Dirichlet boundary condition is imposed since in Scenario 2 the actual value of $V_1(x)$ at $x= -10$ and $x = -15$ is still substantially larger than 0.

In order to improve the performance of the collocation method in Scenario 2, we have computed the solution with the collocation method for this case without applying the state space transformation (3.1), (3.2) (also imposing the boundary condition $V_1(-20) = 0$). To improve the convergence of the policy iteration we employ a preconditioning approach: we use the approximate value function calculated with the state space transformation as the initial guess (${\overline{u}}^0$ in Algorithm 3.1) in the policy iteration. The preconditioing step solves convergence issues and the computed solution agrees with the results of other methods. The computed value and control are depicted as the dashed green line in the top graphs in Fig. 1. Unlike in the case of the method with transformation, the computed control does not exhibit oscillation on $[-20,-10]$.

4 Discretization by the Finite Difference Method

This section is devoted to the description of the finite difference approximation of (1.1). For the sake of brevity, we do not consider the finite difference approximation of $V_2$ since the results of the finite difference approximation are very similar to those obtained by the finite element method in Sect. 6.1 below.

The finite difference method requires the spatial domain to be of finite size. Thus, we consider (1.1) for $x\in ({\underline{x}},\bar{x})$ with ${\underline{x}}<0$ and $\bar{x}\ge {\tilde{x}}$, see Proposition 2.1. We impose the following Dirichlet boundary conditions

$$\begin{aligned} V_1({\underline{x}})=0\quad \text{ and }\quad V_1(\bar{x})= \bar{x}+ c - \frac{\xi }{\gamma _I}I^{nc}, \quad \bar{x}\ge {\tilde{x}}. \end{aligned}$$

(4.1)

We introduce an equidistant grid $\Pi ^h_1:=\{x_0,\ldots ,x_{M-1}\}$ with mesh size $h :=(\bar{x}-{\underline{x}})/(M-1)$, where

$$\begin{aligned} x_i = {\underline{x}}+ ih\quad i = 0,1,\ldots ,M-1\,. \end{aligned}$$

(4.2)

4.1 Upwind Finite Difference Approximation

For $i = 0,\ldots ,M-1$, we denote $b_1^{i}(I):=b_1(x_i,I)$, $p^i_0:=p_0(x_i)$, $p_2:=p_2(I)$, and $f_1^i:=f_1(x_i)$. The finite difference approximation of the value function $V_1$ on the grid (4.2) is denoted by $V_1^i \approx V_1(x_i)$.

We approximate $\partial _xV_1$ by backward, central, or forward differences, which are denoted by $\mathrm {D}_{x}^{-}$, $\mathrm {D}_{x}^{0}$, and $\mathrm {D}_{x}^{+}$, respectively, and defined by

$$\begin{aligned} \mathrm {D}_{x}^{-}V_1^i:=\frac{V_1^{i}-V_1^{i-1}}{h},\quad \mathrm {D}_{x}^{0}V_1^i:=\frac{V_1^{i+1}-V_1^{i-1}}{2h},\quad \mathrm {D}_{x}^{+}V_1^i:=\frac{V_1^{i+1}-V_1^{i}}{h}. \end{aligned}$$

(4.3)

We also introduce the upwind finite difference approximation $\mathrm {D}_{x}^{\pm }V_1^i\approx \partial _xV_1(x_i)$ defined by

$$\begin{aligned} \displaystyle \mathrm {D}_{x}^{\pm }V_1^i&:={\left\{ \begin{array}{ll} \mathrm {D}_{x}^{+}V^i_1, &{} \text{ if } b^i_1(I)>0, \\ \mathrm {D}_{x}^{-}V^i_1,&{} \text{ if } b^i_1(I)\le 0. \end{array}\right. } \end{aligned}$$

(4.4)

Once a numerical approximation of the value function $V_2$ is computed we denote its values on the grid points at $y=0$ by $V_2^{i,0} \approx V_2(x_i,0)$, $i=0,\dots , M-1$ and set the vector $\bar{V}_2:=(V_2^1,\dots , V_2^{M-2})$.

To obtain the finite difference approximation of $V_1$, we replace the derivative in (1.1) by the upwind finite difference (4.4). We set $\bar{V}_1:=(V_1^{1},\dots ,V_1^{M-2})$ and by (4.1) impose

$$\begin{aligned} V_1^{0} = 0 \quad \text{ and }\quad V_1^{M} = \bar{x}+ c - \frac{\xi }{\gamma _I}I^{nc}. \end{aligned}$$

To obtain the finite difference approximation of the optimal control $I^*$, we replace the derivative in (3.14) by the central finite difference $\mathrm {D}_{x}^{0}V_1^ i$, see (4.3). We set $\bar{I}:=(I^1, \dots , I^{M-2})$ with

$$\begin{aligned} I^i =\frac{\gamma _I}{\xi } \frac{\big (V_2^{i,0}-V_1^{i}\big )}{\mathrm {D}_{x}^{0}V_1^{i}},\; i = 1,\ldots ,M-2. \end{aligned}$$

(4.5)

Thus, to (1.1), we associate the following discrete system for all nodes $i = 1,\ldots ,M-2$

$$\begin{aligned} rV_1^i = L_1^M(\bar{V}_1, \bar{V}_2, x_i,\bar{I}) + f_1^i, \end{aligned}$$

(4.6)

where

$$\begin{aligned} L_1^M(\bar{V}_1, \bar{V}_2, x_i,\bar{I}):=b_1^i(I^i)\mathrm {D}_{z}^{\pm }V_1^i - p_0^i V_1^i +p_2(I^i)(V_2^{i,0}-V_1^i). \end{aligned}$$

(4.7)

In the discrete system (4.6), the optimal control $\bar{I}$ given by the formula (4.5) depends implicitly on $\bar{V}_1$ and the central difference of $\bar{V}_1$. Analogically to the collocation method, we use the policy iteration procedure to break the implicit character of the system (4.6). Altogether, the upwind scheme and the policy iteration give the following algorithm:

Algorithm 4.1

(FDM) Set $\bar{I}^{(0)}:=I^{\mathrm {nc}}$ and for $n=1,\dots$ proceed as follows:

Policy evaluation: Find $\bar{V}_1^{(n)}:=\big (V_1^{1,(n)},\ldots ,V_1^{M-2,(n)}\big )$, such that

$$\begin{aligned} rV_1^{i,(n)} = L_1^{M}(\bar{V}_1^{(n)},\bar{V}_2,x_i,\bar{I}^{(n-1)}) + f_1^i. \end{aligned}$$

(4.8)

Policy improvement: Update the control $\bar{I}^{(n)}:=\big (I^{1,(n)},\ldots ,I^{M-2,(n)}\big )$ with

$$\begin{aligned} I^{i,(n)} = \frac{\gamma _I}{\xi } \frac{(V_2^{i,0}-V_1^{i,(n)})}{\mathrm {D}_{x}^{0}V_1^{i,(n)}}. \end{aligned}$$

(4.9)

Stopping criterion: Repeat the loop until

$$\begin{aligned} \frac{\displaystyle \max _{i = 1,\ldots ,M-2.}|I^{i,(n)}-I^{i,(n-1)}|}{\displaystyle \max _{i = 1,\ldots ,M-2.}|I^{i,(n)}|}\le 10^{-6}. \end{aligned}$$

4.1.1 Numerical Experiments

To obtain the numerical approximation of $V_1$ we use the values of the numerical approximation of $V_2$ at $y=0$ computed by the finite element method with mesh size $h = 2^{-11}$ (see Sect. 6.1 below) in all numerical experiments in this section.

Effect of $V_2$ on the solution As mentioned above, we use a value function $V_2$ computed on a (fixed) coarse mesh, i.e. the approximation of $V_2$ is given on $M = 2^{11}$ uniformly distributed grid-points. To compute a value function $V_1$ on a finer mesh, we have to interpolate the available data for $V_2$ over the (finer) mesh on which $V_1$ is computed. Below we examine the effect of different interpolation methods on the numerical solution in Scenario 2. The value function $V_1$ is computed for $x\in (-20,10)$ using Algorithm 4.1.

We compare the effect of linear interpolation and cubic interpolation (see [20]) of $V_2$ on the control. We observe in Fig. 2 (bottom) that algorithm with linearly interpolated $V_2$ produces a control with irregular oscillations while control obtained with cubic interpolation is smooth; note that in Fig. 2 (top) we display the central difference approximation of the gradient of $V_2$. Although, the different interpolation methods for $V_2$ do not influence the overall convergence of the policy algorithm, it is preferable to avoid the oscillations in the control caused by the linear interpolation of $V_2$. Therefore, we employ cubic interpolation of $V_2$ in the remaining numerical experiments.

Fig. 2

Top: Gradient of $V_2$ (left) and the gradient of $V_2$ obtained with different interpolation methods (right). Bottom: Gradient of the control computed with $V_2$ obtained by linear interpolation (left) and the gradient of the controls computed with $V_2$ obtained respectively by linear and cubic interpolation (right)

Effect of the domain on the solution In this second batch of numerical experiments, we consider three domains $\Omega _{10}:=(-10,10)$, $\Omega _{15}:=(-15,10)$, and $\Omega _{20}:=(-20,10)$. We compute the solution for the Scenarios 1, 2, and 3 under different configurations, everything else being equal.

We made the following observations:

• Scenario 2. The solution obtained changes significantly with the domain, see Fig. 3. The reason is that for $x \ge -15$, the value function $V_1(x)$ is far from zero. Thus, if we impose a zero Dirichlet boundary condition at $x = -15$ or above, then the solution obtained deviates significantly from the exact solution.

• Scenarios 1 and 3. The solution obtained is not affected by the change of domain or boundary conditions. At $x = -7$, the value function is almost zero, see Fig. 3. Thus, imposing a zero Dirichlet boundary condition at $x = -10$ does not have a significant effect on the solution.

In general, we observe that smaller values of $\gamma _B$ require the use of larger domains.

Fig. 3

Top: Effect of the domain on $V_1$ (left) and I (right) for Scenario 2 with $M = 2^{15}$. Bottom: The numerical solutions $V_1$ (left) and I (right). are computed in $(-10, 10)$ for Scenarios 1 (red) and 3 (blue); in $(-20,20)$ for Scenario 2 (green)

Discontinuity of the control In [9, Proposition 2], the authors argue that the control is discontinuous at $x = 0$ for certain values of the model parameters, cf., also the discussion at the beginning of Sect. 3.3. Our numerical experiments confirm this statement. The numerical experiments for decreasing grids size in Fig. 4 confirm the theoretical prediction, i.e., the control exhibits a discontinuity at $x = 0$ in Scenarios 1 and 3, which is reflected by the growing values of the gradient of the control. In the case of discontinuity the control is not Lipschitz continuous, which affects the convergence of the policy iteration algorithm; we discus this issue in more detail in Sect. 4.1.2 below.

Fig. 4

Gradient of the control computed by Algorithm 4.1 for Scenarios 1, 2, and 3 (from left to right)

4.1.2 Effect of the Discretization of the Control

We examine the effect of different approximations of the optimal control (3.14) on the convergence behavior of the policy iteration algorithm. The computations below were performed on a fixed grid with $M = 2^{15}$ grid-points. In addition to the central difference approximation (4.5), we consider the backward difference, forward difference and upwind approximation, i.e, the central difference $\mathrm {D}_{x}^{0}$ in the denominator of (4.5) is replaced by $\mathrm {D}_{x}^{-}$, the forward difference $\mathrm {D}_{x}^{+}$, and the upwind difference $\mathrm {D}_{x}^{\pm }$, respectively (with the corresponding modification of the formula (4.9) in the policy improvement step of Algorithm 4.1).

We also consider the following alternative to (4.5)

$$\begin{aligned} I^{i} = \frac{\gamma _I }{2\xi } \Bigg (\frac{V_2^{i,0}-V_1^{i}}{\mathrm {D}_{x}^{-}V_1^{i}} + \frac{V_2^{i,0}-V_1^{i}}{\mathrm {D}_{x}^{+}V_1^{i}}\Bigg ). \end{aligned}$$

(4.10)

which is an average of the respective formulas obtained by the forward and backward differences. The averageing has a regularizing effect in the case of discontinuous control (cf. Fig. 4) and improves the convergence behavior even when the control is not Lipschitz continuous.

Fig. 5

Error evolution of the policy iteration for Scenarios 1, 2, and 3 (from left to right) for difference approximations of the control

The convergence plots of the policy iteration with the considered finite difference approximations of the control for Scenarios 1–3 are displayed in Fig. 5. The results can be summarized as follows:

• The upwind finite difference approximation (the results are not displayed in Fig. 5): this discretization yields control that is optimal on the discrete level; nevertheless, we observe that the policy iteration diverges with the upwind discretization.

• The forward finite difference: the algorithm converges in Scenario 1 and 2 but diverges in Scenario 3.

• The backward finite difference: the algorithm converges in all scenarios, nevertheless one may expect similar problems with convergence as for the forward difference approximation in general (i.e., for different model parameters).

• The central and “averaged” finite differences: in both cases the algorithm converges in every scenario, the “averaged” difference approximation exhibits slightly better convergence behavior overall.

Finally, it is apparent from Fig. 5 that in Scenario 2 (i.e., when there is no discontinuity in the control) the approximation of the control has only minor influence on the convergence behavior of the policy iteration algorithm.

4.1.3 Experimental Rate of Convergence

Next, we study the rate of convergence of the finite difference numerical approximation; the numerical solution is computed on $\Omega _{10}$ in Scenarios 1,3 and on $\Omega _{20}$ in Scenario 2. Since no analytical solution is known for the given problem, we determine the rate of convergence by using a reference solution computed on a fine grid.

We compute a sequence of approximations for $\kappa = 0,\ldots ,5$ on uniformly spaced grids with $M(\kappa ) = 2^{10+\kappa }$ grid-points denoted as $\bar{V}_{1,M(\kappa )}:=(V_{1,M(\kappa )}^1, \dots , V_{1,M(\kappa )}^{M(\kappa )})$ and $\bar{I}_{M(\kappa )} = (I_{M(\kappa )}^1, \dots , I_{M(\kappa )}^{M(\kappa )})$.

We approximate the exact solution $V_1$, I by the respective reference solutions $\bar{V}_{1,M_{\text {ref}}}$, $\bar{I}_{M_{\text {ref}}}$ on the uniformly spaced grid with $M(\kappa )<M_{\text {ref}}$ grid-points. On a reference domain $\Omega _{r}=(-9,9)$, the $L^\infty (\Omega _{r})$-errors $\Vert V_1 - \bar{V}_{1,M(\kappa )}\Vert _{L^\infty (\Omega _{r})}$ and $\Vert I - \bar{I}_{{M(\kappa )}}\Vert _{L^\infty (\Omega _{r})}$ are computed from piecewise linear interpolation of the numerical solution on the respective finite difference grids.

The rate of convergence in the $L^\infty$-norm (analogically in the $L^2$-norm) is determined as

$$\begin{aligned} \frac{\ln \big (\Vert \bar{V}_{1,M_{\text {ref}}} - \bar{V}_{1,{M(\kappa -1)}}\Vert _{L^\infty (\Omega _{r})}\big ) - \ln \big (\Vert \bar{V}_{1,M_{\text {ref}}} - \bar{V}_{1,{M(\kappa )}}\Vert _{L^\infty (\Omega _{r})}\big )}{\ln (h(\kappa -1)) - \ln (h(\kappa ))},\quad \kappa = 1,\ldots ,5. \end{aligned}$$

The results are displayed in Table 3; we observe that the experimental convergence rate is of order about 1 for the variables $V_1$ and I. The convergence rate in the $L^\infty$-norm is reduced to about 1/2 in Scenarios 1 and 3, which reflects the fact that the control is discontinuous. In addition, in Table 4 we display the experimental convergence rates for the “averaged” discretization of the control (4.10) (which has the advantage of a slightly faster convergence of the policy iteration algorithm); the observed convergence rates are similar to those from Table 3, the accuracy of the approximation is slightly worse in Scenario 2.

Table 3 Experimental rate of convergence of the finite difference method

Table 4 Experimental convergence rate for the “averaged” approximation of the control (formula (4.10))

5 Weighted Essentially Non-oscillatory (WENO) Finite–Difference Approximation

In this section we propose an algorithm for the computation of the value function that employs higher-order finite difference discretization in space. The scheme is similar to the policy iteration Algorithm 4.1 with the exception that the value function in the policy evaluation step is obtain as a steady state of a time dependent version of (1.1), i.e., the solution of (1.1) is obtained as $V_1(x) = \lim _{t\rightarrow \infty }w(t,x)$ where w solves the transient PDE $\partial _t w + r w = \sup _{I} {\mathcal {L}}_1 (w, V_2,x, I) + f_1(x)$.

Given the control I we consider the time-dependent counterpart of (1.1) as

$$\begin{aligned} \left\{ \begin{aligned} \partial _t u(t,x)&= \Phi (u, I, x),\quad (t,x)\in (0,\infty )\times \mathbb {R}, \\ u(0,x)&= g(x), \quad x\in \mathbb {R}, \end{aligned} \right. \end{aligned}$$

(5.1)

where

$$\begin{aligned} \Phi (u, I, x):=b_1(x, I)\partial _x u(t,x) - (p_0(x) + p_2(I) + r)u(t,x) + p_2(I)V_2(x, 0) + f_1(x). \end{aligned}$$

(5.2)

Spatial semi-discretization We employ the notation from the previous section, in particular we denote $x_i = i h$, where h is the mesh size. We perform the spatial semi-discretization of (5.1) on the grid (4.2). We use a WENO5 interpolation procedure that employs a stencil with 6 nodes [31],

$$\begin{aligned} \{x_{i-3},\ldots ,x_{i+2}\} \quad \text{ or } \quad \{x_{i-2},\ldots ,x_{i+3}\}\,, \end{aligned}$$

(5.3)

depending on the direction of the velocity, see (5.5). We assign the values to grid points near the boundary of $\Omega _r$; via the boundary condition given by

$$\begin{aligned} g(x):={\left\{ \begin{array}{ll} \dfrac{ c - \dfrac{\xi }{\gamma _I}I^{nc}}{1 + \exp {(-3x)}}, \quad \text{ if } x < 0,\\ \dfrac{x + c - \dfrac{\xi }{\gamma _I}I^{nc}}{1 + \exp {(-3x)}},\quad \text{ if } x \ge 0. \end{array}\right. } \end{aligned}$$

(5.4)

The choice of the function (5.4) is motivated by the fact that the solution to (1.1) is asymptotically 0 for $x\rightarrow -\infty$ and takes the form (2.2) when $x\rightarrow +\infty$

Below, we describe the WENO spatial discretization in the case that $b_1^i(I)\ge 0$, which uses the stencil biased to the left, i.e. on the nodes $\{x_{i-3},\ldots ,x_{i+2}\}$; the case $b_1^i(I) <0$ uses the the nodes $\{x_{i-2},\ldots ,x_{i+3}\}$ and is constructed analogously, cf., [31].

We denote the approximation at the nose $x_i$ as $u_i(t)\approx u(t, x_i)$ and define the following interpolated values

$$\begin{aligned} {[}u_i(t)]_1&= \frac{1}{h}\left( -\frac{1}{3} u_{i-1}(t)-\frac{1}{2} u_{i}(t) + u_{i-1}(t)- \frac{1}{6} u_{i+2}(t)\right) , \\ {[}u_i(t)]_2&= \frac{1}{h}\left( \frac{1}{6} u_{i-2}(t)-u_{i-1}(t) + \frac{1}{2} u_{i}(t)+ \frac{1}{3} u_{i+1}(t)\right) , \\ {[}u_i(t)]_3&= \frac{1}{h}\left( -\frac{1}{3} u_{i-3}(t) + \frac{3}{2}u_{i-2}(t) - 3u_{i-1}(t) + \frac{11}{6} u_{i}(t)\right) . \end{aligned}$$

We define the approximation of $\partial _x u(t,x_i)$ by

$$\begin{aligned} \mathrm {D}_{x}^{w}u_i(t) = \omega _1 [u_i(t)]_1 + \omega _2 [u_i(t)]_2 + \omega _3 [u_i(t)]_3, \end{aligned}$$

(5.5)

where the weights $\omega _i$ are given by

$$\begin{aligned} \omega _i&= \frac{\alpha _i}{\alpha _1 + \alpha _2 + \alpha _3},\quad i = 1,2,3,\quad \text{ with } \nonumber \\ \alpha _i&= \frac{d_i}{c_e + \beta _i},\quad i = 1, 2, 3, \quad d_1 = 3/10,\;d_2=3/5,\; d_3 = 1/10. \end{aligned}$$

(5.6)

The parameter $c_e$ in (5.6) is introduced to avoid a denominator with zero value, in the numerical experiments below we set $c_e = 10^{-6}$. The parameters $\beta _i$ in (5.6), the so-called smoothness indicators, are defined as

$$\begin{aligned} \beta _1&= \frac{13}{12}({\bar{u}}^i - 2 {\bar{u}}^{i+1} + {\bar{u}}^{i+2})^2 + \frac{1}{4}(3{\bar{u}}^i - 4 {\bar{u}}^{i+1} + {\bar{u}}^{i+2})^2,\\ \beta _2&= \frac{13}{12}({\bar{u}}^{i-1} - 2 {\bar{u}}^{i} + {\bar{u}}^{i+1})^2 + \frac{1}{4}({\bar{u}}^{i-1} - 4 {\bar{u}}^{i+1})^2,\\ \beta _3&= \frac{13}{12}({\bar{u}}^{i-2} - 2 {\bar{u}}^{i-1} + {\bar{u}}^{i})^2 + \frac{1}{4}({\bar{u}}^{i-2} - 4{\bar{u}}^{i-1} + 3{\bar{u}}^{i})^2, \end{aligned}$$

where ${\bar{u}}^i = (u^i - u^{i-1})/h$.

Hence, the spatial semi-discretization based on the upwind WENO5 interpolation procedure leads to the following system of ordinary differential equations

$$\begin{aligned} \left\{ \begin{aligned} \partial _tu_i(t)&= \Phi _i(u_i, I), \quad t\in (0,\infty ), \\ u_i(0)&= g(x_i), \end{aligned} \right. \end{aligned}$$

(5.7)

where

$$\begin{aligned} \Phi _i(u_i, I):=b_1(x_i,I)\mathrm {D}_{x}^{w}u_i - (p_0(x_i) + p_2(I) + r)u_i + p_2(I)V_2(x_i, 0) + f_1(x_i). \end{aligned}$$

Time-discretization Given a time step $\Delta t$ and a sequence of discrete time levels $t_\ell = \ell \Delta t$ we denote the fully discrete approximation at $(t_\ell , x_i)$ as $u_i^\ell \approx u_i(t_\ell ))$. The time-discretization of (5.7) is based on the Heun method, which is an explicit one-step 2-stage Runge–Kutta method of order 2 (RK22). The choice of the time-discretization is motivated by the stability properties of the RK22 scheme [30]. Since the scheme is explicit we set $\Delta t = h/2$ to preserve the associated CFL condition.

The fully discrete scheme for the approximation of (5.1) then reads as follows: given an initial guess $\{u_i^0\}_{i=1}^{M-2}$ compute for $\ell =1, \dots$

$$\begin{aligned} u_i^\ell = u_i^{\ell -1} + \Delta t\frac{ (k_1 + k_2)}{2} \qquad i=1,\dots ,M-2\,, \end{aligned}$$

(5.8)

where

$$\begin{aligned} \begin{aligned} k_1&= \Phi _i(u_i^{\ell -1}, I),\\ k_2&= \Phi _i(u_i^{\ell -1} + \Delta t k_1, I)\,. \end{aligned} \end{aligned}$$

We then obtain the following modification of the policy iteration algorithm:

Algorithm 5.1

(WENO) Start with the initial guess $\bar{I}^{(0)}:=I^{\mathrm {nc}}$ and for $n=1,\dots$ iterate as follows:

Policy evaluation:

• 1. Set $u_i^0 =V_1^{i, (n-1)}$ for $n>1$ and $V_1^{i, (0)} = g(x_i)$ for $i=-3,\dots ,M+1$.

• 2. For $\ell =1,\dots$ compute $\{u_i^\ell \}_{i=1}^{M-2}$ using (5.8) with $I\equiv \bar{I}^{(n-1)}$ until $\displaystyle \max _{i = 1,\ldots ,M-2.}|u_i^{\ell }-u_i^{\ell -1}|\le 10^{-6}$.

• 3. Set $V_1^{i,(n)}= u_i^\ell$, $i=1,\dots ,M-2$ and proceed to the policy improvement step.

Policy improvement: Update the control $\bar{I}^{(n)}:=\big (I^{1,(n)},\ldots ,I^{M-2,(n)}\big )$ with

$$\begin{aligned} I^{i,(n)} = \frac{\gamma _I}{\xi } \frac{(V_2^{i,0}-V_1^{i,(n)})}{\mathrm {D}_{x}^{0}V_1^{i,(n)}}. \end{aligned}$$

Stopping criterion: Repeat the loop until

$$\begin{aligned} \frac{\displaystyle \max _{i = 1,\ldots ,M-2.}|I^{i,(n)}-I^{i,(n-1)}|}{\displaystyle \max _{i = 1,\ldots ,M-2.}|I^{i,(n)}|}\le 10^{-6}. \end{aligned}$$

Numerical results We discuss the results for Scenario 1, the behavior of Algorithm 5.1 in the remaining scenarios was similar. We observe very good agreement between the solutions computed with the upwind finite difference scheme and the WENO scheme, see Fig. 6. In Fig. 7 we display the error between the solution of the WENO scheme for decreasing mesh size and the solution of the upwind finite difference scheme on a very fine mesh with $M=2^{20}$ grid points; we observe a linear convergence for the approximation of $V_1$ and convergence of order $\frac{1}{2}$ for the control in the maximum norm. Algorithm 5.1 requires about $30-40$ time more policy iterations than Algorithm 4.1, in Scenario 1 443 policy iterations were required to achieve the convergence criterion. As expected, the number of time-steps required to reach the steady-state in each policy iteration, decreases in later states of the policy iteration algorithm, only two time-steps per policy iteration are required after the 10th policy iteration, see Tables 3 and 4. The advantage of the time-stepping approach in Algorithm 5.1 is that it does not require the solution of a linear system of equations, nevertheless in the present setting Algorithm 4.1 was considerably faster in terms of CPU time due to the lower policy iteration count.

Fig. 6

Comparison of the solutions obtained using the upwind finite difference scheme and the WENO scheme i.e. Algorithms 4.1 and 5.1 for $M=2^{11}$, Scenario 1

Fig. 7

Scenario 1: difference between the solution of the upwind finite difference scheme with $M = 2^{20}$ and the solution of the WENO scheme with $M = 2^{11}, 2^{12}, 2^{13}$ (left $V_1$ and right I)

Table 5 Number of time-marching iteration (WENO5 + RK22) required at each policy iteration

To reduce the number of policy iterations for the WENO scheme we employ a preconditioning approach: we use the control computed by Algorithm 4.1 as an initial guess for Algorithm 5.1. We obtained a reduction in the number of policy iterations as well as in the total number of time-steps, see Table 6.

Table 6 Number of time-marching iteration (WENO5 + RK22) required at each policy iteration with control preconditioning

6 Discretization by the Stabilized Finite Element Method

In this section we introduce stabilized finite element (FEM) schemes for the approximation of (1.1)–(1.2). We note that, in contrast to the collocation method, we do not employ the transformation (3.1) and, similarly as in the previous section, we compute the numerical solution on domains of finite size. We consider (1.1) for $x\in (-20,20)$ or $x\in (-10,10)$ and (1.2) for $(x,y) \in (-20,0)\times (0,20)$ and impose suitable conditions along the boundary of the respective domains.

Throughout the section we use the following notation. By $L^2(\Omega )$ we denote the standard Lebesgue space of square integrable functions on $\Omega$. For $u,v\in L^2(\Omega )$ we denote the ${L}^2$-inner product by $(u,v):=\int _\Omega u(x)v(x)\,\mathrm {d}x$ with the corresponding $L^2$-norm $\Vert u\Vert _{L^2(\Omega )} :=(u,u)^{1/2}$.

6.1 Discretization of the Value in Mode $m=2$

We note that for $x\ge 0$ by Proposition 2.1 the solution of the HJB Eq. (1.2) is given explicitly by (2.1). Hence, similarly as in Sect. 3, we only consider numerical approximation of $V_2$ for $x< 0$ and impose a Dirichlet boundary condition at $x=0$ via (2.1). In addition, in order to solve $V_1$ we only need to obtain the solution $V_2$ at $y=0$. Therefore, to obtain a finite element approximation of $V_2$ we solve the HJB Eq. (1.2) on the bounded domain $\Omega ^- = (-20,0)\times (0,20)$. The boundary of $\Omega ^-$ is denoted as $\partial \Omega ^-$ and the boundary at $x=0$ is denoted as $\Gamma _D :=\partial \Omega ^- \cap (0,y)$, $y\in [0,20]$.

For simplicity we construct the finite element scheme using uniform triangulations $\mathcal {T}_h$ of $\Omega ^-$. For a fixed mesh size parameter $h>0$ the triangulation $\mathcal {T}_h$ is obtained by partitioning $\Omega ^-$ into uniform squares with side h and each square is then sub-divided into four equally sized right-angled triangles. Given a triangulation $\mathcal {T}_h$ of $\Omega ^-$ we denote by $\mathbb {V}_h(\mathcal {T}_h)$ the standard finite element space of continuous functions which are piecewise linear with respect to the triangulation $\mathcal {T}_h$. The set of nodes $\{\mathbf {x}_\ell \equiv (x_\ell , y_\ell )\}_{\ell =1}^{L}$ of $\mathcal {T}_h$ is denoted as $\mathcal {N}_h$. Note that $\mathbb {V}_h(\mathcal {T}_h) = \mathrm {span}\{\varphi _\ell ,\,\, \ell =1,\dots ,L\}$ where $\{\varphi _\ell \}_{\ell =1}^{L}$ are the standard nodal basis functions, s.t. $\varphi _\ell (\mathbf {x}_m) = \delta _{ml}$ for all $\mathbf {x}_m\in \mathcal {N}_h$. We also introduce a finite element space of functions that satisfy the homogeneous Dirichlet boundary condition on $\Gamma _D$ as $\mathbb {V}_{h,0} = \{\phi _h \in \mathbb {V}_h(\mathcal {T}_h),\,\, \phi _h | \Gamma _D = 0\}$.

The standard nodal interpolation operator is denoted as $\mathcal {I}_h: C({\overline{\Omega }}^-) \rightarrow \mathbb {V}_h(\mathcal {T}_h)$, where for $v\in C({\overline{\Omega }}^-)$ the interpolant $\mathcal {I}_h v \in \mathbb {V}_h(\mathcal {T}_h)$ is defined via the relation $\mathcal {I}_h v(\mathbf {x}_\ell ) = v(\mathbf {x}_\ell )$, $\forall \, \mathbf {x}_\ell \in \mathcal {N}_h$, i.e., $\mathcal {I}_h v (\mathbf {x})=\sum _{\ell =1}^L v(\mathbf {x}_\ell ) \varphi _\ell (\mathbf {x})$ for $\mathbf {x}\in {\overline{\Omega }}^-$. For $u,v \in C({\overline{\Omega }}^-)$ define the discrete $L^2$-inner product for $\Omega ^{-} \subseteq \mathbb {R}^2$ as

$$\begin{aligned} \big (u,v\big )_h = \int _{\Omega ^-} \mathcal {I}_h(uv)(x) \mathrm {d}x = \sum _{T\in \mathcal {T}_h}\frac{|T|}{3} \sum _{i=1}^3 u(\mathbf{v}_{T,i})v(\mathbf{v}_{T,i})\,, \end{aligned}$$

(6.1)

where $\mathbf{v}_{T,1},\mathbf{v}_{T,2},\mathbf{v}_{T,3} \in \mathcal {N}_h$ denote the vertices of the triangle $T\in \mathcal {T}_h$.

The finite element space of piecewise linear continuous functions only satisfies $\mathbb {V}_h(\mathcal {T}_h) \subset H^1(\Omega ^-)$, i.e., the second-order derivatives are not defined for $v_h \in \mathbb {V}_h(\mathcal {T}_h)$. For $v_h \in \mathbb {V}_h(\mathcal {T}_h)$ we introduce the discrete second-order operator $\delta _{xx,h}^2:\mathbb {V}_h(\mathcal {T}_h)\rightarrow \mathbb {V}_h(\mathcal {T}_h)$, $\delta _{xx,h}^2\approx \partial _{xx}^2$ as

$$\begin{aligned} (-\delta _{xx,h}^2v_h, \phi _h )_h = (\partial _xv_h, \partial _x\phi _h )\qquad \forall \, \phi _h \in \mathbb {V}_h(\mathcal {T}_h)\,, \end{aligned}$$

(6.2)

and similarly we define $\delta _{yy,h}^2\approx \partial _{yy}^2$. Consequently the discrete Laplace operator $\Delta _h :=\delta _{xx,h}^2+ \delta _{yy,h}^2: \mathbb {V}_h(\mathcal {T}_h) \rightarrow \mathbb {V}_h(\mathcal {T}_h)$ satisfies

$$\begin{aligned} (-\Delta _h v_h, \phi _h )_h = (\nabla v_h, \nabla \phi _h )\qquad \forall \, \phi _h \in \mathbb {V}_h(\mathcal {T}_h)\,. \end{aligned}$$

(6.3)

Let $\{\phi _\ell \}_{\ell =1}^{L}$ be the set of nodal basis functions of $\mathbb {V}_h(\mathcal {T}_h)$, i.e., $\mathbb {V}_h(\mathcal {T}_h)=\mathrm {span}\{\phi _1, \dots , \phi _L\}$. Hence, any $v_h\in \mathbb {V}_h(\mathcal {T}_h)$ can be expressed as $v_h(x) = \sum _{\ell =1}^L v_\ell \phi _\ell (x)$ with coefficients $v_\ell \in \mathbb {R}$. By the definition of the discrete inner product (6.1) and the fact that $\phi _\ell (\mathbf {x}_m) = \delta _{\ell m}$ for $\mathbf {x}_m \in \mathcal {N}_h$, we deduce from (6.2) that

$$\begin{aligned} \partial _{xx}^2v_h(\mathbf {x}_\ell ) = 3|\mathrm {supp}{(\phi _\ell )}|^{-1} (\partial _xv_h, \partial _x\phi _\ell )\quad \forall \, \mathbf {x}_\ell \in \mathcal {N}_h, \end{aligned}$$

where $\mathrm {supp} (\phi _\ell ) = \{x \in \Omega ^-,\,\, \phi _\ell (x) \ne 0\}$. Analogically (6.3) implies that

$$\begin{aligned} \Delta _h v_h(\mathbf {x}_\ell ) = 3|\mathrm {supp}{(\phi _\ell )}|^{-1} (\nabla v_h, \nabla \phi _\ell )\quad \forall \, \mathbf {x}_\ell \in \mathcal {N}_h, \end{aligned}$$

cf. [13, 17, 27].

The finite element approximation of the continuous problem is then constructed as follows. The finite element solution $V_{2,h}\in \mathbb {V}_{h}(\mathcal {T}_h)$ satisfies the Dirichlet boundary condition at $\Gamma _D$, that is $V_{2,h}(\mathbf {x}_\ell ) = V_2(0, y_\ell )$ for $\mathbf {x}_\ell = (0, y_\ell ) \in \mathcal {N}_h\cap \Gamma _D$ (where $V_2(0, \cdot )$ is given by (2.1)) and solves the discrete system

$$\begin{aligned}&\Big (-\tfrac{1}{2}{\sigma ^2} \delta _{yy,h}^2V_{2,h} + \mathbf {b}\cdot \nabla V_{2,h} + (r + p_0) V_{2,h}, \phi _h\Big )_h\nonumber \\&\quad - \Big (h_{\texttt {stab}}\Delta _h V_{2,h},\phi _h\Big )_h = (f_2,\phi _h)_h ~ \forall \, \phi _h\in \mathbb {V}_{h,0}\,, \end{aligned}$$

(6.4)

where $h_{\texttt {stab}}(\mathbf {x}_\ell ) = h |\mathbf {b}(\mathbf {x}_\ell )|$, with $\mathbf {b} = -(b_2,\mu )$.

On setting $\mathcal {L}_{2,h}(V_{2,h}) :=\tfrac{1}{2}{\sigma ^2(y)} \delta _{yy,h}^2V_{2,h} - \mathbf {b}\cdot \nabla V_{2,h} - p_0 V_{2,h}$ we observe that (6.4) is equivalent to

$$\begin{aligned} r \big (V_{2,h}, \phi _h\big )_h = \Big (\mathcal {L}_{2,h}(V_{2,h}) + f_2, \phi _h\Big )_h + \Big (h_{\texttt {stab}}\Delta _h V_{2,h},\phi _h\Big )_h \qquad \forall \, \phi _h\in \mathbb {V}_{h}\,, \end{aligned}$$

which is a discrete counterpart of (1.4) with an additional artificial diffusion term $h_{\texttt {stab}}\Delta _h V_{2,h}$.

The artificial diffusion term guarantees the monotonicity and convergence of the finite element approximation, cf. [17] and [27, Sect. 3.5]. It is well known that the nodal basis functions of $\mathbb {V}_h(\mathcal {T}_h)$ satisfy

$$\begin{aligned} (\nabla \phi _i, \nabla \phi _j) \le 0 \quad \mathrm {for} \,\,i\ne j\,, \end{aligned}$$

(6.5)

for weakly acute triangulation $\mathcal {T}_h$. Since we employ uniform right-angled triangulations (consequently $\mathcal {T}_h$ are weakly acute) it can be shown that the artificial diffusion term $-h_{\texttt {stab}}\Delta _h V_{2,h}$ ensures the monotonicity of the finite element scheme (6.4), [27, Lemma 3.42]. The monotonicity of the finite element approximation for adaptive meshes can be ensured by a suitable choice of the stabilization parameter $h_{\texttt {stab}}$, for more details see [17, Sect. 8] and [27]. By the monotonicity of the numerical approximation standard arguments imply that the finite element solution $V_{2,h}$ of (6.4) converges to the (unique) viscosity solution of (1.2) (considered on $\Omega ^-$), cf. [17]. Furthermore, on noting the representation $V_{2,h} \equiv V_{2,h}(x,y) = \sum _{\ell =1}^{L} v_{2,\ell } \phi _\ell (x,y)$ we observe that (6.4) is equivalent to a system of linear equations for the coefficients $\{v_{2,\ell }\}_{\ell =1}^L$. Hence, owing to the monotonicity of the numerical approximation the associated matrix is a M-matrix, which implies that the numerical approximation (6.4) satisfies a discrete maximum principle, see for instance [27, Corollary 3.43].

We observed that, in the present setting, it is also possible to use the numerical scheme (6.4) without the stabilization (i.e., with $h_{\texttt {stab}} \equiv 0$) and the resulting numerical solution (not displayed) is very similar to those obtained by the stabilized scheme. In general the use of suitable numerical stabilization is advisable to ensure the reliability of numerical approximation.

The graph of the finite element solution $V_{2,h}$ for Scenarios 1–3 (see Sect. 2) computed on a mesh with mesh size $h=20\times 2^{-11}$ is displayed in Fig. 8.

Fig. 8

Plot of the finite element approximation of $V_2$ for Scenario 1–3 (from left to right)

6.2 Discretization of the Value in Mode $m=1$

In this section we propose two adaptive finite element algorithms for the numerical solution of (1.1). The first algorithm uses a stabilized finite element method along with a heuristic mesh refinement strategy, the second adaptive finite element algorithm is based on a least-squares approach.

Since the finite element method requires the spatial domain to be of finite size we consider (1.1) for $x\in \Omega = (-10,10)$. If not mentioned otherwise we impose the following Dirichlet boundary condition

$$\begin{aligned} V_1(-10)=0\quad \text {and}\quad V_1(10)= 10 + c - \frac{\xi }{\gamma _1}I^{nc}\,; \end{aligned}$$

(6.6)

the first condition is motivated by heuristic observations and the second condition is derived from (2.2). We partition $\Omega$ into possibly non-equidistant subintervals, i.e., we denote $T_\ell = (x_{\ell -1}, x_\ell )$, $\ell =1,\dots ,L$ and $\mathcal {T}_h = \bigcup _{\ell =1}^L T_\ell$. The partition $\mathcal {T}_h$ is determined by an adaptive refinement strategy, that is, the computation starts with a coarse initial grid $\mathcal {T}_{h,0}$ and then performs local mesh refinements and coarsening using suitable criteria which will be discussed below. The space of piecewise linear globally continuous functions on $\mathcal {T}_h$ is denoted as $\mathbb {V}_h \equiv \mathbb {V}_h(\mathcal {T}_h)=\mathrm {span}\{\phi _\ell ,\,\,\ell =1,\dots ,L\}$ and $\mathbb {V}_{h,0} :=\{\phi \in \mathbb {V}_h;\,\, \phi (-10) = \phi (10)=0\}$. The value function $V_1$ is then approximated by a continuous and piecewise linear function $V_{1,h} \in \mathbb {V}_h$.

We propose the following adaptive numerical algorithm which combines the policy iteration algorithm (cf., Algorithm 4.1) with iterative mesh refinement. The algorithm requires the values of $V_2(x, 0)$, $x\in \Omega$; we construct an approximation $V_{2,h}\approx V_2(\cdot , 0)$ as follows. For $x\le 0$ we obtain the (piecewise linear) approximation $V_{2,h}\in {\mathbb {V}}_{{\tilde{h}}}\equiv {\mathbb {V}}_{{\tilde{h}}}(\mathcal {T}_{{\tilde{h}}})$ using the stabilized finite element method from Sect. 6.1 on a uniform mesh with mesh size ${\tilde{h}}=20\times 2^{-11}$, see Fig. 8; for $x\ge 0$ the approximation $V_{2,h}$ is obtained from formula (2.1) by piecewise linear interpolation over a uniform mesh with the same mesh size.

Algorithm 6.1

(Adaptive Scheme)

Initialization: Given some initial triangulation $\mathcal {T}_{h,0}$ and the approximation $V_{2,h}\in {\mathbb {V}}_{{\tilde{h}}}$ of $V_2(\cdot , 0)$, set $I_{h,0} :=I^{\mathrm {nc}}$ and iterate for $k = 0,1,\dots$:

(1) Policy iteration:

Set $I_h^{(0)} \equiv I_{h,k}\in \mathbb {V}_h(\mathcal {T}_{h,k})$ and iterate for $n=1,2\dots$:

1A) Policy evaluation: Compute the approximation $V_{1,h}^{(n)} \in \mathbb {V}_h(\mathcal {T}_{h,k})$ of the solution $V^{(n)}_{1}$ of the linear problem

$$\begin{aligned} -b_1(I^{(n-1)}_h) \partial _x V_{1}^{(n)} + (p_0 + p_2(I_h^{(n-1)}) + r) V^{(n)}_{1} = f_1 + p_2(I_h^{(n-1)})V_{2,h} \quad \text {in }\Omega . \end{aligned}$$

(6.7)

1B) Policy improvement: For $P_h:L^2(\Omega )\rightarrow \mathbb {V}_h$ (cf., (6.11) below) update the control as

$$\begin{aligned} I^{(n)}_h = \frac{\gamma _1}{\xi } P_h \left( \frac{ V_{2,h} - V_{1,h}^{(n)}}{\partial _x V_{1,h}^{(n)}}\right) \,. \end{aligned}$$

(6.8)

1C) Stopping criterion: Set $n\rightarrow n+1$ and proceed with step 1A) unless the relative change of the control $I_h^{(n)}$ is below the prescribed tolerance (see, (6.9), (6.12), respectively).

(2) Mesh refinement: Set $V_{1,h}^{k+1}:=V_{1,h}^{(n)}$, $I_{h, k+1} :=I_h^{(n)}$, and construct a new mesh $\mathcal {T}_{h,k+1}$ by local mesh refinement/coarsening that is performed on the basis of a suitable error indicator $\eta _h \equiv \eta _h (V_{1,h}^{k}, I_{h, \ell +1})$.

(3) Stopping criterion: Set $k\rightarrow k+1$ and proceed to step (1) unless the mesh $\mathcal {T}_{h,k}$ (or the error indicator $\eta _h$) satisfies a prescribed stopping criterion.

We note that the use of a suitable operator $P_h:L^2(\Omega )\rightarrow \mathbb {V}_h$ in step 1B) of the above algorithm is necessary because for $V_{1,h} \in \mathbb {V}_h$ the derivative $\partial _x V_{1,h}$ is a piecewise constant function, i.e., $\partial _x V_{1,h}\notin \mathbb {V}_h$. Possible choices of the operator are given implicitly by (6.11) below.

6.2.1 Stabilized Finite Element Approximation with Heuristic Mesh Refinement

In this section we introduce a stabilized finite element method with heuristic mesh refinement strategy for the (linear) subproblem (6.7). The method employs the following stopping criterion in step (1C) of Algorithm 6.1:

$$\begin{aligned} \frac{\Vert I_h^{(n)}-I_h^{(n-1)}\Vert _{L^\infty (\Omega )}}{\Vert I_h^{(n)}\Vert _{L^\infty (\Omega )}}\le \epsilon . \end{aligned}$$

(6.9)

In all considered cases the prescribed tolerance $\epsilon = 10^{-6}$ was typically reached after 5 to 15 iterations; the iteration count is higher in the initial iterations and decreases once a good approximation of the control is available.

To simplify the notation we denote throughout this section $\mathcal {T}_h \equiv \mathcal {T}_h^{(n)}$, $\mathbb {V}_h\equiv \mathbb {V}_h(\mathcal {T}_h^{(n)})$, $V_{1,h} \equiv V_{1,h}^{(n)}$ and $I_h \equiv I_h^{(n-1)}$. We note that the discrete inner product for $\Omega \subseteq \mathbb {R}^1$ takes the form

$$\begin{aligned} \big (u,v\big )_h = \int _{\Omega } \mathcal {I}_h(uv)(x) \mathrm {d}x = \sum _{\ell =1}^{L}\frac{(x_\ell -x_{\ell -1})}{2} \big (u(x_{\ell -1})v(u(x_{\ell -1}) + u(x_{\ell })v(u(x_{\ell })\big )\,. \end{aligned}$$

We employ a 1d counterpart of the stabilized finite element method from Sect. 6.1, i.e., given the control $I_h$ the finite element approximation $V_{1,h}\in \mathbb {V}_{h}$ that satisfies the Dirichlet boundary condition (6.6) and solves

$$\begin{aligned} \Big (-b_1(I_h) \partial _x V_{1,h}&+ \big (p_0(I_h)+p_2+r\big ) V_{1,h}, \phi _h\Big ) \nonumber \\&- \Big (h_{\texttt {stab}}\delta _{xx,h}^2 V_{1,h},\phi _h\Big )_h = (p_2(I_h)V_{2,h}+f_1,\phi _h) \qquad \forall \, \phi _h\in \mathbb {V}_{h,0}\,, \end{aligned}$$

(6.10)

where the discrete second-order term $\delta _{xx,h}^2 V_{1,h}$ is a 1d analogue of (6.2). The artificial diffusion parameter in the above scheme is defined as

$$\begin{aligned} h_{\texttt {stab}}\equiv \sum _{\ell =1}^L h_{\texttt {stab}}(x_\ell )\phi _\ell (x) = \alpha _{\texttt {stab}}\sum _{\ell =1}^L \frac{|\mathrm {supp}(\phi _\ell )|}{4}\big |b_1\big (x_\ell , I_h(x_\ell )\big )\big | \phi _\ell (x)\,, \end{aligned}$$

with $\mathrm {supp} (\phi _\ell ) = \{x \in \Omega ,\,\, \phi _\ell (x) \ne 0\}$, and $\alpha _{\texttt {stab}}=1$, unless mentioned otherwise. We note that, in general, the above choice of the stabilization parameter $h_{\texttt {stab}}$ does not guarantee the M-matrix property of the corresponding linear system of equations (i.e., the monotonicity of the numerical approximation). However, in the current setting the M-matrix property was only violated for very few rows of the matrix (e.g. rows that correspond to the elements with large gradient in the discrete control) and this effect had negligible influence on the numerical approximation. The monotonicity of the numerical approximation can be obtained by choosing a large value of the stablization parameter $\alpha _{\texttt {stab}}$, more sophisticated monotonicity preserving stabilization methods are also available, see for instance in [17].

The discrete control in (6.8) of Algorithm 6.1 reads

$$\begin{aligned} I_h = \sum _{\ell =1}^L I_\ell \phi _\ell (x) =:\frac{\gamma _1}{\xi } P_h\left( \frac{ V_{2,h} - V_{1,h}^{(n)}}{\partial _x V_{1,h}^{(n)}}\right) \qquad \text {with}\qquad I_\ell = \frac{\gamma _1}{\xi } \frac{ V_{2,h}(x_\ell ) - V_{1,h}(x_\ell )}{\frac{1}{2}\big (\partial _x V_{1,h}|_{T_{\ell }} + \partial _x V_{1,h}|_{T_{\ell +1}}\big )}\,. \end{aligned}$$

(6.11)

An alternative choice of the approximation of the optimal control, which is an analogue of the ”averaged” finite difference approximation from Sect. 4.1.2, is given as

$$\begin{aligned} I_\ell = \frac{\gamma _1}{\xi } \big ( V_{2,h}(x_\ell ) - V_{1,h}(x_\ell )\big ) \frac{1}{2} \left( \frac{1}{\partial _x V_{1,h}|_{T_{\ell }}} + \frac{1}{\partial _x V_{1,h}|_{T_{\ell +1}}} \right) \,. \end{aligned}$$

Similarly as in the case of the finite difference approximation, we observe that the above approximation requires similar iteration counts for the convergence as (6.11).

Furthermore, we note that the policy iteration does not converge without any ”averaging” of the (piecewise constant) derivative $\partial _x V_{1,h}$, e.g., for

$$\begin{aligned} I_\ell = \frac{\gamma _1}{\xi } \frac{ V_{2,h}(x_\ell ) - V_{1,h}(x_\ell )}{\partial _x V_{1,h}|_{T_{\ell }}}\,, \end{aligned}$$

or for the choice $I_\ell \equiv {\tilde{I}}_\ell$, with ${\tilde{I}}_\ell$ defined below.

For the adaptive mesh refinement we employ the following local error indicator $\eta _T :=\eta _h|_T$, $T\in \mathcal {T}_h$

$$\begin{aligned} \eta _T = \alpha _\eta \Big \Vert \frac{I_h - {\tilde{I}}_h}{I_h}\Big \Vert _{L^2(T)}^2\,, \end{aligned}$$

with a scaling factor $\alpha _\eta >0$; the error indicator involves a modified control ${\tilde{I}}_h = \sum _{\ell =1}^L {\tilde{I}}_\ell \phi _\ell (x)$ where

$$\begin{aligned} {\tilde{I}}_\ell = \frac{\gamma _1}{\xi } \frac{ V_{2,h}(x_\ell ) - V_{1,h}(x_\ell )}{\left[ \frac{1}{2}\big (\partial _x V_{1,h}|_{T_{\ell }} + \partial _x V_{1,h}|_{T_{\ell +1}}\big ) - \frac{|\mathrm {supp}(\phi _\ell )|}{4}\delta _{x,h}^2V_{1,h}(x_\ell )\right] }\,. \end{aligned}$$

On meshes with uniform mesh size $h \equiv \frac{1}{2} |\mathrm {supp}(\phi _\ell )|$ the denominator in the above expression becomes $\frac{1}{2}\big (\partial _x V_{1,h}|_{T_{\ell }} + \partial _x V_{1,h}|_{T_{\ell +1}}\big ) + \frac{|\mathrm {supp}(\phi _\ell )|}{4}\delta _{x,h}^2V_{1,h}(x_\ell ) \equiv \partial _x V_{1,h}|_{T_{\ell }}$. Hence, the error indicator $\eta _h$ measures the difference between discrete controls obtained by two alternative discrete formulas which both approximate the continuous expression of the optimal control (3.14). Due to the sensitivity of the control with respect to the discretization, the above heuristic mesh refinement strategy produces meshes with good approximation properties, see Table 7 below. We observe that the choice of the scaling factor $\alpha _\eta =2.5$ improves the efficiency of the mesh refinement strategy for the considered simulation parameters.

In Fig. 9 we display the results for Scenario 1 computed on an adaptive mesh with 10945 degrees of freedom. The displayed results indicate growth of the gradient of the control at $x=0$, which indicates a discontinuity that is also captured by the mesh refinement algorithm (the tolerance for the refinement was $\eta _h = \sum _T \eta _T \le 2\times 10^{-5}$). Results for Scenarios 2–3 are not displayed, we observed no significant differences to the results obtained by the finite difference method.

Fig. 9

Plot of the ${2}|\mathrm {supp}(\phi _i)|^{-1} = 2(h_i + h_{i+1})^{-1}$ (left), discrete control (middle) and the gradient of the discrete control (right) for Scenario 1

We remark that the scheme (6.10) can be modified by replacing the $L^2$-inner product with the discrete inner product Sect. ; on a equidistant mesh with mesh size h it holds that $\mathrm {supp} (\phi _\ell ) = 2h$ and the resulting finite element scheme is equivalent to the upwind finite difference approximation (4.6). Nevertheless, this generalization of the finite difference scheme turned out to be unsuitable for the use in conjunction with adaptive mesh refinement as the policy iteration in Algorithm 6.1 did not converge.

6.2.2 Adaptive Least–Squares Finite Element Approximation

In this section we discuss the least-squares finite element method (LSFEM) for the approximation of (6.7). We employ the following mesh dependent stopping criterion in step (1C) of Algorithm 6.1: set $\text {ndof} :=\mathrm {dim} \mathbb {V}_h (\mathcal {T}_{h,k})$, $\epsilon _{k}:=\text {ndof}^{-1}$ ($\text {ndof} :=\dim \mathbb {V}_h(\mathcal {T}_{h,k})$), i.e.,

$$\begin{aligned} \frac{\Vert I_h^{(n)}-I_h^{(n-1)}\Vert _{L^2(\Omega )}}{\Vert I_h^{(n)}\Vert _{L^2(\Omega )}}\le \epsilon _k\,. \end{aligned}$$

(6.12)

As in the previous section we describe the application of the method in the n-th iteration of Algorithm 6.1 and denote $\mathcal {T}_h \equiv \mathcal {T}_h^{(n)}$, $\mathbb {V}_h\equiv \mathbb {V}_h(\mathcal {T}_h^{(n)})$, and $I_h \equiv I_h^{(n-1)}$. The LSFEM method consists of finding the approximation $V_{1,h} \approx V_1^{(n)}$ by the minimization of the functional

$$\begin{aligned} V_{1,h} = {\mathop {{{\,\mathrm{arg\,min}\,}}}\limits _{W_h \in \mathbb {V}_h}}\, \Vert - b_1(I_h) \partial _x W_h + (p_0 + p_2(I_h) + r) W_h - f_1 - p_2(I_h){ V_2(\cdot ,0)} \Vert _{L^2(\Omega )}^2. \end{aligned}$$

(6.13)

Notice that we do not impose any boundary conditions, since they are in general unknown (in particular on the negative axis). The above least squares functional provides the following straightforward way for the construction of mesh refinement indicators. The minimization in (6.13) is equivalent to the minimization of the error $W_h-V_{1}^{(n)}$ over all $W_h \in \mathbb {V}_h$ with respect to the semi-norm

$$\begin{aligned} \Vert \cdot \Vert _{a,I_h} :=\Vert - b_1(I_h) \partial _x \cdot + (p_0 + p_2(I_h) + r) \cdot \Vert _{L^2(\Omega )}. \end{aligned}$$

To minimize the error $\Vert V_{1,h} - V_1^{(n)} \Vert _{a,I_h}$, we want to refine the mesh in regions where this error is large. This motivates the evaluation of the residual on each $T\in \mathcal {T}_h$, leading to the refinement indicators

$$\begin{aligned} \eta _h|_T&:=\Vert -b_1(I_h) \partial _x V_{1,h} + (p_0 + p_2(I_h) + r)V_{1,h} - f_1 - p_2(I_h)V_2(\cdot , 0) \Vert _{L^2(T)} \\&= \Vert - b_1(I_h) \partial _x (V_{1,h}-V_{1}^{(n)}) + (p_0 + p_2(I_h) + r) (V_{1,h} - V_{1}^{(n)}) \Vert _{L^2(T)}. \end{aligned}$$

We use this built-in error control to drive adaptive mesh refinements with Dörfler marking strategy (cf. [26]) and bulk parameter $\Theta = 0.3$.

A downside of the LSFEM is that it is not a monotone scheme, in general. Thus the solution $V_{1,h}$ might oscillate, which results in difficulties in the approximation of the control $I_h$. To stabilize the policy improvement in Algorithm 6.1, our numerical experiments suggested a damping, that is, given some damping parameter $\rho \in (0,1)$ and the operator $P_h$ in (6.11), we replace the policy improvement (6.8) by the damped version

$$\begin{aligned} I^{(n)}_h :=I^{(n-1)}_h + \rho \left( \frac{\gamma _1}{\xi }P_h\left( \frac{ V_{2,h} - V_{1,h}^{(n)}}{\partial _x V_{1,h}^{(n)}}\right) - I^{(n-1)}_h \right) . \end{aligned}$$

(6.14)

Throughout all numerical experiments, we use the damping $\rho = 1/2$. Nevertheless, there are still tiny oscillations of the control near the origin $x=0$, causing difficulties in attaining the stopping criterion (6.12). To circumvent this difficulty, we had to replace the $L^\infty$ norm by the $L^2$ norm.

We terminate our computations if the number k of policy iterations in Algorithm 6.1 exceeds fifteen or the number of degrees of freedom $\text {ndof} = \dim \mathbb {V}_h$ exceeds half a million. All experiments utilize the parameters in Tables 1, 2 and the approximations of $V_2$ already used in Sects. 4 and 6.2.1. The initial grid $\mathcal {T}_0$ is a uniform partition of the domain $\Omega$ into 10 subintervals. We refine the mesh either adaptively or uniformly. The computations utilize the open source tool for solving partial differential equations FEniCS [24].

6.3 Numerical Experiments

In this section we examine the performance of the respective adaptive finite element schemes proposed in Sect. 6.2.1, 6.2.2 for the computation of $V_1$ and the control I. The approximation of $V_2$ is constructed using the (uniform) finite element method from Sect. 6.1 as discussed in Sect. 6.2.1. We compare our results with the finite difference scheme introduced in Sect. 4 which also provides the reference solution computed on a grid with $M=2^{20} \approx 10^6$ grid-points. Table 7 displays the resulting errors on two different underlying grids (the number of degrees of freedom equals roughly 1000 and 15000). To exclude the artificial behavior of the approximations near the left boundary, the errors have been computed on the reduced intervals $\Omega _\text {r} :=[-7,10]$ in Scenario 1 and 3 and $\Omega _\text {r} :=[-17,20]$ in Scenario 2.

It turns out that the adaptive schemes allow for significantly better approximations compared to the (uniformly refined) finite differences scheme, in particular on the coarse grid. This observation underlines the need of adaptive schemes in higher dimensions, where we have to solve such models on coarse grids due to limitations in our computational resources. We do not experience difficulties with the adaptive stabilized FEM in the sense that the policy iterations converge after a finite number of iterations.

The adaptive LSFEM performs well in Scenario 2 and 3, but struggles in Scenario 1: the policy iterates $I_h^{(n)}$ seem to oscillate between several states (cf. Fig. 10) and so do not converge after $\text {ndof}$ exceeds $10^4$. As discussed previously, the failure of convergence of the algorithm is related to the discontinuity in the exact control I (cf. Fig. 9) at the origin $x=0$, which induces oscillation in the computed approximation $V_{1,h}$ and and in the control $I_h$. We overcome the failure of the adaptive LSFEM in Scenario 1 by a domain decomposition approach similar to the one used in the collocation method. More precisely, we use the scheme to solve the problem on the domains $\Omega _l = (-10,0)$ and $\Omega _r = (0,10)$ separately with given boundary data for $V_{1,h}$ in the origin $x=0$. With this splitting, the results obtained by adaptive LSFEM are comparable to those obtained by the FDM and stabilized FEM in Scenario 1, cf. the results in Table 7.

Table 7 Errors of the stabilized adaptive finite element scheme, the finite differences scheme, and the adaptive least-squares scheme

Fig. 10

Zoom at the approximations of the optimal control in Scenario 1 with the FD scheme, the LSFEM with splitting (blue solid line, solutions are graphically indistinguishable), and several (non-convergent) iterates of the policy iteration in the LSFEM without splitting (dashed lines)

In Fig. 11 we display the mesh size distribution for adaptive meshes with $\text {ndof} \approx 1000$ for the considered adaptive finite element methods. We observe that, up to the boundary, the meshes exhibit similar structure; in particular both algorithms exhibit similar behavior near $x=0$. The smaller mesh size near the boundary for the stabilized FEM method is due to the boundary effects and has minor effect on the overall accuracy. The efficiency of the scheme can be increased by a suitable modification of the refinement indicator near the boundary, but we do not pursue this approach for simplicity.

Fig. 11

Plot of the inverse mesh size ${2}|\mathrm {supp}(\phi _i)|^{-1} = 2(h_i + h_{i+1})^{-1}$ in the $\log _2$ scale for Scenario 1–3 (from left to right) for stabilized FEM (solid line) and LSFEM (dotted line)

7 Conclusion

In general, the computed numerical approximations were similar for all considered numerical schemes in all the considered scenarios, at least for certain choices of simulation parameters. Nevertheless, we made some interesting observations (in particular in the solution for $m=1$) that are summarized below. We believe that these observations can be used as a benchmark for the evaluation of the robustness of numerical schemes for Hamilton–Jacobi–Bellman equations.

Convergence of the policy iteration The convergence of the policy iteration proved to be the most challenging aspect of numerical approximation of (1.1). The non-monotone numerical schemes, i.e., the collocation and LSFEM methods, required a splitting/domain decomposition approach in order to obtain the convergence of the policy iteration algorithm in Scenario 1, where the control is discontinuous at $x=0$. In addition, without a good initial guess, the collocation method required the use of the transformation (3.1), (3.2) to ensure convergence. The solution computed with the state state transformation can be used as a preconditioner to obtain an accurate solution on larger domains without transformation. The divergence can be explained by the fact that the policy iteration can be interpreted as a nonlinear fixed-point iteration (cf. [5]) and, in the case of discontinuous control, the corresponding nonlinearity is not Lipschitz continuous. Nevertheless, a rigorous convergence analysis of the policy iteration in the present setting is still open. We note that the application of the splitting/domain decomposition approach may be difficult for more general problems (e.g., in higher dimensions) where the location of the discontinuity is not known a priori. The monotone schemes proved to be more robust in this respect, in particular when a suitable discrete approximation of the optimal control is employed. For instance in the case of the finite element approximation in Scenario 1, a suitable nodal interpolation of the (piecewise constant) gradient of the numerical solution in the formula for the discrete optimal control was essential in order to ensure robust convergence of the nonlinear solver. We observed that the number of policy iterations is independent of the mesh size, nevertheless the convergence of the policy iteration can be affected by ”too harsh” mesh refinement, i.e., on meshes with too sharp differences in the sizes of the neighboring elements. On the other hand, suitable mesh refinement may improve the convergence of the policy iteration, in particular in the case of LSFEM method. We note that an additional benefit of the mesh refinement algorithm is that the iterations counts decrease in the final stages of the mesh refinement loop, i.e., once the mesh is close to the optimal one. This can be attributed to the fact that the final refinement steps typically only introduce minor changes in the control. In the case of the WENO scheme, the convergence of the policy iteration combined with the time-stepping algorithm was robust, but the algorithm required a high number of iterations to reach the prescribed tolerances.

WENO scheme The algorithm produced very similar results as the monotone finite difference and finite element schemes. The disadvantage of the method is the high iteration count of the policy iteration, nevertheless compared to the LSFEM and the collocation method, the convergence of the policy iteration was robust. Suitable preconditioning strategies can improve the efficiency of the algorithm.

Boundary/transformation effects Except for the collocation method, we observed no major differences between the computed approximation of $V_1$ by the respective numerical schemes. On larger domains the collocation method becomes inaccurate due to the transformation (3.1), (3.2) which is required for the convergence of the policy iteration. The transformation causes the collocation points to be concentrated near $x=0$ and the approximation becomes increasingly inaccurate on large domains; a more advanced adaptive collocation strategies could improve the accuracy of the scheme, cf. [4, 35]. Without the transformation the collocation method produces accurate results on large domains as well, but it requires preconditioning to ensure the convergence of the policy iteration algorithm. Compared to other methods, the adaptive LSFEM methods exhibited slightly faster convergence behavior w.r.t. the mesh size, nevertheless due to the fact that the value function $V_1$ is smooth (i.e., the solution does not exhibit any shocks) the differences were not significant. For x close to the origin, the computed optimal controls also exhibited good agreement for all methods, in particular when adaptivity is employed. Nevertheless, near the left boundary the three methods produce very different numerical results. This can be explained by the different approximation of the infinite spatial domain as well as the different choice of the (artificial) boundary conditions. With the exception of the collocation method, all considered numerical schemes exhibit similar behavior for increasing sizes of the (bounded) spatial domain, i.e., the effects of the artificial boundary conditions can be mitigated by the choice of a sufficiently large spatial domain for numerical computations.

Approximation of $V_2$ The numerical solution of $V_2$ enters as a parameter in the numerical approximation of $V_1$ and hence the quality of the approximation of $V_2$ has some influence on the numerical approximation of $V_1$. The numerical approximation $V_2$ computed by collocation method contained minor oscillations, this effect can be attributed to the lack of numerical stabilization in the collocation scheme as well as the aforementioned domain transformation. A close inspection of the related numerical result reveals that these oscillations transfer to the approximation of $V_1$. While the effect was minor and did not have notable impact on the accuracy of numerical solution in the present setup, the results suggest that the use of monotone schemes for $V_2$, such as the stabilized FEM method in Sect. 6.1 produces a more reliable numerical algorithm overall.