Harvesting and seeding of stochastic populations: analysis and numerical approximation

Abstract

We study an ecosystem of interacting species that are influenced by random environmental fluctuations. At any point in time, we can either harvest or seed (repopulate) species. Harvesting brings an economic gain while seeding incurs a cost. The problem is to find the optimal harvesting-seeding strategy that maximizes the expected total income from harvesting minus the cost one has to pay for the seeding of various species. In Hening et al. (J Math Biol 79(2):533–570, 2019b) we considered this problem when one has absolute control of the population (infinite harvesting and seeding rates are possible). In many cases, these approximations do not make biological sense and one must consider what happens when one, or both, of the seeding and harvesting rates are bounded. The focus of this paper is the analysis of these three novel settings: bounded seeding and infinite harvesting, bounded seeding and bounded harvesting, and infinite seeding and bounded harvesting. Even one dimensional harvesting problems can be hard to tackle. Once one looks at an ecosystem with more than one species analytical results usually become intractable. In order to gain information regarding the qualitative behavior of the system we develop rigorous numerical approximation methods. This is done by approximating the continuous time dynamics by Markov chains and then showing that the approximations converge to the correct optimal strategy as the mesh size goes to zero. By implementing these numerical approximations, we are able to gain qualitative information about how to best harvest and seed species in specific key examples. We are able to show through numerical experiments that in the single species setting the optimal seeding-harvesting strategy is always of threshold type. This means there are thresholds $0<L_1<L_2<\infty$ such that: (1) if the population size is ‘low’, so that it lies in $(0, L_1]$, there is seeding using the maximal seeding rate; (2) if the population size ‘moderate’, so that it lies in $(L_1,L_2)$, there is no harvesting or seeding; (3) if the population size is ‘high’, so that it lies in the interval $[L_2, \infty )$, there is harvesting using the maximal harvesting rate. Once we have a system with at least two species, numerical experiments show that constant threshold strategies are not optimal anymore. Suppose there are two competing species and we are only allowed to harvest or seed species 1. The optimal strategy of seeding and harvesting will involve lower and upper thresholds $L_1(x_2)<L_2(x_2)$ which depend on the density $x_2$ of species 2.

Appendices

Appendix A: Properties of the value function

Proposition A.1

Assume we are in the setting of bounded seeding and unbounded harvesting rates. Suppose that there exists number $U>0$ such that

$$\begin{aligned} \sum \limits _{i=1}^d \big [ b_i(x) -\delta (x_i-U)\big ]f_i<0 \quad \text {for}\quad |x|>U. \end{aligned}$$

Then there exists $x^*\in [0, U]^d$ such that

$$\begin{aligned} V(x)= V(x^*) + f\cdot (x-x^*)\quad \text {for}\quad x\in \overline{S} \setminus [0, U]^d. \end{aligned}$$

Moreover,

$$\begin{aligned} V(x)=V^U(x) \quad \text {for}\quad x\in [0, U]^d. \end{aligned}$$

Proof

Fix some $x \in \overline{S}\setminus [0, U]^d$ and $(Y, C)\in {\mathcal {A}}_{x}$, and let X denote the corresponding harvested process. Let $x_i^*= \min \{x_i, U\}$ for $i=1, \ldots , d$ and $x^*=(x_1^*, \ldots , x_d^*)'$.

Let $\varepsilon \in (0, 1)$ be a constant and define

$$\begin{aligned} \Phi _\varepsilon (y)=f \cdot (y-x^*) +\varepsilon , \quad y\in \overline{S}\setminus [0, U]^d. \end{aligned}$$

(A.1)

We can extend $\Phi _\varepsilon (\cdot )$ to the entire $\overline{S}$ so that $\Phi _\varepsilon (\cdot )$ is twice continuously differentiable, $\Phi _\varepsilon (y)\ge 0$ and $f\le \nabla \Phi _\varepsilon (y)$ for all $y\in \overline{S}$. By assumption, we can check that

$$\begin{aligned} ({\mathcal {L}}-\delta )\Phi _\varepsilon (y)=\sum \limits _{i=1}^d \big [ b_i(y) -\delta (y_i-x^*)\big ]f_i-\delta \varepsilon < 0 \quad \text {for} \quad y\in \overline{S}\setminus [0, U]^d. \end{aligned}$$

Choose N sufficiently large so that $|x|< N$. For

$$\begin{aligned} \beta _N=\inf \{t\ge 0: |X(t)|\ge N\}, \quad \gamma _0=\inf \{t\ge 0: X(t)\in [0, x^*]\}, \quad T_N = N\wedge \beta _N\wedge \gamma _0, \end{aligned}$$

we have $T_N \rightarrow \gamma _0$ with probability one as $N \rightarrow \infty$. By Dynkin’s formula,

$$\begin{aligned}&\mathbb {E}_{x} \big [e^{-\delta T_N}\Phi _\varepsilon \left( X(T_N)\right) \big ]-\Phi _\varepsilon (x)\\&\quad = \mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon \left( X(s)\right) ds-\mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}\nabla \Phi _\varepsilon \left( X(s)\right) \cdot dY^c(s)\\&\qquad +\,\mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}\nabla \Phi _\varepsilon \left( X(s)\right) \cdot C(s)ds + \mathbb {E}_{x}\sum \limits _{0\le s\le T_N}e^{-\delta s}\Big [\Phi _\varepsilon \left( X(s)\right) -\Phi _\varepsilon \left( X(s-)\right) \Big ], \end{aligned}$$

where $Y^c(\cdot )$ is the continuous part of $Y(\cdot )$. Let $\Delta Y(s)= Y(s)-Y(s-)$. Since $\nabla \Phi _\varepsilon (X(s))=f$ and $\Phi _\varepsilon \left( X(s)\right) -\Phi _\varepsilon \left( X(s-)\right) =-f\cdot \Delta Y(s)$, we obtain

$$\begin{aligned}&\mathbb {E}_{x} \big [e^{-\delta T_N}\Phi _\varepsilon \left( X(T_N)\right) \big ]-\Phi _\varepsilon (x) \le \mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon (X(s))ds\nonumber \\&\quad -\mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}f\cdot dY^c(s) + \mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}f\cdot C(s)ds - \mathbb {E}_{x}\sum \limits _{0\le s\le T_N}e^{-\delta s}f\cdot \Delta Y(s).\nonumber \\ \end{aligned}$$

(A.2)

Since $\Phi _\varepsilon (y)\ge 0$ and $f<g(y)$ for any $y\in \overline{S}$, it follows from (A.2) that

$$\begin{aligned}&\mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}f\cdot dY(s) - \mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}g(X(s))\cdot C(s)ds \\&\quad \le \Phi _\varepsilon (x) +\mathbb {E}_{x}\int _{0}^{T_N} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon (X(s)) ds. \end{aligned}$$

Letting $N\rightarrow \infty$, by the bounded convergence theorem, we obtain

$$\begin{aligned}&\mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}f\cdot dY(s) - \mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}g(X(s))\cdot C(s)ds \\&\quad \le \Phi _\varepsilon (x) +\mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon (X(s))ds. \end{aligned}$$

As a result

$$\begin{aligned} J(x, Y, C)&\le \mathbb {E}_{x}\Big [\int _{0}^{\gamma _0} e^{-\delta s}f\cdot dY(s) - \int _{0}^{\gamma _0} e^{-\delta s}g(X(s))\cdot C(s)ds + V(X(\gamma _0)) \Big ] \\&\le V(x^*) + \Phi _\varepsilon (x)+\mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon (X(s))ds. \end{aligned}$$

The above implies

$$\begin{aligned} J(x, Y, C)\le V(x^*) + f\cdot (x-x^*) + \varepsilon +\mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _\varepsilon (X(s))ds.\nonumber \\ \end{aligned}$$

(A.3)

Letting $\varepsilon \rightarrow 0$ in (A.3)

$$\begin{aligned} J(x, Y, C)\le V(x^*) + f\cdot (x-x^*)-\mathbb {E}_{x}\int _{0}^{\gamma _0} e^{-\delta s}({\mathcal {L}}-\delta )\Phi _0(X(s))ds, \end{aligned}$$

(A.4)

where $\Phi _0(\cdot )$ is also defined by (A.1) at $\varepsilon =0$. Note that if $\mathbb {P}(\gamma _0=0)<1$, then (A.3) is a strict inequality. On the other hand, it is obvious (by harvesting instantaneously $x-x^*$ at time $t=0$) that

$$\begin{aligned} V(x)\ge V(x^*) + f\cdot (x-x^*). \end{aligned}$$

(A.5)

In view of (A.4) and (A.5), if $x \in \overline{S} \setminus [0, U]^d$, $V(x)=V(x^*) + f\cdot (x-x^*)$. Moreover, it is optimal to instantaneously harvest an amount of $x-x^*$ to drive the population to the state $x^*$ on the boundary of $[0, U]^d$, and then apply an optimal or near-optimal harvesting-seeding policy in ${\mathcal {A}}_{x^*}$. Therefore, if the initial population $x\in [0, U]^d$, it is optimal to apply a harvesting-seeding policy so that the population process stays in $[0, U]^d$ forever. This completes the proof. $\square$

Proposition A.2

Suppose we are in the setting of bounded seeding and harvesting rates, and that Assumption 2.1 is satisfied.

• (a) The value function V is finite and continuous on ${\overline{S}}$.

• (b) The value function V is a viscosity subsolution of (2.19); that is, for any $x^0\in S$ and any function $\phi \in C^2(S)$ satisfying

  $$\begin{aligned} (V-\phi )(x)\ge (V-\phi )(x^0)=0, \end{aligned}$$

  for all x in a neighborhood of $x^0$, we have

  $$\begin{aligned} ({\mathcal {L}}-\delta ) \phi (x^0) + \max \limits _{\xi \in [-\lambda , \mu ]}\Big [\xi ^-\cdot \big (f-\nabla \phi )\left( x^0\right) - \xi ^+ \cdot (g-\nabla \phi )\left( x^0\right) \Big ]\le 0.\nonumber \\ \end{aligned}$$

  (A.6)

• (c) The value function V is a viscosity supersolution of (2.19); that is, for any $x^0\in S$ and any function $\varphi \in C^2(S)$ satisfying

  $$\begin{aligned} (V-\varphi )(x)\le (V-\varphi )(x^0)=0, \end{aligned}$$

  (A.7)

  for all x in a neighborhood of $x^0$, we have

  $$\begin{aligned} ({\mathcal {L}}-\delta ) \varphi (x^0) + \max \limits _{\xi \in [-\lambda , \mu ]}\Big [\xi ^-\cdot \big (f-\nabla \varphi )\left( x^0\right) - \xi ^+ \cdot (g-\nabla \varphi )\left( x^0\right) \Big ]\ge 0. \end{aligned}$$

  (A.8)

• (d) The value function V is a viscosity solution of (2.19).

In the proof, we use the following notation and definitions. For a point $x^0\in S$ and a strategy $Q\in {\mathcal {A}}_{x^0}$, let X be the corresponding process with harvesting and seeding. Let $B_\varepsilon (x^0)=\{x\in S: |x-x^0|<\varepsilon \}$, where $\varepsilon >0$ is sufficiently small so that $\overline{B_\varepsilon (x^0)}\subset S$. Let $\theta =\inf \{t\ge 0: {X}(t)\notin B_\varepsilon (x^0) \}$. For a constant $r>0$, we define $\theta _r=\theta \wedge r$.

Proof

(a) Since the functions $f(\cdot )$, $g(\cdot )$ and the rates $C(\cdot )$, $R(\cdot )$ are bounded, the value function is also bounded. The conclusion then follows by (Krylov 2008, Chapter 3, Theorem 5).

(b) For $x^0\in S$, consider a $C^2$ function $\phi (\cdot )$ satisfying $\phi (x^0)=V(x^0)$ and $\phi (x)\le V(x)$ for all x in a neighborhood of $x^0$. Let $\varepsilon >0$ be sufficiently small so that $\overline{B_\varepsilon (x^0)}\subset S$ and $\phi (x)\le V(x)$ for all $x\in \overline{B_\varepsilon (x_0)}$, where $\overline{B_\varepsilon (x_0)}=\{x\in S: |x-x^0|\le \varepsilon \}$ is the closure of $B_\varepsilon (x^0)$.

Let $\xi \in [-\mu , \lambda ]$ and define $Q\in {\mathcal {A}}_{x^0}$ to satisfy $Q(t)=\xi$ for all $t\in [0, r]$ for a positive constant r. We denote by X the corresponding harvested process with initial condition $x^0$. Then ${X}(t)\in \overline{B_\varepsilon (x^0)}$ for all $0\le t\le \theta$. By virtue of the dynamic programming principle, we have

$$\begin{aligned} \phi (x^0)= & {} V(x^0) \ge \mathbb {E}\bigg [ \int _0^{\theta _r} e^{-\delta s}\Big ( Q^-(s)\cdot f\left( {X}(s)\right) - Q^+(s)\cdot g\left( {X}(s)\right) \Big )ds \nonumber \\&+ e^{-\delta \theta _r} \phi ({X}(\theta _r))\bigg ]. \end{aligned}$$

(A.9)

By the Dynkin formula, we obtain

$$\begin{aligned} \phi (x^0)&= \mathbb {E}e^{-\delta \theta _r} \phi ({X}(\theta _r)) - \mathbb {E}\int _0^{\theta _r} e^{-\delta s} ({\mathcal {L}}-\delta ) \phi ({X}(s))ds\nonumber \\&\quad + \mathbb {E}\int _0^{\theta _r} e^{-\delta s}\Big (Q^-(s)\cdot \nabla \phi \left( {X}(s)\right) - Q^+(s)\cdot \nabla \phi \left( {X}(s)\right) \Big )ds. \end{aligned}$$

(A.10)

A combination of (A.9) and (A.10) leads to

$$\begin{aligned}&0 \ge \mathbb {E}\int _0^{\theta _r} e^{-\delta s}\Big (Q^-(s)\cdot f\left( {X}(s)\right) - Q^+(s)\cdot g\left( {X}(s)\right) \Big )ds \nonumber \\&\qquad +\, \mathbb {E}\int _0^{\theta _r} e^{-\delta s} ({\mathcal {L}}-\delta ) \phi ({X}(s))ds \nonumber \\&\qquad -\, \mathbb {E}\int _0^{\theta _r} e^{-\delta s}\Big (Q^-(s)\cdot \nabla \phi \left( {X}(s)\right) - Q^+(s)\cdot \nabla \phi \left( {X}(s)\right) \Big )ds, \end{aligned}$$

(A.11)

which in turn implies

$$\begin{aligned}&\mathbb {E}\int _0^{\theta _r}e^{-\delta s}\Big [ ({\mathcal {L}}-\delta ) \phi (X(s)) + Q^-(s)\cdot \big (f-\nabla \phi )\left( X(s)\right) \\&\quad - Q^+(s)\cdot (g-\nabla \phi )\left( X(s)\right) \Big ]ds\le 0. \end{aligned}$$

By the continuity of $X(\cdot )$ and the definition of $Q(\cdot )$, we obtain

$$\begin{aligned} ({\mathcal {L}}-\delta ) \phi (x^0) + \xi ^-\cdot \big (f-\nabla \phi )\left( x^0\right) - \xi ^+ \cdot (g-\nabla \phi )\left( x^0\right) \le 0. \end{aligned}$$

This completes the proof of (b).

(c) Let $x^0\in S$ and suppose $\varphi (\cdot )\in C^2(S)$ satisfies (A.7) for all x in a neighborhood of $x^0$. We argue by contradiction. Suppose that (A.8) does not hold. Then there exists a constant $A>0$ such that

$$\begin{aligned} ({\mathcal {L}}-\delta ) \varphi (x^0) + \max \limits _{\xi \in [-\lambda , \mu ]}\Big [\xi ^-\cdot \big (f-\nabla \varphi )\left( x^0\right) - \xi ^+ \cdot (g-\nabla \varphi )\left( x^0\right) \Big ]\le -2A< 0. \end{aligned}$$

(A.12)

Let $\varepsilon >0$ be small enough so that $\overline{B_\varepsilon (x^0)}\subset S$ and for any $x\in \overline{B_\varepsilon (x^0 )}$, $\varphi (x)\ge V(x)$ and

$$\begin{aligned} ({\mathcal {L}}-\delta ) \varphi (x) + \max \limits _{\xi \in [-\lambda , \mu ]}\Big [\xi ^-\cdot \big (f-\nabla \varphi )\left( x\right) - \xi ^- \cdot (g-\nabla \varphi )\left( x\right) \Big ]\le -A <0. \end{aligned}$$

(A.13)

Let $Q\in {\mathcal {A}}_{x^0}$ and ${X}(\cdot )$ be the corresponding process. Recall that $\theta =\inf \{t\ge 0: {X}(t)\notin B_\varepsilon (x^0) \}$ and $\theta _r=\theta \wedge r$ for any $r>0$. It follows from the Dynkin formula that

$$\begin{aligned}&\mathbb {E}e^{-\delta \theta _r} \varphi ({X}(\theta _r)-\varphi (x^0)) \nonumber \\&\quad = \mathbb {E}\int _0^{\theta _r} e^{-\delta s} \Big [({\mathcal {L}}-\delta ) \varphi ({X}(s)) - Q^-(s)\cdot \nabla \varphi (X(s)) + Q^+(s)\cdot \nabla \varphi (X(s))\Big ]ds\nonumber \\&\quad = \int _0^{\theta _r} e^{-\delta s} \Big [({\mathcal {L}}-\delta ) \varphi ({X}(s)) +Q^-(s) \cdot (f- \nabla \varphi )(X(s)) \nonumber \\&\qquad - Q^+(s)\cdot (g-\nabla \varphi ) (X(s))\Big ]ds\nonumber \\&\qquad - \int _0^{\theta _r} e^{-\delta s} \Big [ Q^-(s)\cdot f(X(s)) - Q^+(s)\cdot g(X(s))\Big ]ds . \end{aligned}$$

(A.14)

Equations (A.13) and (A.14) show that

$$\begin{aligned}&\mathbb {E}e^{-\delta \theta _r} \varphi ({X}(\theta _r))-\varphi (x^0)) \nonumber \\&\quad \le \mathbb {E}\int _0^{\theta _r } e^{-\delta s} (-A)ds - \int _0^{\theta _r} e^{-\delta s} \Big ( Q^-(s)\cdot f(X(s)) - Q^+(s)\cdot g(X(s))\Big )ds . \end{aligned}$$

(A.15)

Therefore

$$\begin{aligned} \varphi (x^0)&\ge \mathbb {E}e^{-\delta \theta _r} \varphi ({X}(\theta _r)) + A \mathbb {E}\int _0^{\theta _r} e^{-\delta s} ds\nonumber \\&\quad + \int _0^{\theta _r} e^{-\delta s} \Big ( Q^-(s)\cdot f(X(s)) - Q^+(s)\cdot g(X(s))\Big )ds. \end{aligned}$$

(A.16)

Letting $r\rightarrow \infty$, we have

$$\begin{aligned} V(x^0)&=\varphi (x^0) \ge \mathbb {E}e^{-\delta \theta } \varphi ({X}(\theta )) + A \mathbb {E}\int _0^{\theta } e^{-\delta s} ds\nonumber \\&\quad + \int _0^{\theta } e^{-\delta s} \Big ( Q^-(s)\cdot f(X(s)) - Q^+(s)\cdot g(X(s))\Big )ds. \end{aligned}$$

(A.17)

Set $\kappa _0 = A \mathbb {E}\int _0^{\theta } e^{-\delta s} ds>0$. Taking the supremum over $Q\in {\mathcal {A}}_{x^0}$ we arrive at

$$\begin{aligned} V(x^0)&\ge \kappa _0 + \sup \limits _{Q\in {\mathcal {A}}_{x^0}}\mathbb {E}\bigg [e^{-\delta \theta } \varphi ({X}(\theta )) + \int _0^{\theta } e^{-\delta s} \Big ( Q^-(s)\cdot f(X(s)) \nonumber \\&\quad - Q^+(s)\cdot g(X(s))\Big )ds\bigg ]. \end{aligned}$$

(A.18)

In view of the dynamic programming principle, the preceding inequality can be rewritten as $V(x^0)\ge V(x_0)+\kappa _0>V(x^0)$, which is a contradiction. This implies that (A.8) has to hold and the conclusion follows.

Part (d) follows from (b) and (c). $\square$

Appendix B: Numerical algorithm

We will present the detailed convergence analysis of Theorem 2.6, which is closely based on the Markov chain approximation method developed by Kushner and Dupuis (1992), Kushner and Martins (1991). Theorem 2.8 and Theorem 2.10 can be derived using similar techniques and we therefore omit the details.

1.1 B.1: Transition probabilities for bounded seeding and unbounded harvesting rates

For simplicity, we make use of one more assumption below. This assumption will be used to ensure that the transition probabilities $p^h(x, y|u)$ are well defined. Nevertheless, this is not an essential assumption. There are several alternatives to handle the cases when Assumption B.1 fails. We refer the reader to (Kushner 1990, page 1013) for a detailed discussion. Define for any $x\in \overline{S}$ the covariance matrix $a(x)= \sigma (x)\sigma '(x)$.

Assumption B.1

For any $i=1, \ldots , d$ and $x\in \overline{S}$,

$$\begin{aligned} a_{ii}(x)-\sum \limits _{j: j\ne i}\big |a_{ij}(x)\big |\ge 0. \end{aligned}$$

We define the difference $\Delta X_n^h = X_{n+1}^h-X_{n}^h.$ Denote by $\Delta Y^h_n$ the harvesting amount for the chain at step n. If $\pi ^h_n=i$, we let $\Delta Y^h_n=h\mathbf{e_i}$ and then $\Delta X^h_n=-h\mathbf{e_i}$. If $\pi ^h_n=0$, we set $\Delta Y^h_n=0$. Define

$$\begin{aligned} Y^h_0=0, \quad Y^h_n = \sum \limits _{m=0}^{n-1}\Delta Y^h_m. \end{aligned}$$

For definiteness, if $X^{h}_{n, i}$ is the ith component of the vector $X^h_n$ and $\{j: X^{h}_{n, j}=U\}$ is non-empty, then step n is a harvesting step on species $\min \{j: X_{n, j}^{h}=U\}$. Recall that $u^h_n= (\pi ^h_n, C^h_n)$ for $n\in {\mathbb {Z}}_{\ge 0}$ and $u^h=\{u^h_n\}_n\equiv \{Y^h_n, C^h_n\}_n$ is a sequence of controls. It should be noted that $\pi ^h_n = 0$ includes the case when we seed nothing; that is, $C^h_n = 0$. Denote by ${\mathcal {F}}^h_n=\sigma \{X^h_m,u^h_m, m\le n\}$ the $\sigma$-algebra containing the information from the processes $X^h_m$ and $u^h_m$ between the times 0 and n.

The sequence $u^h= (\pi ^h, C^h)\equiv \{Y^h_n, C^h_n\}_n$ is said to be admissible if it satisfies the following conditions:

• (a) $u^h_n$ is $\sigma \{X^h_0, \ldots , X^h_{n},u^h_0, \ldots , u^h_{n-1}\}-\text {adapted},$

• (b) For any $x\in S_h$, we have

  $$\begin{aligned} \mathbb {P}\{ X^h_{n+1} = x | {\mathcal {F}}^h_n\}= \mathbb {P}\{ X^h_{n+1} = x | X^h_n, u^h_n\} = p^h( X^h_n, x| u^h_n), \end{aligned}$$

• (c) Denote by $X^{h}_{n, i}$ the ith component of the vector $X^h_n$. Then

  $$\begin{aligned} \mathbb {P}\big ( \pi ^h_{n}=\min \{j: X^{h}_{n, j} = U\} | X^{h}_{n, j} = U \text { for some } j\in \{1, \ldots , d \}, {\mathcal {F}}^h_n\big )=1. \end{aligned}$$

• (d) $X^h_n\in S_h$ for all $n\in {\mathbb {Z}}_{\ge 0}$.

The class of all admissible control sequences $u^h$ having the initial state x will be denoted by ${\mathcal {A}}^h_{x}$.

For each $(x, u)\in S_h\times {\mathcal {U}}$, we define a family of interpolation intervals $\Delta t^h (x, u)$. The values of $\Delta t^h (x, u)$ will be specified later. Then we define

$$\begin{aligned} t^h_0 = 0,\quad \Delta t^h_m = \Delta t^h(X^h_m, u^h_m), \quad t^h_n = \sum \limits _{m=0}^{n-1} \Delta t^h_m. \end{aligned}$$

(B.1)

Let $\mathbb {E}^{h, u}_{x, n}$, ${\mathbb {Cov}}^{h, u}_{x, n}$ denote the conditional expectation and covariance given by

$$\begin{aligned} \{X_m^h, u_m^h, m\le n, X_n^h=x, u^h_n=u \}, \end{aligned}$$

respectively. Our objective is to define transition probabilities $p^h (x, y | u)$ so that the controlled Markov chain $\{X^h_n\}$ is locally consistent with respect to the controlled diffusion (2.7) in the sense that the following conditions hold at seeding steps, i.e., for $u=(0, c)$

$$\begin{aligned} \mathbb {E}^{h, u}_{x, n}\Delta X_n^h&= \big ({b}(x)+c\big )\Delta t^h(x, u) + o(\Delta t^h(x, u)),\nonumber \\ Cov^{h, u}_{x, n}\Delta X_n^h&= a(x)\Delta t^h(x, u) + o(\Delta t^h(x, u)),\nonumber \\ \sup \limits _{n, \ \omega } |\Delta X_n^h|&\rightarrow 0 \quad \text {as}\quad h \rightarrow 0. \end{aligned}$$

(B.2)

Using the procedure used by Kushner (1990), for $(x, u)\in S_h\times {\mathcal {U}}$ with $u=(0, c)$, define

$$\begin{aligned} \displaystyle Q_h (x, u)&=\sum \limits _{i=1}^d a_{ii}(x) -\sum \limits _{i, j: i\ne j}\dfrac{1}{2}|a_{ij}(x)| +h\sum \limits _{i=1}^d |b_i(x) + c_i| +h,\nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}|u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_{i}(x)+c_i\big )^+ h }{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x-h \mathbf{e_i}| u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_i(x) +c_i)^- h}{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}+h \mathbf{e_j}| u\right)&= p^h \left( x, x-h\mathbf{e_i}-h\mathbf{e_j}| u\right) = \dfrac{a_{{ij}}^+(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}-h \mathbf{e_j}| u\right)&= p^h \left( x, x-h\mathbf{e_i}+ h \mathbf{e_j}| u \right) = \dfrac{a_{{ij}}^-(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x | u\right)&=\dfrac{h }{ Q_h (x, u)},\qquad \Delta t^h (x, u)=\dfrac{h^2}{Q_h(x, u)}. \end{aligned}$$

(B.3)

Set $p^h \left( x, y|u=(0, c)\right) =0$ for all unlisted values of $y\in S^h$. Assumption B.1 guarantees that the transition probabilities in (B.3) are well-defined. At the harvesting steps, we define

$$\begin{aligned} \displaystyle p^h\left( x, x - h \mathbf{e_i}| u = (i, c)\right) =1, \quad \Delta t^h(x, u=(i, c))=0, \quad i=1, 2, \ldots , d. \end{aligned}$$

(B.4)

Thus, $p^h \left( x, y|u=(i, c)\right) =0$ for all unlisted values of $y\in S^h$. Using the above transition probabilities, we can check that the locally consistent conditions of $\{X^h_n\}$ in (B.2) are satisfied.

1.2 B.2: Continuous–time interpolation and time rescaling

The convergence result is based on a continuous-time interpolation of the chain, which will be constructed to be piecewise constant on the time interval $[t^h_n, t^h_{n+1}), n\ge 0$. We define $n^h(t)=\max \{n: t^h_n\le t\}, t\ge 0$. We first define discrete time processes associated with the controlled Markov chain as follows. Let $B^h_0=M^h_0=0$ and define for $n\ge 1$,

$$\begin{aligned} B^h_n = \sum \limits _{m=0}^{n-1} I_{\{\pi _m^h=0\}} \mathbb {E}^{h}_m \Delta \xi ^h_m,\qquad M^h_n = \sum \limits _{m=0}^{n-1} (\Delta \xi ^h_m - \mathbb {E}^{h}_m \Delta X_m)I_{\{\pi _m^h=0\}}. \end{aligned}$$

(B.5)

The piecewise constant interpolation processes, denoted by $(X^h(\cdot ), Y^h(\cdot ), B^h(\cdot ), M^h(\cdot ), C^h(\cdot ))$ are naturally defined as

$$\begin{aligned} X^h(t)&= X^h_{n^h(t)},\quad C^h(t) = C^h_{n^h(t)}, \nonumber \\ Y^h(t)&= Y^h_{n^h(t)}, \quad B^h(t) = B^h_{n^h(t)}, \quad M^h(t) = M^h_{n^h(t)}, \quad t\ge 0. \end{aligned}$$

(B.6)

Define ${\mathcal {F}}^h(t)=\sigma \{X^h(s), Y^h(s), C^h(s): s\le t\}$. At each step n, we can write

$$\begin{aligned} \Delta X_n^h = \Delta X_n^h I_{\{\text {harvesting step at }n\}}+ \Delta X_n^h I_{\{\text {seeding step at }n\}}. \end{aligned}$$

(B.7)

Thus, we obtain

$$\begin{aligned} X^h_n = x + \sum \limits _{m=0}^{n-1} \Delta X_m^h I_{\{ \pi ^h_m\ge 1\}}+ \sum \limits _{m=0}^{n-1} \Delta X_m^h I_{\{ \pi ^h_m=0\}}. \end{aligned}$$

(B.8)

This implies

$$\begin{aligned} X^h(t) = x + B^h(t) + M^h(t) -Y^h(t). \end{aligned}$$

(B.9)

Recall that $\Delta t^h_m = h^2/Q_h(X^h_m, u^h_m)$ if $\pi ^h_m=0$ and $\Delta t^h_m = 0$ if $\pi ^h_m\ge 1$. It follows that

$$\begin{aligned} B^h(t)&= \sum \limits _{m=0}^{n^h(t)-1} \Big [ b (X^h_m) +C^h_m \Big ]\Delta t^h_m\nonumber \\&=\int _0^t \Big [ b (X^h(s)) + C^h(s) \Big ] ds-\int _{t^h_{n^h(t)}}^t \Big [ b (X^h(s)) +C^h(s)\Big ] ds\nonumber \\&= \int _0^t \Big [ b (X^h(s)) +C^h(s) \Big ]ds + \varepsilon ^h_1(t), \end{aligned}$$

(B.10)

with $\{\varepsilon _1^h(\cdot )\}$ being an ${\mathcal {F}}^h(t)$-adapted process satisfying

$$\begin{aligned} \lim \limits _{h\rightarrow 0} \sup \limits _{t\in [0, T_0]}\mathbb {E}|\varepsilon _1^h(t)|=0 \quad \text {for any }0<T_0<\infty . \end{aligned}$$

We now attempt to represent $M^h(\cdot )$ in a form similar to the diffusion term in (2.7). Factor

$$\begin{aligned} a(x)= \sigma (x)\sigma '(x)=P(x)D^2(x)P'(x), \end{aligned}$$

where $P(\cdot )$ is an orthogonal matrix, $D(\cdot )=\mathrm{diag}\{r_1(\cdot ), ..., r_d (\cdot )\}$. Without loss of generality, we suppose that $\inf \limits _{x}r_i(x)>0$ for all $i=1, \ldots , d$. Define $D_0(\cdot )=\mathrm{diag}\{1/r_1(\cdot ), ..., 1/r_d (\cdot )\}$.

Remark B.2

In the argument above, for simplicity, we assume that the diffusion matrix a(x) is nondegenerate. If this is not the case, we can use the trick from (Kushner and Dupuis 1992, p.288-289) to establish equation (B.12).

Define $W^h(\cdot )$ by

$$\begin{aligned} W^h(t)&= \int _0^t D_0 (X^h(s)) P' (X^h(s))dM^h(s)\nonumber \\&= \sum \limits _{m=0}^{n^h(t)-1} D_0 (X^h_m) P' (X^h_m)(\Delta \xi ^h_m -\mathbb {E}^{h}_m \Delta \xi ^h_m)I_{\{ \pi ^h_m=0\}}. \end{aligned}$$

(B.11)

Then we can write

$$\begin{aligned} M^h(t) =\int _0^t \sigma (X^h(s)) dW^h(s) + \varepsilon _2^h(t), \end{aligned}$$

(B.12)

with $\{\varepsilon _2^h(\cdot )\}$ being an ${\mathcal {F}}^h(t)$-adapted process satisfying

$$\begin{aligned} \lim \limits _{h\rightarrow 0} \sup \limits _{t\in [0, T_0]}\mathbb {E}|\varepsilon _2^h(t)|=0 \quad \text {for any }0<T_0<\infty . \end{aligned}$$

Using (B.10) and (B.12), we can write (B.9) as

$$\begin{aligned} X^h(t) = x + \int _0^t \Big [ b (X^h(s)) +C^h(s) ]ds + \int _0^t \sigma (X^h(s)) dW^h(s) -Y^h(t)+\varepsilon ^h(t), \end{aligned}$$

(B.13)

where $\varepsilon ^h(\cdot )$ is an ${\mathcal {F}}^h(t)$-adapted process satisfying

$$\begin{aligned} \lim \limits _{h\rightarrow 0} \sup \limits _{t\in [0, T_0]}\mathbb {E}|\varepsilon ^h(t)|=0 \quad \text {for any }0<T_0<\infty . \end{aligned}$$

The objective function from (2.12) can be rewritten as

$$\begin{aligned} J^h(x, Y^h, C^h)= \mathbb {E}\left[ \int _0^{\infty } e^{-\delta s} f \cdot dY^{h}(s)- \int _0^{\infty } e^{-\delta s} g(X^h(s))\cdot C^h(s) d(s)\right] .\nonumber \\ \end{aligned}$$

(B.14)

Time rescaling Next we will introduce “stretched-out” time scale. This is similar to the approach previously used by Kushner and Martins (1991) and Budhiraja and Ross (2007) for singular control problems. Using the new time scale, we can overcome the possible non-tightness of the family of processes $\{Y^h(\cdot )\}$.

Define the rescaled time increments $\{\Delta \widehat{t}_n^h: n\in {\mathbb {Z}}_{\ge 0}\}$ by

$$\begin{aligned} \begin{array}{ll} &{}\displaystyle \Delta \widehat{t}^h_n = \Delta t^h_n I_{\{ \pi ^h_n =0 \}} + h I_{\{ \pi ^h_n \ge 1 \}}, \qquad \widehat{t}_0=0, \qquad \widehat{t}_n = \sum \limits _{k=0}^{n-1}\Delta \widehat{t}^h_k, \quad n\ge 1.\\ \end{array} \end{aligned}$$

(B.15)

Definition B.3

The rescaled time process $\widehat{T}^h(\cdot )$ is the unique continuous nondecreasing process satisfying the following:

• (a) $\widehat{T}^h(0)=0$;

• (b) the derivative of $\widehat{T}^h(\cdot )$ is 1 on $(\widehat{t}^h_n, \widehat{t}^h_{n+1})$ if $\pi ^h_n=0$, i.e., n is a seeding step;

• (c) the derivative of $\widehat{T}^h(\cdot )$ is 0 on $(\widehat{t}^h_n, \widehat{t}^h_{n+1})$ if $\pi ^h_n\ge 1$, i.e., n is a harvesting step.

Define the rescaled and interpolated process $\widehat{X}^h(t)= X^h(\widehat{T}^h(t))$ and likewise define $\widehat{Y}^h(\cdot )$, $\widehat{C}^h(\cdot )$, $\widehat{B}^h(\cdot )$, $\widehat{M}^h(\cdot )$, and the filtration $\widehat{{\mathcal {F}}}^h(\cdot )$ similarly. It follows from (B.9) that

$$\begin{aligned} \widehat{X}^h(t)=x+\widehat{B}^h(t) + \widehat{M}^h(t) - \widehat{Y}^h(t). \end{aligned}$$

(B.16)

Using the same argument we used for (B.13) we obtain

$$\begin{aligned} \widehat{X}^h(t) = x + \int _0^t \Big [ b (\widehat{X}^h(s)) + \widehat{C}^h(s) \Big ]d\widehat{T}^h(s) + \int _0^t \sigma (\widehat{X}^h(s)) d \widehat{W}^h(s) - \widehat{Y}^h(t) + \widehat{\varepsilon }^h(t), \end{aligned}$$

(B.17)

with $\widehat{\varepsilon }^h(\cdot )$ is an $\widehat{{\mathcal {F}}}^h(\cdot )$-adapted process satisfying

$$\begin{aligned} \lim \limits _{h\rightarrow 0} \sup \limits _{t\in [0, T_0]}\mathbb {E}|\widehat{\varepsilon }^h(t)|=0 \quad \text {for any }0<T_0<\infty . \end{aligned}$$

(B.18)

Define

$$\begin{aligned} {A}^h(t)=\int _0^t {C}^h(s) ds, \quad \widehat{A}^h(t)=\int _0^t \widehat{C}^h(s) \widehat{T}^h(s), \quad t\ge 0, h> 0. \end{aligned}$$

(B.19)

1.3 Convergence

Using weak convergence methods, we can obtain the convergence of the algorithms. Let $D[0, \infty )$ denote the space of functions that are right continuous and have left-hand limits endowed with the Skorokhod topology. All the weak analysis will be on this space or its k-fold products $D^k[0, \infty )$ for appropriate k.

Theorem B.4

Suppose Assumptions 2.1 and B.1 hold. Let the chain $\{X^h_n \}$ be constructed with transition probabilities defined in (B.3)–(B.4), $X^h(\cdot )$, $W^h(\cdot )$, $Y^h(\cdot )$, and $A^h(\cdot )$ be the continuous-time interpolation defined in (B.5)–(B.6), (B.11), and (B.19). Let $\widehat{X}^h(\cdot )$, $\widehat{W}^h(\cdot )$, $\widehat{Y}^h(\cdot )$, $\widehat{A}^h(\cdot )$ be the corresponding rescaled processes, $\widehat{T}^h(\cdot )$ be the process from Definition B.3, and denote

$$\begin{aligned} \widehat{H}^h(\cdot )=\Big (\widehat{X}^h(\cdot ), \widehat{W}^h(\cdot ), \widehat{Y}^h(\cdot ), \widehat{A}^h(\cdot ), \widehat{T}^h(\cdot )\Big ). \end{aligned}$$

Then the family of processes $(\widehat{H}^h)_{h>0}$ is tight. As a result, $(\widehat{H}^h)_{h>0}$ has a weakly convergent subsequence with limit

$$\begin{aligned} \widehat{H}(\cdot )=\Big (\widehat{X}(\cdot ),\widehat{W}(\cdot ), \widehat{Y}(\cdot ), \widehat{A}(\cdot ), \widehat{T}(\cdot )\Big ). \end{aligned}$$

Proof

We use the tightness criteria used by (Kushner 1984, p. 47). Specifically, a sufficient condition for tightness of a sequence of processes $\zeta ^h(\cdot )$ with paths in $D^k[0, \infty )$ is that for any constants $T_0, \rho \in (0, \infty )$,

$$\begin{aligned}&\mathbb {E}_t^h\big |\zeta ^h(t+s)-\zeta ^h(t)\big |^2\le \mathbb {E}^h_t \gamma (h, \rho ) \quad \text {for all}\quad s\in [0, \rho ], \quad t\le T_0,\\&\quad \lim \limits _{\rho \rightarrow 0}\limsup \limits _{h\rightarrow 0} \mathbb {E}\gamma (h, \rho ) =0. \end{aligned}$$

The proof for the tightness of $\widehat{W}^h(\cdot )$ is standard; see for example Kushner and Martins (1991), Jin et al. (2013). We show the tightness of $\widehat{Y}^h(\cdot )$ to demonstrate the role of time rescaling. Following the definition of “stretched out” timescale, for any constants $T_0, \rho \in (0, \infty )$, $s\in [0, \rho ]$ and $t\le T_0$,

$$\begin{aligned} \begin{array}{ll} \mathbb {E}^h_{t}|\widehat{Y}^h(t+s) - \widehat{Y}^h(t)|^2&{} \le d h^2 \mathbb {E}^h_{t}(\text {number of harvesting steps in}\\ &{} \displaystyle \qquad \text {interpolated interval } [t, t+s) )^2\\ &{}\le d h^2 \max \{1, \rho ^2/h^2 \} \\ &{} \le d (h^2 + \rho ^2). \end{array} \end{aligned}$$

(B.20)

Thus $\{\widehat{Y}^h(\cdot )\}$ is tight. The tightness of $\{\widehat{T}^h(\cdot )\}$ follows from the fact that

$$\begin{aligned} 0\le \widehat{T}^h(t+s)-\widehat{T}^h(t)\le \rho . \end{aligned}$$

Since $|\widehat{A}^h(t+s)-\widehat{A}^h(t)|\le |\widehat{T}^h(t+s)-\widehat{T}^h(t)|\sum _{i=1}^d \lambda _i$, it follows that $\{\widehat{A}^h(\cdot )\}$ is tight. The tightness of $\{\widehat{X}^h(\cdot )\}$ follows from (B.16), (B.20). Hence $\{\widehat{X}^h(\cdot ), \widehat{W}^h(\cdot ), \widehat{Y}^h(\cdot ), \widehat{A}^h(\cdot ), \widehat{T}^h(\cdot )\}$ is tight. By virtue of Prohorov’s Theorem, $\widehat{H}^h(\cdot )$ has a weakly convergent subsequence with the limit $\widehat{H}(\cdot )$. This completes the proof.

$\square$

We proceed to characterize the limit process.

Theorem B.5

Under conditions of Theorem B.4, let $\widehat{{\mathcal {F}}}(t)$ be the $\sigma$-algebra generated by

$$\begin{aligned} \{\widehat{X}(s), \widehat{W}(s), \widehat{Y}(s), \widehat{A}(s), \widehat{T}(s):s \le t\}. \end{aligned}$$

Then the following assertions hold.

• (a) $\widehat{X}(\cdot )$, $\widehat{W}(\cdot )$, $\widehat{Y}(\cdot )$, $\widehat{A}(\cdot )$, and $\widehat{T}(\cdot )$ have continuous paths with probabilty one, $\widehat{Y}(\cdot )$ and $\widehat{T}(\cdot )$ are nondecreasing and nonnegative. Moreover, $\widehat{T}(\cdot )$ is Lipschitz continuous with Lipschitz coefficient 1.

• (b) There exists an $\{\widehat{{\mathcal {F}}}(\cdot )\}$-adapted process $\widehat{C}(\cdot )$ with $\widehat{C}(t)\in [0, \lambda ]$ for any $t\ge 0$, such that $\widehat{A}(t)=\int _0^t \widehat{C}(s)d\widehat{T}(s)$ for any $t\ge 0$.

• (c) $\widehat{W}(t)$ is an $\widehat{{\mathcal {F}}}(t)$-martingale with quadratic variation process $\widehat{T}(t)I_d$, where $I_d$ is the $d\times d$ identity matrix.

• (d) The limit processes satisfy

  $$\begin{aligned} \widehat{X}(t) = x + \int _0^t \big [ b (\widehat{X}(s)) + \widehat{C}(s) \big ]d\widehat{T}(s) + \int _0^t \sigma (\widehat{X}(s)) d \widehat{W}(s) - \widehat{Y}(t). \end{aligned}$$

  (B.21)

Proof

(a) Since the sizes of the jumps of $\widehat{X}^h(\cdot )$, $\widehat{W}^h(\cdot )$, $\widehat{Y}^h(\cdot )$, $\widehat{A}^h(\cdot )$, $\widehat{T}^h(\cdot )$ go to 0 as $h\rightarrow 0$, the limits of these processes have continuous paths with probability one (see (Kushner 1990, p. 1007)). Moreover, $\widehat{Y}^h(\cdot )$ (resp. $\widehat{T}^h(\cdot )$) converges uniformly to $\widehat{Y}(\cdot )$, (resp. $\widehat{T}(\cdot )$) on bounded time intervals. This, together with the monotonicity and non-negativity of $\widehat{Y}^h(\cdot )$ and $\widehat{T}^h(\cdot )$ implies that the processes $\widehat{Y}(\cdot )$ and $\widehat{T}(\cdot )$ are nondecreasing and nonnegative.

(b) Since $|\widehat{A}^h_i(t+s)-\widehat{A}^h_i(t)|\le \lambda _i |\widehat{T}^h(t+s)-\widehat{T}^h(t)|$ for any $t\ge 0, s\ge 0, h>0, i=1, 2,\ldots , d$ and by virtue of Skorohod representation, $|\widehat{A}_i(t+s)-\widehat{A}_i(t)|\le \lambda _i |\widehat{T}(t+s)-\widehat{T}(t)|$ for any $t\ge 0, s\ge 0, i=1, 2,\ldots , d$; that is, each $\widehat{A}_i$ is absolutely continuous with respect to $\widehat{T}$. Therefore, there exists a $[0,\lambda _i]$-valued $\{\widehat{{\mathcal {F}}}(t)\}$-adapted process $\widehat{C}_i(\cdot )$ such that $\widehat{A}_i(t)=\int _0^t \widehat{C}_i(s)d\widehat{T}(s)$ for any $t\ge 0$. Then $C(\cdot )=(C_1(\cdot ), \ldots , C_d(\cdot ))'$ is the desired process.

(c) Let $\widehat{\mathbb {E}}_t^h$ denote the expectation conditioned on $\widehat{{\mathcal {F}}}^h(t)={\mathcal {F}}^h(\widehat{T}^h(t))$. Recall that $W^h(\cdot )$ is an ${\mathcal {F}}^h(\cdot )$- martingale and by the definition of $\widehat{W}^h(\cdot )$, for any $\rho >0$,

$$\begin{aligned}&\displaystyle \widehat{\mathbb {E}}_t^h \big (\widehat{W}^h(t+\rho )-\widehat{W}^h(t)\big )=0, \nonumber \\&\displaystyle \widehat{\mathbb {E}}_t^h \big (\widehat{W}^h(t+\rho )\widehat{W}^h(t+\rho )'-\widehat{W}^h(t)\widehat{W}^h(t)'\big ) = \big (\widehat{T}^h(t+\rho )-\widehat{T}^h(t)\big )I_d + \widehat{\varepsilon }^h(\rho ), \end{aligned}$$

(B.22)

where $\mathbb {E}|\widehat{\varepsilon }^h(\rho )|\rightarrow 0$ as $h\rightarrow 0$. To characterize $\widehat{W}(\cdot )$, let q be an arbitrary integer, $t>0$, $\rho >0$ and $\{t_k: k\le q\}$ be such that $t_k\le t<t+\rho$ for each k. Let $\Psi (\cdot )$ be a real-valued and continuous function with compact support. Then in view of (B.22), we have

$$\begin{aligned} \mathbb {E}\Psi (\widehat{H}^h(t_k), k\le q)\Big [ \widehat{W}^h(t+\rho )-\widehat{W}^h(t)\Big ]=0, \end{aligned}$$

(B.23)

and

$$\begin{aligned}&\mathbb {E}\Psi (\widehat{H}^h(t_k), k\le q)\Big [ \big (\widehat{W}^h(t+\rho )\widehat{W}^h(t+\rho )'-\widehat{W}^h(t)\widehat{W}^h(t)' \nonumber \\&\quad -\big (\widehat{T}^h(t+\rho )-\widehat{T}^h(t)\big )I_d-\widehat{\varepsilon }^h(\rho )\Big ]=0. \end{aligned}$$

(B.24)

By the Skorokhod representation and the dominated convergence theorems, letting $h\rightarrow 0$ in (B.23), we obtain

$$\begin{aligned} \mathbb {E}\Psi (\widehat{H}(t_k), k\le q)\Big [ \widehat{W}(t+\rho )-\widehat{W}(t)\Big ]=0. \end{aligned}$$

(B.25)

Since $\widehat{W}(\cdot )$ has continuous paths with probability one, (B.25) implies that $\widehat{W}(\cdot )$ is a continuous $\widehat{{\mathcal {F}}}(\cdot )$-martingale. Moreover, (B.24) gives us that

$$\begin{aligned} \mathbb {E}\Psi (\widehat{H}(t_k), k\le q)\Big [ \widehat{W}(t+\rho )\widehat{W}(t+\rho )'-\widehat{W}(t)\widehat{W}(t)'-\big (\widehat{T}(t+\rho )-\widehat{T}(t)\big )I_d\Big ]=0.\nonumber \\ \end{aligned}$$

(B.26)

This implies part (c).

(d) The proof of this part is motivated by that of (Kushner and Dupuis 1992, Theorem 10.4.1). By virtue of Skorohod representation,

$$\begin{aligned} \int _0^t \Big [ b (\widehat{X}^h(s)) + \widehat{C}^h(s) \Big ]d\widehat{T}^h(s) \rightarrow \int _0^t \Big [ b (\widehat{X}(s)) + \widehat{C}(s) \Big ]d\widehat{T}(s), \end{aligned}$$

(B.27)

as $h\rightarrow 0$ uniformly in t on any bounded time interval with probability one.

For each positive constant $\rho$ and a process $\widehat{\nu }(\cdot )$, define the piecewise constant process $\widehat{\nu }^\rho (\cdot )$ by $\widehat{\nu }^\rho (t)=\widehat{\nu }(k\rho )$ for $t\in [k\rho , k\rho +\rho ), k\in {\mathbb {Z}}_{\ge 0}$. Then, by the tightness of $(\widehat{X}^h(\cdot ))$, (B.17) can be rewritten as

$$\begin{aligned} \widehat{X}^h(t) = x_0 + \int _0^t \Big [b (\widehat{X}^h(s)) + \widehat{C}^h(s)\Big ] d\widehat{T}^h(s) + \int _0^t \sigma (\widehat{X}^{h, \rho }(s)) d \widehat{W}^h(s) - \widehat{Y}^h(t) + \widehat{\varepsilon }^{h, \rho }(t), \end{aligned}$$

(B.28)

where $\lim \limits _{\rho \rightarrow 0}\limsup \limits _{h\rightarrow 0} \mathbb {E}|\widehat{\varepsilon }^{h, \rho }(t)|=0.$ Owing to the fact that $\widehat{X}^{h, \rho }$ takes constant values on the intervals $[k\rho , k\rho +\rho )$, we have

$$\begin{aligned} \int _0^t \sigma (\widehat{X}^{h, \rho }(s))d\widehat{W}^h(s)\rightarrow \int _0^t \sigma (\widehat{X}^\rho (s))d\widehat{W}(s) \quad \text { as }\quad h\rightarrow 0, \end{aligned}$$

(B.29)

which are well defined with probability one since they can be written as finite sums. Combining (B.27)–(B.29), we have

$$\begin{aligned} \widehat{X}(t)=x_0 +\int _0^t \Big [b(\widehat{X}(s))+\widehat{C}(s)\Big ]d\widehat{T}(s)+\int _0^t \sigma (\widehat{X}^\rho (s))d\widehat{W}(s) - \widehat{Y}(t)+\widehat{\varepsilon }^{\rho }(t), \end{aligned}$$

(B.30)

where $\lim \limits _{\rho \rightarrow 0}E|\widehat{\varepsilon }^{ \rho }(t)|=0.$ Taking the limit $\rho \rightarrow 0$ in the above equation yields the result. $\square$

For $t<\infty$, define the inverse ${\overline{T}}(t)= \inf \{s: \widehat{T}(s)>t\}$. For any process $\widehat{\nu }(\cdot )$, define the time-rescaled process $(\overline{\nu }(\cdot ))$ by $\overline{\nu }(t)= \widehat{\nu }({\overline{T}}(t))$ for $t\ge 0$. Let ${\mathcal {\overline{F}}}(t)$ be the $\sigma$-algebra generated by $\{\overline{X}(s), {\overline{W}}(s), {\overline{Y}}(s), {\overline{C}}(s), {\overline{T}}(s): s\le t\}$. Let $V^h(x)$ and $V^U(x)$ be value the functions defined in (2.13) and (2.9), respectively.

Theorem B.6

Under conditions of Theorem B.4, the following assertions are true.

• (a) $\overline{T}$ is right continuous, nondecreasing, and $\overline{T}(t)\rightarrow \infty$ as $t \rightarrow \infty$ with probability one.

• (b) The processes $\overline{Y}(t)$ and $\overline{C}(t)$ are $\mathcal {\overline{F}}(t)$-adapted. Moreover, $\overline{Y}(t)$ is right-continuous, nondecreasing, nonnegative; $\overline{C}(t)\in [0, \lambda ]$ for any $t\ge 0$.

• (c) $\overline{W}(\cdot )$ is an $\mathcal {\overline{F}}(t)$-adapted standard Brownian motion, and

  $$\begin{aligned} {\overline{X}}(t)=x +\int _0^t \Big [b(\overline{X}(s)) +\overline{C}(s)\Big ] ds+\int _0^t \sigma (\overline{X}(s))d\overline{W}(s)-\overline{Y}(t), \quad t\ge 0.\nonumber \\ \end{aligned}$$

  (B.31)

Proof

(a) We will argue via contradiction that $\widehat{T}(t)\rightarrow \infty$ as $t\rightarrow \infty$ with probability one. Suppose $\mathbb {P}[\sup _{t\ge 0}\widehat{T}(t)<\infty ]>0$. Then there exist positive constants $\varepsilon$ and $T_0$ such that

$$\begin{aligned} \mathbb {P}[\sup \limits _{t\ge 0}\widehat{T}(t)<T_0-1]>\varepsilon . \end{aligned}$$

(B.32)

We first observe that

$$\begin{aligned} t+d|Y^h(t)|\ge \sum \limits _{k=0}^{n^h(t)-1}\Big (\Delta t^h_n I_{\{\pi ^h_k=0\}} + h I_{\{\pi ^h_k\ge 1\}}\Big ). \end{aligned}$$

Since $\widehat{T}^h(\cdot )$ is nondecreasing and $\widehat{T}^h(\widehat{t}^h_n)=t^h_n$,

$$\begin{aligned} \widehat{T}^h\big (t+d|Y^h(t)|\big )&\ge \widehat{T}^h \Big (\sum \limits _{k=0}^{n^h(t)-1}\big ( \Delta t^h_k I_{\{\pi ^h=0\}} + hI_{\{\pi ^h_k\ge 1\}}\big )\Big )\nonumber \\&= \widehat{T}^h(\widehat{t}^h_{n^h(t)})={t}^h_{n^h(t)}\ge t-1. \end{aligned}$$

(B.33)

The last inequality above is a consequence of the inequalities $t^h_{n^h(t)}\le t< t^h_{n^h(t)+1}=t^h_{n^h(t)}+\Delta t^h_{n+1} <t^h_{n^h(t)}+1$.

It follows from (B.9) that for each fixed $t\ge 0$, $\sup \limits _{h}\mathbb {E}\big (|Y^h(t)|\big )<\infty .$ Thus, for a sufficiently large K,

$$\begin{aligned} \mathbb {P}\{ d|Y^h(T_0)| \ge 2K \} \le \dfrac{d\mathbb {E}\big |Y^h(T_0)\big |}{2K}< \dfrac{\varepsilon }{2}. \end{aligned}$$

(B.34)

In views of (B.33) and (B.34), we obtain

$$\begin{aligned} \mathbb {P}\big [\widehat{T}^h(T_0+2K)<T_0-1 \big ]&\le \mathbb {P}\big [\widehat{T}^h\big (T_0+d|Y^h(T_0)\big )<T_0-1 , d|Y^h(T_0)|<2K\big ]\nonumber \\&\quad + \mathbb {P}\big [ d|Y^h(T_0)|\ge 2K\big ]\nonumber \\&< \dfrac{\varepsilon }{2} \qquad \text {for small }h. \end{aligned}$$

(B.35)

Since $\widehat{T}^h$ converges weakly to $\widehat{T}$, it follows from (B.35) that $\liminf \limits _{h\rightarrow 0} \mathbb {P}\big [\widehat{T}^h(T_0+2K) <T_0-1 \big ]\le \varepsilon /2$. This contradicts (B.32) (see (Billingsley 1968, Theorem 1.2.1)). Hence $\widehat{T}(t)\rightarrow \infty$ as $t\rightarrow \infty$ with probability one. Thus ${\overline{T}}(t)<\infty$ for all t and ${\overline{T}}(t)\rightarrow \infty$ as $t\rightarrow \infty$. Since $\widehat{T}(\cdot )$ is nondecreasing and continuous, ${\overline{T}}(\cdot )$ is nondecreasing and right-continuous.

(b) The properties of $\overline{Y}(\cdot )$ follow from the fact that $\widehat{Y}(\cdot )$ is continuous, nondecreasing, nonnegative, and ${\overline{T}}(\cdot )$ is right-continuous. The properties of $\overline{C}(\cdot )$ follow from those of $\widehat{C}(\cdot )$.

(c) Note that although ${\overline{T}}(\cdot )$ might fail to be continuous, $\overline{W}(\cdot )=\widehat{W}({\overline{T}}(\cdot ))$ has continuous paths with probability one. Indeed, consider the tight sequence $\big ({W}^h(\cdot ), \widehat{W}^h(\cdot ), \widehat{T}^h(\cdot )\big )$ with the weak limit $\big (\widetilde{W}(\cdot ), \widehat{W}(\cdot ), \widehat{T}(\cdot )\big )$. Since $\widehat{W}^h(\cdot )=W^h(\widehat{T}^h(\cdot ))$, we must have that $\widehat{W}(\cdot )=\widetilde{W}(\widehat{T}(\cdot ))$. It follows from the definition of ${\overline{T}}(\cdot )$ that for each $t\ge 0$, we have $\widehat{T}({\overline{T}}(t))=t$. Hence $\overline{W}(t)=\widehat{W}({\overline{T}}(t))=\widetilde{W}\big (\widehat{T}({\overline{T}}(t))\big )=\widetilde{W}(t)$. Since the sizes of the jumps of $W^h(\cdot )$ go to 0 as $h\rightarrow 0$, $\widetilde{W}(\cdot )$ also has continuous paths with probability 1. This shows that $\overline{W}(\cdot )=\widehat{W}({\overline{T}}(\cdot ))$ has continuous paths with probability 1. Before characterizing $\overline{W}(\cdot )$, we note that for $t\ge 0$, $\{{\overline{T}}(s)\le t\}=\{\widehat{T}(t)\ge s\}\in \widehat{{\mathcal {F}}}(t)$ since $\widehat{T}(t)$ is $\widehat{{\mathcal {F}}}(t)$-measurable. Thus ${\overline{T}}(s)$ is an $\widehat{{\mathcal {F}}}(t)$-stopping time for each $s\ge 0$. Since $\widehat{W}(t)$ is an $\widehat{{\mathcal {F}}}(t)$-martingale with quadratic variation process $\widehat{T}(t) I_d$,

$$\begin{aligned} \mathbb {E}\big [\widehat{W}({\overline{T}}(t)\wedge n)| \widehat{{\mathcal {F}}}({\overline{=} T}(s))\big ]&=\widehat{W}({\overline{T}}(s)\wedge n), \quad n=1,2, \ldots , \nonumber \\ \mathbb {E}\widehat{W}({\overline{T}}(t)\wedge n)\widehat{W}({\overline{T}}(t)\wedge n)'&=\mathbb {E}\widehat{T}({\overline{T}}(t)\wedge n)I_d, \end{aligned}$$

(B.36)

and $\widehat{T}({\overline{T}}(t)\wedge n)\le \widehat{T}({\overline{T}}(t))=t$. Hence for each fixed $t\ge 0$, the family $\{\widehat{W}({\overline{T}}(t)\wedge n), n\ge 1\}$ is uniformly integrable. By that uniform integrability, we obtain from (B.36) that $E\big [\widehat{W}({\overline{T}}(t))| \widehat{{\mathcal {F}}}({\overline{T}}(s))\big ]=\widehat{W}({\overline{T}}(s))$, that is $E\big [\overline{W}(t)| \overline{{\mathcal {F}}}(s)\big ]=\overline{W}(s)$. This proves that $\overline{W}(\cdot )$ is a continuous $\overline{{\mathcal {F}}}(\cdot )$ -martingale. We next consider its quadratic variation. By the Burkholder–Davis–Gundy inequality, there exists a positive constant K independent of $n=1, 2,\ldots$ such that

$$\begin{aligned} \mathbb {E}|\widehat{W}({\overline{T}}(t)\wedge n)|^2\le K\mathbb {E}\bigg [\Big (\sup \limits _{0\le s\le {\overline{T}}(t)}|\widehat{W}({\overline{T}}(s)\wedge n)|^2 \Big )\bigg ]\le K\mathbb {E}|\widehat{T}({\overline{T}}(t)\wedge n)|\le Kt. \end{aligned}$$

Thus the families $\{\widehat{W}({\overline{T}}(t)\wedge n), n\ge 1\}$ and $\{\widehat{T}({\overline{T}}(t)\wedge n), n\ge 1\}$ are uniformly integrable for each fixed $t\ge 0$. Combining this with the fact that $\widehat{W}(\cdot )$, $\widehat{T}(\cdot )$ have continuous paths, for nonnegative constants $s\le t$, we have

$$\begin{aligned}&\widehat{W}({\overline{T}}(s)\wedge n)\widehat{W}({\overline{T}}(s)\wedge n)' -\widehat{T}({\overline{T}}(s)\wedge n)I_d\nonumber \\&\quad = \mathbb {E}\big [ \widehat{W}({\overline{T}}(t)\wedge n)\widehat{W}({\overline{T}}(t)\wedge n)'-\widehat{T}({\overline{T}}(t)\wedge n)I_d | \widehat{{\mathcal {F}}}({\overline{T}}(s))\big ]\nonumber \\&\quad \rightarrow \mathbb {E}\big [ \widehat{W}({\overline{T}}(t))\widehat{W}({\overline{T}}(t))'-\widehat{T}({\overline{T}}(s))I_d | \widehat{{\mathcal {F}}}({\overline{T}}(s))\big ]\nonumber \\&\quad = \mathbb {E}\big [ \overline{W}(t)\overline{W}(t)'-t I_d | \overline{{\mathcal {F}}}(s)\big ]. \end{aligned}$$

(B.37)

Note that the first equation in (B.37) follows from the martingale property of $\widehat{W}(\cdot )\widehat{W}(\cdot )'-\widehat{T}(\cdot )I_d$ with respect to $\widehat{{\mathcal {F}}}(t).$ Letting $n\rightarrow \infty$ in (B.37), we arrive at

$$\begin{aligned} \mathbb {E}\big [ \overline{W}(t)\overline{W}(t)'-tI_d | {\mathcal {F}}(s)\big ]=\overline{W}(s)\overline{W}(s)'-s I_d. \end{aligned}$$

Therefore, $\overline{W}(\cdot )$ is an $\overline{{\mathcal {F}}}(t)$—adapted standard Brownian motion. A rescaling of (B.21) yields

$$\begin{aligned} \overline{X}(t)=x +\int _0^t \Big [b(\overline{X}(s)) +\overline{C}(s)\Big ]ds+\int _0^t \sigma (\overline{X}(s))d\overline{W}(s)-\overline{Y}(t). \end{aligned}$$

The proof is complete. $\square$

Theorem B.7

Under conditions of Theorem B.4, let $V^h(x)$ and $V^U(x)$ be value functions defined in (2.13) and (2.9), respectively. Then $V^h(x)\rightarrow V^U(x), x\in [0,U]^d$ as $h\rightarrow 0$. If (2.10) holds, then $V^h(x)\rightarrow V(x), x\in [0,U]^d$ as $h\rightarrow 0$.

Proof

We first show that as $h\rightarrow 0$,

$$\begin{aligned} J^h(x, u^h) \rightarrow J(x, \overline{Y}(\cdot ), \overline{C}(\cdot )), \end{aligned}$$

(B.38)

where $u^h=(\pi ^h, C^h)$. Indeed, for an admissible strategy $u^h=(\pi ^h_n, C^h_n)$, we have

$$\begin{aligned} J^h (x, u^h)&= \mathbb {E}\bigg [\sum _{m=1}^{\infty } e^{-\delta t_m^h} f\cdot \Delta Y_{m}^{h}- \sum _{m=1}^{\infty } e^{-\delta t_m^h} g(X^h_m) \cdot C^h_m \Delta t^h_m\bigg ].\nonumber \\&= \mathbb {E}\Big [ \int _0^{\infty } e^{-\delta \widehat{T}^h(t)} f\cdot d\widehat{Y}^{h}(t)-\int _0^{\infty } e^{-\delta \widehat{T}^h(t)} g(\widehat{X}^h(t))\cdot \widehat{C}^{h}(t)d\widehat{T}^h(t)\Big ]. \end{aligned}$$

(B.39)

By a small modification of the proof in Theorem B.6 (a), we have $\widehat{T}^h(t)\rightarrow \infty$ as $t\rightarrow \infty$ with probability 1. It also follows from the representation (B.9) and estimates on $B^h(\cdot )$ and $M^h(\cdot )$ that $\{Y^h(n+1)-Y^h(n): n, h\}$ is uniformly integrable. Thus, by the definition of $\widehat{T}^h(\cdot )$,

$$\begin{aligned} \mathbb {E}\int _{T_0}^{\infty }e^{-\delta \widehat{T}^h(t)} f\cdot d\widehat{Y}^{h}(t)&\le \mathbb {E}\int _{\min \{t:\widehat{T}^h(t)\ge T_0 \}}^{\infty } K e^{-\delta s}\cdot d{Y}^{h}(s)\\&\le \mathbb {E}\int _{T_0 }^{\infty } K e^{-\delta s}\cdot d{Y}^{h}(s)\rightarrow 0, \end{aligned}$$

uniformly in h as $T_0\rightarrow \infty$. In the above argument, we have used that $\widehat{T}^h(T_0)\le T_0$. Then by the weak convergence, the Skohorod representation, and uniform integrability we have for any $T_0>0$ that

$$\begin{aligned} \mathbb {E}\int _0^{T_0} e^{-\delta \widehat{T}^h(t)} f\cdot d\widehat{Y}^{h}(t)\rightarrow \mathbb {E}\int _0^{T_0} e^{-\delta \widehat{T}(t)} f\cdot d\widehat{Y}(t). \end{aligned}$$

Therefore, we obtain

$$\begin{aligned} \mathbb {E}\int _0^{\infty } e^{-\delta \widehat{T}^h(t)} f\cdot d\widehat{Y}^{h}(t)\rightarrow \mathbb {E}\int _0^{\infty } e^{-\delta \widehat{T}(t)} f\cdot d\widehat{Y}(t). \end{aligned}$$

Similarly,

$$\begin{aligned} \mathbb {E}\int _0^{\infty } e^{-\delta \widehat{T}^h(t)} g(\widehat{X}^h(t))\cdot \widehat{C}^h(t) d\widehat{T}^{h}(t)\rightarrow \mathbb {E}\int _0^{\infty } e^{-\delta \widehat{T}(t)} g(\widehat{X}(t))\cdot \widehat{C}(t) d\widehat{T}(t). \end{aligned}$$

On inversion of the timescale, we have

$$\begin{aligned} J^h(x, u^h)\rightarrow \mathbb {E}\Big [\int _0^\infty e^{-\delta t} f\cdot d\overline{Y}(t)-\int _0^\infty e^{-\delta t} g( \overline{X}(t))\cdot d\overline{C}(t)dt\Big ]. \end{aligned}$$

Thus, $J^h(x, u^h)\rightarrow J(x, \overline{Y}(\cdot ), \overline{C}(\cdot ))$ as $h\rightarrow 0$.

Next, we prove that

$$\begin{aligned} \limsup \limits _{h} V^h(x) \le V^U(x). \end{aligned}$$

(B.40)

For any small positive constant $\varepsilon$, let $\{\widetilde{u}^h\}$ be an $\varepsilon$-optimal harvesting strategy for the chain $\{X^h_n\}$; that is,

$$\begin{aligned} V^h(x)=\sup \limits _{u^h} J^h(x, u^h)\le J^h(x, \widetilde{u}^h) + \varepsilon . \end{aligned}$$

Choose a subsequence $\{\widetilde{h}\}$ of $\{h\}$ such that

$$\begin{aligned} \limsup \limits _{{h}\rightarrow 0} V^{{h}}(x)=\lim \limits _{\widetilde{h}\rightarrow 0}V^{\widetilde{h}} (x)\le \limsup \limits _{\widetilde{h}\rightarrow 0} J^{\widetilde{h}}(x, {\widetilde{u}}^{\widetilde{h}})+\varepsilon . \end{aligned}$$

(B.41)

Without loss of generality (passing to an additional subsequence if needed), we may assume that

$$\begin{aligned} \widehat{H}^{\widetilde{h}}(\cdot )= \Big (\widehat{X}^{\widetilde{h}}(\cdot ), \widehat{W}^{\widetilde{h}}(\cdot ), \widehat{Y}^{\widetilde{h}}(\cdot ), \widehat{A}^{\widetilde{h}}(\cdot ), \widehat{T}^{\widetilde{h}}(\cdot )\Big ) \end{aligned}$$

converges weakly to

$$\begin{aligned} \widehat{H}(\cdot )= \Big (\widehat{X}(\cdot ), \widehat{W}(\cdot ), \widehat{Y}(\cdot ), \widehat{A}(\cdot ), \widehat{T}(\cdot )\Big ), \end{aligned}$$

and $\overline{Y}(\cdot )=\widehat{Y}(\overline{T}(\cdot ))$, $\overline{A}(\cdot )=\widehat{A}(\overline{T}(\cdot ))$, $\overline{C}(\cdot )=\widehat{C}(\overline{T}(\cdot ))$. It follows from our claim in the beginning of the proof that

$$\begin{aligned} \lim \limits _{\widetilde{h}\rightarrow 0} J^{\widetilde{h}}(x, {\widetilde{u}}^{\widetilde{h}})= J(x, \overline{Y}(\cdot ), \overline{C}(\cdot ))\le V^U(x), \end{aligned}$$

(B.42)

where $J(x, \overline{Y}(\cdot ), \overline{C}(\cdot ))\le V^U(x)$ since $V^U(x)$ is the maximizing performance function. Since $\varepsilon$ is arbitrarily small, (B.40) follows from (B.41) and (B.42).

To prove the reverse inequality $\liminf \limits _{h} V^h(x)\ge V^U(x)$, for any small positive constant $\varepsilon$, we choose a particular $\varepsilon$-optimal harvesting strategy for (2.7) such that the approximation can be applied to the chain $\{X^h_n\}$ and the associated reward compared with $V^h(x)$. By an adaptation of the method used by Kushner and Martins (1991) for singular control problems, for given $\varepsilon >0$, there is a $\varepsilon$-optimal harvesting strategy $({Y}(\cdot ), {C}(\cdot ))$ for (2.7) in ${\mathcal {A}}_x^U$ with the following properties: There are $T_\varepsilon <\infty$, $\rho >0$, and $\lambda >0$ such that $( {Y}(\cdot ), {C}(\cdot ))$ are constants on the intervals $[n\lambda , n\lambda + \lambda )$; only one of the components of $Y(\cdot )$ can jump at a time and the jumps take values in the discrete set $\{k\rho : k=1, 2, ...\}$; ${Y}(\cdot )$ is bounded and is constant on $[T_\varepsilon , \infty )$; and $C(\cdot )$ takes only finitely many values.

We adapt this strategy to the chain $\{X^h_n\}$ by a sequence of controls $u^h\equiv (Y^h,C^h)$ using the same method as in (Kushner and Martins 1991, p. 1459). Suppose that we wish to apply a harvesting action of “impulsive” magnitude $\Delta y_i$ (that is, for species i) to the chain at some interpolated time $t_0$. Define $n_h=\min \{k: t^h_k\ge t_0\}$, with $t^h_k$ was defined in (B.1). Then starting at step $n_h$, apply $[\Delta y_i/h]$ successive harvesting steps on species i. Let $Y^h(\cdot )$ denote the piecewise interpolation of the harvesting strategy just defined. With the observation above, let $({Y}^h, {C}^h)$ denote the interpolated form of the adaption. By the weak convergence argument analogous to that of preceding theorems, we obtain the weak convergence

$$\begin{aligned} \big (X^h(\cdot ), W^h(\cdot ), Y^h(\cdot ), A^h(\cdot )\big )\rightarrow \big ({X}(\cdot ), {W}(\cdot ),{Y}(\cdot ), {A}(\cdot )\big ), \end{aligned}$$

where $A(t)=\int _0^t C(s)ds$, and the limit solves (2.7). It follows that

$$\begin{aligned} J(x, {Y}(\cdot ), C(\cdot ))\ge V^U(x)-\varepsilon . \end{aligned}$$

By the optimality of $V^h(x)$ and the above weak convergence,

$$\begin{aligned} V^h(x)\ge J^h(x, u^h)\rightarrow J(x, {Y}(\cdot ), {C}(\cdot )). \end{aligned}$$

It follows that $\liminf \limits _{h\rightarrow 0}V^h(x)\ge V^U(x) -\varepsilon$. Since $\varepsilon$ is arbitrarily small, $\liminf \limits _{h\rightarrow 0}V^h(x)\ge V^U(x)$. Therefore, $V^h(x)\rightarrow V^U(x)$ as $h\rightarrow 0$. If (2.10) holds, by Proposition 2.4 we have $V^U(x)=V(x)$ which finishes the proof. $\square$

1.4 Transition probabilities for bounded harvesting and seeding rates

In this case, recall that $u^h_n= (\pi ^h_n, Q^h_n)$ for each n and $u^h=\{u^h_n\}_n$ be a sequence of controls. It should be noted that $\pi ^h_n = 0$ includes the case that we harvest nothing and also seed nothing; that is, $Q^h_n=0$. Note also that ${\mathcal {F}}^h_n=\sigma \{X^h_m, u^h_m, m\le n\}$.

The sequence $u^h= (\pi ^h, Q^h)$ is said to be admissible if it satisfies the following conditions:

• (a) $u^h_n$ is $\sigma \{X^h_0, X^h_1,\ldots , X^h_{n}, u^h_0, u^h_1,\ldots , u^h_{n-1}\}-\text {adapted},$

• (b) For any $x\in S_{h+}$, we have

  $$\begin{aligned} \mathbb {P}\{ X^h_{n+1} = x | {\mathcal {F}}^h_n\}= \mathbb {P}\{ X^h_{n+1} = x | X^h_n, u^h_n\} = p^h( X^h_n, x| u^h_n), \end{aligned}$$

• (c) Let $X^{h}_{n, j}$ be the j th component of the vector $X^h_n$ for $j=1, 2, \ldots , d$. Then

  $$\begin{aligned} \mathbb {P}\big ( \pi ^h_{n}=\min \{j: X^{h}_{n, j} = U+h\} | X^{h}_{n, j} = U+h \text { for some } j\in \{1, \ldots , d \}, {\mathcal {F}}^h_n\big )=1. \end{aligned}$$

• (d) $X^h_n\in S_{h+}$ for all $n\in {\mathbb {Z}}_{\ge 0}$.

Now we proceed to define transition probabilities $p^h (x, y | u)$ so that the controlled Markov chain $\{X^h_n\}$ is locally consistent with respect to the controlled diffusion $X(\cdot )$. For $(x, u)\in S_{h+}\times {\mathcal {U}}$ with $u=(0, q)$, we define

$$\begin{aligned} \displaystyle Q_h (x, u)&=\sum \limits _{i=1}^d a_{ii}(x) -\sum \limits _{i, j: i\ne j}\dfrac{1}{2}|a_{ij}(x)| +h\sum \limits _{i=1}^d |b_i(x) +q_i| +h,\nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}|u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_{i}(x)+q_i\big )^+ h }{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x-h \mathbf{e_i}| u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_i(x) +q_i)^- h}{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}+h \mathbf{e_j}) | u\right)&= p^h \left( x, x-h\mathbf{e_i}-h\mathbf{e_j}| u\right) = \dfrac{a_{{ij}}^+(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}-h \mathbf{e_j}| u\right)&= p^h \left( x, x-h\mathbf{e_i}+ h \mathbf{e_j}| u \right) = \dfrac{a_{{ij}}^-(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x | u\right)&=\dfrac{h }{ Q_h (x, u)},\qquad \Delta t^h (x, u)=\dfrac{h^2}{Q_h(x, u)}. \end{aligned}$$

(B.43)

Set $p^h \left( x, y|u=(0, q)\right) =0$ for all unlisted values of $y\in S_{h+}$. Assumption B.1 guarantees that the transition probabilities in (B.43) are well-defined. At the reflection steps, we define

$$\begin{aligned} \displaystyle p^h\left( x, x - h \mathbf{e_i}| u = (i, q)\right) =1 \quad \text {and} \quad \Delta t^h(x, u=(i, q))=0,\quad i=1, 2, \ldots , d. \end{aligned}$$

(B.44)

Thus, $p^h \left( x, y|u=(i, q)\right) =0$ for all unlisted values of $y\in S_{h+}$.

1.5 Transition probabilities for unbounded seeding and bounded harvesting rates

In this case, recall that $u^h_n= (\pi ^h_n, R^h_n)$ for each n and $u^h=\{u^h_n\}_n$ be a sequence of controls. It should be noted that $\pi ^h_n = 0$ includes the case that we harvest nothing; that is, $R^h_n=0$. Note also that ${\mathcal {F}}^h_n=\sigma \{X^h_m, u^h_m, m\le n\}$.

The sequence $u^h= (\pi ^h, R^h)$ is said to be admissible if it satisfies the following conditions:

• (a) $u^h$ is $\sigma \{X^h_0, X^h_1,\ldots , X^h_{n}, u^h_0, u^h_1,\ldots , u^h_{n-1}\}-\text {adapted},$

• (b) For any $x\in S_{h+}$, we have

  $$\begin{aligned} \mathbb {P}\{ X^h_{n+1} = x | {\mathcal {F}}^h_n\}= \mathbb {P}\{ X^h_{n+1} = x | X^h_n, u^h_n\} = p^h( X^h_n, x| u^h_n), \end{aligned}$$

• (c) Let $X^{h}_{n, j}$ be the j th component of the vector $X^h_n$ for $j=1, 2, \ldots , d$. Then

  $$\begin{aligned} \mathbb {P}\big ( \pi ^h_{n}=\min \{j: X^{h}_{n, j} = U+h\} | X^{h}_{n, j} = U+h \text { for some } j\in \{1, \ldots , d \}, {\mathcal {F}}^h_n\big )=1. \end{aligned}$$

• (d) $X^h_n\in S_{h+}$ for all $n\in {\mathbb {Z}}_{\ge 0}$.

Now we proceed to define transition probabilities $p^h (x, y | u)$ so that the controlled Markov chain $\{X^h_n\}$ is locally consistent with respect to the controlled diffusion $X(\cdot )$. We use the notations as in the preceding case. For $(x, u)\in S_{h+}\times {\mathcal {U}}$ with $u=(0, r)$, we define

$$\begin{aligned} \displaystyle Q_h (x, u)&=\sum \limits _{i=1}^d a_{ii}(x) -\sum \limits _{i, j: i\ne j}\dfrac{1}{2}|a_{ij}(x)| +h\sum \limits _{i=1}^d |b_i(x) -r_i| +h,\nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}|u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_{i}(x)-r_i\big )^+ h }{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x-h \mathbf{e_i}| u\right)&= \dfrac{a_{ii}(x)/2-\sum \limits _{j: j\ne i}|a_{ij}(x )|/2+\big (b_i(x) -r_i)^- h}{Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}+h \mathbf{e_j}) | u\right)&= p^h \left( x, x-h\mathbf{e_i}-h\mathbf{e_j}| u\right) = \dfrac{a_{{ij}}^+(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x+h\mathbf{e_i}-h \mathbf{e_j}| u\right)&= p^h \left( x, x-h\mathbf{e_i}+ h \mathbf{e_j}| u \right) = \dfrac{a_{{ij}}^-(x)}{2Q_h (x, u)}, \nonumber \\ \displaystyle p^h \left( x, x | u\right)&=\dfrac{h }{ Q_h (x, u)},\qquad \Delta t^h (x, u)=\dfrac{h^2}{Q_h(x, u)}. \end{aligned}$$

(B.45)

Set $p^h \left( x, y|u=(0, r)\right) =0$ for all unlisted values of $y\in S_{h+}$. Assumption B.1 guarantees that the transition probabilities in (B.45) are well-defined. At the reflection steps, we define

$$\begin{aligned} \displaystyle p^h\left( x, x - h \mathbf{e_i}| u = (i, r)\right) =1 \quad \text {and}\quad \Delta t^h(x, u=(i, r))=0,\quad i=1, 2, \ldots , d. \end{aligned}$$

(B.46)

As a result, $p^h \left( x, y|u=(i, r)\right) =0$ for all unlisted values of $y\in S_{h+}$. At the seeding steps, we define

$$\begin{aligned} p^h\left( x, x + h \mathbf{e_i}| u = (-i, r)\right) =1 \quad \text {and}\quad \Delta t^h(x, u=(-i, r))=0,\quad i=1, 2, \ldots , d. \end{aligned}$$

Thus, $p^h \left( x, y|u=(-i, r)\right) =0$ for all unlisted values of $y\in S_{h+}$.