A diffusion model of dynamic participant inflow management

Abstract

This paper studies a diffusion control problem motivated by challenges faced by public health agencies who run clinics to serve the public. A key challenge for these agencies is to motivate individuals to participate in the services provided. They must manage the flow of (voluntary) participants so that the clinic capacity is highly utilized, but not overwhelmed. The organization can deploy costly promotion activities to increase the inflow of participants. Ideally, the system manager would like to have enough participants waiting in a queue to serve as many individuals as possible and efficiently use clinic capacity. However, if too many participants sign up, resulting in a long wait, participants may become irritated and hesitate to participate again in future. We develop a diffusion model of managing participant inflow mechanisms. Each mechanism corresponds to choosing a particular drift rate parameter for the diffusion model. The system manager seeks to balance three different costs optimally: (i) a linear holding cost that captures the congestion concerns, (ii) an idleness penalty corresponding to wasted clinic capacity and negative impact on public health, and (iii) costs of promotion activities. We show that a nested-threshold policy for deployment of participant inflow mechanisms is optimal under the long-run average cost criterion. In this policy, the system manager progressively deploys mechanisms in increasing order of cost, as the number of participants in the queue decreases. We derive explicit formulas for the queue length thresholds that trigger each promotion activity, providing the system manager with guidance on when to use each mechanism.

Proofs

Proof of Proposition 1

Let $\delta (\cdot )$ be a feasible policy for formulation (2)–(7) and set $\theta (t) = \sum _{k=1}^{K} \mu _k \delta _k(t)$ for $t \ge 0$. Clearly, $\theta (t) \in A$ and is feasible for $c(\theta (t)) \le \sum _{k=1}^{K} c_k \mu _k \delta _k(t)$ for $t \ge 0$. Similarly, given a feasible policy $\theta (\cdot )$ for formulation (9)–(13), we set

$$\begin{aligned} \delta _k(t) = \min \Bigg \{ 1, \frac{[\theta (t) - \sum _{l=1}^{k-1} \mu _l]^+}{\mu _k} \Bigg \}, \,\, k = 1, \dots , K. \end{aligned}$$

(46)

Clearly, $\delta _k(t) \in [0, 1]$ for all $t \ge 0$. Thus, $\delta (\cdot )$ is feasible for formulation (2)–(7). Also, it is easy to see from Eq. (8) that $\sum _{k=1}^{K} c_k \mu _k \delta _k(t) = c(\theta (t))$ for $t \ge 0$. These imply that the two formulations have the same long-run average cost. $\blacksquare$

Proof of Lemma 4

First, we prove that $v_{\beta _2}(x) > v_{\beta _1}(x)$ for $x>0$ and $\beta _2> \beta _1 > \underline{\beta }$. Fix $\beta _2> \beta _1 > \underline{\beta }$. Define $\tilde{\phi }(y) = \phi (y) - \theta _K y$ and note that $\tilde{\phi }$ is a decreasing function. Then substituting $\phi (y) = \tilde{\phi }(y) + \theta _K y$ in Eq. (24) and rearranging the terms, we arrive at the following: For $i=1,2$,

$$\begin{aligned} v_{\beta _i}'(x) + \frac{2 \theta _K}{\sigma ^2} v_{\beta _i}(x) = \frac{2 \beta _i}{\sigma ^2} + \frac{2}{\sigma ^2} \tilde{\phi }(p - v_{\beta _i}(x)) + \frac{2 \theta _K p}{\sigma ^2} - \frac{2hx}{\sigma ^2}. \end{aligned}$$

(47)

We argue by contradiction. Suppose $v_{\beta _1}(x) > v_{\beta _2}(x)$ for some $x>0$. Let $x^* = \inf \{ x\ge 0: v_{\beta _1}(x) \ge v_{\beta _2}(x) \}$. If $x^* > 0$, then by continuity of $v_{\beta _i}(\cdot )$, we conclude that

$$\begin{aligned} v_{\beta _1}(x^*) = v_{\beta _2}(x^*) \quad \text{ and } \quad v_{\beta _1}(x) < v_{\beta _2}(x) \quad \text{ on } [0, x^*). \end{aligned}$$

(48)

Multiplying both sides of Eq. (47) with $\exp \{2 \theta _K x/\sigma ^2\}$ and integrating over $[0, x^*]$ give the following:

$$\begin{aligned}&\exp \left\{ \frac{2 \theta _K}{\sigma ^2} x^* \right\} v_{\beta _i}(x^*) \!=\! \int _0^{x^*} \frac{2 \beta _i}{\sigma ^2} \exp \! \left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} dx \!+\! \frac{2}{\sigma ^2 }\! \int _0^{x^*}\!\! \tilde{\phi }( p \!-\! v_{\beta _i}(x) ) \exp \!\left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} dx \nonumber \\&\quad + \frac{2 \theta _K p}{\sigma ^2} \int _0^{x^*} \exp \left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} dx - \frac{2}{\sigma ^2 } \int _0^{x^*} \exp \left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} h x \,dx, \,\,\, i=1,2 \end{aligned}$$

(49)

Considering Eq. (49) for $\beta _1$ and $\beta _2$ and taking the difference give

$$\begin{aligned} 0&= \int _0^{x^*} \frac{2 (\beta _2 - \beta _1)}{\sigma ^2} \exp \left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} dx \nonumber \\ {}&\quad + \frac{2}{\sigma ^2} \int _0^{x^*} [ \tilde{\phi }( p - v_{\beta _2}(x) ) - \tilde{\phi }( p - v_{\beta _1}(x) )] \exp \left\{ \frac{2 \theta _K}{\sigma ^2} x \right\} dx > 0, \end{aligned}$$

(50)

where the inequality follows from Eq. (49) and the monotonicity of $\tilde{\phi }$. The contradiction reached in Eq. (50) shows $v_{\beta _2}(x) > v_{\beta _1}(x)$ when $x > 0$.

Next, we consider the case of $x^*=0$. If $x^* = 0$, then there exists a sequence $\{x_n\}$ such that $x_n \downarrow 0$ as $n \rightarrow \infty$ and $v_{\beta _1}(x_n) \ge v_{\beta _2}(x_n)$. In particular,

$$\begin{aligned} \frac{v_{\beta _1}(x_n)}{x_n} \ge \frac{v_{\beta _2}(x_n)}{x_n} \quad \text{ for } n\ge 1. \end{aligned}$$

(51)

Because $v_{\beta _1}(0) = v_{\beta _2}(0)$, taking the limit in Eq. (51) as $n \rightarrow \infty$ gives $v_{\beta _2}'(0) \le v_{\beta _1}'(0)$. Combining this with Eq. (24) gives

$$\begin{aligned} \beta _2 + \phi (p) - h(0) \le \beta _1 + \phi (p) - h(0), \end{aligned}$$

or, $\beta _1 \le \beta _2$, which is a contradiction, proving $v_{\beta _2}(x) > v_{\beta _2}(x)$ for $x>0$.

Second, we prove that $v_{\beta }(x)$ is continuous in $\beta$ on $(\beta , \infty )$. To this end, fix $\beta _2> \beta _1 > \underline{\beta }$ and note from Eq. (24) that

$$\begin{aligned} \frac{1}{2} \sigma ^2 v_{\beta _i}'(x) = \beta _i - hx - \phi (p - v_{\beta _i}(x)), \,\,\, x \ge 0, i=1,2. \end{aligned}$$

Integrating both sides of this on [0, y], we have that

$$\begin{aligned} v_{\beta _2}(y)&= \frac{2}{\sigma ^2} \beta _2 y - \frac{h y^2}{\sigma ^2} -\frac{2}{\sigma ^2} \int _{0}^{y} \phi (p - v_{\beta _2}(s)) ds, \nonumber \\ v_{\beta _1}(y)&= \frac{2}{\sigma ^2} \beta _1 y - \frac{h y^2}{\sigma ^2} -\frac{2}{\sigma ^2} \int _{0}^{y} \phi (p - v_{\beta _1}(s)) ds. \nonumber \end{aligned}$$

Taking the difference gives

$$\begin{aligned} v_{\beta _2}(y) - v_{\beta _1}(y) = \frac{2}{\sigma ^2} (\beta _2 - \beta _1) y - \frac{2}{\sigma ^2} \int _{0}^{y} [\phi (p - v_{\beta _2}(s)) - \phi (p - v_{\beta _1}(s)) ] ds. \end{aligned}$$

Taking the absolute value of both sides and using the Lipschitz continuity of $\phi$ (see Lemma 1), we conclude that

$$\begin{aligned} |v_{\beta _2}(y) - v_{\beta _1}(y)| \le \frac{2}{\sigma ^2} |\beta _2 - \beta _1| y + \frac{2L}{\sigma ^2} \int _{0}^{y} |v_{\beta _2}(s) - v_{\beta _1}(s)| ds. \end{aligned}$$

Note that letting $F(y) = v_{\beta _2}(y) - v_{\beta _1}(y)$, we have for $y \in [0, x]$ that

$$\begin{aligned} |F(y)| \le \frac{2}{\sigma ^2} |\beta _2 - \beta _1| x + \int _{0}^{y} F(s)ds. \end{aligned}$$

Thus, by Gronwall’s inequality (Lemma 3), we write

$$\begin{aligned} |v_{\beta _2}(y) - v_{\beta _1}(y)| \le \frac{2x}{\sigma ^2} (\beta _2 - \beta _1) +\exp \left\{ \frac{2L}{\sigma ^2} y \right\} , \,\, \forall y \in [0,x]. \nonumber \end{aligned}$$

In particular, $v_{\beta }(x)$ is continuous in $\beta$. $\blacksquare$

Proof of Lemma 5

We proceed in several steps. First, we prove that for all $\beta > \underline{\beta }$, $v_{\beta }(\cdot )$ strictly increases its maximum. Suppose not. Then by continuity of $v_{\beta }$ and its derivative, there exists $x_2> x_1 > 0$ such that

$$\begin{aligned} 0 = v'(x_1)&\le v'(x_2), \nonumber \\ v(x_1)&= v(x_2). \nonumber \end{aligned}$$

Then we write by Eq. (24) that

$$\begin{aligned} \beta&= \frac{1}{2} \sigma ^2 v_{\beta }'(x_1) + hx_1 - \phi (p - v_{\beta }(x_1)), \end{aligned}$$

(52)

$$\begin{aligned} \beta&= \frac{1}{2} \sigma ^2 v_{\beta }'(x_2) + hx_2 - \phi (p - v_{\beta }(x_2)). \end{aligned}$$

(53)

Subtracting (52) from (53) yields:

$$\begin{aligned} 0 = \frac{1}{2} \sigma ^2 v_{\beta }'(x_2) + h(x_2 - x_1) > 0, \end{aligned}$$

which is a contradiction.

Second, we show that $v_{\beta }$ is either strictly increasing or it is first increasing then decreasing. Because $v_{\beta }$ strictly increases to its maximum, if it never reaches its maximum, then we are done. Otherwise, let $x_0 < \infty$ denote the maximizer of $v_{\beta }(x)$. It suffices to show that $v_{\beta }(x)$ is decreasing on $(x_0, \infty )$. Suppose not. Then there exists $x_1$, $x_2$ such that

$$\begin{aligned}&x_0< x_1< x_2, \\&v_{\beta }'(x_1)< 0 < v_{\beta }'(x_2), \\&v_{\beta }(x_1) = v_{\beta }(x_2). \end{aligned}$$

It also follows from Eq. (24) that

$$\begin{aligned} \beta&= \frac{1}{2} \sigma ^2 v_{\beta }'(x_1) + hx_1 - \phi (p - v_{\beta }(x_1)), \end{aligned}$$

(54)

$$\begin{aligned} \beta&= \frac{1}{2} \sigma ^2 v_{\beta }'(x_2) + hx_2 - \phi (p - v_{\beta }(x_2)). \end{aligned}$$

(55)

Subtracting (54) from (55) yields:

$$\begin{aligned} 0 = \frac{1}{2} \sigma ^2 (v_{\beta }'(x_2) -v_{\beta }'(x_1))+ h(x_2 - x_1) > 0, \end{aligned}$$

which is a contradiction.

Third, we show that for $\beta > \underline{\beta }$, if $v_{\beta }$ is increasing, then $\lim _{x \rightarrow \infty } v_{\beta }(x) =\infty$. Suppose not, i.e., $v_{\beta }$ is bounded. Then $v_{\beta }'(x) \rightarrow 0$ as $x \rightarrow \infty$. Recall from Eq. (24) that

$$\begin{aligned} \frac{1}{2} \sigma ^2 v_{\beta }'(x) = \beta - hx + \phi (p - v_{\beta }(x)), \end{aligned}$$

which implies that $\phi (p - v_{\beta }(x)) - hx \rightarrow \beta$ for $x \rightarrow \infty$. This is possible only if $\phi (p - v_{\beta }(x)) = \theta _0 (p - v_{\beta }(x))$ for x large and that $\theta _0 p - \theta _0 v(x) - hx \rightarrow p$ as $x \rightarrow \infty$. That is,

$$\begin{aligned} v(x) \sim \frac{h}{|\theta _0|}x \quad \text{ as } x \rightarrow \infty , \end{aligned}$$

which contradicts that $v_{\beta }$ is bounded.

So far, we proved that for each $\beta > \underline{\beta }$, it belongs to either ${\mathcal {I}}$ or ${\mathcal {N}}$. Thus, $(\underline{\beta }, \infty ) = {\mathcal {I}} \cup {\mathcal {N}}$. Clearly ${\mathcal {I}} \cap {\mathcal {N}} = \emptyset$.

To conclude the proof, it remains to show that if $\beta _1 \in {\mathcal {I}}$, then $\beta _2 \in {\mathcal {I}}$ for all $\beta _2 > \beta _1$. Let $\beta _1 \in {\mathcal {I}}$ and suppose that there exists $\beta _2 > \beta _1$ such that $\beta _2 \notin {\mathcal {I}}$. That is, $v_{\beta _2}$ is first increasing, then decreasing. Then because $v_{\beta _1}(x) \rightarrow \infty$ as $x \rightarrow \infty$, there exists $x_0$ such that $v_{\beta _1}(x_0) > v_{\beta _2}(x_0)$, which contradicts that $v_{\beta }(x)$ is increasing in $\beta$ (see Lemma 4). $\blacksquare$

Proof of Lemma 6

We first prove that ${\mathcal {I}} \not = \emptyset$ and $\left( p | \theta _0 | + \frac{h \sigma ^2 }{2 | \theta _0| }, \infty \right) \subset {\mathcal {I}}$. That is, for $\beta > p | \theta _0 | + \frac{h \sigma ^2 }{2 | \theta _0| }$, $v_{\beta }(x)$ is strictly increasing in x and that $v_{\beta }(x) \rightarrow \infty$ as $x \rightarrow \infty$. Recall from Eq. (24) that

$$\begin{aligned} v'_{\beta }(x)&= \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi ( p - v_{\beta }(x) ) - \frac{2h}{\sigma ^2} x, \,\,\, x \ge 0 \end{aligned}$$

(56)

$$\begin{aligned} v_{\beta }(0)&= 0. \end{aligned}$$

(57)

Also note that $\phi (y) \ge \theta _0 y$ for all y. Thus,

$$\begin{aligned} v'_{\beta }(x)&\ge \frac{2 \beta }{\sigma ^2} + \frac{2 \theta _0}{\sigma ^2} ( p - v_{\beta }(x) ) - \frac{2h}{\sigma ^2} x, \,\,\, x \ge 0. \end{aligned}$$

(58)

Rearranging the terms, we write

$$\begin{aligned} v'_{\beta }(x) + \frac{2 \theta _0}{\sigma ^2} v_{\beta }(x) \ge \frac{2 (\beta + \theta _0 p)}{\sigma ^2} - \frac{2h}{\sigma ^2} x. \end{aligned}$$

(59)

Multiplying both sides by $\exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\}$ gives the following

$$\begin{aligned} \left[ \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} v_{\beta }(x) \right] ' \ge \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} \frac{2 (\beta + \theta _0 p)}{\sigma ^2} - \frac{2h}{\sigma ^2}x \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} . \end{aligned}$$

Integrating both sides and using Eq. (57) give the following:

$$\begin{aligned} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} v_{\beta }(x) \ge \frac{2 (\beta + \theta _0 p)}{\sigma ^2} \frac{\sigma ^2}{2 \theta _0} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} \Bigg |_0^x - \frac{2h}{\sigma ^2} \int _0^x z \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} dz. \end{aligned}$$

That is,

$$\begin{aligned} v_{\beta }(x) \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} \ge \left( \frac{\beta }{\theta _0} + p \right) \left[ \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} - 1\right] - \frac{2h}{\sigma ^2 } \int _0^x z \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} dz, \end{aligned}$$

(60)

where the last term can be integrated by parts as follows:

$$\begin{aligned} \frac{2h}{\sigma ^2} \int _0^x z \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} dz&= \frac{2h}{\sigma ^2} \frac{\sigma ^2}{2 \theta _0} z \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} \Bigg |_0^x - \frac{2h}{\sigma ^2 } \int _0^x \frac{\sigma ^2}{2 \theta _0} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} dz \\ {}&= \frac{h}{\theta _0} z \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} \Bigg |_0^x - \frac{h}{\theta _0} \int _0^x \exp \left\{ \frac{2 \theta _0}{\sigma ^2} z\right\} dz \\ {}&= \frac{h}{\theta _0} x \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x \right\} - \frac{h}{\theta _0} \frac{\sigma ^2}{2 \theta _0} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x \right\} \Bigg |_0^x \\ {}&= \frac{h}{\theta _0} x \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x \right\} - \frac{h \sigma ^2}{2 \theta _0^2} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x \right\} + \frac{h \sigma ^2}{2 \theta _0^2} \end{aligned}$$

Substituting this into Eq. (60) gives the following:

$$\begin{aligned}&v_{\beta }(x) \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} \ge \left( \frac{\beta }{\theta _0} + p \right) \left[ \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} - 1\right] - \frac{h}{\theta _0} x \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x \right\} \\ {}&\qquad \qquad \qquad \qquad \qquad \quad + \frac{h \sigma ^2}{2 \theta _0^2} \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} - \frac{h \sigma ^2}{2 \theta _0^2} \\&= \exp \left\{ \frac{2 \theta _0}{\sigma ^2} x\right\} \left[ \frac{\beta }{\theta _0} + p - \frac{h}{\theta _0}x + \frac{h \sigma ^2}{2 \theta _0^2} \right] - \left( \frac{\beta }{\theta _0} + p + \frac{h \sigma ^2}{2 \theta _0^2} \right) . \end{aligned}$$

Therefore, we conclude that

$$\begin{aligned} v_{\beta }(x) \ge \left( -\frac{\beta }{\theta _0} - p - \frac{h \sigma ^2}{2 \theta _0^2} \right) \exp \left\{ \frac{-2 \theta _0}{\sigma ^2} x\right\} + \left( \frac{\beta }{\theta _0} + p -\frac{h}{\theta _0}x + \frac{h \sigma ^2}{2 \theta _0^2} \right) . \end{aligned}$$

(61)

Thus, for $\beta > p | \theta _0| + \frac{h \sigma ^2}{2 | \theta _0|}$, we have that $v_{\beta }(x) \rightarrow \infty$ as $x \rightarrow \infty$. Moreover, note from Eq. (59) that

$$\begin{aligned} v_{\beta }'(x) \ge \frac{2( \beta + \theta _0 p)}{\sigma ^2} - \frac{2h}{\sigma ^2} x - \frac{2 \theta _0}{\sigma ^2} v_{\beta }(x). \end{aligned}$$

(62)

Because $\theta _0 < 0$ by assumption, substituting (61) into (62) gives

$$\begin{aligned} v_{\beta }'(x) \ge - \frac{2h}{\theta _0} + \frac{2}{\sigma ^2} \left( \beta - p |\theta _0| - \frac{h \sigma ^2}{2 | \theta _0|} \right) \exp \left\{ -\frac{2 \theta _0 x }{\sigma ^2} \right\} > 0, \end{aligned}$$

where the last inequality follows because $\theta _0 < 0$ and $\beta > p |\theta _0| + \frac{h \sigma ^2}{2 |\theta _0|}$. This proves that $v_{\beta }(x)$ is strictly increasing for $\beta > p |\theta _0| + \frac{h \sigma ^2}{2 |\theta _0|}$, proving the claim.

Next, we prove ${\mathcal {N}} \not = \emptyset$ by contradiction. As a preliminary, define $q = \inf \{ y \in [0,p]: \phi (y) = -\underline{\beta }\}$ and note that $\phi (0) = 0$, $\phi$ is decreasing (0, q) and increasing on (q, p). We have the following three cases to consider:

Case 1. $q=p$. In this case, $\phi$ is decreasing on (0, p] and $\theta _k < 0$.

Case 2. $q<p$ and $\phi (p) \ge 0$. In this case, $\phi (p) = \max _{y \in [0,p]} \phi (y)$ and $\theta _k > 0$.

Case 3. $q<p$ and $\phi (p) < 0$. In this case, $\phi (0) =0= \max _{y \in [0,p]} \phi (y)$ and $\theta _k \ge 0$.

Suppose ${\mathcal {N}} = \emptyset$, i.e., $v_{\beta }(x)$ increases strictly to $\infty$ for all $\beta > \underline{\beta }$ by Lemma 5. In particular, its inverse $v_{\beta }^{-1}$ is well defined and strictly increasing. We will illustrate a $\beta \in ( \underline{\beta }, \underline{\beta }+ 1)$ in each case that contradict this. To this end, for $\beta \in ( \underline{\beta }, \underline{\beta }+ 1)$, let

$$\begin{aligned} \bar{x}(\beta )&= \inf \{ x \ge 0 : v_{\beta }(x) = p \} = v_{\beta }^{-1}(p), \\ x_0(\beta )&= \inf \{ x \ge 0 : p - v_{\beta }(x) = q \} = v_{\beta }^{-1}(p-q), \\ \hat{x}(\beta )&= \inf \{ x \ge 0 : v_{\beta }(x) = c_k \} = v_{\beta }^{-1}(c_k), \\ \tilde{x}(\beta )&= \inf \left\{ x \ge 0 : v_{\beta }(x) = \frac{p - c_k}{2} \right\} = v_{\beta }^{-1} \left( \frac{p - c_k}{2} \right) . \end{aligned}$$

Next, we consider each of the above cases.

Case 1. In this case, $x_0(\beta ) = 0$ because $v_{\beta }(0)=0$ and $p - v_{\beta }(0) = p-q$. Moreover, for $x \in [0, \hat{x}(\beta ))$, we have that

$$\begin{aligned} \phi (p - v_{\beta }(x)) = \phi (p) - \theta _k v_{\beta }(x) > \phi (p). \end{aligned}$$

(63)

Recall from Eq. (24) that

$$\begin{aligned} v_{\beta }'(x)&= \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi ( p - v_{\beta }(x)) - \frac{2h}{\sigma ^2}x, \,\,\, x \ge 0 \end{aligned}$$

(64)

$$\begin{aligned} v_{\beta }(0)&= 0. \end{aligned}$$

(65)

For $\varepsilon \in (0,1)$, let $\beta = \underline{\beta }+ \varepsilon$. Then substituting (63) into (64) gives

$$\begin{aligned} v_{\beta }'(x)&= \frac{2 \beta }{\sigma ^2} + \frac{2 \varepsilon }{\sigma ^2} + \frac{2 \phi (p)}{\sigma ^2} - \frac{2 \theta _k}{\sigma ^2} v_{\beta }(x) - \frac{2h}{\sigma ^2} x \\ {}&= \frac{2 \varepsilon }{\sigma ^2} - \frac{2 \theta _k}{\sigma ^2} v_{\beta }(x) - \frac{2h}{\sigma ^2} x, \end{aligned}$$

because $\phi (p) = -\underline{\beta }$. Rearranging the terms further gives

$$\begin{aligned} v_{\beta }' + \frac{2 \theta _k}{\sigma ^2} v_{\beta }(x) = \frac{2 \varepsilon }{\sigma ^2} - \frac{2h}{\sigma ^2}x, \,\,\, x \in [0, \hat{x}(\beta )]. \end{aligned}$$

Multiplying both sides by $\exp \left\{ \frac{2 \theta _k}{\sigma ^2}x \right\}$ gives

$$\begin{aligned} \left[ \exp \left\{ \frac{2 \theta _k }{\sigma ^2} x \right\} v_{\beta }(x) \right] ' = \frac{2 \varepsilon }{\sigma ^2} \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} - \frac{2h}{\sigma ^2} x \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} . \end{aligned}$$

Integrating both sides from 0 to $x \in (0, \hat{x}(\beta ))$ gives:

$$\begin{aligned} \exp \left\{ \frac{2 \theta _k }{\sigma ^2} x \right\} v_{\beta }(x)&= \frac{2 \varepsilon }{\sigma ^2} \frac{\sigma ^2}{2 \theta _k} \exp \left\{ \frac{2 \theta _k }{\sigma ^2} z \right\} \Bigg |_0^x - \int _o^x \frac{2h}{\sigma ^2} z \exp \left\{ \frac{2 \theta _k}{\sigma ^2} z \right\} dz \nonumber \\ {}&= \frac{\varepsilon }{\theta _k} \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} - \frac{\varepsilon }{\theta _k} - \int _0^x \frac{2h}{\sigma ^2} z \exp \left\{ \frac{2 \theta _k}{\sigma ^2} z \right\} dz. \end{aligned}$$

(66)

As done earlier in the proof, the last term on the right-hand side of (66) can be integrated by parts to give the following:

$$\begin{aligned} \frac{2h}{\sigma ^2} \int _0^x z \exp \left\{ \frac{2 \theta _k}{\sigma ^2} z \right\} dz = \frac{h}{\theta _k} x \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} - \frac{h \sigma ^2}{2\theta _k^2} x \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} + \frac{h \sigma ^2}{2\theta _k^2}. \end{aligned}$$

Substituting this into Eq. (66) gives

$$\begin{aligned} \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} v_{\beta }(x) = \left[ \frac{\varepsilon }{\theta _k} - \frac{h}{\theta _k}x + \frac{h \sigma ^2}{2 \theta _k^2}\right] \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} - \left( \frac{\varepsilon }{\theta _k} + \frac{h \sigma ^2}{2 \theta _k^2} \right) . \end{aligned}$$

Thus, we have that

$$\begin{aligned} v_{\beta }(x) = \left[ \frac{\varepsilon }{\theta _k} - \frac{h}{\theta _k}x + \frac{h \sigma ^2}{2 \theta _k^2}\right] - \left( \frac{h \sigma ^2}{2 \theta _k^2} + \frac{\varepsilon }{\theta _k} \right) \exp \left\{ -\frac{2 \theta _k}{\sigma ^2} x \right\} \end{aligned}$$

Differentiating both sides gives

$$\begin{aligned} v_{\beta }'(x) = -\frac{h}{\theta _k} + \frac{h}{\theta _k} \exp \left\{ -\frac{2 \theta _k}{\sigma ^2} x \right\} + \frac{2 \varepsilon }{\sigma ^2} \exp \left\{ -\frac{2 \theta _k}{\sigma ^2} x \right\} . \end{aligned}$$

Rearranging the terms, we can rewrite this as follows:

$$\begin{aligned} v_{\beta }'(x) = \exp \left\{ -\frac{2 \theta _k}{\sigma ^2} x \right\} \left[ \frac{h}{\theta _k} \left( 1 - \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} \right) + \frac{2 \varepsilon }{\sigma ^2}\right] . \end{aligned}$$

(67)

Next, we verify that for $\beta \in (\underline{\beta }, \underline{\beta }+ 1)$,

$$\begin{aligned} \tilde{x}(\beta ) \ge \frac{(p-c_k) \sigma ^2}{4(\underline{\beta }+1)}. \end{aligned}$$

(68)

Recall that $\hat{x}(\beta ) \ge \tilde{x}(\beta )$. Note by definition of $\tilde{x}(\beta )$ that

$$\begin{aligned} v_{\beta }(\tilde{x}(\beta )) = \frac{p-c_k}{2}. \end{aligned}$$

(69)

Also note from Eq. (24) that

$$\begin{aligned} v_{\beta }'(x)&= \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi (p - v_{\beta }(x)) - \frac{2h}{\sigma ^2} x, \,\,\, x \in (0, \hat{x}(\beta )) \nonumber \\ {}&\le \frac{2 \beta }{\sigma ^2} \nonumber \\ {}&\le \frac{2(\underline{\beta }+ 1)}{\sigma ^2}, \end{aligned}$$

(70)

where the first inequality follows because $\phi (p - v_{\beta }(x)) < 0$ for $x \in (0, \hat{x}(\beta ))$, and the second inequality follows because we restrict attention to $\beta \in (\underline{\beta }, \underline{\beta }+ 1)$. Integrating both sides of Eq. (70) on $(0, \hat{x}(\beta ))$ gives

$$\begin{aligned} \frac{p - c_k}{2} = v_{\beta }(\tilde{x}(\beta )) \le \frac{2 (\underline{\beta }+1)}{\sigma ^2} \tilde{x}(\beta ), \end{aligned}$$

which follows from (69)–(70) and establishes (68).

Consider the derivative of the second term of the product on the right-hand side of Eq. (67):

$$\begin{aligned} \frac{d}{dx} \left[ \frac{h}{\theta _k} \left( 1 - \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} \right) + \frac{2 \varepsilon }{\sigma ^2}\right] = -\frac{2h}{\sigma ^2} \exp \left\{ \frac{2 \theta _k}{\sigma ^2} x \right\} < 0. \end{aligned}$$

Thus, we conclude from Eq. (68) that

$$\begin{aligned} \frac{h}{\theta _k} \left( 1 - \exp \left\{ \frac{2 \theta _k}{\sigma ^2} \tilde{x}(\beta ) \right\} \right) + \frac{2 \varepsilon }{\sigma ^2} < \frac{h}{\theta _k} \left( 1 - \exp \left\{ \frac{ \theta _k (p - c_k)}{2(\underline{\beta }+1)} \right\} \right) + \frac{2 \varepsilon }{\sigma ^2}. \end{aligned}$$

In particular, combining this with Eq. (67) gives

$$\begin{aligned} v_{\beta }'(\tilde{x}(\beta )) \le \exp \left\{ -\frac{2 \theta _k}{\sigma ^2} \tilde{x}(\beta ) \right\} \left[ \frac{h}{\theta _k} \left( 1 - \exp \left\{ \frac{ \theta _k (p - c_k)}{2(\underline{\beta }+1)} \right\} \right) + \frac{2 \varepsilon }{\sigma ^2} \right] . \end{aligned}$$

Letting $\varepsilon \in \left( 0, \frac{\sigma ^2}{4} \frac{h}{| \theta _k|} \left( 1 - \exp \left\{ \frac{ \theta _k (p - c_k)}{2(\underline{\beta }+1)} \right\} \right) \wedge 1 \right)$, we conclude that $v_{\beta }'(\tilde{x}(\beta )) < 0$, where $\beta = \underline{\beta }+ \varepsilon$, which contradicts that $v_{\beta }$ is strictly increasing.

To conclude the proof, we consider cases 2 and 3 and treat them simultaneously. Recall that $x_0(\beta ) = v_{\beta }^{-1}(p-q)$, i.e., $v_{\beta }(x_0(\beta )) = p - q$. Also note from Eq. (24) that

$$\begin{aligned} v_{\beta }'(x) = \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi ( p - v_{\beta }(x)) - \frac{2h}{\sigma ^2}x, \,\,\, x \in (0, \bar{x}(\beta )). \end{aligned}$$

In particular,

$$\begin{aligned} v_{\beta }'(x) \le \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi (p)^+, \end{aligned}$$

(71)

where $\phi (p)^+ = \max \{ \phi (p), 0\}$. This follows because $\phi (p)^+ \ge \phi (y)$ for $y \in [0, p]$ in cases 2 and 3. Integrating both sides of (71) on $(0, x_0(\beta ))$ gives

$$\begin{aligned} p-q = v_{\beta } (x_0(\beta )) \le \frac{2}{\sigma ^2} (\beta + \phi (p)^+) x_o(\beta ), \end{aligned}$$

from which it follows that

$$\begin{aligned} x_0(p) \ge \frac{\sigma ^2}{2} \frac{p-q}{\beta + \phi (p)^+} \ge \frac{\sigma ^2}{2} \frac{p-q}{\underline{\beta }+1+ \phi (p)^+}, \end{aligned}$$

(72)

where the last inequality follows because we restrict attention to $\beta \in (\underline{\beta }, \underline{\beta }+1)$.

For $\varepsilon \in (0, 1)$, letting $\beta = \underline{\beta }+ \varepsilon$, we note from Eq. (24) that

$$\begin{aligned} v_{\beta }'( x_0(\beta ))&= \frac{2(\underline{\beta }+ \varepsilon )}{\sigma ^2} + \frac{2}{\sigma ^2} \phi ( p - v_{\beta }( x_0(\beta ))) - \frac{2h}{\sigma ^2} x_0(\beta ) \nonumber \\ {}&= \frac{2 \varepsilon }{\sigma ^2} + \frac{2 \beta }{\sigma ^2} + \frac{2}{\sigma ^2} \phi (q) - \frac{2 h}{\sigma ^2} x_0(\beta ) \nonumber \\&= \frac{2 \varepsilon }{\sigma ^2} - \frac{2 h}{\sigma ^2} x_0(\beta ), \end{aligned}$$

(73)

where the last equality follows because $\phi (q){=}-\!\underline{\beta }$. Letting $\varepsilon {\in }( 0, \frac{h \sigma ^2}{4} \frac{p-q}{\underline{\beta }{+}1{+}\phi (p)^{+}}\wedge 1 )$, we conclude from (73) that for $\beta = \underline{\beta }+ \varepsilon$,

$$\begin{aligned} v_{\beta }'(x_0(\beta )) = \frac{2}{\sigma ^2} (\varepsilon - h x_0(\beta )) \le \frac{2}{\sigma ^2} \left( \varepsilon - \frac{\sigma ^2 h}{2} \frac{p-q}{\underline{\beta }+ 1 + \phi (p)^+} \right) < 0, \end{aligned}$$

where the first inequality follows from (72). This contradicts that $v_{\beta }$ is strictly increasing, concluding the proof. $\blacksquare$

Proof of Lemma 9

To begin, we fix and admissible policy $\theta (\cdot )$ and construct a process that is stochastically larger than the state process Z under $\theta (\cdot )$. Recall that under an admissible policy $\theta (\cdot )$, there exists $\bar{z} > 0$ such that $\theta (z) \le \theta _{j^*} <0$ for $z \ge \bar{z}$. Fix an admissible policy $\theta (\cdot )$ and the associated $\bar{z}$. Also, let $\tilde{B}$ be a $(\theta _j^*, \sigma ^2)$ Brownian motion. It is straightforward to argue that $\bar{z} + \tilde{Z}(t)$ is stochastically larger than the (scaled) queue length process $Z(\cdot )$ under $\theta (\cdot )$.

To establish part i), it suffices to show the following (see Harrison [25]):

$$\begin{aligned} {\mathbb {E}} \left[ \int _0^t (f'(Z(s)))^2 ds \right] < \infty \quad \forall t > 0. \end{aligned}$$

(74)

To verify (74), recall that $f'(z) = v(z) - p$ and consider the following:

$$\begin{aligned} {\mathbb {E}} \left[ \int _0^t (v(Z(s)) - p)^2 ds \right] =&\,{\mathbb {E}} \left[ \int _0^t ( v^2(Z(s)) - 2p v(Z(s)) + p^2) ds \right] \nonumber \\ \le&{\mathbb {E}} \int _0^t v^2(Z(s)) ds +2 p {\mathbb {E}} \int _0^t v(Z(s)) ds + p^2. \end{aligned}$$

(75)

Recall that $v(\cdot )$ has linear growth. Thus, there exists $K_1, K_2 > 0$ such that

$$\begin{aligned} v(z) \le K_1 + K_2 z, \,\,\, z > 0. \end{aligned}$$

(76)

Substituting this into Eq. (75) yields

$$\begin{aligned} {\mathbb {E}} \left[ \int _0^t ( v(Z(s)) - p)^2 ds \right] \le&\,\, {\mathbb {E}} \int _0^t (K_1^2 + 2 K_1 K_2 Z(s) + K_2^2 Z^2(s)) ds \\&+ 2p {\mathbb {E}} \int _0^t (K_1 + K_2 Z(s)) ds + p^2. \end{aligned}$$

Therefore, it suffices to show that ${\mathbb {E}}[Z(t)] < \infty$ and ${\mathbb {E}}[Z^2(t)] < \infty$ for $t \ge 0$, but because

$$\begin{aligned} {\mathbb {E}}[Z(s)]&\le \bar{z} + {\mathbb {E}}[\tilde{Z}(t)], \,\, t\ge 0, \\ {\mathbb {E}}[Z^2(s)]&\le {\mathbb {E}}[ (\bar{z} + \tilde{Z}(t))^2 ], \,\, t\ge 0, \end{aligned}$$

and that ${\mathbb {E}}[ \tilde{Z}(t)] < \infty$ and ${\mathbb {E}}[ \tilde{Z}^2(t)] < \infty$ for $t \ge 0$, see Harrison [25], the result follows.

To establish part ii), note that

$$\begin{aligned} f(z) = \int _0^z (v(z) - p) ds = \int _0^z v(s) ds - pz. \end{aligned}$$

In particular, we have

$$\begin{aligned} |f(z)| \le \int _0^z |v(s)| ds + pz. \end{aligned}$$

Because $v(\cdot )$ has linear growth (see Eq. (76)), we conclude that

$$\begin{aligned} |f(z)| \le K_1 z + \frac{K_2}{2} z^2 + pz. \end{aligned}$$

Thus, we have that

$$\begin{aligned} {\mathbb {E}} |f(Z(t))|&\le (K_1 + p) {\mathbb {E}} Z(t) + \frac{K_2}{2} {\mathbb {E}} Z(t)^2 \nonumber \\&\le (K_1 + p) {\mathbb {E}} (\bar{z} + \tilde{Z}(t)) + \frac{K_2}{2} {\mathbb {E}} (\bar{z} + \tilde{Z}(t))^2, \end{aligned}$$

(77)

where the last inequality follows $(\bar{z} + \tilde{Z}(t))$ is stochastically larger than Z(t).

Also note that $\tilde{Z}(t) \Rightarrow \tilde{Z}^*$ as $t \rightarrow \infty$, where $\tilde{Z}^*$ is an exponential random variable with mean $\sigma ^2/2| \theta _{j}|$ [25]. Therefore, we deduce from Eq. (77) that

$$\begin{aligned} \overline{\lim }_{t \rightarrow \infty } {\mathbb {E}} |f(Z(t))| \le (p + K_1) ( \bar{z} + {\mathbb {E}} Z^*) + \frac{K_2}{2} {\mathbb {E}}( \bar{z} + Z^*)^2. \end{aligned}$$

Thus,

$$\begin{aligned} \lim _{t \rightarrow \infty } \frac{{\mathbb {E}} |f(Z(t))| }{t} = 0. \end{aligned}$$

$\blacksquare$