Asymptotically optimal index policies for an abandonment queue with convex holding cost

Abstract

We investigate a resource allocation problem in a multi-class server with convex holding costs and user impatience under the average cost criterion. In general, the optimal policy has a complex dependency on all the input parameters and state information. Our main contribution is to derive index policies that can serve as heuristics and are shown to give good performance. Our index policy attributes to each class an index, which depends on the number of customers currently present in that class. The index values are obtained by solving a relaxed version of the optimal stochastic control problem and combining results from restless multi-armed bandits and queueing theory. They can be expressed as a function of the steady-state distribution probabilities of a one-dimensional birth-and-death process. For linear holding cost, the index can be calculated in closed-form and turns out to be independent of the arrival rates and the number of customers present. In the case of no abandonments and linear holding cost, our index coincides with the $c\mu$-rule, which is known to be optimal in this simple setting. For general convex holding cost, we derive properties of the index value in limiting regimes: we consider the behavior of the index (i) as the number of customers in a class grows large, which allows us to derive the asymptotic structure of the index policies, (ii) as the abandonment rate vanishes, which allows us to retrieve an index policy proposed for the multi-class M/M/1 queue with convex holding cost and no abandonments, and (iii) as the arrival rate goes to either 0 or $\infty$, representing light-traffic and heavy-traffic regimes, respectively. We show that Whittle’s index policy is asymptotically optimal in both light-traffic and heavy-traffic regimes. To obtain further insights into the index policy, we consider the fluid version of the relaxed problem and derive a closed-form expression for the fluid index. The latter is shown to coincide with the index values for the stochastic model in asymptotic regimes. For arbitrary convex holding cost the fluid index can be seen as the $Gc\mu /\theta$-rule; that is, including abandonments into the generalized $c\mu$-rule ($Gc\mu$-rule). Numerical experiments for a wide range of parameters have shown that the Whittle index policy and the fluid index policy perform very well for a broad range of parameters.

Appendices

Appendix 1: Proof of Proposition 1

In Proposition 1 we aim to prove that the threshold policy $\varphi =n$ is an optimal solution of the relaxed problem (8). In order to do so, we are left to prove the convexity of the value function $V$. We will therefore prove that the value function that corresponds to the truncated system $V^L(m)$ (truncated by $L>1$) is convex. Having done this, due to the result in [14], Th. 3.1] we have that $V^L(m)\rightarrow V(m)$ as $L\rightarrow \infty$ and hence, the convexity of $V^L$ for all $L$ will imply convexity of the function $V$. In order to apply [14], Th. 3.1] we need to make sure that the conditions required are satisfied. We therefore check the conditions required by [14], Th. 3.1] in Appendix 1, and prove the convexity of $V^L$ in Appendix 1.

1.1 Conditions to be checked for [14, Th. 3.1]

Let us first present the following definition:

Definition 6

A function $f:E\longrightarrow \mathbb {R}_+$ is a moment function if there exists an increasing sequence of finite sets $E_r\uparrow E$, $r\rightarrow \infty$, such that $\inf \{f(m):m\notin E_r\}\rightarrow \infty$ as $r\rightarrow \infty$. (Where $E$ is the state space).

Let us define $q^{\varphi ,L}(m,m-1)=\mu S^\varphi (m)+\theta 'S^\varphi (m)-\theta (m-S^\varphi (m))$, and recall that $q^{\varphi ,L}(m,m+1)=\lambda \left( 1-\frac{m}{L}\right)$. The conditions to be checked in [14], Th. 3.1] are the following:

• 1. There exists a moment function $f:\mathbb {N}\cup \{0\}\longrightarrow \mathbb {R}_+$, constants $\alpha , \beta >0$ and $M>0$ such that

  $$\begin{aligned} \sum _{\tilde{m}=0}^\infty q^{\varphi ,L}(m,\tilde{m})f(\tilde{m})\le -\alpha f(m)+\beta 1\!\!1_{\{m< M\}}(m), \text { for all }\varphi , L, \end{aligned}$$

  where $\varphi$ defines the policy followed, $L$ is the truncating parameter and $q^{\varphi ,L}(m,\tilde{m})$ the transition rate from $m$ to $\tilde{m}$ under $\varphi$ and $L$.

• 2. $(S^\varphi (m),L)\mapsto q^{\varphi ,L}(m,\tilde{m})$ and $(S^\varphi (m),L)\mapsto \sum _{\tilde{m}}q^{\varphi ,L}(m,\tilde{m})f(\tilde{m})$ are continuous functions in $S^\varphi (m)$ and $L$ for all $m$ and $\tilde{m}$.

We define $f(m):=\mathrm {e}^{\epsilon m}$, where $\epsilon >0$. We can construct $E_r=\{0,\ldots ,r\}$ such that $E_r$ is finite, $E_r\uparrow \mathbb {N}\cup \{0\}$ as $r\rightarrow \infty$ and $\inf \{f(m):m\notin E_r\}\rightarrow \infty$. The objective is then to see, that there exists $\epsilon >0$, an $M>0$ and a constant $\alpha >0$, such that

$$\begin{aligned} \sum _{\tilde{m}=0}^\infty q^{\varphi ,L}(m,\tilde{m})f(\tilde{m})\le -\alpha f(m), \text { for all } m\ge M; \end{aligned}$$

that is,

$$\begin{aligned} \begin{aligned}&\lambda \left( 1-\frac{m}{L}\right) \mathrm {e}^{\epsilon (m+1)}+\left( (\mu +\theta ')S^{\varphi }(m)+\theta (m-S^\varphi (m))\right) \mathrm {e}^{\epsilon (m-1)}\\&\quad -\left( \left( 1-\frac{m}{L}\right) +(\mu +\theta ')S^{\varphi }(m)+\theta (m-S^{\varphi }(m))\right) \mathrm {e}^{\epsilon m}\le -\alpha \mathrm {e}^{\epsilon m}, \text { for all } m\ge M. \end{aligned} \end{aligned}$$

After some algebra we get

$$\begin{aligned} \begin{aligned}&\lambda \left( 1\!-\!\frac{m}{L}\right) (\mathrm {e}^{\epsilon }-1)\!+\!\left( (\mu +\theta '-\theta )S^\varphi (m)+\theta m\right) (\mathrm {e}^{-\epsilon }-1)\le \!-\!\alpha , \text { for all } m\ge M. \end{aligned} \end{aligned}$$

Note that $\lambda (1-m/L)(\mathrm {e}^{-\epsilon }-1)$ can be upper bounded by a constant, $\kappa _1$, and $(\mu +\theta '-\theta )S^\varphi (m)(\mathrm {e}^{\epsilon }-1)$ can be upper bounded by $\kappa _2$. Besides, $\theta m(\mathrm {e}^{-\epsilon }-1)<0$. Hence, we can find $M$ large enough so that $-\theta m(\mathrm {e}^{-\epsilon }-1)\ge \kappa _1+\kappa _2$ for all $m\ge M$. This proves that condition $(1)$ is satisfied.

Condition (2), i.e., the continuity of the functions $(S^\varphi (m),L)\mapsto q^{\varphi ,L}(m,\tilde{m})$ and $(S^\varphi (m),L)\mapsto \sum _{\tilde{m}}q^{\varphi ,L}(m,\tilde{m})f(\tilde{m})$ in $L$ and $S^\varphi (m)$, is satisfied by the definition of the transition rates.

1.2 Convexity of $V^L$

For clarity we define $\omega :=\mu +\theta '-\theta$ throughout this proof. W.l.o.g. assume $\lambda +\mu + \theta '+\theta L=1$. For $n\in \{0,1,\ldots , L\}$ we define $V^L_t(n)$ by $V^L_0(n)=0$ and

$$\begin{aligned} \begin{aligned} V^L_{t+1}(n) =\,&\lambda \left( 1-\frac{n}{L}\right) V^L_t(\min \{n+1,L\}) \\&+\min \Big \{-W +\tilde{C}(n,0) + \omega V^L_t(n), \tilde{C}(n,1) +\omega V^L_t((n-1)^+)\Big \}\\&+\theta n V^L_t((n-1)^+) +\lambda \frac{n}{L} V^L_t(n)+(L-n+1)\theta V^L_t(n). \end{aligned} \end{aligned}$$

We will prove that $V_t^L$ is a convex function for $n\le L-1$; that is,

$$\begin{aligned} 2 V_t^L(n) \le V_t^L((n-1)^+) + V_t^L(n+1), \text{ for } n \le L-1. \end{aligned}$$

(29)

The function $V^L_t$ being convex, for any $t$, implies convexity of $V^L$ and concludes the proof.

In order to prove convexity of $V_t^{L}$ we first prove that $V^L_t(\cdot )$ is a non-decreasing function. The proof follows by induction: $V^L_0(n)=0$ is non-decreasing for $t=0$, then we assume $V_t^L(n)$ is non-decreasing and we prove that

$$\begin{aligned} V_{t+1}^L(n+1)-V_{t+1}^L(n)\ge 0\hbox { for all } n\le L-1. \end{aligned}$$

(30)

Let us first consider the terms multiplied by $\lambda$ in $V_{t+1}^L(n+1)-V_{t+1}^L(n)$; that is,

$$\begin{aligned}&\lambda \left( 1-\frac{n+1}{L}\right) V_t^{L}(\min \{n+2,L\})+\lambda \frac{n+1}{L}V_t^L(\min \{n+1,L\})\\&\quad -\lambda \left( 1-\frac{n}{L}\right) V_t^L(\min \{n+1,L\})-\lambda \frac{n}{L}V_t^L(n)\\&\,\ge \lambda \left( 1-\frac{n+1}{L}\right) \left( V_t^L(\min \{n+2,L\})-V_t^L(\min \{n+1,L\})\right) \\&\quad \,\,+\lambda \frac{n}{L}\left( V_t^L\left( \min \{n+1,L\}-V_t^L(n)\right) \right) \ge 0, \end{aligned}$$

where the last inequality holds due to the non-decreasingness of $V^L_t(n)$. Let us now consider the terms multiplied by $\theta$ in $V_{t+1}^L(n+1)-V_{t+1}^L(n)$, namely,

$$\begin{aligned}&\theta (n+1)V_t^L(n)+(L-n-1)\theta V_t^L(\min \{n+1,L\})\\&\quad -\theta n V_t^L((n-1)^+)-(L-n)\theta V_t^L(n)\\&\quad \!\ge \, \theta n \left( V_t^L(n)\!-\!V_t^L((n-1)^+)\right) \!+\!(L-n-1)\left( V_t^L(\min \{n+1,L\})\!-\!V_t^L(n)\right) \!\ge \! 0, \end{aligned}$$

where, again, the last inequality holds due to $V_t^L(n)$ being non-decreasing for all $n\le L-1$. Finally, let us consider the $\min$-terms in $V_{t+1}^L(n+1)-V_{t+1}^L(n)$. It is straightforward that

$$\begin{aligned}&\min \big \{-W +\tilde{C}(\min \{n+1,L\},0) + (\mu +\theta ')V^L_t(\min \{n+1,L\}),\\&\qquad \qquad \tilde{C}(\min \{n+1,L\},1) +(\mu +\theta ')V^L_t(n)\big \}\\&-\min \big \{-W +\tilde{C}(n,0) + (\mu +\theta ')V^L_t(n),\\&\qquad \qquad \tilde{C}(n,1) +(\mu +\theta ')V^L_t((n-1)^+)\big \}\ge 0,\end{aligned}$$

due to $\tilde{C}$ and $V_t^L$ being non-decreasing. This proves (30) and hence we showed that $V^L_t(n)$ is non-decreasing.

Equation (29) for $n=0$ follows directly from $V_t^L(\cdot )$ being non-decreasing. In the remainder of the proof we therefore prove Eq. (29) for $n\ge 1$.

We will prove convexity (29) by induction on $t$. Since $V_0^L(n)=0$, it holds for $t=0$. Now assume $V^L_t$ is convex. For $1\le n \le L-1$ we have

$$\begin{aligned} 2V^L_{t+1}(n)&=\, 2\lambda \left( 1-\frac{n}{L }\right) V_t^L(n+1) \nonumber \\&\quad + 2\lambda \frac{n }{L} V_t^L(n ) + 2\theta n V_t^L(n-1)+2 (L-n +1)\theta V_t^L(n)\nonumber \\&\quad + 2\min \Big \{-W +\tilde{C}(n,0) +\omega V_t^L(n), \tilde{C}(n,1) +\omega V_t^L(n-1)\Big \}. \end{aligned}$$

(31)

We need to show that this is less than or equal to $V^L_{t+1}(n-1)+V^L_{t+1}(n+1)$, which is given by

$$\begin{aligned}&\lambda \left( 1-\frac{n-1}{L}\right) V_t^L(n)+ \lambda \left( 1-\frac{n+1}{L}\right) V_t^L(n+2) + \lambda \frac{n-1}{L} V_t^L(n-1) \nonumber \\&\quad +\lambda \frac{n+1}{L}V_t^L(n+1)+ \theta (n-1) V_t^L((n-2)^+)+ \theta (n+1) V_t^L(n) \nonumber \\&\quad +(L-n+2)\theta V_t^L(n-1)+ (L-n)\theta V_t^L(n+1)\nonumber \\&\quad + \min \left\{ -W + \tilde{C}(n-1,0)+ \omega V_t^L(n-1), \tilde{C}(n-1,1)+ \omega V_t^L((n-2)^+)\right\} \nonumber \\&\quad + \min \left\{ -W+\tilde{C}(n+1,0) + \omega V_t^L(n+1), \tilde{C}(n+1,1)+\omega V_t^L(n)\right\} . \end{aligned}$$

(32)

We first consider the two terms multiplied by $\lambda$ in (31) and show that they are smaller than or equal to

$$\begin{aligned} \lambda&\left( 1-\frac{n-1}{L}\right) V_t^L(n) + \lambda \left( 1-\frac{n+1}{L}\right) V_t^L(n+2) \nonumber \\&+ \lambda \frac{n-1}{L} V_t^L(n-1) + \lambda \frac{n+1}{L} V_t^L(n+1). \end{aligned}$$

(33)

When $1\le n<L-1$, then for the terms multiplied by $\lambda$ in (31) we can write

$$\begin{aligned} 2\left( 1-\frac{n}{L }\right) V_t^L(n+1) + 2 \frac{n }{L} V_t^L(n )&=2\left( 1-\frac{n+1}{L }\right) V_t^L(n+1) + 2 \frac{n }{L} V_t^L(n )\nonumber \\&\quad +\frac{2}{L}V_t^L(n+1) \nonumber \\&\le \left( 1-\frac{n-1}{L }\right) V_t^L(n)\nonumber \\&\quad -\frac{2}{L}V_t^L(n)+\left( 1-\frac{n+1}{L }\right) V_t^L(n+2)\nonumber \\&\quad + 2 \frac{n }{L} V_t^L(n )+\frac{2}{L}V_t^L(n+1), \end{aligned}$$

(34)

by convexity of $V_t^L$. Since, by convexity, $2\frac{n-1}{L} V_t^L(n) \le \frac{n-1}{L} (V_t^L(n-1)+V_t^L(n+1))$, we obtain that (34) is smaller than or equal to (33). When $n=L-1$, it reduces to verifying $2(1-2/L) V_t^L(L-1) \le (1-2/L)(V_t^L(L-2) + V_t^L(L))$, which follows from convexity of $V_t^L$.

For the terms multiplied by $\theta$, we need to show for $1\le n\le L-1$ that

$$\begin{aligned}&2nV_t^L(n-1)+2V_t^L(n)+2(L-n)V_t^L(n)\\&\le (n-1)V_t^L((n-2)^+)+(n+1)V_t^L(n)+2V_t^L(n-1)\\&\quad +(L-n)(V_t^L(n-1)+V_t^L(n+1)). \end{aligned}$$

We apply the inequality $2V_t^L(n-1)\le V_t^L((n-2)^+)+V_t^L(n)$ on the right-hand-side and the whole initial inequality reduces to

$$\begin{aligned}&2nV_t^L(n-1)+2(L-n)V_t^L(n)\le n(V_t^L((n-2)^+)+V_t^L(n))\\&\quad + (L-n)(V_t^L(n-1)+V_t^L(n+1)), \end{aligned}$$

which holds by convexity of $V_t^L$.

We now consider the $\min$-terms. We will condition on the possible optimal actions in states $n-1$ and $n+1$. Since at time $t$ we have that $V_t^L$ is convex, the optimal actions satisfy the monotonicity property. Denote by $a^*_n\in \{0,1\}$ the optimal action in state $n$, with action 0 (1) being passive (active). Then, by monotonicity there are the following three possibilities: $(a^*_{n-1},a^*_{n+1})$ equals $(0,0), (0,1)$ or $(1,1)$. First assume $a^* = (0,1)$. Then, we obtain for $1\le n\le L-1$ that

$$\begin{aligned}&2\min \left\{ -W +\tilde{C}(n,0) +\omega V_t^L(n),\tilde{C}(n,1)+ \omega V_t^L(n-1)\right\} \nonumber \\&\le -W +\tilde{C}(n,0)+\omega V_t^L(n)+\tilde{C}(n,1)+ \omega V_t^L(n-1) \nonumber \\&\le -W +\tilde{C}(n-1,0) +\omega V_t^L(n)+\tilde{C}(n+1,1)+ \omega V_t^L(n-1)\nonumber \\&= \min \big \{-W +\tilde{C}(n-1,0) +\omega V_t^L(n-1), \tilde{C}(n-1,1)+ \omega V_t^L((n-2)^+)\big \}, \nonumber \\&\quad + \min \big \{-W +\tilde{C}(n+1,0)+ \omega V_t^L(n+1), \tilde{C}(n+1,1)+ \omega V_t^L(n)\big \}, \end{aligned}$$

(35)

where in the second inequality we used that $C$, and hence $\tilde{C}$, satisfies (2). In the case $a^* =(1,1)$ we obtain for $1\le n\le L-1$ that

$$\begin{aligned}&2\min \left\{ -W +\tilde{C}(n,0) +\omega V_t^L(n),\tilde{C}(n,1)+ \omega V_t^L(n-1)\right\} \nonumber \\&\le 2\tilde{C}(n,1)+2\omega V_t^L(n-1)\nonumber \\&\le \tilde{C}(n-1,1)+\tilde{C}(n+1,1)+ \omega (V_t^L((n-2)^+)+V_t^L(n))\nonumber \\&= \min \left\{ -W +\tilde{C}(n-1,0) +\omega V_t^L(n-1), \tilde{C}(n-1,1)+ \omega V_t^L((n-2)^+)\right\} , \nonumber \\&\quad + \min \left\{ -W +\tilde{C}(n+1,0)+ \omega V_t^L(n+1), \tilde{C}(n+1,1)+ \omega V_t^L(n)\right\} . \end{aligned}$$

(36)

In the second inequality we used the convexity of $C$ (and hence of $\tilde{C}$) and the convexity of $V_t^L$.

When $a^* =(0,0)$ we obtain for $1\le n\le L-1$ that

$$\begin{aligned}&2\min \left\{ -W +\tilde{C}(n,0) +\omega V_t^L(n),\tilde{C}(n,1)+ \omega V_t^L(n-1)\right\} \nonumber \\&\le -2W +2\tilde{C}(n,0) +2\omega V_t^L(n)\nonumber \\&\le -2W+\tilde{C}(n-1,0)+\tilde{C}(n+1,0) +\omega V_t^L(n-1)+\omega V_t^L(n+1)\nonumber \\&= \min \left\{ -W +\tilde{C}(n-1,0) +\omega V_t^L(n-1), \tilde{C}(n-1,1)+ \omega V_t^L((n-2)^+)\right\} , \nonumber \\&\quad + \min \left\{ -W +\tilde{C}(n+1,0)+ \omega V_t^L(n+1), \tilde{C}(n+1,1)+ \omega V_t^L(n)\right\} . \end{aligned}$$

(37)

In the second inequality we used the convexity of $C$ (and hence of $\tilde{C}$) and the convexity of $V_t^L$.

Hence, we have that (31) is less than or equal to $V^L_{t+1}(n-1)+V^L_{t+1}(n+1)$, hence $V_{t+1}^L$ is convex. This concludes the proof for convexity of $V^L_t(\cdot )$. Since $V_t^L\rightarrow V^L$ as $t\rightarrow \infty$ [34], Chap. 9.4], convexity of $V_t^L(\cdot )$ implies convexity of $V^L(\cdot )$.

Appendix 2: Proof of Theorem 1

In this section, we prove that the steps described in Theorem 1 indeed defines Whittle’s index correctly. To do so let us assume that the steps stop at iteration $J\in \mathbb {N}\cup \{\infty \}$, and hence $n_{J}=\infty$. We further set $W_i:=W_J$ and $n_i=\infty$ for all $i\in \{J+1,\ldots \}\cup \{\infty \}$. We will prove that $W_0<W_1<W_2<\cdots$, and note that by construction $n_i$ for $i\in \mathbb {N}\cup \{0,\infty \}$ is an increasing sequence. Let us prove $W_{i}<W_{i+1}$ for all $i\in \{0,1,2,\dots \}\cup \{\infty \}$. We have from the characterization of $W_i$ that

$$\begin{aligned}&\frac{{\mathbb E}(\tilde{C} (N^{n_{i+1}},S^{n_{i+1}}(N^{n_{i+1}})) )-{\mathbb E}(\tilde{C} (N^{n_{i-1}},S^{n_{i-1}}(N^{n_{i-1}})) )}{\sum _{m=0}^{n_{i+1}}\pi ^{n_{i+1}}(m)-\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)}\\&\quad > \frac{{\mathbb E}(\tilde{C} (N^{n_i},S^{n_i}(N^{n_i})) )-{\mathbb E}\left( \tilde{C} \left( N^{n_{i-1}},S^{n_{i-1}}\left( N^{n_{i-1}}\right) \right) \right) }{\sum _{m=0}^{n_i}\pi ^{n_i}(m)-\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)}\\&\Longrightarrow \left( {\mathbb E}(\tilde{C} (N^{n_{i+1}},S^{n_{i+1}}(N^{n_{i+1}})) )-{\mathbb E}(\tilde{C} (N^{n_{i-1}},S^{n_{i-1}}(N^{n_{i-1}})) )\right) \\&\quad \qquad \left( \sum _{m=0}^{n_i}\pi ^{n_i}(m)-\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)\right) \\&\quad > \left( {\mathbb E}(\tilde{C} (N^{n_i},S^{n_i}(N^{n_i})) )-{\mathbb E}(\tilde{C} (N^{n_{i-1}},S^{n_{i-1}}(N^{n_{i-1}})) )\right) \\&\qquad \quad \left( \sum _{m=0}^{n_{i+1}}\pi ^{n_{i+1}}(m)-\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)\right) , \end{aligned}$$

and adding ${\mathbb E}(\tilde{C} (N^{n_i},S^{n_i}(N^{n_i}))(\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)-\sum _{m=0}^{n_i}\pi ^{n_i}(m)))$ on both sides of the inequality, after some algebra we obtain

$$\begin{aligned}&W_{i+1}=\frac{{\mathbb E}(\tilde{C} (N^{n_{i+1}},S^{n_{i+1}}(N^{n_{i+1}})) )-{\mathbb E}(\tilde{C} (N^{n_i},S^{n_i}(N^{n_i})) )}{\sum _{m=0}^{n_{i+1}}\pi ^{n_{i+1}}(m)-\sum _{m=0}^{n_i}\pi ^{n_i}(m)}\\&\quad > \frac{{\mathbb E}(\tilde{C} (N^{n_i},S^{n_i}(N^{n_i})) )-{\mathbb E}(\tilde{C} (N^{n_{i-1}},S^{n_{i-1}}(N^{n_{i-1}})) )}{\sum _{m=0}^{n_i}\pi ^{n_i}(m)-\sum _{m=0}^{n_{i-1}}\pi ^{n_{i-1}}(m)}=W_i. \end{aligned}$$

To prove that the steps given in Theorem 1 indeed define the Whittle index we have to show that,

• 1. The threshold policy $-1$ is optimal for problem (8) for all $W$ such that $W<W_{0}$.

• 2. The threshold policy $n_i<\infty$ is optimal for problem (8) for all $W$ such that $W_{i}< W<W_{i+1}$.

• 3. And finally that the policy $\infty$ is optimal for problem (8) for all $W$ such that $\infty >W> W_J$ and $J<\infty$.

To show 1., note that for all $W<W_0$

$$\begin{aligned}&W\sum _{m=0}^n\pi ^n(m)<{\mathbb E}(\tilde{C} (N^{n},S^{n}(N^{n})) )-{\mathbb E}\left( \tilde{C} \left( N^{-1},S^{-1}\left( N^{-1}\right) \right) \right) ,\\&\Longrightarrow {\mathbb E}\left( \tilde{C} \left( N^{-1},S^{-1}\left( N^{-1}\right) \right) \right) <{\mathbb E}(\tilde{C} (N^{n},S^{n}(N^{n})) )-W\sum _{m=0}^n\pi ^n(m), \forall n; \end{aligned}$$

that is, $g^{(-1)}(W)<g^{(n)}(W)$ for all $n\in \mathbb {N}^0$, and hence $g(W)=g^{(-1)}(W)$. Policy $-1$ is therefore optimal for problem (8) for $W<W_0$.

We will prove 2. by induction. Observe from the definition of $n_0$ that for all $n\ge 0$

$$\begin{aligned} {\mathbb E}(\tilde{C} (N^{n_0},S^{n_0}(N^{n_0})) )-W_0\sum _{m=0}^{n_0}\pi ^{n_0}(m)\le {\mathbb E}(\tilde{C} (N^n,S^n(N^n)) )-W_0\sum _{m=0}^{n}\pi ^n(m); \end{aligned}$$

that is, $g^{(n_0)}(W_0)\le g^{(n)}(W_0)$, for all $n\ge 0$. Besides, we trivially have that $g^{(n_0)}(W_0)\le g^{(-1)}(W_0)$. We have proven in the proof of Proposition 2 that $\sum _{m=0}^n\pi ^n(m)$ is strictly increasing in $n$, and therefore for all $n\le n_0$ and $W_0< W$

$$\begin{aligned}&{\mathbb E}(\tilde{C} (N^{n_0},S^{n_0}(N^{n_0})) )-W\sum _{m=0}^{n_0}\pi ^{n_0}(m)\le {\mathbb E}(\tilde{C} (N^n,S^n(N^n)) )-W\sum _{m=0}^{n}\pi ^n(m)\\&\Longrightarrow g^{(n_0)}(W)\le g^{(n)}(W). \end{aligned}$$

In particular, $g^{(n_0)}(W)\le g^{(n)}(W)$ is satisfied for all $W_0<W<W_1$ and $n\le n_0$. Similarly, from the definition of $W_1$ we have that $g^{(n_0)}(W_1)\le g^{(n)}(W_1)$ for all $n\ge n_0+1$, and again using that $\sum _{m=0}^n\pi ^n(m)$ is strictly increasing we obtain $g^{(n_0)}(W)\le g^{(n)}(W)$ for all $W_0<W<W_1$ and $n\ge n_0+1$.

We have therefore proven that $g^{(n_0)}(W)\le g^{(n)}(W)$ for all $n$ and $W_0<W<W_1$; that is, policy $n_0$ is optimal for all $W$ such that $W_0<W<W_1$. This establishes the first step of the induction, $i=0$. Let us now assume that it holds for step $i-1\ge 0$; that is, $n_i$ is an optimal policy for problem (8), given $W$ such that $W_{i-1}<W<W_{i}$, and let us assume $n_{i}<\infty$. The definition of $W_i$ together with the fact that $n_{i-1}$ is optimal for the choice of $W$ such that $W_{i-1}<W<W_{i}$, imply

$$\begin{aligned} g^{(n_{i-1})}(W_i)=g^{(n_i)}(W_i)\le g^{(n)}(W_i), n\ge 0. \end{aligned}$$

Recall that $\sum _{m=0}^n\pi ^n(m)$ is strictly increasing in $n$ and therefore

$$\begin{aligned} g^{(n_i)}(W)\le g^{(n)}(W), n\le n_i, W_i<W<W_{i+1}. \end{aligned}$$

Besides, from the definition of $W_{i+1}$ we have

$$\begin{aligned} g^{(n_i)}(W)\le g^{(n)}(W), n\ge n_i+1, W_i<W<W_{i+1}. \end{aligned}$$

We therefore have obtained that threshold policy $n_i$ is optimal for problem (8) given $W$ such that $W_i<W<W_{i+1}$.

Finally, we prove 3. for $J<\infty$, note that from the induction followed in the previous point we have that

$$\begin{aligned} g^{(n_{J-1})}(W_J)=g^{(n_J)}(W_J)\le g^{(n)}(W_J), n\ge 0, \end{aligned}$$

and the fact that $\sum _{m=0}^n\pi ^n(m)$ is increasing in $n$ gives that

$$\begin{aligned} g^{(n_J)}(W)<g^{(n)}(W), n\le n_J=\infty , W_J<W. \end{aligned}$$

Which concludes the proof of the theorem.

Appendix 3: Proof of Proposition 3

For ease of notation we omit the subscript $k$ from the notation in the proof. To calculate Whittle’s index as in Theorem 1 we need to consider the monotone policies $n$ and $n-1$ in which the server is active in states $m\ge n+1$ and $m\ge n$, respectively.

Let us consider the policy $n$ first. Let $f^{n}(ab)$ and $f^{n}(ser)$ denote the fraction of customers that end up abandoning and being served, respectively. A rate conservation argument implies that all arriving users either abandon or are served, thus $\lambda = \lambda f^{n}(ab) + \lambda f^{n}(ser)$. Conditioning on the state, the rate of abandonment from the system can be written as $\sum _{m=0}^\infty \theta m \pi ^{n}(m) + (\theta '-\theta ) \sum _{m=n+1}^\infty \pi ^{n}(m)$, and equating the rates we get the relation

$$\begin{aligned} \theta {\mathbb E}(N^{n}) + (\theta '-\theta ) \sum _{m=n+1}^\infty \pi ^{n}(m) = \lambda f^{n}(ab) = \lambda (1-f^{n}(ser)). \end{aligned}$$

(38)

Conditioning on the state, the rate of service is given by $\sum _{m=n+1}^\infty \mu \pi ^{n}(m)$, and we get the relation

$$\begin{aligned} \lambda f(ser)=\mu \sum _{m=n+1}^\infty \pi ^{n}(m), \end{aligned}$$

and substituting in (38) we get

$$\begin{aligned} {\mathbb E}(N^n) = \frac{\lambda }{\theta } + \frac{\theta - \theta ' - \mu }{\theta } \sum _{m=n+1}^\infty \pi ^{n}(m), \end{aligned}$$

where $N^{n}$ denotes the stationary number of class-$k$ customers in the system under the threshold policy $n$. We calculate now the average holding cost. Plugging the holding cost $C_k(n_k,a)=c_k (n_k-a)^+ + c_k'a$ in the total cost relation (4) we get $\tilde{C}(n, a)= \tilde{c} n + a (\tilde{c}' -\tilde{c})$, where the constants $\tilde{c}$ and $\tilde{c}'$ are defined in Sect. 6.1. The average cost then becomes

$$\begin{aligned} {\mathbb E}( \tilde{C}(N^{n},S^{n}(N^{n})))= & {} \tilde{c} {\mathbb E}(N^n) + (\tilde{c}' -\tilde{c}) \sum _{m=n+1}^\infty \pi ^{n}(m) \\= & {} \tilde{c} \frac{\lambda }{\theta } + \left( \frac{\tilde{c} ( \theta -\theta ' - \mu )}{\theta } + \tilde{c}' - \tilde{c} \right) \sum _{m=n+1}^\infty \pi ^{n}(m)\\= & {} \tilde{c} \frac{\lambda }{\theta } + \left( \tilde{c}' - \frac{\tilde{c} ( \theta ' + \mu )}{\theta } \right) \sum _{m=n+1}^\infty \pi ^{n}(m). \end{aligned}$$

We substitute now all the terms in (15) to get

$$\begin{aligned} W(n) = \frac{\tilde{c} (\mu + \theta ' )}{\theta } -\tilde{c}', \end{aligned}$$

(39)

which concludes the proof.

Appendix 4: Proof of Proposition 4

For ease of notation we drop the dependency on $k$ throughout the proof.

The index in the case $\mu +\theta '=\theta$ was obtained in (16), therefore we assume $\mu +\theta '>\theta$ throughout the proof. First of all recall that the steady-state probabilities $\pi ^n(i)$ for policy $n$ and state $i$ are given by (11). To compute Whittle’s index for large values of $n$, we need to compute $\pi ^n(i)-\pi ^{n-1}(i), \forall i\ge 0$. Let us start with $i=0$; that is,

$$\begin{aligned} \pi ^n(0) -\pi ^{n-1}(0)= & {} \dfrac{\left( \pi ^{n-1}(0)\right) ^{-1}-\left( \pi ^n(0)\right) ^{-1}}{\left( \pi ^n(0)\pi ^{n-1}(0)\right) ^{-1}}\\= & {} \left( \sum _{i=1}^\infty \prod _{m=1}^i\dfrac{q^{n-1}(m-1,m)}{q^{n-1}(m,m-1)}\right. \\&\left. \quad -\sum _{i=1}^\infty \prod _{m=1}^i\dfrac{q^n(m-1,m)}{q^n(m,m-1)}\right) \pi ^n(0)\pi ^{n-1}(0). \end{aligned}$$

The following observations on the transition rates will be used throughout the proof:

$$\begin{aligned} q^n(m,m-1)= & {} q^{n-1}(m,m-1),\,\,\, \forall \, m\ne n, m\ge 1, \end{aligned}$$

(40)

$$\begin{aligned} q^n(m-1,m)= & {} q^{n-1}(m-1,m),\,\,\, \forall \, m\ge 1. \end{aligned}$$

(41)

Taking these relations into account together with the fact that $q^n(n,n-1)-q^{n-1}(n,n-1)=\theta -\mu -\theta '$, we get after some calculations

$$\begin{aligned} \begin{aligned} \pi ^n(0) -\pi ^{n-1}(0)&=\pi ^n(0)\pi ^{n-1}(0) \sum _{i=n}^\infty \prod _{\begin{array}{c} m=1\\ m\ne n \end{array}}^i \dfrac{q^{n}(m-1,m)}{q^{n}(m,m-1)}\\&\quad \left( \dfrac{1}{q^{n-1}(n,n-1)}-\dfrac{1}{q^n(n,n-1)}\right) \\&=\pi ^n(0)\pi ^{n-1}(0)\frac{\theta -\mu -\theta '}{q^{n-1}(n,n-1)}\sum _{i=n}^\infty \prod _{m=1}^i\dfrac{q^{n}(m-1,m)}{q^{n}(m,m-1)}.\end{aligned}\end{aligned}$$

Since $q^{n}(m-1,m)=\lambda$ for all $m\ge 1$, $q^{n}(m,m-1)=\theta m$ for all $1\le m\le n-1$ and $q^{n}(m,m-1)=\mu +\theta '+\theta (m-1)$ for all $m\ge n$, together with $\pi ^n(0)$ given as in (11), we observe that

$$\begin{aligned} \frac{\pi ^n(0)\pi ^{n-1}(0)}{q^{n-1}(n,n-1)}\in \mathcal {O}\left( \frac{1}{n}\right) \quad \hbox { and }\quad \sum _{i=n}^\infty \prod _{m=1}^i\dfrac{q^{n}(m-1,m)}{q^{n}(m,m-1)}\in \mathcal {O}\left( \frac{1}{n!}\right) . \end{aligned}$$

We then get that

$$\begin{aligned} \pi ^n(0)-\pi ^{n-1}(0)\in \mathcal {O}\left( \frac{1}{nn!}\right) . \end{aligned}$$

(42)

We can now compute $\pi ^n(i)-\pi ^{n-1}(i)$, for all $0<i\le n-1$. Using  (41), we obtain for $i\le n-1$,

$$\begin{aligned} \pi ^n(i) - \pi ^{n-1}(i)&=\prod _{m=1}^i\dfrac{q^n(m-1,m)}{q^n(m,m-1)}(\pi ^n(0)-\pi ^{n-1}(0)). \end{aligned}$$

Due to (42) and since $q^n(m,m-1)=\theta m$, and $q^n(m-1,m)=\lambda$ for all $m \le n-1$, we obtain for $i\le n-1$

$$\begin{aligned} \pi ^n(i) - \pi ^{n-1}(i)=\frac{\lambda ^i}{i!\theta ^i}\left( \pi ^n(0)-\pi ^{n-1}(0)\right) \in \mathcal {O}\left( \frac{1}{nn!}\right) . \end{aligned}$$

(43)

For states $i \ge n$, with $n$ sufficiently large, we have the following:

$$\begin{aligned} \pi ^n(i)-\pi ^{n-1}(i)&=\prod _{m=1}^i\dfrac{q^n(m-1,m)}{q^n(m,m-1)} \pi ^n(0)\nonumber \\&\quad -\prod _{m=1}^i\dfrac{q^{n-1}(m-1,m)}{q^{n-1}(m,m-1)} \left( \pi ^{n}(0)-\pi ^n(0)+\pi ^{n-1}(0)\right) .\nonumber \end{aligned}$$

From observation (42), together with $\prod _{m=1}^i\dfrac{q^{n-1}(m-1,m)}{q^{n-1}(m,m-1)}\in \mathcal {O}\left( \frac{1}{i!}\right) ,$ we obtain

$$\begin{aligned} \pi ^n(i)-\pi ^{n-1}(i)= & {} \mathcal {O}\left( \frac{1}{i!n!n}\right) \nonumber \\&+\prod _{m=1}^i\dfrac{q^n(m-1,m)}{q^n(m,m-1)}\pi ^n(0)-\prod _{m=1}^i\dfrac{q^{n-1}(m-1,m)}{q^{n-1}(m,m-1)}\pi ^{n}(0).\nonumber \end{aligned}$$

After some calculations and by observations (40) and (41) we obtain

$$\begin{aligned} \pi ^n(i)-\pi ^{n-1}(i)= & {} \left( \dfrac{1}{q^{n}(n,n-1)}-\dfrac{1}{q^{n-1}(n,n-1)}\right) \prod _{\begin{array}{c} m=1\\ m\ne n \end{array}}^i \dfrac{q^{n}(m-1,m)}{q^{n}(m,m-1)}\nonumber \\&+\mathcal {O}\left( \frac{1}{i!n!n}\right) \nonumber \\= & {} \frac{\mu +\theta '-\theta }{q^{n-1}(n,n-1)}\pi ^n(i)+\mathcal {O}\left( \frac{1}{i!n!n}\right) , \end{aligned}$$

(44)

for $i\ge n$. Recall from (18) that Whittle’s index can be written as $d(\mu +\theta ')-d'\theta '+W^c(n)$, where $W^c(n)$ corresponds to the holding costs only. $W^c(n)$ can be written as follows

$$\begin{aligned} W^c(n)&=\frac{\xi _1(n)+\xi _2(n)+\xi _3(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}(\pi ^n(m)-\pi ^{n-1}(m))}=\frac{\xi _1(n)+\xi _2(n)+\xi _3(n)}{\pi ^n(n)+\mathcal {O}(1/n!n)}, \end{aligned}$$

(45)

with

$$\begin{aligned} \xi _1(n)&:=\sum _{i=1}^{n-1}C(i,0)\left( \pi ^n(i)-\pi ^{n-1}(i)\right) ,\nonumber \\ \xi _2(n)&:=C(n,0)\pi ^n(n)-C(n,1)\pi ^{n-1}(n),\nonumber \\ \xi _3(n)&:=\sum _{i=n+1}^\infty C(i,1)\left( \pi ^n(i)-\pi ^{n-1}(i)\right) . \end{aligned}$$

(46)

Recall now the assumption that the holding costs $C(n,1)$ and $C(n,0)$ are upper bounded by polynomials of finite degrees $P<\infty$ and $Q<\infty$, respectively. Hence, we can write $C(n,a)=E(n,a)+o(1)$, for large values of $n$, where $E(n,1)$ $=\sum _{i=0}^{P}C^{(P,i)}n^i$, with $C^{(P,i)}:=\lim _{n\rightarrow \infty }\frac{C(n,1)-\sum _{j=i+1}^{P}C^{(P,j)}n^j}{n^{i}},$ and $E(n,0)=\sum _{i=0}^{Q}E^{(Q,i)}n^i$, with $E^{(Q,i)}:=\lim _{n\rightarrow \infty }\frac{C(n,0)-\sum _{j=i+1}^{Q}E^{(Q,j)}n^j}{n^{i}}.$ We assume w.l.o.g. that $P$ is such that $C^{(P,P)}>0$ and $Q$ such that $E^{(Q,Q)}>0$. We then have

$$\begin{aligned} \begin{aligned} \xi _1(n)&=\sum _{i=1}^{n-1}E(i,0)\left( \pi ^n(i)-\pi ^{n-1}(i)\right) +o(1),\\ \xi _2(n)&=E(n,0)\pi ^n(n)-E(n,1)\pi ^{n-1}(n)+o(1),\\ \xi _3(n)&=\sum _{i=n+1}^\infty E(i,1)\left( \pi ^n(i)-\pi ^{n-1}(i)\right) +o(1). \end{aligned} \end{aligned}$$

We now define $\hat{\xi _1}:=\sum _{i=1}^{n-1}E(i,0)(\pi ^n(i)-\pi ^{n-1}(i))$, and with the result obtained in Eq. (43) we have that for large values of $n$, $\hat{\xi }_1(n)\in \mathcal {O}\left( \frac{n^{Q-1}}{n!}\right) \subset o(1).$

Hence, for large values of $n$, $\xi _1(n)\in o(1)$. Let us now define $\hat{\xi }_2(n)$ $:=E(n,0)\pi ^n(n)-E(n,1)\pi ^{n-1}(n)$. Using (40) and (41) we have after some calculations,

$$\begin{aligned} \hat{\xi }_2(n)=\frac{\prod _{m=1}^{n}q^n(m-1,m)}{\prod _{m=1}^{n-1}q^n(m,m-1)}\bigg (\frac{E(n,0)\pi ^n(0)}{q^n(n,n-1)}-\frac{E(n,1)\pi ^{n-1}(0)}{q^{n-1}(n,n-1)}\bigg ). \end{aligned}$$

We recall that $q^{n-1}(n,n-1)=\mu +\theta '+\theta (n-1)$ and $q^n(n,n-1)=\theta n$, which together with (42) give, after some calculations,

$$\begin{aligned} \hat{\xi }_2(n)= & {} \prod _{m=1}^{n}\frac{q^{n}(m\!-\!1,m)}{q^{n}(m,m\!-\!1)}\frac{\theta n}{q^{n-1}(n,n\!-\!1)}\left( \left( E(n,0)\!-\!E(n,1)\right) \pi ^n(0)\!+\!\mathcal {O}\left( \frac{n^{P-1}}{n!}\right) \right) \\&+\pi ^n(n)(\mu \!+\!\theta '\!-\!\theta )\frac{E(n,0)}{q^{n-1}(n,n-1)}. \end{aligned}$$

Since for large values of $n$

$$\begin{aligned} \prod _{m=1}^{n}\frac{q^{n}(m-1,m)}{q^{n}(m,m-1)}\frac{\theta n}{q^{n-1}(n,n-1)}\cdot \mathcal {O}\left( \frac{n^{P-1}}{n!}\right) \subset \mathcal {O}\left( \frac{n^{P-1}}{(n!)^2}\right) \subset o(1), \end{aligned}$$

we conclude that

$$\begin{aligned} \xi _2(n)=\frac{\pi ^n(n)}{q^{n-1}(n,n-1)}\bigg (\theta n(E(n,0)-E(n,1))+(\mu +\theta '-\theta )E(n,0)\bigg )+o(1). \end{aligned}$$

(47)

Finally, we compute $\hat{\xi }_3(n):=\sum _{i=n+1}^\infty E(i,1)(\pi ^n(i)-\pi ^{n-1}(i))$. From (44) we see that

$$\begin{aligned} \hat{\xi }_3(n)= \frac{\mu +\theta '-\theta }{q^{n-1}(n,n-1)}\sum _{i=n+1}^\infty E(i,1)\pi ^n(i)+\sum _{i=n+1}^\infty E(i,1)\cdot \mathcal {O}\left( \frac{1}{i!n!n}\right) . \end{aligned}$$

Since for large values of $n$, $\sum _{i=n+1}^\infty E(i,1)\cdot \mathcal {O}\left( \frac{1}{i!n!n}\right) \subset \mathcal {O}\left( \frac{n^{P-1}}{i!n!}\right) \subset o(1),$ we obtain

$$\begin{aligned} \xi _3(n)=\frac{\mu +\theta '-\theta }{q^{n-1}(n,n-1)}\sum _{i=n+1}^\infty E(i,1)\pi ^n(i)+o(1). \end{aligned}$$

(48)

Now using $\xi _1\in o(1)$, the expression for $\xi _2(n)$ in (47), and (48), and letting $n$ be large, we see that $\frac{\xi _1(n)}{\pi ^n(n)}\in o(1),$ and,

$$\begin{aligned} \frac{\xi _2(n)}{\pi ^n(n)}&=\frac{\theta n (E(n,0)-E(n,1))}{\mu +\theta '+\theta (n-1)}+\frac{(\mu +\theta '-\theta )E(n,0)}{\mu +\theta '+\theta (n-1)}+o(1)\\&=E(n,0)-E(n,1)+ \frac{(\mu +\theta '-\theta )}{\theta n}E(n,0)+o(1)\\&=E(n,0)-E(n,1)+ \frac{(\mu +\theta '-\theta )}{\theta }\sum _{j=1}^{Q}E^{(P,j)}n^{j-1}+o(1), \end{aligned}$$

and

$$\begin{aligned} \frac{\xi _3(n)}{\pi ^n(n)}= & {} \frac{\mu +\theta '-\theta }{\mu +\theta '+\theta (n-1)}\cdot \sum _{i=n+1}^\infty E(i,1)\prod _{m=n+1}^i\frac{\lambda }{\mu +\theta '+\theta (m-1)}+o(1)\\= & {} \frac{\mu +\theta '-\theta }{\theta n}\sum _{i=n+1}^\infty \sum _{j=0}^{P}C^{(P,j)}i^j\left( \frac{\lambda }{\theta m}\right) ^{i-n}+o(1). \end{aligned}$$

Define $\tilde{W}^c(n)$ as $W^c(n)$ for large values of $n$. Substituting the expressions for $\xi _1(n)/\pi ^n(n), \xi _2(n)/\pi ^n(n)$ and $\xi _3(n)/\pi ^n(n)$ in Eq. (45), we obtain

$$\begin{aligned} \tilde{W}^c(n)= & {} (E(n,0)-E(n,1))+ (\mu +\theta '-\theta )/\theta \\&\times \left( \sum _{j=1}^{Q}E^{(Q,j)}n^{j-1}+ \sum _{i=2}^{P} C^{(P,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}\right) +o(1), \end{aligned}$$

as $n\rightarrow \infty$; that is, the expression in Eq. (19). $E(n,a)$ being non-decreasing together with Condition 2 implies that $\tilde{W}^c$ is non-decreasing, and hence $W^\infty$ as well, which concludes the proof.

Appendix 5: Proof of Propostion 5

For ease of notation we drop the dependency on $k$ throughout the proof.

The index in the case $\mu +\theta '=\theta$ was obtained in (16), therefore we assume $\mu +\theta '>\theta$ throughout the proof. Recall from (18) that Whittle’s index can be written as $d(\mu +\theta ')-d'\theta '+W^c(n)$, where $W^c(n)$ corresponds to the holding costs only. Recall from (45) that $W^c(n)$ can be written as

$$\begin{aligned} W^c(n)&=\frac{\xi _1(n)+\xi _2(n)+\xi _3(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}(\pi ^n(m)-\pi ^{n-1}(m))} \end{aligned}$$

(49)

with $\xi _i(n)$ for $i\in \{1,2,3\}$ as given in Eq. (46).

Let us first compute $\lim _{\lambda \rightarrow 0}\pi ^{n-1}(0)/\pi ^n(0)$, since this result will be used later in the proof. Recall the expression of the steady-state probabilities as defined in (11). Using this together with (40) and (41) we obtain

$$\begin{aligned} \lim _{\lambda \rightarrow 0}\frac{\pi ^{n-1}(0)}{\pi ^n(0)}&=\lim _{\lambda \rightarrow 0}\frac{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^n(i,i-1)}}{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1} (i,i-1)}}\nonumber \\ \!&=\!1\!+\!\lim _{\lambda \rightarrow 0}\frac{\sum _{m=n}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^n(i,i-1)}\!-\!\sum _{m=n}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i\!-\!1)}}\nonumber \\&=1+\lim _{\lambda \rightarrow 0}\frac{\sum _{m=n}^\infty \left( \frac{\lambda ^m(\mu +\theta '+\theta (n-1))}{\theta n\prod _{i=1}^mq^{n-1}(i,i-1)}-\frac{\lambda ^m \theta n}{\theta n\prod _{i=1}^mq^{n-1}(i,i-1)}\right) }{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}\nonumber \\&=1+\frac{(\mu +\theta '-\theta )}{\theta n}\cdot \lim _{\lambda \rightarrow 0}\frac{\mathcal {O}(\lambda ^n)}{1+\mathcal {O}(\lambda )}=1. \end{aligned}$$

(50)

From this last result we observe the following:

$$\begin{aligned} \lim _{\lambda \rightarrow 0}\frac{\lambda ^n/(\theta ^nn!)}{1-\pi ^{n-1}(0)/\pi ^n(0)}&=\lim _{\lambda \rightarrow 0}\frac{\lambda ^n/(\theta ^{n}n!)}{-\frac{(\mu +\theta '-\theta )}{\theta n}\left( \frac{\frac{\lambda ^n}{(\mu +\theta '+\theta (n-1))\theta ^{n-1}(n-1)!}+\mathcal {O}(\lambda ^{n+1})}{1+\mathcal {O}(\lambda )}\right) }\nonumber \\&\!=\!\lim _{\lambda \rightarrow 0}\!-\!\frac{\mu \!+\!\theta '\!+\!\theta (n\!-\!1)}{\mu \!+\!\theta '-\theta }\!+\!o(1)\!=\!-\frac{\mu +\theta '\!+\!\theta (n-1)}{\mu \!+\!\theta '-\theta }. \end{aligned}$$

(51)

Let us now consider the first term in (49); that is,

$$\begin{aligned}&\frac{\sum _{m=0}^{n-1}C(m,0)(\pi ^n(m)-\pi ^{n-1}(m))}{\sum _{m=0}^{n}\pi ^n(m)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\frac{\sum _{m=0}^{n-1}C(m,0)\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}(\pi ^n(0)-\pi ^{n-1}(0))}{\pi ^n(n)+\sum _{m=0}^{n-1}\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}(\pi ^n(0)-\pi ^{n-1}(0))}\nonumber \\&\quad =\frac{\sum _{m=0}^{n-1}C(m,0)\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}}{\frac{\pi ^n(n)}{\pi ^n(0)-\pi ^{n-1}(0)}+\sum _{m=0}^{n-1}\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}}\nonumber \\&\quad =\frac{\sum _{m=0}^{n-1}C(m,0)\frac{\lambda ^m}{\prod _{i=1}^{m}q^n(i,i-1)}}{\frac{\lambda ^n/(\theta ^nn!)}{1-\pi ^{n-1}(0)/\pi ^n(0)}+\sum _{m=0}^{n-1}\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}}. \end{aligned}$$

(52)

where in the first inequality we used the conditions (40) and (41). In order to obtain the limit of (52) as $\lambda \rightarrow 0$ we substitute the result obtained in (51), and we obtain the following:

$$\begin{aligned}&\lim _{\lambda \rightarrow 0}\frac{\xi _1(n)}{\sum _{m=0}^{n}\pi ^n(m)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\lim _{\lambda \rightarrow 0}\frac{\sum _{m=0}^{n-1}C(m,0)\frac{\lambda ^m}{\prod _{i=1}^{m}q^n(i,i-1)}}{\frac{\lambda ^n/(\theta ^nn!)}{1-\pi ^{n-1}(0)/\pi ^n(0)}+\sum _{m=0}^{n-1}\prod _{i=1}^{m}\frac{q^n(i-1,i)}{q^n(i,i-1)}}\nonumber \\&\quad =\lim _{\lambda \rightarrow 0}\frac{C(0,0)+\mathcal {O}(\lambda )}{-\frac{\mu +\theta '+\theta (n-1)}{\mu +\theta '-\theta }+1+\mathcal {O}(\lambda )}=-C(0,0)\frac{(\mu +\theta '-\theta )}{\theta n}. \end{aligned}$$

(53)

Let us now consider the second term in (49); that is,

$$\begin{aligned}&\frac{C(n,0)\pi ^n(n)-C(n,1)\pi ^{n-1}(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}\pi ^n(n)-\sum _{m=0}^{n-1}\pi ^n(n-1)}\nonumber \\&\quad =\frac{C(n,0)-C(n,1)\frac{\pi ^{n-1}(n)}{\pi ^n(n)}}{1+\frac{1}{\pi ^n(n)}(\pi ^n(0)-\pi ^{n-1}(0))\sum _{m=0}^{n-1}\frac{\lambda ^m}{\theta ^mm!}}\nonumber \\&\quad =\frac{C(n,0)-C(n,1)\frac{\theta n\pi ^{n-1}(0)}{(\mu +\theta '+\theta (n-1))\pi ^n(0)}}{1+\frac{\theta ^nn!}{\lambda ^n}(1-\pi ^{n-1}(0)/\pi ^n(0))\sum _{m=0}^{n-1}\frac{\lambda ^m}{\theta ^mm!}}. \end{aligned}$$

(54)

Substituting the results obtained in (50) and (51) in the expression of Eq. (54) we obtain

$$\begin{aligned}&\lim _{\lambda \rightarrow 0}\frac{\xi _2(n)}{\sum _{m=0}^n\pi ^n(n)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\lim _{\lambda \rightarrow 0}\frac{C(n,0)-C(n,1)\left( \frac{\theta n}{\mu +\theta '+\theta (n-1)}\right) (1+\mathcal {O}(\lambda ^n))}{1-\frac{\mu +\theta '-\theta }{\mu +\theta '+\theta (n-1)}(1+\mathcal {O}(\lambda ))}\nonumber \\&\quad =\lim _{\lambda \rightarrow 0}\frac{C(n,0)(\mu +\theta '+\theta (n-1))-C(n,1)\theta n+\mathcal {O}(\lambda ^n)}{\theta n(1+\mathcal {O}(\lambda ))}\nonumber \\&\quad =C(n,0)-C(n,1)+C(n,0)\frac{(\mu +\theta '-\theta )}{\theta n}+\mathcal {O}(\lambda ). \end{aligned}$$

(55)

To conclude the proof we need to analyze the third term in (49); that is,

$$\begin{aligned}&\frac{\sum _{m=n+1}^\infty C(m,1)\pi ^n(m)-\sum _{m=n+1}^\infty C(m,1)\pi ^{n-1}(m)}{\pi ^n(n)+\sum _{m=0}^{n-1}\pi ^n(m)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\\&\quad =\frac{\lambda ^n\sum _{m=n+1}^\infty \frac{\lambda ^{m-n}}{\prod _{i=1}^{n-1}q^n(i,i-1)\prod _{i=n+1}^mq^n(i,i-1)}\left( \frac{\pi ^n(0)}{q^n(n,n-1)}-\frac{\pi ^{n-1}(0)}{q^{n-1}(n,n-1)}\right) }{\lambda ^n\left( \frac{\pi ^n(0)}{\theta ^nn!}+\frac{1}{\lambda ^n}\left( \pi ^n(0)-\pi ^{n-1}(0)\right) \sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}\right) }\\&\quad =\frac{\sum _{m=n+1}^\infty \frac{\lambda ^{m-n}}{\prod _{i=n+1}^{m}q^n(i,i-1)}\left( 1-\frac{\theta n\pi ^{n-1}(0)}{(\mu +\theta '+\theta (n-1))\pi ^n(0)}\right) }{\left( 1+\frac{\theta ^n n!}{\lambda ^n}\left( 1-\frac{\pi ^{n-1}(0)}{\pi ^n(0)}\right) \sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}\right) }. \end{aligned}$$

In the last expression we substitute the results obtained in (50) and (51), and we show that

$$\begin{aligned}&\lim _{\lambda \rightarrow 0}\frac{\xi _3(n)}{\sum _{m=0}^n\pi ^n(n)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\lim _{\lambda \rightarrow 0} \frac{\sum _{m=n+1}^\infty \frac{\lambda ^{m-n}}{\prod _{i=n+1}^{m}q^n(i,i-1)}\left( 1-\frac{\theta n\pi ^{n-1}(0)}{(\mu +\theta '+\theta (n-1))\pi ^n(0)}\right) }{\left( 1+\frac{\theta ^n n!}{\lambda ^n}\left( 1-\frac{\pi ^{n-1}(0)}{\pi ^n(0)}\right) \sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}\right) }\nonumber \\&\quad =\lim _{\lambda \rightarrow 0} \frac{\sum _{m=n+1}^\infty \frac{\lambda ^{m-n}}{\prod _{i=n+1}^{m}q^n(i,i-1)}\left( 1-\frac{\theta n}{\mu +\theta '+\theta (n-1)}\left( 1+\mathcal {O}(\lambda ^n)\right) \right) }{\left( 1-\left( \frac{\mu +\theta '-\theta }{\mu +\theta '+\theta (n-1)}+\mathcal {O}(\lambda )\right) \sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}\right) }\nonumber \\&\quad =\lim _{\lambda \rightarrow 0}\frac{\mathcal {O}(\lambda )}{\frac{\theta n}{\mu +\theta '+\theta (n-1)}+\mathcal {O}(\lambda )}=0. \end{aligned}$$

(56)

We now substitute the results obtained in Eqs. (53),  (55) and (56) in $\lim _{\lambda \rightarrow 0}W^c(n)$, and we obtain

$$\begin{aligned} \lim _{\lambda \rightarrow 0}W^c(n)=C(n,0)-C(n,1)+\frac{(\mu +\theta '-\theta )}{\theta n}(C(n,0)-C(0,0)). \end{aligned}$$

Appendix 6: Proof of Proposition 6

For ease of notation we drop the dependency on $k$ throughout the proof.

The index in the case $\mu +\theta '=\theta$ was obtained in (16), therefore we assume $\mu +\theta '>\theta$ throughout the proof. Recall from (18) that Whittle’s index can be written as $d(\mu +\theta ')-d'\theta '+W^c(n)$, where $W^c(n)$ corresponds to the holding costs only. Recall from (45) that $W^c(n)$ can be written as

$$\begin{aligned} W^c(n)&=\frac{\xi _1(n)+\xi _2(n)+\xi _3(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}(\pi ^n(m)-\pi ^{n-1}(m))} \end{aligned}$$

(57)

with $\xi _i(n)$ for $i\in \{1,2,3\}$ as given by Eq. (46).

We first compute $\pi ^{n-1}(0)/\pi ^{n}(0),$ which will be used later in the proof:

$$\begin{aligned} \frac{\pi ^{n-1}(0)}{\pi ^n(0)}&=\frac{\sum _{m=0}^\infty \frac{ \lambda ^m}{\prod _{i=1}^mq^n(i,i-1)}}{\sum _{m=0}^\infty \frac{ \lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}\nonumber \\&=\left( 1+\frac{\sum _{m=n}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^n(i,i-1)}-\sum _{m=n}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}\right) \nonumber \\&=1+\frac{(\mu +\theta '-\theta )}{\theta n}\cdot \frac{\sum _{m=n}^\infty \frac{ \lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}{\sum _{m=0}^\infty \frac{ \lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}} \end{aligned}$$

(58)

$$\begin{aligned}&=1+\frac{\mu +\theta '-\theta }{\theta n}(1+o(1)). \end{aligned}$$

(59)

We now proceed to compute (57) as $\lambda \rightarrow \infty$. Let us begin by computing the term that corresponds to $\xi _1(n)$. We have, after some algebra,

$$\begin{aligned} \frac{\xi _1(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}\left( \pi ^n(m)-\pi ^{n-1}(m)\right) }&=\frac{\sum _{m=0}^{n-1} C(m,0)\left( \pi ^n(m)-\pi ^{n-1}(m)\right) }{\sum _{m=0}^{n}\pi ^n(m)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&=\frac{\sum _{m=0}^{n-1}C(m,0)\frac{\lambda ^m}{\theta ^mm!}}{\frac{\lambda ^n/(\theta ^nn!)}{1-\pi ^{n-1}(0)/\pi ^n(0)}+\sum _{m=0}^{n-1}\frac{\lambda ^m}{\theta ^mm!}}, \end{aligned}$$

(60)

which after substitution of (59) reduces to

$$\begin{aligned}&\frac{\xi _1(n)}{\pi ^n(n)+\sum _{m=0}^{n-1}(\pi ^n(m)-\pi ^{n-1}(m))} =\mathcal {O}\left( \frac{1}{\lambda }\right) , \end{aligned}$$

(61)

as $\lambda \uparrow \infty$, for all $n$. We are now interested in computing the second term in (57) as $\lambda \rightarrow \infty$. Using (59) we obtain

$$\begin{aligned}&\lim _{\lambda \rightarrow \infty }\frac{\xi _2(n)}{\sum _{m=0}^n\pi ^n(n)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\lim _{\lambda \rightarrow \infty }\frac{ C(n,0)- C(n,1)\frac{\pi ^{n-1}(n)}{\pi ^{n}(n)}}{1+\frac{\pi ^n(0)-\pi ^{n-1}(0)}{\pi ^n(n)}\sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}}\nonumber \\&\quad =\lim _{\lambda \rightarrow \infty }\frac{ C(n,0)- C(n,1)\frac{\theta n}{\mu +\theta '+\theta (n-1)}\frac{\pi ^{n-1}(0)}{\pi ^{n}(0)}}{1+\frac{1-\pi ^{n-1}(0)/\pi ^n(0)}{\lambda ^n/(\theta ^n n!)}\sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}}\nonumber \\&\quad = C(n,0)- C(n,1), \end{aligned}$$

(62)

for all $n$. We are left with the third term in (57); that is,

$$\begin{aligned}&\frac{\xi _3(n)}{\sum _{m=0}^n\pi ^n(m)-\sum _{m=0}^{n-1}\pi ^{n-1}(m)}\nonumber \\&\quad =\frac{\sum _{m=n+1}^\infty C(m,1)\frac{ \lambda ^m}{\prod _{i=1}^{n-1}q^n(i,i-1)\prod _{i=n+1}^{m}q^n(i,i-1)}\left( \frac{\pi ^n(0)}{\theta n}-\frac{\pi ^{n-1}(0)}{\mu +\theta '+\theta (n-1)}\right) }{\pi ^n(n)+(\pi ^n(0)-\pi ^{n-1}(0))\sum _{m=0}^{n-1}\frac{\lambda ^m}{m!\theta ^m}}\nonumber \\&\quad =\frac{\sum _{m=n+1}^\infty C(m,1)\frac{\lambda ^m}{\prod _{i=1}^{m}q^n(i,i-1)}\left( \frac{\theta n}{\mu +\theta '+\theta (n-1)}\left( 1-\frac{\pi ^{n-1}(0)}{\pi ^{n}(0)}\right) +\frac{\mu +\theta '-\theta }{\mu +\theta '+\theta (n-1)}\right) }{\lambda ^n/(\theta ^{n}n!)+(1-\pi ^{n-1}(0)/\pi ^n(0))\sum _{m=0}^{n-1}\frac{ \lambda ^m}{m!\theta ^m}}, \end{aligned}$$

(63)

where in the second step we used that $\prod _{i=1}^{n-1}q^n(i,i-1)\prod _{i=n+1}^{m}q^n(i,i-1)=\prod _{i=1}^mq^n(i,i-1)/\theta n$. After substituting (58) in the latter equation and some algebra, we obtain that (63) can be written as

$$\begin{aligned}&\frac{\mu +\theta '-\theta }{\theta n}\frac{\sum _{m=n+1}^\infty C(m,1)\frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}{\frac{\lambda }{\theta n}\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}(1+o(1))}.\\ \end{aligned}$$

Hence the third term as $\lambda \rightarrow \infty$ simplifies to

$$\begin{aligned}&\frac{\mu +\theta '-\theta }{\theta }\frac{\sum _{m=n+1}^\infty C(m,1)\frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}{\frac{\lambda }{\theta }\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{i=1}^mq^{n-1}(i,i-1)}}+o(1)\nonumber \\&\quad =\frac{\mu +\theta '-\theta }{\theta }\frac{\sum _{m=n+1}^\infty C(m,1)\pi ^{n-1}(m)}{\lambda /\theta }+o(1). \end{aligned}$$

(64)

The latter equality follows due to $\pi ^{n-1}(0)=\left( \sum _{m=0}^{\infty }\frac{\lambda ^m}{\prod _{j=1}^mq^{n-1}(j,j-1)}\right) ^{-1}$. We now write (64) as follows:

$$\begin{aligned}&\frac{\mu +\theta '-\theta }{\theta }\left( \frac{\sum _{m=0}^\infty C(m,1)\pi ^{n-1}(m)}{\lambda /\theta }-\frac{\sum _{m=0}^n C(m,1)\pi ^{n-1}(m)}{\lambda /\theta }\right) \nonumber \\&\quad = \frac{\mu +\theta '-\theta }{\theta }\frac{\mathbb {E}(C(N^{n-1},1))}{\lambda /\theta }\left( 1-\frac{\sum _{m=0}^n C(m,1)\frac{\lambda ^m}{\prod _{j=1}^m q^{n-1}(j,j-1)}}{\mathcal {O}(\lambda ^n)+\sum _{m=n+1}^\infty \frac{\lambda ^m}{\prod _{j=1}^mq^{n-1}(j,j-1)}}\right) , \end{aligned}$$

(65)

where

$$\begin{aligned} \mathbb {E}(C(N^{n-1},1))=\frac{\sum _{m=0}^{\infty }C(m,1)\frac{\lambda ^m}{\prod _{j=1}^{m}q^{n-1}(j,j-1)}}{\sum _{m=0}^\infty \frac{\lambda ^m}{\prod _{j=1}^mq^{n-1}(j,j-1)}}. \end{aligned}$$

We then have that if there exists $z\ge 1$ such that $\frac{\mathbb {E}(C(N^{n-1},1))}{\lambda ^z}\rightarrow 0$, as $\lambda \rightarrow \infty$, then (65) reduces to

$$\begin{aligned} \frac{\mu +\theta '-\theta }{\theta }\frac{\mathbb {E}(C(N^{n-1},1))}{\lambda /\theta }+o(1). \end{aligned}$$

Hence, together with Eqs. (57), (61) and (62) we obtain that

$$\begin{aligned} W^{c}(n)= C(n,0)- C(n,1)+\frac{\mu +\theta '-\theta }{\theta }\frac{\mathbb {E}(C(N^{n-1},1))}{\lambda /\theta }+o(1), \end{aligned}$$

as $\lambda \rightarrow \infty$. This concludes the proof.$\square$

Appendix 7: Proof of Proposition 7

For ease of notation, we omit the class index $k$ in the proof.

Since $\theta '=\theta$ we have $\mu +\theta '>\theta$. Since $d'=d=0$, $\theta '=\theta$ and $C(n,a)= C(n)$, we can write $\tilde{C}(n,a)= C(n)$. Hence, we are interested in the following limit:

$$\begin{aligned} \begin{aligned} \lim _{\theta \rightarrow 0} \theta W(n)&=\lim _{\theta \rightarrow 0}\dfrac{\theta \sum _{m=0}^{\infty }{C}(m)\left( \pi ^{n}(m)-\pi ^{n-1}(m)\right) }{\sum _{m=1}^{n-1}\left( \pi ^{n}(m)-\pi ^{n-1}(m)\right) +\pi ^{n}(n)}\\&=\varepsilon _1(n)\varepsilon _2(n), \end{aligned} \end{aligned}$$

with

$$\begin{aligned} \varepsilon _1(n)=\lim _{\theta \rightarrow 0}\dfrac{\theta }{\sum _{m=1}^{n-1}\left( \pi ^{n}(m)-\pi ^{n-1}(m)\right) +\pi ^{n}(n)}, \end{aligned}$$

and

$$\begin{aligned}\varepsilon _2(n) =\lim _{\theta \rightarrow 0}\sum _{m=0}^{\infty } C(m) \left( \pi ^{n}(m)-\pi ^{n-1}(m)\right) . \end{aligned}$$

Consider $\varepsilon _2(n)$. We have

$$\begin{aligned} \pi ^{n}(0)-\pi ^{n-1}(0)\xrightarrow {\theta \rightarrow 0}0, \end{aligned}$$

hence

$$\begin{aligned}&\pi ^{n}(m)-\pi ^{n-1}(m)\xrightarrow {\theta \rightarrow 0}0, \qquad \forall m<n-1,\\&\pi ^{n}(n-1)-\pi ^{n-1}(n-1)\xrightarrow {\theta \rightarrow 0}(\rho -1), \end{aligned}$$

and

$$\begin{aligned} \pi ^{n}(m)-\pi ^{n-1}(m)\xrightarrow {\theta \rightarrow 0}\rho ^{m-n}(1-\rho )^2, \qquad \forall m\ge n. \end{aligned}$$

This gives

$$\begin{aligned} \varepsilon _2(n)&= - C( n-1)(1-\rho ) + \frac{(1-\rho )}{\rho }\sum _{m=n}^\infty C(m)(1-\rho )\rho ^{m-n+1} \\&=\frac{(1-\rho ) }{\rho }(- C( n-1) + \sum _{m=0}^\infty C(m+ n -1)(1-\rho )\rho ^{m}). \end{aligned}$$

After some algebra and using that $\pi ^n(n)\xrightarrow {\theta \rightarrow 0}(1-\rho )^{-1}$ (as pointed out in Sect. 7), we obtain $\varepsilon _1(n) =1/\mu$. This concludes the proof.$\square$

Appendix 8: Proof of Theorem 2

We first assume there exists a $k$ such that $C_k(0,1)>0$. Let us consider that $W=0$, and from (22) we know that necessarily $\mathcal {C}^{{REL}}(0) \le \mathcal {C}^{OPT}$. We also consider the policy $\bar{u} \in \mathcal {U}$ that takes active action when the total number of customers in the system is 0, and is passive otherwise. Note that policy $\bar{u}$ does not take any scheduling decision. Since $\mu _k + \theta '_k \ge \theta _k$, for all $k$, the queue length under policy $\bar{u}$ stochastically upper bounds any policy $u \in \mathcal {U}$. Note that under the assumption $C_k(0,0)\ge C_k(0,1),$ $\forall k,$ it holds from (2) that, for all $n,$ $C_k(n,0)\ge C_k(n,1),$ which implies that $W_k(n)$ is always positive; see Sect. 5. Hence, it follows $\mathcal {C}^{WI} \le \mathcal {C}^{\bar{u}}$. We will now show that $\frac{\mathcal {C}^{\bar{u}}- \mathcal {C}^{{REL}}(0) }{\mathcal {C}^{OPT}} \rightarrow 0$ as $\lambda \rightarrow 0$, which in view of (22) implies the optimality of Whittle’s index policy.

We have $W_k(0)=C_k(0,0)-C_k(0,1)\ge 0,$ for all $k$. Setting $W=0$, it follows that for every class ${\textit{REL}}(0)$ is the threshold policy with threshold $-1$; that is, class-$k$ always activates for any state $n_k>-1$. Hence, under policy $REL(0)$ the steady-state probabilities for class-$k$ are given by (11) with threshold $n=-1$. It then follows that

$$\begin{aligned} \mathcal {C}^{{REL}(0)}(0)&= \sum _{k=1}^K \sum _{m=0}^\infty C_k(m,1) \pi _k^{-1}(m)\nonumber \\&= \sum _{k=1}^K C_k(0,1) \pi _k^{-1}(0) +\sum _{k=1}^KC_k(1,1)\frac{\lambda \gamma _k}{\mu _k+\theta _k'}\pi ^{-1}_k(0)+ \mathcal {O}(\lambda ^2), \end{aligned}$$

(66)

as $\lambda \downarrow 0$. We have $\pi _k^{-1}(0) = (1 + \mathcal {O}(\lambda ))^{-1}$, hence $\mathcal {C}^{{REL}(0)}(0) = \sum _{k=1}^K C_k(0,1) + \mathcal {O}(\lambda )$.

Under policy $\bar{u}\in \mathcal {U}$, every class $k$ behaves as an $M/M/\infty$ queue with arrival rate $\lambda \gamma _k$ and departure rate $\theta _k n_k$. We then have $\mathcal {C}^{\bar{u}} = \sum _{k=1}^K C_k(0,1) \mathrm {e}^{-\lambda \gamma _k/\theta _k} + \sum _{k=1}^K \sum _{m=1}^\infty C_k(m,0) \frac{ (\lambda \gamma _k)^m}{\theta _k^m m!} \mathrm {e}^{-\lambda \gamma _k/\theta _k} = \sum _{k=1}^K C_k(0,1) + \mathcal {O}(\lambda )$.

Hence,

$$\begin{aligned} \mathcal {C}^{\bar{u}}- \mathcal {C}^{{REL}(0)}(0)= & {} \mathcal {O}(\lambda ). \end{aligned}$$

(67)

We now note that in the limit $\lambda \rightarrow 0$, $\mathcal {C}^{OPT} \ge \mathcal {C}^{{REL}(0)}(0) = \mathcal {O}(1)$. Together with (22) and (67), we thus conclude that

$$\begin{aligned} \lim _{\lambda \downarrow 0} \frac{\mathcal {C}^{WI} - \mathcal {C}^{OPT} }{\mathcal {C}^{OPT}} \le \lim _{\lambda \downarrow 0} \frac{\mathcal {C}^{\bar{u}}- \mathcal {C}^{{REL}(0)}(0) }{\mathcal {C}^{OPT}} =0. \end{aligned}$$

In the case $C_k(0,1)=0$, then $C_k(0,0)\ge C_k(0,1)$ for all $k$ and $W_k(0)=C_k(0,0)-C_k(0,1)\ge 0$, for all $k$. Setting $W=0$, it follows that $REL(0)$ is the policy that activates class-$k$ for any $n_k\ge 0$. We consider $\bar{u}$ to be the policy that takes active action when the total number of customers in the system is 0 or 1, and is passive otherwise. Then

$$\begin{aligned} \mathcal {C}^{\bar{u}}=&\sum _{k=1}^KC_k(1,1)\frac{\lambda \gamma _k}{\mu _k+\theta _k'}\pi ^{\bar{u}}_k(0)+\sum _{k=1}^K\sum _{m=2}^\infty C_k(m,0)\frac{(\lambda \gamma _k)^m}{(\mu _k+\theta _k')\theta ^{m-1}m!}\pi ^{\bar{u}}_k(0), \end{aligned}$$

and $\pi ^{\bar{u}}_k(0)=\left( 1+\frac{\lambda \gamma _k}{\mu _k+\theta '_k}+\frac{(\lambda \gamma _k)^2}{(\mu _k+\theta _k')2\theta _k}+\mathcal {O}(\lambda ^3)\right) ^{-1}$ as $\lambda \rightarrow 0$. We have that $\pi ^{-1}_k(0)=\pi ^{\bar{u}}_k(0)+\mathcal {O}(\lambda ^2)$ as $\lambda \rightarrow 0$. Then the term that corresponds to $C(1,1)$ in $\mathcal {C}^{\bar{u}}$ and $\mathcal {C}^{{REL}(0)}(0)$ as given in (66) coincide up to a $\mathcal {O}(\lambda ^2)$ term. Hence, $\mathcal {C}^{\bar{u}}- \mathcal {C}^{{REL}(0)}(0) = \mathcal {O}(\lambda ^2)$ and $\mathcal {C}^{OPT} \ge \mathcal {C}^{{REL}(0)}(0) = \mathcal {O}(\lambda )$, which lead to the desired result $\lim _{\lambda \downarrow 0} \frac{\mathcal {C}^{WI} - \mathcal {C}^{OPT} }{\mathcal {C}^{OPT}} =0.$

Appendix 9: Proof of Theorem 3

Let us assume for the sake of clarity that there are only two classes of customers. It extends trivially to the general case of $k$ classes. We further assume w.l.o.g. $\bar{k}=2$, hence

$$\begin{aligned} \lim _{\lambda \rightarrow \infty }\frac{W_2(\lambda \gamma _2/\theta _2)}{ W_1(\lambda \gamma _1/\theta _1)}>1. \end{aligned}$$

We prove Theorem 3 as follows:

• Step 1 We assume that Whittle’s index is either constant (linear holding cost case), or strictly increasing; the general case follows similarly. We prove that there exists $\bar{W}(\lambda )$ such that $\lim _{\lambda \rightarrow \infty }\mathcal {C}^{{REL}(\bar{W}(\lambda ))}(\bar{W}(\lambda ))-\mathcal {C}^{{REL}(\bar{W}(\lambda ))}=0$, i.e., the optimal solution of the relaxed problem is feasible for the original problem.

• Step 2 From Step 1 we deduce that in the limit, ${REL}(\bar{W}(\lambda ))$ will only serve class $\bar{k}$ with probability 1, it becomes feasible for the original problem, i.e., ${REL}(\bar{W}(\lambda )) \in \mathcal {U}$ as $\lambda \rightarrow \infty$, and hence equivalent to Whittle’s index policy. Therefore $\lim _{\lambda \rightarrow \infty } \mathcal {C}^{{REL}(\bar{W}(\lambda ))}- \mathcal {C}^{WI}=0 .$

• Step 3 Applying the result in Eq. (22) we obtain $\lim _{\lambda \rightarrow \infty }\mathcal {C}^{WI}-\mathcal {C}^{OPT}=0$.

Let us first assume that $W_k(n_k)$ is constant for $k=1,2$, as is the case for linear holding cost. Later on we solve the case in which $W_k(n_k)$ in strictly increasing, following the steps above. If $W_k(n_k)$ is a constant for $k=1,2$, from the assumption in the statement, we can find $\bar{W}$ a constant such that $W_1(n_1)\le \bar{W}\le W_2(n_2)$. It then follows trivially that the relaxed policy becomes feasible for the original problem taking $W=\bar{W}$ and equivalent to the Whittle index policy. In view of (22) this in particular implies $\lim _{\lambda \rightarrow \infty }\mathcal {C}^{WI}-\mathcal {C}^{OPT}=0$.

We now assume $W_k(n_k)$ is strictly increasing for $k=1,2$. We will denote $b_k=\gamma _k/\theta _k$ for $k=1,2$.

Step 1 From the assumption $\lim _{\lambda \rightarrow \infty }W_2(\lambda b_2)/W_1(\lambda b_1)>1$, where $W_k(\lambda b_k)$ is a continuous non-decreasing function in $\lambda b_k$. It then follows that there exists $\bar{W}(\lambda )$ a continuous non-decreasing function in $\lambda$, such that

$$\begin{aligned} \lim _{\lambda \uparrow \infty }\frac{W_1(\lambda b_1)}{\bar{W}(\lambda )}<1, \lim _{\lambda \uparrow \infty }\frac{W_2(\lambda b_2)}{\bar{W}(\lambda )}>1. \end{aligned}$$

We have assumed $W_k(\lambda b_k)$ to be increasing, and hence it is invertible. We then obtain

$$\begin{aligned} \lim _{\lambda \uparrow \infty }\frac{\lambda b_1}{(W_1)^{-1}(\bar{W}(\lambda ))}<1, \end{aligned}$$

(68)

$$\begin{aligned} \lim _{\lambda \uparrow \infty }\frac{\lambda b_2}{(W_2)^{-1}(\bar{W}(\lambda ))}<1. \end{aligned}$$

(69)

The optimal policy of the relaxed problem is to serve all customers whose index is greater than $\bar{W}(\lambda )$. Together with (68) and (69) we will now prove that the optimal policy for the relaxed problem becomes feasible for the original problem taking $W=\bar{W}(\lambda )$ as $\lambda \rightarrow \infty$. Hence,

$$\begin{aligned} \lim _{\lambda \uparrow \infty } \mathcal {C}^{{REL}(\bar{W}(\lambda ))}(\bar{W}(\lambda ))-\mathcal {C}^{{REL}(\bar{W}(\lambda ))}=0. \end{aligned}$$

From (7) we have

$$\begin{aligned}&\mathcal {C}^{{REL}(\bar{W})}(\bar{W}(\lambda ))\nonumber \\&\quad =\sum _{k=1}^2\mathbb {E}\left( \tilde{C}(N_k^{\bar{W}(\lambda )}, S^{\bar{W}(\lambda ))}\left( N_k^{\bar{W}(\lambda )}\right) \right) \nonumber \\&\qquad -\bar{W}(\lambda )\left( 1-2+\sum _{k=1}^2\left( 1-\mathbb {E}\left( 1\!\!1_{S^{\bar{W}(\lambda )}(N_k^{\bar{W}(\lambda )})=1}\right) \right) \right) . \end{aligned}$$

(70)

Due to the independence of the classes of customers in the relaxed problem (note that in the relaxed problem serving one of the classes does not mean we can not serve the other) we can write

$$\begin{aligned}&\lim _{\lambda \uparrow \infty }\sum _{k=1}^2\left( 1-\mathbb {E}\left( 1\!\!1_{S^{\bar{W}(\lambda )}(N_k^{\bar{W}(\lambda )})=1}\right) \right) \nonumber \\&\quad =2-\left( \lim _{\lambda \uparrow \infty }\mathbb {P}(W_1(N_1)>\bar{W}(\lambda ))+\mathbb {P}(W_2(N_2)>\bar{W}(\lambda ))\right) \nonumber \\&\quad =2-\left( \lim _{\lambda \uparrow \infty }\mathbb {P}(N_1>(W_1)^{-1}(\bar{W}(\lambda )))+\mathbb {P}(N_2>(W_2)^{-1}(\bar{W}(\lambda )))\right) . \end{aligned}$$

(71)

Let us then compute $\lim _{\lambda \uparrow \infty }\mathbb {P}(N_k>(W_k)^{-1}(\bar{W}(\lambda )))=\lim _{\lambda \uparrow \infty }\mathbb {P}(N_k$ $>\lfloor {(W_k)^{-1}(\bar{W}(\lambda ))}\rfloor )$. To do so we first note that for a given $f(\lambda )$

$$\begin{aligned} \mathbb {P}(N_k>f(\lambda ))= & {} \sum _{m=f(\lambda )}^\infty \frac{(\lambda \gamma _k)^m}{\theta ^{f(\lambda )}f(\lambda )!\prod _{j=f(\lambda )+1}^m(\mu _k+\theta _k'+\theta _k(j-1))}\\&\cdot \frac{1}{\sum _{r=0}^{f(\lambda )}\frac{(\lambda \gamma _k)^r}{\theta _k^r r!}+\sum _{r=f(\lambda )+1}^\infty \frac{(\lambda \gamma _k)^r}{\theta _k^{f(\lambda )}f(\lambda )!\prod _{j=f(\lambda )+1}^r(\mu _k+\theta _k'+\theta _k(j-1))}}. \end{aligned}$$

Assume $f(\lambda )$ is a positive non-decreasing function in $\lambda$. Then, in the limit as $\lambda \rightarrow \infty$ we have

$$\begin{aligned}&\lim _{\lambda \uparrow \infty }\mathbb {P}(N_k>f(\lambda ))\nonumber \\&\quad =\lim _{\lambda \uparrow \infty }\sum _{m=f(\lambda )}^\infty \frac{(\lambda \gamma _k)^m}{\theta _k^{m}m!+\mathcal {O}(\theta _k^{m-1}(m-1)!)}\frac{1}{\sum _{j=0}^{f(\lambda )}\frac{(\lambda \gamma _k)^j}{\theta _k^jj!}+\sum _{j=f(\lambda )+1}^{\infty }\frac{(\lambda \gamma _k)^j}{\theta _k^j j!+\mathcal {O}(\theta _k^{j-1}(j-1)!)}}\nonumber \\&\quad =\lim _{\lambda \uparrow \infty }\sum _{m=f(\lambda )}^\infty \frac{(\lambda b_k)^m}{m!}\frac{1}{\sum _{j=0}^{\infty }\frac{(\lambda b_k)^j}{ j!}}=\lim _{\lambda \uparrow \infty }\frac{\sum _{m=0}^\infty \frac{(\lambda b_k)^m}{m!}-\sum _{m=0}^{f(\lambda )-1}\frac{(\lambda b_k)^m}{ m!}}{\sum _{j=0}^\infty \left( \lambda b_k\right) ^j\frac{1}{j!}}\nonumber \\&\quad =1-\lim _{\lambda \uparrow \infty }\mathrm {e}^{-\lambda b_k}\sum _{m=0}^{f(\lambda )-1}\frac{(\lambda b_k)^m}{m!}=1-\lim _{\lambda \uparrow \infty }\mathrm {e}^{-\lambda b_k}\sum _{m=0}^{f(\lambda )}\frac{(\lambda b_k)^m}{m!}=P(f(\lambda ),\lambda b_k)\nonumber \\&\quad ={\left\{ \begin{array}{ll} 0, &{} \hbox {if } \lim _{\lambda \uparrow \infty }\frac{\lambda b_k}{f(\lambda )}<1 ,\\ 1, &{} \hbox { if} \lim _{\lambda \uparrow \infty }\frac{\lambda b_k}{f(\lambda )}>1. \end{array}\right. } \end{aligned}$$

(72)

The last equality follows from the auxiliary Lemma 1 (see Appendix 10). We now take $f^k(\lambda )=\lfloor {(W_k)^{-1}(\bar{W}(\lambda ))}\rfloor$, for $k=1,2$. Then from (72) together with (68) and (69) we obtain that $\mathbb {P}(N_1>f^1(\lambda ))=0$, and $\mathbb {P}(N_2>f^2(\lambda ))=1$. Hence, (71) = 1, which implies

$$\begin{aligned} 1-2+\sum _{k=1}^2\left( 1-\mathbb {E}\left( 1\!\!1_{S^{\bar{W}(\lambda )}\left( N_k^{\bar{W}(\lambda )}\right) =1}\right) \right) =0. \end{aligned}$$

(73)

From (70) we then obtain

$$\begin{aligned}&\lim _{\lambda \rightarrow \infty }\mathcal {C}^{{REL}(\bar{W}(\lambda ))}(\bar{W}(\lambda ))=\lim _{\lambda \rightarrow \infty }\sum _{k=1}^2\mathbb {E}\left( \tilde{C}\left( N_k^{\bar{W}(\lambda )}, S^{\bar{W}(\lambda )}\left( N_k^{\bar{W}(\lambda )}\right) \right) \right) \\&\quad =\lim _{\lambda \rightarrow \infty }\mathcal {C}^{\textit{REL}(\bar{W}(\lambda ))}. \end{aligned}$$

Step 2 Since ${\textit{REL}}(\bar{W}(\lambda ))$ will only serve class $2$ with probability 1, it becomes feasible for the original problem, i.e., ${\textit{REL}}(\bar{W}(\lambda )) \in \mathcal {U}$ as $\lambda \rightarrow \infty$, and hence equivalent to Whittle’s index policy; that is,

$$\begin{aligned}&\lim _{\lambda \rightarrow \infty }\mathcal {C}^{\textit{REL}(\bar{W}(\lambda ))}=\lim _{\lambda \rightarrow \infty }\mathcal {C}^{WI}. \end{aligned}$$

Step 3 In view of Eq. (22) and the result in Step 2 we obtain

$$\begin{aligned} \lim _{\lambda \rightarrow \infty }\mathcal {C}^{WI}-\mathcal {C}^{\textit{OPT}}=0. \end{aligned}$$

This concludes the proof.$\square$

Appendix 10: Auxiliary Lemma 1

Lemma 1

Let $f(\lambda )$ be a positive continuous non-decreasing function in $\lambda$, and let $b>0$ be some constant. We further define $P(y,\tilde{\lambda }):=1-\mathrm {e}^{-\tilde{\lambda }}\sum _{m=0}^{y}\frac{\tilde{\lambda }^m}{m!}$. Then

$$\begin{aligned} \lim _{\lambda \rightarrow \infty } P(f(\lambda ),\lambda b)={\left\{ \begin{array}{ll} 0, &{} \text { if }\lim _{\lambda \rightarrow \infty }\frac{\lambda b}{f(\lambda )}<1,\\ 1, &{} \text { if }\lim _{\lambda \rightarrow \infty }\frac{\lambda b}{f(\lambda )}>1. \end{array}\right. } \end{aligned}$$

Proof

Let us first note that $P(f(\lambda ),\lambda )=\frac{1}{f(\lambda )!}\int _0^{\lambda b}\mathrm {e}^{-t}t^{f(\lambda )}\mathrm {d}t;$ see [1]. That is,

$$\begin{aligned} P(f(\lambda ),\lambda )&=\frac{\mathrm {e}^{-f(\lambda )}}{f(\lambda )!}\int _0^{\lambda b}\mathrm {e}^{f(\lambda )-t}t^{f(\lambda )}\mathrm {d}t\nonumber \\&=\frac{f(\lambda )^{f(\lambda )+1}\mathrm {e}^{-f(\lambda )}}{f(\lambda )!}\int _0^{\lambda b}\left( \mathrm {e}^{1-\frac{t}{f(\lambda )}}\frac{t}{f(\lambda )}\right) ^{f(\lambda )}\mathrm {d}\left( \frac{t}{f(\lambda )}\right) \nonumber \\&=\frac{f(\lambda )^{f(\lambda )+1}\mathrm {e}^{-f(\lambda )}}{f(\lambda )!}\int _0^{\lambda b/f(\lambda )}(\mathrm {e}^{1-u}u)^{f(\lambda )}\mathrm {d}u. \end{aligned}$$

(74)

We recall Stirling’s formula $f(\lambda )!=\sqrt{2\pi }f(\lambda )^{f(\lambda )+1/2}\mathrm {e}^{-f(\lambda )}(1+\mathcal {O}(\frac{1}{f(\lambda )}))$, from which we obtain

$$\begin{aligned}&\frac{f(\lambda )^{f(\lambda )+1}\mathrm {e}^{-f(\lambda )}}{f(\lambda )!}=\frac{\sqrt{\frac{f(\lambda )}{2\pi }}}{1+\mathcal {O}(\frac{1}{f(\lambda )})}\nonumber \\&\quad =\sqrt{\frac{f(\lambda )}{2\pi }}\left( 1+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) \right) . \end{aligned}$$

(75)

Let us first analyze the case $\lim _{\lambda \rightarrow \infty }(\lambda b)/f(\lambda )<1$. Then there exists $\epsilon >0$ such that $0\le (\lambda b)/f(\lambda )\le 1-\epsilon$ for large enough $\lambda$. Hence, $0\le \mathrm {e}^{1-u}u\le \mathrm {e}^{\epsilon }(1-\epsilon )<1$ for all $0\le u\le (\lambda b)/f(\lambda )$, and therefore from Eqs. (74) and (75) we obtain

$$\begin{aligned} P(f(\lambda ),\lambda )\le & {} \sqrt{\frac{f(\lambda )}{2\pi }}\left( 1+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) \right) \cdot \frac{\lambda b}{f(\lambda )}\left( \mathrm {e}^{\epsilon }(1-\epsilon )\right) ^{f(\lambda )}\nonumber \\= & {} \mathcal {O}\left( \frac{1}{\sqrt{f(\lambda )}}(\mathrm {e}^{\epsilon }(1-\epsilon ))^{f(\lambda )}\right) , \end{aligned}$$

(76)

for $\lambda$ large enough. Since, $\mathrm {e}^{\epsilon }(1-\epsilon )<1$, we have $\lim _{\lambda \rightarrow \infty }\mathcal {O}\left( \frac{1}{\sqrt{f(\lambda )}}(\mathrm {e}^{\epsilon }(1-\epsilon ))^{\kappa _\lambda }\right) =0$. Hence, from (76) we obtain $P(f(\lambda ),\lambda )=0$.

We now analyze the case $\lim _{\lambda \rightarrow \infty }(\lambda b)/f(\lambda )>1$. Then there exists $\epsilon >0$ such that $0\le (\lambda b)/f(\lambda )\ge 1+\epsilon$ for $\lambda$ large enough. The function $\mathrm {e}^{1-u}u$ can also be written as

$$\begin{aligned} \mathrm {e}^{1-u}u=\mathrm {e}^{1-u-\log (1-(1-u))}=\mathrm {e}^{-\sum _{i=2}^\infty \frac{1}{i}(1-u)^i}. \end{aligned}$$

From the latter and the saddle point method [17], p. 174] we have that for $\lambda$ large enough

$$\begin{aligned} \int _0^{\lambda b/f(\lambda )}(\mathrm {e}^{1-u}u)^{f(\lambda )}\mathrm {d}u= & {} \int _{-\infty }^{\infty }\mathrm {e}^{-\frac{1}{2}f(\lambda )(1-u)^2}\mathrm {d}u+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) \\= & {} \sqrt{\frac{2\pi }{f(\lambda )}}+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) . \end{aligned}$$

From Eq. (74) together with Eq. (75), we then obtain

$$\begin{aligned} P(f(\lambda ),\lambda )= & {} \sqrt{\frac{f(\lambda )}{2\pi }}\left( 1+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) \right) \left( \sqrt{\frac{2\pi }{f(\lambda )}}+\mathcal {O}\left( \frac{1}{f(\lambda )}\right) \right) \\= & {} 1+\mathcal {O}\left( \frac{1}{\sqrt{f(\lambda )}}\right) , \end{aligned}$$

for $\lambda$ large enough. From the latter we obtain, $\lim _{\lambda \rightarrow \infty }P(f(\lambda ),\lambda )=1$, if $\lim _{\lambda \rightarrow \infty }\lambda b/f(\lambda )>1$.$\square$

Appendix 11: Proof of Theorem 4

Throughout the proof we drop the dependency on $k$.

We first prove that $w^{(1)}, w^{(2)}$ and $w^{(3)}$ are non-decreasing and continuous functions. For that recall that the function $C(m,a)$ is convex, which implies

$$\begin{aligned}&t C(m,a)+(1-t)C(m',a)\ge C(t m+(1-t)m'), \forall t\in [0,1],\\&\quad \Longrightarrow C(m,a)-C(m',a)\ge \frac{C(t m+(1-t)m')-C(m',a)}{t}\\&\quad \Longrightarrow C(m,a)-C(m',a)\ge \lim _{t\rightarrow 0}\frac{C(t m+(1-t)m')-C(m',a)}{t}\\&\quad =(m-m')\frac{\mathrm {d}C(m',a)}{\mathrm {d}m'}. \end{aligned}$$

From the latter we deduce

$$\begin{aligned} \frac{C(m,a)-C(m',a)}{m-m'}\ge \frac{\mathrm {d}C(m',a)}{\mathrm {d}m'}\ge \frac{C(m',a)-C(m'',a)}{m'-m''}, \end{aligned}$$

for all $m''\le m'\le m$. Then $(C(m,a)-C(m,a))(m'-m'')\ge (C(m',a)-C(m',a))(m-m')$. Adding and subtracting $C(m'',a)(m'-m'')$ in the LHS of the inequality and after some algebra, we obtain

$$\begin{aligned} \frac{C(m,a)-C(m'',a)}{m-m''}\ge \frac{C(m',a)-C(m'',a)}{m'-m''}. \end{aligned}$$

Hence, $\frac{C(m,a)-C(m'',a)}{m-m''}$ is non-decreasing in $m$. Similarly in $m''$. The latter directly implies that functions $w^{(1)}$ and $w^{(3)},$ under the assumption $\mu +\theta '\ge \theta$, are non-decreasing. To prove that $w^{(2)}$ is also non-decreasing, let us prove that $\mathrm {d}w^{(2)}(m)/\mathrm {d}m>0$ for all $\max (0,\lambda /(\mu +\theta '-\theta ))\le m\le \lambda /\theta$. We write

$$\begin{aligned} \frac{\mathrm {d} w^{(2)}(m)}{\mathrm {d}m}&=2\left( \frac{\mathrm {d}C(m,0)}{\mathrm {d}m}-\frac{\mathrm {d}C(m,1)}{\mathrm {d}m}\right) +\frac{(\lambda -\theta m)}{\theta }\frac{\mathrm {d}^2C(m,1)}{\mathrm {d}m^2}\\&\quad +\frac{(\theta m+\mu +\theta '-\theta -\lambda )}{\theta }\frac{\mathrm {d}^2C(m,0)}{\mathrm {d}m^2}. \end{aligned}$$

The first term is positive because of Eq. (23). Convexity of $C(\cdot ,\cdot )$ implies that the second and the third terms are positive in the interval $[\max (0,\lambda /(\mu +\theta '-\theta )),\lambda /\theta ]$. This implies that the function $w^{(2)}$ is also non-decreasing in $m$. Continuity of $w^{(1)}, w^{(2)}$ and $w^{(3)}$ follows from the fact that

$$\begin{aligned} \lim _{m\uparrow (\lambda -(\mu +\theta '-\theta ))/\theta }\frac{ C\left( \frac{\lambda -(\mu +\theta '-\theta )}{\theta },1\right) - C(m,1)}{(\lambda -(\mu +\theta '-\theta ))/\theta -m} = \frac{\hbox {d} C(m,1)}{\hbox {d} m}, \end{aligned}$$

hence $\lim _{{m\uparrow (\lambda -(\mu +\theta '-\theta ))/\theta }} w^{(1)}(m)=w^{(2)}((\lambda -(\mu +\theta '-\theta ))/\theta ),$ and

$$\begin{aligned} \lim _{m\downarrow \lambda /\theta } \frac{ C(m,0)-C(\lambda /\theta ,0)}{ m-\lambda /\theta }= \frac{\hbox {d} C(m,0)}{\hbox {d}m}, \end{aligned}$$

hence $\lim _{m\downarrow \lambda /\theta } w^{(3)}(m) = w^{(2)}(\lambda /\theta )$.

Having proved that $w(\cdot )$ is non-decreasing and continuous, we are left to prove that the optimal control for problem (26) is $s^*(t)=1$ when $W<w(m(t))$ and $s^*(t)=0$ when $W\ge w(m(t))$. In order to do so, we start by characterizing the optimal equilibrium point. Recall that an equilibrium point $(\bar{m}, \bar{s})$ of $\frac{\mathrm {d}m(t)}{\mathrm {d}t}$ is such that

$$\begin{aligned} 0=\lambda -(\mu +\theta '-\theta )\bar{s}-\theta \bar{m}, \end{aligned}$$

with $\bar{s}\in [0,\min \{1,\frac{\lambda }{\mu +\theta '-\theta }\}]$ and $\bar{m}= (\lambda - \bar{s} (\mu +\theta '-\theta ))/\theta$, hence $\bar{m} \in [ \max (0, (\lambda -(\mu +\theta '-\theta ))/\theta ), \lambda /\theta ]$. The optimal equilibrium point $(m^*,s^*)$ minimizes $EC(\bar{s}, W)$. We first prove that $EC(\bar{s}, W)$ is a convex function in $\bar{s}\in [0,\min \{1,\frac{\lambda }{\mu +\theta '-\theta }\}]$, by checking that $\frac{\mathrm {d}}{\mathrm {d}\bar{s}}\left( \frac{\mathrm {d}EC(\bar{s}, W)}{\mathrm {d}\bar{s}}\right) ~>~0$. After some algebra, we obtain that

$$\begin{aligned}&\frac{\mathrm {d}}{\mathrm {d}\bar{s}}\left( \frac{\mathrm {d}EC(\bar{s}, W)}{\mathrm {d}\bar{s}}\right) \\&=\frac{(\mu +\theta '-\theta )}{\theta }\left( \frac{\mathrm {d}\tilde{C}(\bar{m},0)}{\mathrm {d}\bar{m}}-\frac{\mathrm {d}\tilde{C}(\bar{m},1)}{\mathrm {d}\bar{m}}\right) \\&\quad +\frac{(\mu +\theta '-\theta )}{\theta }\left( \frac{\mathrm {d}^2\tilde{C}(\bar{m},0)}{\mathrm {d}m^2}\frac{(-\lambda +(\mu +\theta '-\theta )+\theta \bar{m})}{\theta }\right. \\&\quad \qquad \left. +\,\frac{\mathrm {d}^2\tilde{C}(\bar{m},1)}{\mathrm {d}m^2}\frac{(\lambda -\theta \bar{m})}{\theta }\right) >0. \end{aligned}$$

The inequality follows from $\bar{m} \in [(\lambda -(\mu +\theta '-\theta ) ) /\theta ,\lambda /\theta ]$, $\mu +\theta '\ge \theta$ and convexity of $C(\cdot ,\cdot )$.

Let us assume from now on that $\min \{1,\lambda /(\mu +\theta '-\theta )\}=1$. The proof when $\min \{1,\lambda /(\mu +\theta '-\theta )\}=\lambda /(\mu +\theta '-\theta )$ follows similarly. Having proved convexity of $EC(\bar{s}, W)$, we can distinguish the following three cases:

• (1) Case 1 $\frac{\mathrm {d}EC(\bar{s}, W)}{\mathrm {d}\bar{s}} \le 0$ for all $\bar{s}\in [0,1]$, hence the optimal equilibrium point satisfies $s^*=1$, $m^*=\lambda /(\mu +\theta '-\theta )$.

• (2) Case 2 $\frac{\mathrm {d}EC(s^*, W)}{\mathrm {d}s^*} = 0$, hence the optimal equilibrium point satisfies $s^*\in [0,1]$, $m^*\in [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$.

• (3) Case 3 $\frac{\mathrm {d}EC(\bar{s}, W)}{\mathrm {d}\bar{s}} \ge 0$ for all $\bar{s}\in [0,1]$, hence the optimal equilibrium point satisfies $s^*=0$, $m^*=\lambda /\theta$.

In the case $\min \{1,\lambda /(\mu +\theta '-\theta )\}=\lambda /(\mu +\theta '-\theta )$, only Case 2 and 3 hold.

Now note that

$$\begin{aligned} \frac{\mathrm {d}EC(\bar{s},W)}{\mathrm {d}\bar{s}}&\ge 0\Leftrightarrow W\ge \tilde{C}(\bar{m},0)+\tilde{C}(\bar{m},1)\nonumber \\&\quad \quad +(1-\bar{s})\frac{\mathrm {d}\bar{m}}{\mathrm {d}\bar{s}}\frac{\mathrm {d} \tilde{C}(\bar{m},0)}{\mathrm {d}\bar{m}}+\bar{s}\frac{\mathrm {d}\bar{m}}{\mathrm {d}\bar{s}}\frac{\mathrm {d}\tilde{C}(\bar{m},1)}{\mathrm {d}\bar{m}}, \end{aligned}$$

(77)

which after substitution of $\bar{s}=\frac{\lambda -\theta \bar{m}}{\mu +\theta '-\theta }$ and the expression for $\mathrm {d}\bar{m}/\mathrm {d}\bar{s}=-\frac{\mu +\theta '-\theta }{\theta }$; gives that (77) is equivalent to

$$\begin{aligned}&W\ge \tilde{C}(\bar{m},0)- \tilde{C}(\bar{m},1)+\frac{(\lambda -\theta \bar{m})\frac{\mathrm {d}\tilde{C}(\bar{m},1)}{\mathrm {d}\bar{m}}-(\lambda -(\mu +\theta '-\theta )-\theta \bar{m})\frac{\mathrm {d}\tilde{C}(\bar{m},0)}{\mathrm {d}\bar{m}})}{\theta }; \end{aligned}$$

that is,

$$\begin{aligned}&W\ge \tilde{C}(\bar{m},0)-\tilde{C}(\bar{m},1)+w^{(2)}(\bar{m}). \end{aligned}$$

Hence, in Case 3 the $W$ is such that $W\ge \tilde{C}(\bar{m},0)-\tilde{C}(\bar{m},1)+w^{(2)}(\bar{m})$ for all $\bar{m}\in [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$, and in particular $W\ge w(\lambda /\theta )$.

Similarly, being in Case 1 implies $W\le w(\lambda /(\mu +\theta '-\theta ))$.

In Case 2, from $\mathrm {d}EC(s^*,W)/\mathrm {d}s^*=0$ we obtain, $W=\tilde{C}(m^*,0)- \tilde{C}(m^*,1)+w^{(2)}(m^*)=w(m^*)$, for $m^*\in [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$, since $EC^*(W)=(1-s^*)(\tilde{C}(m^*,0)-W) +s^*\tilde{C}(m^*,1)$, $s^*=(\lambda -\theta m^*)/(\mu +\theta '-\theta )$ and $\mathrm {d}m^*/\mathrm {d}s^*=-(\mu +\theta '-\theta )/\theta$. The function $w(m)$ being non-decreasing in particular implies that in Case 2, $W$ is such that $w(\lambda /(\mu +\theta '-\theta ))\le W\le w(\lambda /\theta )$.

The objective is to find the control $u$ that minimizes the total bias cost; that is, the cost and subsidy obtained over time minus the optimal cost in equilibrium, denoted as

$$\begin{aligned}&J^u(m(0),W):= \int _0^{\infty } \big ( \tilde{C}( m(t),s^u(t))-W(1-s^u(t)) - EC^*(W)\big )\hbox {d}t. \end{aligned}$$

(78)

We define $J(m(0),W)=\min _u J^u(m(0),W)$. The theory of optimal control shows that a sufficient condition in order for a control to be bias optimal is that it solves the Hamilton–Jacobi–Bellman (HJB) equation, [34]:

$$\begin{aligned} 0=\min \{\mathcal {J}_0(m,W),\mathcal {J}_1(m,W)\}, \hbox { for all } m, \end{aligned}$$

(79)

where

$$\begin{aligned} \mathcal {J}_0(m,W)&=\tilde{C}(m,0)-W-EC^*(W)+(\lambda -\theta m)\frac{\partial J(m,W)}{\partial m}, \end{aligned}$$

(80)

$$\begin{aligned} \mathcal {J}_1(m,W)&=\tilde{C}(m,1)-EC^*(W)+(\lambda -(\mu +\theta '-\theta )-\theta m)\frac{\partial J(m,W)}{\partial m}, \end{aligned}$$

(81)

and the function $J(m,W)$ is continuous and differentiable. The reader is referred to [12] for a derivation of the HJB. For a given $W$, we consider the policy that prescribes being passive, $s(t)=0$, in all states $m$ for which $W\ge w(m)$, and active, $s(t)=1$, in all states $m$ for which $W<w(m)$. Observe that due to $w(m)$ being non-decreasing, this will be a threshold policy. That is, there exists $n(W)\in \mathbb {Z}_+$ for which $W> w(m)$ for all $m\le n(W)$ and $W\le w(m)$ if $m\ge n(W)$. We refer to this policy as threshold policy $n(W)$. We want to prove that the policy $n(W)$ satisfies the HJB. To do so let us define $J^{n(W)}(m,W)$ for a given $W$ as the cost under policy $n(W)$, starting at state $m$ and up to equilibrium; that is,

$$\begin{aligned} J^{n(W)}(m,W)&=\int _0^{t_0}\tilde{C}(m^{n(W)}(t),s_0)-W(1-s_0)-EC^*(W)\mathrm {d}t\nonumber \\&\quad +\int _{t_0}^\infty \tilde{C}(m^{n(W)}(t),s_1)-W(1-s_1)-EC^*(W)\mathrm {d}t, \end{aligned}$$

(82)

where $s_0=s(0)$, $s_1=1-s_0$, and $t_0\ge 0$, the time at which threshold $n(W)$ is reached. Note that $s_0=0$ if $m(0)=m\le n(W)$ and $s_0=1$ otherwise. The function $J^{n(W)}(m,W)$ can be written as the sum of two terms, the first term corresponding to the phase from the starting point $m$ up to the time the threshold is reached, $t_0$. In this phase the control equals $s_0$. Once the threshold is reached, a switch in the control happens and therefore the second term corresponds to the phase from the switch time $t_0$ until the equilibrium is reached. In this phase the control equals $s_1$. This is due to threshold policies having at most one switch in the control.

Let us assume $m(0)=m\le n(W)$, which implies $s_0=0$ and $s_1=1$, then from (82)

$$\begin{aligned} \frac{\mathrm {d}J^{n(W)}(m,W)}{\mathrm {d}m}&=\frac{\mathrm {d}t_0}{\mathrm {d}m}\biggl (\tilde{C}(n(W),0)-W-EC^*(W)-\tilde{C}(n(W),1)+EC^*(W)\biggr )\nonumber \\&\quad +\int _0^{t_0}\frac{\mathrm {d}\tilde{C}\left( m^{n(W)}(t),0\right) }{\mathrm {d}t}\frac{\mathrm {d}t}{\mathrm {d}m}\mathrm {d}t+\int _{t_0}^\infty \frac{\mathrm {d}\tilde{C}\left( m^{n(W)}(t),1\right) }{\mathrm {d}t}\frac{\mathrm {d}t}{\mathrm {d}m}\mathrm {d}t. \end{aligned}$$

(83)

Policy $n(W)$ implies $\mathrm {d}m^{n(W)}(t)/\mathrm {d}t=\lambda -\theta m^{n(W)}(t)$ for all $t\in [0,t_0]$. Then,

$$\begin{aligned}&\frac{\mathrm {d}m^{n(W)}(t)}{\mathrm {d}t}=\lambda -\theta m^{n(W)}(t)\Rightarrow m^{n(W)}(t)=\left( m-\frac{\lambda }{\theta }\right) \mathrm {e}^{-\theta t}+\frac{\lambda }{\theta }\\&\quad \Rightarrow t_0=-\frac{1}{\theta }\log \left( \frac{n(W)-\lambda /\theta }{m-\lambda /\theta }\right) \Rightarrow \frac{\mathrm {d}t_0}{\mathrm {d}m}=\frac{1}{\theta m-\lambda }, \end{aligned}$$

and $\mathrm {d}t/\mathrm {d}m=1/f^0(m)$. Substituting the latter in Eq. (83), we obtain for all $m\le n(W)$

$$\begin{aligned} \frac{\partial J^{n(W)}(m,W)}{\partial m}=\frac{\tilde{C}(m,0)-W-EC^*(W)}{\theta m-\lambda }, \end{aligned}$$

and similarly for all $m>n(W)$

$$\begin{aligned} \frac{\partial J^{n(W)}(m,W)}{\partial m}=\frac{\tilde{C}(m,1)-EC^*(W)}{\mu +\theta '-\theta +\theta m-\lambda }. \end{aligned}$$

For all $m\le n(W)$, the action under policy $n(W)$ is to keep the bandit passive. In addition, when substituting $\frac{\partial J^{n(W)}(m,W)}{\partial m}$ in (80), we obtain $\mathcal {J}_0(m,W)=0$. In order for the threshold policy $n(W)$ to satisfy the HJB in (79), we therefore need to prove that $\mathcal {J}_1(m,W)\ge 0$. Substituting $\frac{\partial J^{n(W)}(m,W)}{\partial m}$ in (81) we obtain that this is equivalent to

$$\begin{aligned}&\mathcal {J}_1(m,W)\ge 0\nonumber \\&\Leftrightarrow W\ge \tilde{C}(m,0)-\tilde{C}(m,1)+\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta -\theta m}(\tilde{C}(m,1)-EC^*(W)), \end{aligned}$$

(84)

for all $m\notin [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$ with $m\le n(W)$, and

$$\begin{aligned}&\mathcal {J}_1(m,W)\ge 0\nonumber \\&\Leftrightarrow W\le \tilde{C}(m,0)-\tilde{C}(m,1)+\frac{-(\mu +\theta '-\theta ))}{\lambda -\mu -\theta '+\theta -\theta m}(\tilde{C}(m,1)-EC^*(W)), \end{aligned}$$

(85)

for all $m\in [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$ with $m\le n(W)$. If (84) is satisfied for $m\notin [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }]$ and (85) for $m\in [\frac{\lambda }{\mu +\theta '-\theta },\frac{\lambda }{\theta }],$ then the action under policy $n(W)$ is to keep the bandit passive.

Assume now $m>n(W)$. Hence, the action under policy $n(W)$ is to keep the bandit active. Substituting $\frac{\partial J^{n(W)}(m,W)}{\partial m}=\frac{EC^*(W)-\tilde{C}(m,1)}{\lambda -\mu -\theta '+\theta -\theta m}$ in (81) we then obtain $\mathcal {J}_1(m,W)=0$. In order for the threshold policy $n(W)$ to satisfy the HJB in (79), we need therefore to prove that $\mathcal {J}_0(m,W)\ge 0$. Substituting $\frac{\partial J^{n(W)}(m,W)}{\partial m}=\frac{EC^*(W)-\tilde{C}(m,1)}{\lambda -\mu -\theta '+\theta -\theta m}$ in (80), this is equivalent to

$$\begin{aligned}&\mathcal {J}_0(m,W)\ge 0\nonumber \\&\Leftrightarrow W\le \tilde{C}(m,0)-\tilde{C}(m,1)+\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta -\theta m}(\tilde{C}(m,1)-EC^*(W)), \end{aligned}$$

(86)

for all $m>n(W)$. If (86) is satisfied in $m>n(W)$ then the action under policy $n(W)$ is to keep the bandit active.

Hence, if conditions (84)–(86) are satisfied, then threshold policy $n(W)$ is optimal. It remains to be proved that conditions (84)–(86) are satisfied. This will be done in the remainder of the proof for the three different cases.

Let us first assume that $m^*=\lambda /(\mu +\theta '-\theta )$ and $W\le w(\lambda /(\mu +\theta '-\theta ))$, that is, Case 1, then $EC^*(W)=\tilde{C}(\frac{\lambda }{\mu +\theta '-\theta },1)$. Recall that threshold policy $n(W)$ implies that $W\ge w(m)$ for all $m\le n(W)$ and $W\le w(m)$ if $m\ge n(W)$. Hence, $W\le w(\lambda /(\mu +\theta '-\theta ))$ and $w(m)$ being non-decreasing imply that $n(W)\le \lambda /(\mu +\theta '-\theta )$. Conditions (84)–(86) reduce then to the following: the HJB is satisfied if and only if $W\ge (\le ) \tilde{C}(m,0)-\tilde{C}(m,1)+w^{(1)}(m)$ for all $m\le (\ge )n(W)$. This is equivalent to $W\ge (\le ) w(m)$ for all $m\le (\ge )n(W)$, since $w^{(1)}(m)$ is non-decreasing and $W\le w(\lambda /(\mu +\theta '-\theta ))$. Hence, in Case 1 the threshold policy $n(W)$ satisfies the HJB and is hence optimal.

Similarly, if $m^*=\lambda /\theta$ and $W\ge w(\lambda /\theta )$, that is, Case 3, then $EC^*(W)=\tilde{C}(\lambda /\theta ,0)-W$. Since under threshold policy $n(W)$, $W\ge w(m)$ for all $m\le n(W)$ and $W\le w(m)$ if $m\ge n(W)$, $w(m)$ being non-decreasing implies $n(W)\ge \lambda /\theta$. Using $EC^*(W)=\tilde{C}(\lambda /\theta ,0)-W$, we obtain that conditions (84)–(86) simplify to $W\ge (\le ) \tilde{C}(m,0)-\tilde{C}(m,1)+w^{(3)}(m)$, for all $m\le (\ge )n(W)$. This is equivalent to $W\ge (\le ) w(m)$ for all $m\le (\ge )n(W)$, due to $w^{(3)}(m)$ being non-decreasing and $W\ge w(\lambda /\theta )$. Hence, in Case 3, threshold policy $n(W)$ satisfies the HJB and is hence optimal.

We are left with Case 2 in which $W$ is such that $\frac{\mathrm {d}E(s^*,W)}{\mathrm {d}s^*}=0$, and $s^*\in [0,1]$, that is, $w(\lambda /(\mu +\theta '-\theta ))\le W\le w(\lambda /\theta )$. In addition $W=w(m^*)$, hence $n(W)=m^*$, by definition of $n(W)$. In this setting we have that

$$\begin{aligned} EC^*(W)=(1-s^*)\left( \tilde{C}(m^*,0)-W\right) +s^*\tilde{C}(m^*,1). \end{aligned}$$

Substituting the latter in Conditions (84) and (86) the conditions simplify to

$$\begin{aligned}&W\ge (\le ) \tilde{C}(m,0)-\tilde{C}(m,1)\nonumber \\&\quad +\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta -\theta m}\left( \tilde{C}(m,1)-(1-s^*)\left( \tilde{C}(m^*,0)-W\right) -s^*\tilde{C}(m^*,1)\right) , \end{aligned}$$

(87)

for all $m\le \lambda /(\mu +\theta '-\theta )$( $m\ge \lambda /\theta$).

Condition (85) and (86) reduce to

$$\begin{aligned}&W\le \tilde{C}(m,0)-\tilde{C}(m,1)\nonumber \\&\quad +\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta -\theta m}\left( \tilde{C}(m,1)-(1-s^*)\left( \tilde{C}(m^*,0)-W\right) -s^*\tilde{C}(m^*,1)\right) , \end{aligned}$$

(88)

for all $m\in [\frac{\lambda }{\mu +\theta '-\theta },m^*]$ and

$$\begin{aligned}&W\le \tilde{C}(m,0)-\tilde{C}(m,1)\nonumber \\&+\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta -\theta m}\left( \tilde{C}(m,1)-(1-s^*)(\tilde{C}(m^*,0)-W)-s^*\tilde{C}(m^*,1)\right) , \end{aligned}$$

(89)

for all $m\in [m^*,\lambda /\theta ]$.

Taking into account that $\lambda -\mu -\theta '+\theta -\theta m\ge 0$, for all $m< \lambda /(\mu +\theta '-\theta )$, and $\lambda -\mu -\theta '+\theta -\theta m\le 0$ otherwise, and that $\lambda -s^*(\mu +\theta '-\theta )-\theta m\ge 0$, for all $m\le m^*$, and $\lambda -s^*(\mu +\theta '-\theta )-\theta m\le 0$ otherwise, Conditions (87)–(89) reduce to the following:

$$\begin{aligned} W\ge (\le )&\bigg (\tilde{C}(m,0)-\tilde{C}(m,1)+\frac{-(\mu +\theta '-\theta )}{\lambda -\mu -\theta '+\theta +\theta m}\nonumber \\&\cdot \left( \tilde{C}(m,1)-(1-s^*)\tilde{C}(m^*,0)-s^*\tilde{C}(m^*,1)\right) \bigg )\nonumber \\&\cdot \frac{\lambda -\mu -\theta '+\theta -\theta m}{\lambda -s^*(\mu +\theta '-\theta )-\theta m}, \end{aligned}$$

(90)

for all $m\le m^*(m\ge m^*)$. After some algebra, the latter gives

$$\begin{aligned} W&\ge (\le )\tilde{C}(m,0)-\tilde{C}(m,1)+\frac{(\lambda -\theta m^*)}{\theta }\frac{(\tilde{C}(m,1)-\tilde{C}(m^*,1))}{m-m^*}\nonumber \\&\quad +\frac{(\mu +\theta '+\theta (m^*-1)-\lambda )}{\theta }\frac{(\tilde{C}(m^*,0)-\tilde{C}(m,0))}{m^*-m}, \end{aligned}$$

(91)

for all $m\le (\ge )m^*$. Since $w^{(2)}(\cdot )$ and $w(\cdot )$ are non-decreasing, in order to prove (90) it therefore suffices to prove that the RHS in (91) is a non-decreasing function and that RHS in  (91)$\xrightarrow {m\rightarrow m^*}C(m^*,0)-C(m^*,1)+w^{(2)}(m^*)$. Let us denote the RHS in (91) by $\tilde{W}(m)$. Convexity of $\tilde{C}(\cdot ,\cdot )$ and $\tilde{C}(\cdot ,\cdot )$ being non-decreasing imply $\tilde{W}(m)$ is non-decreasing. Now note that

$$\begin{aligned}&\lim _{m\rightarrow m^*}\tilde{W}(m)\nonumber \\&=C(m^*,0)-C(m^*,1)+d(\mu +\theta ')-d'\theta '\nonumber \\&\quad +\frac{(\mu +\theta '-\theta (m^*-1)-\lambda )\frac{\mathrm {d}C(m^*,0)}{\mathrm {d}m^*}}{\theta }\frac{(\lambda -\theta m^*)\frac{\mathrm {d}C(m^*,1)}{\mathrm {d}m^*}}{\theta }\nonumber \\&=C(m^*,0)-C(m^*,1)+d(\mu +\theta ')-d'\theta '+w^{(2)}(m^*)=W. \end{aligned}$$

(92)

Hence, for all $m\le (\ge )m^*$, we have $\tilde{W}(m)\le (\ge )C(m^*,0)-C(m^*,1)+d(\mu +\theta ')-d'\theta '+w^{(2)}(m^*)$. In other words,threshold policy $n(W)=m^*$ satisfies the HJB and is hence optimal.

Appendix 12: Proof of Proposition 9

We drop the dependency on $k$ throughout the proof.

As $n\rightarrow \infty$, then the fluid index is given by $w(n)=C(n,0)-C(n,1)+d(\mu +\theta ')-d'\theta '+w^{(3)}(n)$. We have assumed that $C(n,a)$, $a=0,1$, are upper bounded by a polynomial of degree $P$. Therefore, we can write $C(n,a)=E(n,a)+o(1)$ for large values of $n$, where $E(n,1)=\sum _{i=0}^{P}C^{(P,i)}n^i$, with

$$\begin{aligned} C^{(P,i)}:=\lim _{n\rightarrow \infty }\frac{C(n,1)-\sum _{j=i+1}^{P}C^{(P,j)}n^j}{n^{i}}, \end{aligned}$$

and $E(n,0)=\sum _{i=0}^{Q}E^{(Q,i)}n^{i}$, with

$$\begin{aligned} E^{(Q,i)}:=\lim _{n\rightarrow \infty }\frac{C(n,0)-\sum _{j=i+1}^{Q}E^{(Q,j)}n^j}{n^{i}}, \end{aligned}$$

Then, as $n\rightarrow \infty$, $w(n)=w^\infty (n)+o(1)$, where $w^\infty (n)=d(\mu +\theta ')-d'\theta '$ $+w^c(n)+o(1)$, and

$$\begin{aligned} w^c(n)=E(n,0)-E(n,1)+\frac{(\mu +\theta '-\theta )}{\theta }\frac{\left( E(n,0)-E(\lambda /\theta ,0)\right) }{n-\lambda /\theta }. \end{aligned}$$

Note that $(E(n,0)-E(\lambda /\theta ,0))/(n-\lambda /\theta )$ for large values of $n$ can equivalently be written as

$$\begin{aligned}&\frac{\sum _{i=0}^{Q}E^{(Q,i)}n^i-\sum _{i=0}^{Q}E^{(Q,i)}(\lambda /\theta )^i}{n-\lambda /\theta }\nonumber \\&\quad =\sum _{i=0}^{Q}E^{(Q,i)}\frac{(n^i-(\lambda /\theta )^i)}{n-\lambda /\theta }=\sum _{i=2}^{Q}E^{(Q,i)}\left( \sum _{j=0}^i\left( \frac{\lambda }{\theta }\right) ^jn^{i-1-j}\right) \nonumber \\&\quad =\frac{E(n,0)}{n}+\frac{E^{(Q,1)}\left( \frac{\lambda }{\theta }\right) +E^{(Q,2)}\left( \frac{\lambda }{\theta }\right) ^2+\ldots +E^{(Q,Q)}\left( \frac{\lambda }{\theta }\right) ^{Q}}{n}\nonumber \\&\quad \quad + \sum _{i=2}^{Q} E^{(Q,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}\nonumber \\&\quad =\frac{E(n,0)}{n}+\sum _{i=2}^{Q} E^{(Q,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}+o(1). \end{aligned}$$

(93)

We then compute $\lim _{n\rightarrow \infty }W(n)/w(n)$, which by the result in (19) is equivalent to

$$\begin{aligned}&\lim _{n\rightarrow \infty }\frac{W(n)}{w(n)}=\lim _{n\rightarrow \infty }\frac{W^\infty (n)+o(1)}{w^\infty (n)+o(1)}=\lim _{n\rightarrow \infty }\frac{d(\mu +\theta ')-d'\theta '+W^c(n)+o(1)}{d(\mu +\theta ')-d'\theta '+w^c(n)+o(1)}\\&\quad =\lim _{n\rightarrow \infty }\frac{E(n,0)-E(n,1)+\frac{(\mu +\theta '-\theta )}{\theta }\left( \frac{E(n,0)}{n}+\sum _{i=2}^{P} C^{(P,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}\right) +\mathcal {O}(1)}{E(n,0)-E(n,1)+\frac{(\mu +\theta '-\theta )}{\theta }\left( \frac{E(n,0)}{n}+\sum _{i=2}^{Q} E^{(Q,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}\right) +\mathcal {O}(1)}\\&\quad =1+o(1),\end{aligned}$$

which follows from the fact that both in the denominator and numerator the highest term comes from $E(n,0)-E(n,1)+\frac{(\mu +\theta '-\theta )}{\theta }\frac{E(n,0)}{n}$. This concludes the proof for the expression in (27).

Let us now obtain the expression in (28) with the extra assumptions $P=Q$ and $C^{(P,i)}=E^{(P,i)}$ for all $i\in \{2,\ldots ,P\}$. Observe that under this assumption we obtain from (93) that $(E(n,0)-E(\lambda /\theta ,0))/(n-\lambda /\theta )$, for large values of $n$, can be written as

$$\begin{aligned} \frac{E(n,0)}{n}+\sum _{i=2}^{P} C^{(P,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}+o(1), \end{aligned}$$

and hence,

$$\begin{aligned} w^\infty (n)= & {} d_k(\mu +\theta ')-d'\theta '+E(n,0)-E(n,1)\\&+\;\frac{(\mu +\theta '-\theta )}{\theta }\bigg (\frac{E(n,0)}{n}+\sum _{i=2}^{P} C^{(P,i)} \sum _{j=0}^{i-2} n^{i-2-j} \left( \frac{\lambda }{\theta }\right) ^{j+1}\bigg )+o(1). \end{aligned}$$

Then, by the result in (19) we have $W^\infty (n)=w^\infty (n)+o(1),$ and hence $W(n)$ $=w(n)+o(1)$ for large values of $n$ which concludes the proof of (28).