On the computation of Whittle’s index for Markovian restless bandits

Abstract

The multi-armed restless bandit framework allows to model a wide variety of decision-making problems in areas as diverse as industrial engineering, computer communication, operations research, financial engineering, communication networks etc. In a seminal work, Whittle developed a methodology to derive well-performing (Whittle’s) index policies that are obtained by solving a relaxed version of the original problem. However, the computation of Whittle’s index itself is a difficult problem and hence researchers focused on calculating Whittle’s index numerically or with a problem dependent approach. In our main contribution we derive an analytical expression for Whittle’s index for any Markovian bandit with both finite and infinite transition rates. We derive sufficient conditions for the optimal solution of the relaxed problem to be of threshold type, and obtain conditions for the bandit to be indexable, a property assuring the existence of Whittle’s index. Our solution approach provides a unifying expression for Whittle’s index, which we highlight by retrieving known indices from literature as particular cases. The applicability of finite rates is illustrated with the machine repairmen problem, and that of infinite rates by an example of communication networks where transmission rates react instantaneously to packet losses.

Appendices

A Proof of Propositions

In this section we provide the proofs of different propositions. For ease of notation, we removed the subscript k from all the proofs in this section.

1.1 A.1 Proof of Proposition 1

Proof

Since ${\mathcal {U}}_{REL}$ is non-empty, there exists a stationary optimal policy $\phi ^*$ that optimally solves the subproblem (6) for a bandit. Define $n^*= \min \{ m\in \{ 0,1,\ldots \}: S^{\phi ^*}(m)=1 \}$. This implies $S^{\phi ^*}(m)=0~\forall ~m<n^*$ and $S^{\phi ^*}(n^*)=1$. From the structure on the transition rates and jump probabilities in (i) of Proposition 1, we have $q_k^1(N, N+i) = 0,~\forall ~i\ge 1$, $\ q_k^0(N, N+i) = 0,~\forall ~i\ge 2$, and $p_k^a(N, N+i) =0,~\forall ~i\ge 1, a =0, 1$. The above transition structure ensures that all the states $m> n^*$ are transient. Hence $\pi ^{\phi ^*}(m)=0 ~\forall ~ m >n^*$. Thus, the following holds under the optimal policy $\phi ^*$:

$$\begin{aligned} {\mathbb {E}}(C(N^{\phi ^*}, S^\phi (\vec {N}^{\phi ^*})))= & {} \sum _{m=0}^{n^*-1}C(m,0)\pi ^{\phi ^*}(m) + C(n^*,1)\pi ^{\phi ^*}(n^*),\\ {\mathbb {E}}(f(N^{\phi ^*}, S^\phi (\vec {N}^{\phi ^*})))= & {} \sum _{m=0}^{n^*-1}f(m,0)\pi ^{\phi ^*}(m) + f(n^*,1)\pi ^{\phi ^*}(n^*), \end{aligned}$$

and lump-sum cost under the optimal policy $\phi ^*$ is given by:

$$\begin{aligned} {\mathbb {E}}(C^{\infty , \phi ^*}({N}^{\phi ^*}, S^{\phi ^*}(\vec {N}^{\phi ^*}))= & {} \sum _{{\tilde{n}}}\sum _{m}{\mathbb {E}}\left( q^{S^{\phi ^*}(\vec {N}^{\phi ^*})}(N^{\phi ^*}, {\tilde{n}}) \times {\mathcal {I}}({\tilde{n}}, S^{\phi ^*}(\vec {M}^{\phi ^*}(\vec {N}^{\phi ^*}, {\tilde{n}})))\right. \\&\left. \times { p^{S^{\phi ^*}(\vec {M}^{\phi ^*}(\vec {N}^{\phi ^*}, {\tilde{n}})}({\tilde{n}},m) \times L^\infty ({\tilde{n}}, m, S^{\phi ^*}(\vec {M}^{\phi ^*}(\vec {N}^{\phi ^*}, {\tilde{n}})) }\right) , \end{aligned}$$

From Markov chain theory, the average number of times state y is visited in the next decision epoch under action a given the current state x can be written as:

$$\begin{aligned} \lim _{N \rightarrow \infty }\frac{1}{N} \sum _{n=1}^N 1^a_{\{X_n=x,~ X_{n+1} = y\}}=\pi (x)q^a(x,y). \end{aligned}$$

Given that the parameters satisfy (i) of Proposition 1, the lump-sum cost can equivalently be written as;

$$\begin{aligned}&{\mathbb {E}}(C^{\infty , \phi ^*}({N}^{\phi ^*}, S^{\phi ^*}(\vec {N}^{\phi ^*}))\\&\quad = \sum _{n=0}^{n^*-1}\sum _{m=0}^n\left( p^0(n,m) L^\infty (n, m, 0)\left[ \sum _{k=0, k \ne n}^{n^* -1}q^0(k,n)\pi ^{\phi ^*}(k) + q^1(n^*,n)\pi ^{\phi ^*}(n^*)\right] \right) \\&\qquad + \sum _{m=0}^{n^*}\left( p^1(n^*,m) L^\infty (n^*, m, 1)\left[ \sum _{k=0}^{n^* -1}q^0(k,n^*)\pi ^{\phi ^*}(k)\right] \right) . \end{aligned}$$

In the above expected lump-sum cost, the first term is the contribution in cost due to transition from states $0,1,2, \cdots , n^*-1$ and the second term is that for the transition from state $n^*$. Additionally, it exploits the fact that $\pi ^{\phi ^*}(m)=0 ~\forall ~ m >n^*$. It follows from the expressions of the above expected costs that the long run average cost under the optimal policy $\phi ^*$,

$$\begin{aligned} {\mathbb {E}}(C(N^{\phi ^*}, S^{\phi ^*}(N^{\phi ^*})))~+{\mathbb {E}}(C^{\infty , \phi ^*}(N^{\phi ^*}, S^{\phi ^*}(N^{\phi ^*}))) - W {\mathbb {E}}(f(N^{\phi ^*}, S^{\phi ^*}(N^{\phi ^*}))), \end{aligned}$$

is the same as the long run average cost under a 0–1 type threshold policy with threshold $n^*$,

$$\begin{aligned} {\mathbb {E}}(C(N^{n^*}, S^{n^*}(N^{n^*})))~+{\mathbb {E}}(C^{\infty , n^*}(N^{n^*}, S^{n^*}(N^{n^*}))) - W {\mathbb {E}}(f(N^{n^*}, S^{n^*}(N^{n^*}))). \end{aligned}$$

Thus, a 0–1 type of threshold policy with threshold $n^*$ is optimal when (i) is satisfied. The alternate rates (ii) can be proven to result in 0–1 type threshold optimality along the similar lines by considering the set $\max \{ m\in \{ 0,1,\ldots \}: S^{\phi ^*}(m)=0 \}$. $\square$

1.2 A.2 Proof of Proposition 2

Proof

We will focus on 0–1 type of threshold policies throughout the proof. The case of threshold policies of type 1–0 can be proven similarly. Since an optimal solution of problem (6) is of threshold type for a given subsidy W, the optimal average cost will be $g(W) := \min \nolimits _n{g^{(n)}(W)}$ where

$$\begin{aligned} g^{(n)}(W) = {\mathbb {E}}(T(N^n, S^n(N^n))) - W {\mathbb {E}}(f(N^n, S^n(N^n))). \end{aligned}$$

We denote the minimizer of g(W) by n(W). Note that the function g(W) is a lower envelope of affine non-increasing functions of W due to the non-negative nature of ${\mathbb {E}}(f(\cdot ))$. It thus follows that g(W) is a concave non-increasing function.

It follows that the right derivative of g(W) in W is given by $-{\mathbb {E}}(f(N^{n(W)}, S^{n(W)}(N^{n(W)})))$. Since g(W) is concave in W, the right derivative is non-increasing in W. Together with the fact ${\mathbb {E}}(f(N^n, S^n(N^n)))$ is strictly increasing in n, it hence follows that n(W) is non-decreasing in W. Since an optimal policy is of 0–1 threshold type, the set of states where it is optimal to be passive can be written as $D(W) = \{m: m \le n(W)\}$. Since n(W) is non-decreasing, by definition this implies that bandit k is indexable. $\square$

1.3 A.3 Proof of Proposition 3

We will focus on 0–1 type of threshold policies throughout the proof. Let ${\tilde{W}}(n)$ be the value for subsidy such that the average cost under threshold policy n is equal to that under threshold policy $n-1$. By using (6), we have ${\mathbb {E}}(T(N^n, S^n(N^n))) - {\tilde{W}}(n) {\mathbb {E}}(f(N^n, S^n(N^n))) = {\mathbb {E}}(T(N^{n-1}, S^{n-1}(N^{n-1}))) - {\tilde{W}}(n) {\mathbb {E}}(f(N^{n-1}, S^{n-1}(N^{n-1})))$. Hence, ${\tilde{W}}(n)$ is given by,

$$\begin{aligned} \frac{{\mathbb {E}}(T^n(N^n, S^n({N}^n))) - {\mathbb {E}}(T^{n-1}(N^{{n-1}}, S^{{n-1}}({N}^{{n-1}})))}{{\mathbb {E}}(f(N^{n}, S^n({N}^{n}))) - {\mathbb {E}}(f(N^{{n-1}}, S^{{n-1}}({N}^{{n-1}})))}, \end{aligned}$$

which is the same as (7). Since ${\tilde{W}}(n)$ is monotone, it can be verified by exploiting threshold optimality that $g({\tilde{W}}(n)) = g^{(n)}({\tilde{W}}(n)) = g^{(n-1)}({\tilde{W}}(n))$. Similarly, $g({\tilde{W}}(n-1)) = g^{(n-1)}({\tilde{W}}(n-1)) = g^{(n-2)}({\tilde{W}}(n-1))$. Further, monotonicity of ${\tilde{W}}(n)$ implies the following two possibilities:

• 1. Non-decreasing nature, i.e., ${\tilde{W}}(n-1)\le {\tilde{W}}(n)$

• 2. Non-increasing nature, i.e., ${\tilde{W}}(n-1)\ge {\tilde{W}}(n)$

But ${\tilde{W}}(n-1)\ge {\tilde{W}}(n)$ results in a contradiction from indexability and 0–1 type of threshold optimality. Thus, ${\tilde{W}}(n)$ has to be non-decreasing, i.e., ${\tilde{W}}(n-1)\le {\tilde{W}}(n)$.

It follows from indexability and 0–1 type of threshold optimality that for all $W \le {\tilde{W}}(n)$, the set of states where it is optimal to be passive, D(W), satisfies $D(W) \subseteq \{m:m\le n-1\}$. Again from indexability in a similar way, $D(W) \supseteq {\{m:m\le n-1\}}$ for all $W \ge {\tilde{W}}(n-1)$. Thus, for ${\tilde{W}}(n-1) \le W \le {\tilde{W}}(n)$, $\{m:m\le n-1\} \subseteq D(W) \subseteq \{m:m\le n-1\}$ which implies that threshold policy $n-1$ is optimal for all ${\tilde{W}}(n-1) \le W \le {\tilde{W}}(n)$ and hence $g(W) = g^{(n-1)}(W)$ for ${\tilde{W}}(n-1) \le W \le {\tilde{W}}(n)$. Hence, ${{\tilde{W}}}(n)$ is the smallest value of the subsidy such that activating the bandit in state n becomes optimal, that is, Whittle’s index is given by $W(n) = {\tilde{W}}(n)$.

B Machine repairman problem

In this section, we provide the details to obtain the stationary distribution, prove indexability and derive Whittle’s index for two specific models of the machine repairman problem of Sect. 5.

1.1 B.1 Stationary distribution

In this section, we determine the stationary distribution under a 0–1 type of threshold policy n. Thus, action $a=0$ is taken in states $0,1,2,\cdots , n$ and action $a=1$ in states $n+1, n+2,\ldots$

The transition diagram for the evolution of Markov chain is shown in Fig. 2.

Fig. 2

Transition diagram under the threshold policy n for machine repairman problem

The balance equations for the stationary distribution under the threshold policy n are given by

$$\begin{aligned} \lambda (0)\pi ^n(0)= & {} {\psi (1)\pi ^n(1) + \psi (2)\pi ^n(2) + \cdots + \psi (n)\pi ^n(n)+ r(n+1)\pi ^n(n+1)}, \nonumber \\ \lambda (m)\pi ^n(m)= & {} (\lambda (m+1) + \psi (m+1))\pi ^n(m+1)~\text { for }m=0,~1,~2,\ldots ,~n-1,\nonumber \\ \lambda (n)\pi ^n(n)= & {} r(n+1)\pi ^n({n+1}). \end{aligned}$$

(15)

Using $\sum \nolimits _{m=1}^{n+1}\pi ^n(m) = 1$, one obtains

$$\begin{aligned} \pi _k^{n_k}(m_k)= & {} \frac{P_{m_k}}{\lambda _k(m_k)\left( \sum \nolimits _{i=0}^{n_k}\frac{P_i}{\lambda _k(i)} +\frac{P_{n_k}}{r_k(n_k+1)}\right) }~\forall ~m_k=0,1,2,\ldots n_k,\nonumber \\ \pi _k^{n_k}{(n_k+1)}= & {} \frac{P_{n_k}}{r_k(n_k+1)\left( \sum \nolimits _{i=0}^{n_k}\frac{P_i}{\lambda _k(i)} +\frac{P_{n_k}}{r_k(n_k+1)}\right) }, \nonumber \\ \pi _k^{n_k}(m_k)= & {} 0 ~\forall ~m_k = n_k+2, \cdots \end{aligned}$$

(16)

where $P_i = \prod \nolimits _{j=1}^ip_k(j)$ and $p_k(j) = \frac{\lambda _k(j)}{\lambda _k(j) + \psi _k(j)}$; $P_0 = 1$.

1.2 B.2 Indexability

Lemma B. 1

Machine k is indexable if the repair rates are non-decreasing in their state, i.e., $r_k(n) \le r_k(n+1)~\forall ~n$, and $r_k(1)>0$. In particular, all machines are indexable for state-independent repair rates.

Proof

From Proposition 3, it follows that machine k is indexable if ${\mathbb {E}}(f_k(N_k^{n}, S_k^{n}({N}_k^{n})))$ is strictly increasing in n. Recall that $f_k(n, a) = {\mathbf {1}}_{\{ a = 0\}}$. Under the 0–1 type of threshold structure policy, with threshold n, we have

$$\begin{aligned} {\mathbb {E}}(f_k(N_k^{n}, S_k^{n}({N}_k^{n}))) =\sum \limits _{m=0}^{n}\pi _k^{n}(m). \end{aligned}$$

Thus, machine k is indexable if $\sum \nolimits _{m=0}^{n}\pi _k^{n}(m)$ is strictly increasing in n. Since $\pi _k^{n}(m)=0$ for $m>n+1$, this is equivalent to proving that $\pi _k^{n}(n+1)$ is strictly decreasing in n. From Eq. (16) and some algebra, we obtain that

$$\begin{aligned} \pi _k^{n}(n+1) - \pi _k^{n-1}(n) = \frac{\left( \frac{\lambda _k(n)(r_k(n) -r_k(n+1))-\psi _k(n)r_k(n+1)}{\lambda _k(n) + \psi _k(n)}\right) \sum \nolimits _{i=0}^{n-1}\frac{P_i}{\lambda _k(i)} - r_k(n+1)\frac{ P_{n}}{\lambda _k(n)}}{r_k(n)r_k(n+1)\left( \sum \nolimits _{i=0}^{n}\frac{P_i}{\lambda _k(i)} +\frac{P_{n}}{r_k(n+1)}\right) \left( \sum \nolimits _{i=0}^{n+1}\frac{P_i}{\lambda _k(i)} +\frac{P_{n+1}}{r_k(n+2)}\right) }. \end{aligned}$$

Note that the denominator is strictly positive. Since $r_k(n)$ is non-decreasing and $r_k(1)>0$, the numerator is strictly negative. That is, the result follows. $\square$

1.3 B.3 Whittle’s index: Proof of Proposition 4

Since a 0–1 type of threshold policy is optimal, using Proposition 3, the Whittle index is given by Eq. (7), i.e.,

$$\begin{aligned} W_k(n) = \frac{{\mathbb {E}}(C_k(N_k^n, S_k^n(N_k^n))) - {\mathbb {E}}(C_k(N_k^{n-1}, S_k^{n-1}(N_k^{n-1})))}{\sum \nolimits _{m=0}^n\pi _k^n(m)-\sum \nolimits _{m=0}^{n-1}\pi _k^{n-1}(m)}, \end{aligned}$$

(17)

if (17) is non-decreasing. The expected cost under threshold policy n in the nominator is given by

$$\begin{aligned} {\mathbb {E}}(C_k(N_k^n, S_k^n(N_k^n))) = \sum \limits _{m=1}^{n}\left[ \psi _k(m)L_k^b(m) + C_k^{d}(m)\right] \pi _k^n(m) + r_k(n+1)L_k^r(n+1)\pi _k^n(n+1). \end{aligned}$$

Using the expression for the stationary distribution as derived in “Appendix B.1”, we obtain that the denominator of (17) simplifies to

$$\begin{aligned} \sum \limits _{i=0}^{n} \pi _k^{n}(i) - \sum \limits _{i=0}^{n-1} \pi _k^{{n}-1}(i) = \frac{\frac{P_{n-1}}{r_k(n)}\sum \nolimits _{i=0}^{n}\frac{P_i}{\lambda _k(i)} - \frac{P_{n}}{r_k(n+1)}\sum \nolimits _{i=0}^{n-1}\frac{ P_i}{\lambda _k(i)}}{\left( \sum \nolimits _{i=0}^{n}\frac{P_i}{\lambda _k(i)} +\frac{P_{n}}{r_k(n+1)}\right) \left( \sum \nolimits _{i=0}^{n-1}\frac{P_i}{\lambda _k(i)} +\frac{P_{n-1}}{r_k(n)}\right) }, \end{aligned}$$

where $P_i = \prod \nolimits _{j=1}^ip_k(j)$ and $p_k(j) = \frac{\lambda _k(j)}{\lambda _k(j) + \psi _k(j)}$; $P_0 = 1$. After some algebra, we obtain that (17) simplifies to the one stated in Proposition 4.

1.4 B.4 Model 1: deterioration cost per unit

We consider now a particular case when there are no breakdowns. Thus, $\psi _k(n_k) = 0$ and $L_k^b(n_k) = 0$. This simplifies since $p_k(j) = 1$ and $P_i =1$, and hence the expression in Proposition 4 simplyfies to

$$\begin{aligned} \frac{ \left( \sum \nolimits _{i=1}^{n} \frac{C_k^{d}(i)}{\lambda _k(i)} + L_k^{r}(n+1)\right) \left( \sum \nolimits _{i=0}^{n-1}\frac{1}{\lambda _k(i)} + \frac{1}{r_k(n)}\right) - \left( \sum \nolimits _{i=1}^{n-1} \frac{C_k^{d}(i)}{\lambda _k(i)} + {L_k^{r}(n)}\right) \left( \sum \nolimits _{i=0}^{n}\frac{1}{\lambda _k(i)} + \frac{1}{r_k(n+1)}\right) }{\frac{1}{r_k(n)}\sum \nolimits _{i=0}^{n}\frac{1}{\lambda _k(i)} - \frac{1}{r_k(n+1)}\sum \nolimits _{i=0}^{n-1}\frac{1}{\lambda _k(i)}}. \end{aligned}$$

(18)

If in addition $r_k(n)= r_k$ for all n, we obtain (after some algebra) from Eq. (18) that

$$\begin{aligned} W_k(n)= r_k\left[ \sum \limits _{i=0}^{n-1}\frac{C_k^{d}(n) - C_k^{d}(i)}{\lambda _k(i)} + \frac{C_k^{d}(n) - r_k L_k^{r}}{r_k}\right] . \end{aligned}$$

(19)

In addition,

$$\begin{aligned} W_k(n) - W_k(n+1) = r_k \left[ (C_k^{d}(n) -C_k^{d}(n+1))\left( \sum \limits _{i=0}^n \frac{1}{\lambda _k(i)} + \frac{1}{r_k} \right) \right] , \end{aligned}$$

which is negative when the $C_k^d(n)$ is non-decreasing.

1.5 B.5 Model 2: lump-sum cost for breakdown

Here, we assume that $C_k^{d}(n_k) = 0$, $r_k(n)= r_k(n+1)=r_k$, $L_k^r(n)= R_k, L_k^b(n) = B_k~\forall ~n$, and $\psi _k(n)$ is an increasing sequence. From Proposition 4, Whittle’s index simplifies to (9). Hence, $W_k(n) - W_k(n+1)$ simplifies to:

$$\begin{aligned} W_k(n) - W_k(n+1) = \frac{r_kB_k\left( \frac{P_n}{r_k} +\sum \nolimits _{i=0}^{n}\frac{P_i}{\lambda _k(i)} \right) \left( \frac{1}{\psi (n+1) }-\frac{1}{\psi (n)}\right) }{\left( (1-p_k(n))\sum \nolimits _{i=0}^{n}\frac{P_i}{\lambda _k(i)} + \frac{p_k(n)P_n}{\lambda _k(n)} \right) \left( (1-p_k(n+1))\sum \nolimits _{i=0}^{n+1}\frac{P_i}{\lambda _k(i)} + \frac{p_k(n+1)P_{n+1}}{\lambda _k(n+1)}\right) }, \end{aligned}$$

which is negative under the increasing breakdown rates assumption.

C Congestion control in TCP

In Sect. 6.1 we described a TCP model, where multiple users (flows) are trying to transmit packets through a bottleneck router as shown in Fig. 3.

Fig. 3

A bottleneck router in TCP with multiple flows (Avrachenkov et al. 2013)

1.1 C.1 Stationary distribution

Under a 1–0 type threshold policy n, action $a=1$ is taken in states $0,1,2, \cdots , n$ and action $a=0$ in states $n+1, n+2,\ldots$ When action $a=0$ is taken at state $n+1$, the state instantaneously changes to $S:=\max \{\lfloor \gamma .(n+1) \rfloor , 1\}$. Figure 4 shows the rates and its stationary distribution is given by

$$\begin{aligned} \pi ^n(m)= & {} 0;~m = 0,1,2,\ldots ,S-1,\\ \pi ^n(m)= & {} \frac{1}{n-S+1};~m = S, S+1, \ldots ,n. \end{aligned}$$

Fig. 4

Transition diagram under the threshold policy ‘n’ for TCP congestion control problem

We then obtain

$$\begin{aligned} {\mathbb {E}}(f_k(N_k^{n}, S_k^{n}({N}_k^{n}))) =\sum \limits _{m=0}^{n}m\pi _k^{n}(m)= \frac{n^2 + n - S(S-1)}{2(n-S+1)}, \end{aligned}$$

where $S = \max \{\lfloor \gamma _k.(n+1) \rfloor , 1\}$. It can be easily argued that

$$\begin{aligned} {\mathbb {E}}(f_k(N_k^{n}, S_k^{n}({N}_k^{n}))) - {\mathbb {E}}(f_k(N_k^{n-1}, S_k^{n-1}({N}_k^{n-1}))) = 1/2>0. \end{aligned}$$

(20)

Thus, ${\mathbb {E}}(f_k(N_k^{n}, S_k^{n}({N}_k^{n})))$ is strictly increasing in n and the result follows.

1.2 C.2 Expression of Whittle’s index: Proof of Lemma 6.1

Since 1–0 type of threshold policies are optimal, using Proposition 3, the Whittle index is given by

$$\begin{aligned} W_k(n) = \frac{{\mathbb {E}}(T_k^n(N_k^n, S_k^n({N}_k^n))) - {\mathbb {E}}(T_k^{n-1}(N_k^{{n-1}}, S_k^{{n-1}}({N}_k^{{n-1}})))}{{\mathbb {E}}(f_k(N_k^{n}, S_k^n({N}_k^{n}))) - {\mathbb {E}}(f_k(N_k^{{n-1}}, S_k^{{n-1}}({N}_k^{{n-1}})))}, \end{aligned}$$

(21)

if Eq. (21) is non-increasing in n.

We have

$$\begin{aligned} {\mathbb {E}}(T_k^n(N_k^n, S_k^n({N}_k^n))) = \sum _{m=0}^nC_k(m,1)\lambda _k \pi _k^n(m) + C_k(n+1,0)\lambda _k\pi _k^n(n+1), \end{aligned}$$

which simplifies to

$$\begin{aligned} \mathbb {E}(T_k^n(N_k^n, S_k^n({N}_k^n))) = {\left\{ \begin{array}{ll} \frac{\lambda _k\sum \nolimits _{m=S}^{n}(1-(1+m)^{1-\alpha })}{(n-S+1)(1-\alpha )} &{} \quad \text {if } \alpha \ne 1,\\ \frac{-\lambda _k\sum \nolimits _{m=S}^{n}\log (1+m)}{(n-S+1)} &{} \quad \text {if } \alpha = 1; \end{array}\right. } \end{aligned}$$

Together with (20) and Eq. (21), results in the Whittle index as stated in Lemma 6.1.

D Content delivery network

We consider here the content delivery network as described in Sect. 6.2, see also Fig. 5

Fig. 5

Optimal clearing framework as single-armed restless bandit

1.1 D.1 Stationary distribution

Under a 0–1 type of threshold policy n, action $a=0$ is taken in states $0,1,2\ldots , n$ and action $a=1$ in states $n+1, n+2,\ldots$ The transition diagram is shown in Fig. 6.

Fig. 6

Transition diagram under the threshold policy n in the content delivery network

The balance equations under this chain are

$$\begin{aligned} \pi ^n(0)\lambda (0)= & {} \pi ^n(1)\theta (1) + \lambda (n)\pi ^n(n), \end{aligned}$$

(22)

$$\begin{aligned} (\lambda (k) + \theta (k))\pi ^n(k)= & {} \lambda (k-1)\pi ^n({k-1})+ \theta (k+1)\pi ^n({k+1})\quad \text { for }\quad k=1,~2,\ldots , n-1, \end{aligned}$$

(23)

$$\begin{aligned} \lambda (n-1)\pi ^n({n-1})= & {} \theta (n)\pi ^n(n) + \lambda (n)\pi ^n(n), \end{aligned}$$

(24)

which together with the normalization condition $\sum \nolimits _{i=0}^n\pi ^n(i) = 1$ results in the following stationary distribution:

$$\begin{aligned} \pi ^{n}(m)= & {} \frac{\pi ^{n}{(n)}\lambda (n)}{\lambda (m)}\left[ 1+\sum \limits _{i=1}^{n-m}p(m+1,m+i) \right] ~\forall ~m=0,1,2,\ldots n-1,\nonumber \\ \pi ^{n}{(n)}= & {} \left( 1+\sum \limits _{k=0}^{n-1}\frac{\lambda (n)}{\lambda (k)}\left[ 1+\sum \limits _{i=1}^{n-k}p(k+1,k+i)\right] \right) ^{-1}, \end{aligned}$$

(25)

$$\begin{aligned} \pi ^{n}(m)= & {} 0 ~\forall ~m = n+1, \cdots , \end{aligned}$$

(26)

where $p(k+1, k+i) = \dfrac{\theta (k+1)\theta (k+2)\ldots \theta (k+i)}{\lambda (k+1)\lambda (k+2)\ldots \lambda (k+i)}~\forall ~i\ge 1$.

The summation term in denominator of (25) is strictly increasing in n if $\lambda (n)$ is non-decreasing. Thus, $\pi ^{n}{(n)}$ will be strictly decreasing in n under non-decreasing assumption on $\lambda (n)$.

1.2 D.2 Whittle’s index: Proof of Lemma 6.2

The expected cost under threshold policy n is given by

$$\begin{aligned} {\mathbb {E}}(T^n(N^n, S^n({N}^n))) = \sum \limits _{i=1}^{n}(iC^h(i) + \theta (i)L^a(i))\pi ^n(i) + \lambda (n)L_s^\infty (n+1) \pi ^{n}(n). \end{aligned}$$

Similarly, for the threshold policy $n-1$

$$\begin{aligned} {\mathbb {E}}(T^{n-1}(N^{n-1}, S^{n-1}({N}^{n-1})))= & {} \sum \limits _{i=1}^{n-1}(iC^h(i) + \theta (i)L^a(i))\pi ^{n-1}(i)\\&+ \lambda (n-1)L_s^\infty (n) \pi ^{{n-1}}(n-1). \end{aligned}$$

From Proposition 3, we get the expression as stated in Lemma 6.2.