Dynamic scheduling with reconfiguration delays

Abstract

We consider scheduling in networks with interference constraints and reconfiguration delays, which may be incurred when one service schedule is dropped and a distinct service schedule is adopted. Reconfiguration delays occur in a variety of communication settings, such as satellite, optical, or delay-tolerant networks. In the absence of reconfiguration delays it is well known that the celebrated Max-Weight scheduling algorithm guarantees throughput optimality without requiring any knowledge of arrival rates. As we will show, however, the Max-Weight algorithm may fail to achieve throughput optimality in case of nonzero reconfiguration delays. Motivated by the latter issue, we propose a class of adaptive scheduling algorithms which persist with the current schedule until a certain stopping criterion is reached, before switching to the next schedule. While earlier proposed Variable Frame-Based Max-Weight (VFMW) policies belong to this class, we also present Switching-Curve-Based (SCB) policies that are more adaptive to bursts in arrivals. We develop novel Lyapunov drift techniques to prove that this class of algorithms under certain conditions achieves throughput optimality by dynamically adapting the durations of the interswitching intervals. Numerical results demonstrate that these algorithms significantly outperform the ordinary Max-Weight algorithm, and that SCB policies yield a better delay performance than VFMW policies.

Appendices

Appendix 1: Proof of Lemma 1

Let the state of the process be given by $\left( \mathbf{Q}(t), I(t), \varrho (t) \right)$ where $\mathbf{Q}(t)$ is the queue length vector at the start of the slot t, I(t) indicates which queue is being served, and $\varrho (t)$ denotes the number of reconfiguration slots to go (if any). Under the ordinary Max-Weight policy, if $Q_1(t) > Q_2(t)$ and $I(t) = 2$, then reconfiguration is invoked so that $I(t+1) = 1, \varrho (t+1) = 1$, and service is not applied in slot t. The opposite is done if $Q_2(t) > Q_1(t)$. Otherwise the current service vector is applied, followed by any queue arrivals, as explained in Sect. 2.1. (Note that the state variable $\varrho (t)$ is redundant.) With the state $\left( \mathbf{Q}(t), I(t), \varrho (t) \right)$ the process forms a time-homogeneous Markov chain with time index $t \in {\mathbb {N}}_0$.

Let $\mathcal{D} \doteq \left\{ \mathbf{Q}: | Q_1 - Q_2 | \le 3 \right\}$; elementary considerations show that the main diagonal $Q_1 = Q_2$ is reached with probability 1 for $p \in (0,1/2)$. Once the main diagonal is reached the process will be confined to $\mathcal{D}$. Restricting to slots where reconfiguration is not invoked, a slot t is said to be a freeze slot if $\mathbf{Q}(t) = \mathbf{Q}(t+1)$ and is said to be progressive otherwise. Finally, further consideration shows that reconfiguration occurs infinitely often a.s. with at most four progressive slots occurring between two slots where reconfiguration is invoked, again once the main diagonal is reached.

To show the Markov chain is transient (instability), let $S_k$ be the slot in which the kth reconfiguration takes place and let $M_k = Q_1(t_k) + Q_2(t_k)$ be the total queue length at time $S_k$, $k = 0, 1, 2, \dots$. Consider the interval between one reconfiguration and the next, where we will examine the mean drift of $M_k$. Since a freeze slot occurs with probability $p (1 - p)$, the number of freeze slots until the occurrence of a progressive one is geometrically distributed with mean $\frac{p (1 - p)}{1 - p + p^2}$. Furthermore, the total drift along the main diagonal per service slot is $- (1 - 2p)$, which is negative (toward the origin) for $p < 1/2$.

From the above it follows that the accumulated drift during a service interval is no less than

$$\begin{aligned} - 4 (1 - 2 p) \times \left( 1 + \frac{p (1 - p)}{1 - p + p^2}\right) = \frac{- 4 (1 - 2 p)}{1 - p + p^2}. \end{aligned}$$

During a reconfiguration slot, the positive drift (away from the origin) is 2 p. Thus, the total accumulated drift along the main diagonal must be positive when

$$\begin{aligned} 2 p - \frac{4 (1 - 2 p)}{1 - p + p^2} > 0, \end{aligned}$$

which amounts to $p > 0.42049$.

We have thus shown that there is a $p < 1/2$ such that the sequence $M_k$ satisfies

$$\begin{aligned} {\mathbb E}\left[ M_{k+1} | M_k \right] \ge M_k + \theta , \end{aligned}$$

where $\theta > 0$ is a fixed constant. We now show that $M_k \rightarrow \infty$ as $k \rightarrow \infty$ almost surely, which in turn implies $Q_\ell (t) \rightarrow \infty$ as $t \rightarrow \infty$, $\ell = 1, 2$. Introduce $Z_k = M_{k+1} - M_k$, $k = 0, 1, 2, \dots$, and observe that $Z_k \le 6$. Additionally, define $R_k = \frac{1}{M_k+1}$, and observe

$$\begin{aligned} {\mathbb E}\left[ \frac{1}{M_{k+1} + 1} | {\mathcal {F}}_k\right]= & {} \frac{1}{M_k + 1} - {\mathbb E}\left[ \frac{Z_k}{(M_k + 1) (M_k + 1 + Z_k)}\right] \\\le & {} \frac{1}{M_k + 1} - \frac{\theta }{(M_k + 1) (M_k + 1 + 6)}, \end{aligned}$$

from which it follows that $R_k$ is a nonnegative supermartingale where ${\mathcal {F}}_k$ is an appropriate sigma algebra at time slot $t_k$. Hence $\lim _{k \rightarrow \infty }R_k \le 1$ exists almost surely, and so does $\lim _{k \rightarrow \infty }M_k$. Finally,

$$\begin{aligned} {\mathbb E}\left[ R_\infty \right] \le \liminf _{k \rightarrow \infty }{\mathbb E}\left[ R_k\right] \le 1 - \sum _{k=0}^{\infty } {\mathbb E}\left[ \frac{\theta }{(M_k + 1) (M_k + 7)}\right] , \end{aligned}$$

which implies

$$\begin{aligned} \liminf _{k \rightarrow \infty }{\mathbb E}\left[ \left( \frac{\theta }{(M_k + 7)}\right) ^2\right] \le \liminf _{k \rightarrow \infty }{\mathbb E}\left[ \frac{\theta }{(M_k + 1) (M_k + 7)}\right] = 0. \end{aligned}$$

This shows that $\liminf _{k \rightarrow \infty }(M_k+7)^{-2} = 0$ almost surely by Fatou’s lemma, and since actually there is a limit, we have $\lim _{k \rightarrow \infty }M_k = \infty$ almost surely. This then implies $\lim _{k \rightarrow \infty }Q_\ell (t_k) = \infty$ almost surely, since $\vert Q_1 - Q_2 \vert \le 3$. Moreover, if $t \in [t_k, t_{k+1})$, then $\vert Q_\ell (t) - Q_\ell (t_k) \vert \le 4$, $\ell = 1, 2$, so that $Q_\ell (t) \rightarrow \infty$ as $t \rightarrow \infty$ almost surely. Since $\liminf _{t \rightarrow \infty }{\mathbb E}\left[ Q_\ell (t)\right] \ge {\mathbb E}\left[ \lim _{t \rightarrow \infty }Q_\ell (t)\right] = \infty$, the system is not strongly stable. $\square$

Appendix 2: Proof of Lemma 2

On occurrence of $S_k < \infty$, condition (iii) in Theorem 1 implies that ${\mathbb E}\left[ \chi _k | {\mathcal {F}}_{S_k}\right] \le T_r + \sqrt{c_2} F(\mathbf{Q}(S_k)) < \infty$, so that $\chi _k$ is finite almost surely. Using the Lindley recursion in (1) for time slots $t = S_k, \dots , S_k + \chi _k - 1$, we obtain

$$\begin{aligned} Q_\ell (S_k+\chi _k) \le \max \left\{ Q_\ell (S_k) - \sum _{s=T_r-1}^{\chi _k-1} I_\ell (S_k+s), 0\right\} + \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s). \end{aligned}$$

(26)

Note that if $\sum _{s=T_r-1}^{\chi _k-1} I_\ell (S_k+s)$, representing the total service opportunity given to queue $\ell$ during the $k^{\text {th}}$ service interval, is smaller than $Q_\ell (S_k)$, then the inequality in (26) in fact holds with equality. Otherwise, the first term is 0 and (26) holds with inequality, because some of the arrivals during the service interval might depart before the end of the service interval.

Squaring both sides of (26), using $\max \{0,x\}^2 \le x^2$ and $I_\ell (t) \le \mu _{\max }$ for all $\ell = 1, \dots , N$, $t = S_k + T_r - 1, \dots , S_k + \chi _k - 1$, we obtain

$$\begin{aligned} Q_\ell (S_k+\chi _k)^2- & {} Q_\ell (S_k)^2 \le \chi _k^2 \mu _{\max }^2 + \left( \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) ^2 \nonumber \\- & {} 2 Q_\ell (S_k) \left( \sum _{s=T_r-1}^{\chi _k-1} I_\ell (S_k+s) - \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) . \end{aligned}$$

(27)

Summing (27) over the queues, taking conditional expectations, and then using Wald’s equality (or optional stopping), we derive

$$\begin{aligned} \varDelta _{S_k}\le & {} N \mu _{\max }^2 {\mathbb E}\left[ \chi _k^2 | {\mathcal {F}}_{S_k}\right] + \sum _{\ell = 1}^{N} {\mathbb E}\left[ \left( \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) ^2 | {\mathcal {F}}_{S_k}\right] \\&+\, 2 {\mathbb E}\left[ \chi _k | {\mathcal {F}}_{S_k}\right] \sum _{\ell = 1}^{N} \lambda _\ell Q_\ell (S_k) - 2 \sum _{\ell = 1}^{N} Q_\ell (S_k) {\mathbb E}\left[ \sum _{s=T_r-1}^{\chi _k-1} I_\ell (S_k+s) | {\mathcal {F}}_{S_k}\right] , \nonumber \end{aligned}$$

(28)

where we used the fact that the arrival processes are i.i.d. over time and independent of the queue lengths.

Now observe that for any arrival rate vector $\lambda \in \Lambda _0^0$, there exist real numbers $\beta ^1, \dots , \beta ^{|{\mathcal {I}}|}$ such that $\beta ^j \ge 0$ for all $j = 1, \dots , |{\mathcal {I}}|$, $\sum _{j=1}^{|{\mathcal {I}}|} \beta ^j = 1 - \epsilon$ for some $\epsilon > 0$ and $\mathbf {\lambda } = \sum _{j=1}^{|{\mathcal {I}}|} \beta ^j \mathbf{I}^j$.

Substituting the latter expression in (28) and using conditions (ii) and (iii) of Theorem 1, we obtain

$$\begin{aligned} \varDelta _{S_k}\le & {} N \mu _{\max }^2 (T_r^2 + c_2 (F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) )^2) + \sum _{\ell = 1}^{N} {\mathbb E}\left[ \left( \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) ^2 | {\mathcal {F}}_{S_k}\right] \nonumber \\&+\,2 {\mathbb E}\left[ \chi _k | {\mathcal {F}}_{S_k}\right] \mathbf{Q}(S_k) \cdot \sum _{j=1}^{|{\mathcal {I}}|} \beta ^j \mathbf{I}^j - 2 {\mathbb E}\left[ \chi _k - T_r | {\mathcal {F}}_{S_k}\right] \mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k)\nonumber \\\le & {} N \mu _{\max }^2 (T_r^2 + c_2 (F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) )^2) + \sum _{\ell = 1}^{N} {\mathbb E}\left[ \left( \sum _{\tau =0}^{\chi _k-1} A_\ell (S_k+\tau )\right) ^2 | {\mathcal {F}}_{S_k}\right] \nonumber \\&- 2 c_1 \epsilon (1 - \delta (||\mathbf{Q}(S_k)||)) F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) \mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k)+ 2 T_r \mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k), \nonumber \\ \end{aligned}$$

(29)

where in the last inequality we used the fact that $\mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k)\ge \mathbf{Q}(S_k)\cdot \mathbf{I}$ for all $\mathbf{I}\in {\mathcal {I}}$ by definition of the Max-Weight service vector.

Applying Lemma 3 to the second term in (29), we derive

$$\begin{aligned}&{\mathbb E}\left[ \left( \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) ^2 | {\mathcal {F}}_{S_k}\right] \\&\quad \le 2 {\mathbb E}\left[ \sum _{s=0}^{\chi _k-1} A_{\max }^2 - \lambda _\ell ^2 | {\mathcal {F}}_{S_k}\right] + 2 {\mathbb E}\left[ \left( \sum _{s=0}^{\chi _k-1} \lambda _\ell \right) ^2 | \mathcal{F}_{S_k}\right] \\&\quad = 2 \left( A_{\max }^2 - \lambda _\ell ^2\right) {\mathbb E}\left[ \chi _k | \mathbf{Q}(S_k)\right] + 2 \lambda _\ell ^2 {\mathbb E}\left[ \chi _k^2 | {\mathcal {F}}_{S_k}\right] . \end{aligned}$$

Using conditions (ii) and (iii) of Theorem 1, we obtain

$$\begin{aligned} {\mathbb E}\left[ \left( \sum _{s=0}^{\chi _k-1} A_\ell (S_k+s)\right) ^2 | {\mathcal {F}}_{S_k}\right] \le d_1 (F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) )^2, \end{aligned}$$

for some constant $d_1 < \infty$. Substituting the latter inequality in (29), we derive

$$\begin{aligned} \varDelta _{S_k}\le & {} N \mu _{\max }^2 T_r^2 + F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) \\&\times \Bigg ((c_2 N \mu _{\max }^2 + d_1) F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) - 2 \left( c_1 (1 - \delta (||\mathbf{Q}(S_k)||)) \epsilon - \frac{T_r}{F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) }\right) \\&\times \mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k)\Bigg ). \end{aligned}$$

Since $F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right)$ is a monotonically increasing function of $\vert \vert \mathbf{Q}(S_k) \vert \vert$, and $\delta (\vert \vert \mathbf{Q}(S_k) \vert \vert )$ is a monotonically decreasing function of $\vert \vert \mathbf{Q}(S_k) \vert \vert$, there exists a constant $d_2$ such that if $\vert \vert \mathbf{Q}(S_k) \vert \vert > d_2$, then $c_1 (1 - \delta (||\mathbf{Q}(S_k)||)) \epsilon - \frac{T_r}{F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) } > \delta _1 > 0$, yielding

$$\begin{aligned} \varDelta _{S_k}\le & {} N \mu _{\max }^2 T_r^2 + (c_2 N \mu _{\max }^2 + d_1) (F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) )^2\\&- 2 \delta _1 F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) \mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k). \end{aligned}$$

Hence, for $\vert \vert \mathbf{Q}(S_k) \vert \vert > d_2$, we use $\mathbf{Q}(S_k)\cdot \mathbf{I}^*(S_k)\ge \frac{1}{N} \sum _{\ell = 1}^{N} Q_\ell (t_k)$ to arrive at

$$\begin{aligned} \varDelta _{S_k} \le N \mu _{\max }^2 T_r + (c_2 N \mu _{\max }^2 + d_1) (F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) )^2 - \frac{2 \delta _1}{N} F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) \vert \vert \mathbf{Q}(S_k) \vert \vert . \end{aligned}$$

Since $F(\cdot )$ is a sublinear function, it follows that there exist fixed constants $c_3 < \infty$, $\eta = \delta _1 / N$ such that

$$\begin{aligned} \varDelta _{S_k} \le c_3 - \eta F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) \vert \vert \mathbf{Q}(S_k) \vert \vert . \end{aligned}$$

This completes the proof. $\square$

Appendix 3: Proof of Lemma 4

Divide the state space into the set ${\mathcal {E}}$ containing the ergodic states and its complement ${\mathcal {T}}$ consisting of the null and transient states. By Markov’s inequality, given $\varepsilon > 0$, there exists $C_\varepsilon$ such that

$$\begin{aligned} {\mathbb P}\left\{ \vert \vert \mathbf{Q}(t) \vert \vert \le C_\varepsilon \right\} \ge 1 - \varepsilon \qquad \qquad \text{ for } \text{ all } t \in {\mathbb {N}}_0. \end{aligned}$$

Let $E_t$ be the event that $(\mathbf{Q}(t), J(t)) \in {\mathcal {E}}$. Since there are only finitely many states in ${\mathcal {T}}\cap \{\vert \vert \mathbf{Q} \vert \vert \le C_\varepsilon \}$, we may apply Theorem 5 in [13] on page 389, and the reverse Fatou lemma to obtain

$$\begin{aligned} {\mathbb P}\left\{ E_t~i.o.\right\} \ge \limsup _{t \rightarrow \infty } {\mathbb P}\left\{ \vert \vert \mathbf{Q}(t) \vert \vert \le C_\varepsilon \right\} \ge 1 - \varepsilon . \end{aligned}$$

Since $\varepsilon > 0$ is arbitrary, the above implies that an ergodic state is entered with probability 1.

The existence of a random variable $Q_\infty$ with the stationary distribution follows directly from irreducibility and [14, Sect. XI.8, Theorem 1, p. 379]. Finally,

$$\begin{aligned} {\mathbb E}\left[ \vert \vert \mathbf{Q}_\infty \vert \vert \right] \le \liminf {\mathbb E}\left[ \vert \vert \mathbf{Q}(t) \vert \vert \right] \le E_{\max } < \infty \end{aligned}$$

as in Theorem 5.3 in [3, p. 32], as the distribution of $\mathbf{Q}(t)$ converges weakly to a unique stationary distribution. $\square$

Appendix 4: Proof of Corollary 2

For the sake of the proof, we may as well suppose that the sequence $S_k < \infty$ almost surely. Otherwise, the system reconfigures only finitely many times as it is 1-Harris recurrent, so that the bound given in (31) holds with $K_T = 1$. Hence we may suppose that there is a reconfiguration state $\mathbf{s} = (\mathbf{Q}, \mathbf{I}, \mathbf{Q}, \varrho _0)$ which occurs i.o. almost surely, by assumption. Call an occurrence of $\mathbf{s}$ a renewal. Now define $T_M$ to be the number of slots used in taking M steps of the process $\left( \mathbf{Q}(\tau _m), \mathbf{I}(\tau _m), {\mathbbm {1}}^{(m)}\right)$ starting from $m=0$,

$$\begin{aligned} T_M = \sum _{m=0}^{M-1} \sum _{k = 0}^{m} \chi _k {\mathbbm {1}}^{(m,k)}+ \sum _{m=0}^{M-1} {\mathbbm {1}}^{(m)}\ge M, \end{aligned}$$

because multiple slots are used after each reconfiguration.

It is obvious that

$$\begin{aligned} \sum _{k = 0}^{m} \chi _k {\mathbbm {1}}^{(m,k)}\le \sum _{k = 0}^{m} \chi _k \vert \vert \mathbf{Q}(S_k) \vert \vert {\mathbbm {1}}^{(m,k)}+ \sum _{k = 0}^{m} \chi _k {\mathbbm {1}}^{(m,k)}{\mathbb I}\left[ \mathbf{Q}(\tau _m)= 0\right] . \end{aligned}$$

(30)

Taking expectations and bounding the last term in (30), we obtain

$$\begin{aligned} \sum _{k=0}^{m} {\mathbb E}\left[ \chi _k; A_{m,k}, \mathbf{Q}(\tau _m)=0\right]= & {} \sum _{k=0}^{m} {\mathbb E}\left[ \chi _k; A_{m,k}, \mathbf{Q}(S_k)= 0\right] \\\le & {} \sum _{k=0}^{m} {\mathbb E}\left[ T_r + \sqrt{c_2} F(0); A_{m,k}\right] \\\le & {} T_r + \sqrt{c_2}F(0) =: \mu _0. \end{aligned}$$

The equality holds since $A_{m,k}\in {\mathcal {F}}_{S_k}$ and since $\mathbf{Q}(S_k)= 0$ must have occurred. The first inequality follows on applying (12). The last inequality holds since the events $A_{m,k}$ are disjoint.

Taking expectations and turning to the first term in the right-hand side of (30), we may again apply (12), as the event $A_{m,k}\in {\mathcal {F}}_{S_k}$, to obtain

$$\begin{aligned} {\mathbb E}\left[ \vert \vert \mathbf{Q}(S_k) \vert \vert \chi _k; A_{m,k}\right] \le c {\mathbb E}\left[ F(\vert \vert \mathbf{Q}(\tau _m) \vert \vert ) \vert \vert \mathbf{Q}(\tau _m) \vert \vert ; A_{m,k}\right] \end{aligned}$$

for some finite positive constant $c < \infty$. Combining the above two inequalities, it follows that

$$\begin{aligned} \limsup _{M \rightarrow \infty } \frac{1}{M} {\mathbb E}\left[ T_M\right] \le 1 + \mu _0 + \limsup _{M \rightarrow \infty } \frac{c}{M} \sum _{m=0}^{M-1} \sum _{k = 0}^{m} {\mathbb E}\left[ F(\vert \vert \mathbf{Q}(\tau _m)) \vert \vert ) \vert \vert \mathbf{Q}(\tau _m) \vert \vert ; A_{m,k}\right] . \end{aligned}$$

Equation (18) now shows that

$$\begin{aligned} \limsup _{M \rightarrow \infty } \frac{1}{M} {\mathbb E}\left[ T_M\right] = K_T < \infty . \end{aligned}$$

(31)

Now let $U_r$ be the number of slots between renewal r and renewal $r+1$ with a delay of V slots until the first renewal. The assumption of 1-Harris recurrence implies $V < \infty$. Let $\mu _R = {\mathbb E}\left[ U_1\right] \le \infty$ be the expected number of slots between renewals.

Let R(M) be the number of renewals over the first M steps, so that

$$\begin{aligned} \sum _{r=1}^{R(M)-1} U_r \le T_M. \end{aligned}$$

(32)

Since $\mathbf{s}$ is an ergodic state,

$$\begin{aligned} \lim _{M \rightarrow \infty } \frac{R(M)}{M} = \phi _{\mathbf{s}}, \end{aligned}$$

with $\phi _{\mathbf{s}} \in (0, 1]$ by the renewal theorem. Moreover, by the extended Strong Law of Large Numbers,

$$\begin{aligned} \lim _{M \rightarrow \infty } \frac{1}{R(M)} \sum _{r=1}^{R(M)} U_r = \mu _R \text{ almost } \text{ surely. } \end{aligned}$$

Rewriting (32), we obtain

$$\begin{aligned} \frac{R(M)}{M} \times \frac{1}{R(M)} \sum _{r=1}^{R(M)} U_r \le \frac{T_M}{M}. \end{aligned}$$

Now let $M \rightarrow \infty$ along any sequence so that the term on the right-hand side converges to $\liminf _{M \rightarrow \infty } \frac{T_M}{M} \ge 1$, then

$$\begin{aligned} \phi _{\mathbf{s}} \mu _R \le \liminf _{M \rightarrow \infty } \frac{T_M}{M}. \end{aligned}$$

Using Fatou’s lemma, we deduce

$$\begin{aligned} {\mathbb E}\left[ \liminf _{M \rightarrow \infty } \frac{T_M}{M}\right] \le \liminf _{M \rightarrow \infty } {\mathbb E}\left[ \frac{T_M}{M}\right] \le \limsup _{M \rightarrow \infty } {\mathbb E}\left[ \frac{T_M}{M}\right] = K_T < \infty . \end{aligned}$$

It follows that $\liminf _{M \rightarrow \infty } \frac{T_M}{M}$ is almost surely finite, and therefore $\mu _R < \infty$.

The 1-Harris recurrence of the chain $\left( \mathbf{Q}(t), \mathbf{I}(t), \mathbf{Q}(S_k(t)), \varrho (t) \right)$ follows immediately as $\mathbf{s}$ has been shown to be an ergodic state within that chain. This completes the proof. $\square$

Appendix 5: Proof of Lemma 5

To obtain a lower bound for $\chi _k$, we consider the arrivals and departures separately, since the triangle inequality shows that the change in queue length cannot exceed their sum, at any stage. (Thus the sum can only exceed the BB threshold at $\chi _k$ or an earlier slot.) Moreover, we may count virtual departures (when queues are empty) because we are only concerned with a lower bound.

Define $T_E$ to be the first time slot when the sum of arrivals and virtual departures, $\Sigma _j,~j \ge T_r$,

$$\begin{aligned} \Sigma _j \doteq \sum _{s=0}^{j- 1} \vert \vert \mathbf{A}(S_k+s) \vert \vert + \sum _{s=T_r}^{j} \vert \vert \mathbf{I}(S_k+s) \vert \vert \end{aligned}$$

reaches the threshold level $\theta _{BB}F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right)$ after reconfiguration at $S_k$. $T_E \ge T_r$ is therefore a ${\mathcal {F}}_{S_k}$ stopping time. Let $M_j \doteq \Sigma _j - {\mathbb E}\left[ \Sigma _j | \mathcal{F}_{S_k+j-1}\right] ,~j \ge 1$, with $M_0=0$, so that $M_j$ is a $\mathcal{F}_{S_k+j}$ martingale, null at 0, and bounded in $\mathcal{L}^1$. Clearly $T_E$ is almost surely finite and has finite expectation, since all the summands are positive and have strictly positive expectation. It follows that condition (16) of Corollary 5 in [10, p. 243], holds. By the definition of $T_E$, at stopping it holds that

$$\begin{aligned} \sum _{s=0}^{T_E - 1} \vert \vert \mathbf{A}(S_k+s) \vert \vert + \sum _{s=T_r}^{T_E} \vert \vert \mathbf{I}(S_k+s) \vert \vert \ge \theta _{BB}F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) , \end{aligned}$$

and from optional stopping we obtain

$$\begin{aligned}&{\mathbb E}\left[ \sum _{s=0}^{T_E - 1} \vert \vert \mathbf{A}(S_k + s) \vert \vert + \sum _{s=T_r}^{T_E} \vert \vert \mathbf{I}(S_k+s) \vert \vert \right] \\&\qquad = {\mathbb E}\left[ T_E\right] \sum _{\ell =1}^{N} \lambda _\ell + ({\mathbb E}\left[ T_E\right] - T_r+1) \sum _{\ell =1}^{N} \mathbf{I}_\ell . \end{aligned}$$

Therefore,

$$\begin{aligned} {\mathbb E}\left[ \chi _k | {\mathcal {F}}_{S_k}\right] \ge {\mathbb E}\left[ T_E | {\mathcal {F}}_{S_k}\right] \ge \frac{\theta _{BB}F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right) }{\sum _{\ell =1}^{N} (\lambda _\ell + \mathbf{I}_\ell )}, \end{aligned}$$

which yields the stated lower bound on ${\mathbb E}\left[ \chi _k | {\mathcal {F}}_{S_k}\right]$ with $\delta = 0$ and

$$\begin{aligned} c_1 = \frac{\theta _{BB}}{\vert \vert \mathbf {\lambda } \vert \vert + N \mu _{\max }}. \end{aligned}$$

$\square$

Appendix 6: Proof of Lemma 6

For any arrival rate vector $\mathbf {\lambda }$ in the interior of the stability region $\mathbf {\Lambda }$, there exists an $\epsilon > 0$ such that $\mathbf {\lambda } + \epsilon \varvec{1} \in \mathbf {\Lambda }$. Therefore, the Max-Weight service vector $\mathbf{I}^*(S_k)= [\mathbf{I}_1^*, \dots , \mathbf{I}_N^*]$ satisfies

$$\begin{aligned} \sum _{\ell = 1}^{N} Q_\ell (S_k) \mathbf{I}_\ell ^* \ge \mathbf{Q}(S_k)\cdot \mathbf {\lambda } + \epsilon \vert \vert \mathbf{Q}(S_k) \vert \vert , \end{aligned}$$

yielding

$$\begin{aligned} \sum _{\ell = 1}^{N} Q_\ell (S_k) (\mathbf{I}_\ell ^* - \lambda _\ell ) \ge \epsilon \sum _{\ell = 1}^{N} Q_\ell (S_k). \end{aligned}$$

Since some queues might have negative contributions to the sum on the left-hand side, we have

$$\begin{aligned} \sum _{\ell : \mathbf{I}_\ell ^* > \lambda _\ell } Q_\ell (S_k) (\mathbf{I}_\ell ^* -\lambda _\ell ) \ge \epsilon \sum _{\ell = 1}^{N} Q_\ell (S_k). \end{aligned}$$

Let $\ell _0$ be the queue with the maximum contribution to the sum on the left-hand side. We have

$$\begin{aligned} Q_{\ell _0}(S_k) \ge \frac{\epsilon ||\mathbf{Q}(S_k)||}{N \left( I^*_{\ell _0} - \lambda _{\ell _0} \right) }. \end{aligned}$$

Since there are only finitely many possibilities for the strictly positive denominator, there is an $A_\epsilon > 0$ and an $\ell _0$ such that

$$\begin{aligned} Q_{\ell _0}(S_k) \ge A_\epsilon \vert \vert \mathbf{Q}(S_k) \vert \vert \end{aligned}$$

for every $\mathbf{Q}(S_k)\in {\mathbb {N}}_0^N$. Since the function $F(\cdot )$ is sublinear, it follows that we may take

$$\begin{aligned} {\mathcal {C}}= \{\vert \vert \mathbf{Q} \vert \vert \le D\} \end{aligned}$$

for some finite positive constant D, so that $Q_{\ell _0} \ge \lceil \theta _{BB}F(\vert \vert \mathbf{Q} \vert \vert ) \rceil$. Since it is also the case that the drift of queue $\ell _0$ is strictly negative, the proof is complete. $\square$

Appendix 7: Proof of Lemma 7

Let $D = I_{\ell ^*} > \lambda _{\ell ^*} = a$ be the number of packets that can be served at queue $\ell ^*$ under the Max-Weight service vector in each time slot. (For compactness, we drop the queue index $\ell ^*$ from the notation in the remainder of the proof.) Thus the queue length evolves according to the usual Lindley recursion

$$\begin{aligned} Q(t+1) = \max \{Q(t) - D + A(t), 0\}. \end{aligned}$$

Let $T_1$ be the random amount of time for the queue length to reduce by at least one packet, assuming that the initial number of packets is D or larger. Thus $T_1$ is the so-called descending ladder index of the associated random walk. It is well known [14], pages 396–397, that ${\mathbb E}\left[ T_1\right] < \infty$, since the random walk has strictly negative drift. If the number of arrivals per slot has finite variance $\sigma _A^2 < \infty$, it is reasonable to suppose that this implies $\sigma _{T_1}^2 = {\mathbb V}\left[ T_1\right] < \infty$, and we now proceed to show that this is indeed the case.

Suppose we start a busy period with the service of $Q(1) = D$ packets. Arrivals may take place during this time slot, and so the busy period will continue under the recursion

and we stop at step $T_1 = n \in {\mathbb {N}}$ as soon as $[Q(n) - D + A(n)]^+ \in \{0, 1, \dots , D - 1\}$, which is the first time the number of packets falls by at least one. Further define $S(n) = \sum _{k=1}^{n} A(k)$ and $U(n) = S(n) - n a$, which we may take as null for $n = 0$. By construction, U(n) is a $\mathcal{L}_2$ martingale. At the stopping time $T_1$, the following equality holds:

$$\begin{aligned} S(T_1) = D T_1 - \varepsilon _{T_1}, \end{aligned}$$

where $\varepsilon _{T_1} \in \{1, \dots , D\}$. This holds since full use has been made of the service in each time slot except for the slot following $T_1$.

Although not needed, it is the case that optional stopping holds for the martingale U(n) with respect to $T_1$. This follows since ${\mathbb E}\left[ |A(1) - a |\right] = M < \infty$ and the sequence is independent. Finally, since ${\mathbb E}\left[ T_1\right] < \infty$, we find that

$$\begin{aligned} {\mathbb E}\left[ \sum _{k=1}^{T_1} {\mathbb E}\left[ |A(k) - a | | {\mathcal {F}}_{k-1}\right] \right] = M {\mathbb E}\left[ T_1\right] < \infty . \end{aligned}$$

Hence, from [10, Chapter 7.4, Corollary 5, p. 243], we have ${\mathbb E}\left[ U(T_1)\right] = 0$ which shows that

$$\begin{aligned} (D - a) {\mathbb E}\left[ T_1\right] = {\mathbb E}\left[ \varepsilon _{T_1}\right] , \end{aligned}$$

yielding ${\mathbb E}\left[ T_1\right] = 1/(1-a)$ in the special case $D = 1$.

We also obtain from Theorem 7 of [10, p. 245], that

$$\begin{aligned} {\mathbb E}\left[ U^2(T_1)\right] \le \sigma _A^2 {\mathbb E}\left[ T_1\right] = {\mathbb E}\left[ \sum _{k=1}^{T_1} (A(k) - a)^2\right] , \end{aligned}$$

since the A(k) are independent. Substituting, we find

$$\begin{aligned} (D-a)^2 {\mathbb E}\left[ T_1^2\right] - 2 (D-a) {\mathbb E}\left[ T_1 \varepsilon _{T_1}\right] + {\mathbb E}\left[ \varepsilon _{T_1}^2\right] \le \sigma _A^2 {\mathbb E}\left[ T_1\right] , \end{aligned}$$

which shows that ${\mathbb V}\left[ T_1\right] < \infty$. $\square$

Appendix 8: Proof of Lemma 8

For compactness, define $M = \theta _{BB}F\left( \vert \vert \mathbf{Q}(S_k)\vert \vert \right)$, where we suppose that $M > 0$. Next suppose that we are outside the compact set $\mathcal{C}$ shown to exist in Lemma 6, but where the inequality on the right is multiplied by 3. Denote by $\ell ^*$ the index of the queue as given in the lemma. It follows that $Q_{\ell ^*} \ge 3 M$. Let Y be the amount of time for $Q_{\ell ^*}$ to decrease by 2 M, and as we have seen in Lemma 7 such a decrease is possible. The value 2 M is considered, since $Q_{\ell ^*}$ may increase by up to M packets without causing an immediate reconfiguration after $T_r$ time slots. The behavior of $Q_{\ell ^*}$ now follows a discrete-time random walk on ${\mathbb {N}}_0$. Clearly, $\chi _k \le Y + T_r$, and Y is finite almost surely since $Q_{\ell ^*}$ has strictly negative drift.

Let $T_{\ell ^*}$ be the amount of time for $Q_{\ell ^*}$ to reduce by at least one packet. Lemma 7 gives ${\mathbb E}\left[ T_{\ell ^*}^2\right] < \infty$.

The random variable Y is stochastically smaller than the sum of 2 M i.i.d. copies $\tau _1, \dots , \tau _{2 M}$ of $T_{\ell ^*}$, since each descent is by at least one. The Cauchy–Schwarz inequality [34, p. 62], then implies

$$\begin{aligned} {\mathbb E}\left[ Y^2\right] \le {\mathbb E}\left[ \left( \sum _{k=1}^{2 M} \tau _k\right) ^2\right] \le 4 M^2 {\mathbb E}\left[ T_{\ell ^*}^2\right] < \infty . \end{aligned}$$

Since $F(y) \uparrow \infty$ as $y \rightarrow \infty$ and given the definition of M, it follows that there is a constant $c_2$ such that (22) holds for all $\mathbf{Q}(S_k)\not \in {\mathcal {C}}$.

If $\mathbf{Q}(S_k)\in \mathcal C$, then there exists a finite constant $A \ge \max _{\mathbf{Q}\in {\mathcal {C}}} F(\vert \vert \mathbf{Q} \vert \vert )$. Moreover, the set ${\mathcal {C}}_A$ of points which are within A of some point in ${\mathcal {C}}$ is also compact. Let $B = \sup _{\mathbf{Q}\in {\mathcal {C}}_A} \vert \vert \mathbf{Q} \vert \vert$. Then if there are more than B arrivals in a given time slot, the stopping criterion must be triggered, if it has not been reached already. The time until B arrivals have occurred has a geometric distribution with finite first and second moments, since the distribution of the number of arrivals has unbounded support. It follows that (22) holds for all $\mathbf{Q}\in {\mathbb {N}}_0^N$ for $c_2$ sufficiently large. $\square$