Optimal scheduling of parallel queues using stochastic flow models

Abstract

We consider a classic scheduling problem for optimally allocating a resource to multiple competing users and place it in the framework of Stochastic Flow Models (SFMs). We derive Infinitesimal Perturbation Analysis (IPA) gradient estimators for the average holding cost with respect to resource allocation parameters. These estimators are easily obtained from a sample path of the system without any knowledge of the underlying stochastic process characteristics. Exploiting monotonicity properties of these IPA estimators, we prove the optimality of the well-known cμ-rule for an arbitrary finite number of queues and stochastic processes under non-idling policies and linear holding costs. Further, using the IPA derivative estimates obtained along with a gradient-based optimization algorithm, we find optimal solutions to similar problems with nonlinear holding costs extending current results in the literature.

Appendix: Proofs of the Lemmas

Proof of Lemma 1

If the start of a NEPn is due to a positive jump in α _n (t) at t _k , it is an exogenous event and by Eq. 13 the Lemma’s statement immediately follows. If α _n (t) is continuous at t _k , we have $\alpha_{n}(t_{k}^{-})=\alpha_{n}(t_{k}^{+})=\mu_{n} \theta_{n,m}(t_{k})$. Then, using Eq. 4, we see that $f_{n} (t_{k}^{-})-f_{n}(t_{k}^{+})=0$. By Assumption 1 no other inflow rate can be discontinuous at t _k . Therefore, $\alpha_{l}(t_{k} ^{-})=\alpha_{l}(t_{k}^{+})$ for all l ≠ n. Looking at Eq. 3, we find that $u_{l}(t_{k}^{-})=u_{l}(t_{k}^{+})$ for l ≠ n, so that from Eq. 4 we get $f_{l}(t_{k}^{-})-f_{l}(t_{k}^{+})=0$. Therefore, in the case of an endogenous event at t _k we find that $[f_{l}(t_{k} ^{-})-f_{n}(t_{l}^{+})]=0$ for all l = 1,...,N and the proof is complete.□

Proof of Lemma 2

Starting with part (a) for n = 1, by the definition of η _1,m we have $\theta_{1}(\eta_{1,m}^{+})=\frac{\alpha_{1}(\eta_{1,m} )}{\mu_{1}}$. Consequently, $\Delta\theta_{1}(\eta_{1,m})=\frac{\alpha _{1}(\eta_{1,m})}{\mu_{1}}-\theta_{1}(\eta_{1,m}^{-})$. At η _1,m the system switches to the cμ-rule, so the total residual resource capacity is allocated to queue 2, i.e., $\theta_{2}(\eta_{1,m}^{+})=1-\frac{\alpha _{1}(\eta_{1,m})}{\mu_{1}}$, $\theta_{n}(\eta_{1,m}^{+})=0$ for n > 2. Applying Eq. 20, part (a) of the lemma is proved.

For part (b), let us first prove Δθ _n (η _k,m ) = 0 for all k ≠ n,n − 1. If n < k then by Eq. 19 queue n is empty before η _k,m . By Assumption 1, α _n (t) is continuous at η _k,m , so Eq. 3 gives $\theta_{n}(\eta_{k,m}^{-} )=\theta_{n}(\eta_{k,m}^{+})=\frac{\alpha_{n}(\eta_{k,m})}{\mu_{k}}$ which in turn results in Δθ _n (η _k,m ) = 0 for n < k. On the other hand, when n > k + 1, according to the cμ-rule, no resource is given to queue n before or after η _k,m and we get $\theta_{n}(\eta_{k,m}^{-})=\theta _{n}(\eta_{k,m}^{+})=0$, thereby, yielding Δθ _n (η _k,m ) = 0 for n > k + 1. Combining these two results establishes Δθ _n (η _k,m ) = 0 for all k ≠ n,n − 1. Next, since n > 2, the cμ-rule applies to the interval before and after η _{n − 1,m} . At $\eta_{n-1,m}^{-}$ according to the cμ-rule we have $\theta_{n}(\eta_{n-1,m}^{-})=0$. Right after η _{n − 1,m} , queue n takes over the total residual resource capacity, which gives $\theta_{n}(\eta_{n-1,m}^{+})=1-\sum_{k=1}^{n-1} \frac{\alpha_{k}(\eta_{n-1,m})}{\mu_{k}}$ and $\Delta\theta_{n}(\eta _{n-1,m})=1-\sum_{k=1}^{n-1}\frac{\alpha_{k}(\eta_{n-1,m})}{\mu_{k}}$. Finally, by the cμ-rule, we have $\theta_{n}(\eta_{n,m}^{-})=1-\sum _{k=1}^{n-1}\frac{\alpha_{k}(\eta_{n,m})}{\mu_{k}}$. After η _n,m , queue n enters an EP and, according to Eq. 3, $\theta_{n}(\eta _{n,m}^{+})=\frac{\alpha_{n}(\eta_{n,m})}{\mu_{n}}$. It follows that $\Delta\theta_{n}(\eta_{n,m})=-\big(1-\sum_{k=1}^{n}\frac{\alpha_{k} (\eta_{n,m})}{\mu_{k}}\big).$ This completes part (b) of the proof.□

Proof of Lemma 3

Consider Eq. 12 with t _k  = t _m . By Eq. 10 we have $\frac{\partial x_{n}(t_{m}^{-})}{\partial\theta_{i,m}}=0$ for any i. Moreover, t _m is the start of a NEP because N _m  = N, i.e., all queues are non-empty in [t _m ,t _m + 1). Therefore, by Lemma 1, we have $\frac{\partial t_{m}}{\partial\theta_{i,m}}[f_{n}(t_{m}^{-})-f_{n}(t_{m} ^{+})]=0$ for all i = 1,...,N − 1. Finally, by Eqs. 3, 4, and 8 only f _i (τ) and f _N (τ) are dependent on θ _i,m (t) in the interval [t _m ,η _1,m ). Specifically, we have f _i (τ) = α _i (t) − μ _i θ _i,m (τ) and $f_{N}(\tau )=\alpha_{N}(\tau)-\mu_{N}(1-\sum_{n=1}^{N-1}\theta_{n,m}(\tau))$ resulting in $\frac{\partial f_{i}(\tau)}{\partial\theta_{i,m}}=-\mu_{i}$ and $\frac{\partial f_{N}(t)}{\partial\theta_{i,m}}=\mu_{N}$. The proof is complete by applying Eq. 12.□

Proof of Lemma 4

Let us define the set A = {η _1,m ,...,η _N,m }. In view of Eq. 12, we first show that $\frac{\partial t_{l}} {\partial\theta_{i,m}}[f_{n}(t_{l}^{-})-f_{n}(t_{l}^{+})]=0$ for any l ≥ m + 1 such that $t_{l}\not \in A$. There are only two possibilities for this:

Case 1

t _l  ∈ Σ _p , p = 1,...,k and k ≤ j. In this case, queue p which was emptied at some η _p,m  ≤ η _k,m becomes non-empty again, i.e., t _l is the start of an NEP. It then follows from Lemma 1 that $\frac{\partial t_{l}}{\partial\theta_{i,m} }[f_{n}(t_{l}^{-})-f_{n}(t_{l}^{+})]=0,$ for all i = 1,...,N − 1.

Case 2

t _l  ∈ Γ _p , p = 1,...,k and k ≤ j. This case is contingent upon the previous one, since it corresponds to the end of a NEP for a queue p with the NEP starting at some time $t_{l^{\ast} }<t_{l}$ such that $t_{l^{\ast}}\in(\eta_{k,m},\eta_{k+1,m})$ (as illustrated in Fig. 8) for some k ≤ j. We limit ourselves to only one such event occurring in (η _k,m ,η _k + 1,m ), thereby having $t_{l^{\ast}}=t_{l-1}$, since the extension to more events of this kind is straightforward. Since the cμ-rule applies for t ≥ η _1,m , the resource switches all its available capacity from queue k + 1 to queue p in the interval [t _l − 1,t _l ). Using Eq. 14, we will show that $\frac{\partial t_{l}}{\partial\theta_{i,m}}=0$. Since t _l − 1 is the start of a NEPp, by Lemma 1 we have $\frac{\partial t_{l-1}} {\partial\theta_{i,m}}[f_{n}(t_{l-1}^{-})-f_{n}(t_{l-1}^{+})]=0$. Moreover, x _p (t) = 0 for $t\in\lbrack\eta_{k,m},t_{l-1})$ and $\frac{\partial x_{p}(t_{l-1}^{-})}{\partial\theta_{i,m}}=0$. In addition, by Eqs. 4 and 5, for any $\tau\in\lbrack t_{l-1},t_{l})$, f _p (τ) is not a function of θ _i,m and we have $\frac{\partial f_{p}(\tau )}{\partial\theta_{i,m}}=0$. Utilizing these results in Eq. 12 with t _k  = t _l − 1 and n = p, we get

$$ \frac{\partial x_{p}(t_{l}^{-})}{\partial\theta_{i,m}}=\frac{\partial x_{p}(t_{l-1}^{-})}{\partial\theta_{i,m}}=0. $$

Using this in Eq. 14 gives $\frac{\partial t_{l}}{\partial\theta_{i,m} }=0$.

Fig. 8

Lemma 4—a re-start of a NEP of a higher priority queue in an interval [η _k,m ,η _k + 1,m )

We can now directly focus on proving Eq. 24. Let $t\in\lbrack \eta_{j,m},\eta_{j+1,m})$ and suppose the last NEP start or end event prior to t occurs at t _l  ≥ t _m + 1. Clearly, $\frac{\partial f_{n}(\tau )}{\partial\theta_{i,m}}=0$ for any $\tau\in\lbrack t_{l},t)$ and any n. Therefore, applying Eq. 12 at t _l gives

$$ \frac{\partial x_{n}(t)}{\partial\theta_{i,m}}=\frac{\partial x_{n}(t_{l} ^{-})}{\partial\theta_{i,m}}+\frac{\partial t_{l}}{\partial\theta_{i,m}} \left[f_{n}\left(t_{l}^{-}\right)-f_{n}\left(t_{l}^{+}\right)\right]\mathrm{,}\text{ \ }t\in\lbrack t_{l},t). $$

Based on our analysis of the two cases above, if $t_{l}\not \in A$ we get $\frac{\partial t_{l}}{\partial\theta_{i,m}}[f_{n}(t_{l}^{-})-f_{n}(t_{l} ^{+})]=0$ which yields $\frac{\partial x_{n}(t)}{\partial\theta_{i,m}} =\frac{\partial x_{n}(t_{l}^{-})}{\partial\theta_{i,m}}$. If t _l  ∈ A, since $t\in\lbrack\eta_{j,m},\eta_{j+1,m})$, it follows that t _l  ≡ η _j,m . In this case, by Assumption 1, α _n (t) is continuous at η _j,m and $f_{n}(\eta_{j,m}^{-})-f_{n}(\eta_{j,m} ^{+})=\mu_{n}\Delta\theta_{n}(\eta_{j,m})$. Regarding $\frac{\partial x_{n}(t_{l}^{-})}{\partial\theta_{i,m}}$ in the equation above, we look at the interval [t _l − 1,t _l ) and apply the same analysis at t _l − 1. It is now clear that by doing this recursively backwards in time we either encounter an event at t _r  < t _l − 1 with $\frac{\partial t_{r}}{\partial\theta_{i,m} }[f_{n}(t_{r}^{-})-f_{n}(t_{r}^{+})]=0$ or some η _k,m with k ≤ j, in which case a term μ _n Δθ _n (η _k,m ) is added on. The recursion ends at η _1,m where Eq. 24 is recovered and the lemma is proved.□

Proof of Lemma 5

The case where η _N,m  > T is trivial since after T the derivatives vanish. When η _N,m  ≤ T, recall that all queues become empty right after η _N,m , i.e., x _n (η _N,m ) = 0 for all n = 1,...,N and $\frac{\partial x_{n}(\eta_{N,m})}{\partial\theta_{i,m}} =0$. Since η _N,m  > t _m + 1, all functions f _n (τ) in Eq. 4 become independent of θ _i,m for every $\tau \in\lbrack\eta_{N,m},T)$. As a result, all x _n (t) are independent of θ _i,m for the same interval. Let us consider the event time derivatives $\frac{\partial t_{k}}{\partial\theta_{i,m}}$ for t _k  > η _N,m . If t _k  ∈ Λ _n or Σ _n , Eq. 13 or Lemma 1 gives $\frac{\partial t_{k}}{\partial\theta_{i,m}} [f_{n}(t_{k}^{-})-f_{n}(t_{k}^{+})]=0$ for any n and i. If t _k  ∈ Γ _n for some n, its associated NEP must have started at some t _l such that after η _N,m  < t _l  < t _k . Since, x _n (t) = 0 for $t\in \lbrack\eta_{n,m},t_{l})$, we get $\frac{\partial x_{n}(t)}{\partial \theta_{i,m}}=0$. Moreover, by Lemma 1 we have $\frac{\partial t_{l}}{\partial\theta_{i,m}}[f_{n}(t_{l}^{-})-f(t_{l}^{+})]=0$. Also, by the definition of θ _i,m (t), f _n (τ) is independent of θ _i,m (t) over the interval [t _l ,t _k ). Starting from t _l and using all these observations in Eq. 12, we find that $\frac{\partial x_{n} (t)}{\partial\theta_{i,m}}=0$ over the associated NEPn. Specifically, we find $\frac{\partial x_{n}(t_{k}^{-})}{\partial\theta_{i,m}}=0$. Using this in Eq. 14 implies that $\frac{\partial t_{k}}{\partial\theta_{i,m}}=0$ for all t _k  > η _N,m such that t _k  ∈ Γ _n for some n. Since the event time derivatives do not contribute a nonzero value for any k such that t _k  > η _N,m and since f _n (τ) is independent of θ _i,m for all $\tau\in\lbrack\eta_{N,m},T)$, the lemma’s statement is easily concluded by Eq. 12.□

Proof of Lemma 6

By Lemma 3 for n < i we get

$$ \frac{\partial x_{n}(\eta_{1,m}^{-})}{\partial\theta_{i,m}} =0.$$

(49)

Invoking Eq. 14 with t _k  = η _1,m immediately proves Eq. 25 for n = 1, i.e. $\frac{\partial\eta_{1,m}}{\partial \theta_{i,m}}=0.$. Considering n ≥ 2, using Lemma 4 for interval [η _{n − 1,m} ,η _n,m ) and $t=\eta_{n,m}^{-}$ we get

$$ \frac{\partial x_{n}(\eta_{n,m}^{-})}{\partial\theta_{i,m}}=\frac{\partial x_{n}(\eta_{1,m}^{-})}{\partial\theta_{i,m}}+\sum_{k=1}^{n-1}\frac {\partial\eta_{k,m}}{\partial\theta_{i,m}}\mu_{n}\Delta\theta_{n}(\eta _{k,m}).$$

(50)

By Eq. 49 the first term in the above equation is zero. Assuming i > 2, evaluating Eq. 50 for n = 2 gives

$$ \frac{\partial x_{2}(\eta_{2,m}^{-})}{\partial\theta_{i,m}}=\frac{\partial \eta_{1,m}}{\partial\theta_{i,m}}\mu_{2}\Delta\theta_{2}(\eta_{1,m})=0. $$

Inserting this result in Eq. 14 for t _k  = η _2,m implies that $\frac{\partial\eta_{2,m}}{\partial\theta_{i,m}}=0$. Repeating the same process for n = 3,...,i − 1 completes the proof of the lemma.□

Proof of Lemma 7

We proceed by considering the cases i = n and i < n separately.

• i = n: When n = 1, it is easy to verify the Lemma’s claim since, according to Eq. 14, we have

  $$ \frac{\partial\eta_{1,m}}{\partial\theta_{1,m}}=\frac{-\frac{\partial x_{1}(\eta_{1,m}^{-})}{\partial\theta_{1,m}}}{f_{1}(\eta_{1,m}^{-})} $$

  Multiplying both numerator and denominator by μ ₁ and using Eq. 26 gives

  $$ \frac{\partial\eta_{1,m}}{\partial\theta_{1,m}}=\frac{r_{1,m}}{\mu_{1}} \frac{\partial x_{1}(\eta_{1,m}^{-})}{\partial\theta_{1,m}}. $$

  When n > 1 we use Lemma 4 with j = n − 1 to get, for any $t\in\lbrack\eta_{n-1,m},\eta_{n,m})$,

  $$ \frac{\partial x_{n}(t)}{\partial\theta_{i,m}}=\frac{\partial x_{n}(\eta _{1,m}^{-})}{\partial\theta_{i,m}}+\sum_{k=1}^{n-1}\frac{\partial\eta_{k,m} }{\partial\theta_{i,m}}\mu_{n}\Delta\theta_{n}(\eta_{k,m}). $$

  Because i = n in each term of the sum above, we have k < i, hence, by Lemma 6, $\frac{\partial\eta_{k,m}}{\partial\theta_{i,m}}=0$ for k = 1,...,i − 1 and we conclude that

  $$ \frac{\partial x_{n}(t)}{\partial\theta_{i,m}}=\frac{\partial x_{n}(\eta _{1,m}^{-})}{\partial\theta_{i,m}}=\frac{\partial x_{n}(\eta_{1,m}^{-} )}{\partial\theta_{n,m}}. $$

  By Eq. 22 we have

  $$ \frac{\partial x_{n}(\eta_{1,m}^{-})}{\partial\theta_{n,m}}=\frac{\mu_{n}} {\mu_{1}}\frac{\partial x_{1}(\eta_{1,m}^{-})}{\partial\theta_{1,m}}. $$

  (51)

  Using this expression in Eq. 14 yields

  $$ \frac{\partial\eta_{n,m}}{\partial\theta_{n,m}}=\frac{-\mu_{n}}{f_{n} (\eta_{n,m}^{-})}\frac{1}{\mu_{1}}\frac{\partial x_{1}(\eta_{1,m}^{-} )}{\partial\theta_{1,m}}=\frac{r_{n,m}}{\mu_{1}}\frac{\partial x_{1} (\eta_{1,m}^{-})}{\partial\theta_{1,m}} $$

  (52)

  which completes the proof of the lemma for n = i.

• i < n: By Lemma 3, because n ≠ i we have $\frac{\partial x_{n}(\eta_{1,m}^{-})}{\partial\theta_{i,m}}=0$. Therefore, invoking Lemma 4, for any $t\in\lbrack \eta_{n-1,m},\eta_{n,m})$ we have:

  $$ \frac{\partial x_{n}(t)}{\partial\theta_{i,m}}=\sum_{k=1}^{n-1}\frac {\partial\eta_{k,m}}{\partial\theta_{i,m}}\mu_{n}\Delta\theta_{n}(\eta_{k,m}). $$

  However, by Lemma 6, $\frac{\partial\eta_{k,m}}{\partial \theta_{i,m}}=0$ when k < i. Therefore, the above equation reduces to

  $$ \frac{\partial x_{n}(t)}{\partial\theta_{i,m}}=\sum_{k=i}^{n-1}\frac {\partial\eta_{k,m}}{\partial\theta_{i,m}}\mu_{n}\Delta\theta_{n}(\eta _{k,m}). $$

  (53)

  Using this expression in Eq. 14 yields the event time derivative

  $$ \frac{\partial\eta_{n,m}}{\partial\theta_{i,m}}=r_{n,m}\sum_{k=i}^{n-1} \frac{\partial\eta_{k,m}}{\partial\theta_{i,m}}\Delta\theta_{n}(\eta _{k,m}). $$

  (54)

  According to part (b) of Lemma 2, Δθ _n (η _k,m ) can only be nonzero when k = n − 1. Thus, the above expression reduces to

  $$ \frac{\partial\eta_{n,m}}{\partial\theta_{i,m}}=r_{n,m}\frac{\partial \eta_{n-1,m}}{\partial\theta_{i,m}}\Delta\theta_{n}(\eta_{k,m} ). $$

  (55)

  This is clearly a recursive equation in n. Starting with n = i + 1 and proceeding forward in time we find that

  $$ \frac{\partial\eta_{n,m}}{\partial\theta_{i,m}}=\frac{\partial\eta_{i,m} }{\partial\theta_{i,m}}\prod_{k={i+1}}^{n}r_{k,m}\Delta\theta_{k}(\eta _{k-1,m}). $$

  Now using Eq. 52 with n replaced by i to find $\frac{\partial \eta_{i,m}}{\partial\theta_{i,m}}$ and inserting the result in the above equation gives the final result as

  $$ \frac{\partial\eta_{n,m}}{\partial\theta_{i,m}}=\frac{r_{i,m}}{\mu_{1}} \frac{\partial x_{1}(\eta_{1,m}^{-})}{\partial\theta_{1,m}}\prod_{k=i+1} ^{n}r_{k,m}\Delta\theta_{k}(\eta_{k-1,m}) $$

  (56)

  which completes the proof.□