Markov decision processes in service facilities holding perishable inventory

Abstract

In this article, we consider a single server queueing system with finite waiting space N (including one customer in service) and an inventory is attached with the maximum capacity S. The arrival of customer at the system is according to independent Poisson Processes with rate λ through a single channel. The service time is exponentially distributed with mean 1/μ and the item in stock has exponential life time with perishable rate γ(>0). When we place the order due to the demand of the customers, we assume that the lead time of procurement of item is exponentially distributed with parameter δ. Our object is to make a decision at each state of the system to operate the server by minimizing the entire service cost. The problem is modelled as a Markov decision problem by using the value iteration algorithm to obtain the minimal average cost of the service. The unique equilibrium probability distributions {p(q, i)} is also obtained by using Matrix geometric form in which the two dimensional state space contains infinite queue length and finite capacity of inventory. Numerical examples are provided to obtain the optimal average cost.

Appendices

Appendix A

Proof of Theorem 1: To prove the first inequality, choose any stationary policy R. By the definition of (Tv)_(q,i), we have for any state (q,i)∈E ₁ that

$$ {\left( {Tv} \right)_{{\left( {q,i} \right)}}} \leqslant {c_{{\left( {q,i} \right)}}}(a) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}(a){v_{{\left( {r,j} \right)}}},\quad \quad a \in A\left( {q,i} \right)} $$

(17)

where the equality sign holds for $a = {R_{{\left( {q,i} \right)}}}(v)$. Choosing a = R _(q,i) in (17) gives,

$$ {\left( {Tv} \right)_{{\left( {q,i} \right)}}} \leqslant {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}} \right) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right){v_{{\left( {r,j} \right)}}},\quad \quad \left( {q,i} \right) \in {E_1}} $$

(18)

Define the lower bound

$$ m = \mathop{{min}}\limits_{{\left( {q,i} \right) \in {E_1}}} \left\{ {{{\left( {Tv} \right)}_{{\left( {q,i} \right)}}} - {v_{{\left( {q,i} \right)}}}} \right\} $$

Since $m \leqslant {\left( {Tv} \right)_{{\left( {q,i} \right)}}} - {v_{{\left( {q,i} \right)}}}$ for all (q,i), it follows from (18) that $m + {v_{{\left( {q,i} \right)}}} \leqslant {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}} \right) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right){v_{{\left( {r,j} \right)}}}}$ for all (q,i)∈E ₁ and so

$$ {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}} \right) - m + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right){v_{{\left( {r,j} \right)}}} \geqslant {v_{{\left( {q,i} \right)}}}} $$

(19)

By the improvement theorem, Let g and ${v_{{(q,i)}}}$, $(q,i) \in {E_1}$, be given numbers. Suppose that the stationary policy $\overline R$ has the property

$$ {c_{{\left( {q,i} \right)}}}\left( {{{\overline R }_{{\left( {q,i} \right)}}}} \right) - g + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{{\overline R }_{{\left( {q,i} \right)}}}} \right){v_{{\left( {r,j} \right)}}} \leqslant {v_{{\left( {q,i} \right)}}}} $$

(20)

Then the long run average cost of policy $\overline R$ satisfies ${g_{{\left( {q,i} \right)}}}\left( {\overline R } \right) \leqslant g,\;\left( {q,i} \right) \in {E_1}$

Equation (19) gives that

$$ {g_{{\left( {q,i} \right)}}}(R) \geqslant m,\quad \quad \left( {q,i} \right) \in {E_1}. $$

This inequality holds for each policy R and so ${g^{*}} = \mathop{{min}}\nolimits_R {g_{{\left( {q,i} \right)}}}(R) \geqslant m$ proving the first inequality in (5). The proof of the last inequality in (5) is very similar.

By the definition of policy R(v)

$$ {\left( {Tv} \right)_{{\left( {q,i} \right)}}} = {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}(v)} \right) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {\left. {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right)(v)} \right){v_{{\left( {r,j} \right)}}},\;\left( {q,i} \right) \in {E_1}.} $$

(21)

Define the upper bound

$$ M = \mathop{{max}}\limits_{{\left( {q,i} \right) \in {E_1}}} \left\{ {{{\left( {Tv} \right)}_{{\left( {q,i} \right)}}} - {v_{{\left( {q,i} \right)}}}} \right\}. $$

Since $M \geqslant {\left( {Tv} \right)_{{\left( {q,i} \right)}}} - {v_{{\left( {q,i} \right)}}}$ for all $\left( {q,i} \right) \in {E_1}$, we obtain from (21) that

$$ {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}(v)} \right) - M + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}(v)} \right){v_{{\left( {r,j} \right)}}} \leqslant {v_{{\left( {q,i} \right)}}},\;\left( {q,i} \right) \in {E_1}.} $$

(22)

Hence by Eq. (20), ${g_{{\left( {q,i} \right)}}}\left( {R(v)} \right) \leqslant M$ for all $\left( {q,i} \right) \in {E_1}$, proving the last inequality in (5). This completes the proof.

Appendix B

Proof of Theorem 2: By the definition of policy R(n),

$$ {V_n}\left( {q,i} \right) = {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}(n)} \right) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {\left. {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right)(n)} \right){V_{{n - 1}}}\left( {r,j} \right),\;\left( {q,i} \right) \in {E_1}.} $$

(23)

In the same way as (18) was obtained, we find for any policy R that

$$ {c_{{\left( {q,i} \right)}}}\left( {{R_{{\left( {q,i} \right)}}}} \right) + \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right){V_{{n - 1}}}\left( {r,j} \right) \geqslant {V_n}\left( {q,i} \right),\left( {q,i} \right) \in {E_1}.} $$

(24)

Taking n = k in (23) and taking n = k + 1 and R = R(k) in (24) gives

$$ {V_{{k + 1}}}\left( {q,i} \right) - {V_k}\left( {q,i} \right) \leqslant \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {\left. {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right)(k)} \right)\left\{ {{V_k}\left( {r,j} \right) - {V_{{k - 1}}}\left( {r,j} \right)} \right\}} $$

(25)

Similarly, by taking n = k + 1 in (23) and taking n = k and $R = R\left( {k + 1} \right)$ in (24), we find

$$ {V_{{k + 1}}}\left( {q,i} \right) - {V_k}\left( {q,i} \right) \geqslant \sum\limits_{{\left( {r,j} \right) \in {E_1}}} {\left. {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{\left( {q,i} \right)}}}} \right)\left( {k + 1} \right)} \right)\left\{ {{V_k}\left( {r,j} \right) - {V_{{k - 1}}}\left( {r,j} \right)} \right\},} $$

(26)

Since ${V_k}\left( {r,j} \right) - {V_{{k - 1}}}\left( {r,j} \right) \leqslant {M_k}$ for all $\left( {r,j} \right) \in {E_1}$ and $\sum\limits_{{\left( {r,j} \right) \in {E_1}}} {\left. {p_{{\left( {q,i} \right)}}^{{\left( {r,j} \right)}}\left( {{R_{{(q,i)}}}} \right)(k)} \right) = 1,}$ it follows from (25) that ${V_{{k + 1}}}\left( {q,i} \right) - {V_k}\left( {q,i} \right) \leqslant {M_k}$ for all $\left( {q,i} \right) \in {E_1}$. This gives ${M_{{k + 1}}} \leqslant {M_k}$. Similarly, we obtain from (26) that ${m_{{k + 1}}} \geqslant {m_k}$.