Fluid Approximations to Markov Decision Processes with Local Transitions

  • Alexey PiunovskiyEmail author
  • Yi Zhang
Part of the Systems & Control: Foundations & Applications book series (SCFA)


In this chapter we indicate that when it is scaled properly, an absorbing Markov decision process (MDP) model with an expected total undiscounted cost can be approximated by two of its deterministic and continuous analogues, namely, the standard fluid model and the refined fluid model, at least when the scaling parameter \(n = 1,2,\ldots \) grows large. We obtain the level of accuracy of such fluid approximations by showing that the absolute difference between the objective function of the scaled MDP model and the one of the (standard and refined) fluid model goes to zero as fast as \(\frac{1} {n}.\) Under some extra conditions, we obtain that given a particular type of policy solving the refined fluid model, it can be translated into a policy, which is nearly optimal for the scaled MDP model.



Mr. Mantas Vykertas kindly helped us improve the English presentation of this chapter. We thank the referee for valuable comments, too.


  1. 1.
    Altman, E.: Constrained Markov Decision Processes. Chapman and Hall/CRC, Boca Raton (1999)zbMATHGoogle Scholar
  2. 2.
    Avrachenkov, K., Ayesta, U., Piunovskiy, A.: Convergence of trajectories and optimal buffer sizing for AIMD congestion control. Perform. Evaluation. 67, 501–527 (2010)CrossRefGoogle Scholar
  3. 3.
    Avrachenkov, K., Piunovskiy, A., Zhang, Y.: Asymptotic fluid optimality and efficiency of tracking policy for bandwidth-sharing networks. J. Appl. Probab. bf 48, 90–113 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Bäuerle, N.: Optimal control of queueing networks: an approach via fluid models. Adv. Appl. Prob. 34, 313–328 (2002)zbMATHCrossRefGoogle Scholar
  5. 5.
    Bertsekas, D., Shreve, S.: Stochastic Optimal Control. Academic Press, NY (1978)zbMATHGoogle Scholar
  6. 6.
    Jacko, P., Sansó, B.: Optimal anticipative congestion control of flows with time-varying input stream. Perform. Evaluation. 69, 86–101 (2012)CrossRefGoogle Scholar
  7. 7.
    Hernández-Lerma, O., Lasserre, J.: Discrete-time Markov Control Processes. Springer-Verlag, NY (1996)CrossRefGoogle Scholar
  8. 8.
    Dai, J.: On positive Harris recurrence of multiclass queueing networks: a unified approach via fluid limit models. Ann. Appl. Prob. 5, 49–77 (1995)zbMATHCrossRefGoogle Scholar
  9. 9.
    Foss, S., Kovalevskii, A.: A stability criterion via fluid limits and its application to a Polling system. Queueing. Syst. 32, 131–168 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Mandelbaum, A., Pats, G.: State-dependent queues: approximations and applications. In Kelly, F., Williams, R. (eds.) Stochastic Networks, pp. 239–282. Springer, NY (1995)Google Scholar
  11. 11.
    Piunovskiy, A., Clancy, D.: An explicit optimal intervention policy for a deterministic epidemic model. Optim. Contr. Appl. Met. 29, 413–428 (2008)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Piunovskiy, A.: Random walk, birth-and-death process and their fluid approximations: absorbing case. Math. Meth. Oper. Res. 70, 285–312 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Piunovskiy, A., Zhang, Y.: Accuracy of fluid approximations to controlled Birth-and-Death processes: absorbing case. Math. Meth. Oper. Res. 73, 159–187 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Piunovskiy, A., Zhang, Y.: On the fluid approximations of a class of general inventory level-dependent EOQ and EPQ models. Adv. Oper. Res. (2011) doi: 10.1155/2011/301205MathSciNetGoogle Scholar
  15. 15.
    Puterman, M.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, NY (1994)zbMATHGoogle Scholar
  16. 16.
    Zhang, Y., Piunovskiy, A., Ayesta, U., Avrachenkov, K.: Convergence of trajectories and optimal buffer sizing for MIMD congestion control. Com. Com. 33, 149–159 (2010)zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of Mathematical SciencesUniversity of LiverpoolLiverpoolUK

Personalised recommendations