Mean-Field Analysis of Markov Models with Reward Feedback

  • Anton Stefanek
  • Richard A. Hayden
  • Mark Mac Gonagle
  • Jeremy T. Bradley
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7314)


We extend the population continuous time Markov chain formalism so that the state space is augmented with continuous variables accumulated over time as functions of component populations. System feedback can be expressed using accumulations that in turn can influence the Markov chain behaviour via functional transition rates. We show how to obtain mean-field differential equations capturing means and higher-order moments of the discrete populations and continuous accumulation variables. We also provide first- and second-order convergence results and suggest a novel normal moment closure that can greatly improve the accuracy of means and higher moments.

We demonstrate how such a framework is suitable for modelling feedback from globally-accumulated quantities such as energy consumption, cost or temperature. Finally, we present a worked example modelling a hypothetical heterogeneous computing cluster and its interaction with air conditioning units.


Markov Model Process Algebra Discrete Population Discrete State Space Component Population 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Billingsley, P.: Convergence of Probability Measures. John Wiley & Sons (1968)Google Scholar
  2. 2.
    Bortolussi, L.: Hybrid Limits of Continuous Time Markov Chains. In: QEST 2011, pp. 3–12. IEEE, Aachen (2011)Google Scholar
  3. 3.
    Bortolussi, L., Galpin, V., Hillston, J.: HYPE with stochastic events. EPTCS 57, 120–133 (2011)CrossRefGoogle Scholar
  4. 4.
    Cain, M.: The moment-generating function of the minimum of bivariate normal random variables. The American Statistician 48(2), 124–125 (1994)MathSciNetGoogle Scholar
  5. 5.
    Davis, M.H.A.: Markov models and optimization. Chapman & Hall/CRC (1993)Google Scholar
  6. 6.
    Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley (2005)Google Scholar
  7. 7.
    Gast, N., Bruno, G.: A mean field model of work stealing in large-scale systems. In: SIGMETRICS, p. 13. ACM, New York (2010)Google Scholar
  8. 8.
    Gast, N., Gaujal, B.: Mean field limit of non-smooth systems and differential inclusions. SIGMETRICS Performance Evaluation Review 38(2), 30–32 (2010)CrossRefGoogle Scholar
  9. 9.
    Gillespie, C.S.: Moment-closure approximations for mass-action models. IET Systems Biology 3(1), 52–58 (2009)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hayden, R.A., Bradley, J.T.: A fluid analysis framework for a Markovian process algebra. Theoretical Computer Science 411(22-24), 2260–2297 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Hayden, R.A., Stefanek, A., Bradley, J.T.: Fluid computation of passage-time distributions in large Markov models. Theoretical Computer Science 413(1), 106–141 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Hillston, J.: Fluid flow approximation of PEPA models. In: QEST, pp. 33–42. IEEE (September 2005)Google Scholar
  13. 13.
    Horton, G., Kulkarni, V.G., Nicol, D.M., Trivedi, K.S.: Fluid stochastic Petri nets: Theory, applications, and solution techniques. European Journal of Operational Research 105(1), 184–201 (1998)zbMATHCrossRefGoogle Scholar
  14. 14.
    Kallenberg, O.: Foundations of Modern Probability. Springer (2002)Google Scholar
  15. 15.
    Khadim, U.: A comparative study of process algebras for hybrid systems. Computer Science Report 06–23, Technische Universiteit Eindhoven (2006)Google Scholar
  16. 16.
    Klebaner, F.C.: Introduction to stochastic calculus with applications, 2nd edn. Imperial College Press (2006)Google Scholar
  17. 17.
    Rawson, A., Pfleuger, J., Cader, T.: Data Center Power Efficiency Metrics: PUE and DCiE. The Green Grid (2007)Google Scholar
  18. 18.
    de Souza e Silva, E., Gail, R.: An algorithm to calculate transient distributions of cummulative rate and impulse-based rewards. Communications in Statistics: Stochastic Models 14(3), 509–536 (1998)Google Scholar
  19. 19.
    Silva, M., Júlvez, J., Mahulea, C., Vázquez, C.R.: On fluidization of discrete event models: observation and control of continuous Petri nets. Discrete Event Dynamic Systems 21(4), 427–497 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Stefanek, A., Hayden, R.A., Bradley, J.T.: A new tool for the performance analysis of massively parallel computer systems. In: QAPL. Electronic Proceedings in Theoretical Computer Science (March 2010)Google Scholar
  21. 21.
    Stefanek, A., Hayden, R.A., Bradley, J.T.: Fluid Analysis of Energy Consumption using Rewards in Massively Parallel Markov Models. In: Computing, p. 121. ACM Press (2011)Google Scholar
  22. 22.
    Stefanek, A., Hayden, R.A., Bradley, J.T.: GPA – A Tool for Fluid Scalability Analysis of Massively Parallel Systems. In: QEST, pp. 147–148. IEEE (September 2011)Google Scholar
  23. 23.
    Tang, Q., Gupta, S., Varsamopoulos, G.: Energy-efficient thermal-aware task scheduling for homogeneous high-performance computing data centers: A cyber-physical approach. IEEE Transactions on Parallel and Distributed Systems 19(11), 1458–1472 (2008)CrossRefGoogle Scholar
  24. 24.
    Telek, M., Rácz, S.: Numerical analysis of large Markovian reward models. Performance Evaluation, 36–37, 95–114 (August 1999)Google Scholar
  25. 25.
    Tribastone, M., Gilmore, S., Hillston, J.: Scalable Differential Analysis of Process Algebra Models. IEEE Transactions on Software Engineering 38(1), 205–219 (2012)CrossRefGoogle Scholar
  26. 26.
    Whitt, W.: Internet supplement to Stochastic-Process Limits (2002),

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Anton Stefanek
    • 1
  • Richard A. Hayden
    • 1
  • Mark Mac Gonagle
    • 1
  • Jeremy T. Bradley
    • 1
  1. 1.Department of ComputingImperial College LondonLondonUK

Personalised recommendations