Journal of Statistical Physics

, Volume 151, Issue 1–2, pp 254–276

Stochastic Approximation to Understand Simple Simulation Models



This paper illustrates how a deterministic approximation of a stochastic process can be usefully applied to analyse the dynamics of many simple simulation models. To demonstrate the type of results that can be obtained using this approximation, we present two illustrative examples which are meant to serve as methodological references for researchers exploring this area. Finally, we prove some convergence results for simulations of a family of evolutionary games, namely, intra-population imitation models in n-player games with arbitrary payoffs.


Stochastic approximation Mean dynamic Markov models Evolutionary games 


  1. 1.
    Banisch, S., Lima, R., Araújo, T.: Agent based models and opinion dynamics as Markov chains. Social Networks. doi:10.1016/j.socnet.2012.06.001
  2. 2.
    Barrat, A., Barthélemy, M., Vespignani, A.: Dynamical Processes on Complex Networks. Cambridge University Press, Cambridge (2008) MATHCrossRefGoogle Scholar
  3. 3.
    Beggs, A.W.: Stochastic evolution with slow learning. J. Econ. Theory 19(2), 379–405 (2002) MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Beggs, A.W.: On the convergence of reinforcement learning. J. Econ. Theory 122(1), 1–36 (2005) MathSciNetMATHCrossRefGoogle Scholar
  5. 5.
    Beggs, A.W.: Large deviations and equilibrium selection in large populations. J. Econ. Theory 132(1), 383–410 (2007) MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Benaïm, M., Le Boudec, J.-Y.: A class of mean field interaction models for computer and communication systems. Perform. Eval. 65(11–12), 823–838 (2008) CrossRefGoogle Scholar
  7. 7.
    Benaim, M., Weibull, J.W.: Deterministic approximation of stochastic evolution in games. Econometrica 71(3), 873–903 (2003) MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Benveniste, A., Priouret, P., Metivier, M.: Adaptive Algorithms and Stochastic Approximations. Springer, New York (1990) MATHCrossRefGoogle Scholar
  9. 9.
    Binmore, K.G., Samuelson, L., Vaughan, R.: Musical chairs: modeling noisy evolution. Games Econ. Behav. 11(1), 1–35 (1995) MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Börgers, T., Sarin, R.: Learning through reinforcement and replicator dynamics. J. Econ. Theory 77(1), 1–14 (1997) CrossRefGoogle Scholar
  11. 11.
    Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, New York (2008) Google Scholar
  12. 12.
    Borrelli, R.L., Coleman, C.S.: Differential Equations: A Modeling Approach. Prentice-Hall, Englewood Cliffs (1987) Google Scholar
  13. 13.
    Boylan, R.T.: Laws of large numbers for dynamical systems with randomly matched individuals. J. Econ. Theory 57(2), 473–504 (1992) MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Boylan, R.T.: Continuous approximation of dynamical systems with randomly matched individuals. J. Econ. Theory 66(2), 615–625 (1995) MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Erev, I., Roth, A.E.: Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Am. Econ. Rev. 88(4), 848–881 (1998) Google Scholar
  16. 16.
    Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley Series in Probability and Statistics (2005) MATHGoogle Scholar
  17. 17.
    Fudenberg, D., Kreps, D.M.: Learning mixed equilibria. Games Econ. Behav. 5(3), 320–367 (1993) MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    Fudenberg, D., Levine, D.K.: The Theory of Learning in Games. MIT Press, Cambridge (1998) MATHGoogle Scholar
  19. 19.
    Galán, J.M., Latek, M.M., Rizi, S.M.M.: Axelrod’s metanorm games on networks. PLoS ONE 6(5), e20474 (2011) ADSCrossRefGoogle Scholar
  20. 20.
    Gilbert, N., Troitzsch, K.G.: Simulation for the Social Scientist. McGraw-Hill, New York (2005) Google Scholar
  21. 21.
    Gintis, H.: Markov models of social dynamics: theory and applications. ACM Trans. Intell. Syst. Technol. (in press). Available at,
  22. 22.
    Gleeson, J.P., Melnik, S., Ward, J.A., Porter, M.A., Mucha, P.J.: Accuracy of mean-field theory for dynamics on real-world networks. Phys. Rev. E 85(2), 026106 (2012) ADSCrossRefGoogle Scholar
  23. 23.
    Hairer, E., Nørsett, S., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (1993) MATHGoogle Scholar
  24. 24.
    Hofbauer, J., Sandholm, W.H.: On the global convergence of stochastic fictitious play. Econometrica 70(6), 2265–2294 (2002) MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    Hopkins, E.: Two competing models of how people learn in games. Econometrica 70(6), 2141–2166 (2002) MathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    Huet, S., Deffuant, G.: Differential equation models derived from an individual-based model can help to understand emergent effects. J. Artif. Soc. Soc. Simul. 11(2), 10 (2008) MathSciNetGoogle Scholar
  27. 27.
    Izquierdo, L.R., Izquierdo, S.S., Gotts, N.M., Polhill, J.G.: Transient and asymptotic dynamics of reinforcement learning in games. Games Econ. Behav. 61(2), 259–276 (2007) MathSciNetMATHCrossRefGoogle Scholar
  28. 28.
    Izquierdo, S.S., Izquierdo, L.R., Gotts, N.M.: Reinforcement learning dynamics in social dilemmas. J. Artif. Soc. Soc. Simul. 11(2), 1 (2008) Google Scholar
  29. 29.
    Izquierdo, L., Izquierdo, S., Galán, J.M., Santos, J.I.: Techniques to understand computer simulations: Markov chain analysis. J. Artif. Soc. Soc. Simul. 12(1), 6 (2009) Google Scholar
  30. 30.
    Izquierdo, S.S., Izquierdo, L.R., Vega-Redondo, F.: The option to leave: conditional dissociation in the evolution of cooperation. J. Theor. Biol. 267(1), 76–84 (2010) MathSciNetCrossRefGoogle Scholar
  31. 31.
    Kotliar, G., Savrasov, S.Y., Haule, K., Oudovenko, V.S., Parcollet, O., Marianetti, C.A.: Electronic structure calculations with dynamical mean-field theory. Rev. Mod. Phys. 78(3), 865–951 (2006) ADSCrossRefGoogle Scholar
  32. 32.
    Kulkarni, V.G.: Modeling and Analysis of Stochastic Systems. Chapman & Hall/CRC, London (1995) MATHGoogle Scholar
  33. 33.
    Kushner, H.: Stochastic approximation: a survey. WIREs: Comput. Stat. 2(1), 87–96 (2010) CrossRefGoogle Scholar
  34. 34.
    Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications. Springer, New York (2003) MATHGoogle Scholar
  35. 35.
    Lambert, M.F.: Numerical Methods for Ordinary Differential Systems. Wiley, Chichester (1991) MATHGoogle Scholar
  36. 36.
    Ljung, L.: Analysis of recursive stochastic algorithms. IEEE Trans. Autom. Control AC-22 (4), 551–575 (1977) MathSciNetCrossRefGoogle Scholar
  37. 37.
    López-Pintado, D.: Diffusion in complex social networks. Games Econ. Behav. 62(2), 573–590 (2008) MATHCrossRefGoogle Scholar
  38. 38.
    Macy, M.W., Flache, A.: Learning dynamics in social dilemmas. Proc. Natl. Acad. Sci. USA 99(3), 7229–7236 (2002) ADSCrossRefGoogle Scholar
  39. 39.
    Morozov, A., Poggiale, J.C.: From spatially explicit ecological models to mean-field dynamics: the state of the art and perspectives. Ecol. Complex. 10, 1–11 (2012) CrossRefGoogle Scholar
  40. 40.
    Norman, M.F.: Some convergence theorems for stochastic learning models with distance diminishing operators. J. Math. Psychol. 5(1), 61–101 (1968) MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    Norman, M.F.: Markov Processes and Learning Models. Academic Press, New York (1972) MATHGoogle Scholar
  42. 42.
    Perc, M., Szolnoki, A.: Coevolutionary games—a mini review. Biosystems 99(2), 109–125 (2010) CrossRefGoogle Scholar
  43. 43.
    Rozonoer, L.I.: On deterministic approximation of Markov processes by ordinary differential equations. Math. Probl. Eng. 4(2), 99–114 (1998) MATHCrossRefGoogle Scholar
  44. 44.
    Sandholm, W.H.: Deterministic evolutionary dynamics. In: Durlauf, S.N., Blume, L.E. (eds.) The New Palgrave Dictionary of Economics, vol. 2. Palgrave Macmillan, New York (2008) Google Scholar
  45. 45.
    Sandholm, W.H.: Stochastic imitative game dynamics with committed agents. J. Econ. Theory 147(5), 2056–2071 (2012) MathSciNetMATHCrossRefGoogle Scholar
  46. 46.
    Szabó, G., Fáth, G.: Evolutionary games on graphs. Phys. Rep. 446(4–6), 97–216 (2007) MathSciNetADSCrossRefGoogle Scholar
  47. 47.
    Vega-Redondo, F.: Complex Social Networks. Cambridge University Press, Cambridge (2007) MATHCrossRefGoogle Scholar
  48. 48.
    Vega-Redondo, F.: Economics and the Theory of Games. Cambridge University Press, Cambridge (2003) CrossRefGoogle Scholar
  49. 49.
    Weiss, P.: L’hypothèse du champ moléculaire et la propriété ferromagnétique. J. Phys. 6, 661–690 (1907) MATHGoogle Scholar
  50. 50.
    Young, H.P.: Individual Strategy and Social Structure: An Evolutionary Theory of Institutions. Princeton University Press, Princeton (1998) Google Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Segismundo S. Izquierdo
    • 1
    • 2
    • 4
  • Luis R. Izquierdo
    • 2
    • 3
  1. 1.Department of Industrial Organization, EIIUniversidad de ValladolidValladolidSpain
  2. 2.InSiSoc, Social Systems Engineering CentreValladolidSpain
  3. 3.Department of Civil EngineeringUniversidad de BurgosBurgosSpain
  4. 4.Departamento de Organización de EmpresasEscuela de Ingenierías IndustrialesValladolidSpain

Personalised recommendations