Journal of Statistical Physics

, Volume 151, Issue 1–2, pp 254–276 | Cite as

Stochastic Approximation to Understand Simple Simulation Models

  • Segismundo S. IzquierdoEmail author
  • Luis R. Izquierdo


This paper illustrates how a deterministic approximation of a stochastic process can be usefully applied to analyse the dynamics of many simple simulation models. To demonstrate the type of results that can be obtained using this approximation, we present two illustrative examples which are meant to serve as methodological references for researchers exploring this area. Finally, we prove some convergence results for simulations of a family of evolutionary games, namely, intra-population imitation models in n-player games with arbitrary payoffs.


Stochastic approximation Mean dynamic Markov models Evolutionary games 



The authors gratefully acknowledge financial support from the Spanish Ministry of Education (JC2009-00263) and MICINN (CONSOLIDER-INGENIO 2010: CSD2010-00034, and DPI2010-16920). We also thank two anonymous reviewers for their useful comments.


  1. 1.
    Banisch, S., Lima, R., Araújo, T.: Agent based models and opinion dynamics as Markov chains. Social Networks. doi: 10.1016/j.socnet.2012.06.001
  2. 2.
    Barrat, A., Barthélemy, M., Vespignani, A.: Dynamical Processes on Complex Networks. Cambridge University Press, Cambridge (2008) zbMATHCrossRefGoogle Scholar
  3. 3.
    Beggs, A.W.: Stochastic evolution with slow learning. J. Econ. Theory 19(2), 379–405 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Beggs, A.W.: On the convergence of reinforcement learning. J. Econ. Theory 122(1), 1–36 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Beggs, A.W.: Large deviations and equilibrium selection in large populations. J. Econ. Theory 132(1), 383–410 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Benaïm, M., Le Boudec, J.-Y.: A class of mean field interaction models for computer and communication systems. Perform. Eval. 65(11–12), 823–838 (2008) CrossRefGoogle Scholar
  7. 7.
    Benaim, M., Weibull, J.W.: Deterministic approximation of stochastic evolution in games. Econometrica 71(3), 873–903 (2003) MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Benveniste, A., Priouret, P., Metivier, M.: Adaptive Algorithms and Stochastic Approximations. Springer, New York (1990) zbMATHCrossRefGoogle Scholar
  9. 9.
    Binmore, K.G., Samuelson, L., Vaughan, R.: Musical chairs: modeling noisy evolution. Games Econ. Behav. 11(1), 1–35 (1995) MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Börgers, T., Sarin, R.: Learning through reinforcement and replicator dynamics. J. Econ. Theory 77(1), 1–14 (1997) CrossRefGoogle Scholar
  11. 11.
    Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, New York (2008) Google Scholar
  12. 12.
    Borrelli, R.L., Coleman, C.S.: Differential Equations: A Modeling Approach. Prentice-Hall, Englewood Cliffs (1987) Google Scholar
  13. 13.
    Boylan, R.T.: Laws of large numbers for dynamical systems with randomly matched individuals. J. Econ. Theory 57(2), 473–504 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Boylan, R.T.: Continuous approximation of dynamical systems with randomly matched individuals. J. Econ. Theory 66(2), 615–625 (1995) MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Erev, I., Roth, A.E.: Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Am. Econ. Rev. 88(4), 848–881 (1998) Google Scholar
  16. 16.
    Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley Series in Probability and Statistics (2005) zbMATHGoogle Scholar
  17. 17.
    Fudenberg, D., Kreps, D.M.: Learning mixed equilibria. Games Econ. Behav. 5(3), 320–367 (1993) MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Fudenberg, D., Levine, D.K.: The Theory of Learning in Games. MIT Press, Cambridge (1998) zbMATHGoogle Scholar
  19. 19.
    Galán, J.M., Latek, M.M., Rizi, S.M.M.: Axelrod’s metanorm games on networks. PLoS ONE 6(5), e20474 (2011) ADSCrossRefGoogle Scholar
  20. 20.
    Gilbert, N., Troitzsch, K.G.: Simulation for the Social Scientist. McGraw-Hill, New York (2005) Google Scholar
  21. 21.
    Gintis, H.: Markov models of social dynamics: theory and applications. ACM Trans. Intell. Syst. Technol. (in press). Available at,
  22. 22.
    Gleeson, J.P., Melnik, S., Ward, J.A., Porter, M.A., Mucha, P.J.: Accuracy of mean-field theory for dynamics on real-world networks. Phys. Rev. E 85(2), 026106 (2012) ADSCrossRefGoogle Scholar
  23. 23.
    Hairer, E., Nørsett, S., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (1993) zbMATHGoogle Scholar
  24. 24.
    Hofbauer, J., Sandholm, W.H.: On the global convergence of stochastic fictitious play. Econometrica 70(6), 2265–2294 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Hopkins, E.: Two competing models of how people learn in games. Econometrica 70(6), 2141–2166 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Huet, S., Deffuant, G.: Differential equation models derived from an individual-based model can help to understand emergent effects. J. Artif. Soc. Soc. Simul. 11(2), 10 (2008) MathSciNetGoogle Scholar
  27. 27.
    Izquierdo, L.R., Izquierdo, S.S., Gotts, N.M., Polhill, J.G.: Transient and asymptotic dynamics of reinforcement learning in games. Games Econ. Behav. 61(2), 259–276 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Izquierdo, S.S., Izquierdo, L.R., Gotts, N.M.: Reinforcement learning dynamics in social dilemmas. J. Artif. Soc. Soc. Simul. 11(2), 1 (2008) Google Scholar
  29. 29.
    Izquierdo, L., Izquierdo, S., Galán, J.M., Santos, J.I.: Techniques to understand computer simulations: Markov chain analysis. J. Artif. Soc. Soc. Simul. 12(1), 6 (2009) Google Scholar
  30. 30.
    Izquierdo, S.S., Izquierdo, L.R., Vega-Redondo, F.: The option to leave: conditional dissociation in the evolution of cooperation. J. Theor. Biol. 267(1), 76–84 (2010) MathSciNetCrossRefGoogle Scholar
  31. 31.
    Kotliar, G., Savrasov, S.Y., Haule, K., Oudovenko, V.S., Parcollet, O., Marianetti, C.A.: Electronic structure calculations with dynamical mean-field theory. Rev. Mod. Phys. 78(3), 865–951 (2006) ADSCrossRefGoogle Scholar
  32. 32.
    Kulkarni, V.G.: Modeling and Analysis of Stochastic Systems. Chapman & Hall/CRC, London (1995) zbMATHGoogle Scholar
  33. 33.
    Kushner, H.: Stochastic approximation: a survey. WIREs: Comput. Stat. 2(1), 87–96 (2010) CrossRefGoogle Scholar
  34. 34.
    Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications. Springer, New York (2003) zbMATHGoogle Scholar
  35. 35.
    Lambert, M.F.: Numerical Methods for Ordinary Differential Systems. Wiley, Chichester (1991) zbMATHGoogle Scholar
  36. 36.
    Ljung, L.: Analysis of recursive stochastic algorithms. IEEE Trans. Autom. Control AC-22 (4), 551–575 (1977) MathSciNetCrossRefGoogle Scholar
  37. 37.
    López-Pintado, D.: Diffusion in complex social networks. Games Econ. Behav. 62(2), 573–590 (2008) zbMATHCrossRefGoogle Scholar
  38. 38.
    Macy, M.W., Flache, A.: Learning dynamics in social dilemmas. Proc. Natl. Acad. Sci. USA 99(3), 7229–7236 (2002) ADSCrossRefGoogle Scholar
  39. 39.
    Morozov, A., Poggiale, J.C.: From spatially explicit ecological models to mean-field dynamics: the state of the art and perspectives. Ecol. Complex. 10, 1–11 (2012) CrossRefGoogle Scholar
  40. 40.
    Norman, M.F.: Some convergence theorems for stochastic learning models with distance diminishing operators. J. Math. Psychol. 5(1), 61–101 (1968) MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Norman, M.F.: Markov Processes and Learning Models. Academic Press, New York (1972) zbMATHGoogle Scholar
  42. 42.
    Perc, M., Szolnoki, A.: Coevolutionary games—a mini review. Biosystems 99(2), 109–125 (2010) CrossRefGoogle Scholar
  43. 43.
    Rozonoer, L.I.: On deterministic approximation of Markov processes by ordinary differential equations. Math. Probl. Eng. 4(2), 99–114 (1998) zbMATHCrossRefGoogle Scholar
  44. 44.
    Sandholm, W.H.: Deterministic evolutionary dynamics. In: Durlauf, S.N., Blume, L.E. (eds.) The New Palgrave Dictionary of Economics, vol. 2. Palgrave Macmillan, New York (2008) Google Scholar
  45. 45.
    Sandholm, W.H.: Stochastic imitative game dynamics with committed agents. J. Econ. Theory 147(5), 2056–2071 (2012) MathSciNetzbMATHCrossRefGoogle Scholar
  46. 46.
    Szabó, G., Fáth, G.: Evolutionary games on graphs. Phys. Rep. 446(4–6), 97–216 (2007) MathSciNetADSCrossRefGoogle Scholar
  47. 47.
    Vega-Redondo, F.: Complex Social Networks. Cambridge University Press, Cambridge (2007) zbMATHCrossRefGoogle Scholar
  48. 48.
    Vega-Redondo, F.: Economics and the Theory of Games. Cambridge University Press, Cambridge (2003) CrossRefGoogle Scholar
  49. 49.
    Weiss, P.: L’hypothèse du champ moléculaire et la propriété ferromagnétique. J. Phys. 6, 661–690 (1907) zbMATHGoogle Scholar
  50. 50.
    Young, H.P.: Individual Strategy and Social Structure: An Evolutionary Theory of Institutions. Princeton University Press, Princeton (1998) Google Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Segismundo S. Izquierdo
    • 1
    • 2
    • 4
  • Luis R. Izquierdo
    • 2
    • 3
  1. 1.Department of Industrial Organization, EIIUniversidad de ValladolidValladolidSpain
  2. 2.InSiSoc, Social Systems Engineering CentreValladolidSpain
  3. 3.Department of Civil EngineeringUniversidad de BurgosBurgosSpain
  4. 4.Departamento de Organización de EmpresasEscuela de Ingenierías IndustrialesValladolidSpain

Personalised recommendations