Skip to main content
Log in

Probabilistic Approach to Finite State Mean Field Games

  • Published:
Applied Mathematics & Optimization Submit manuscript

Abstract

We study mean field games and corresponding N-player games in continuous time over a finite time horizon where the position of each agent belongs to a finite state space. As opposed to previous works on finite state mean field games, we use a probabilistic representation of the system dynamics in terms of stochastic differential equations driven by Poisson random measures. Under mild assumptions, we prove existence of solutions to the mean field game in relaxed open-loop as well as relaxed feedback controls. Relying on the probabilistic representation and a coupling argument, we show that mean field game solutions provide symmetric \(\varepsilon _N\)-Nash equilibria for the N-player game, both in open-loop and in feedback strategies (not relaxed), with \(\varepsilon _N\le \frac{\text {constant}}{\sqrt{N}}.\) Under stronger assumptions, we also find solutions of the mean field game in ordinary feedback controls and prove uniqueness either in case of a small time horizon or under monotonicity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Here only the open-loop part of the chattering lemma is needed, which is well known, and so we postpone the proof of the lemma to Sect. 5, where we also give the feedback part.

References

  1. Aldous, D.: Stopping times and tightness. Ann. Probab. 6(2), 335–340 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  2. Basna, R., Hilbert, A., Kolokoltsov, V.N.: An epsilon-Nash equilibrium for non-linear Markov games of mean-field-type on finite spaces. Commun. Stoch. Anal. 8(4), 449–468 (2014)

    MathSciNet  Google Scholar 

  3. Bayraktar, E., Cohen, A.: Analysis of a finite state many player game using its master equation. arXiv:1707.02648 [math.AP] (July, 2017)

  4. Benazzoli, C., Campi, L., Di Persio, L.: Mean-field games with controlled jumps. arXiv:1703.01919 [math.PR] (March, 2017)

  5. Bensoussan, A., Frehse, J., Yam, P.: Mean Field Games and Mean Field Type Control Theory. Springer Briefs in Mathematics. Springer, New York (2013)

    Book  MATH  Google Scholar 

  6. Bensoussan, A., Sung, K., Yam, S., Yung, S.: Linear-quadratic mean field games. J. Optim. Theory Appl. 169(2), 496–529 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  7. Brunick, G., Shreve, S.E.: Mimicking an Itô process by a solution of a stochastic differential equation. Ann. Appl. Probab. 23(4), 1584–1628 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cardaliaguet, P.: Notes on Mean Field Games. Technical report. Université de Paris - Dauphine (September, 2013)

  9. Cardaliaguet, P., Delarue, F., Lasry, J.-M., Lions, P.-L.: The master equation and the convergence problem in mean field games. arXiv:1509.02505 [math.AP] (September, 2015)

  10. Carmona, R., Delarue, F.: Probabilistic analysis of mean-field games. SIAM J. Control Optim. 51(4), 2705–2734 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Carmona, R., Lacker, D.: A probabilistic weak formulation of mean field games and applications. Ann. Appl. Probab. 25(3), 1189–1231 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  12. Carmona, R., Wang, P.: Finite state mean field games with major and minor players. arXiv:1610.05408 [math.PR] (October, 2016)

  13. Carmona, R., Delarue, F., Lachapelle, A.: Control of McKean-Vlasov dynamics versus mean field games. Math. Financ. Econ. 7(2), 131–166 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  14. Cecchin, A., Pelino, G.: Convergence, fluctuations and large deviations for finite state mean field games via the master equation. arXiv:1707.01819 [math.AP] (July, 2017)

  15. Doncel, J., Gast, N., Gaujal, B.: Mean-field games with explicit interactions. Preprint hal-01277098 (February, 2016)

  16. Fan, K.: Fixed-point and minimax theorems in locally convex topological linear spaces. Proc. Natl Acad. Sci. USA 38, 121–126 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  17. Fischer, M.: On the connection between symmetric \(N\)-player games and mean field games. Ann. Appl. Probab. 27(2), 757–810 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  18. Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions. Applications of Mathematics, vol. 25, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  19. Fournier, N., Guillin, A.: On the rate of convergence in Wasserstein distance of the empirical measure. Probab. Theory Relat. Fields 162, 707–738 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  20. Gomes, D.A., Mohr, J., Souza, R.R.: Continuous time finite state mean field games. Appl. Math. Optim. 68(1), 99–143 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  21. Gomes, D.A., Nurbekyan, L., Pimentel, E.A.: Economic Models and Mean-Field Games Theory. \(30^\circ \) Colóquio Brasileiro de Matemática. IMPA, Rio de Janeiro (2015)

  22. Graham, C.: McKean-Vlasov Itô–Skorohod equations, and nonlinear diffusions with discrete jump sets. Stoch. Process. Appl. 40(1), 69–82 (1992)

    Article  MATH  Google Scholar 

  23. Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Applications of Mathematics, vol. 30. Springer, New York (1996)

    Book  MATH  Google Scholar 

  24. Huang, M., Malhamé, R.P., Caines, P.E.: Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. Commun. Inf. Syst. 6(3), 221–252 (2006)

    MathSciNet  MATH  Google Scholar 

  25. Ikeda, N., Watanabe, S.: Stochastic Differential Equations and Diffusion Processes. North-Holland Mathematical Library, vol. 24, 2nd edn. North Holland/Kodansha, Amsterdam (1989)

    MATH  Google Scholar 

  26. Jacod, J.: Multivariate point processes: predictable projection, Radon–Nikodym derivatives, representation of martingales. Z. Wahrscheinlichkeitstheorie verw. Geb. 31, 235–253 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  27. Jacod, J.: Calcul Stochastique et Problèmes de Martingales. Lecture Notes in Mathematics, vol. 714. Springer, Berlin (1979)

    Book  MATH  Google Scholar 

  28. Kallenberg, O.: Random Measures, 4th edn. Akademie Verlag, Berlin (1986)

    MATH  Google Scholar 

  29. Kallenberg, O.: Foundations of Modern Probability, 2nd edn. Springer, Berlin (2001)

    MATH  Google Scholar 

  30. Kolokoltsov, V.N., Li, J., Yang, W.: Mean field games and nonlinear Markov processes. arXiv:1112.3744 [math.PR] (2011)

  31. Kushner, H.J.: Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems. Systems and Control: Foundations and Applications, vol. 3. Birkhäuser, Boston (1990)

    Book  MATH  Google Scholar 

  32. Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time. Applications of Mathematics, vol. 24, 2nd edn. Springer, New York (2001)

    Book  MATH  Google Scholar 

  33. Lacker, D.: Mean field games via controlled martingale problems: existence of Markovian equilibria. Stoch. Process. Appl. 125(7), 2856–2894 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  34. Lacker, D.: A general characterization of the mean field limit for stochastic differential games. Probab. Theory Relat. Fields 165(3), 581–648 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  35. Lasry, J.-M., Lions, P.-L.: Mean field games. Jpn. J. Math. 2(1), 229–260 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  36. Sznitman, A.-S.: Topics in propagation of chaos. In: Hennequin, P.-L. (ed) Ecole d’Eté de Probabilités de Saint-Flour XIX - 1989. Lecture Notes in Mathematics, vol. 1464, pp. 165–251. Springer, Berlin (1991)

  37. Turchi, N.: Approximate Nash equilibria in N-player games based on Poisson random measures. Master’s Thesis, University of Padua (2015)

Download references

Acknowledgements

The Alekos Cecchin is supported by the Ph.D. Program in Mathematical Sciences, Department of Mathematics, University of Padua (Italy) and Progetto Dottorati - Fondazione Cassa di Risparmio di Padova e Rovigo (CaRiPaRo). The Markus Fischer acknowledges partial support through the research projects “Mean Field Games and Nonlinear PDEs” (CPDA157835) of the University of Padua and “Nonlinear Partial Differential Equations: Asymptotic Problems and Mean-Field Games” of the Fondazione CaRiPaRo. Both authors thank an anonymous referee for her/his helpful critique and detailed comments and suggestions.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Markus Fischer.

Appendix: Relaxed Poisson Measures

Appendix: Relaxed Poisson Measures

In order to state the definition of the relaxed Poisson random measure we first need to define the canonical space of integer valued random measures on a metric space E. Following Jacod [27], the setting is:

  • \(\overline{\Omega }\) is the set of sequences \((t_n,\,y_n) \subset [0,\, +\,\infty ] \times E\) such that \((t_n)\) is increasing and \(t_n < t_{n+1}\) if \(t_n < +\infty ;\) set \(t_0:=0\) and \(t_\infty := \lim _n t_n;\)

  • if \(\overline{\omega } = (t_n,\,y_n)_{n\in \mathbb {N}}\) write \(T_n(\overline{\omega }):=t_n\) and \(Y_n(\overline{\omega }):=y_n;\)

  • the canonical random measure is

    $$\begin{aligned} \overline{\mathcal {N}}(\overline{\omega },\, B) := \sum _{n\in \mathbb {N}} \mathbb {1}_{\{T_n(\overline{\omega })<\infty \}} \delta _{(T_n(\overline{\omega }),\, Y_n(\overline{\omega })} (B) \end{aligned}$$

    for any \(B\in \mathcal {B}([0,\, +\,\infty [ \times E );\)

  • \(\overline{\mathcal {G}}_t := \sigma (\overline{\mathcal {N}}(\cdot ,\, B) : B\in \mathcal {B}([0,\, t] \times E )),\)\(\overline{\mathcal {F}}_0\) is given, \(\overline{\mathcal {F}}_t = \overline{\mathcal {F}_0} \vee (\cap _{s<t} \overline{\mathcal {G}}_s),\)\( \overline{\mathcal {F}}=\overline{\mathcal {F}}_\infty \) and \(\overline{\mathbb {F}}= (\overline{\mathcal {F}}_t)_{t\ge 0}.\)

The filtered space \((\overline{\Omega },\,\overline{\mathcal {F}},\,\overline{\mathbb {F}})\) is then the canonical space of integer valued random measures onE. A probability measure on it is the law of an integer valued random measure on E,  given an initial condition on \(\overline{\mathcal {F}_0}.\) Note that the canonical measure \(\overline{\mathcal {N}}\) is not the identity: for this reason we can work with \(\mathcal {M} = \mathcal {M}([0,\, +\,\infty [ \times E)\) as the state space of a random measure. Moreover, the set of integer valued random measures is vaguely closed in \(\mathcal {M}\): see Theorem 15.7.4 in Kallenberg [28] and the references therein.

Let now \(\Theta \) be any integer valued random measure defined on a filtered probability space \((\Omega ,\,\mathcal {F},\,\mathbb {F},\,P).\) It is determined by a sequence of stopping times \(T_n\) and random variables \(X_n\) which are \(\mathcal {F}_{T_n}\)-measurable. To any \(\Theta \) is associated its compensator, that is, a positive random measure \(\eta \) on E such that

  1. (1)

    \(\eta ([0,\,t]\times B)_{t\ge 0}\) is predictable for any \(B\in \mathcal {B}(E);\)

  2. (2)

    \((\Theta ([0,\,t\wedge T_n]\times B) -\eta ([0,\,t\wedge T_n]\times B))_{t\ge 0}\) is an \(\mathbb {F}\)-martingale for each n and B

  3. (3)

    \(\eta (\{t\}\times E)\le 1\) for each t and \(\eta ([T_\infty ,\, \infty [ \times E)=0.\)

The compensator exists and is unique (up to a modification on a P-null set) for any \(\Theta .\) The proof can be found in Jacod [26], where the author also shows that a process with the above properties uniquely determines an integer valued random measure.

Consider then an arbitrary measurable space \((\Omega ',\,\mathcal {F}')\) and define \(\Omega := \overline{\Omega } \times \Omega '.\) Set \(\overline{\mathcal {F}_0} := \{\varnothing ,\,\overline{\Omega }\}\) and \(\mathcal {F}_0 :=\overline{\mathcal {F}_0}\otimes \mathcal {F}'.\) The canonical random measure \(\overline{\mathcal {N}}\) on \(\overline{\Omega }\) is extended to \(\Omega \) via \((T_n,\,Y_n)\cdot (\overline{\omega },\,\omega '):=(T_n,\,Y_n)\cdot (\overline{\omega }).\) Set \(\mathcal {F}_t:= \overline{\mathcal {F}}_t \vee \mathcal {F}_0.\)

Theorem 10

(Jacod [26]) Let \(P_0\) be a probability measure on \((\Omega ,\, \mathcal {F}_0)\) and \(\eta \) a predictable random measure satisfying (1) and (3). Then there exists a unique probability measure P on \((\Omega ,\,\mathcal {F}_\infty )\) whose restriction to \(\mathcal {F}_0\) is \(P_0\) and for which \(\eta \) is the compensator of \(\overline{\mathcal {N}}.\)

By means of this theorem, we are able to define properly a relaxed Poisson measure. Consider a relaxed control \(((\Omega '',\, \mathcal {F}'',\, P'';\, \mathbb {F}''),\, \rho ,\, \xi ,\, \mathcal {N}) \in \mathcal {R}\) and let \(\Omega '= \mathcal {D}\times \Sigma \times \overline{\Omega }\) be the state space of the process \(\rho ,\) the initial distribution \(\xi \) and the Poisson random measure \(\mathcal {N}.\) The \(\sigma \)-algebra \(\mathcal {F}'\) is generated by the processes and \(P_0\) is the joint law of \((\rho ,\, \xi ,\, \mathcal {N}).\) So a relaxed Poisson measure \(\mathcal {N}_\rho ,\) related to the relaxed control \(\rho ,\) is an integer valued random measure on \([0,\,T]\times U \times A\) whose compensator \(\eta ,\) calculated on \([0,\,t],\)\(U_0,\)\(A_0,\) is \(\nu (U_0) \rho ([0,\,t]\times A_0).\) Its law is uniquely determined on \(\overline{\Omega }\) and thus has the martingale properties (2.17) and (2.18). Moreover, the joint law of \((\mathcal {N}_\rho ,\,\rho ,\,\xi ,\,\mathcal {N})\) is uniquely determined.

We can give an explicit construction of \(\mathcal {N}_\rho .\) Let \(\rho \in \mathcal {R}\) and \((\alpha _n)\) be a sequence in \(\mathcal {A}\) which tends to \(\rho \) in the sense of Lemma 8, the chattering lemma. Denote by \(\rho ^{\alpha _n}\) the relaxed control representation of \(\alpha _n\) and construct \(\mathcal {N}_{\alpha _n}\) as in (2.19): \(\mathcal {N}_{\alpha _n}(t,\,U_0,\,A_0) := \int _0^t \int _{U_0}\mathbb {1}_{A_0}(\alpha _n(s)) \mathcal {N}(ds,\,du).\) Then, by Theorem 1, the sequence \((X_{\alpha _n},\, \rho ^{\alpha _n},\,\mathcal {N}_{\alpha _n})\) is tight and any subsequence converges in distribution to \((X_{\rho },\, \rho ,\,\mathcal {N}_{\rho }).\) The marginals are uniquely defined in this way, while to show that the joint law of \((\rho ,\,\mathcal {N}_{\rho })\) is unique we need to invoke the above Theorem 10.

1.1 Proof of Lemma 1

Let \(m\in \mathcal {L}\) be fixed, which we shall omit. Let \(\mathcal {Z}\) be the space of stochastic processes with paths in \(D([0,\,T],\,\Sigma )\) and equip it with the norm \(||X||= E[\sup _{0\le t \le T} |X(t)|].\) Let \(\rho \in \mathcal {R}\) and define the map \(G:\mathcal {Z} \longrightarrow \mathcal {Z}\) by

$$\begin{aligned} G_t(X):= \xi + \int _0^t\int _U\int _A f\left( s,\,X(s^-),\, u,\, a\right) \mathcal {N}_\rho (ds,\,du,\,da) \end{aligned}$$

for any \(X\in \mathcal {Z}.\) If we prove that this map is a contraction in the norm \(||\cdot ||,\) then pathwise existence and uniqueness of solutions to Eq. (2.20) follow. We have, for any \(X, \,Y \in \mathcal {Z},\)

$$\begin{aligned}&\left| G_t(X)-G_t(Y)\right| \\&\quad \le \int _0^t\int _U\int _A \left| f\left( s,\,X(s^-),\, u,\, a\right) - f\left( s,\,Y(s^-),\, u,\, a\right) \right| \mathcal {N}_\rho (ds,\,du,\,da), \end{aligned}$$

hence

$$\begin{aligned}&E\left[ \sup _{0\le t\le T} |G_t(X)-G_t(Y)|\right] \\&\quad \le E\int _0^T\int _U\int _A | f(s,\,X(s),\, u,\, a) - f(s,\,Y(s),\, u,\, a)|\rho _s(da) \nu (du) ds\\&\quad \le K_1 E \int _0^T \int _A |X(s) -Y(s)| \rho _s(da) ds \le K_1 T E \left[ \sup _{0\le t\le T} |X(s) -Y(s)|\right] \end{aligned}$$

thanks to (2.1) and the fact that \(\rho _s\) is a probability measure. Therefore G is a contraction if \(T <\frac{1}{K_1},\) and so uniqueness is proved for small time horizon; but then iterating the same argument, we have uniqueness for any T.

Consider now \(\widehat{\gamma }\in \widehat{\mathbb {A}}\) and define \(\widehat{G}:\mathcal {Z} \longrightarrow \mathcal {Z}\) by

$$\begin{aligned} \widehat{G}_t(X):= \xi + \int _0^t\int _U\int _A f\left( s,\,X(s^-),\, u,\, a\right) \mathcal {N}_{\rho ^{\widehat{\gamma },X}}(ds,\,du,\,da) \end{aligned}$$

for any process \(X\in \mathcal {Z}.\) Then for any X and Y we have \(||\widehat{G}(X) - \widehat{G}(Y)|| \le ||Z_1|| + ||Z_2||\) where

$$\begin{aligned} Z_1(t) := \int _0^t\int _U\int _A \left| f\left( s,\,X(s^-),\, u,\, a\right) - f\left( s,\,Y(s^-),\, u,\, a\right) \right| \mathcal {N}_{\rho ^{\widehat{\gamma },Y}}(ds,\,du,\,da) \end{aligned}$$

and

$$\begin{aligned} Z_2(t) := \int _0^t\int _U\int _A \left| f\left( s,\,X(s^-),\, u,\, a\right) \right| \left| \mathcal {N}_{\rho ^{\widehat{\gamma },X}}-\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}\right| (ds,\,du,\,da), \end{aligned}$$

where \(|\Theta |\) denotes the total variation of the signed measure \(\Theta \) defined for any \(C\in \mathcal {B}([0,\,T]\times U \times A)\) by \(|\Theta |(C) := \sup _{E\subseteq C} |\Theta (E)|;\) while the total variation norm is \(||\Theta ||_{TV} = |\Theta |([0,\,T]\times U \times A).\) The first term \(Z_1\) is bounded as above yielding \(||Z_1||\le K_1 T||X-Y||.\) For the second term, we use \(|f|\le d\) to obtain

$$\begin{aligned} \sup _{0\le t\le T} Z_2(t) \le d \left\| \mathcal {N}_{\rho ^{\widehat{\gamma },X}}-\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}\right\| _{TV} = d \sup _{E\subset [0,T]\times U \times A} \left| \mathcal {N}_{\rho ^{\widehat{\gamma },X}}(E)-\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}(E)\right| . \end{aligned}$$

Thanks to (2.17) and (2.13), we have \(E||\mathcal {N}_{\rho ^{\widehat{\gamma },X}}-\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}||_{TV} \le 2T\nu (U),\) saying that the right-hand side above is finite P-a.s. Since the measure \(\mathcal {N}_{\rho ^{\widehat{\gamma },X}}-\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}\) is integer valued, we can assume that the above supremum is attained on a set \(C(\omega )\) for P-a.e. \(\omega ,\) giving thus a random set C. Moreover, we may assume that on such a set the random measure considered is positive. The martingale property (2.18) now gives

$$\begin{aligned} ||Z_2||&\le d E \left[ \mathcal {N}_{\rho ^{\widehat{\gamma },X}}(C) -\mathcal {N}_{\rho ^{\widehat{\gamma },Y}}(C) \right] \\&=d \left| E \int _0^T \int _U\int _A \mathbb {1}_{C}(t,\,u,\,a)[\widehat{\gamma }(t,\,X(t)) - \widehat{\gamma }(t,\,Y(t))](da)\nu (du)dt\right| \\&\le d E \int _0^T |\widehat{\gamma }(t,\,X(t)) - \widehat{\gamma }(t,\,Y(t))|(A) \nu (U) dt\\&\le 2 \nu (U) d E \int _0^T |X(t) -Y(t)| dt \le K_1 T ||X-Y||, \end{aligned}$$

where in the last line above we have used the fact that \(\widehat{\gamma }\) is a probability measure and \(|x-y| \ge 1\) for each \(x\ne y\in \Sigma .\) Therefore, for \(T<\frac{1}{2 K_1},\) the map \(\widehat{G}\) is a contraction; the claim follows iterating the above procedure.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cecchin, A., Fischer, M. Probabilistic Approach to Finite State Mean Field Games. Appl Math Optim 81, 253–300 (2020). https://doi.org/10.1007/s00245-018-9488-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00245-018-9488-7

Keywords

Mathematics Subject Classification

Navigation