On the asymptotic nature of first order mean field games

For a class of finite horizon first order mean field games and associated N-player games, we give a simple proof of convergence of symmetric N-player Nash equilibria in distributed open-loop strategies to solutions of the mean field game in Lagrangian form. Lagrangian solutions are then connected with those determined by the usual mean field game system of two coupled first order PDEs, and convergence of Nash equilibria in distributed Markov strategies is established.


Introduction
The purpose of this article is to illustrate a simple way of establishing convergence of open-loop Nash equilibria in the case of first-order non-stationary Mean Field Games (MFGs). Introduced by J.-M. Lasry and P.-L. Lions and, independently, by M. Huang, R.P. Malhamé and P.E. Caines about fifteen years ago (cf. [34,31]), mean field games are limit models for non-cooperative symmetric N -player differential games as the number of players N tends to infinity; see, for instance, the lecture notes [13] and the recent two-volume work [18]. The notion of solution usually adopted for the prelimit models is that of a Nash equilibrium. A standard way of making the connection with the limit model rigorous is to show that a solution of the mean field game yields approximate Nash equilibria for the N -player games, with approximation error vanishing as N → ∞. In the opposite direction, one aims to prove that a sequence of N -player Nash equilibria converges, as N tends to infinity, to the mean field game limit.
When Nash equilibria are considered in stochastic open-loop strategies, then their convergence is well understood and can be established under mild conditions; see [28] and [32], both for finite horizon games with general, possibly degenerate, Brownian dynamics. The convergence analysis is much harder when Nash equilibria are defined over Markov feedback strategies with full state information.
A first result in this setting was given by Gomes, Mohr, and Souza [29] for continuous time games with finite state space. There, convergence of Markovian Nash equilibria is proved, but only if the time horizon is small enough. A breakthrough was achieved by Cardaliaguet, Delarue, Lasry, and Lions in [15]. In the setting of games with non-degenerate Brownian dynamics, possibly including common noise, convergence to the mean field game limit is established there for arbitrary time horizon provided the so-called master equation associated with the mean field game possesses a unique sufficiently regular solution. In this case, the convergence analysis can be refined, yielding not only convergence of minimal costs and propagation of chaos for the Nash equilibrium state trajectories, but also fluctuation and large deviations results for the associated empirical measures; see Cecchin and Pelino [19] and, independently, Bayraktar and Cohen [7] for finite state games and the papers by Delarue, Lacker, and Ramanan [23,24] for Brownian dynamics without or with common noise.
Well-posedness of the master equation implies uniqueness of solutions for the mean field game. But also the situation where the mean field game possesses multiple solutions, while the N -player Nash equilibria in full Markov feedback strategies are still uniquely determined, occurs. In this case, the convergence problem is in part open. The most general result appears to be the recent preprint [33] by Lacker. There, for a class of games with non-degenerate Brownian dynamics, it is shown that all limit points of the N -player Nash equilibria are concentrated on weak solutions of the mean field game; these are more general than randomizations of ordinary ("strong") solutions of the mean field game. In the three recent works [36], [25] and [20], the authors present case studies, giving criteria for characterizing those mean field game solutions that can be attained as limits of feedback Nash equilibria with full state information.
Here, we consider a much simpler situation: The underlying dynamics are deterministic with direct control of players' states; randomness enters only through the players' initial positions, which are assumed to be independently and identically distributed. Thanks to the deterministic dynamics without explicit interaction, players can directly control their entire state trajectories. Thus, the set of strategies (or actions) of each player consists of trajectory-valued functions depending on time and player's own position. We call these actions distributed open-loop strategies. By considering randomizations of these strategies, obtaining what we call randomized distributed open-loop strategies, we obtain existence of symmetric Nash equilibria in this new class of actions (through a standard fixed point argument), as well as precompactness of sequences of such equilibria. Convergence to the mean field game equilibrium in Lagrangian form (see e.g. [16,8,17,10])) along weakly converging subsequences of Nash equilibria is then established in Theorem 3.1 by showing that the variance of the associated empirical measures (evaluated by integrating over test functions from a countable family) tends to zero as N → ∞. Thus, to obtain convergence, we essentially prove a law of large numbers for a triangular array (Y N i ) i∈{1,...,N },N ∈N where Y N 1 , . . . , Y N N are independent and identically distributed with common marginal law that however depends on N . Let us point out that after obtaining a suitable compactness property for the set of admissible strategies, the convergence result in Theorem 3.1 can actually be deduced from the more general results in [28] or [32]. Thus, the main purpose of this result is therefore to give a simple proof of convergence, which takes advantage of the structure of the dynamics and the deterministic nature of the underlying differential games. We believe that the techniques considered here could be useful in order to justify the asymptotic nature of more sophisticated deterministic MFGs (see e.g. [10,11] dealing with state constrained problems).
In the second part of this article, we consider additional second order assumptions on the data and we assume that the initial distribution is absolutely continuous with respect to the Lebesgue measure. In this framework, and following [16], we link the notion of Lagrangian MFG equilibrium to the well known PDE characterization of mean field games in terms of two coupled first order partial differential equations, namely a backward first order Hamilton-Jacobi-Bellman equation and a forward continuity equation; see equation (M F G) in Subsection 4.1 below. Under the stronger assumptions mentioned above, for each N ∈ N, any symmetric randomized distributed open-loop Nash equilibrium of the N -player game can be identified with a symmetric distributed open-loop Nash equilibrium (non-randomized). Moreover, the strategies associated to such equilibria can be described by controls which are feedback with respect to the individual states. We call these actions distributed Markov strategies. The latter are determined by the solutions of a coupled system of first order PDEs analogous to the mean field game PDE system; see Eq. (M F G N ) in Subsection 4.2. This is in contrast with N -player Nash equilibria defined over Markov feedback strategies with full state information, which are determined through a system of N coupled PDEs. Building on the equivalence of characterizations and the convergence result in Theorem 3.1, we also establish in Theorem 4.1 the convergence of solutions of (M F G N ) to solutions of (M F G), as well as the convergence of the Nash equilibria in distributed Markov strategies.
The rest of this paper is organized as follows. In Section 2, we introduce the N -player games together with some notation and our standing assumptions. Existence of Nash equilibria in randomized distributed open-loop strategies is verified in Proposition 2.1. In Section 3, the associated mean field game is introduced in Lagrangian form; see Definition 3.1 and Proposition 3.1. We show convergence of symmetric N -player randomized distributed open-loop Nash equilibria to the mean field game limit in Theorem 3.1. Section 4 links, under additional assumptions, the mean field game in Lagrangian form with the mean field game PDE system. Similarly, N -player Nash equilibria in distributed Markov strategies are described in terms of a PDE system analogous to the limit system, but dependent on the number of players N . The corresponding convergence results are given in Theorem 4.1 and Corollary 4.1, respectively.

The N -player game
Before introducing the N -player deterministic differential game that we are interested in, let us first fix some notations. Given a Polish space (X, d X ), we denote by P(X) the set of probability measures over X. If (Y, d Y ) is another Polish space and ϕ : X → Y a Borel measurable function, then the push-forward ϕ♯m ∈ P(Y ) of a measure m ∈ P(X) by ϕ is defined by where B(Y ) denotes the σ-algebra of Borel sets in Y . Given p ∈ [1, +∞) we denote by P p (X) the set of probability measures µ over X such that X d X (x, x 0 ) p dµ(x) < +∞ for some x 0 ∈ X. The set P p (X) is endowed with the Monge-Kantorovich distance d p : where π i : X × X → X (i = 1, 2) is the projection on the i-th coordinate, that is, π i (x 1 , x 2 ) := x i . It is well known (see e.g. [37]) that in the particular case p = 1 we have where Lip 1 (X) denotes the set of Lipschitz functions on X with Lipschitz constant equal to one. We will fix as data some functions ℓ : and a probability measure m 0 ∈ P(R d ). The functions ℓ, Φ will represent running and terminal costs, respectively, while m 0 will be an initial state distribution. We will assume that: (A1) (i) The functions ℓ and Φ are continuous. Moreover, the following properties hold true: where we denote by | · | the Eulidean norm in R d .
Choose T > 0, the finite time horizon, and set Γ : The space Γ is naturally endowed with the topology of uniform convergence. Let W 1,q ([0, T ]; R d ) denote the Sobolev space of R d -valued absolutely continuous functions that possess first order weak sense partial derivatives in L q ((0, T ); R d ).
In order to introduce the game that we will consider, assume first that there are N individuals (N ≥ 2), which, from now on, will be called players, positioned at x 1 , . . . , x N ∈ R d at time t = 0. For i ∈ {1, . . . , N }, player i chooses a trajectory γ i ∈ A(x i ), where Given these initial positions, the cost j N i : where, for notational convenience, we have denoted In the game that we will consider, the initial position of each player is random, independent of the initial positions of the other players and with the same law m 0 . In this new context, we define the set of distributed open-loop strategies of the players as A := γ : R d → Γ γ is Borel measurable and γ(x) ∈ A(x) for all x ∈ supp(m 0 ) .
For notational simplicity, for γ ∈ A, we will write γ x := γ(x). Given a profile of actions (γ 1 , . . . , γ N ) ∈ A N , it is natural to define the cost that it induces on player i as the mean with respect to the initial conditions of costs having the form (2.4). Namely, the cost function J N i : A N → R for player i is defined as Recalling (2.5) and defining In particular, the costs are symmetric, i.e. for every permutation σ : {1, . . . , N } → {1, . . . , N }, we have that ). Let us recall the classical notion of Nash equilibrium when applied to the game defined by the N players, the action set A (which is the same for all players) and the individual costs The existence of a Nash equilibrium for this symmetric game is not immediate because of the lack of compactness of the set A. The following simple result shows that the time derivatives of the strategies in a Nash equilibrium configuration (provided that it exists) enjoy a uniform boundedness property in L q ((0, T ); R d ). This fact will allow us to reduce the set of action strategies in Definition 2.1.
Lemma 2.1. Assume that (A1) holds and that (γ 1,N , . . . , γ N,N ) ∈ A N is a Nash equilibrium. Then, there exists a constant C > 0, independent of N , such that Proof. Fix i ∈ {1, . . . , N } and define j i : . By Fubini's theorem, and (2.7), for all γ ∈ A we have where γ x (t) := x for all t ∈ [0, T ]. Assumption (A1) implies that j 1 (γ x ) is bounded by a constant, which is uniform for x ∈ supp(m 0 ) and independent of N . Using this fact, the first inequality in (2.3) easily yields (2.8).
It remains to prove the claim. First note that if (γ x n ) is a sequence in S N i (x) converging to γ x uniformly in [0, T ], then, by the first inequality in ( , then, passing to the limit along a subsequence in the relation Using this fact and the weak lower Arguing as before, using the first inequality in (2.3), we obtain that (γ xn n ) is bounded in L q ([0, T ]; R d ). This implies that, up to some subsequence, γ xn n converge uniformly to some Using that M is closed, w.r.t. the uniform convergence, we get that γx ∈ M . On the other hand, noticing that which implies that the latter set is closed. Now, we focus our attention on the existence of Nash equilibria for the described game. Using Lemma 2.1, a reformulation of the set of admissible strategies and cost functionals of the N -players game will be useful. Let C > 0 be given by Lemma 2.1 and define the set Since supp(m 0 ) is compact, Hölder's inequality yields the existence of a compact set K C ⊆ R d such that Using this fact and arguing as in the proof of Lemma 2.1, we have that Q C is compact as a subset of Γ, that is, when it is endowed with the topology of uniform convergence. Given γ i ∈ A, let us set m i := γ i ♯m 0 ∈ P(Γ). By (2.1), for any profile of strategies (γ i ) N i=1 ∈ A N , the cost for player i is given by This expression for the cost motivates a relaxation of the game with strategies in A by considering strategies taking values in P(Γ). Let us define the set A rel of randomized distributed open-loop strategies by Lemma 2.2. The set A rel is convex and compact as a subset of P(Γ).
Proof. Convexity follows directly from the definition. On the other hand, since A rel ⊆ P(Q C ) and P(Q C ) is compact as a subset of P(Γ) (because Q C is a compact subset of Γ), it suffices to check that A rel is closed in P(Γ), but this follows directly from [3, Proposition 5.1.8] and the fact that Q C is closed.
Therefore, by (2.11), [3, Proposition 7.1.5] and Lemma 2.2, the set Motivated by (2.12), we introduce the new relaxed game which has A rel as set of strategies for each player and, given a strategy profile (m j ) N j=1 ⊆ A N rel , the cost for player i is given by Note that the this game is still symmetric. In this framework, a profile of strategies (m 1,N , . . . , m N, . . , µ n ). Therefore, passing to the limit in (2.15) we obtain that J N rel (ν;μ, . . . ,μ) ≤ J N rel (µ ′ ;μ, . . . ,μ) ∀ µ ′ ∈ A rel , i.e.ν ∈ S N (μ)∩M , which implies the closedness {µ ∈ A rel | S N (µ)∩M = ∅} and the upper semicontinuity of S N . Using the properties above, the existence of a fixed point for S N follows from the Kakutani-Fan-Glicksberg fixed-point theorem (see e.g. [1,Corollary 17.55]).
rel be a Nash equilibrium for the game defined by the cost J N rel and the set of strategies A rel . Definel : and assume that for m 0 -almost every x ∈ R d the optimization problem By definition of Nash equilibrium and Fubini's theorem, for all m ∈ A rel we have that By the desintegration theorem (see e.g. [ Define the set-valued functionŜ N : Arguing as in the proof of Lemma 2.1, we have the existence of a Borel measurable selection , which, by assumption, is m 0 -uniquely determined. Moreover, by definition, γ N ∈ A. Now, let us definem := γ N ♯m 0 ∈ A rel . Then, by (2.19), taking m =m, and (2.20), we have that Since the integrand in the expression above is non-negative, by definition of γ Remark 2.2. An example of application of Corollary 2.1 is provided in Section 4.2 below.

Convergence to a mean field game equilibrium
In this section, we study the limit behavior, as N → ∞, of symmetric Nash equilibria in randomized distributed open-loop strategies. The existence of such Nash equilibria is ensured by Proposition 2.1. We begin by defining the limit object, i.e. the MFG equilibrium. Then we will prove that any cluster point of the sequence (m N ) is a MFG equilibrium.
Let us define J : It is straightforward to check that m ∈ P 1 (Γ) implies that e t ♯m ∈ P 1 (R d ) for all t ∈ [0, T ], which implies that J is well-defined. Following the terminology in [35], we consider next the notion of Lagrangian MFG equilibrium (see e.g. [16,8,17,10]). Definition 3.1. We say that m * ∈ P 1 (Γ) is a Lagrangian MFG equilibrium if e 0 ♯m * = m 0 and Reasoning as in the proof of Lemma 2.1, assumption (A1) implies that if m * is a Lagrangian MFG equilibrium, then supp(m * ) ⊆ Q C and, hence, m * ∈ A rel . Thus, m * is a Lagrangian MFG if and only if m * ∈ A rel , e 0 ♯m * = m 0 and We still denote by J the restriction of J to Q C × A rel and recall that Q C , endowed with the topology of uniform convergence, is a compact set. For later use, let us state the following simple results.
Lemma 3.1. Suppose that (A1) holds. Then, the following assertions hold true: (i) The relative topology on A rel , as a subset of P 1 (Γ), coincides with the topology induced by the narrow convergence.
(ii) The function J is lower semicontinuous in Q C × A rel .
(iii) For all γ ∈ Q C , the function J(γ, ·) is bounded, uniformly in γ, and continuous in A rel .
Proof. Assertion (i) follows from the fact that both topologies coincide on P(Q C ), because Q C is a compact subset of Γ. In particular, if m n → m narrowly in A rel , then for all t ∈ [0, T ] we have that (3.3) e t ♯m n → e t ♯m in P 1 (R d ).
If m * is a Lagrangian MFG equilibrium, we will denote by {m x * | x ∈ R d } the m 0 -uniquely determined Borel family of probability measures on Q C satisfying that m x * (Q C \ A(x)) = 0 and dm * (γ) = dm x * (γ) ⊗ dm 0 (x). The existence of such a family is ensured by the disintegration theorem (see e.g. [3, Theorem 5.3.1]). We have the following equivalent characterization of a Lagrangian MFG equilibrium.
Proposition 3.1. The following assertions are equivalent: (iii) The following inequality holds true: Proof. Let us prove the equivalence between (i) and (ii). Let m * ∈ A rel be a Lagrangian MFG equilibrium.
we obtain a contradiction with (3.2). Conversely, suppose that (ii) holds and that m * is not a Lagrangian MFG equilibrium. Then, defining which is an open set and, hence, belongs to B(Q C ), we have that which is impossible because (3.4) implies that the r.h.s. above is equal to 0.
Let us now prove that (ii) ⇔ (iii). Notice that Analogously, given m ∈ A rel , we disintegrate it w.r.t. m 0 and write dm(γ) = dm x (γ) ⊗ dm 0 (x), where {m x | x ∈ R d } is a m 0 -uniquely determined Borel family of probability measures on Q C such that m x (Q C \ A(x)) = 0 for m 0 -a.e. x ∈ R d . Thus, If (ii) holds, then for m 0 -a.e. x ∈ R d and m x * -a.e. γ ∈ Q C we have Integrating both sides of (3.8), first with respect to dm x (γ ′ ) and then with respect to dm x * (γ), and using (3.6)-(3.7) we obtain (3.5). Conversely, using the notations introduced above, suppose that (3.5) holds and letγ ∈ A be a Borel measurable selection of S (the existence of such selection can be justified arguing exactly as in the proof of Lemma 2.1). Then, taking the measure m ∈ A rel defined by dm(γ) = dδγx(γ) ⊗ dm 0 (x) in (3.5) and using (3.6), we deduce that and, hence, m x * -almost every γ belongs to S(x). The conclusion follows. Now, consider the symmetric N -player game defined in Section 2, with randomized distributed openloop strategies, and let (m N , . . . , m N ) ∈ A N rel be a symmetric equilibrium. Our main result in this section, stated in the next theorem, shows that any limit point of this sequence is a Lagrangian MFG equilibrium.
Theorem 3.1. Suppose that (A1) holds. Then any limit point m * of (m N ) N ∈N (there exist at least one) is a Lagrangian MFG equilibrium. Moreover, if (m N k ) k∈N is a subsequence of (m N ) N ∈N converging to m * , then sup t∈[0,T ] d 1 (e t ♯m N k , e t ♯m * ) → 0 as k → ∞.
Proof. We only prove the first assertion, since, having this result, the second one follows directly from Remark 2.1. For N ≥ 2, let Y N 1 , . . . , Y N N be independent and identically distributed (i.i.d.) Q C -valued random variables with common distribution m N defined on some probability space (Ω N , F N , P N ). We denote by E N the expectation with respect to P N .
Notice that Y N i and µ N,i are independent for every i, while µ N,1 , . . . , µ N,N are identically distributed (not independent in general) with common distribution depending on N . Moreover, for every i ∈ {1, . . . , N }, be a converging subsequence of (m N ) N ∈N with limit m * for some m * ∈ A rel . The existence of such a subsequence follows from the compactness of A rel . Let us prove that (µ N k ,1 ) k∈N converges in distribution to the deterministic limit m * . Let T ⊂ C b (Q C ) be countable and measure determining (or separating). Thus, T is a countable collection of bounded continuous functions on Q C such that two probability measures ν,ν ∈ P(Q C ) are equal whenever ψ dν = ψ dν for all ψ ∈ T . Observe that T can be chosen countable since Q C is a Polish space, hence separable, under the supremum norm topology.
For ψ ∈ T set By construction and symmetry, for every i ∈ {1, . . . , N }, Since Y N 1 , . . . , Y N N are independent and the functions in T are bounded, it follows that Moreover, considering the subsequence (m N k ) k∈N , we have that Since P(Q C ) is compact, the set P(P(Q C )) is also compact. Thus, there exists a sub-subsequence (N k l ) l∈N such that (µ N k l ,1 ) l∈N converges in distribution to µ for some P(Q C )-valued random variable µ defined on some probability space (Ω, F , P). We denote by E the expectation under P. Then, by the continuous mapping theorem (see e.g. [9, Theorem 5.1]) and (3.11), we have that On the other hand, thanks to (3.10), lim l→∞ v N k l ψ = 0. It follows that for every ψ ∈ T , QC ψ(γ) dµ(γ) = m * ψ = QC ψ(γ) dm * (γ) P-almost surely.
Since T is measure determining, we find that µ ω = m * for P-almost all ω ∈ Ω.
As we can always choose converging (sub-)subsequences, we deduce that (µ N k ,1 ) k∈N converges in distribution to m * . By independence of Y N k 1 and µ N k ,1 , and using [9, Theorem 3.2], we find that This implies, thanks to (3.9), Lemma 3.1(ii) and [3, Lemma 5.1.7], that Let m ∈ A rel . By Lemma 3.1(iii) and dominated convergence, it follows that Passing to the limit in the Nash equilibrium inequality . . , m N k ), and using (3.12)-(3.13), we obtain that (3.5) holds. The result now follows from Proposition 3.1.

The first order Mean Field Game system
In this section, we first discuss, following [16], the relation between the notion of Lagrangian MFG equilibrium and the first order PDE system introduced by Lasry and Lions in [34, Section 2.5] for some particular data. Next, in Section 4.2, we consider symmetric randomized distributed open-loop Nash equilibria for the N -player game, which, thanks to Corollary 2.1 and Assumption (A2) below, can be identified with symmetric distributed open-loop Nash equilibria (non-randomized). Arguing as in the MFG limit, we connect these equilibria with a first order PDE system which is similar to the one appearing in the limit case. Consequently, analytic techniques can also be used in order study the limit behavior of these equilibria as the number of players tends to infinity.
Let L : , is bounded from below and (i.1) there exist C L > 0, L > 0 such that ii) The functions f and g are continuous. Moreover, for every m ∈ P 1 (R d ) the functions f (·, m) and g(·, m) belong to C 2 (R d ) and there exists a constant C f,g > 0 such that where, for h = f , g, we have set h(·, m) C 2 := sup (iii) The measure m 0 is absolutely continuous w.r.t. to the Lebesgue measure L d , with density still denoted by m 0 , and has a compact support.
A typical example of function L satisfying (A2)(i) is given by (iii) By the first estimate in (4.2) we have that ∂ ξ H(ξ, x) is characterized as the unique solution α(ξ, x) of the optimization problem max α∈R d {ξ · α − L(α, x)}. As a consequence of this fact and the first relation in (4.3), we obtain the existence of c H > 0 such that Moreover, from the convexity of L(x, ·), for all ξ, x ∈ R d we have that ∂ ξ H(ξ, x) is the unique solution to Using the relation above, the regularity L ∈ C 2 (R d × R d ), the first estimate in (4.2) and the implicit function theorem, we obtain that Let us define ℓ : Clearly, Assumption (A2) implies that ℓ, Φ and m 0 satisfy (A1).

4.1.
Lagrangian MFG equilibria and the MFG PDE system. As pointed out in [16], under (A2) the existence of a Lagrangian equilibrium for the MFG problem defined by ℓ, Φ and m 0 , is equivalent the existence of a solution (u, ρ) of the following PDE system, which was first introduced in [34], In the system above, u : The existence of such measurable selection follows from the fact that the above set-valued map has a closed graph (thanks to the semi-concavity property of u, see e.g. [12,Proposition 3.3.4]). Moreover, The function ρ : [0, T ] → P(R d ) is a solution to the second equation, with the associated initial condition, if ρ ∈ C([0, T ]; P 1 (R d )), and the equation is satisfied in the sense of distributions, i.e. for all φ ∈ C ∞ (R d ) with compact support we have Note that by the previous considerations, the second term in the right-hand-side of (4.8) is well-defined. Any pair (u, ρ) satisfying (M F G) is called an equilibrium of the first order MFG problem.
For the sake of completeness, let us provide the main arguments that justify the equivalence between both notions of equilibria. Let m * ∈ A rel be a Lagrangian MFG equilibrium. Let us define ρ : [0, T ] → P(R d ) by ρ(t) := e t ♯m * for all t ∈ [0, T ], and the value function u : Since m * ∈ A rel , we have that ρ ∈ C([0, T ]; P 1 (R d )). Using this fact, assumption (A2) and [5, Proposition 1.1 and Remark 1.1], we obtain that u is a viscosity solution of (4.10) Moreover, by [21, Theorem 2.1], the Hamilton-Jacobi-Bellman equation above admits a comparison principle, which implies that u is its unique viscosity solution. We will need the following result, whose proof follows from standard arguments.
is non-empty. Moreover, there exists a constant C > 0, independent of (x, t), such that (ii) The value function u is globally Lipschitz.
(iii) The value function u is locally semi-concave w.r.t. to the space variable, uniformly in t ∈ [0, T ]. More precisely, for any compact set K ⊆ R d there exists a constant C K , independent of t, such that for every λ ∈ [0, 1], x, y ∈ K, such that the segment [x, y] is contained in K, the following inequality holds Proof. The proof being standard, we only sketch the main ideas. The fact that S(x, t) is non-empty follows directly from (A2) and the direct method in the Calculus of Variations. Moreover, by (A2) and arguing as in the proof of Lemma 2.1 we obtain the existence of c > 0, independent of (x, t, ρ), such that (4.14) u(x, t) = inf T t ℓ(γ(s), γ(s), ρ(s))ds + Φ(γ(T ), ρ(T )) γ ∈ W 1,2 ([0, T ]; R d ), γ(t) = x, and T 0 |γ(s)| 2 ds ≤ c .
Using this fact, the Euler-Lagrange equation associated to any element γ * ∈ S(x, t), the second estimate in (4.2) and arguing as in the proof of [12, Theorem 6.2.5], we easily obtain (4.12), which proves assertion (i). In order to prove (ii), notice that (i) implies that the value function can also be written as , the uniform Lipschitz property for f and g in (A2)(ii), and the second estimate in (4.2), we easily obtain that u(·, t) is globally Lipschitz, with a Lipschitz constant which is independent of t ∈ [0, T ]. Similarly, using (4.15) and the estimate (4.1), we get that u(x, ·) is globally Lipschitz, with a Lipschitz constant which is independent of x ∈ R d . Assertion (ii) follows. Finally, assertion (iii) follows directly from [12, Theorem 6.4.1]. Now, let us consider the set-valued map Since Lemma 4.1(ii) implies that u(·, 0) is a.e. differentiable, [12,Corollary 6.4.10] yields that for a.e.
x ∈ R d we have S(x) = {γ x } for someγ x ∈ A(x). Now, as in the proof of Proposition 3.1, let γ * ∈ A be a Borel measurable selection of S. Then, for a.e. x ∈ R d we have that γ x * =γ x . Thus, Proposition 3.1(ii) yields m x * = δ γ x * for a.e. x ∈ supp(m 0 ) and, hence, m * = γ * ♯m 0 . In particular, On the other hand, by [12, Theorem 6.4.9, Theorem 6.3.3 and Theorem 6.4.8], for a.e. x ∈ R d , we have where we underline that u is differentiable w.r.t. to its first argument at the point (γ x * (t), t) if t ∈ (0, T ) (see [12,Theorem 6.4.7]). Denoting by still by ∇u a measurable selection of (x, t) → D + x u(x, t), for every φ ∈ C ∞ (R d ) with compact support and t ∈ [0, T ], we have which implies that ρ satisfies (4.8) and, hence, the couple (u, ρ) solves (M F G). Notice that under (A2) a Lagrangian MFG equilibrium m * exists (see Remark 3.1) and, hence, the previous arguments show, in particular, the existence of at least one solution (u, ρ) to (M F G).
Conversely, if (u, ρ) solves (M F G), then the first equation therein implies that u and ρ are still related by (4.9). By the second equation in (M F G) and [3, Theorem 8.2.1], there exists a probability measure m * ∈ P(Γ) such that ρ(t) = e t ♯m * for all t ∈ [0, T ] and, considering the disintegration dm * (γ) = dm x * (γ) ⊗ dm 0 (x), for a.e. x ∈ supp(m 0 ) the support of the measure m x * is contained in the set of solutions of (4.16). By Lemma 4.1 and arguing as in the proof of [13,Lemma 4.11], we have that every solution to (4.16) solves the optimization problem in the r.h.s. of (4.9) with t = 0. Thus, by Proposition 3.1(ii) we obtain that m * is a Lagrangian MFG equilibrium. Notice also that S(x) being a singleton for a.e. x ∈ R d , the previous argument shows, in particular, that [0, T ] ∋ t → γ (·) * (t)♯m 0 ∈ P 1 (R d ) is the unique solution in C([0, T ]; P 1 (R d )) of the continuity equation In addition to the relation between Lagrangian MFG equilibria and the solutions of (M F G), assumption (A2) has also consequences on the regularity of the time marginals {ρ(t) | t ∈ [0, T ]} as the following result shows. Proposition 4.1. In addition to (A2)(iii), assume that m 0 ∈ L p (R d ) for some p ∈ (1, +∞] and let (u, ρ) be a solution to (M F G). Then, the following assertions hold true: (ii) For all t ∈ [0, T ] the measure ρ(t) is absolutely continuous w.r.t. the Lebesgue measure. Moreover, the density of ρ(t), that we will still denote by ρ(t), belongs to L p (R d ) and there exists a constant c 2 > 0, independent of p ∈ (1, +∞] and t ∈ [0, T ], such that Proof. Assertion (i) follows directly from the formula ρ(t) = γ (·) * (t)♯m 0 , where γ x * ∈ S(x) for all x ∈ R d , Lemma 4.1(i) and the fact that supp(m 0 ) is compact. In order to prove (ii), let β ∈ C ∞ (R d ), nonnegative, with support contained in the unit ball and such that R d β(x)dx = 1. For ε > 0, let us define β ε (x) := ε −d β(x/ε), u ε (x, t) := (β ε * u(·, t)) (x) and consider the equation For every x ∈ R d , let us define γ x ε ∈ C 1 ([0, T ]; R d ) as the unique solution to (4.20) [3,Proposition 8.1.8], equation (4.19) admits a unique solution in C([0, T ]; P 1 (R d )), which is given by ρ ε (t) := γ (·) ε (t)♯m 0 for all t ∈ [0, T ]. Moreover, by a standard change of variable argument (see e.g. [2, Section 2]), for every t ∈ [0, T ] we have that ρ ε (t) is absolutely continuous, with density given by where, for each y ∈ R d , Y (y, ·) is defined as the unique solution tȯ . Let us assume that p ∈ (1, +∞). By a change of variable again, we obtain that As a consequence, Tr(L ε (x, t)) is bounded from above by a constant which is independent of ε > 0 small enough, x ∈ O, and t ∈ [0, T ]. Thus, by (4.23) and taking the power 1/p in (4.22), there exists c 2 > 0, independent of ε, t and p, such that The previous estimate shows the existence ofρ ∈ L ∞ ([0, T ]; L p (R d )) and a sequence (ρ εn ) n∈N such that, as n → ∞, ε n → 0 and ρ εn →ρ ∈ L ∞ ([0, T ]; L p (R d )) in the weak* topology. By dominated convergence, we have that ∂ ξ H(∇u εn , ·) → ∂ ξ H(∇u, ·) in L 1 ([0, T ]; L s (R d )) for any s ∈ [1, +∞). As a consequence,ρ satisfies estimate (4.24) and, passing to the limit in (4.19), we get that the measure  [26], in order to establish L p -estimates for the time evolving distributions describing equilibria in optimal-exit MFGs.

4.2.
The N -player equilibria: associated time marginals and value functions. Let us consider the game with N players defined in Section 2 with ℓ and Φ given by (4.7). Let (m N , . . . , m N ) ∈ A N rel be a symmetric equilibrium in randomized distributed open-loop strategies for the N -player game. Note that if for h = f , g we define we have that f N and g N satisfy the assumptions for f and g in (A2)(ii) (with the same constant C f,g ). As a consequence of this fact, the results in [12,Chapter 6] and Corollary 2.1 we obtain the existence of γ N ∈ A such that m N = γ N ♯m 0 , i.e. (m N , . . . , m N ) can be identified with the non-randomized symmetric equilibrium in distributed open-loop strategies given by (γ N , . . . , γ N ) ∈ A N . Furthermore, setting ρ N (t) := e t ♯m N = γ (·) In particular, if γ ′ N ∈ A is different from γ N but coincides with it on a set A such that m 0 (A) = 1, then we also have that m N = γ ′ N ♯m 0 . For the sake of simplicity, we have chosen to represent always m N via a measurable selection γ N of the set-valued map S N . Notice that (A2) and the results in [12,Chapter 6] imply that γ x N is uniquely defined for a.e. x ∈ R d . (ii) Exactly as in the limit case (see Lemma 4.1(i)), we have the existence of a constant C > 0, independent of (x, t) and N ∈ N, such that As a consequence, there exists a compact set K C ⊆ R d such that γ x N (t) ∈ K C for all N ∈ N, x ∈ supp(m 0 ) and t ∈ [0, T ]. In particular, the representation ρ N (t) = e t ♯m N = γ  Arguing as in the previous subsection, the pair (u N , ρ N ) solves Conversely, associated to any solution (u N , ρ N ) we have the existence of a symmetric equilibrium in distributed open-loop strategies (γ N , . . . , γ N ) ∈ A N for the N -player game. Moreover, using again the results in [12,Chapter 6], any γ N ∈ A defining such equilibrium satisfies x ∈ supp(m 0 ). Thus, we can think of the r.h.s. above as an optimal control which is feedback with respect to the individual states. We call (γ N , . . . , γ N ) a Nash equilibrium in distributed Markov strategies for the N -player game. As a consequence of the previous discussion, such equilibria exist for all N ∈ N provided that (A2) holds true.
Let us consider a sequence (γ N ) N ∈N of elements in A defining Nash equilibria in distributed Markov strategies for the N -player games. Theorem 3.1 yields the existence of a Lagrangian equilibrium m * ∈ A rel and a subsequence (γ N k ) k∈N such that γ N k ♯m 0 → m * as k → ∞. Let γ * ∈ A be such that m * = γ * ♯m 0 and γ x * ∈ S(x) for all x ∈ R d , i.e. an equilibrium in distributed Markov strategies for the MFG. Our aim now is to study the convergence of the associated time marginals ρ N k to ρ, the convergence of the associated value functions u N k to u and, finally, the convergence of γ x N k to γ x * for a.e. x ∈ R d . We will need the following preliminary result. Lemma 4.2. Assume (A2)(ii) and let K ⊆ R d be a nonempty compact set. Consider a sequence of measures (µ k ) k∈N ⊆ P 1 (R d ) such that supp(µ k ) ⊆ K, for all k ∈ N, and, as k → ∞, µ k → µ for some µ ∈ P 1 (R d ). Then, for any sequence (x k ) k∈N and x ∈ R d such that x k → x, we have (4.29) h(x, µ) = lim where h = f , g and h k is defined by (4.25).
Proof. Notice that (A2)(ii) implies that Now, let Y k 1 , . . . , Y k k be independent and identically distributed K-valued random variables, defined on some probability space (Ω k , F k , P k ), with common distribution µ k . Using that P(P(K)) is compact and arguing as in the proof of Theorem 3.1, we obtain that, as k → ∞, the P 1 (R d )-valued random sequence converges in distribution to the deterministic measure µ. Since (4.30) can be written as relation (4.29) follows by letting k → ∞.
Theorem 4.1. Assume that (A2) holds. Then, the following assertions hold true: Let γ x,t N k ∈ S N (x, t) and γ x,t ∈ C([t, T ]; R d ) be a cluster point of (γ x,t N k ) k∈N , with respect to the uniform convergence. The existence of γ x,t is ensured by (4.26) and the Arzelà-Ascoli theorem. Up to the extraction of a subsequence, we can assume that lim inf k→∞ u N k (x, t) = lim k→∞ u N k (x, t) and lim k→∞ γ x,t N k = γ x,t in C([t, T ]; R d ). Using estimate (4.26) again, we get thatγ x,t exists andγ x,t N k →γ x,t in the weak* topology in L ∞ ([0, T ]; R d ). By the weak lower semi-continuity of the cost functional we obtain u(x, t) ≤ T t L(−γ x,t (s), γ x,t (s)) + f (γ x,t (s), ρ(s)) ds + g(γ x,t (T ), ρ(T )) ≤ lim inf k→∞ u N k (x, t).
Thus, by (4.31) we get the pointwise convergence lim k→+∞ u N k (x, t) = u(x, t) ∀ (x, t) ∈ R d × [0, T ], and hence, using that u N k is Lipschitz continuous, with a Lipschitz constant which is independent of k, assertion (ii) follows from the Arzelà-Ascoli theorem. Finally, (iii) is a consequence of the previous analysis with t = 0 and the fact that [12,Corollary 6.4.10] implies that for a.e. x ∈ R d the optimization problem associated with u(x, 0) admits a unique solution.
Recall that, as in the case of (M F G), to each solution (u N , ρ N ) of (M F G N ) we can associate a symmetric equilibrium (m N , . . . , m N ) ∈ A N rel of the N -player game. As a consequence of this fact, Theorem 3.1 and Theorem 4.1, we have the following result.  Let ((u N , ρ N )) N ∈N be a sequence of solutions to (M F G N ) (N ∈ N). Then, there exists a solution (u, ρ) ∈ C([0, T ]; P 1 (R d )) to (M F G) such that, up to some subsequence, u N → u uniformly over compact subsets of R d × [0, T ] and ρ N → ρ in C([0, T ]; P 1 (R d )).

Remark 4.5.
If h = f , g satisfies then the solution (u, ρ) to (M F G) is unique (see [34] and [30,Corollary 5.2]). Since any Lagrangian equilibrium m * can be represented by γ * ♯m 0 , where γ x * ∈ S(x) is uniquely determined for a.e. x ∈ R d , the Lagrangian equilibrium must also be unique. In this case, the results in Theorem 4.1 hold for the entire sequence (u N , ρ N , γ N ) and the result in Corollary 4.1 holds for the entire sequence (u N , ρ N ).
Finally, let us point out that the convergence result in Corollary 4.1 can also be established directly, without appealing to Theorem 3.1, under a stronger regularity assumption than (A2)(ii). Indeed, assume that, in addition to (A2), m 0 ∈ L p (R d ) for some p ∈  Therefore, there exists C > 0 and a compact set K ⊆ R d , both independent of N , such that γ x N (t) ∈ K for a.e. x ∈ supp(m 0 ) and all t ∈ [0, T ]. In particular, supp(ρ N (t)) ⊆ K and d 1 (ρ N (s), ρ N (t)) ≤ C|s − t| for all s, t ∈ [0, T ] and N ∈ N. This implies the existence of ρ ∈ C([0, T ]; P 1 (R d )) such that, up to some subsequence, ρ N → ρ in C([0, T ]; P 1 (R d )) as N → ∞. Since Lemma 4.2 implies that f N (·, ρ N (·)) converges uniformly to f (·, ρ(·)) on compact subsets of R d × [0, T ], standard stability results for viscosity solutions of Hamilton-Jacobi-Bellman equations imply that, up to some subsequence, u N → u uniformly on compact subsets of R d × [0, T ], u being the unique viscosity solution to (4.10). In particular, u N being locally semi-concave with respect to the space variable, uniformly in N , for all t ∈ [0, T ] we have that ∇u N (x, t) → ∇u(x, t) for a.e. x ∈ R d . Using that u N is globally Lipschitz, uniformly in N , by dominated convergence we deduce that (4.32) ∂ ξ H (∇u N (·, ·), ·) → ∂ ξ H (∇u(·, ·), ·) in L 1 ([0, T ]; L s (R d )) for any s ∈ [1, +∞).
On the other hand, using again the uniform local semiconcavity of u N (·, t) and arguing as in the proof Proposition 4.1 we get the existence of c 3 > 0, independent of N , such that (4.33) ρ N (t) L p ≤ c 3 m 0 L p ∀ t ∈ [0, T ].
Using this bound, we obtain that ρ(t) is absolutely continuous w.r.t. the Lebesgue measure for a.e. t ∈ [0, T ], and its density, denoted likewise by ρ(t), satisfies ρ(t) L p ≤ c 3 m 0 L p for a.e. t ∈ [0, T ]. Since ρ ∈ C([0, T ]; P 1 (R d )), the previous bound implies that for all t ∈ [0, T ], the measure ρ(t) is absolutely continuous w.r.t. the Lebesgue measure and the estimate ρ(t) L p ≤ c 3 m 0 L p holds. Moreover, using (4.32)-(4.33) we can pass to the limit in the second equation (M F G N ) to obtain that the pair (u, ρ) solves (M F G).