Subgame Maxmin Strategies in Zero-Sum Stochastic Games with Tolerance Levels

We study subgame ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}-maxmin strategies in two-player zero-sum stochastic games with a countable state space, finite action spaces, and a bounded and universally measurable payoff function. Here, ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document} denotes the tolerance function that assigns a nonnegative tolerated error level to every subgame. Subgame ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}-maxmin strategies are strategies of the maximizing player that guarantee the lower value in every subgame within the subgame-dependent tolerance level as given by ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}. First, we provide necessary and sufficient conditions for a strategy to be a subgame ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}-maxmin strategy. As a special case, we obtain a characterization for subgame maxmin strategies, i.e., strategies that exactly guarantee the lower value at every subgame. Secondly, we present sufficient conditions for the existence of a subgame ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}-maxmin strategy. Finally, we show the possibly surprising result that each game admits a strictly positive tolerance function ϕ∗\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^*$$\end{document} with the following property: if a player has a subgame ϕ∗\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^*$$\end{document}-maxmin strategy, then he has a subgame maxmin strategy too. As a consequence, the existence of a subgame ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document}-maxmin strategy for every positive tolerance function ϕ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi $$\end{document} is equivalent to the existence of a subgame maxmin strategy.


Introduction
We consider two-player zero-sum stochastic games with a countable state space, finite action spaces, and a bounded and universally measurable payoff function. Such a game models a repeated interaction between two players with opposite objectives. The environment in which the interaction takes place is fully characterized by the state variable. The transition from one state variable to the next one is influenced by both players as well as an element of chance. Throughout the paper, we take the perspective of the maximizing player. We are interested in strategies of the maximizing player that guarantee the lower value at every subgame and call such strategies subgame maxmin strategies. Under the assumptions as made in the paper, the value may not exist, which explains why we consider the lower value instead.
As the name subgame maxmin strategy suggests, this concept is closely related to the concept of a subgame perfect equilibrium as defined in Selten ([19]). In two-player zerosum games where the value exists, for conditions see Maitra and Sudderth ([13]) and Martin ([14]), the notions of a subgame maxmin strategy and a subgame minmax strategy coincide with the notion of a subgame optimal strategy. Moreover, in such games a strategy profile is a subgame perfect equilibrium if and only if it consists of a pair of subgame optimal strategies.
As illustrated by the Big Match, a game introduced in Gillette ( [9]) and analyzed in Blackwell and Ferguson ( [2]), even if the value exists, it is not guaranteed that optimal strategies exist, so a fortiori, subgame optimal strategies and subgame perfect equilibria may not exist. A large part of the literature therefore focuses on so-called subgame perfectequilibria as defined in Radner ([17]). This equilibrium concept is more permissive than a subgame perfect equilibrium and consists of a strategy pair such that every player obtains the value at each history up to a fixed error term of /2.
Instead of having a fixed error term at each subgame, we allow the error term to vary across different subgames. This error term is expressed as a function φ of the histories and is called the tolerance function. The central topic of this paper is the concept of a subgame φ-maxmin strategy. This is a strategy of the maximizing player that guarantees the lower value at every subgame within the allowed tolerance level. Intuitively, a subgame φ-maxmin strategy performs sufficiently well across all subgames. This type of strategy is related to the concept of φ-tolerance equilibrium as defined in Flesch and Predtetchinski ( [6]). Indeed, if the value exists, then a strategy profile in which both players use a subgame (φ/2)-optimal strategy is a φ-tolerance equilibrium.
One motivation for letting the tolerance level vary across subgames is given by Mailath, Postlewaite, and Samuelson ( [11]) when introducing the concept of a contemporaneous perfect -equilibrium. The authors focus on games in which the payoff function of the players is given by the discounted sum of periodic rewards. Due to this discounting, there exists a period after which the maximal total discounted reward a player can receive is smaller than . If the allowed tolerance level is fixed across all subgames, any strategy will be an -maxmin strategy of a subgame in such a period. Therefore, the concept of subgame -maxmin strategy does not impose any restrictions on the actions chosen at a very distant future. The issue here is that it would be more intuitive to discount not only the reward but also the allowed tolerance level.
Additional motivation for letting the tolerance level vary across subgames stems from the fact that the notion of what is considered sufficiently good might be relative. For instance, Tversky and Kahneman [21] observe that people evaluate decisions with respect to a reference level. They find that significantly more people were willing to exert extra effort to save $5 on a $15 purchase than to save $5 on a $125 purchase. To apply this to the context of zero-sum games, consider the following game to which we will refer as the high stakes-low stakes game. In the first stage of the game, a chance move determines whether the player will engage in the high stakes or the low stakes variant of this game. The high stakes and the low stakes games are identical in terms of possible strategies. The only difference is that the payoffs in the high stakes game are a thousand fold the payoffs in the low stakes game. Furthermore, assume that in the high stakes subgame, the payoff of player 1 ranges between 0 and 2000 and the value is 1000, while in the low stakes subgame the payoff of player 1 ranges between 0 and 2 and the value is 1. Players that evaluate decisions with respect to a reference level may label a strategy which guarantees a payoff of 999 in the high stakes game as sufficiently good. However, in the low stakes game, 0 corresponds to the minimum payoff. Allowing the tolerance level to vary across subgames can therefore be used to accommodate such behavioral effects into the model of zero-sum stochastic games.
Another case where history-dependent tolerance levels are natural is the following. In situations that commonly occur, a player may use a familiar strategy irrespective of the scale of the payoffs. To understand this better, imagine a player who is highly experienced in playing a certain zero-sum game. He has a trusted strategy which guarantees him the value of this game within some error. Now, consider the high stakes-low stakes game again. The player might well use the trusted strategy in both the low stakes and the high stakes subgame. Therefore, the error related to this strategy will be relative with respect to the value of the respective subgame.
Finally, in the class of stochastic games as identified in Flesch, Thuijsman, and Vrieze ( [8]), the only way to obtain -optimality is to use strategies that are called improving. Improving strategies are closely related to subgame φ-maxmin strategies such that the tolerance level in some subgames is smaller than the tolerance level at the root.
With respect to the concept of subgame φ-maxmin strategies, this paper attempts to provide answers to the following two fundamental questions: 1. For positive tolerance functions φ, when does a subgame φ-maxmin strategy exist? 2. How is the existence of a subgame φ-maxmin strategy related to the existence of a subgame maxmin strategy?
To answer these questions we will first derive necessary and sufficient conditions for a strategy to be a subgame φ-maxmin strategy. This is done in Sect. 4. As a special case of these necessary and sufficient conditions, we obtain a characterization of subgame maxmin strategies. For the special class of positive and negative stochastic games, a related characterization of subgame maxmin strategies was obtained by Flesch, Predtetchinski, and Sudderth ( [7]). The necessary and sufficient conditions for strategies to be subgame φ-maxmin can be split into a local condition and an equalizing condition. Informally, the local condition states that the lower value one expects to get in the next subgame should always be at least the lower value of the current subgame. The equalizing condition requires that for every strategy of the other player, a subgame φ-maxmin strategy almost surely results in a play with an eventually good enough payoff, where eventually good enough means being at least the lower value in very deep subgames up to the allowed tolerance level.
In Sect. 5, we consider the question of existence of subgame φ-maxmin strategies for positive tolerance functions φ. We prove that for a positive tolerance function φ, a subgame φ-maxmin strategy exists if every play is either a point of upper semicontinuity of the payoff function or if the sequence of tolerance levels which occur along the play has a positive lower bound. The novel aspect of the theorem is the way in which it encompasses a number of important special cases, some of which have been studied extensively in the literature. One special case of interest is when the payoff function is upper semicontinuous everywhere.
In this case, the existence of subgame maxmin strategies follows by a result of Laraki, Maitra, and Sudderth ( [10]). Another is the case when along each play tolerance levels remain bounded away from zero. This latter case subsumes the familiar class of all positive constant tolerance functions. When the tolerance level is equal to a constant at each history, subgame φ-maxmin strategies are exactly the so-called subgame -optimal strategies. Existence of subgame -optimal strategies for each > 0 is established in Mashiah-Yaakovi ( [15]).
Our technique relies on switching between strategies: one starts by following a φ(x 0 ) 2optimal strategy until a history (say h) is reached where it ceases to be φ(h)-optimal. At that moment, one discards the original strategy and switches instead to a φ(h) 2 -optimal strategy, and so on. This technique is well-attested in the literature, and in particular is used in Mashiah-Yaakovi ( [15]). We refine the method to accommodate both types of plays: those where the tolerance levels remain bounded away from zero, and those where they converge to zero (the latter are, by assumption, the plays where the payoff function is upper semicontinuous).
In Sect. 6, we study the relation between the existence of subgame φ-maxmin strategies and subgame maxmin strategies. Our main result states that the existence of a particular subgame φ * -maxmin strategy, with φ * > 0, is equivalent to the existence of a subgame maxmin strategy. Consequently, the existence of subgame φ-maxmin strategies for all positive tolerance functions φ is equivalent to the existence of a subgame maxmin strategy. In the special case of an upper semi-continuous payoff function, the results of Sects. 5 and 6 imply the existence of a subgame maxmin strategy. As noted above, this result is not new, and is a consequence of Laraki, Maitra, and Sudderth ( [10]).
The connection between existence of subgame φ-maxmin strategies for every positive tolerance function φ and the existence of subgame maxmin strategies is not only useful to further understand the results obtained by Laraki, Maitra, and Sudderth ( [10]) but also highlights an important and surprising difference between subgame φ-maxmin strategies and the closely related concept of subgame -maxmin strategies. Indeed, the existence of a subgame -maxmin strategy for every > 0 does not imply the existence of a subgame maxmin strategy.
The rest of the paper is structured as follows. In Sect. 2, we formulate the model setup, and in Sect. 3, we formally define the main concepts. Then in Sect. 4, we discuss the necessary and sufficient conditions for a strategy to be a subgame φ-maxmin strategy and give a characterization for subgame maxmin strategies. We continue in Sect. 5 by providing sufficient conditions to guarantee the existence of a subgame φ-maxmin strategy. Then in Sect. 6, we explain why the existence of subgame φ-maxmin strategies for every positive tolerance function φ is equivalent to existence of a subgame maxmin strategy. Finally, in Sect. 7, we discuss the importance of the assumptions we made and whether they might be relaxed.

Two-Player Zero-Sum Stochastic Games
We consider a two-player zero-sum stochastic game with finitely many actions and countably many states. The payoff function is required to be bounded and universally measurable. The model encompasses all two-player zero-sum games with perfect information and stochastic moves. Actions, states, and histories The action sets of players 1 and 2 are given by the finite sets A and B, respectively. The state space is given by the countable set X . Let x 0 denote the initial state and define the set Z = A × B × X . The game consists of an infinite sequence of one-shot games. At the initial state x 0 , the one-shot game G(x 0 ) is played as follows: Players 1 and 2 simultaneously select an action from their respective action sets, denoted by a 1 and b 1 , respectively. Then, the next state x 1 is selected according to the transition probability q(·|x 0 , a 1 , b 1 ). At the new state x 1 , this process repeats itself and both players play the one-shot game G(x 1 ) by selecting actions a 2 and b 2 from their respective action sets. The next state x 2 is selected according to the transition probability q(·|x 0 , a 1 , b 1 , x 1 , a 2 , b 2 ). This goes on indefinitely and creates a play p = (x 0 , a 1 , b 1 , x 1 , a 2 , b 2 , . . .). Note that the transition probability can depend on the entire history.
For every t ∈ N = {0, 1, 2, . . .}, let H t = {x 0 } × Z t denote the set of all histories that are generated after t one-shot games. The set H 0 consists of the single element x 0 . For t ≥ 1, x t ) = t denote the number of one-shot games that occurred during the history h. We refer to h as the length of the history h. For all t ≤ h , the restriction of the history h to the first t one-shot games is denoted by h is a prefix of p. For every h ∈ H, let P(h) = {p ∈ P|h ≺ p} denote the cylinder set consisting of all plays which extend history h.
We endow Z with the discrete topology and P with the product topology. The collection of all cylinder sets is a basis for the product topology on P.
For t ∈ N, let F t be the sigma-algebra generated by the collection of cylinder sets Let F ∞ be the sigma-algebra generated by ∪ t∈N F t . This is exactly the Borel sigma-algebra generated by the product topology on P. The sigma-algebra of universally measurable subsets of P is denoted by F . Elements of F are sets that belong to the completion of each Borel probability measure on P. For details of the definition of the sigma-algebra F , the reader is referred to Appendix A. It holds that F t ⊆ F t+1 ⊆ · · · ⊆ F ∞ ⊆ F . The set of plays P together with the universally measurable sigma-algebra F determines a measurable space (P, F ). A stochastic variable is a universally measurable function from P to R. Strategies Let (A) denote the set of probability measures over the action set of player 1 and (B) the set of probability measures over the action set of player 2. A behavioral strategy for player 1 is a function σ : H → (A). Hence, at each history player 1 chooses a mixed action. A pure strategy for player 1 is a function that assigns to every history an action, with a minor abuse of notation, σ : H → A. Similarly, one can define a behavioral and a pure strategy τ for player 2. Let S 1 and S 2 denote the sets of behavioral strategies of players 1 and 2, respectively.
It follows from the Ionescu Tulcea extension theorem that every history h ∈ H and strategy profile (σ, τ ) ∈ S 1 × S 2 determine a probability measure P h,σ,τ on the measurable space denotes the Borel sigma-algebra over the set of plays extending the history h. The measure P h,σ,τ can be extended to the measurable space (P, F ∞ ) in the obvious way. Taking the completion of the probability space (P, F ∞ , P h,σ,τ ) yields the probability space (P, F , P c h,σ,τ ). With a minor abuse of notation, we will omit the superscript c and write P h,σ,τ in the remainder of this paper. Payoff function We assume that the payoff function u : P → R of player 1 is bounded, where we define M = sup p∈P |u( p)|, and universally measurable. We also assume the game to be zero-sum. The payoff function of player 2 is therefore given by −u. We denote the game as described above by x 0 (u). Throughout the paper, we take the point of view of player 1. This is without loss of generality, since the situation of player 2 in the game x 0 (u) is identical to that of player 1 in the game x 0 (−u).
The expected payoff of player 1 corresponding to strategy profile (σ, τ ) ∈ S 1 × S 2 at history h ∈ H is given by E h,σ,τ [u] , where the expectation is taken with respect to the probability measure P h,σ,τ . The expected payoff of player 1 at the history x 0 is denoted by We imposed the condition above that the payoff function u is universally measurable. As Borel measurable functions are also universally measurable, this condition is weaker than requiring that u be Borel measurable. The main difference between these two conditions is the following. When the payoff function u is Borel measurable, the game is known to admit a value (cf. Sect. 3 for the formal definition). However, it is not provable from the standard ZFC axioms of set theory 1 that our games with a universally measurable payoff function also admit a value. Therefore, instead of the value, in the remaining part of the paper we will use the lower value (cf. Sect. 3 for the formal definition). We would like to emphasize that except for the difference between the concepts of the value and the lower value, assuming Borel measurable payoffs would not lead to any simplification in the proofs. So the reader unfamiliar with universally measurable functions can read the entire paper imagining Borel measurable payoffs.

Subgame -Maxmin Strategies
We start this section by defining subgame φ-maxmin strategies. Then, we discuss an illustrative example. Finally, we define and examine guarantee levels of strategies.

Definition of Subgame -Maxmin Strategies
For every game x 0 (u), for every history h ∈ H, we define the lower value v(h) and the Because the payoff function u is assumed to be bounded, we have that v(h), v(h) ∈ R. Therefore, the lower and upper value exist in every subgame of x 0 (u). Furthermore, we we say that the subgame at history h has a value and we denote it by v(h). The lower value, the upper value, and the value at the initial state x 0 are denoted by v, v, and v, respectively. If u is Borel measurable, then the value exists by Maitra and Sudderth ([13]) and Martin ([14]). Since we do not assume u to be Borel measurable, we present our results in terms of the lower value. Even if the value exists, player 1 may not have a strategy that guarantees the value in each subgame and the literature has therefore studied subgame -optimal strategies. These are strategies which guarantee the value in each subgame up to an allowed error term of . If the payoff function is bounded and Borel measurable, it has been shown by Mashiah-Yaakovi ( [15]) that for each > 0 player 1 has a subgame -optimal strategy. The concept . . . 0 Fig. 1 Characterization of φ such that pure subgame φ-maxmin strategies exist of a subgame -optimal strategy has a constant error term across all subgames. However, as argued in the introduction, there are instances in which it is more natural to consider a variable error term. Therefore, instead of considering a constant error term, we follow Flesch and Predtetchinski ( [6]), who allow the error term to vary across histories in their investigation of the φ-tolerance equilibrium. This leads us to the study of subgame φ-maxmin strategies, where φ : H → [0, ∞) is a tolerance function assigning an allowed tolerance level to each history.
In case φ is identically equal to zero, we omit it from the notation, and simply refer to a subgame maxmin strategy.
A subgame φ-maxmin strategy guarantees at each history h of the game the lower value up to the tolerance level φ(h). If the tolerance function is such that for some ≥ 0, for every h ∈ H, φ(h) = , then we refer to the strategy as a subgame -maxmin strategy. If the value exists, then the notion of subgame -maxmin strategy coincides with the notion of subgame -optimal strategy. The following example illustrates that even for a strictly positive tolerance function φ player 1 may not have subgame φ-maxmin strategies. Interestingly, however, player 1 has a subgame -maxmin strategy for every positive > 0, which in fact is pure. We also give an example of a tolerance function φ for which player 1 has a subgame φ-maxmin strategy, but not a pure such strategy. Fig. 1 corresponds to a two-player zero-sum stochastic game in which the state space is trivial and the second player is a dummy player. Whenever the state space or action sets are degenerate, the corresponding states and actions are omitted from the notation in examples. The set of actions of player 1 is A = {c, q}, where c stands for continue and q for quit. The game stops as soon as player 1 chooses to quit. If player 1 decides to quit at period t, then his payoff is t/(t + 1). If player 1 never quits, his payoff is 0.

Example 3.2 The decision problem depicted in
In this game, player 1 has a subgame -maxmin strategy for every positive > 0. For example, the strategy which quits whenever quitting leads to a payoff of at least 1 − .
We now turn to the existence of a subgame φ-maxmin strategy. Clearly, there exist no subgame maxmin strategy. As we will see later, Theorem 6.1 then implies that there is some strictly positive tolerance function φ for which there does not exist a subgame φ-maxmin strategy. Intuitively, such a tolerance function forces player 1 to continue with such a large probability that the total probability of never quitting becomes nearly one.
Interestingly, there is a tolerance function φ for which player 1 has a subgame φ-maxmin strategy, but for which he has no pure strategy that is subgame φ-maxmin. To see this, define the tolerance function φ by φ(c t ) = 1 − 1 2 · t t+1 − 1 2 · t+1 t+2 for each t ∈ N. Now consider the strategy σ which always chooses action c with probability 1/2 and action q with probability 1/2. In the subgame at any history c t , it quits immediately with probability 1/2 and quits at a later period with probability 1/2. In the former case, it gives payoff t t+1 , whereas in the latter case, it gives an expected payoff of at least t+1 t+2 . Hence, the strategy σ is a subgame φ-maxmin strategy. Yet, there is no pure strategy that is subgame φ-maxmin. Indeed, the tolerance function does not allow a pure strategy to quit at any period, whereas a pure strategy that always continues is not subgame φ-maxmin either. As we show in the working paper version of our paper, Flesch, Herings, Maes, and Predtetchinski ( [5]), a pure subgame φ-maxmin strategy exists if and only if the following two conditions hold: (1) for every t ∈ N, φ(c t ) > 0, and

Guarantee Levels
To identify subgame φ-maxmin strategies, it is useful to define the function u : The payoff u(σ, h) corresponds to the guarantee level that player 1 is expected to receive at history h when playing the strategy σ .
For every strategy profile (σ, τ ) ∈ S 1 × S 2 , for every t ∈ N, define the stochastic variables , respectively, for each play p ∈ P. All three stochastic variables are F t -measurable. We have U t σ ≤ U t σ,τ and U t σ ≤ V t everywhere on P. The next lemma states the submartingale property of guarantee levels. It says that the guarantee level that player 1 can expect to receive increases over time.

Lemma 3.3 (Submartingale property of guarantee levels)
Let a strategy profile (σ, τ ) ∈ S 1 × S 2 , t ∈ N, and a history h ∈ H t of length t be given. The process We have that To prove that the process (U t+n σ ) n∈N is a P h,σ,τ -submartingale, note that the conditional expectation E h,σ,τ (U t+n+1 σ | F t+n ), when evaluated at a play p ∈ P(h), equals E p| t+n ,σ,τ (U t+n+1 σ ), which, by the claim already proven, is at least u(σ, p| t+n ) = U t+n (σ )( p).

n-Day Maxmin Strategies and Equalizing Strategies
The goal of this section is to provide necessary and sufficient conditions for a strategy to be subgame φ-maxmin and to provide a characterization of subgame maxmin strategies. Definition 4.1 A strategy σ ∈ S 1 is an n-day φ-maxmin strategy in the game x 0 (u) if for every t ∈ N, for every history h ∈ H t of length t, and for every strategy τ ∈ S 2 , (4.1) When φ = 0, we use the terms n-day maxmin and equalizing to mean n-day 0-maxmin and 0-equalizing, respectively.
The first definition is very intuitive. It says that player 1 should play in such a way that, on average, the lower value increases over time. The notion of 1-day maxmin strategies is particularly well-known in dynamic programming and stochastic games, see Puterman ([16]). A simple characterization of 1-day maxmin strategies is provided in following theorem.

Theorem 4.3
Consider a strategy σ ∈ S 1 in the game x 0 (u). The following three conditions are equivalent: 1. For each n ∈ N, σ is an n-day maxmin strategy. 2. σ is a 1-day maxmin strategy.

For each history h ∈ H t of length t and each strategy
Proof That condition 1 implies condition 2 is obvious. That condition 2 implies condition 3 follows by the law of iterated expectations. Finally, that condition 3 implies condition 1 follows from the properties of a submartingale.
Note that the equivalence in Theorem 4.3 is no longer true for nonzero tolerance functions. For example, if φ > 0, then a 1-day φ-maxmin strategy is generally not 2-day φ-maxmin.
A strategy is φ-equalizing if, roughly speaking, it almost surely results in a play with an eventually good enough payoff, where eventually good enough means being about as large as the lower value in very deep subgames.
The following example illustrates both the notion of a 1-day maxmin strategy and an equalizing strategy. Fig. 2. At every history, the active player can choose to continue (c) or to quit (q). As soon as a player decides to quit the game ends

Example 4.4 Consider the centipede game shown in
A pure strategy σ ∈ S 1 is then easily seen to be a 1-day maxmin strategy if and only if σ (c 2t ) = c for every t ∈ N.
On the other hand, a pure strategy σ ∈ S 1 is equalizing if and only if for infinitely many t ∈ N it holds that σ (c 2t ) = q. Indeed, for a strategy σ such that q is chosen for infinitely many t ∈ N the generated play ends with q after each history h ∈ H irrespective of τ ∈ S 2 , which shows that σ is equalizing. On the other hand, if σ is such that q is chosen for finitely many t ∈ N, then there is a history h ∈ H such that player 1 always plays continue after this history. If τ ∈ S 2 is such that player 2 always chooses continue, then the play c ∞ with u(c ∞ ) = 0 results, which is strictly less than lim sup t→∞ v(c t ) = 1, so such a σ is not equalizing.
It follows that player 1 does not have a pure strategy which is both 1-day maxmin and equalizing.
The following theorem states sufficient conditions under which a strategy σ of player 1 is a subgame φ-maxmin strategy.
Proof Let φ 1 , φ 2 , and σ be such that the conditions in the theorem are satisfied. We show that σ is a subgame φ-maxmin strategy.
Fix a history h ∈ H t and a strategy τ ∈ S 2 . Then, we have that where the second inequality holds since σ is an n-day φ 1 -maxmin strategy, the third inequality is by Fatou lemma, and the last inequality holds since σ is φ 2 -equalizing.
According to Theorem 4.5, to conclude that σ is a subgame φ-maxmin strategy, it is enough to find tolerance functions φ 1 and φ 2 such that at each history their sum does not exceed φ and the strategy σ is both n-day φ 1 -maxmin and φ 2 -equalizing.
Particularly natural are situations where the tolerance level does not increase as time progresses. More formally, the tolerance function φ is said to be non-increasing if φ(h) ≥ φ(h ) whenever h ≺ h . The following result states necessary conditions for a strategy to be subgame φ-maxmin. Theorem 4.6 (Necessary condition) Let σ ∈ S 1 be a subgame φ-maxmin strategy in the game x 0 (u). Then, it holds that: 1. For every n ∈ N, σ is an n-day φ-maxmin strategy. 2. If the tolerance function φ is non-increasing, then σ is φ-equalizing.
Proof Let σ ∈ S 1 be a subgame φ-maxmin strategy in the game x 0 (u). Take a history h ∈ H t of length t and a strategy τ ∈ S 2 .
We prove condition 1. We have . The second inequality holds by Lemma 3.3. The next equation holds since t is the length of history h, and the final inequality holds since σ is a subgame φ-maxmin strategy.
We prove condition 2. We have, P h,σ,τ -almost surely, where the equality is by Levy's zero-one law (Lemma A.1 in Appendix A), the first inequality follows since σ is a subgame φ-maxmin strategy, and the second inequality holds since φ is non-increasing.
Notice that in the case of a nonzero tolerance function, the necessary and sufficient conditions do not coincide, and we do not obtain a characterization of subgame φ-maxmin strategies. We now turn to the case where the tolerance function φ is identically equal to 0.

Corollary 4.7 A strategy σ ∈ S 1 is a subgame maxmin strategy in the game x 0 (u) if and only if it is 1-day maxmin and equalizing.
When we compare the sufficient conditions of Theorem 4.5 to the sufficient conditions of Corollary 4.7, we notice that in the case of a non-zero tolerance function we require the strategy to be n-day φ-maxmin. In the case of a zero tolerance function, the corresponding sufficient conditions only require the strategy to be 1-day maxmin. The reason for this difference is that we should avoid that strategies accumulate the allowed tolerance levels, causing them to become too permissive over time. The following example illustrates this issue. Fig. 3. Each period the decision maker can choose to continue (c) or to quit (q). Notice that v(c t ) = 1/(t + 1) and hence lim t→∞ v(c t ) = 0. Any strategy of player 1 is therefore equalizing. Furthermore, it is clear that in this decision problem the subgame maxmin strategy is unique and requires the decision maker to quit at each time. Now suppose the decision maker has the following tolerance function:

Example 4.8 Consider the decision problem depicted in
Consider the strategy σ where the decision maker always chooses to continue. From the definition of the tolerance function, it follows that the strategy σ is a 1-day φ-maxmin It is also easily seen that the strategy σ is equalizing. Nevertheless, it is clear that σ is not a subgame φ-maxmin strategy.
The underlying problem with the strategy σ is that every time the decision maker chooses to continue this causes an acceptable loss in the value, but over time these losses add up to an unacceptable loss. The requirement that for every n ∈ N a strategy is n-day φ-maxmin guarantees that the accumulated losses over any finite period of time remain acceptable.
If we require a strategy to be subgame maxmin, then we do not tolerate any losses. Therefore, the accumulation problem mentioned above will never occur, and it will be sufficient to only require that the strategy is 1-day maxmin. ♦ Example 4.9 is such that player 1 has a maxmin strategy in every subgame, but has no subgame maxmin strategy. From Corollary 4.7, it follows that any subgame maxmin strategy needs to be both 1-day maxmin and equalizing. The example presents a game where all the strategies are 1-day maxmin but none of them is equalizing and therefore subgame maxmin strategies do not exist.

Example 4.9
Consider the following perfect information game. Both players have two actions, left (L) and right (R), so A = B = {L, R}. The players take turns playing an action, which generates a play p ∈ {L, R} N . Let r 1 ( p) and r 2 ( p) denote the number of times that player 1 and player 2, respectively, play action R during the play p and define r ( p) = min{r 1 ( p), r 2 ( p)}. Player 1 obtains a payoff of 0 if both players choose R infinitely often. When at least one of them chooses R only a finite number of times, then player 1 receives a payoff of r ( p)/(r ( p) + 1). The payoff function u is therefore obtained by defining, for p ∈ {L, R} N , At each history h ∈ H, the value of the game exists and is given by v(h) = r 2 (h)/(r 2 (h)+ 1), where r 2 (h) denotes the number of times player 2 has chosen R in the history h. Indeed, player 1 can guarantee this payoff by choosing the action R max{r 2 (h) − r 1 (h), 0} times after history h. Player 2 can guarantee to lose at most this amount by playing only left after history h. Hence at every history h ∈ H, player 1 has a maxmin strategy.
For every history h ∈ H where player 1 takes an action and for every action a ∈ A, we have that r 2 (ha) = r 2 (h) and v(ha) = v(h). Therefore, all strategies of player 1 are 1-day maxmin.
On the other hand, no equalizing strategy exists for player 1. To see this, take any strategy σ ∈ S 1 for player 1 and consider the strategy τ ∈ S 2 in which player 2 always chooses R. Let p be the play generated by the strategy profile (σ, τ ). Then, we have that u( p) < 1 and lim t→∞ v( p |t ) = 1. It follows that σ is not equalizing. Using Corollary 4.7, we conclude that player 1 does not have a subgame maxmin strategy.

The case of an upper semi-continuous payoff function
The remainder of this section is devoted to the case where the payoff function is upper semicontinuous. We argue that in this case, any strategy of player 1 is equalizing. Because all upper semi-continuous functions are Borel measurable and because we assumed finite action sets and a countable state space, the value exists, see Maitra and Sudderth ([13]) and Martin ([14]), and the lower value equals the value.
The function u is upper semi-continuous at a play p ∈ P if for every sequence { p t } t∈N of plays converging to p it holds that Since this holds for every > 0, the lemma follows.
In view of Lemma 4.10, we obtain the following corollary to Theorems 4.5, 4.6, and 4.7.
Corollary 4.11 Let x 0 (u) be such that u is upper semi-continuous. Then, each strategy of player 1 is equalizing. The strategy σ ∈ S 1 is a subgame φ-maxmin strategy if and only if for every n ∈ N it is an n-day φ-maxmin strategy. The strategy σ ∈ S 1 is a subgame maxmin strategy if and only if it is a 1-day maxmin strategy.

Example 4.12 (Staying in the set game)
Let some subset X * of X be given. For a play . . ), we define u( p) to be 1 if x t ∈ X * for every t ∈ N and to be 0 otherwise. Maitra and Sudderth ( [12]) refer to such a payoff function as "staying in a set" and in the computer science literature it goes under the name of "safety objective," see Bruyaère ( [4]). Since u is upper semi-continuous, any strategy σ ∈ S 1 is equalizing.

Example 4.13
Consider again the centipede game shown in Figure 2, but with one slight modification. If the game continues indefinitely, then player 1 receives a payoff of 2 instead of 0. The payoff function is now upper semi-continuous. As argued in Example 4.4, the strategy σ in which player 1 continues at each history is 1-day maxmin. Because of Corollary 4.11, we can conclude that the strategy σ is a subgame maxmin strategy.
Note that the limit average payoff is generally not upper semi-continuous. Hence, in games where the payoff function is given by the limit average payoff, there may exist strategies of the maximizing player which are not equalizing.

Existence of subgame -maxmin strategies
The goal of this section is to give sufficient conditions for the existence of a subgame φmaxmin strategy if the tolerance function φ is positive. We will prove the following theorem.
Theorem Let a game x 0 (u) and a tolerance function φ > 0 be given such that for every p ∈ P at least one of the following two conditions holds: 1. (point of upper semicontinuity) The function u is upper semi-continuous at p. 2. (positive limit inferior) lim inf t→∞ φ( p |t ) > 0.
Then, there exists a subgame φ-maxmin strategy in the game x 0 (u).
Two special cases of the above theorem are worth noting. One is when the payoff function is everywhere upper semicontinuous. In this special case, the existence of a φ-maxmin strategy (in fact, of a maxmin strategy) follows by a result of Laraki, Maitra, and Sudderth ( [10]). We note that the latter result is not entirely subsumed by the theorem above, since the authors allow for a general Borel state space. Another special case of interest is when along each play, the tolerance remains bounded away from zero (i.e., if each play satisfies condition 2 of the theorem). This special case, to the best of our knowledge, has not been explicitly considered in the literature. However, it does encompass the familiar situation of a constant tolerance function: φ(h) ≡ > 0. For such tolerance functions, φ-maxmin strategies are exactly subgame -optimal strategies, and existence follows by Proposition 11 in Mashiah-Yaakovi ( [15]). Note that the case when all plays satisfy condition 2 of the theorem is more general than the case of a constant tolerance function, since no uniform lower bound on the tolerance is imposed. A novel feature of the above theorem is the way it combines these various special cases. We note that the theorem remains original even if we restrict our attention to Borel measurable payoff functions.
The main technique used in the proof of this theorem are switching strategies. The idea of switching between progressively "more optimal" strategies is certainly not new. A similar construction is used for example in Rosenberg, Solan, and Vieille ( [18]), Solan and Vieille ( [20]), and Mashiah-Yaakovi ( [15]). However, the method needs to be fine-tuned, in order for it to accommodate both the plays where the tolerance remains bounded away from zero (condition 2), as well as those where the payoff function is upper semicontinuous (condition 1). It is not straightforward to use this proof technique when we have a general Borel measurable state space as the selection of switching strategies would need to satisfy measurability conditions. It is therefore an interesting open question whether the result can be extended to that case.
The switching strategy σ φ is formally defined in Sect. 5.1. To analyze its properties, we define in Sect. 5.2, for every k ∈ N, a strategy σ k that coincides with σ φ as long as at most k switches have been made and it stops switching thereafter. We also present three technical lemmas for finite switching strategies in this subsection, the most important one showing that the guaranteed expected payoff of a strategy σ k is increasing in k. Sect. 5.3 uses the lemmas of Sect. 5.2 to show the desired existence result in Theorem 5.7.

Definition of the Switching Strategy
Fix a tolerance function φ > 0. For every history h ∈ H, player 1 has a (φ(h)/2)-maxmin strategy for the subgame at history h, denoted by σ h . The function ψ : H → H maps histories into histories and is such that player 1 is going to use strategy σ ψ(h) at subgame h ∈ H. The function ψ is used to describe when player 1 switches strategies and is recursively defined by setting ψ(x 0 ) = x 0 and, for every h ∈ H, for every z ∈ Z, The condition u(σ ψ(h) , hz) ≥ v(hz)−φ(hz) verifies whether the strategy to which player 1 switched last, σ ψ(h) , is a φ(hz)-maxmin strategy for the subgame at history hz. If this is the case, then there is no need to switch and ψ(hz) = ψ(h). Otherwise, player 1 switches to σ hz , which is achieved by setting ψ(hz) = hz. Formally, we define the switching strategy The following example illustrates the switching strategy σ φ and shows that it may not be subgame φ-maxmin.
Let h ∈ H be the active history and let k ∈ N be such that h = c 2k or h = c 2k+1 . The strategy σ h in which player 1 chooses continue at periods 0, 2, . . . , 2k and chooses quit at every later period, i.e., is a maxmin strategy at subgame h. We now consider the switching strategy σ φ . We show by induction that for every h ∈ H , for every z ∈ Z , it holds that ψ(hz) = hz if hz is an active history of player 1 and ψ(hz) = h if hz is an active history of player 2. The statement trivially holds for the initial history. Let h be an active history of player 2 and let t ∈ N be such that h = c 2t+1 . Since σ h = σ c 2t is a maxmin strategy at subgame h, it holds that ψ(c 2t+1 ) = c 2t . Let h be an active history of player 1 and let t ∈ N \ {0} be such that h = c 2t . We have that so ψ(c 2t ) = c 2t . Since the tolerance function φ is so stringent, it forces player 1 to switch at each of his active histories. For every t ∈ N, it holds that σ φ (c 2t ) = σ c 2t (c 2t ) = c, so under σ φ player 1 chooses c at each of his active histories. The strategy σ φ is not a subgame φ-maxmin strategy as it fails to be φ-equalizing, see Example 4.4.

Definition and Analysis of Finite Switching Strategies
Given the switching strategy σ φ , for every k ∈ N, we define the strategy σ k : H → (A) such that it coincides with σ φ as long as at most k switches have been made and it stops switching thereafter. Formally, we recursively define the function κ : H → N which counts the number of switches along a history h by setting κ(x 0 ) = 0 and, for all histories h, hz ∈ H, For every k ∈ N, we define the stopping time T k : P → N ∪ {∞} by T k ( p) = inf{t ∈ N|κ( p |t ) = k}, p ∈ P. (5. 2) The stopping time T k indicates the time at which switch k occurred. Since, for t < ∞, the expression T k ( p) ≤ t only depends on the history up to period t, it holds that the set { p ∈ P|T k ( p) ≤ t} is F t -measurable and T k is a stopping time indeed.
We now formally define σ k : H → (A). Take any p ∈ P(h) and let If κ(h) > k, then the time at which switch k has occurred is the same for every p ∈ P(h), so it holds that σ k is well-defined. For every k ∈ N, let R k ⊆ P be the set of plays along which at least k switches occur, Furthermore, let R ∞ ⊆ P denote the set of plays along which infinitely many switches occur, The remainder of the section is devoted to the proof of three technical lemmas concerning the finite switching strategies. These lemmas are needed in the analysis of σ φ .
Consider the strategies σ k , σ k+1 , . . . . All these strategies agree with σ φ for as long as σ φ does not require more than k switches. Consequently, the measures that these strategies induce on P assign the same probability to any event that is "determined" before switch k + 1 occurs, i.e., to any event in the sigma-algebra F T k+1 .

Lemma 5.2
Let a strategy τ ∈ S 2 , a history h ∈ H, and some k ∈ N be given. For σ = σ k , σ k+1 , . . . , σ φ , the probability measures P h,σ,τ all coincide on the sigma-algebra F T k+1 . Furthermore, these probability measures all agree on each universally measurable subset of P \ R k+1 .
Proof A set A of the universally measurable sigma-algebra F is called agreeable if for σ = σ k , σ k+1 , . . . , σ φ the measures P h,σ,τ all assign the same probability to A.
We argue first that each cylinder set in F T k+1 is agreeable. A cylinder set P(h) is a member of F T k+1 if and only if κ(h) ≤ k + 1. Let a cylinder set P(h ) in F T k+1 be given. Since κ(h ) ≤ k + 1, we know that κ(h ) ≤ k for each history h preceding h . It follows that σ k (h ) = σ k+1 (h ) = · · · = σ φ (h ). Since this applies to each history h that precedes h , the set P(h ) is agreeable. Now take any E ∈ F T k+1 . For t ∈ N, let E t = E ∩ {p ∈ P | T k+1 ( p) = t} and let E ∞ = E ∩ {p ∈ P | T k+1 ( p) = ∞}. To show that E is agreeable, it suffices to show that the sets E t and E ∞ are.
Let some t ∈ N be given. We know that E t is a member of F t . Consequently, E t can be written as a disjoint union of cylinder sets in F t , say E t = ∪{C n | n ∈ N}, with each C n a member of F t . Since each C n is a subset of the set { p ∈ P | T k+1 ( p) = t}, it is a member of F T k+1 , so is agreeable by the result of the second paragraph in the proof. It now follows that E t is agreeable.
To show that E ∞ is agreeable, we make use of the fact that E ∞ is a Borel set and of the regularity of σ on the Borel sigma-algebra. Let O be any open subset of P containing E ∞ . The set O can be written as a disjoint union of cylinder sets, say O = ∪{P(h n ) | n ∈ N}. Without loss of generality it holds that for every n ∈ N the set P(h n ) has a point in common with E ∞ . Thus in particular, there is p ∈ P(h n ) with T k+1 ( p) = ∞. This implies that κ(h n ) ≤ k and hence that P(h n ) is a member of F T k+1 . We conclude that each P(h n ) is agreeable by the result of the second paragraph in the proof. It follows that O is an agreeable set.
To prove the second claim, we notice that all Borel subsets of P \ R k+1 = {p ∈ P | T k+1 ( p) = ∞} are members of F T k+1 , so are agreeable. The result for universally measurable subsets of { p ∈ P | T k+1 ( p) = ∞} follows since each such set differs from a Borel set by a negligible set.
Let T be a stopping time. We define the stochastic variable U T σ by letting it agree with U t σ on the set { p ∈ P : T ( p) = t} for each t ∈ N and by letting it agree with U ∞ σ on the set { p ∈ P : T ( p) = ∞}, where U ∞ σ is some suitably chosen F ∞ -measurable stochastic variable. The next lemma relates the guaranteed expected payoffs of strategies σ k and σ k+1 at the moment of switch k + 1. We write t to denote the F t -measurable stochastic variable given by t ( p) = φ( p |t ).
Proof Let some p ∈ P be given. We distinguish the following two cases. Case 1: T k+1 ( p) < ∞. In this case at least k + 1 switches occur along the play p. For h = p |T k+1 ( p) , we have the following inequalities where the first inequality holds since σ k is not a φ(h)-maxmin strategy for the subgame at history h and the second inequality holds because the strategy σ k+1 is a (φ(h)/2)-maxmin strategy for the subgame at history h. Since I (T k+1 ( p) < ∞) = 1, (5.6) holds. Case 2: T k+1 ( p) = ∞. In this case, we have σ k+1 ( p). Thus, (5.6) holds as an equality.
The following lemma states the intuitive property that for histories with less than k + 1 switches or histories at which switch k + 1 occurs, strategy σ k+1 guarantees at least the same payoff to player 1 than strategy σ k .

Lemma 5.4 Let t ∈ N and a history h ∈ H t of length t be given. Let k ∈ N be such that T k+1 ( p) ≥ t for every p ∈ P(h). Then, it holds that u(σ k , h) ≤ u(σ k+1 , h).
Proof Fix strategy τ ∈ S 2 of player 2.
We first define U ∞ σ k . Consider the probability measure Q on the measurable space (P, F ) given by Q(A) = k∈N 2 −k−1 P h,σ k ,τ (A) for each A ∈ F . Letū be an F ∞ -measurable stochastic variable with the property thatū = u, Q-almost surely. Since P h,σ k ,τ is absolutely continuous with respect to Q, it holds thatū = u, P h,σ k ,τ -almost surely, for every k ∈ N. We define U ∞ σ k to be equal toū, for every k ∈ N. We now obtain the following inequalities. First, by Lemma 3.3 and the optional sampling theorem with unbounded stopping times as presented in Theorem 8. 16 2 We verify that the conditions of Theorem 8.16 in Yeh ( [22]) applied to the process (U n σ ) n≥t on the measurable space (P, F , P h,σ,τ ) with filtration (F n ) n≥t are satisfied. The process (U n σ ) n≥t is a P h,σ,τ -submartingale with respect to the filtration (F n ) n≥t by Lemma 3.3. Take ξ of Theorem 8. 16  Secondly, from the fact that U T k+1 σ k is an F T k+1 -measurable stochastic variable, we obtain by Lemma 5.2 that Thirdly, we know that where the first of these inequalities follows from Lemma 5.3 and the second one follows by Lemma 3.3 and the optional sampling theorem with unbounded stopping times as presented in Theorem 8.16 of Yeh ([22], p. 139). Taking the expectation of the last array of inequalities with respect to P h,σ k+1 ,τ and making use of the law of iterated expectation yields Combining these facts yields u(σ k , h) ≤ E h,σ k+1 ,τ [u]. Taking the infimum over all strategies τ of player 2 completes the proof.

Properties of the Switching Strategy
We show first that the switching strategy σ φ is always n-day φ-maxmin for every n ∈ N.

Lemma 5.5 Let a game x 0 (u) and a tolerance function φ > 0 be given. For every n ∈ N, the switching strategy σ φ is n-day φ-maxmin.
Proof Let a history h ∈ H with length t be given. Let k = κ(h) denote the number of switches that have occurred along the history h. For every τ ∈ S 2 , we obtain the chain of inequalities where the first inequality holds since σ k is a φ(h)-maxmin strategy for the subgame at history h, the second inequality holds by Lemma 5.4 since k ≤ t ≤ t +n, the third inequality holds by Lemma 3.3, and the fourth one follows since U t+n σ t+n ≤ V t+n . The final equality follows from the fact that the stochastic variable V t+n is F t+n -measurable and the fact that the strategy σ t+n coincides with σ φ at least until time t + n.
The next lemma says that along almost any play p ∈ P only finitely many switches occur or the tolerance level goes to zero. This lemma is crucial to show the equalizing property of switching strategies.

Lemma 5.6
For every history h ∈ H, for every strategy τ ∈ S 2 of player 2, it holds that Proof Although it is possible that player 1 switches infinitely many times along a play p, and therefore incurs infinitely many increases in his guarantee level along this play, we show first that the total overall increase in his guarantee level is bounded, since the payoff function itself is a bounded function. More formally, let t ∈ N, a history h ∈ H t of length t, and a strategy τ ∈ S 2 of player 2 be given. Let κ(h) denote the number of switches that have occurred along the history h. We show that where we recall that M = sup p∈P |u( p)|.
As in the proof of Lemma 5.4, letū be any F ∞ -measurable stochastic variable with the property that, for every k ∈ N,ū = u, P h,σ k ,τ -almost surely. For every k ∈ N, we define U ∞ σ k =ū. Let some k > κ(h) be given. For every p ∈ P(h), it holds that t ≤ T k ( p) ≤ T k+1 ( p), so Lemma 3.3 and the optional sampling theorem with unbounded stopping times as presented in Theorem 8.16 of Yeh ([22], p. 139) implies where the two equalities follow from Lemma 5.2 and the fact that both U T k σ k and U T k+1 σ k are F T k+1 -measurable stochastic variables, and the inequality follows by taking the expectation on both sides of inequality (5.8) and the law of iterated expectation.
Using Lemma 5.3, we can conclude that Summing the preceding inequality over k = κ(h) + 1, . . . , K , we obtain The result follows by taking the limit as K → ∞.
The latter expression is finite by (5.7). Thus ∞ k=k X k has a finite mean with respect to the probability measure P h,σ φ ,τ . Hence ∞ k=k X k < ∞ holds P h,σ φ ,τ -almost surely. This implies that X k → 0 holds P h,σ φ ,τ -almost surely.
We are now in a position to state the main result of this section, which gives sufficient conditions for the existence of subgame φ-maxmin strategies.
Theorem 5.7 Let a game x 0 (u) and a tolerance function φ > 0 be given such that for every p ∈ P at least one of the following two conditions holds: Then, there exists a subgame φ-maxmin strategy in the game x 0 (u).
Proof We define the tolerance function φ by Thus, φ is a non-increasing tolerance function with φ ≤ 1 2 φ. We show that σ φ is φ -equalizing. Since 2φ ≤ φ and σ φ is an n-day φ -maxmin strategy for every n ∈ N by Lemma 5.5, it then follows from Theorem 4.5 that σ φ is a subgame φ-maxmin strategy.
Let U denote the set of plays p ∈ P at which u is upper semi-continuous and let I denote the set of plays p ∈ P such that lim inf t→∞ φ( p |t ) > 0. By the assumption of the theorem it holds that P = U ∪ I. By the definition of φ , we have lim inf t→∞ φ ( p |t ) > 0 for each p ∈ I.
Let some t ∈ N, a history h ∈ H t , and a strategy τ ∈ S 2 be given.
where we remind the reader that R k contains the plays along which at least k switches occur.
Let k ∈ N and p ∈ J k be given. Exactly k switches occur along the play p and the last switch occurs at time T k ( p). By our construction of σ φ , this means that for each time n > T k ( p), the strategy σ k is a φ ( p |n )-maxmin strategy for the subgame at history p |n , so . Taking the limit as n goes to infinity, and making use of the fact that p ∈ J k , we obtain Step 3: P h,σ φ ,τ (I ∩ R ∞ ) = 0.
Recall that by Lemma 5.6, T k · I (T k < ∞) converges to 0 as k goes to infinity, P h,σ φ ,τalmost surely. Also recall that R ∞ is the set of plays where infinitely many switches occur. Thus I (T k < ∞) is identically equal to 1 on R ∞ . Furthermore, lim inf k→∞ T k > 0 everywhere on I. We conclude that T k · I (T k < ∞) does not converge to zero on I ∩ R ∞ and the result follows.

Using Lemma 5.2 twice yields
We now have where the last equality follows from Step 3. Finally, we obtain where the last equality follows from the fact that the sets U and I cover P(h).
Theorem 5.7 generalizes Proposition 11 in Mashiah-Yaakovi ( [15]), where the existence of a subgame -optimal strategy in a two-player zero-sum stochastic game with Borel measurable payoff functions is shown. The tolerance function φ there is given by φ(h) = for every h ∈ H, so for every play the tolerance function has a positive limit inferior. Theorem 5.7 yields the existence of a subgame -maxmin strategy. Because the Borel measurability of the payoff function guarantees the existence of the value (Maitra and Sudderth,[13], and Martin, [14]), this is equivalent to proving the existence of a subgame -optimal strategy.
In Sect. 6, we argue that Theorem 5.7 also provides further insight into the main result of Laraki, Maitra, and Sudderth ( [10]). There the authors prove among other things that if the payoff function is bounded and upper semi-continuous, then the first player has a subgame optimal strategy.

Subgame maxmin strategies
The goal of this section is to explore the relationship between the concept of a subgame maxmin strategy and that of a subgame φ-maxmin strategy for φ > 0. The main result of this section is the following theorem. Theorem 6.1 For every game x 0 (u), there exists a tolerance function φ * > 0 such that the following statements are equivalent: 1. The game x 0 (u) has a subgame maxmin strategy. 2. The game x 0 (u) has a subgame φ * -maxmin strategy.
Because every subgame maxmin strategy is a subgame φ-maxmin strategy, statement 1 clearly implies statement 2. To prove the converse, we define a tolerance function φ * > 0; the precise definition is given in Subsection 6.3. Intuitively, we fix a positive sequence (δ t ) t∈N with the property that ∞ t=0 δ t < ∞. Then, we choose φ * such that (1) φ * ( p |t ) < δ t for very play p ∈ P and every t ∈ N and (2) every one-day φ * -maxmin action is δ t -close to the region of optimal mixed actions in the one-day game. The first condition implies that tolerance function φ * decreases rapidly to zero, i.e., ∞ t=0 φ * ( p |t ) < ∞. By assumption, player 1 has a subgame φ * -maxmin strategy. We obtain a subgame maxmin strategy as follows: at every history h t we pick a mixed action which guarantees the lower value v(h t ) in the one-day game and is δ t -close to the mixed action chosen by the subgame φ * -maxmin strategy. This method creates a strategy which is clearly one-day maxmin. The condition that the tolerance function φ * fastly decreases to 0 is then used to show that expected payoffs from a history h t with length t obtained by the subgame φ * -maxmin strategy and the constructed strategy are within a distance of M · ∞ n=t δ n , where M = sup p∈P |u( p)|. From this, the equalizing property of the constructed strategy follows. Then, Corollary 4.7 from Sect. 4 lets us conclude that the constructed strategy is indeed a subgame maxmin strategy. Theorem 6.1 has the following corollary.
We would like to make two remarks. First, as the payoff function u need not be continuous, one cannot simply use a continuity argument to prove that statement 2 of Corollary 6.2 implies statement 1. Second, note that the existence of a subgame -maxmin strategy for every > 0 is a weaker requirement than statements 1 and 2 of Corollary 6.2. Indeed, as Example 3.2 already illustrated, there exist games which admit a subgame -maxmin strategy for every > 0 but do not admit a subgame maxmin strategy. The special case where the payoff-function is upper semi-continuous deserves some additional attention. From Maitra and Sudderth ([13]) and Martin ([14]), it follows that every two-player zero-sum stochastic game with a countable state space, finite action sets, and a Borel measurable payoff function admits a value. Because every upper semi-continuous payoff function is Borel measurable, the existence of the value is guaranteed. From Theorem 5.7, we obtain the existence of a subgame φ-optimal strategy for every φ > 0. Combining this with Corollary 6.2 shows that player 1 has a subgame optimal strategy. Hereby, we obtain a special case of the result by Laraki, Maitra, and Sudderth ( [10]), where the authors allow the state space to be a Borel subset of a Polish space and transition probabilities to be determined by a Borel transition function.
When we are interested in pure strategies that guarantee the maxmin levels, we can strengthen the result of Theorem 6.1. In the context of simultaneous move games, pure strategies are of course rather restrictive. Still, there are important classes of games, such as perfect information games, in which they are natural and play a prominent role.

Theorem 6.3
For every game x 0 (u), there exists a tolerance function φ * > 0 such that the following statements are equivalent: 1. The pure strategy σ ∈ S 1 is a subgame maxmin strategy. 2. The pure strategy σ ∈ S 1 is a subgame φ * -maxmin strategy.
The tolerance function φ * in Theorem 6.3 will be defined based on the data of the game and the lower values in the subgames.
This section is structured as follows. We start by proving Theorem 6.3, which is easier than Theorem 6.1 and helps us explain the main ideas. Then, we turn to the proof of Theorem 6.1.

The proof of Theorem 6.3
In this subsection, we prove Theorem 6.3. Since the statement of this theorem is about pure strategies, the proof is less technical and the intuition is more transparent.
We only need to prove that there is φ * > 0 such that statement 2 of Theorem 6.3 implies statement 1 of Theorem 6.3. From now on, for any t ∈ N, history h ∈ H t with some final state x, and mixed actions m 1 ∈ (A) and m 2 ∈ (B), we denote the expectation of the lower value at the next stage by |h, a, b) · v(h, a, b, x ).
The proof consists of two steps.
For every t ∈ N, for every history h ∈ H t , there exists a number d(h) > 0 such that for every action a ∈ A of player 1 either one of the following holds: • Action a guarantees that the lower value does not drop in expectation: for every m 2 ∈ (B), E h,a,m 2 V t+1 ≥ v(h). • There exists a mixed action m 2 ∈ (B) for player 2 such that, if player 1 uses action a, the lower value drops in expectation by more than d(h): This statement is true because the action set A of player 1 is finite. We define φ * > 0 as follows. For every t ∈ N, for every history h ∈ H t , let φ * (h) = min{d(h), 2 −t }. The term 2 −t is included so that the tolerance levels tend to 0 along each play. For every p ∈ P, it holds that lim t→∞ φ * ( p |t ) = 0.
Step 2. If the pure strategy σ ∈ S 1 is a subgame φ * -maxmin strategy, then σ is a subgame maxmin strategy. Let the pure strategy σ ∈ S 1 be subgame φ * -maxmin. We verify that σ satisfies the conditions of Corollary 4.7.
First, we show that σ is 1-day maxmin. Fix t ∈ N and h ∈ H t . Let a = σ (h) denote the action that σ recommends at h. Then, for every mixed action m 2 ∈ (B) and strategy τ ∈ S 2 with m 2 = τ (h), we have where the first inequality follows from the fact that σ is subgame φ * -maxmin and by condition 1 of Theorem 4.6, and the second inequality follows from the definition of φ * (h). Therefore, by the choice of d(h) in Step 1, for every m 2 ∈ (B), it holds that Hence, for every strategy τ ∈ S 2 , it holds that Thus, σ is 1-day maxmin.
We show that σ is equalizing. Take some τ ∈ S 2 . Because σ is subgame φ * -maxmin, for every p ∈ P(h), for every n ≥ t, we have where * n ( p) = φ * ( p |n ). We conclude that where the equality follows from Lemma A.1 in Appendix A. We have shown that σ is equalizing.

The One-Shot Game 7 h
To prove Theorem 6.1, we first analyze a one-shot zero-sum game in this subsection. For each history, the one-shot game is such that the payoff is given by the lower value at the next stage. In the next subsection, we use these one-shot games to construct the tolerance function φ * > 0. For some t ∈ N, let h ∈ H t be a history in the game x 0 (u). The one-shot zero-sum game ϒ h is played as follows. Player 1 chooses an action a ∈ A and player 2 simultaneously chooses an action b ∈ B. Then, player 1 receives from player 2 the amount E h,a,b V t+1 . As the action sets A and B are finite, the game ϒ h has a value, which we denote by w(h). Furthermore, both players have optimal mixed actions in the game ϒ h .
The following lemma states that the value w(h) of the one-shot game ϒ h equals the lower value v(h) at the history h in the original game x 0 (u). In case the value exists, this is a classic result of dynamic programming.

Lemma 6.4 For every history h ∈ H, we have w(h) = v(h).
Proof The proof is by contradiction. Fix t ∈ N and a history h ∈ H t . Now suppose that We derive a contradiction by showing that, in the subgame of x 0 (u) at history h, player 1 can guarantee an expected payoff of at least v(h) + δ/2.
Let m 1 ∈ (A) be an optimal mixed action for player 1 in the one-shot game ϒ h . Let σ ∈ S 1 be such that σ (h) = m 1 and such that it induces a (δ/2)-maxmin strategy for the subgame at each history in period t + 1, i.e., for every h ∈ H t+1 , for every τ ∈ S 2 , Then, for every τ ∈ S 2 , it holds that where the second equality follows from the fact that (U n σ,τ ) n≥t is a P h,σ,τ -martingale, the first inequality follows from (6.2), and the second inequality follows from the choice of m 1 .
We derive a contradiction by showing that, for every strategy of player 1, there is a strategy for player 2 such that the expected payoff is at most v(h) − δ/2 in the subgame of x 0 (u) at history h.
Fix σ ∈ S 1 and let m 1 = σ (h). Let m 2 ∈ (B) be an optimal mixed action for player 2 in the one-shot game ϒ h . Let τ ∈ S 2 be such that τ (h) = m 2 and the expected payoff under (σ, τ ) in the subgame at each history h at period t + 1 is at most the lower value v(h ) + δ/2, i.e., for every h ∈ H t+1 , where the first inequality follows from the choice of m 2 , the second inequality follows from (6.3), and the penultimate equality follows from the fact that (U n σ,τ ) n≥t is a P h,σ,τ -martingale. Because σ is chosen arbitrarily, we have derived a contradiction with the definition of the lower value v(h).
The total variation distance between two mixed actions m 1 , n 1 ∈ (A) of player 1 is defined as The total variation distance between two probability measures P and P on (P, F ) is defined as Let t ∈ N and a history h ∈ H t be given. Let O h ⊆ (A) denote the set of optimal mixed actions of player 1 in the one-shot game ϒ h . By Lemma 6.4 it holds that Note that O h is a compact subset of (A). For every m 1 ∈ (A), the distance of m 1 to O h is defined by Due to the compactness of O h , the minimum is attained. For δ > 0, let D δ h be the set of mixed actions of player 1 which have a distance of at least δ to the set O h , so The mixed actions in D δ h are not optimal in the one-shot game ϒ h . The following lemma says that the loss in utility caused by these mixed actions has a positive lower bound.

Lemma 6.5 Let h ∈ H and δ > 0 be given. If D δ
h is non-empty, then there is > 0 such that for every m 1 Since B is finite, the minimum exists. For every m 1 ∈ D δ h , we have m 1 / ∈ O h , and therefore there exists m 2 ∈ (B) such that E h,m 1 ,m 2 V t+1 < v(h). The function e δ h is therefore a positive and continuous function on a compact set, so has a positive minimum.

Definition of the Tolerance Function *
In this subsection, we define a positive tolerance function φ * . Fix a positive and decreasing sequence (δ t ) t∈N such that ∞ t=0 δ t < ∞. For example, δ t = 2 −t for every t ∈ N. Note that lim t→∞ δ t = 0.
For every t ∈ N, for every history h ∈ H t , we define the constant c(h) as follows. If the set D δ t h is non-empty, then c(h) is equal to the positive number of Lemma 6.5 and c(h) = δ t otherwise. We define We summarize the properties of the tolerance function φ * : The importance of ∞ t=0 δ t < ∞ is underlined by the following lemma. Recall that M = sup p∈P |u( p)|. Lemma 6.6 Let the strategies σ, σ ∈ S 1 be such that, for every t ∈ N, for every history h ∈ H t , σ (h) − σ (h) TV ≤ δ t . Then, for every strategy τ ∈ S 2 , for every t ∈ N, and for every history h ∈ H t , Proof Let τ ∈ S 2 be given. It follows from a more general result in Abate, Redig, and Tkachev ([1], theorem 1) that, for every t ∈ N, for every history h ∈ H t , For completeness, we provide a direct proof of this inequality in Lemma B.1 in Appendix B. The claim of Lemma 6.6 follows directly. Lemma 6.6 says the following. Consider two arbitrary strategies σ, σ ∈ S 1 such that the total variation distance between the mixed actions at every history in period t is at most δ t . Now consider t ∈ N, a history h ∈ H t , and a strategy τ ∈ S 2 of player 2. Then, the expected payoffs under (σ, τ ) and (σ , τ ) in the subgame at h differ at most the constant M times ∞ n=t δ n . Note that this bound does not depend on the strategy τ and it only depends on the history h through its period t. Moreover, these bounds tend to 0 as t goes to infinity.
The importance of property 4 of the tolerance function is shown by the following lemma. It says that if σ ∈ S 1 is a subgame φ * -maxmin strategy, then for every h ∈ H the mixed action σ (h) is close to the set of optimal mixed actions O h in the one-shot game ϒ h . Lemma 6.7 Let σ ∈ S 1 be a subgame φ * -maxmin strategy. Then, for every t ∈ N, for every Proof Because σ is a subgame φ * -maxmin strategy, it follows from condition 1 of Theorem 4.6 that for every mixed action m 2 ∈ (B) of player 2 If D δ t h is empty, then there is nothing to prove. If D δ t h is non-empty, then property 4 of φ * shows that σ (h) / ∈ D δ t h .

The proof of Theorem 6.1
Proof Let x 0 (u) be a game and take the tolerance function φ * as defined in Subsection 6.3. We only need to show that statement 2 implies statement 1. Let σ ∈ S 1 be a subgame φ * -maxmin strategy of x 0 (u).
With the help of σ, we define a strategy σ * ∈ S 1 in Step 1 of the proof. Then, it is shown that σ * is a subgame maxmin strategy of x 0 (u) in Steps 2 and 3 of the proof by verifying that σ * satisfies the conditions of Corollary 4.7.
Step 1: Definition of σ * ∈ S 1 . Take t ∈ N and a history h ∈ H t . In view of Lemma 6.7, it holds that Step 2: σ * is 1-day maxmin.
Step 3: σ * is equalizing. Consider some τ ∈ S 2 . In view of (6.8), we can apply Lemma 6.6 to conclude that, for every t ∈ N, for every h ∈ H t , Hence, for every t ∈ N, for every history h ∈ H t , and for every n ≥ t, Because σ is subgame φ * -maxmin, for every history h ∈ H, we have Thus, for every history h ∈ H it holds that where the first equality is due to Lemma A.1 in Appendix A, the second equality follows from (6.9), the inequality is by (6.10), and the last equality is a consequence of property 3 of φ * from Subsection 6.3.
Note that it is not necessary that the initial δ t are small, or that for initial histories the tolerance level φ * (h) is small. The conditions put on the sequence (δ t ) t∈N are there to ensure that the equalizing condition for subgame maxmin strategies is satisfied. The equalizing condition essentially only cares about what happens to the payoff at deep subgames. The requirement that the sequence (δ t ) t∈N decreases to zero very fast guarantees that lim t→∞ P p |t ,σ,τ − P p |t ,σ,τ TV = 0. This allows us to use the φ * -equalizing property of the subgame φ * -maxmin strategy to show that the strategy σ * satisfies the equalizing condition.

Discussion
Countable action sets. We have assumed throughout the paper that the action sets are finite. If one player has countably many actions, several of our arguments and results are no longer valid. For example, consider the following game. In the initial state, player 1's action set is N, player 2 has only one action, and if player 1 chooses action n ∈ N then regardless how the play continues, the payoff is 1 − 1 n+1 . This is essentially a one-shot game. Clearly, for every tolerance function φ > 0 player 1 has a pure subgame φ-maxmin strategy, i.e. he can play action n with n sufficiently large. Yet, player 1 has no subgame maxmin strategy. So the statements of Theorems 6.1 and 6.3 and Corollary 6.2 are not valid for this game. One-player games. Suppose that player 2 has a trivial action set, i.e. B is a singleton. Even though this is essentially a game with only one player, as shown in Example 3.2, it may happen that player 1 has a subgame φ-maxmin strategy for a certain tolerance function φ, but he has no pure strategy that is subgame φ-maxmin. Nevertheless, if B is a singleton and the conditions of Theorem 5.7 are satisfied, then player 1 has a pure subgame φ-maxmin strategy. Indeed, player 1 has a pure δ-maxmin strategy for each δ > 0. Hence, the switching strategy used in the proof of Theorem 5.7 will also be pure. Discounted games. Our model encompasses discounted stochastic games with finite action and countable state spaces, provided that the immediate payoffs are bounded. Since these games admit a value and the discounted payoffs are continuous, it follows by Theorem 5.7 that player 1 has a subgame φ-optimal strategy for every tolerance function φ > 0. Therefore, Corollary 6.2 implies the well-known fact that player 1 has a subgame 0-optimal strategy. Our proof does not directly imply that this strategy can be chosen to be stationary.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Appendix A: Universal Measurability
This appendix contains a review of the definitions of universally measurable sets, integrals of universally measurable functions, stopping times, and conditional expectations, as well as a technical lemma that is used in the paper. Universally measurable sets Let (P, F ∞ ) be a measurable space, where F ∞ denotes the Borel sigma-algebra and let M denote the collection of all probability measures over this measurable space. For each probability measure P ∈ M, we can extend the probability space (P, F ∞ , P) to a complete probability space (P, F P , P c ) by including all P-negligible sets.
To be more precise, let F 0 P = {Q ⊂ P | ∃Q ∈ F ∞ such that P(Q ) = 0 and Q ⊆ Q } be the set of all subsets of P-negligible sets of F ∞ . We define and we define P c : F P → [0, 1] by It can be shown that (P, F P , P c ) is a probability space. Now define It can be shown that F is a sigma-algebra that contains the Borel sigma-algebra F ∞ as a proper subset. The collection F is the universally measurable sigma-algebra and the elements of F are called universally measurable sets.

Integrals of universally measurable functions
. . , Z n } is a partition of P with Z i ∈ F for all i = 1, . . . , n. The expected value of a simple universally measurable payoff with respect to a probability measure P is defined as Let G denote the set of simple universally measurable functions. The expected value of a bounded universally measurable function u : P → R is then given by p∈P u( p)P(dp) = sup g∈G g≤u p∈P g( p)P(dp) = inf g∈G g≥u p∈P g( p)P(dp).
Stopping times A stopping time is a function T : P → N ∪ {∞} such that for each t ∈ N the set { p ∈ P | T ( p) = t} is an element of F t . Given a stopping time T , let F T denote the sigma-algebra of sets A ∈ F ∞ such that A ∩ {p ∈ P | T ( p) = t} ∈ F t for each t ∈ N. For every t ∈ N, let X t be an F t measurable stochastic variable, and let X ∞ be an F ∞ measurable stochastic variable. The stochastic variable X T is defined by letting it coincide with X t on { p ∈ P | T ( p) = t} for each t ∈ N and with X ∞ on { p ∈ P | T ( p) = ∞}. Following Yeh ([22], Theorem 3.28), it holds that X T is F T measurable. Conditional expectations Consider a bounded stochastic variable F : P → R, a strategy profile (σ, τ ) ∈ S 1 × S 2 , and a history h ∈ H of length . Let some t ≥ be given. The conditional expectation of F with respect to the sigma-algebra F t and the measurable space P h,σ,τ -almost surely. In the first equality, we use the definition of the stochastic variable U t σ,τ . The second equality follows from the fact that u =ū P h,σ,τ -almost surely. The third equality follows from Levy's zero-one law (see e.g., Bogachev, [3], Example 10.3.15). The last equality follows from the fact thatū is F ∞ measurable. This is becauseū is Borel measurable and F ∞ is the Borel sigma-algebra on P. The fact that u =ū P h,σ,τ -almost surely concludes the proof.

Appendix B: The Proof of Inequality (6.7)
The following statement follows from the more general result of Theorem 1 in Abate, Redig, and Tkachev ( [1]). For completeness, we provide a direct proof in this section.
Lemma B.1 Let σ, σ ∈ S 1 be such that, for every t ∈ N, for every h ∈ H t , σ (h) − σ (h) TV ≤ δ t . Then, for every strategy τ ∈ S 2 , for every t ∈ N, and for every history h ∈ H t , The class V ⊆ F is called an inner (outer) approximating class for the class W ⊆ F if V is closed under unions (intersections) and if for every > 0, and for every probability measure P ∈ M and every set W ∈ W there exists a set V ∈ V such that V ⊆ W (V ⊇ W ) and |P(W ) − P(V )| ≤ . In the following lemma, we use Lemma B.2 to simplify the computation of the total variation norm. Instead of having to compute the supremum over all sets of the universally measurable sigma-algebra, it will be sufficient to compute the supremum over the subclass of open sets O t , t ∈ N, defined by Because of the outer regularity of Borel measures on metric spaces, we have that the class of open sets is an outer approximating class for the class of Borel sets. In addition, we have that the class of Borel sets is an inner approximating class for the class of universally measurable sets. Indeed, for any probability measure and any universally measurable set, there exists a Borel set which is contained in the universally measurable set and has the same probability. Repeated application of Lemma B.2 concludes the proof.

Lemma B.2 If
Proof of Lemma B.1 Fix t ∈ N and a history h t ∈ H t . We prove by induction that for every n ≥ t, for every O ∈ O n+1 ,

Induction basis (n = t).
We prove that for every O ∈ O t+1 , P h t ,σ,τ (O) − P h t ,σ ,τ (O) ≤ δ t . Fix a set O ∈ O t+1 . We define Let A h t = {a ∈ A|∃(b, x) ∈ B × X : (a, b, x) ∈ Z h t } be the projection of the set Z h t on the set A. Let x t denote the state at the history h t . We have that We find that