Faster algorithms for extensive-form game solving via improved smoothing functions

Abstract

Sparse iterative methods, in particular first-order methods, are known to be among the most effective in solving large-scale two-player zero-sum extensive-form games. The convergence rates of these methods depend heavily on the properties of the distance-generating function that they are based on. We investigate both the theoretical and practical performance improvement of first-order methods (FOMs) for solving extensive-form games through better design of the dilated entropy function—a class of distance-generating functions related to the domains associated with the extensive-form games. By introducing a new weighting scheme for the dilated entropy function, we develop the first distance-generating function for the strategy spaces of sequential games that has only a logarithmic dependence on the branching factor of the player. This result improves the overall convergence rate of several FOMs working with dilated entropy function by a factor of $\Omega (b^dd)$, where b is the branching factor of the player, and d is the depth of the game tree. Thus far, counterfactual regret minimization methods have been faster in practice, and more popular, than FOMs despite their theoretically inferior convergence rates. Using our new weighting scheme and a practical parameter tuning procedure we show that, for the first time, the excessive gap technique, a classical FOM, can be made faster than the counterfactual regret minimization algorithm in practice for large games, and that the aggressive stepsize scheme of CFR+ is the only reason that the algorithm is faster in practice.

A Notation and results described from an EFG perspective

In the body of the paper we described our results using notation oriented toward the convex-optimization perspective on treeplexes: the results are for general treeplexes, and so we view them as a general convex set. In this section we give an overview of our results from a standard EFG-specific perspective, in the hopes that it may be helpful for researchers who are familiar with that literature, but not the first-order methods literature and convex analysis.

First we define notation, then we give a description of how our treeplex results map onto traditional EFG notation, and finally we give a description of the dilated entropy smoothed best response tree traversal and EGT in terms of this notation.

1.1 A.1 Extensive-form games and the sequence-form

Extensive-form games (EFGs) can be thought of as a game tree, where each node in the tree corresponds to some history of actions taken by all players. Each node belongs to some player, and the actions available to the player at a given node are represented by the branches. Uncertainty is modeled by having a special player, Nature, that moves with some predefined fixed probability distribution over actions at each node belonging to Nature. EFGs model imperfect information by having groups of nodes in information sets, which is a group of nodes all belonging to the same player such that the player cannot distinguish among them. Finally we assume perfect recall, which requires that no player ever forgets their past actions (equivalently, for each information set there is only a single possible last action taken by the player to whom the information set belongs).

Definition 4

A two-player extensive-form game with imperfect information and perfect recall$\Gamma$ is a tuple $(H, Z, A, P, \sigma _c, \mathscr {I}, u)$ composed of:

• H: a finite set of possible sequences (or histories) of actions, such that the empty sequence $\emptyset \in H$, and every prefix z of h in H is also in H.

• $Z\subseteq H$: the set of terminal histories, i.e. those sequences that are not a proper prefix of any sequence.

• A: a function mapping $h \in H{\setminus } Z$ to the set of available actions at non-terminal history h.

• P: the player function, mapping each non-terminal history $h \in H{\setminus } Z$ to $\{1, 2, c\}$, representing the player who takes action after h. If $P(h)=c$, the player is Nature.

• $\sigma _0$: a function assigning to each $h\in H{\setminus } Z$ such that $P(h) = 0$ a probability mass function over A(h).

• $\mathscr {I}_i$, for $i\in \{1,1\}$: partition of $\{h\in H: P(h)=i\}$ with the property that $A(h)=A(h')$ for each $h,h'$ in the same set of the partition. For notational convenience, we will write A(I) to mean A(h) for any of the $h\in I$, where $I\in \mathscr {I}_i$. $\mathcal{I}_i$ is the information partition of player i, while the sets in $\mathcal{I}_i$ are called the information sets of player i.

• $u_i$: utility function mapping $z\in Z$ to the utility (a real number) gained by player i when the terminal history is reached.

We further assume that all players have perfect recall.

A strategy for a player i is usually represented in behavioral form, which consists of probability distributions over actions at each information set in $\mathscr {I}_i$. In this paper we will focus on an alternative, but strategically equivalent, representation of the set of strategies, called the sequence form [17, 32, 37]. In the sequence form, actions are instead represented by sequences. A sequence $\sigma _i$, is an ordered list of actions taken by player i on the path to some history h. In perfect-recall games, all nodes in an information set $I\in \mathscr {I}_i$ correspond to the same sequence for player i, we let ${\mathrm{seq}}(I)$ denote this sequence. Given a sequence $\sigma _i$ and an action a that Player i can take immediately after $\sigma _i$, we let $\sigma _ia$ denote the resulting new sequence. Instead of directly choosing the probability to put on an action, in the sequence form the probability of playing the entire sequence is chosen, this is called the realization probability and is denoted by $r_(\sigma _i)$. A choice of realization probabilities for every sequence belonging to Player i is called a realization plan and is denoted $r_i:\Sigma _i \rightarrow [0,1]^{|\Sigma _i|}$. This representation relies on perfect recall: for any information set $I\in {\mathscr {I}}_i$ we have that each action $a\in A(I)$ is uniquely represented by a single sequence $\sigma _i = {\mathrm{seq}}(I)a$, since ${\mathrm{seq}}(I)$ corresponds to exactly one sequence. In particular, this gives us a simple way to convert any strategy in sequence form to a behavioral strategy: the probability of playing action $a\in A(I)$ at information set I is simply $\frac{r_i({\mathrm{seq}}(I)a)}{r_i({\mathrm{seq}}(I))}$.

1.2 A.2 Mapping between convex analysis notation and EFG notation

First we describe the treeplex. A treeplex Q is used to model the sequence-form strategy space of each player. Thus in an EFG, the treeplexes would be the sets of realization probabilities $\Sigma _1,\Sigma _2$. In the BSPP (1) the typical representation would be that $\mathcal{X}=\Sigma _1$ and $\mathcal{Y}= \Sigma _2$. For the remainder we will describe the notation in terms of Player 1 and $\Sigma _1$. The set of simplexes in $\Sigma _1$, with indices denoted by $S_{\Sigma _1}$, is the set of information sets $\mathscr {I}_1$ where Player 1 acts. The set $\mathcal{D}_j^i$ of simplexes reached immediately after taking branch i in simplex j is the set of potential information sets where Player 1 may have to act next. Which one is reached of course depends on which actions are taken by Nature and Player 2. Another way to put this is that $\mathcal{D}_j^i$ corresponds to the set of information sets $I\in \mathscr {I}_1$ such that $\sigma _1(I) = \sigma _1$, where $\sigma _1$ is the sequence corresponding to taking the action i at simplex j. The table below gives an overview of treeplex notation and how it corresponds to EFG notation and concepts. Not all our concepts are easily mapped to existing EFG ideas. For example $M_Q$ and $M_{Q,r}$, the maximum $\ell _1$ norm of Q and the r-depth-limited maximum $\ell _1$ norm, are still most easily thought of in terms of norms. It is the maximum number of information sets with nonzero probability of being reached when player 1 has to follow a pure strategy while the other player may follow a mixed strategy. Intuitively the maximum $\ell _1$ norm of $\Sigma _1$ measures the branching factor associated with observable opponent actions and Nature actions that cause Player 1 to reach different information sets, while not measuring branching factor associated with Player 1 choosing actions at an information set (since the $\ell _1$ norm sums to one at such information set).

Treeplex notation	EFG meaning
Q	$\Sigma _1$, the set of realization probabilities
$S_{\Sigma _1}$	$\mathscr {I}_1$, the set of information-set indices into the treeplex $\Sigma _1$
$\mathcal{D}_j^i$	Set of information sets in $\Sigma _1$ such that the sequence corresponding to branch i at simplex j is the parent sequence
$q_{p_j}$	The parent sequence $\sigma _1(I_j)$, where $I_j$ is the information set
	corresponding to simplex j
$d_j$	The length of the longest possible sequence of actions starting at $I_j$
$b_{\Sigma _1}^j$	The length of $\sigma _1(I_j)$

Our dilated entropy construction using the weights described in recurrence (6) can now be described in terms of EFG notation as follows:

$$\begin{aligned} \begin{array}{ll} \alpha _j = 1 + \max _{a\in A_{I_j}}\sum _{k \in \mathcal{D}^a_{I_j}} \frac{\alpha _k\beta _k}{\beta _k - \alpha _k},&{}\quad \forall I_j \in \mathscr {I}_1,\\ \beta _j> \alpha _j,&{}\quad \forall I_j \in \mathscr {I}_1~\text {s.t.}~ length(\sigma _1(I_j)) > 0,\\ \beta _j = \alpha _j,&{}\quad \forall I_j \in \mathscr {I}_1~\text {s.t.}~ length(\sigma _1(I_j)) = 0. \end{array} \end{aligned}$$

(18)

If we then instantiate EGT with this DGF and use Theorem 3 we get the following convergence rate:

$$\begin{aligned} \frac{\max _{\sigma _1 \in \Sigma _1, \sigma _2 \in \Sigma _2}|g_1(\sigma _1,\sigma _2)|\, \sqrt{M_{\Sigma _1}^22^{d_{\Sigma _1}+2}M_{\Sigma _2}^22^{d_{\Sigma _2}+2}}\, \max _{I\in \mathscr {I}}\mathop {\mathrm{log}}|A_I|}{\varepsilon }. \end{aligned}$$

1.3 A.3 EGT described as a tree traversal

Here we explain how to implement the $\mathop {\mathrm{Prox}}$ operation when using the dilated entropy function, as well as how to compute smoothed best responses, i.e. $x_{\mu _1}(y)$ or $y_{\mu _2}(x)$ in Algorithm 2. Throughout we will present algorithms for computing everything from the perspective of a player trying to minimize their opponent’s utility, rather than maximize their own.

First, given $y^t$, the gradient for Player 1 is $Ay^t$ where A is the payoff matrix for Player 2. This can be implemented as follows: create an all-zero vector g of dimension $|\Sigma _1|$. Traverse the game tree, and for each leaf z add $\pi _0(z) y^t[\sigma _2(z)] u_2(z)$ at the entry in g corresponding to $\sigma _1(z)$.

Pseudocode for computing a smoothed best response is given in Algorithm 3. This gives an algorithm for using the dilated entropy function with the negative entropy plus a constant term at each simplex $\Delta _n$: $\sum _{x}x\mathop {\mathrm{log}}(x) + \mathop {\mathrm{log}}(n)$. By adding $\mathop {\mathrm{log}}(n)$ we ensure that the function is never negative; it is zero at $x_i = \frac{1}{n}$ for all i. Since the constant does not change the second-order derivatives of the dilated entropy function we retain the same strong convexity properties.

The smoothed best response implementation given here modifies the gradient g in place. Thus it is important that g is not used to represent the gradient after a call to the function. However, the modified g can be useful because the entry in g corresponding to the empty sequence then contains the value of the smoothed best response function, which is needed for verifying e.g. the excessive gap condition.

The algorithm for smoothed best response calculation takes as input a vector g which is usually gradient at the current iteration. This vector g is of length equal to the number of sequences for the player. We assume that the sequences are ordered so that $g[I_{start},I_{end}]$ denotes the subset of g corresponding to entries for all the sequences that have their last action taken at I. Note that in setting the value of of the indices in x corresponding to I, $x[I_{start},I_{end}]$, we assume that $\exp$ is an index-wise exponential operator and offset is subtracted from each entry.

Given our implementation for smoothed best responses above, the computation of the proximal operator $\mathop {\mathrm{Prox}}_{x}(g)$ can be performed easily: it is simply a smoothed best response where we shift the gradient g by $\nabla \omega (x)$, where x is the point that we use as the prox center. The following algorithm shifts g in place:

Note that this implementation of the prox mapping does not give the actual value of the objective, only the strategy vector that minimizes the objective. For smoothed best response the objective could be read off the entry of the modified g at the empty sequence, but it does not hold the correct value for the prox mapping. Unlike for smoothed best response, none of our EGT variants rely on the prox objective.

Once these primitives have been implemented, the high-level steps of Algorithms 1 and 2 are easy to implement. First $\mu _1$ and $\mu _2$ are set to appropriate initial values (for example via the theory or the $\mu$-fitting approach that we use), and initial sequence-form strategies $x^0,y^0$ are computed for Players 1 and 2 using the above procedures. Then, we just take repeated alternating steps for Players 1 and 2, where the stepsize can either be set to $\frac{2}{t+3}$, or chosen aggressively via heuristics. The most powerful stepsize heuristic is checking whether the excessive gap condition $\bar{\phi }_{\mu _2}(x^t) \le \bar{\phi }_{\mu _1}(y^t)$ is maintained after every iteration, and then decreasing $\tau$ and redoing the most recent step when that condition fails. Given an implementation of smoothed best response, the excessive gap value can be computed as the sum of the smoothed best response values (this works because smoothed best response was implemented to give the value for each player when they are trying to minimize their opponent’s utility). The Step algorithm with these primitives is shown below (assuming that A is the sequence-form payoff matrix for Player 2 and $\beta _1,\beta _2$ are the information-set weights in the dilated entropy for the players):

Finally the EGT algorithm is straightforward as it just iterates calls to StepEFG. The initial points can be computed via SmoothedBR and ProxCenterGradient just as in StepEFG.