Robustness and Sample Complexity of Model-Based MARL for General-Sum Markov Games

Abstract

Multi-agent reinforcement learning (MARL) is often modeled using the framework of Markov games (also called stochastic games or dynamic games). Most of the existing literature on MARL concentrates on zero-sum Markov games but is not applicable to general-sum Markov games. It is known that the best response dynamics in general-sum Markov games are not a contraction. Therefore, different equilibria in general-sum Markov games can have different values. Moreover, the Q-function is not sufficient to completely characterize the equilibrium. Given these challenges, model-based learning is an attractive approach for MARL in general-sum Markov games. In this paper, we investigate the fundamental question of sample complexity for model-based MARL algorithms in general-sum Markov games. We show two results. We first use Hoeffding inequality-based bounds to show that $\tilde{{\mathcal {O}}}( (1-\gamma )^{-4} \alpha ^{-2})$ samples per state–action pair are sufficient to obtain a $\alpha$-approximate Markov perfect equilibrium with high probability, where $\gamma$ is the discount factor, and the $\tilde{{\mathcal {O}}}(\cdot )$ notation hides logarithmic terms. We then use Bernstein inequality-based bounds to show that $\tilde{{\mathcal {O}}}( (1-\gamma )^{-1} \alpha ^{-2} )$ samples are sufficient. To obtain these results, we study the robustness of Markov perfect equilibrium to model approximations. We show that the Markov perfect equilibrium of an approximate (or perturbed) game is always an approximate Markov perfect equilibrium of the original game and provide explicit bounds on the approximation error. We illustrate the results via a numerical example.

Appendices

Appendix

Proof of Theorem 4

Let $V_*$ denote the optimal value function for MDP ${\mathcal {M}}$ and $V_{\pi }$ denote the value function for policy $\pi$ in MDP ${\mathcal {M}}$. Let ${\hat{\pi }}_*$ be the optimal policy for $\widehat{{\mathcal {M}}}$ and ${\hat{\pi }}$ be an $\alpha _{\textrm{opt}}$-optimal policy for $\widehat{{\mathcal {M}}}$. From triangle inequality, we have

$$\begin{aligned} \Vert V_* - V_{{\hat{\pi }}} \Vert _{\infty } \le \Vert V_* - {\hat{V}}_{{\hat{\pi }}_*} \Vert _{\infty } + \Vert {\hat{V}}_{{\hat{\pi }}_*} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } + \Vert V_{{\hat{\pi }}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } . \end{aligned}$$

(42)

Now we bound the three terms separately. For the first term, we have

$$\begin{aligned} \Vert V_* - {\hat{V}}_{{\hat{\pi }}_*} \Vert _{\infty }&{\mathop {\le }\limits ^{(a)}}\max _{s\in \mathcal {S}} \biggl | \max _{a\in \mathcal {A}} \biggl [ (1-\gamma )r(s, a) + \gamma \sum _{s' \in \mathcal {S}} {\textsf{P}}(s' | s, a) V_*(s') \\&\qquad \qquad - (1-\gamma ){\hat{r}}(s, a) - \gamma \sum _{s' \in \mathcal {S}} \widehat{{\textsf{P}}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}_*}(s') \biggr ] \biggr | \\&{\mathop {\le }\limits ^{(b)}}(1-\gamma ) \max _{(s,a) \in \mathcal {S}\times \mathcal {A}} \bigl | r(s, a) - {\hat{r}}(s, a) \bigr | \\&\quad + \gamma \max _{(s,a) \in \mathcal {S}\times \mathcal {A}} \biggl | \sum _{s' \in \mathcal {S}} {\textsf{P}}(s' | s, a) V_*(s') - {\textsf{P}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}_*}(s') \biggr | \\&\quad + \gamma \max _{(s,a) \in \mathcal {S}\times \mathcal {A}} \biggl | \sum _{s' \in \mathcal {S}} {\textsf{P}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}_*}(s') - \widehat{{\textsf{P}}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}_*}(s') \biggr | \\&{\mathop {\le }\limits ^{(c)}}(1 - \gamma ) \varepsilon + \gamma \Vert V_* - {\hat{V}}_{{\hat{\pi }}_*}\Vert _{\infty } + \gamma \varDelta _{{\hat{\pi }}_*}, \end{aligned}$$

where (a) relies on the fact that $\max f(x) \le \max | f(x) - g(x) | + \max g(x)$, (b) follows from triangle inequality, and (c) follows from the definition of $\varepsilon$ and $\varDelta _{{\hat{\pi }}}$. Therefore,

$$\begin{aligned} \Vert V_* - {\hat{V}}_{{\hat{\pi }}_*} \Vert _{\infty } \le \varepsilon + \frac{\gamma \varDelta _{{\hat{\pi }}_*}}{1-\gamma }. \end{aligned}$$

(43)

For the second term of (42), we have

$$\begin{aligned} \Vert {\hat{V}}_{{\hat{\pi }}_{*}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } \le \alpha _{\textrm{opt}}, \end{aligned}$$

(44)

since ${\hat{\pi }}$ is an $\alpha _{\textrm{opt}}$-optimal policy of $\widehat{{\mathcal {M}}}$.

For the third term of (42), we have

$$\begin{aligned} \Vert V_{{\hat{\pi }}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty }&= \max _{s\in \mathcal {S}} \biggl | \sum _{a\in \mathcal {A}} {\hat{\pi }}(a| s) \biggl [ (1-\gamma )r(s, a) + \gamma \sum _{s' \in \mathcal {S}} {\textsf{P}}(s' | s, a) V_{{\hat{\pi }}}(s') \\&\qquad \qquad - (1-\gamma ) {\hat{r}}(s,a) - \gamma \sum _{s' \in \mathcal {S}} \widehat{{\textsf{P}}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}}(s') \biggr ] \biggl | \\&{\mathop {\le }\limits ^{(d)}}(1-\gamma )\max _{s\in \mathcal {S}} \biggl | \sum _{a\in \mathcal {A}} {\hat{\pi }}(a| s) \bigl [ r(s,a) - {\hat{r}}(s, a) \bigr ] \biggr | \\&\quad + \gamma \max _{s\in \mathcal {S}} \biggl | \sum _{a\in \mathcal {A}} {\hat{\pi }}(a| s) \biggl [ \sum _{s' \in \mathcal {S}} \bigl [ {\textsf{P}}(s' | s, a) V_{{\hat{\pi }}}(s') - {\textsf{P}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}}(s') \bigr ] \biggr ] \biggr | \\&\quad + \gamma \max _{s\in \mathcal {S}} \biggl | \sum _{a\in \mathcal {A}} {\hat{\pi }}(a| s) \biggl [ \sum _{s' \in \mathcal {S}} \bigl [ {\textsf{P}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}}(s') - \widehat{{\textsf{P}}}(s' | s, a) {\hat{V}}_{{\hat{\pi }}}(s') \bigr ] \biggr ] \biggr | \\&{\mathop {\le }\limits ^{(e)}}(1-\gamma )\varepsilon + \gamma \Vert V_{{\hat{\pi }}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } + \gamma \varDelta _{{\hat{\pi }}} \end{aligned}$$

where (d) follows from triangle inequality and (e) follows from the definition of $\varepsilon$ and $\varDelta _{{\hat{\pi }}}$. Therefore,

$$\begin{aligned} \Vert V_{{\hat{\pi }}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } \le \varepsilon + \frac{\gamma \varDelta _{{\hat{\pi }}}}{1-\gamma }. \end{aligned}$$

(45)

The result of Theorem 4 then follows by substituting (43)–(45) in (42). $\square$

Proof of Theorem 5

We follow the same notation as in Appendix A. In addition, let $\mathcal {B}_*$ and $\mathcal {B}_\pi$ denote the Bellman operators for the true model and let ${\hat{\mathcal {B}}}_*$ and ${{\hat{\mathcal {B}}}}_\pi$ denote the Bellman operators for the approximate model. As in Appendix A, the approximation error can be bounded as

$$\begin{aligned} \Vert V_* - V_{{\hat{\pi }}} \Vert _{\infty } \le \Vert V_* - {\hat{V}}_{{\hat{\pi }}_*} \Vert _{\infty } + \Vert {\hat{V}}_{{\hat{\pi }}_*} - {\hat{V}}_{{\hat{\pi }}}\Vert _{\infty } + \Vert V_{{\hat{\pi }}} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty }. \end{aligned}$$

(46)

Recall that $V_{{\hat{\pi }}} = \mathcal {B}_{{\hat{\pi }}} V_{{\hat{\pi }}}$ and ${\hat{V}}_{{\hat{\pi }}} = {\hat{\mathcal {B}}}_{{\hat{\pi }}} {\hat{V}}_{{\hat{\pi }}}$. Moreover, ${\hat{V}}_* = {\hat{V}}_{{\hat{\pi }}_*}$. Therefore, from triangle inequality, we have

$$\begin{aligned} \Vert V_* - V_{{\hat{\pi }}} \Vert _{\infty }&\le \Vert V_* - {\hat{V}}_{*} \Vert _{\infty } + \Vert {\hat{V}}_{{\hat{\pi }}_*} - {\hat{V}}_{{\hat{\pi }}}\Vert _{\infty } + \Vert \mathcal {B}_{{\hat{\pi }}} V_{{\hat{\pi }}} - \mathcal {B}_{{\hat{\pi }}} V_* \Vert _{\infty } \nonumber \\&\quad + \Vert \mathcal {B}_{{\hat{\pi }}} V_* - {\hat{\mathcal {B}}}_{{\hat{\pi }}} V_* \Vert _{\infty } + \Vert {\hat{\mathcal {B}}}_{{\hat{\pi }}} V_* - {\hat{\mathcal {B}}}_{{\hat{\pi }}} {\hat{V}}_{*} \Vert _{\infty } + \Vert {\hat{\mathcal {B}}}_{{\hat{\pi }}} {\hat{V}}_{*} - {\hat{\mathcal {B}}}_{{\hat{\pi }}} {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } \nonumber \\&{\mathop {\le }\limits ^{(a)}}\Vert V_* - {\hat{V}}_{*} \Vert _{\infty } + \Vert {\hat{V}}_{{\hat{\pi }}_*} - {\hat{V}}_{{\hat{\pi }}}\Vert _{\infty } + \gamma \Vert V_{{\hat{\pi }}} - V_* \Vert _{\infty } \nonumber \\&\quad + \Vert \mathcal {B}_{{\hat{\pi }}} V_* - {\hat{\mathcal {B}}}_{{\hat{\pi }}} V_* \Vert _{\infty } + \gamma \Vert V_* - {\hat{V}}_{*} \Vert _{\infty } + \gamma \Vert {\hat{V}}_{*} - {\hat{V}}_{{\hat{\pi }}} \Vert _{\infty } \end{aligned}$$

(47)

where (a) from the contraction property of the Bellman operators. Rearranging terms and using (44), we get that

$$\begin{aligned} \Vert V_* - V_{{\hat{\pi }}} \Vert _{\infty }&\le \frac{1}{1-\gamma } \bigl [ (1+\gamma ) \Vert V_* - {\hat{V}}_{*} \Vert _{\infty } + (1 + \gamma ) \alpha _{\textrm{opt}} + \Vert \mathcal {B}_{{\hat{\pi }}} V_* - {\hat{\mathcal {B}}}_{{\hat{\pi }}} V_* \Vert _{\infty } \bigr ] \end{aligned}$$

(48)

Now the first term in (48) can be simplified similar to (43), where in step (b) of (43), we need to add and subtract $\sum _{s' \in \mathcal {S}} \widehat{{\textsf{P}}}(s' | s, a) V_{\pi _*}(s')$. Using this, we would obtain

$$\begin{aligned} \Vert V_* - {\hat{V}}_{*} \Vert _{\infty } \le \varepsilon + \frac{\gamma {{\overline{\varDelta }}}_{\pi _*}}{1-\gamma }. \end{aligned}$$

(49)

The last term in (48) can be simplified as follows:

$$\begin{aligned} \Vert \mathcal {B}_{{\hat{\pi }}} V_* - {\hat{\mathcal {B}}}_{{\hat{\pi }}} V_* \Vert _{\infty }&\le \max _{s\in \mathcal {S}} \sum _{a\in \mathcal {A}} {\hat{\pi }}(a| s) \biggl [ | r(s,a) - {\hat{r}}(s,a) | \nonumber \\&\qquad \qquad + \gamma \biggl | \sum _{s' \in \mathcal {S}} {\textsf{P}}(s'|s,a) V_*(s') - \sum _{s' \in \mathcal {S}} \widehat{{\textsf{P}}}(s' | s, a) V_*(s') \biggr | \biggr ] \nonumber \\&\le \varepsilon + \gamma {{\overline{\varDelta }}}_{\pi _*} \end{aligned}$$

(50)

The result of Theorem 5 follows by substituting (49) and (50) in (48). $\square$