Approachability in Stackelberg Stochastic Games with Vector Costs

Abstract

The notion of approachability was introduced by Blackwell (Pac J Math 6(1):1–8, 1956) in the context of vector-valued repeated games. The famous ‘Blackwell’s approachability theorem’ prescribes a strategy for approachability, i.e., for ‘steering’ the average vector cost of a given agent toward a given target set, irrespective of the strategies of the other agents. In this paper, motivated by the multi-objective optimization/decision-making problems in dynamically changing environments, we address the approachability problem in Stackelberg stochastic games with vector-valued cost functions. We make two main contributions. Firstly, we give a simple and computationally tractable strategy for approachability for Stackelberg stochastic games along the lines of Blackwell’s. Secondly, we give a reinforcement learning algorithm for learning the approachable strategy when the transition kernel is unknown. We also recover as a by-product Blackwell’s necessary and sufficient conditions for approachability for convex sets in this setup and thus a complete characterization. We give sufficient conditions for non-convex sets.

Appendix

Consider a controlled Markov chain $\{s_n\}$ with a finite state space S, compact metric action space U with metric d and running cost k(i, u), with transition probabilities p(j | i, u). Assume k, p to be continuous in u. Also assume that Assumption 1 is true. The dynamic programming equation then is

$$\begin{aligned} V(i) = \min _u\left( k(i,u) - \kappa + \sum _jp(j | i,u)V(j)\right) , \quad \ i \in S. \end{aligned}$$

Then, $\kappa$ is the optimal cost. Let $U^*(i)$ denote the set of minimizers on the right-hand side, which will perforce be compact and non-empty by standard arguments. Suppose

$$\begin{aligned} d(a_n, U^*(s_n)) \rightarrow 0. \end{aligned}$$

Then, we have

$$\begin{aligned} V(s_n) - k(s_n, a_n) - \kappa + E[V(s_{n+1})|s_n, a_n]\rightarrow & {} 0 \\ \Longrightarrow \ \lim _{n\uparrow \infty }\frac{1}{n}\sum _{m = 0}^{n-1}E[k(s_m, a_m)] - \kappa= & {} \lim _{n\uparrow \infty }\frac{1}{n}(E[V(s_{n})] - E[V(s_0)]) = 0. \end{aligned}$$

Hence, $\{a_n\}$ is optimal.