A leader–follower partially observed, multiobjective Markov game

Abstract

The intent of this research is to generate a set of non-dominated finite-memory policies from which one of two agents (the leader) can select a most preferred policy to control a dynamic system that is also affected by the control decisions of the other agent (the follower). The problem is described by an infinite horizon total discounted reward, partially observed Markov game (POMG). For each candidate finite-memory leader policy, we assume the follower, fully aware of the leader policy, determines a (perfect memory) policy that optimizes the follower’s (scalar) criterion. The leader–follower assumption allows the POMG to be transformed into a specially structured, partially observed Markov decision process that we use to determine the follower’s best response policy for a given leader policy. We then approximate the follower’s policy by a finite-memory policy. Each agent’s policy assumes that the agent knows its current and recent state values, its recent actions, and the current and recent possibly inaccurate observations of the other agent’s state. For each leader/follower policy pair, we determine the values of the leader’s criteria. We use a multi-objective genetic algorithm to create the next generation of leader policies based on the values of the leader criteria for each leader/follower policy pair in the current generation. Based on this information for the final generation of policies, we determine the set of non-dominated leader policies. We present an example that illustrates how these results can be used to support a manager of a liquid egg production process (the leader) in selecting a sequence of actions to maximize expected process productivity while mitigating the risk due to an attacker (the follower) who seeks to contaminate the process with a chemical or biological toxin.

Appendix

Proof of Proposition 1

Assume v and $\varGamma$ are such that

$$\begin{aligned} v({\mathscr {I}}^F(t)) = \min \left\{ \sum \gamma ({\mathscr {I}}^L(t, \tau )) P({\mathscr {I}}^L(t, \tau )| {\mathscr {I}}^F(t)){:}\; \gamma \in \varGamma (s^F(t))\right\} , \end{aligned}$$

where the sum is over all $I^L(t, \tau )$. Then the analysis, following the same line of arguments in Smallwood and Sondik (1973), shows that

$$\begin{aligned} h^F({\mathscr {I}}^F(t), a^F(t), v) = \min \left\{ \sum \gamma '({\mathscr {I}}^L(t, \tau )) P({\mathscr {I}}^L(t, \tau )| {\mathscr {I}}^F(t)){:}\; \gamma ' \in \varGamma '(s^F(t), a^F(t))\right\} , \end{aligned}$$

where the sum is over all ${\mathscr {I}}^L(t, \tau )$. If $\gamma ' \in \varGamma '(s^F(t), a^F(t))$ then $\gamma '$ is of the form

$$\begin{aligned} \gamma ({\mathscr {I}}^L(t, \tau ))&= \sum _{a^L(t)}P(a^L(t)| {\mathscr {I}}^L(t, \tau ))\left[ c^F(s(t), a(t))\right. \\&\quad + \beta \sum _{s(t+1)} \sum _{z(t+1)} \gamma ^{i,j}(z^L(t+1), s^L(t+1), a^L(t), \\&\quad \left. {\mathscr {I}}^L(t, \tau - 1)) P(z(t+1), s(t+1)| s(t), a(t))\right] , \end{aligned}$$

where $\gamma ^{i,j}$ can be any element in $\varGamma (s^F(t+1))$ for each $s^F(t+1)=i$ and $z^F(t+1)=j$. And $\{z^L(t+1),s^L(t+1),a^L(t),{\mathscr {I}}^L(t,\tau -1)\}={\mathscr {I}}^L(t+1,\tau )$. Then,

$$\begin{aligned}{}[H^Fv]({\mathscr {I}}^F(t))=\min \left\{ \sum _{{\mathscr {I}}^L(t,\tau )} \gamma ''({\mathscr {I}}^L(t,\tau ))P({\mathscr {I}}^L(t,\tau )|I^F(t)){:}\; \gamma '' \in \varGamma ''(s^F(t))\right\} , \end{aligned}$$

where $\varGamma ''(s^F(t))=\cup _{a^F(t)}\varGamma '(s^F(t),a^F(t)).$

The operator $H^F$ is a contraction operator on the Banach space comprised of all functions mapping ${\mathscr {I}}^F(t)$ into the real line, having as its norm the supremum norm, and as a result, the sequence $\{v^{n}\}$, where $v^{n+1} = H^Fv^{n}$, converges to $v^F$ for any given $v^{0}$. The above result indicates that $H^F$ preserves piecewise linearity and concavity and in the limit preserves concavity. $\square$

Proof of Proposition 2

We remark that since $(\pi ^L, \pi ^F)$ is assumed given, $P(s(t),a(t)|{\mathscr {I}}^F(t,\tau ), {\mathscr {I}}^L(t,\tau ))$ is well defined. Assume there is a function g such that

$$\begin{aligned} v({\mathscr {I}}^L(t)) = \sum g({\mathscr {I}}^L(t,\tau ),{\mathscr {I}}^F(t,\tau )) P({\mathscr {I}}^F(t,\tau )|{\mathscr {I}}^L(t)), \end{aligned}$$

where the sum is over all ${\mathscr {I}}^F(t,\tau )$. Then it is straightforward to show that there is a function $g'$ such that

$$\begin{aligned} h^L_i({\mathscr {I}}^L(t),v) = \sum g'({\mathscr {I}}^L(t,\tau ), {\mathscr {I}}^F(t,\tau ))P({\mathscr {I}}^F(t,\tau )|{\mathscr {I}}^L(t)), \end{aligned}$$

where the sum is over all ${\mathscr {I}}^F(t,\tau )$, and

$$\begin{aligned} g'({\mathscr {I}}^L(t, \tau ), {\mathscr {I}}^F(t, \tau ))&= \sum \nolimits ^1 P(s(t),a(t)|{\mathscr {I}}^F(t, \tau ),{\mathscr {I}}^L(t, \tau ))\left\{ c^L_i(s(t),a(t))\right. \\&\quad +\beta \sum \nolimits ^2 g[({\mathfrak {z}}^L(t+1), {\mathscr {I}}^L(t, \tau -1)),\\&\quad \left. ({\mathfrak {z}}^F(t+1), {\mathscr {I}}^F(t, \tau -1))]P(z(t+1),s(t+1)|s(t),a(t)) \right\} , \end{aligned}$$

and where ${\mathfrak {z}}^k(t)=\{z^k(t),s^k(t),a^k(t-1) \}$, $\sum ^1$ is over all s(t) and a(t), and $\sum ^2$ is over all $z(t+1)$ and $s(t+1)$.The result follows directly from the following facts:

• The operator $H^L$, where $[H^Lv]({\mathscr {I}}^L(t)) = h^L_i({\mathscr {I}}^L(t), v)$, is a contraction operator on the Banach space comprised of all functions mapping the set of all ${\mathscr {I}}^L(t)$ into the real line, having as its norm the supremum norm.

• As a result, the sequence $\{v^{n}\}$, where $v^{n+1} = H^Lv^{n}$, converges to $v^L$ for any given $v^{0}$.

$\square$

Determine $y^k(t+1)$, given $y^k(t), z^k(t+1), s^k(t+1)$ and $a^k(t)$: Let $\varsigma ^k(t)=\{z^k(t),s^k(t),a^k(t-1) \}$ and $\varsigma (t)=\{\varsigma ^L(t), \varsigma ^F(t) \}$. Without loss of generality, we determine $y^F(t+1)$, given $y^F(t)$ and $\varsigma ^F(t+1)$. Note,

${\mathscr {I}}^L(t+1,\tau )=\{\varsigma ^L(t+1), {\mathscr {I}}^L(t,\tau -1) \}$ and ${\mathscr {I}}^F(t+1)=\{\varsigma ^F(t+1), {\mathscr {I}}^F(t) \}$. Then,

$$\begin{aligned} P(\varsigma ^L(t+1), {\mathscr {I}}^L(t,\tau -1)|\varsigma ^F(t+1),{\mathscr {I}}^F(t)) =\sum _{\varsigma '}P(\varsigma ^L(t+1), {\mathscr {I}}^L(t,\tau )|\varsigma ^F(t+1),{\mathscr {I}}^F(t)), \end{aligned}$$

where $\varsigma '=\varsigma ^L(t-\tau +1).$

Note

$$\begin{aligned} P(\varsigma (t+1),{\mathscr {I}}^L(t,\tau )|{\mathscr {I}}^F(t)) =P(\varsigma ^L(t+1),{\mathscr {I}}^L(t,\tau )|\varsigma ^F(t+1), {\mathscr {I}}^F(t))P(\varsigma ^F(t+1)|{\mathscr {I}}^F(t)) \end{aligned}$$

and that

$$\begin{aligned} P(\varsigma ^F(t+1)|{\mathscr {I}}^F(t))=\sum _{\varsigma ''} \sum _{{\mathscr {I}}}P(\varsigma (t+1),{\mathscr {I}}^L(t,\tau )| {\mathscr {I}}^F(t)), \end{aligned}$$

where $\varsigma ''=\varsigma ^L(t+1), {\mathscr {I}}={\mathscr {I}}^L(t,\tau )$.

Now,

$$\begin{aligned} P(\varsigma (t+1),{\mathscr {I}}^L(t,\tau )|{\mathscr {I}}^F(t)) =P(\varsigma (t+1)|{\mathscr {I}}^L(t,\tau ),{\mathscr {I}}^F(t)) P({\mathscr {I}}^L(t,\tau )|{\mathscr {I}}^F(t)). \end{aligned}$$

Then,

$$\begin{aligned} P(\varsigma (t+1)|{\mathscr {I}}^L(t,\tau ),{\mathscr {I}}^F(t))&=P(z(t+1),s(t+1)|a(t),{\mathscr {I}}^L(t,\tau ),{\mathscr {I}}^F(t))\\&\quad \times P(a(t)|{\mathscr {I}}^L(t,\tau ),{\mathscr {I}}^F(t)). \end{aligned}$$

Thus, we note that $P(\varsigma ^L(t+1),{\mathscr {I}}^L(t,\tau -1)|\varsigma ^F(t+1), {\mathscr {I}}^F(t))$ is a function of $\{P({\mathscr {I}}^L(t,\tau )|{\mathscr {I}}^F(t))\}$, which is the result.

Example 1

Theoretical analysis on the quality of finite-memory policy has been examined by White and Scherer (1994). Below is an example from our numerical experiment. Parameter values for this example are presented in Chang (2015). The resulting $\gamma$ vectors for $s^F=1$ are:

$(s^L, z^L)$	$(s_1,z_1)$	$(s_1,z_2)$	$(s_2, z_1)$	$(s_2, z_2)$
$a^F = a_1$	[4.4784	4.1677	4.4784	4.1677]
$a^F = a_2$	[4.7137	2.5929	4.7137	2.5929]

At the related information pattern (see Chang 2015 for details), the follower will select action $a_2$ for $y^F(t)=P(s^L(t),z^L(t)|{\mathscr {I}}^F(t))$ where $P(s_1,z_1|{\mathscr {I}}^F(t)) + P(s_2,z_1|{\mathscr {I}}^F(t)) \ge 0.85$. Let $\tau = 1$. Drawing 500 samples of $y^F(t-1)$ from a uniform distribution over $S^L\times Z^L$ indicates that the resulting approximate finite memory policy is $P(a^F(t)=a_2|{\mathscr {I}}^F(t,\tau =1)) = 0.938$.