Semantics and algorithms for trustworthy commitment achievement under model uncertainty

Abstract

We focus on how an agent can exercise autonomy while still dependably fulfilling commitments it has made to another, despite uncertainty about outcomes of its actions and how its own objectives might evolve. Our formal semantics treats a probabilistic commitment as constraints on the actions an autonomous agent can take, rather than as promises about states of the environment it will achieve. We have developed a family of commitment-constrained (iterative) lookahead algorithms that provably respect the semantics, and that support different tradeoffs between computation and plan quality. Our empirical results confirm that our algorithms’ ability to balance (selfish) autonomy and (unselfish) dependability outperforms optimizing either alone, that our algorithms can effectively handle uncertainty about both what actions do and which states are rewarding, and that our algorithms can solve more computationally-demanding problems through judicious parameter choices for how far our algorithms should lookahead and how often they should iterate.

Appendix

Here we present all the technical proofs of the theorems in this article.

Proof of Theorem 1

Note that the belief is a sufficient statistic: given history $h_t$ at time step t and the corresponding belief $b_t$ consistent with $h_t$, one does not need any other information in $h_t$ besides $b_t$ to predict the future state transitions and reward after time step t. Therefore, solving problem (4) is equivalent to solving a constrained MDP, where the MDP is the belief MDP defined as the tuple $\langle {\mathcal {B}}, {\mathcal {A}}, b_0, {\tilde{P}}, {\tilde{R}} \rangle$ with finite state space of beliefs, and the constraint comes from the semantics of commitment c. Our CCFL method can be viewed as a standard linear programming approach to solving a finite state constrained MDP. $\square$

Proof of Theorem 2

It is sufficient to show (1) any policy in ${\varPi }_c \cap {\varPi }_L$ can be derived from a feasible solution to the program in Fig. 5, and (2) any feasible solution to the program derives a policy in ${\varPi }_c \cap {\varPi }_L$.

To show (1), for any policy $\pi \in {\varPi }_c\cap {\varPi }_L$, we are going to define vectors $m^\pi$ and $n^\pi$ such that with $m^\pi$ treated as x and $n^\pi$ treated as y, $m^\pi$ and $n^\pi$ satisfy the constraints of the program in Fig. 5, and the L-updates policy $\pi$ can be derived via Eq. (11). Specifically, given any policy $\pi \in {\varPi }_c\cap {\varPi }_L$, let $n^\pi$ be its belief-action occupancy measure for beliefs in ${\mathcal {B}}_{\le L}^{b_0}$, and $m^\pi$ be its state-action occupancy measure for states from time step L on:

$$\begin{aligned} \forall b\in {\mathcal {B}}_{\le L}^{b_0}, a\quad n^\pi (b,a) = \Pr (B_t=b,A_t=a|B_0=b_0;\pi ) \end{aligned}$$

where t is the time of belief b, and

$$\begin{aligned} \forall s, a \quad m^\pi _{b_L,k}(s,a) ={\left\{ \begin{array}{ll} \Pr (S_t=s,A_t=a, B_L=b_L,k|B_0=b_0;\pi ) &{} t \ge L\\ 0 &{} t < L \end{array}\right. } \end{aligned}$$

where t is the time of state s. Then, with $m^\pi$ treated as x and $n^\pi$ treated as y, $m^\pi$ and $n^\pi$ satisfy the constraints of the program in Fig. 5, and the L-updates policy $\pi$ can be derived via Eq. (11).

To show (2), given a feasible solution x, y to the program, let policy $\pi$ be the derived policy via (11). Then $\pi$ is in ${\varPi }_L$ by definition. Further we have $m^\pi _{b_L,k}(s,a)=x_{b_L,k}(s,a), n^\pi (b,a)=y(b,a)$, where $m^\pi$ and $n^\pi$ are defined as above. Therefore $\pi$ is also in ${\varPi }_c$ because x satisfies commitment constraints (12i), (12h). $\square$

Proof of Theorem 3

By Theorem 2, CCL with boundary L finds the optimal policy in ${\varPi }_c\cap {\varPi }_L$. Therefore, it is sufficient to show

$$\begin{aligned} \forall L>0, {\varPi }_0 \subseteq {\varPi }_L. \end{aligned}$$

This holds because given any Markov policy $\pi _0\in {\varPi }_0$ we can define an L-updates policy $\pi _L\in {\varPi }_L$ that is equivalent to $\pi _0$:

$$\begin{aligned} \pi _L (a|h_t) = {\left\{ \begin{array}{ll} \pi _L (a|b_t)=\pi _0 (a|s_t) &{} t < L\\ \pi _L (a|s_t, b_L)=\pi _0 (a|s_t)&{} t \ge L \end{array}\right. }. \end{aligned}$$

Thus, we know that $\pi _0\in {\varPi }_L$. $\square$

Proof of Theorem 4

It is sufficient to show that the statement holds when $L'=L+1$. We next show that when $P_k=P_{k'} ~\forall k, k'$, given any policy $\pi _L \in {\varPi }_{L}$, there exists an $(L+1)$-updates policy, $\pi _{L+1}$, that mimics $\pi _L$ , and therefore $V^{\pi _L^*}_{\mu _0}(s_0) \le V^{\pi _{L+1}^*}_{\mu _0}(s_0)$.

For the first L actions, an $(L+1)$-updates policy can map the current belief to a distribution of the next actions identical to $\pi _{L}$, and the action that is going to be taken at time step L by $\pi _{L}$ can also be recovered by an $(L+1)$-updates policy, which gives

$$\begin{aligned} \pi _{L+1} (a|h_t) = {\left\{ \begin{array}{ll} \pi _{L+1} (a|b_t) =\pi _{L} (a|b_t) &{} t < L\\ \pi _{L+1} (a|b_L) = \pi _{L} (a|s_L, b_L) &{} t=L \end{array}\right. }. \end{aligned}$$

Under any L-updates policy $\pi _L$, and conditioned on being in belief $b_{L+1}$ at time step $L+1$, the agent thereafter selects actions according to $\pi _L(\cdot |s_t,b_L)$ with probability that the agent was in belief $b_L$ at time step L: $\Pr (b_L|b_{L+1};\pi _L)$. If the transition dynamics does not vary across MDPs in the environment, it is well known [26] that a Markov policy $\pi _{b_{L+1}}(\cdot |s_t), t\ge L+1$ is sufficient to recover the state occupancy measure of $\pi _L$ starting at belief $b_{L+1}$. Then $\pi _{L+1}$ can also recover $\pi _{L}$ for $t\ge L+1$ by demonstrating that $\pi _{b_{L+1}}$ satisfies

$$\begin{aligned} \pi _{L+1} (a|h_t) =\pi _{L+1} (a|s_t, b_{L+1}) = \pi _{b_{L+1}}(a|s_t) \qquad \text {for } t\ge L+1. \end{aligned}$$

This concludes the proof. $\square$

Proof of Theorem 5

In the proof of Theorem 4, we have shown that for any L-updates policy $\pi _L$ there exists an $(L+1)$-update policy that is able to mimic $\pi _L$ up to time step $L+1$. Provided that $P_k=P_{k'} ~\forall k, k'$, one can find a Markov policy that mimics $\pi _L$ starting at any belief at time step $L+1$. When $P_k=P_{k'} ~\forall k, k'$ does not hold, however, this Markov policy in general does not exist, and therefore no $(L+1)$-update policy is able to mimic $\pi _L$. Inspired by this, we next give an example as a formal constructive proof.

Consider the example shown in Fig. 13. The environment has 10 locations $\{0,1,\ldots ,9\}$, action space $\{up, down\}$, time horizon $T=4$, and $K=2$ possible MDPs. The agent starts in location 0 at time step $t=0$ with a prior probability of 0.8 for MDP $k=1$ and a prior probability of 0.2 for MDP $k=2$. In MDP $k=1$, no matter which action the agent takes, it transits to location 1 or 2 uniformly at random at time step $t=1$, and then to location 3 with probability one at time step $t=2$. Starting from location 3, on taking action up (down) the agent transits to the upper (lower) location to the right. The transition dynamics of MDP $k=2$ is the same as MDP $k=1$ until the agent reaches location 3, and thereafter the transition is flipped: starting from location 3, on taking action up (down) the agent transits to the lower (upper) location to the right. In both MDPs, the agent will receive large negative reward ($-\infty$) in location 7 and 8. In MDP $k=1$, the agent will receive $+$ 1 reward if it reaches location 6. There is no reward elsewhere. The agent commits to reaching location 9 with probability 0.5. Consider the following $(L=)1$-updates policy: if the agent was in location 1 at time step $t=1$, always choose action up; if the agent was in location 2 at time step $t=1$, always choose action down. Under this $(L=)1$-updates policy the probability of reaching the commitment location 9 is 0.5 and the expected reward is $0.8\times 0.5\times 1=0.4$. Now consider $(L=)2$-updates policies. Because the agent is in location 3 with probability one at time step $t=2$. An $(L=)2$-updates policy amounts to a Markov policy for time steps $t\ge 2$. Further the agent should minimize the probability of reaching location 7 and 8 that yields large negative reward. One can verify that the only Markov policy for time steps $t\ge 2$ that avoids reaching location 7 and 8 while respecting the commitment semantics is to always choose action down, whose expected reward is 0, smaller than that of the $(L=)1$-updates policy. $\square$

Fig. 13

Example as a proof of Theorem 5

Proof of Theorem 6

We need to show $\pi _{IL}$ satisfies Eq. (3), i.e.,

$$\begin{aligned} \mathop {\mathrm{Pr}}\limits _{k\sim \mu _0} ( S_T \in {\varPhi }| S_0=s_0,k;\pi _{IL}) \ge \rho . \end{aligned}$$

Let $\pi _{L}$ be the CCL L-updates policy derived from the program in Fig. 5. The above inequality holds because:

$$\begin{aligned}&\mathop {\mathrm{Pr}}\limits _{k\sim \mu _0}(S_T\in {\varPhi }| S_0=s_0,k; \pi _{IL} )\\&\quad =\sum \limits _{b_I\in {\mathcal {B}}_{I}^{b_0}}\mathop {\mathrm{Pr}}\limits _{k\sim \mu _0}(B_I=b_I | S_0=s_0, k;\pi _{IL} )\Pr (S_T\in {\varPhi }| B_I=b_I; \pi _{IL} ) \\&\qquad \hbox {(law of total probability)}\\&\quad =\sum \limits _{b_I\in {\mathcal {B}}_{I}^{b_0}}\mathop {\mathrm{Pr}}\limits _{k\sim \mu _0}(B_I=b_I | S_0=s_0, k;\pi _{L} )\Pr (S_T\in {\varPhi }| B_I=b_I; \pi _{IL} ) \\&\qquad (\pi _L \, \hbox {and} \, \pi _{IL} \, \hbox {are identical in the first} \, I \, \hbox {steps)} \\&\quad \ge \sum \limits _{b_I\in {\mathcal {B}}_{I}^{b_0}}\mathop {\mathrm{Pr}}\limits _{k\sim \mu _0}(B_I=b_I | S_0=s_0, k;\pi _{L})\Pr (S_T\in {\varPhi }| B_I=b_I; \pi _{L} ) \\&\quad =\mathop {\mathrm{Pr}}\limits _{k\sim \mu _0}(S_T\in {\varPhi }| S_0=s_0, k;\pi _L ) \qquad \hbox {(law of total probability)} \\&\quad \ge \rho \qquad {(\pi _L\in {\varPi }_c)} \end{aligned}$$

The first inequality holds because CCIL iteratively applies L-step lookahead with the commitment probability achieved by the policy of the previous iteration. This concludes the proof. $\square$