Deep Reinforcement Learning with Temporal Logics

Abstract

The combination of data-driven learning methods with formal reasoning has seen a surge of interest, as either area has the potential to bolstering the other. For instance, formal methods promise to expand the use of state-of-the-art learning approaches in the direction of certification and sample efficiency. In this work, we propose a deep Reinforcement Learning (RL) method for policy synthesis in continuous-state/action unknown environments, under requirements expressed in Linear Temporal Logic (LTL). We show that this combination lifts the applicability of deep RL to complex temporal and memory-dependent policy synthesis goals. We express an LTL specification as a Limit Deterministic Büchi Automaton (LDBA) and synchronise it on-the-fly with the agent/environment. The LDBA in practice monitors the environment, acting as a modular reward machine for the agent: accordingly, a modular Deep Deterministic Policy Gradient (DDPG) architecture is proposed to generate a low-level control policy that maximises the probability of the given LTL formula. We evaluate our framework in a cart-pole example and in a Mars rover experiment, where we achieve near-perfect success rates, while baselines based on standard RL are shown to fail in practice.

Appendix: Proof of Theorem 1

Theorem 1. Let $\varphi$ be a given LTL formula and $\mathfrak {M}_\mathfrak {A}$ be the product MDP constructed by synchronising the MDP $\mathfrak {M}$ with the LDBA $\mathfrak {A}$ associated with $\varphi$. Then the optimal stationary Markov policy on $\mathfrak {M}_\mathfrak {A}$ that maximises the expected return, maximises the probability of satisfying $\varphi$ and induces a finite-memory policy on the MDP $\mathfrak {M}$.

Proof. Assume that the optimal Markov policy on $\mathfrak {M}_\mathfrak {A}$ is ${\pi ^\otimes }^*$, namely at each state $s^\otimes$ in $\mathfrak {M}_\mathfrak {A}$ we have

$$\begin{aligned} {\pi ^\otimes }^*(s^\otimes )=\arg \!\!\!\!\sup \limits _{\pi ^\otimes \in \mathcal {D}^\otimes } {U}^{\pi ^\otimes }(s^\otimes )=\arg \!\!\!\!\sup \limits _{\pi ^\otimes \in \mathcal {D}^\otimes }\mathbb {E}^{\pi ^\otimes } [\sum _{n=0}^{\infty } \gamma ^n~ R(s^\otimes _n,a_n)|s^\otimes _0=s^\otimes ], \end{aligned}$$

(15)

where $\mathcal {D}^\otimes$ is the set of stationary deterministic policies over the state space $\mathcal {S}^\otimes$, $\mathbb {E}^{\pi ^\otimes } [\cdot ]$ denotes the expectation given that the agent follows policy $\pi ^\otimes$, and $s_0^\otimes ,a_0,s_1^\otimes ,a_1,\ldots$ is a generic path generated by the product MDP under policy $\pi ^\otimes$.

Recall that an infinite word $w \in {\Sigma }^\omega ,~\Sigma =2^\mathcal {AP}$ is accepted by the LDBA $\mathfrak {A}=(\mathcal {Q},q_0,\Sigma , \mathcal {F}, \varDelta )$ if there exists an infinite run $\theta \in \mathcal {Q}^\omega$ starting from $q_0$ where $\theta [i+1] \in \varDelta (\theta [i],\omega [i]),~i \ge 0$ and, for each $F_j \in \mathcal {F}$, $inf (\theta ) \cap F_j \ne \emptyset ,$ where $inf (\theta )$ is the set of states that are visited infinitely often in the sequence $\theta$. From Definition 8, the associated run $\theta$ of an infinite path in the product MDP $\rho = s^\otimes _0 \xrightarrow {a_0} s^\otimes _1 \xrightarrow {a_1} ...$ is $\theta = L^\otimes (s^\otimes _0)L^\otimes (s^\otimes _1)...$. From Definition 9 and (10), and since for an accepting run $inf (\theta ) \,\cap \, F_j \ne \emptyset ,~\forall F_j \in \mathcal {F}$, all accepting paths starting from $s_0^\otimes$, accumulate infinite number of positive rewards $r_p$ (see Remark 2).

In the following, by contradiction, we show that any optimal policy ${\pi ^\otimes }^*$ satisfies the property with maximum possible probability. Let us assume that there exists a stationary deterministic Markov policy ${\pi ^\otimes }^+\ne {\pi ^\otimes }^*$ over the state space $\mathcal {S}^\otimes$ such that probability of satisfying $\varphi$ under ${\pi ^\otimes }^+$ is maximum.

This essentially means in the product MDP $\mathfrak {M}_\mathfrak {A}$ by following ${\pi ^\otimes }^+$ the expectation of reaching the point where $inf (\theta ) \cap F_j \ne \emptyset ,~\forall F_j \in \mathcal {F}$ and positive reward is received ever after is higher than any other policy, including ${\pi ^\otimes }^*$. With a tuned discount factor $\gamma$, e.g. (1),

$$\begin{aligned} \mathbb {E}^{{\pi ^\otimes }^+} [\sum _{n=0}^{\infty } \gamma ^n~ R(s^\otimes _n,a_n)|s^\otimes _0=s^\otimes ] > \mathbb {E}^{{\pi ^\otimes }^*} [\sum _{n=0}^{\infty } \gamma ^n~ R(s^\otimes _n,a_n)|s^\otimes _0=s^\otimes ] \end{aligned}$$

(16)

This is in contrast with optimality of ${\pi ^\otimes }^*$ (15) and concludes ${\pi ^\otimes }^*={\pi ^\otimes }^+$. Namely, an optimal policy that maximises the expected return also maximises the probability of satisfying LTL property $\varphi$. It is easy to see that the projection of policy ${\pi ^\otimes }^*$ on MDP $\mathfrak {M}$ is a finite-memory policy $\pi ^*$.    $\Box$