PerceMon: Online Monitoring for Perception Systems

Abstract

Perception algorithms in autonomous vehicles are vital for the vehicle to understand the semantics of its surroundings, including detection and tracking of objects in the environment. The outputs of these algorithms are in turn used for decision-making in safety-critical scenarios like collision avoidance, and automated emergency braking. Thus, it is crucial to monitor such perception systems at runtime. However, due to the high-level, complex representations of the outputs of perception systems, it is a challenge to test and verify these systems, especially at runtime. In this paper, we present a runtime monitoring tool, PerceMon that can monitor arbitrary specifications in Timed Quality Temporal Logic (TQTL) and its extensions with spatial operators. We integrate the tool with the CARLA autonomous vehicle simulation environment and the ROS middleware platform while monitoring properties on state-of-the-art object detection and tracking algorithms.

A Semantics for STQL

Consider a data stream $\xi$ consisting of frames containing objects and annotated with a time stamp. Let $i \in \mathbb {N}$ be the current frame of evaluation, and let $\xi _i$ denote the $i^{th}$ frame. We let $\epsilon : V_t \cup V_f \rightarrow \mathbb {N}\cup \{\mathrm {NaN}\}$ denote a mapping from a pinned time or frame variable to a frame index (if it exists), and let $\zeta : V_o \rightarrow \mathbb {N}$ be a mapping from an object variable to an actual object ID that was assigned by a quantifier. Finally, we let $\mathcal {O}(\xi _i)$ denote the set of object IDs available in the frame $i$, and let $t(\xi _i)$ output the timestamp of the given frame.

Let $\llbracket \varphi \rrbracket$ be the quality of the STQL formula, $\varphi$, at the current frame $i$, which can be recursively defined as follows:

• For the propositional and temporal operations, the semantics simply follows the Boolean semantics for LTL or MTL, i.e.,

  

• For constraints on time and frame variables,

  $$\begin{aligned} \llbracket x - \textsf {C\_TIME}\sim c \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \epsilon (x) - t(\xi _i) \sim c \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket f - \textsf {C\_FRAME}\sim c \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \epsilon (f) - i \sim c \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \end{aligned}$$

• For operations on object variables,

  $$\begin{aligned} \llbracket \{id_j = id_j\} \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \zeta (id_j) = \zeta (id_k) \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {C}}\,}}(id_j) = c \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \mathcal {O}(\xi _i)(\zeta (id_j)).\text {class} = c \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {C}}\,}}(id_j) = {{\,\mathrm{\mathsf {C}}\,}}(id_k) \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \mathcal {O}(\xi _i)(\zeta (id_j)).\text {class} \\ \qquad \qquad = \mathcal {O}(\xi _i)(\zeta (id_k)).\text {class} \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {P}}\,}}(id_j) \sim r \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \mathcal {O}(\xi _i)(\zeta (id_j)).\text {prob} \sim r \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {P}}\,}}(id_j) \sim r \times {{\,\mathrm{\mathsf {P}}\,}}(id_k) \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } \mathcal {O}(\xi _i)(\zeta (id_j)).\text {prob} \sim r \\ \qquad \qquad \times \mathcal {O}(\xi _i)(\zeta (id_k)).\text {prob} \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \end{aligned}$$

• For the area, latitudinal offset, and longitudinal offset,

  $$\begin{aligned} \llbracket {{\,\mathrm{\mathsf {Area}}\,}}(\mathcal {T}_1) \sim r \rrbracket&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } {{\,\mathrm{\mathsf {Area}}\,}}({{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}_1, \xi , \zeta )) \sim r \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {Lat}}\,}}(id_1, \mathrm {CRT}_1) \sim r \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } f_{lat}{}(id_1, \mathrm {CRT}_1, \xi , i, \epsilon , \zeta ) \sim r \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \\ \llbracket {{\,\mathrm{\mathsf {Lon}}\,}}(id_1, \mathrm {CRT}_1) \sim r \rrbracket (\xi , i, \epsilon , \zeta )&= {\left\{ \begin{array}{ll} \top ,\quad \text {if } f_{lon}(id_1, \mathrm {CRT}_1, \xi , i, \epsilon , \zeta ) \sim r \\ \bot ,\quad \text {otherwise.} \end{array}\right. } \end{aligned}$$

  where, $\sim \in \{ <,>,\le ,\ge \}$, and

  • $f_{lat}$ computes the lateral distance of the $\mathrm {CRT}$ point of an object identified by $\mathcal {O}(\zeta (id_1))$ from the Longitudinal axis;

  • $f_{lon}$ computes the longitudinal distance of the $\mathrm {CRT}$ point of an object identified by $\mathcal {O}(\zeta (id_1))$ from the Lateral axis; and

  • ${{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}, \xi , \zeta )$ is the compound spatial object created after set operations on bounding boxes (defined below).

• And, finally, for the spatial existence operator,

  

  Here, the compound spatial function, ${{\,\mathrm{\mathfrak {U}}\,}}$ is defined as follows:

  $$\begin{aligned} {{\,\mathrm{\mathfrak {U}}\,}}(\varnothing , \xi , \zeta )&= \emptyset \\ {{\,\mathrm{\mathfrak {U}}\,}}(\mathbb {U}, \xi , \zeta )&= \mathbb {U}\\ {{\,\mathrm{\mathfrak {U}}\,}}({{\,\mathrm{\mathfrak {BB}}\,}}(id), \xi , \zeta )&= \zeta (id).\text {bbox} \\ {{\,\mathrm{\mathfrak {U}}\,}}(\overline{\mathcal {T}}, \xi , \zeta )&= \mathbb {U}\setminus {{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}, \xi , \zeta ) \\ {{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}_1 \sqcup \mathcal {T}_2, \xi , \zeta )&= {{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}_1, \xi , \zeta ) \cup {{\,\mathrm{\mathfrak {U}}\,}}(\mathcal {T}_2, \xi , \zeta ) \end{aligned}$$