A Formally Verified, Optimized Monitor for Metric First-Order Dynamic Logic

Abstract

Runtime monitors for rich specification languages are sophisticated algorithms, especially when they are heavily optimized. To gain trust in them and safely explore the space of possible optimizations, it is important to verify the monitors themselves. We describe the development and correctness proof in Isabelle/HOL of a monitor for metric first-order dynamic logic. This monitor significantly extends previous work on formally verified monitors by supporting aggregations, regular expressions (the dynamic part), and optimizations including multi-way joins adopted from databases and a new sliding window algorithm.

1 Introduction

As the complexity of IT systems increases, so does the complexity and importance of their verification. Research in runtime verification (RV) has developed well-established formal techniques that can often be applied more easily than traditional formal methods such as model checking. RV is based on dynamic analysis, trading off completeness for efficiency. It is mechanized using monitors, which are algorithms that search sequences of events, either offline from log files or online, for patterns indicating faults.

Monitors must be trusted when they are used as verifiers. This trust can be justified by checking the monitors themselves for correctness [16, 17, 31, 36, 41, 42, 44, 45, 49]. Recently, a simplified version of the algorithm used in the MonPoly tool [8, 9] has been formalized and proved correct in Isabelle/HOL [45] (Sect. 2). MonPoly and its formal counterpart, called VeriMon, are both monitors for metric first-order temporal logic (MFOTL). However, VeriMon only supports a restricted fragment of this logic and lacks many optimizations that are necessary for an acceptable and competitive performance.

We present a formally verified monitor, VeriMon+, that substantially extends and improves VeriMon. VeriMon+ closes all expressiveness gaps between MonPoly and VeriMon. It supports aggregation operators like sum and average [7] similar to those found in database query languages, arbitrary negations of closed formulas, the unbounded ${\Circle }$ (Next) operator, and constraints involving terms (e.g., $P(x) \wedge y = x + 2$). Due to space limitations, our focus (Sect. 3) will be primarily on aggregations, our largest addition. Moreover, VeriMon+ exceeds MFOTL in expressiveness by featuring a significantly richer specification language, metric first-order dynamic logic (MFODL). To our knowledge, it is the first monitor for MFODL with past and bounded future operators (Sect. 4). This logic combines MFOTL with regular expressions, similar to linear dynamic logic [22] but enriched with metric constraints, aggregations, and first-order quantification.

We have also implemented and proved correct several new optimizations. First, to speed up the evaluation of conjunctions, we integrated an efficient algorithm for multi-way joins [38, 39], which we generalized to include anti-joins (Sect. 5). [2] Second, we developed a specialized sliding window algorithm to evaluate the Since and Until operators more efficiently (Sect. 6). VeriMon+ is executable via the generation of OCaml code from Isabelle. To this end, we augmented the code generation setup for IEEE floating point numbers in OCaml [50] with a linear ordering, which is needed for efficient set and mapping data structures.

The result of our efforts is both a verified monitor and a tool for evaluating unverified monitors. Since MFODL is extremely expressive, this gives us very wide scope. For example, we discovered previously unknown bugs in MonPoly via differential testing (Sect. 7), extending a previous case study [45]. As this experience suggests, and we firmly believe, formal verification is the most reliable way to obtain correct, optimized monitors.

In sum, our main contribution is a verified monitor for MFODL with aggregations, a highly expressive specification language that combines regular expressions and first-order temporal logic. Our monitor includes optimizations that are novel in the context of first-order monitoring. Our formalization is publicly available [20, 21].

Related Work. We refer to a recent book [4] for an introduction to runtime verification. The main families of specification languages in this domain are extensions of LTL [13, 26, 46], automata [3], stream expressions [19], and rule systems [23]. We combine two expressive temporal logics and their corresponding monitoring algorithms. MFOTL, implemented in MonPoly [7,8,9], supports first-order quantification over parametrized events, but it cannot express all regular patterns. Metric dynamic logic (MDL), implemented in Aerial [12], supports regular expressions, but it is not first-order. VeriMon+ is based on VeriMon [45], which only supports a fragment of MFOTL and is inefficient (Sect. 7). We refer to [45, Section 1] for an overview of related monitor formalizations. Relational database systems have been formalized by Malecha et al. [33] and by Benzaken et al. [15]. These works use binary joins only, which are not worst-case optimal.

Another efficient first-order monitor, DejaVu [25], supports past-only first-order temporal logic. It uses binary decision diagrams (BDDs) and does not restrict the use of negation, unlike MonPoly, which uses finite tables. DejaVu’s performance is incomparable to MonPoly’s and it is unclear whether multi-way joins can improve conjunctions of BDDs. Aerial and VeriMon+ evaluate regular expressions using derivatives [2, 18], which also have been used for timed regular expressions [47]. Quantified regular expressions [1, 34] extend regular expressions with data and aggregations. They can be evaluated efficiently, but can neither express metric constraints nor future modalities directly.

2 A Verified Monitor for Metric First-Order Temporal Logic

VeriMon [45] is a formally verified monitor for a large fragment of MFOTL [8]. The monitor takes an MFOTL formula, which may be open, and incrementally processes an infinite stream of time-stamped events. It outputs for every stream position the set of variable assignments that satisfy the formula. Thus, the monitor can be used to extract data from the stream. Typically, one is interested in the violations of a property specified as an MFOTL formula, which can be obtained by monitoring the negated formula.

We give an overview of MFOTL and VeriMon. We also cover some of the smaller additions in our new monitor, VeriMon+, highlighted in gray. For readability, we liberally use abbreviations and symbolic notation, departing mildly from Isabelle’s syntax.

Fig. 1.

Syntax and semantics of MFOTL as presented in [45], with additions in gray

Figure 1 shows MFOTL’s syntax and semantics. Events have a name () and a list of parameters of type . In VeriMon+, is a disjoint union of integers, double-precision floats, and strings. Multiple events are grouped together into a database () if they are considered to occur simultaneously. We call an infinite stream of databases, augmented with their corresponding time-stamps, an event stream or . Time-stamps () are modeled as natural numbers (). We write $\mathsf {T}\ \sigma \ i$ to denote the time-stamp of the ith database of the event stream $\sigma$. The predicate expresses that the time-stamps are monotone, i.e., $\mathsf {T}\ \sigma \ i \le \mathsf {T}\ \sigma \ (i+1)$ for all $i \ge 0$, and always eventually strictly increasing, i.e., $\forall t.\ \exists i.\ t < \mathsf {T}\ \sigma \ i$. Consecutive time-points i can have the same time-stamp.

Terms and formulas are represented by the datatypes and , respectively. Our formalization uses de Bruijn indices for free and bound variables (constructor ). In examples, we prefer the standard named syntax (and omit ). The type $\mathcal {I}$ models nonempty, possibly unbounded intervals over . We write $n \in _\mathcal {I}I$ for n’s membership in I, and [a, b] for the unique interval satisfying $n \in _\mathcal {I}[a,b]$ iff $a \le n \le b$. The right bound b is of type , i.e., either a natural number or infinity $\infty$ for an unbounded interval.

The functions and (Fig. 1) define MFOTL’s semantics. Both take a variable assignment v, which is a list of type whose ith element $v\,\mathbin {!}\, i$ is the value assigned to the variable with index i. The function evaluates terms under a given assignment. The expression is true iff the formula $\varphi$ is satisfied by v at time-point i in the trace $\sigma$. VeriMon+ adds arithmetic operators and type conversions to terms, as well as the predicates $\prec$ and $\preceq$. Their semantics on is lifted from the corresponding operations on integers, floats, and strings, whenever they are meaningful. The ordering $\le$ on is total: strings are compared lexicographically and .

VeriMon computes sets of satisfactions (i.e., satisfying assignments) by recursion over the formula’s structure. It represents these sets as finite tables, to which it applies standard relational operations such as the natural join ($\bowtie$) and union. Tables are sets of tuples, which are lists of optional values; missing values are denoted by $\bot$. This representation allows us to use tuples with the same length across subformulas with different free variables. The predicate defines the well-formed tuples for a given length n and a set of variables V. We also refer to V as the columns of a tuple (or table).

The set of satisfactions may be infinite. VeriMon supports only a fragment of MFOTL for which all computed tables are finite. The predicate (omitted) defines the monitorable fragment [45]. It accepts only certain combinations of operators and constrains the free variables of subformulas. Also, the intervals of all $\mathbin {\mathsf {U}}$ operators must be bounded.

Fig. 2.

Simplified state of a Since operator and its update

VeriMon’s interface consists of two functions and . The former initializes the monitor’s state, and the latter updates it with a new time-stamped database to report any new satisfactions. We require that satisfactions are reported for every time-point and in order. Note that a formula containing a future operator such as $\mathbin {\mathsf {U}}$ cannot necessarily be evaluated at time-point i after observing the ith database. Therefore, the output for several time-points may become available at once, so returns a list of pairs of time-points and tables.

We describe the evaluation of $\alpha \mathbin {\mathsf {S}}_{[a,\,b]} \beta$ in more detail. This formula is equivalent to the disjunction of $\alpha \mathbin {\mathsf {S}}_{[c,\,c]} \beta$ for all c such that $a \le c \le b$. Suppose that the most recent time-point is i with time-stamp $\tau$. The monitor’s state for $\alpha \mathbin {\mathsf {S}}_{[a,\,b]} \beta$ consists of a list of tables $T_c$ with the satisfactions of $\alpha \mathbin {\mathsf {S}}_{[c,\,c]} \beta$ at time-point i, along with the corresponding time-stamps $\tau - c$. VeriMon also stores the satisfactions $T_c$ (and time-stamps) for ${0 \le c < a}$, which are not yet in the interval. Figure 2 (left) depicts a state, where we assume for simplicity that we store a table for every time-stamp between $\tau - b$ and $\tau$. (In reality, time-stamps not in the trace do not have a corresponding entry in this list.) The state is updated for every new time-point with time-stamp $\tau '$, for which we already know the satisfactions $R_\alpha$ and $R_\beta$ of the subformulas $\alpha$ and $\beta$. In Fig. 2, we distinguish whether $\tau '$ equals $\tau$ (otherwise $\tau ' > \tau$ by monotonicity). The update consists of three steps: (1) remove tables that fall out of the interval; (2) evaluate the conjunction of each remaining table with $R_\alpha$ using a relational join; and (3) add the new tuples from $R_\beta$, either by inserting them into the most recent table $T_0$ or by adding a new table, depending on whether $\tau '$ equals $\tau$. Finally, we take the union of all tables within the interval to obtain the satisfactions of $\alpha \mathbin {\mathsf {S}}_{[a,b]} \beta$.

We summarize VeriMon’s correctness, which we also prove for VeriMon+. It relates the monitor’s implementation to its specification , which defines the expected output on a stream prefix. The first result shows that characterizes an MFOTL monitor, where means that $\pi$ is a prefix of $\sigma$, and converts v to an assignment by mapping $\bot$ to an unspecified value.[2]

Lemma 1

([45], Lemma 2). Suppose that is true. Then, is sound and eventually complete, i.e., for all prefixes $\pi$ of trace $\sigma$, time-points i, and tuples v,

• (a) , and

• (b) .

Above, is the smallest number larger than all free variables of $\varphi$, written . The next result establishes the implementation’s correctness using the state invariant (omitted). Let convert lists into sets, be the last time-stamp in $\pi$, and $\pi _1 \mathbin {@} \pi _2$ be the concatenation of $\pi _1$ and $\pi _2$.

Theorem 1

([45], Theorem 1). The initialization establishes the invariant and the update preserves the invariant and its output can be described in terms of :

• (a) If , then .

• (b) Let . If and , then and .

3 Aggregations

Basin et al. [7] extended MFOTL with a generic aggregation operator. This operator was inspired by the group-by clause and aggregation functions of SQL. It first partitions the satisfying assignments of its subformula into groups, and then computes a summary value, such as count, sum, or average, for each group. We formalized the aggregation operator’s semantics, added an evaluation algorithm to VeriMon+, and proved its correctness.

Consider the formula . The aggregation operator has four parameters: a result variable (s), the aggregation type (), an aggregation term (first x), and a list of variables that are bound by the operator and thus excluded from grouping (second x). When evaluated, the above formula yields a set of tuples (s, g). There is one such tuple for every value of g with at least one P event that has g’s value as its first parameter. The values of g partition the satisfactions of P(g, x) into groups. For every group, the sum over the values of x in that group is assigned to the variable s.

We added the constructor . Consider the instance $y \leftarrow \varOmega \;t;b.\ \varphi$. The operator binds b variables simultaneously in the formula $\varphi$ and in the term t, over which we aggregate. In examples, we list the bound variables explicitly instead of writing the number b. The remaining free variables (possibly none) of $\varphi$ are used for grouping. The variable y receives the result of the aggregation operation $\varOmega = (\omega ,d)$, where $\omega$ is one of (count), , , , (average), or (median). The default value d, which we usually omit, determines the result for empty groups (e.g., 0 for ). The formula’s free variables are those of $\varphi$ excluding the b bound variables, plus y. [3]

Figure 3 shows the semantics of the aggregation operator $y \leftarrow \varOmega \;t;b.\ \varphi$. The assignment v determines both a group and a candidate value $v \mathbin {!} y$ for the aggregation’s result on that group. The function checks whether the value is correct. First, it computes the set M, which encodes a multiset in the form of pairs (x, c), where c is x’s multiplicity. This multiset contains the values of the term t under all assignments $z \mathbin {@} v$ that satisfy $\varphi$, where z is an assignment to the bound variables. The expression stands for the cardinality of Z when it is finite, and $\infty$ otherwise. Then, compares $v \mathbin {!} y$ to the result of the aggregation operation $\varOmega$ on M, which is given by (omitted).[2]

We extended the predicate with sufficient conditions that describe when the aggregation formula $y \leftarrow \varOmega \;t;b.\ \varphi$ has finitely many satisfactions. We require that $\varphi$ satisfies , that the variable y is not free in $\varphi$ excluding the b bound variables, and that all bound variables and the variables in t occur free in $\varphi$.[2] We adopted the convention [7] that an aggregation formula is not satisfied when M is empty, unless all free variables of $\varphi$ are bound by the operator. Otherwise, there would be infinitely many groups (and hence, satisfactions) with the aggregate value , assuming that $\varphi$ is .

Fig. 3.

Semantics and evaluation of the aggregation operator

Figure 3 also defines , which evaluates the aggregation operator. It takes a table R with $\varphi$’s satisfactions, and returns a table with the aggregation operator’s satisfactions. The first argument n controls the length of the tuples in the tables (Sect. 2). The argument $g_0$ specifies whether all free variables of $\varphi$ are bound by the operator. The remaining arguments y, $\varOmega$, b, and t are those of the operator. We write $f \mathbin {\text {`}}X$ for the image of X under f.

In , we first check whether $g_0 \wedge R = \{\}$ is true to handle the special case mentioned above. (The expression is a table with a single tuple of length n that assigns a to variable y.) Otherwise, we compute the aggregate value separately for each group k. The set of groups is obtained by discarding the first b values of each tuple in R. To every group k, we apply the lambda-term to augment the tuple with the aggregate value. The set G contains all tuples in the group. Note that these tuples extend k with assignments to the b bound variables. Then, we compute the image of G under the term t, which is evaluated by (omitted). Finally, we obtain the multiset M by counting how many tuples in G map to each value in the image.

4 Regular Expressions

VeriMon+ extends VeriMon’s language by generalizing MFOTL’s temporal operators to regular expressions. The resulting metric first-order dynamic logic (MFODL) can be seen [24, §3.16] as the “supremum” (in the sense of combining features) of metric dynamic logic (MDL) [12] and MFOTL [8]. Peycheva’s master’s thesis [40] develops a monitor for past-only MFODL. We give the first formal definition of MFODL with past and future operators. We also define a fragment whose formulas can be evaluated using finite relations (Sect. 4.1). This fragment guides our evaluation algorithm’s design (Sect. 4.2).

Fig. 4.

Syntax and semantics of MFODL (left) and conversion of MFOTL into MFODL (right)

Figure 4 (left) defines the syntax and semantics of our variant of regular expressions. The type is parametrized by a type variable , which is used in the ${\_}?$ constructor. The semantics is given by and assigns to each expression a binary relation ($\otimes$) on natural numbers. Intuitively, a pair $(i,\,j)$ is in the relation assigned to r when r matches the portion of a trace from i to j. The trace notion is abstracted away in via the argument , which indicates whether a parameter of type may advance past a given point.

In more detail, the wildcard operator matches all pairs $(i,\,j)$, where $j = i + k$; we write $\star$ for the useful special case . The test ${x}?$ only matches pairs of the form $(i,\, i)$ that pass . The semantics of alternation ($+$) as union ($\cup$), concatenation ($\,\cdot \,$) as relation composition ($\bullet$), and Kleene star ($\_^*$) as reflexive-transitive closure ($\_^*$) is standard.

Figure 4 (left) also shows ’s extension with two constructors that use regular expressions. The regular expression’s parameter nests a recursive occurrence of , i.e., our regular expressions’ leaves are formulas, which in turn may further nest regular expressions, and so on. MDL’s syntax is often presented as a mutually recursive datatype [12]. Our nested formulation is beneficial because it lets us formalize regular expressions independently, for use in different applications (e.g., monitors for MDL and MFODL).

In terms of their semantics, the two new operators naturally generalize the $\mathbin {\mathsf {S}}_I$ and $\mathbin {\mathsf {U}}_I$ operators. The past match operator  is satisfied at i if there is an earlier time-point j subject to the same temporal constraint I as in the satisfaction of $\mathbin {\mathsf {S}}_I$ and moreover the regular expression r matches from j to i. For the future match operator  , the situation is symmetric with the existentially quantified j being a future time-point. In both cases, the parameter of is recursively instantiated with the satisfaction predicate .

We can embed MFOTL into MFODL by expressing the temporal operators using semantically equivalent formulas built from regular expressions (Fig. 4, right). Thus, we could in principle remove the operators ${\CIRCLE }$, $\mathbin {\mathsf {S}}$, ${\Circle }$, and $\mathbin {\mathsf {U}}$ from and use regular expressions instead. We prefer to keep these operators in as this allows us to optimize their evaluation in a way that is not available for the more general match operators (Sect. 6).

We conclude MFODL’s introduction with an example. Many systems for user authentication follow a policy like: “A user should not be able to authenticate after entering the wrong password three times in a row within the last 10 minutes.” We write for the event “User u entered the wrong password” and for “User u has successfully authenticated.” Additionally, we abbreviate ${\varphi }? \cdot \star$ by $\varphi$. (This abbreviation is only used when $\varphi$ appears in a regular expression position, e.g., as an argument of $\,\cdot \,\!$). Then the formula

expresses this policy’s violations: its satisfying assignments are precisely the users that successfully authenticate after entering wrong credentials for three times in the last 600 seconds, without intermediate successful authentications. We can express this property in MFOTL using three nested $\mathbin {\mathsf {S}}$ operators, one for each of the subformulas. [2] Yet, it is unclear which intervals to put as arguments to $\mathbin {\mathsf {S}}$ beyond the fact that they should sum up to 600. The rather impractical solution exploits that there are only finitely many ways to split the intervals due to their bounds being natural numbers and constructs the disjunction of all possible splits ($180\,901$ in this case). MFODL remediates this infeasible construction.

Fig. 5.

Safety conditions for MFODL

Fig. 6.

Simplified state of a past match operator and its update

4.1 Finitely Evaluable Regular Expressions

Following MonPoly’s design [8], VeriMon+ represents all sets of satisfying assignments with finite tables. The databases occurring in the trace are all finite, yet their combination may not be. Therefore, MonPoly and VeriMon+ work with syntactic restrictions that ensure that all sets that arise are finite. For example, negation must occur under a conjunction $\alpha \wedge \lnot \beta$, where the free variables of $\beta$, written , are contained in those of $\alpha$. We say that $\lnot \beta$ is guarded by $\alpha$ and compute $\alpha \wedge \lnot \beta$ as the anti-join ($\triangleright$) of the corresponding tables. For disjunctions $\alpha \vee \beta$, we must have . Similar restrictions also apply to temporal operators: to evaluate $\alpha \mathbin {\mathsf {S}}_I \beta$ and $\alpha \mathbin {\mathsf {U}}_I \beta$ we require .

We derive a new sufficient criterion for match operators to have finitely many satisfying assignments. To develop some intuition, we first consider several examples that result in infinite tables. The first example is any expression with a Kleene star as the topmost operator. The formula is satisfied at all points i for all assignments v (regardless of r’s free variables) since $0 \in _\mathcal {I}I$ and any (i, i) matches $r^*$. Thus, when we evaluate $\varphi$ at i, we can choose i as the witness for the existential quantifier in the definition of . It follows that Kleene stars must be guarded by a finite table.

The union of two finite tables is finite only if the tables have the same columns (assuming an infinite domain ). This explains the requirement for the subformulas of $\vee$ to have the same variables, [2] but a similar requirement is needed for the $+$ of regular expressions. Perhaps more surprisingly, concatenation can also hide a union: consider and assume that s matches (j, i) for some $j < i$. By the semantics of concatenation, we can split the satisfactions of $\varphi$ into those that use $s^*$’s matching pair (i, i) (i.e., the satisfying assignments of  at i) and those that do not. To combine these assignments it seems necessary to take the union of the satisfaction of $\varphi$ and  , which in turn requires these formulas to have the same free variables, or equivalently (overloading notation to apply to regular expression). The future match operator behaves symmetrically, requiring the side condition for .

MonPoly also allows the left subformula of $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$ to be negated: $(\lnot \alpha )\mathbin {\mathsf {S}}_I \beta$ and $(\lnot \alpha )\mathbin {\mathsf {U}}_I \beta$. Hence, we should support the MFODL variants  and , but also generalize these patterns to flexibly support negated tests.

Our solution to these issues comprises the predicates shown in Fig. 5. The predicate on regular expressions is parametrized by two flags: distinguishing whether the expression occurs under a past or a future match operator and determining whether the tests may be negated and other safety conditions relaxed. The most interesting cases are those for concatenation. There, in addition to the side conditions, only one argument is checked recursively in the same mode as the overall expression. The other argument is checked using the mode, in which side conditions are skipped, except for the requirement that (possibly negated) formulas under the test operators are safe. The context parameter dictates which argument keeps, and which changes, the mode.

4.2 Evaluation Algorithm

The evaluation algorithm’s structure for the past match operator (Fig. 6) closely resembles the evaluation of $\alpha \mathbin {\mathsf {S}}_I \beta$ (Fig. 2). What is different is the data that is stored for each time-stamp and the way we update it. For $\mathbin {\mathsf {S}}$, each stored table $T_c$ corresponds to the satisfactions of $\alpha \mathbin {\mathsf {S}}_{[c,\,c]} \beta$. For , each $T_c$ is a mapping from a regular expression s to the table denoting the satisfactions of . (We represent mappings here by plain functions for readability.) Clearly, this mapping’s domain must be finite. We restrict it to the finite set $\varDelta (r)$ of right partial derivatives [2, 12] of the overall regular expression r, which correspond to the states of a non-deterministic automaton that matches r from right to left.

Fig. 7.

The core evaluation functions for MFODL

Partial derivatives allow us to extend satisfactions of for $s \in \varDelta (r)$ at time-point i to satisfactions of for $s \in \varDelta (r)$ at time-point $i+1$. The Since operator’s counterpart of this extension is the join with $R_\alpha$, the new satisfactions of $\alpha$, which is performed for all $T_c$s for every update. Here, the extension function $\delta _R$ inputs a function R assigning the new satisfactions for all tests occurring in r (possibly with a negation stripped) and updates the mapping $T_c$. It is defined as where $\delta$ is defined recursively on the structure of regular expressions as shown in Fig. 7. The first parameter of $\delta$ uses continuation passing style. It builds up a regular expression context that we use when evaluating the leaves. It is thus guaranteed that if we apply $\delta$ to any regular expression $s \in \varDelta (r)$, all calls to T will apply T to some $s' \in \varDelta (r)$.

The function $\delta$ uses the recursive function in its definition. This function computes the assignments that give rise to matches of the form (i, i) under the assumption that a guard (in form of the table X) is given. For $\delta$, the recursive call acts as ’s guard.

The function is used to update the state with satisfying assignments at the newly added time-point (Fig. 6). It is only specified for expressions satisfying and uses for subexpressions that only satisfy . The recursive structure of and follows the one of . We write and use $\cup$ to denote the pointwise union of mappings in Fig. 6. The Since operator’s counterpart of this update is the addition of the satisfactions for the subformula $\beta$ (Fig. 2).

The above description just sketches our evaluation algorithm and our formalization provides full details. Our proofs establish the monitor’s overall correctness, which amounts to the same statement as Theorem 1 but now covers the syntax and semantics extended with the match operators (and aggregations). In particular, the formalization also includes the future match operators for which the evaluation uses similar ideas (partial derivatives), but in a symmetric fashion following the definition of .

5 Multi-way Join

The natural join $\bowtie$ is a central operation in first-order monitors. Not only is it used to evaluate conjunctions; temporal operators also crucially rely on it. Despite this operation’s importance, both MonPoly and VeriMon [2] naively compute $A\bowtie B$ as nested unions: , where joins two tuples v and w if possible, and $\lceil \_\rceil$ converts the optional result into a set. In this section, we describe a recent development from database theory that we formalize and extend to optimize the computation of joins.

Fig. 8.

Multi-way join algorithm

Ngo et al. [37] and Veldhuizen [48] have developed worst-case optimal multi-way join algorithms that compute the natural join of multiple tables. Here, optimality means that the algorithm never constructs an intermediate result that is larger than the maximum size of all input tables and the overall output. This strictly improves over any evaluation plan using binary joins: There are tables A, B, and C such that the size of $A \bowtie B \bowtie C$ is linear in $|A| = |B| = |C|$, but any plan constructs a quadratic intermediate result from the binary join it evaluates first [39, Fig. 2]. The key idea of the multi-way join is to build the result table column-wise, adding one or more columns at a time, while taking all tables that refer to the currently added columns into account. All intermediate results are restrictions of the overall result to the processed columns, and thus not larger than the overall result.

Figure 8 shows our formalization of the multi-way join algorithm following Ngo et al.’s unified presentation [39] but generalizing it to support anti-joins $\triangleright$; these additions are highlighted in gray. A is a set of s, i.e., tables annotated with the columns (represented by ) they have. The main function, , takes as input a set of columns V and two queries and . It computes the multi-way join of while subtracting the tuples of tables in . For example, computes $A \bowtie B \bowtie C \triangleright D \triangleright E$.

The algorithm proceeds by recursion on V. The base case in which V is empty or a singleton set is evaluated directly using intersections and unions. We first describe the recursive structure of the original algorithm [39], obtained by ignoring the highlighted anti-join additions in Fig. 8. The algorithm is parametrized by the function, which partitions V into two nonempty sets I and J that each determine the number of columns and the order in which they are added. Ngo et al. [39] show how different multi-way join algorithms [37, 48] can be obtained by using specific instances of . We use a heuristic to pick first the column i that maximizes the number of tuples in it affects (by setting $I = \{i\}$). The partitioning only affects performance, not correctness.

Once I and J are fixed, the algorithm constructs a reduced query by focusing on tables that have a column in I. Furthermore, it restricts their columns to I via the overloaded notation $\_ \downarrow I$, which denotes the restriction of tuples (by setting the optional values for columns outside I to $\bot$ [45]), annotated tables, and queries (Fig. 8).

Next, is evaluated recursively, yielding table $A^I$ with columns I. We now consider tables that have a column in J. This yields a second reduced query , which is, however, not restricted to J. Keeping the columns I in allows us to focus on tuples in that match some $t \in A^I$, i.e., coincide with t for all values in columns I. The function performs this matching. For each tuple $t \in A^I$, it creates the query consisting of tables from restricted to t-matching tuples (in database terminology this is a semi-join) further restricted to columns J. These queries are again solved recursively, each resulting in a table $A_t$ with columns J. The final step consists of merging the tuples t with $A_t$. Since t and A have disjoint columns I and J, the function call will return some result (which we extract via ) for all $v \in A$.

We extend the algorithm to support anti-joins by introducing a second query , which we think of as being negated. It is not possible to split ’s tables column-wise. Instead, our generalization processes tables with columns U from once the positive query has accumulated a superset of U as its columns. This is an improvement over the naive strategy of computing first and only then removing tuples from it.

The correctness of relies on several side conditions, e.g., no input table may have zero columns and V must be the union of the columns in the positive query. A wrapper function takes care of these corner cases, e.g., by computing V from and . We omit ’s straightforward definition, but show its correctness property (which only differs from ’s correctness by having fewer assumptions):

In words: whenever is nonempty and all tables in and fit their declared columns, a tuple z belongs to the output of iff it has the correct columns and matches all positive tables from and does not match any negative ones from .

The multi-way join algorithm is integrated in VeriMon+ by adding a new constructor to the formula datatype. At least one of the subformulas of must be non-negated, and the columns of the negative subformulas must be a subset of the positive ones. Since MonPoly’s parser, which we reuse in VeriMon+, generates formulas with binary conjunctions, we have defined a semantics-preserving preprocessing function (omitted), which rewrites nested binary conjunctions into .

6 Sliding Window Algorithm

To evaluate the temporal operators $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$, VeriMon computes the union of tables that are associated with time-stamps within the operator’s interval. These sets of time-stamps often overlap between consecutive monitor steps. The sliding window algorithm (SWA) [10] is an efficient algorithm for combining the elements of overlapping sequences with an associative operator. It improves over the naive approach that recomputes the combination (here, the union) from scratch for every sequence. MonPoly uses SWA for the special cases and , where . However, SWA was not designed for the evaluation of arbitrary $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$ operators. For these, the tables in the sequence must be joined with the left subformula’s results in every monitor step. In a separate work [27, 28], we formally verified SWA’s functional correctness (but not its optimality) and extended it with a join operation to support arbitrary $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$ operators.

SWA is overly general: it supports any associative operator, not just the union of tables. We conjecture that the generic SWA algorithm is not optimal in the special case needed for $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$. To optimize the evaluation of the $\mathbin {\mathsf {S}}$ and $\mathbin {\mathsf {U}}$ operators in VeriMon+, we abstracted the individual steps of their evaluation in one locale for each of them (Sect. 6.1). We then instantiated the locales with specialized sliding window algorithms (Sect. 6.2). Due to space limitations, we only describe the optimization for the Since operator here.

6.1 Integration into the Monitor

Recall the evaluation of Since in VeriMon (Sect. 2). First, VeriMon updates the operator’s state with a new time-stamp $\tau '$ and the satisfactions $R_\alpha$ and $R_\beta$ for the subformulas $\alpha$ and $\beta$. Second, it evaluates the state to obtain the satisfactions for $\alpha \mathbin {\mathsf {S}}_I \beta$.

Let denote the type of the $\mathbin {\mathsf {S}}$ operator’s state in VeriMon. In VeriMon+, we define a locale that abstracts the update and evaluation of an optimized state and relates the optimized state[2] to VeriMon’s original state (Fig. 9). We provide additional constant arguments for evaluating the $\mathbin {\mathsf {S}}$ operator in a record . It consists of the $\mathbin {\mathsf {S}}$ operator’s interval, the arguments to characterizing the satisfactions of the two subformulas, and a Boolean value denoting whether the left subformula occurs negated. The predicate relates an optimized state to VeriMon’s state with respect to the given and a current time-stamp. The function returns an initial optimized state. The next three functions , , and correspond to the three steps in which VeriMon’s state is updated (Sect. 2), except that now they act on the optimized state. Finally, evaluates the optimized state to obtain the satisfactions of the $\mathbin {\mathsf {S}}$ operator at the current time-point. The omitted locale assumptions state that all operations preserve and that returns the union computed on any VeriMon state related by .

Fig. 9.

The locale for evaluating the Since operator (assumptions omitted)

6.2 The Specialized Algorithm

VeriMon’s state for the $\mathbin {\mathsf {S}}$ operator consists of a list of tables $T_c$ with the satisfactions of formulas $\alpha \mathbin {\mathsf {S}}_{[c,\,c]} \beta$, along with the corresponding time-stamps. VeriMon stores the satisfactions $T_c$ (and time-stamps) for all c that do not exceed the interval’s upper bound.

In our optimized state, we partition the list of tables $T_c$ into a list for time-stamps that are not yet in the interval and a list for time-stamps that are already in the interval. The state also contains a mapping that assigns to each tuple occurring in some table $T_c$ in the interval the latest time-stamp in the interval for which this tuple occurs in the respective table. Finally, the state contains a mapping that assigns to each tuple occurring in some table $T_c$ in the entire state the earliest time-stamp for which this tuple occurs in the respective table. (For efficiency, we delete tuples from lazily, i.e., only at defined garbage collection points, such that the mapping may even contain tuples from some $T_c$ that already has fallen out of the interval.)

The state is initialized via to consist of empty lists and empty mappings. The function drops tables from that fall out of the interval based on a newly received time-stamp. It also removes those tuples from whose latest occurrence (which is stored in this mapping) has fallen out of the interval. Then it moves tables that newly enter the interval from to , and updates the tuples from these moved tables in to the most recent time-stamp $\tau$ for which they now occur in the interval, but only if maps the tuple to a time-stamp that is at most $\tau$.

The function only modifies the mappings and by removing tuples that are not matched by any tuple in the given table $R_\alpha$. The function appends the new table $R_\beta$ to (or directly , if $0\in _\mathcal {I}I$), adds the tuples from $R_\beta$ that were not in to that mapping, and, if ${0\in _\mathcal {I}I}$, updates the tuples from $R_\beta$ in the mapping to the current time-stamp. Finally, returns the keys of the mapping , in particular without computing any unions. In other words, contains precisely the tuples that are in the interval and have not been removed by joins. Crucially, and unlike in VeriMon’s state, the join operation does not change the tables $T_c$ in our optimized state. This functionality is implemented more efficiently by filtering the two mappings and .

Fig. 10.

An example of updating the optimized state for the formula $P(x)\mathbin {\mathsf {S}}_{[2,4]}Q(x)$

Example

Figure 10 shows how the optimized state for the formula $P(x)\mathbin {\mathsf {S}}_{[2,4]}Q(x)$ is updated. In total, four time-points are processed. The first two columns show the time-stamp and database for each time-point. The other columns show the state after applying the step named in the third column. Each step corresponds to a function in the locale. The satisfactions $\{\}, \{\}, \{b, c\}, \{a\}$ returned by can be read off from the mapping after each time-point’s last step. We omit steps that do not change the state.

The first row shows the initial state. For the first time-point, the steps with time-stamp 1 and with the table $\{\}$ (as there are no P events) do not change the initial state. Then, appends the table $\{a, b, c\}$ with the parameters of the Q events to (as $0 \not \in _\mathcal {I}[2,4]$) and adds its elements to .

For the second time-point, VeriMon+ applies with time-stamp 2. Again this step has no effect: ’s first entry is not moved to as the difference $2-1$ to the current time-stamp is not in [2, 4]. Next, with the table $\{b, c\}$ (from the P events) removes a from , but not from . Finally, appends the table $\{\}$ (as there are no Q events) to .

For the third time-point, moves ’s first entry to because the time-stamp difference $3-1$ is in [2, 4]. The values b, c of that entry are added to because maps them to a time-stamp $\le 1$. Note that a is not added, as it is not contained in . The step with the table $\{b, c\}$ does not change the state. The step appends $\{a, b\}$ to . Now, a is added to , whereas b is already contained in and its value is not updated.

When the fourth time-point is processed, the first two observed time-stamps fall out of the interval and discards their entries from and , and their values from but not from . As before, the last table $\{a, b\}$ in is moved to and its elements are added to . As time has progressed by more than the upper bound of the interval [2, 4], triggers garbage collection, which removes the key c from . The join operation further removes b from and . Finally, appends $\{\}$ to .

7 Evaluation

We perform two kinds of experiments. First, we carry out differential testing [35] of VeriMon+ against three (unverified) state-of-the-art monitors: MonPoly [9], Aerial [11], and Hydra [43]. Second, we compare VeriMon+’s performance to these monitors on representative formulas. VeriMon+ reuses MonPoly’s log and formulas parsers and user interface. The verified monitor’s code extracted from Isabelle is integrated with these unverified components in about 170 lines of unverified OCaml code. Our implementation and our experiments are available [5]. Of the above monitors, only VeriMon+ supports full MFODL. MonPoly supports a monitorable fragment of MFOTL with bounded future operators and aggregations. Aerial and Hydra support the propositional fragment of MFODL.

Differential Testing. To validate the results produced by unverified monitors, we generate random stream prefixes and formulas, invoke the monitors, and compare their results to VeriMon+’s. For this purpose, we developed a random stream and formula generator. It takes as parameters the formula size (in terms of number of operators) and the number of free variables that occur in the formula. The generator can be configured to generate formulas within the fragments of MFODL supported by the different monitors we evaluate.

Our tests uncovered several classes of inputs where MonPoly’s output deviated from VeriMon+’s. Here, we show one example and refer to our extended report [6] for a comprehensive overview. Namely, formulas of the form $m \leftarrow \varOmega \;x;x. \ {\blacklozenge }_{I} \alpha$, where , , and $0\notin _\mathcal {I}I$, were evaluated in MonPoly using a specialized algorithm, which incorrectly updated the satisfactions of $\alpha$ when they fell out of the interval I.

Aerial’s and Hydra’s output mostly coincided with VeriMon+’s. However, we noticed that Hydra’s output is not as eager as it could be at the end of the stream prefix. For example, is satisfied at time-point 0 of the prefix $(\{\},0), (\{\},1)$ due to the existence of time-point 1, where can be evaluated. The subformula cannot be evaluated at time-point 1. This prevents Hydra from outputting this verdict at 0.

Performance Evaluation. To assess VeriMon+’s performance, we selected four formulas, shown in Fig. 11, which exercise the optimizations (multi-way join and sliding window) and the language features (aggregations and regular expressions) we have introduced. The formula is derived from the star conjunctive query, commonly used as a benchmark for joins [14]. We use it to evaluate our multi-way join (for $N=10$) and sliding window (for $N=30$) implementations. The formula is a commonly used aggregation query, which computes the most frequently occurring value of the event P’s second parameter. Finally, checks if events P and Q alternate over the last 10 time units.

We generate random stream prefixes with a time span of 60 time units containing events P, Q, and R, [2] each with two integer parameters sampled uniformly at random from the set $\{1,2,\ldots ,10^9\}$. Our stream generator is parametrized by the event rate (i.e., by the number of events with the same time-stamp). Since VeriMon+ reuses MonPoly’s formula and log parsing infrastructure, there is an additional (conceptually unnecessary) overhead caused by converting the data structures to match the appropriate interfaces. In cases where the monitoring task is easy, this becomes the bottleneck and MonPoly performs better than VeriMon+. To make the monitoring task difficult for , we sample the value of the first parameter of each event (the common variable x) using the Zipf distribution. Thus, some parameter values occur frequently. This results in large intermediate tables, which are problematic for binary joins.

Figure 11 shows that VeriMon+ outperforms MonPoly on the formulas. The results confirm the feasibility of monitoring aggregations and regular expressions with VeriMon+. Specialized algorithms remain more performant on problems in their domain.

Fig. 11.

Time (s)/memory (MB) usage of the monitors ( , )

8 Conclusion

We have presented a verified monitor, competitive with the state-of-the-art, for the expressive specification language metric first-order dynamic logic. Our formalization comprises roughly $15\,000$ lines of Isabelle code, distributed over the four features we presented: regular expressions ($2\,000$), terms and aggregations (750), multi-way join ($3\,300$), and the sliding window algorithm ($3\,000$). Isabelle extracts a $7\,500$ line OCaml program from our formalization. This code includes efficient libraries representing sets and mappings via red–black trees introduced transparently into the formalization via the Containers framework [32]. We also use and extend a formalization of IEEE floating point numbers [50].

We have made additional contributions from the algorithmic perspective. Our monitor is the first monitoring algorithm for MFODL with aggregations. Moreover, our specialized sliding window algorithm improves over the existing generic algorithm [10]. Our usage of multi-way joins in the context of first-order monitoring is also novel, as is our extension of the multi-way join algorithm to handle anti-joins. It would be interesting to investigate the optimality of this extension and further consider a multi-way-like evaluation of an arbitrary Boolean combination of finite tables.

Our focus was on extending the verified monitor’s specification language and improving its algorithms. As next steps, we plan to further improve performance by refining our algorithms to imperative data structures following Lammich’s methodology [29, 30].