Optimization-Based Model Checking and Trace Synthesis for Complex STL Specifications

Abstract

Techniques of light-weight formal methods, such as monitoring and falsification, are attracting attention for quality assurance of cyber-physical systems. The techniques require formal specs, however, and writing right specs is still a practical challenge. Commonly one relies on trace synthesis—i.e. automatic generation of a signal that satisfies a given spec—to examine the meaning of a spec. In this work, motivated by 1) complex STL specs from an automotive safety standard and 2) the struggle of existing tools in their trace synthesis, we introduce a novel trace synthesis algorithm for STL specs. It combines the use of MILP (inspired by works on controller synthesis) and a variable-interval encoding of STL semantics (previously studied for SMT-based STL model checking). The algorithm solves model checking, too, as the dual of trace synthesis. Our experiments show that only ours has realistic performance needed for the interactive examination of STL specs by trace synthesis.

The authors are supported by ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), the START Grant No. JPMJST2213, the ASPIRE grant No. JPMJAP2301, JST. S.S. is supported by KAKENHI No. 23KJ1011, JSPS. Z.Z. is supported by JSPS KAKENHI Grant No. JP23K16865 and No. JP23H03372.

1 Introduction

Safety and quality assurance of cyber-physical systems (CPSs) is an important and multifaceted problem. The pervasiveness and safety-critical nature of CPSs makes the problem imminent and pressing; at the same time, the problem comes with very different flavors in different application domains, calling for different solutions. For example, in the aerospace domain, full formal verification all the way up from the codebase seems feasible [33]. Such is a luxury that the automotive domain may not afford, however, because of short product cycles, dependence on third-party (thus black-box) components, heterogeneous environmental uncertainties, and fierce competition (thus tight budget).

The above limitations in the automotive domain point, in the formal methods terms, to the absence of white-box system models. This has led to the flourish of light-weight formal methods, such as monitoring [8], runtime verification, and hybrid system falsification [16]. These are logic-based methods that operate on formal specifications, often given in signal temporal logic (STL) [24]. These methods give up comprehensive guarantee due to the absence of white-box system models; yet their values in practical usage scenarios are widely acknowledged.

Trace Synthesis and Model Checking. In this paper, we are motivated by some automotive instances of the trace synthesis problem: it asks to synthesize an execution trace $\sigma$ of a system $\mathcal {M}$ that satisfies a given STL specification $\varphi$. There are two major approaches to trace synthesis for CPSs.

One common approach is via hybrid system falsification [16]: here, we try many input signals $\tau$ for $\mathcal {M}$, iteratively modifying them in the direction of satisfying $\varphi$; the quantitative robust semantics of STL [17] serves as an objective function that allows hill-climbing optimization. It is notable that the system model $\mathcal {M}$ can be black-box: we do not need to know its internal working; it is enough to compute the execution trace $\mathcal {M}(\tau )$ under given input $\tau$. Falsification has attracted a lot of interest especially in the automotive domain; see e.g. [16].

We take the other approach to trace synthesis, namely as the dual of the model checking problem. Here model checking decides if, under any input $\tau$, the execution trace $\mathcal {M}(\tau )$ satisfies $\varphi$. Our choice of this approach may be puzzling—it requires a white-box model $\mathcal {M}$, but it is rare in the automotive domain.

Analyzing Specifications (Rather Than Models). Our choice of the model checking approach to trace synthesis comes from the following basic scope of the paper: we use trace synthesis to analyze the quality of specifications (specs).

This is in stark contrast with many falsification tools whose scope is analyzing models. There, a model $\mathcal {M}$ is extensive and complex (typically a Simulink model of an actual product), and counterexample traces are used for “debugging” $\mathcal {M}$.

In this paper, instead, a model $\mathcal {M}$ is simple and white-box (it can even be the trivial model, where the input and output are the same), but a spec $\varphi$ tends to be complex. One typical usage scenario for our framework is when $\varphi$ is a normative rule—such as a law, a traffic rule, or a property required in an international standard—in which case $\varphi$ is imposed on many different systems (e.g. different vehicle models). Then $\mathcal {M}$ should be a simple overapproximation of a variety of systems, rather than a detailed system model.

Another typical usage scenario of our framework is an early “requirement development” phase of the V-model of the automotive system design. Here, engineers fix specs that pin down the later development efforts, in which those specs get refined and realized. They want to confirm that the specs are sensible (e.g. there is no mutual conflict) and faithful to their intentions. Since a system is yet to be developed, a system model $\mathcal {M}$ cannot be detailed.

Fig. 1.

Rear-end near collision

Motivating Example. More specifically, the current work is motivated by the work [30] on formalizing disturbance scenarios in the ISO 34502 standard for automated driving vehicles. There, a vehicle dynamics model is simple (the scenarios should apply to different vehicle models—see above), but STL formulas are complex. It is observed that existing algorithms have a hard time handling the complexity of specs (see §6 for experiments). This motivated our current technical development, namely a trace synthesis algorithm that exploits white-box models and MILP optimization for efficiency.

The following example illustrates the challenge encountered in [30].

Example 1.1

(rear-end near collision). We would like to express, in STL, a rear-end near collision scenario for two cars. It refers to those driving situations where a rear car $\textrm{Car}_{\textrm{r}}$ comes too close to a front car $\textrm{Car}_{\textrm{f}}$. We assume a single-lane setting (Fig. 1), so we can ignore lateral dynamics.

Consider the following STL formulas. Here, $x_{\textrm{f}}, v_{\textrm{f}}, a_{\textrm{f}}$ are the variables for the position, velocity, and acceleration of $\textrm{Car}_{\textrm{f}}$; the other variables are for $\textrm{Car}_{\textrm{r}}$.

$$\begin{aligned} \begin{array}{rl} \texttt{danger} &{} \quad :\equiv \quad x_{\textrm{f}} - x_{\textrm{r}} \le 10 \\ \mathtt {dyn\_{inv}} &{} \quad :\equiv \quad x_{\textrm{f}} - x_{\textrm{r}} \ge 0\,\wedge \, 2 \le v_{\textrm{f}} \le 27 \,\wedge \, 2 \le v_{\textrm{r}} \le 27 \\ \texttt{trimming} &{} \quad :\equiv \quad (\Diamond \texttt{danger}) \Rightarrow \bigl ((\Box _{[0, 0.2]} a_{\textrm{r}} \ge 0.5) \mathbin {\mathcal {U}} \texttt{danger}\bigr ) \\ \texttt{RNC1} &{} \quad :\equiv \quad \Box (\mathtt {dyn\_inv}\wedge \texttt{trimming}) \wedge \Diamond _{[0, 9]}\Box _{[0,1]}\texttt{danger} \end{array} \end{aligned}$$

(1)

The last formula $\texttt{RNC1}$ formalizes rear-end near collision; in particular, its subformula $\Diamond _{[0, 9]}\Box _{[0,1]}\texttt{danger}$ requires that $\texttt{danger}$ occurs within 9 s and persists for at least one second.

The formula $\texttt{RNC1}$ comes with two auxiliary conditions: $\mathtt {dyn\_inv}$ and $\texttt{trimming}$. We shall now exhibit their content and why they are needed. In fact, these conditions arose naturally in the course of trace synthesis, the problem of our focus.

Specifically, in [30], we conducted trace synthesis repeatedly in order to 1) illustrate the meaning of STL specifications and 2) confirm that they reflect informal intentions. The generated traces were animated for graphical illustration. This workflow is much like in the tool STLInspector [31].

The formula $\mathtt {dyn\_inv}$ imposes basic constraints on the dynamics of the cars. In the trace synthesis in [30], without this basic constraint, we obtained a number of nonsensical example traces in which a car warps and instantly passes the other, drives much faster than the legal maximum, and so on.

The formula $\texttt{trimming}$ requires $\textrm{Car}_{\textrm{r}}$ to accelerate until $\texttt{danger}$ occurs. It was added to limit a generated trace to an interesting part. For example, a trace can have $\texttt{danger}$ only after a 8-s pacific journey; animating this whole trace can easily bore users. The condition trims such a trace to the part where $\textrm{Car}_{\textrm{r}}$ is accelerating towards $\texttt{danger}$.

The dynamics model used in [30] is the following simple one:

$$\begin{aligned} \dot{x_{\textrm{f}}}=v_{\textrm{f}}, \; \dot{v_{\textrm{f}}}=a_{\textrm{f}}; \qquad \dot{x_{\textrm{r}}}=v_{\textrm{r}}, \; \dot{v_{\textrm{r}}}=a_{\textrm{r}}. \; \end{aligned}$$

(2)

This relates x, v and a in the spec (1). The double integrator model is certainly simplistic, but it suffices the purpose in [30] of illustrating and confirming specs.

Remark 1.2

In [30], after illustrating and confirming STL specs through trace synthesis, the final goal was to use them for monitoring actual driving data. Neither the dynamics model (2) nor the condition $\mathtt {dyn\_inv}$ is really relevant to monitoring—actual driving data should comply with them anyway. In contrast, $\texttt{trimming}$ is important, in order to extract only relevant parts of the data.

Technical Solution: MILP-Based Trace Synthesis. We present a novel trace synthesis algorithm. Note that it also solves the dual problem, namely STL model checking. It originates from two recent lines of work: MILP-based optimal control [14, 28, 29] and SMT-based STL model checking [7, 23, 34].

The controller synthesis techniques in [14, 28, 29] exploit mixed-integer linear programming (MILP) for efficiency. The optimal control problem that they solve can be specialized to our trace synthesis problem (detailed discussions come later). But we found their capability of handling complex specs (as in Ex. 1.1) limited, largely because of their constant-interval encoding to MILP.

We solve this challenge by our novel variable-interval encoding of the STL semantics to MILP. It is inspired by the stable partitioning technique introduced in [7]: the technique is used in [7, 23, 34] for logical encoding towards SMT-based model checking; we use it for numerical encoding to MILP. This way we will solve the bounded trace synthesis problem—in the sense that variability of the truth values of the relevant formulas is bounded—much like in [7, 23, 34]. For our MILP encoding, however, we need special care since MILP does not accommodate strict inequalities (partitions such as $\dotsc , (\gamma _{i-1},\gamma _{i}),\{\gamma _{i}\}, (\gamma _{i},\gamma _{i+1}), \dotsc$ in [7] cannot be expressed). We therefore use a novel technique called $\delta$-stable partitioning.

Overall, our algorithm works as follows. We assume that a system model $\mathcal {M}$ can be MILP-encoded, either exactly or approximately. Some model families are discussed in §5. This assumption, combined with our key technique of variable-interval MILP encoding of STL, reduces trace synthesis to an MILP problem, which we solve by Gurobi Optimizer [20]. We conduct experimental evaluation to confirm the scalability of our algorithm, especially for complex specs (§6).

Our algorithm is anytime (i.e. interruptible): even if the budget runs out in the course of optimization, a best-effort result (the trace that is the closest to a solution so far) is obtained. A similar benefit is there in case there is no execution trace $\sigma$ that satisfies the spec $\varphi$: we obtain a trace $\sigma '$ that is the closest to satisfy $\varphi$. Accommodation of parameters is another advantage thanks to our use of MILP; we exploit it for parameter mining for PSTL formulas. See §3.

Both controller synthesis techniques [14, 28, 29] and SMT-based model checking techniques [7, 23, 34] can be used for trace synthesis. The methodological differences are discussed later in §1; experimental comparison is made in §6.

Contributions and Organization. We summarize our contributions.

• We introduce an optimization-based algorithm for bounded trace synthesis for STL specs. It assumes that a system model is white-box and MILP-encodable; it also solves the dual problem (namely bounded model checking).

• As a key element, we introduce a variable-interval encoding of STL to MILP.

• MILP encodings of some system models, notably rectangular hybrid automata and double integrator dynamics (suited for the automotive domain).

• We experimentally confirm scalability of our algorithm, especially for complex specs. Comparison is made with MILP-based optimal control [14], SMT-based model checking [34], and optimization-based falsification [11, 37].

• Through the algorithm, case studies and experiments, we argue for the importance and feasibility of spec analysis for CPSs.

After exhibiting preliminaries on STL and stable partitioning in §2, we formulate our problems (bounded trace synthesis, model checking, etc.) in §3. In §4 we present a novel variable-interval MILP encoding of STL; in §5 we discuss MILP encoding of a few families of models. Our main algorithm combines these two encodings. In §6 we present experiment results.

Related Work I: Optimal STL Control with MILP. The works [14, 28, 29] inspire our use of MILP for STL. Their problem is optimal controller synthesis under STL constraints, i.e. to find an input signal $\tau$ to a system model $\mathcal {M}$ so that 1) the output signal $\mathcal {M}(\tau )$ satisfies a given STL spec $\varphi$ and 2) it optimizes $J(\mathcal {M}(\tau ))$, where J is a given objective function. This problem subsumes our problem of trace synthesis, by taking a constant function as J.

The algorithms in [14, 28, 29] reduce their problem to MILP by a constant-interval encoding of the robust semantics [13, 17] of STL (an enhanced encoding is presented in [22]). Specifically, their system model is discrete-time dynamics $x(t+\varDelta t)=f_{\textrm{d}}(x(t), u(t), w(t))$ with a constant interval $\varDelta t$.

In contrast, in our variable-interval encoding (§4), continuous time is discretized into the intervals $\dotsc , (\gamma _{i-1},\gamma _{i}),\{\gamma _{i}\}, (\gamma _{i},\gamma _{i+1}), \dotsc$ where the end points $\gamma _{i}$ are also variables in MILP. This is advantageous not only in modeling precision but also in scalability: for system models that are largely continuous, constant-interval discretization incurs more integer variables in MILP, hampering the performance of MILP solvers. See §6 for experimental comparison.

Related Work II: SMT-Based STL Model Checking. Our key technical element (a variable-interval MILP encoding of STL) uses the idea of stable partitioning from [7, 23, 34]. They solve bounded STL model checking, and also its dual (trace synthesis). The main difference is the class of system models $\mathcal {M}$ accommodated. SMT solvers accommodate more theories than MILP solving, and thus allows encoding of a greater class of models. In contrast, by restricting the model class to MILP-encodable, our algorithm benefits speed and scalability (MILP is faster than SMT). Iterative optimization in MILP also makes our algorithm an anytime one. Native support of parameter synthesis is another plus.

Other Related Work. Optimization-based falsification has its root in the quantitative robust semantics of STL [13, 17]; the successful combination with stochastic optimization metaheuristics has made falsification an approach of both scientific and industrial interest. See the ARCH competition report [16] for state-of-the-art. Falsification is most of the time thought of as search-based testing; therefore, unlike the model checking approach, the absence of counterexamples is usually not proved. Exceptions are [25, 35] where they strive for probabilistic guarantees for such absence.

The current work is motivated by the observation that falsification solvers often struggle in trace synthesis for complex STL specs, even if a system model is simple. It is known that specs with more connectives pose a performance challenge, and many countermeasures are proposed, including [2] (for temporal operators) and [36, 37] (for Boolean connectives).

2 Preliminaries

We let $\mathbb {N}, \mathbb {R}$ denote the sets of natural numbers and reals, respectively; $\mathbb {R}_{\ge 0}$ denotes an obvious subset. The set $\overline{\mathbb {R}}=\mathbb {R}\cup \{-\infty ,\infty \}$ is that of extended reals. The set $\mathbb {B} = \{\top , \bot \}$ is for Boolean truth values. The powerset of a set X is denoted by $\wp (X)$. An interval is a subset of $\mathbb {R}_{\ge 0}$ of the form (a, b), [a, b), (a, b], or [a, c], where $a<b$ and $a\le c$. Therefore a singleton $\{a\}$ is an interval.

Definition 2.1

(linear predicate pand $\llbracket {p}\rrbracket , \pi _{p}$ ). Given a set V of variables, a (closed) linear predicate is a function $p:\mathbb {R}^V \rightarrow \mathbb {B}$ defined as follows, using some $c \in \mathbb {R}^V$ and $b \in \mathbb {R}$: $p(x) = \top$ if and only if $c^{\top }x + b \ge 0$. We write $\llbracket {p}\rrbracket$ for the closed half-space $\{ x \mid p(x) = \top \} \subseteq \mathbb {R}^V$.

For the above p, we define a function $\pi _p(x):\mathbb {R}^V \rightarrow \mathbb {R}$ by $\pi _{p}(x) {:}{=}c^{\top }x + b$. This is understood as the degree of satisfaction (or violation, if negative) of a linear predicate p by $x\in \mathbb {R}^V$. Indeed, $\pi _{p}(x)$ is the (signed) Euclidean distance to the boundary of $\llbracket {p}\rrbracket$, assuming that the Euclidean norm of c is $\Vert c \Vert = 1$.

Definition 2.2

(signal). Let V be a finite set of variables and T a positive real. A signal over V with a time horizon T is a function $\sigma : [0, T] \rightarrow \mathbb {R}^V$. We write $\textbf{Signal}_V^T$ for the set of all signals over V with time horizon T, or simply $\textbf{Signal}_V$ when T is clear from the context.

If necessary, the domain [0, T] of $\sigma$ can be extended to $\mathbb {R}_{\ge 0}$ by setting $\sigma (t) {:}{=}\sigma (T)$ for all $t > T$. This allows us to define the notion of t-postfix, which will serve as the basis of the STL semantics (§2.1). Precisely, the t-postfix of $\sigma$ is a signal $\sigma ^t$ defined by $\sigma ^t(t') {:}{=}\sigma (t+t')$. The domain of $\sigma ^t$ can be chosen freely but we set it to [0, T] for consistency.

Definition 2.3

(system model, trace set $\mathcal {L}(\mathcal {M})$ ). Let $V,V'$ be finite sets of variables. A system model $\mathcal {M}$ from $V'$ to V with a time horizon T is a function $\mathcal {M}:\textbf{Signal}^T_{V'}\rightarrow \wp (\textbf{Signal}_{V}^T)$. The trace set $\mathcal {L}(\mathcal {M}) {:}{=}\textstyle \bigcup _{\tau \in \textbf{Signal}_{V'}^T}\mathcal {M}(\tau )$ is the set of all output signals of $\mathcal {M}$ where an input signal $\tau$ can vary.

We allow system models to be nondeterministic (note the the powerset construction $\wp$); the models in §1 were deterministic for simplicity. A special case of the above is when $V'=\emptyset$, that is, when $\mathcal {M}$ does not have any input. In this case, a system model $\mathcal {M}$ can be identified with a subset $\mathcal {L}(\mathcal {M})\subseteq \textbf{Signal}_{V}$.

Example 2.4

($\mathcal {M}_{\textrm{RNC}}$ ). The dynamics model in Ex. 1.1 is formalized as a system model $\mathcal {M}_{\textrm{RNC}}$ whose input variables (in $V'$) are $a_{\textrm{f}}, v^{\textrm{init}}_{\textrm{f}},x^{\textrm{init}}_{\textrm{f}}, a_{\textrm{r}}, v^{\textrm{init}}_{\textrm{r}},x^{\textrm{init}}_{\textrm{r}}$, and output variables (in V) are $a_{\textrm{f}}, v_{\textrm{f}},x_{\textrm{f}}, a_{\textrm{r}}, v_{\textrm{r}},x_{\textrm{r}}$. Here, the input is acceleration rates ($a_{\textrm{f}},a_{\textrm{r}}$) and the initial values of velocities and positions (modeled using signals $v^{\textrm{init}}_{\textrm{f}}$ etc. for convenience). The time horizon T of $\mathcal {M}$ represents its simulation time; here we set $T=20$. Given an input signal $\tau$, the output $\mathcal {M}(\tau )$ is a singleton $\mathcal {M}(\tau )=\{\sigma \}$, and $\sigma$ is determined by the ODE (2). Specifically, $\sigma (t)(a_{\textrm{f}})=\tau (t)(a_{\textrm{f}})$, $\sigma (t)(v_{\textrm{f}})=\tau (0)(v^{\textrm{init}}_{\textrm{f}})+\int _{0}^{t}\tau (t')(a_{\textrm{f}})\,\textrm{d}t'$, and so on.

2.1 Signal Temporal Logic

Definition 2.5

(signal temporal logic (STL)). In STL, an atomic proposition over a variable set V is represented as $p:\equiv (f(\vec {w}) \ge 0)$, where is a function that maps a $V$-dimensional vector $\vec {w}$ to a real. The syntax of an STL formula $\varphi$ (over V) is defined as follows: $\varphi :\equiv p \mid \bot \mid \top \mid \lnot \varphi \mid \varphi _1\vee \varphi _2\mid \varphi _1\wedge \varphi _2\mid \Diamond _I\varphi \mid \Box _I\varphi \mid \varphi _1 \mathbin {\mathcal {U}_{I}} \varphi _2 \mid \varphi _1 \mathbin {\mathcal {R}_{I}} \varphi _2$, where I is a nonsingular closed time interval, and $\Diamond _{I}$, $\Box _{I}, \mathbin {\mathcal {U}_{I}}$, $\mathbin {\mathcal {R}_{I}}$ are temporal operators eventually, always, until and release. Implication is defined: $\varphi _1 \Rightarrow \varphi _2 :\equiv \lnot \varphi _1 \vee \varphi _2$. We write temporal operators without the subscript I when $I=[0,\infty ]$ (e.g., $\Diamond$). Note that we do not lose generality by restricting the inequality in $p :\equiv (f(\vec {w}) \ge 0)$. Indeed, $\le , <, >$ can be encoded using (a combination of) $-f$ and $\lnot$.

The set $\text {Sub}(\varphi )$ collects all subformulas of an STL formula $\varphi$; the set $\text {AP}(\varphi )$ collects all atomic propositions $\alpha$ occurring in $\varphi$.

Proposition 2.6

Every STL formula has a formula in the negation normal form (NNF)—i.e. one in which negation $\lnot$ appears only in front of atomic propositions—that is semantically equivalent.    $\square$

Assumption 2.7

We assume that each atomic proposition p is a linear predicate (Def. 2.1), that is, $f(x)=c^\top x+b$ with some $c\in \mathbb {R}^{V}, b\in \mathbb {R}$ in each $p :\equiv (f(\vec {w}) \ge 0)$.

The Boolean semantics $\sigma \models \varphi$ and robust semantics $\llbracket {\sigma ,\varphi }\rrbracket \in \overline{\mathbb {R}}$ of STL are standard. See [32, Appendix A].

PSTL is a parametric extension of STL. It is from [4]; see also [9]. Its definition is in [32, Appendix A]. The semantics of PSTL formula is defined naturally by fixing $\vec {u},\vec {v}$; see Prob. 3.3 for the specific forms we use.

2.2 Finite Variability

The satisfiability checking problem for STL—this is equivalent to the model checking problem under the trivial (identity) system model—is already EXPSPACE-complete [3]. To ease computational complexity, bounded model checking has been a common approach [23, 26]. Its main idea is to bound the number of time-points at which the truth value of each subformula can vary.

Definition 2.8

(finite variability [27]). A (finite) partition $\mathcal {P}$ of an interval $D \subseteq \mathbb {R}$ is a sequence $\mathcal {P} = (J_{i})_{i = 1}^{N}$ of nonempty and mutually disjoint intervals such that $\bigcup _{i=1}^N J_{i} = D$, and $\sup (J_{i}) \le \inf (J_{i'})$ for any $i < i'$. A Boolean signal $q:\mathbb {R}_{\ge 0} \rightarrow \mathbb {B}$ is constant on an interval $J \subseteq \mathbb {R}_{\ge 0}$ if $q(t) = q(t')$ for any $t, t' \in J$. We say q(t) has N-bounded variability if there exists a partition $\mathcal {P}$ of $[0, \infty )$ and q(t) is constant on every interval $J \in \mathcal {P}$.

Let $\sigma :[0, T] \rightarrow \mathbb {R}^V$ be a signal and $\varphi$ be an STL formula over V. We say that $\sigma$ has the N-bounded variability with respect to $\varphi$ if the Boolean (truth value) signal $t \mapsto (\sigma ^t \models \varphi )$ has the N-bounded variability. We say $\sigma$ is finitely variable with respect to $\varphi$ if it has the N-bounded variability for some N.

Finally, we say $\sigma$ has the hereditary N-bounded variability with respect to $\varphi$ if, for each subformula $\psi \in \textrm{Sub}(\varphi )$, $\sigma$ has the N-bounded variability with respect to $\psi$. We write N-HBV for the hereditary N-bounded variability.

Lemma 2.9

([7]). Let $\varphi$ be an STL formula. A signal $\sigma$ has the N-HBV with respect to $\varphi$ for some N if and only if it is finitely variable with respect to each atomic proposition $p\in \textrm{AP}(\varphi )$ occurring in $\varphi$.    $\square$

The following is the basis of bounded model checking in [7, 23].

Definition 2.10

(stable partition). Let $\sigma$ be a signal, $\varphi$ be an STL formula, and $\mathcal {P}$ be a partition of [0, T] such that every $J \in \mathcal {P}$ is singular or open. Intuitively, $\mathcal {P}$ looks like $\{\gamma _0\},(\gamma _{0},\gamma _{1}),\{\gamma _{1}\},(\gamma _{1},\gamma _{2}),\dotsc , \{\gamma _N\}$. We say $\mathcal {P}$ is a stable partition for $\sigma$ and $\varphi$ if $t \mapsto \sigma ^t \models \psi$ is constant on J for each $J \in \mathcal {P}$, $\psi \in \textrm{Sub}(\varphi )$.

3 Problem Formulation

We formulate our problems and discuss their mutual relationship. The next problem is studied in [7, 23, 34].

Problem 3.1

(bounded STL model checking). Given an STL formula $\varphi$ (over V), a system model $\mathcal {M}$ (from $V'$ to V) with time horizon T, and a variability bound $N\in \mathbb {N}$, decide if the following is true or not: $\sigma \models \varphi$ holds for an arbitrary trace $\sigma \in \mathcal {L}(\mathcal {M})$ (cf. Def. 2.3) that has the hereditary N-bounded variability (N-HBV) with respect to $\varphi$.

The following is the dual of Prob. 3.1, and is our main scope.

Problem 3.2

(bounded STL trace synthesis). Given $\varphi ,\mathcal {M},T$ and N as in Prob. 3.1, find a trace $\sigma \in \mathcal {L}(\mathcal {M})$ such that 1) $\sigma$ has the N-HBV with respect to $\varphi$ and 2) $\sigma \models \varphi$ holds, or prove that such $\sigma$ does not exist.

Prob. 3.2 resembles the falsification problem [17]: given $\mathcal {M}$ (that can be black-box) and $\varphi '$, find a counterexample input $\tau$ such that $\mathcal {M}(\tau )\not \models \varphi '$. The emphases and the settings are often different though; see §1.

The following is a special case of the STL parameter mining problem; see e.g. [9, § 3.5]. Recall from [32, Def. A.3] that $\varphi _{\vec {u},\vec {v}}$ instantiates parameters $\vec {p},\vec {q}$ in $\varphi$ with real values $\vec {u},\vec {v}$ from the domains P, Q, respectively.

Problem 3.3

(bounded existential parameter mining). Let $\varphi$ be a PSTL formula over parameters $(\vec {p},\vec {q})$, and $\mathcal {M},T$ and N be as in Prob. 3.1. Find the set $\bigl \{\,(\vec {u},\vec {v})\in P\times Q\,\big |\, \sigma \models \varphi _{\vec {u},\vec {v}} \text { for some }\sigma \in \mathcal {L}(\mathcal {M})\text { that has the } N\text {-HBV wrt. } \varphi \,\bigr \}.$

In §6, we study a further special case where there is only one parameter p and the goal is to find the maximum p in the above set.

4 Variable-Interval Encoding of STL to MILP

4.1 $\delta$-Stable Partitions

We shall adapt the idea of stable partitioning [7], reviewed in §2.2, to the current MILP setting. A major difference we need to address is that SMT is symbolic while MILP is numerical: most MILP solvers do not distinguish < from $\le$ and do not accommodate strict inequalities. See e.g. [20].

In order to address this difference, we develop a theory of $\delta$-stable partitions. Here is its outline. Firstly, we replace partitions $\dotsc ,(\gamma _{i-1},\gamma _{i}),\{\gamma _{i}\},(\gamma _{i},\gamma _{i+1}), \dotsc$ used in [7] (see also Def. 2.10) with new “partitions” $\dotsc ,[\gamma _{i-1},\gamma _{i}],[\gamma _{i},\gamma _{i+1}], \dotsc$. The latter can be expressed only using $\le$; but they have overlaps (at $\gamma _{i}$). The original stability notion (see §2.2) does not fit the new partition notion—it requires “constantly true” or “never true,” and prohibits overlaps. Therefore we introduce $\delta$-stability; it requires either “constantly true” or “never robustly true.”

Fig. 2.

A stable partition (cf. [7]) for $\sigma$ and $\varphi :\equiv x \ge 1$. The symbols $\top$ and $\bot$ denote the (constant) truth value of $\varphi$ each interval $J_i$.

Fig. 3.

A $\delta$-stable partition (Def. 4.7) for $\sigma$ and $\varphi$. Here $\varphi ^{\delta }\equiv (x\ge 1+\delta )$. $\top$ and $\bot$ are much like in Fig. 2; the symbol ? indicates that the truth value is not necessarily constant. In some regions (shaded), $\sigma ^t \models \varphi$ is true but $\sigma ^t \models \varphi ^\delta$ is not.

Example 4.1

Let $\sigma$ be a continuous signal. Suppose that a sequence $\mathcal {P} = {(J_i)}_{i=1}^M$ is a stable partition for $\sigma$ and an STL formula $\varphi$, as illustrated in Fig. 2.

In this paper, since MILP solvers do not accommodate strict inequalities, we are forced to use closed intervals; see $\varGamma _{1},\dotsc , \varGamma _{4}$ in Fig. 3. Notice that the truth value of the formula $\varphi$ not constant in $\varGamma _{2}$ or $\varGamma _{4}$. To regain stability, we introduce the $\delta$-tightening $\varphi ^{\delta }$ of the formula $\varphi$ with some $\delta > 0$ (Def. 4.4); here $\varphi ^{\delta }\equiv (x\ge 1+\delta )$. Then the truth value of $\varphi ^{\delta }$ (instead of $\varphi$) is constantly false in $\varGamma _{2}$ and $\varGamma _{4}$, that is, $\varphi$ is “never $\delta$-robustly true” in $\varGamma _{2}$ and $\varGamma _{4}$.

Definition 4.2

(timed state sequence). A time sequence of [0, T] is a sequence $\varGamma =(0 = \gamma _0 < \dots < \gamma _N = T)$. Such a time sequence induces a “partition of [0, T] with singular overlaps,” namely $\varGamma =\bigl ([\gamma _{i-1}, \gamma _{i}]\bigr )_{i=1}^N$. We identify it with the original time sequence, writing $\varGamma _i$ for the interval $[\gamma _{i-1}, \gamma _{i}]$.

Given a time sequence, a timed state sequence over V is a sequence $\varsigma = \bigl ((x_0, \gamma _0), \dots , (x_N, \gamma _N)\bigr )$, where $x_0, \dots , x_N$ in $\mathbb {R}^V$.

In MILP, it is efficient to represent signals as (continuous) piecewise-linear signals, so that values within an interval can be deduced by linear interpolation.

Definition 4.3

(piecewise-linear signal). Given a timed state sequence $\varsigma = ((x_0, \gamma _0), \dots , (x_N, \gamma _N))$, the signal $\varsigma ^\textrm{pwl}:[0, \gamma _N] \rightarrow \mathbb {R}^V$ is defined by the following linear interpolation: $\varsigma ^\textrm{pwl}(t) {:}{=}(1- \lambda ) x_{i-1} + \lambda x_{i}$ if $\gamma _{i-1} \le t \le \gamma _i$ (where $\lambda = \frac{1}{\gamma _{i} - \gamma _{i-1}} (t - \gamma _{i-1})$).

In this paper, a piecewise-linear signal is a signal of the form $\varsigma ^\textrm{pwl}$ for some timed state sequence $\varsigma$. Note that it is continuous everywhere, and is linear everywhere except for only finitely many points. Obviously, $\varsigma ^\textrm{pwl}$ is finitely variable with respect to any linear predicate p (Def. 2.1).

Definition 4.4

($\delta$ -tightening of linear predicates). Let $\delta > 0$ be a positive real and p be a linear predicate defined by $p(x) = \top \iff c^{\top }x + b \ge 0$. The $\delta$-tightening of p is a linear predicate defined by $p^\delta (x) = \top \iff c^{\top }x + b \ge \delta$.

Note that $p^\delta$ is stronger than p, i.e., $[\![p^\delta ]\!] \subsetneq [\![ p ]\!]$. We further extend the concept of $\delta$-tightening for general STL formulas in NNF (cf. Prop. 2.6). Let $p^-$ be the linear predicate defined by $p^-(x) = \top \iff -c^{\top }x - b \ge 0$.

Definition 4.5

($\delta$ -tightening of STL formulas in NNF). Let $\varphi$ be an STL formula in NNF. The $\delta$-tightening $\varphi ^\delta$ of $\varphi$ is the STL formula obtained from $\varphi$ by replacing all occurrences of atomic predicates p (resp. $\lnot p$) by $p^\delta$ (resp. $(p^-)^\delta$).

The $\delta$-tightening construction is related to robust semantics [32, Def. A.2].

Proposition 4.6

Let $\sigma$ be a signal, $\varphi$ be an STL formula in NNF, and $\delta >0$. Then $\sigma \models \varphi ^\delta$ implies $[\![\sigma , \varphi ]\!] \ge \delta$.    $\square$

Since the closed halfspace $[\![p^-]\!]$ coincides with the closure of the open halfspace $\mathbb {R}^V \setminus [\![p]\!]$, the robust semantics is not affected by the difference between $p^-$ and $\lnot p$. For simplicity, in the following, we assume that any STL formula in NNF does not contain negation, i.e., $\lnot p$ is replaced by a new atomic proposition $p^-$.

We are ready to define $\delta$-stability.

Definition 4.7

($\delta$ -stability). Let $\varphi$ be an STL formula over V in NNF, $\sigma \in \textbf{Signal}_{V}^T$ be a signal, and $\varGamma =(\gamma _0, \dots , \gamma _N)$ be a time sequence (Def. 4.2) with $\gamma _{N}=T$. We say $\varGamma$ is $\delta$-stable for $\sigma$ and $\varphi$ if, for each $i \in [1,N]$ and each subformula $\psi \in \textrm{Sub}(\varphi )$, either of the following holds: 1) $\sigma ^t \models \psi$ for each $t\in \varGamma _i$, or 2) $\sigma ^t \not \models \psi ^\delta$ for each $t\in \varGamma _i$.

In the above definition, in each interval $\varGamma _{i}$, a subformula $\psi$ is either 1) always true or 2) never robustly true. The two conditions are not mutually exclusive—both hold if $\sigma ^{t}\models \psi \wedge \lnot \psi ^{\delta }$ for all $t\in \varGamma _{i}$.

The next notion of conservative valuation records which of 1) and 2) is true in each interval. It conservatively approximates the actual truth of $\varphi$ (Fig. 3).

Definition 4.8

(conservative valuation). Let $\varphi$ be an STL formula in NNF, and $\varGamma =(\gamma _0, \dots , \gamma _N)$ be a time sequence . A valuation of $\varphi$ in $\varGamma$ is a function $\varTheta : \textrm{Sub}(\varphi ) \times [1,N] \rightarrow \mathbb {B}$ that assigns, to each subformula and index of the intervals of $\varGamma$, a Boolean truth value. Let $\sigma$ be a signal with a time horizon $T = \gamma _N$. We say that $\varTheta$ is a conservative valuation of $\varphi$ in $\varGamma$ on $\sigma$ (up to $\delta$) if 1) $\varTheta (\psi , i) = \top$ implies that, for each $t\in \varGamma _{i}$, $\sigma ^t \models \psi$ holds; and 2) $\varTheta (\psi , \varGamma _{i}) = \bot$ implies, for each $t\in \varGamma _i$, $\sigma ^t \not \models \psi ^\delta$.

We simply write $\langle \psi \rangle _i$ for $\varTheta (\psi , i)$ when $\varTheta$ is clear from context.

Suppose there exists a conservative valuation $\varTheta$ of an STL formula $\varphi$ in a time sequence $\varGamma$ on a signal $\sigma$ up to $\delta$. Then $\varGamma$ is $\delta$-stable for $\sigma$ and $\varphi$.

We shall argue in §4.2 that, for each piecewise-linear signal $\sigma$ (Def. 4.3), an STL formula $\varphi$, there is a time sequence $\varGamma$ in which $\varphi$ is $\delta$-stable on $\sigma$. We start with a special case where $\varphi$ is an atomic proposition p.

Fig. 4.

A conservative valuation $\varTheta$ of a linear predicate p on $\sigma$. The red segments are assigned $\top$ by $\varTheta$. (Color figure online)

Definition 4.9

Let $x, x' \in \mathbb {R}^V$, and p be a linear predicate on V. We say $(x, x')$ is a $\delta$-crossing pair with respect to p if $x\in \llbracket {p^\delta }\rrbracket$ and $x'\not \in \llbracket {p^\delta }\rrbracket$ (cf. Def. 2.1), or vice versa. A $\delta$-crossing pair is stationary if $x\in \llbracket {p}\rrbracket$ and $x'\in \llbracket {p}\rrbracket$.

Lemma 4.10

Let p be a linear predicate and $\sigma$ be a piecewise-linear signal. There is a time sequence $\varGamma =(\gamma _{0}, \dots ,\gamma _{N})$ such that, for any $i \in [1,N]$, 1) $\sigma$ is linear in the interval $[\gamma _{i-1}, \gamma _{i}]$, and 2) if $(\sigma (\gamma _{i-1}), \sigma (\gamma _{{i}}))$ is a $\delta$-crossing pair, it is stationary. It follows that there is a conservative valuation $\varTheta$ of p in $\varGamma$ on $\sigma$.

Proof

The lemma argues that, whenever $\sigma$ enters or leaves $\llbracket {p^{\delta }}\rrbracket$, it has to do so via $\llbracket {p}\rrbracket \setminus \llbracket {p^{\delta }}\rrbracket$. See Fig. 4. This can be enforced by adding suitable points to $\varGamma$, exploiting continuity of $\sigma$ (Def. 4.3) and the intermediate value theorem.    $\square$

We note another advantage of $\delta$-stable partitions: the number of invervals is roughly halved compared to (original) stable partitions (see Figs. 2 and 3). This advantage may be exploited also in SMT-based approaches [7] for scalability.

4.2 Variable-Interval MILP Encoding

Our MILP encoding of STL relies on the constructs in §4.1. For the purpose of trace synthesis for an STL formula $\varphi$, our basic strategy is to search for 1) a time sequence $\varGamma =(\gamma _0, \dots , \gamma _N)$ (i.e. a “partition,” see Def. 4.2) and 2) a valuation $\varTheta : \textrm{Sub}(\varphi ) \times [1,N] \rightarrow \mathbb {B}$, such that

• $\varTheta$ is consistent in the sense that the truth values assigned to subformulas comply with the STL semantics (§2.1);

• $\varTheta$ is fulfilling in the sense that it assigns $\top$ to the top-level formula $\varphi$ in $\varGamma _{1}$ (the first interval); and

• $\varTheta$ is realizable in the sense that there is a piecewise-linear trace $\sigma \in \mathcal {L}(\mathcal {M})$ of $\mathcal {M}$ that yields $\varTheta$. That is, precisely, $\varTheta$ must be a conservative valuation of $\varphi$ in $\varGamma$ on $\sigma$ (Def. 4.8).

The entities $\varGamma ,\varTheta$ we search for are expressed as MILP variables, and the above three conditions are expressed as MILP constraints. We describe these MILP variables and constraints in the rest of the section. The constraints expressing $\sigma \in \mathcal {L}(\mathcal {M})$ require system model encoding and are thus deferred to later sections.

Variables. We use the following MILP variables. Their collection is denoted by $\textbf{Var}(\varphi , N)$. Here $N\in \mathbb {N}$ is a constant for variability bound (Prob. 3.2).

• Real-valued variables $\{ \gamma _{0}, \dots ,\gamma _{N} \}$ for a time sequence $\varGamma$.

• Boolean variables $\{ \langle \psi \rangle _{i} \mid 1 \le i \le N, \psi \in \textrm{Sub}(\varphi ) \}$ for the value $\varTheta (\psi ,i)$ of a valuation $\varTheta$ that we search for.

• Real-valued variables $\{ x_{i,v}\mid 0\le i\le N , v\in V\}$ for the values of a piecewise-linear trace $\sigma \in \mathcal {L}(\mathcal {M})$.

• Boolean variables $\{ \zeta ^p_{i}, \zeta ^{\delta , p}_{i} \mid 0 \le i \le N, p \in \textrm{AP}(\varphi ) \}$ for the truth values of p and $p^\delta$ at time $\gamma _{i}$. These variables are used to detect crossing pairs (Def. 4.9).

• Real-valued variables $\{ S^\psi _{i} \mid 0 \le i \le N, \Box _{I}\psi \in \textrm{Sub}(\varphi ) \}$. This auxiliary variable records for how long $\psi$ has been true before $\gamma _{i}$.

• Real-valued variables $\{ P^\psi _{i} \mid 0 \le i \le N, \Diamond _{I}\psi \in \textrm{Sub}(\varphi ) \}$. This auxiliary variable records for how long $\psi$ has been false before $\gamma _{i}$.

By an assignment we refer to a function $\boldsymbol{v}:\textbf{Var}(\varphi , N)\rightarrow \mathbb {R}$ such that $\boldsymbol{v}(y)\in \{0,1\}$ for each Boolean variable y. The MILP problem is to find an assignment $\boldsymbol{v}$ that optimizes an objective under given constraints.

Notation 4.11

In what follows, as a notational convention, we simply write a variable y for the value $\boldsymbol{v}(y)$ when the assignment $\boldsymbol{v}$ is clear from context. We further write $\varsigma$ for the timed state sequence composed of the time sequence $\{\gamma _{0}, \dots ,\gamma _{N}\}$ and the trace values $\{ x_{j,v}\mid 0\le j\le N , v\in V\}$.

Note that, in this paper, we encode the Boolean semantics of STL [32, Def. A.1], unlike [28, 29] where the robust semantics is encoded in a constant-interval manner. The combination of variable-interval encoding and quantitative robust semantics is future work; for example, a quantitative extension of $\delta$-stable partitions (§4.1) seems quite nontrivial.

Shorthands for Propositional Connectives. We use standard shorthands for Boolean connectives in MILP constraints (such as $\lnot A, A\wedge B$ where A, B are Boolean variables). See [32, Appendix B] for the formal encodings.

Realizability Constraints: Traces and Atomic Propositions. We need to constrain $\gamma _0,\dots ,\gamma _N$ to be a time sequence (Def. 4.2), using some constant $\varepsilon >0$ and letting $\cdots \ge \varepsilon$ stand for $\cdots >0$.

$$\begin{aligned} \begin{array}{ll} \gamma _0 = 0,\quad \gamma _N = T,\quad \gamma _{i} - \gamma _{i-1} \ge \varepsilon & \quad \text {for all }i \in [1,N] \end{array} \end{aligned}$$

(3)

For each i and $p\in \textrm{AP}(\varphi )$ (say p is defined by $c^{\top }x + b \ge 0$), the variables $\zeta ^p_i, \zeta ^{\delta ,p}_i$ are constrained as follows,

$$\begin{aligned} \begin{array}{ll} \zeta ^p_i = 1 \;\Rightarrow \; c^{\top }x_i + b \ge 0\qquad \qquad \qquad &{} \zeta ^p_i = 0 \;\Rightarrow \; c^{\top }x_i + b \le -\varepsilon \\ \zeta ^{\delta , p}_i = 1 \;\Rightarrow \; c^{\top }x_i + b \ge \delta &{} \zeta ^{\delta , p}_i = 0 \;\Rightarrow \; c^{\top }x_i + b \le \delta -\varepsilon \end{array} \end{aligned}$$

(4)

Moreover, we impose the following to ensure that $\varGamma$ is the one in Lem. 4.10:

$$\begin{aligned} \begin{array}{ll} \zeta ^{\delta , p}_i = 0 \wedge \zeta ^{\delta , p}_{i+1} = 1 \;\Rightarrow \; \zeta ^p_i = 1 , \quad \zeta ^{\delta , p}_i = 1 \wedge \zeta ^{\delta , p}_{i+1} = 0 \;\Rightarrow \; \zeta ^p_{i+1} = 1 \end{array} \end{aligned}$$

(5)

Under constraints (3) to (5), $\varGamma$ is $\delta$-stable for $\varsigma ^\textrm{pwl}$ (cf. Def. 4.3) and p, by Lem. 4.10. By the definition of $\delta$-stability, we can now constrain the variable $\langle p \rangle _i$ by $\langle p \rangle _i = \zeta ^{p,\delta }_{i-1} \vee \zeta ^{p,\delta }_{i}$ for each i and $p\in \textrm{AP}(\varphi )$.

Remark 4.12

Note that $\varepsilon$ must be chosen to be small enough for the completeness of the encoding (Thm. 4.18). Thereafter we assume that, given a piecewise-linear signal $\sigma$ and an STL formula $\varphi$, $\varepsilon$ is small enough to find a $\delta$-stable partition for $\sigma$ and $\varphi$, and we omit $\varepsilon$ from the constraints for simplicity.

Consistency Constraints I: Boolean Connectives. We can directly encode conjunction $\bigwedge _{j=1}^m \psi _j$ in STL by recursively applying the shorthand $\wedge$ in [32, Appendix B]: $\textstyle \langle \bigwedge _{j=1}^m \psi _j \rangle _i = \langle \psi _1 \rangle _i \wedge \langle \bigwedge _{j=2}^m \psi _j \rangle _i$ for each $i\in [1,N]$. It is known that the following alternative encoding avoids auxiliary variables $\langle \bigwedge _{j=k}^m \psi _j \rangle _i$ (where k varies): for each $i\in [1,N]$, $\langle \bigwedge _{j=1}^m \psi _j \rangle _i \ge 1 - m + \sum _{j=1}^m \langle \psi _j \rangle _i$ and $\textstyle \langle \bigwedge _{j=1}^m \psi _j \rangle _i \le \langle \psi _j \rangle _i$. An encoding for disjunction is given similarly: $\textstyle \langle \bigvee _{j=1}^m \psi _j \rangle _i \le \sum _{j=1}^m \langle \psi _j \rangle _i$, $\langle \bigvee _{j=1}^m \psi _j \rangle _i \ge \langle \psi _j \rangle _i$.

Consistency Constraints II: Unbounded Temporal Modalities. For temporal operators with $I=[0,\infty )$, the following encodings are straightforward.

$$\begin{aligned} \begin{array}{ll} \langle \psi _{1} \mathcal {U}\ \psi _{2} \rangle _{i} = \langle \psi _{2} \rangle _{i} \vee (\langle \psi _{1} \mathcal {U}\ \psi _{2} \rangle _{i+1} \wedge \langle \psi _{1} \rangle _{i}),\quad \\ \langle \psi _{1} \mathcal {R}\ \psi _{2} \rangle _i = \langle \psi _{2} \rangle _{i} \wedge (\langle \psi _{1} \mathcal {R}\ \psi _{2} \rangle _{i+1} \vee \langle \psi _{1} \rangle _{i}) \quad &{} \quad \text {for each}\,i\in [1,N-1]\text {,} \\ \langle \psi _{1} \mathcal {U}\ \psi _{2} \rangle _N = \langle \psi _{2} \rangle _{N}, \quad \langle \psi _{1} \mathcal {R}\ \psi _{2} \rangle _{N} = \langle \psi _{2} \rangle _{N} &{} \quad \text {for }i=N\text {.} \end{array} \end{aligned}$$

(6)

The encodings for $\Diamond ,\Box$ are special cases.

Consistency Constraints III: Bounded Temporal Modalities. This is the most technically involved part. The challenge here is that the stability for $\Box _{[a,b]} \psi$ is not guaranteed by the stability for $\psi$ (similarly for $\Diamond _{[a,b]}\psi$). Therefore we need additional MILP constraints for the stability for $\Box _{[a,b]} \psi$.

Our encoding is inspired by the results from [26]; ours is simpler thanks to our theory in §4.1 where intervals are all closed.

Recall that we use the variables $S^\psi _{i},P^\psi _{i}$ for this purpose. We focus on $\Box _{[a,b]} \psi$; the encoding of $\Diamond _{[a,b]} \psi$ is similar. The constraints on $S^\psi _{i}$ are as follows.

$$\begin{aligned} \begin{array}{ll} S^\psi _0 = 0, \qquad \langle \psi \rangle _{i} = 0 \;\Rightarrow \; S^\psi _i = 0, \\ \langle \psi \rangle _{i} = 1 \;\Rightarrow \; S^\psi _i \ge S^\psi _{i-1} + (\gamma _i - \gamma _{i-1}) &{} \quad \text {for each }i\in [1,N]\text {.} \end{array} \end{aligned}$$

It follows that, for any non-negative real number $L\in [0,\gamma _j)$, we have $S^\psi _j \le L$ if and only if there exists $k\in [1, j]$ such that $\langle \psi \rangle _k = 0$ and $\gamma _j - \gamma _k \le L$.

We proceed to the constraints that describe the relationship between $S^\psi _i$ and the semantics of $\Box _{I} \psi$. Suppose $\varGamma = ( \gamma _0,\dots ,\gamma _N )$ is $\delta$-stable for a signal $\sigma$ and $\psi$. Let us write $\gamma _{N+1} = \infty$ and $\langle \psi \rangle _{N+1} = \langle \psi \rangle _{N}$ for simplicity.

We consider consistency for the positive and negative cases separately. For the positive one (i.e. $\langle \Box _{[a,b]} \psi \rangle _i = 1$), the following observation is used.

Proposition 4.13

Let $\varphi \equiv \Box _I \psi$ be an STL formula in NNF, and $\varTheta$ be a conservative valuation of $\psi$ in $\varGamma = (\gamma _0,\dots ,\gamma _N)$ on a signal $\sigma$. Given $i\in [1,N]$, suppose $(\varGamma _i + I) \cap (\gamma _{j-1}, \gamma _{j}] \ne \emptyset$ implies $\langle \psi \rangle _j = 1$ for each $j \in [i, N + 1]$. Then $\sigma ^t \models \varphi$ holds for any $t \in \varGamma _i$.    $\square$

Prop. 4.13 leads to the following MILP constraint:

$$\begin{aligned} \begin{array}{ll} \lnot \langle \varphi \rangle _{i} \vee (\gamma _{i} + b \le \gamma _{j-1}) \vee (\gamma _{i - 1} + a > \gamma _{j}) \vee \langle \psi \rangle _{j} & \; \text {for each}\, i\in [1,N]\text {,}j \in [i, N+1]\text {.} \end{array} \end{aligned}$$

The constraint itself does not follow the MILP format; we can nevertheless express it in MILP using an auxiliary Boolean variable $Z_f$. Specifically, an inequality $f(x) \ge 0$ in a disjunctive constraint is constrained by $Z_f = 1 \Rightarrow f(x) \ge 0$.

For the consistency in the negative case (i.e. $\langle \Box _{[a,b]} \psi \rangle _i = 0$), the counterpart of Prop. 4.13 also involves $S^\psi _j$. See below; it leads to an MILP constraint much like Prop. 4.13 does.

Proposition 4.14

Suppose $\varphi$, $\sigma$, $\varGamma$, and $\varTheta$ are as in Prop. 4.13. For any $t \in \varGamma _i$, $\sigma ^t \not \models \varphi ^\delta$ holds if the following conditions are satisfied for each $j \in [i, N]$:

$$\begin{aligned} {\left\{ \begin{array}{ll} S^\psi _j \le b - a &{} \text {if }\gamma _j \in (\gamma _{i-1} + b, \gamma _i + b), \\ S^\psi _j \le \gamma _j - \gamma _i - a &{} \text {if }\gamma _i + b \in [\gamma _{j-1}, \gamma _j], \\ S^\psi _N \le \max (0, \gamma _N - \gamma _{i} - a) &{} \text {if }\gamma _i + b > \gamma _N. \end{array}\right. } \end{aligned}$$

(7)

Proof

Let $j_t \in [i, N+1]$ be the unique index such that $t + b \in [\gamma _{j_t-1}, \gamma _{j_t})$. When $j_t \le N$ and $\gamma _{j_t} < \gamma _i + b$, we have $\gamma _{j_t} \in (\gamma _{i-1} + b, \gamma _i + b)$ and by assumption $S^\psi _{j_t} \le b - a$. There is $k \in [1, j_t]$ such that $\langle \psi \rangle _k = 0$ and $\gamma _k \ge \gamma _{j_t} - b + a > t + a$. We obtain $\varGamma _{k} \cap (t + [a, b]) \ne \emptyset$ and then $\sigma ^t \not \models \varphi ^\delta$ holds. The other cases can be checked in a similar manner.    $\square$

Remark 4.15

For Prop. 4.13, the converse of the statement does not hold. This is because $\sigma ^t \models \psi$ does not guarantee $\langle \psi \rangle _i{:}{=}\varTheta (\psi , i)=1$ where $t \in \varGamma _i$—we allow $\langle \psi \rangle _i = 0$ when $\sigma ^t \models \psi \wedge \lnot \psi ^\delta$. It is similar for Prop. 4.14. However, this does not affect the completeness of the encoding (Thm. 4.18): while the converse of Prop. 4.13 does not hold for fixed $\varGamma$, in our workflow we also search for $\varGamma$, in which case it is easily shown that the MILP constraints derived from Prop. 4.13 are complete. The same is true for Prop. 4.14.

The remaining cases ($\varphi \equiv \psi _1 \mathbin {\mathcal {U}_{I}} \psi _2$ and $\varphi \equiv \psi _1 \mathbin {\mathcal {R}_{I}} \psi _2$) can be reduced to the cases for $\Box _{I}$ and $\Diamond _{I}$. It is by the rewriting techniques shown in [12]:

$$\begin{aligned} \psi _1 \mathbin {\mathcal {U}_{[a, b]}} \psi _2 \quad \sim \ \quad & \Diamond _{[a,b]}\psi _2 \wedge \Box _{[0, a]} (\psi _1 \mathcal {U}\ \psi _2), \end{aligned}$$

(8)

$$\begin{aligned} \psi _1 \mathbin {\mathcal {R}_{[a, b]}} \psi _2 \quad \sim \ \quad & \Box _{[a,b]}\psi _2 \vee \Diamond _{[0, a]} (\psi _1 \mathcal {R}\ \psi _2). \end{aligned}$$

(9)

These equivalences hold in both Boolean and robust semantics.

Correctness of Encoding. Let $\textbf{Enc}_{\textbf{STL}}(\varphi , N, T, \delta )$ denote the polyhedron defined by the above MILP constraints. It is correct in the following sense; see also the goal we announced in the beginning of §4.2. Its proof is by induction on $\varphi$.

Lemma 4.16

Let $\varphi$ be an STL formula in NNF, $N \in \mathbb {N}$, $T > 0$ and $\delta > 0$. Given an assignment $\boldsymbol{v}:\textbf{Var}(\varphi , N)\rightarrow \mathbb {R}$ that lies in $\textbf{Enc}_{\textbf{STL}}(\varphi , N, T, \delta )$, let $\varGamma$, $\varsigma$ be the time sequence and the timed state sequence determined by $\boldsymbol{v}$, and define a valuation $\varTheta$ by $\varTheta (\psi , i) {:}{=}\langle \psi \rangle _i$ (cf. Def. 4.8). Then $\varTheta$ is a conservative valuation of $\varphi$ in $\varGamma$ on the signal $\varsigma ^\textrm{pwl}$.    $\square$

We define $\textbf{Enc}(\varphi , \mathcal {M}, N, T, \delta )$ by the intersection of $\textbf{Enc}_{\textbf{STL}}(\varphi , N, T, \delta )$, the MILP encoding $\textbf{Enc}_{\textbf{model}}(\mathcal {M}, N, T)$ of a system model $\mathcal {M}$, and $\langle \varphi \rangle _1 = 1$.

Theorem 4.17

(soundness). Let $\varphi$ be an STL formula in NNF, $\mathcal {M}$ be a model with a time horizon T, $N \in \mathbb {N}$ and $\delta > 0$. If an assignment $\boldsymbol{v}$ lies in $\textbf{Enc}(\varphi , \mathcal {M}, N, T, \delta )$, the induced $\varsigma ^\textrm{pwl}$ has $\varsigma ^\textrm{pwl} \in \mathcal {L}(\mathcal {M})$ and $\llbracket {\varsigma ^\textrm{pwl}, \varphi }\rrbracket \ge 0$.    $\square$

Theorem 4.18

(completeness). Assume the setting of Thm. 4.17. If there is piecewise-linear $\sigma \in \mathcal {L}(\mathcal {M})$ such that $\llbracket {\sigma , \varphi }\rrbracket \ge \delta$, there is an assignment $\boldsymbol{v}$ that lies in $\textbf{Enc}(\varphi , \mathcal {M}, N, T, \delta )$ for some $N \in \mathbb {N}$.    $\square$

5 System Models and Their MILP Encoding

We introduce the MILP encoding $\textbf{Enc}_{\textbf{model}}(\mathcal {M}, N, T)$ for some families of models $\mathcal {M}$. We introduce an exact encoding for rectangular hybrid automata (RHAs), and an approximate one for HAs with closed-form solutions. We also introduce a refinement of the latter—it is more precise and efficient—restricting to double integrator dynamics. The last is useful for automotive examples such as Ex. 1.1.

We defer the discussion of RHAs for the space reason; see [32, Appendix C]. We thus focus on the other two families.

5.1 HAs with Closed-Form Solutions

Fig. 5.

MILP encoding of f(t)

Here we are interested in hybrid automata (HAs) whose continuous flow dynamics at each control mode has a closed-form solution. The basic idea is simple and it is illustrated in Fig. 5, where the solution f(t) of dynamics (blue) is approximated by a piecewise linear function (red). Such MILP encoding is standard; see e.g. [5].

We formalize this intuition. Firstly, to accommodate input signals $\tau \in \textbf{Signal}_{V'}$ (Def. 2.3), we extend the HA definition so that some variables $x^{\textrm{in}}$ can be designated to be input variables. This means that there are no ODEs whose left-hand side is $\dot{ x^{\textrm{in}}}$, and that the variable updates associated with mode transitions never change $x^{\textrm{in}}$.

Then the above “closed-form solution” assumption on an HA $\mathcal {H}$ is precisely described as follows. Let $\vec {x^{\textrm{in}}}=(x^{\textrm{in}}_{1},\dotsc ,x^{\textrm{in}}_{k})$ enumerate $\mathcal {H}$’s input variables, and $\vec {x}=(x_{1},\dotsc ,x_{l})$ enumerate its other variables. We assume that, for the flow dynamics at each control mode u, there is a closed-form solution

Here, the variable t is the elapsed time since the arrival at the current control mode u; the variables $\vec {x^{\textrm{in}}}$ refer to the input variables (their values are assumed to be constant within the same mode); and the variables $\vec {x_{0}}$ refer to the initial values of $\vec {x}$ on the arrival at u. The assumption holds in many examples, such as polynomial dynamics.

Let us motivate the assumption. A closed-form solution $f_{u}$ helps precision: in piecewise linear approximation such as in Fig. 5, errors do not accumulate over time; in contrast, if a closed-form solution is not given, our alternative will be numerical integration e.g. by the Euler method, where errors accumulate. The linearity assumption in (10) is there for MILP encoding; see below.

Our approximate MILP encoding poses the closed-form solution assumption and follows the intuition of Fig. 5. Specifically, 1) it fixes a constant $\varDelta t\in \mathbb {R}_{\ge 0}$ as a sampling interval; 2) it obtains a family $\bigl (\, f_{u}(k\cdot \varDelta t, \vec {x^{\textrm{in}}},\vec {x}_{0}) \,\bigr )_{k}$ of linear functions over the variables $\vec {x^{\textrm{in}}},\vec {x}_{0}$; and 3) the value of $\vec {x}$ at the elapsed time t is expressed by the linear interpolation

$$\begin{aligned} \textstyle \frac{(k+1)\varDelta t -t}{\varDelta t} f_{u}(k\varDelta t, \vec {x^{\textrm{in}}},\vec {x}_{0}) + \frac{t-k\varDelta t}{\varDelta t} f_{u}\bigl ((k+1)\varDelta t, \vec {x^{\textrm{in}}},\vec {x}_{0}\bigr ), \end{aligned}$$

(11)

where k is such that $k\varDelta t\le t \le (k+1)\varDelta t$. This encoding of flow dynamics is combined with the HA structure, much like in [32, Appendix C], yielding an approximate MILP encoding of the whole HA.

The above encoding has two sources of numerical errors. One is linear interpolation. Errors caused by it are illustrated in Fig. 5 as the vertical margin between blue and red.

The other source is binary expansion [18, 19], a standard MILP technique for encoding bilinear functions. Indeed, in (11), $t, \vec {x^{\textrm{in}}},\vec {x}_{0}$ are all continuous variables in MILP, and the expression (11) can contain their products. The linearity assumption in (10) has been posed to restrict (11) to bilinear.

5.2 HAs with Double Integrator Dynamics

Our next focus is a special case of the model family of §5.1, where each continuous flow is double integrator dynamics. This is important because 1) it gets rid of one of the two error sources in §5.1, namely linear interpolation, by the trapezoidal rule, and 2) it can be used for many automotive dynamics models (cf. Ex. 1.1).

The trapezoidal rule is a basic technique in numerical integration [6], where $\int _{a}^{b}g(t)\,\textrm{d}t$ is approximated by $(b-a)\frac{g(a)+g(b)}{2}$. For double integrator dynamics, we apply the trapezoidal rule to the velocity v, and it is exact since v’s evolution is linear. This allows us to express the position x in the bilinear form $x= t\cdot \frac{v_{0} + v}{2}$, using the variables t (elapsed time), $v_{0}$ (initial velocity), and v (current velocity). Thus we can dispose of the sampling points and their interpolation (11) in §5.1.

We exploit this encoding for our automotive case studies such as Ex. 1.1.

6 Implementation and Experiments

We implemented, in Python, our MILP encodings of the STL semantics (§4) and two model families, namely RHAs [32, Appendix C] and double integrator dynamics (§5.2; multiple modes are not supported since our benchmarks do not need them). The hyperparameter $\delta$ in our encoding is fixed at 0.1 for all benchmarks. The resulting MILP constraints are solved by Gurobi Optimizer [20]. This prototype implementation is called STLts—STL trace synthesizer.

Our experiments are designed to address the following research questions.

• RQ1 Assess the effect of variability bounds N (Prob. 3.2) on the performance.

• RQ2 Compare the performance of STLts with optimization-based falsification, and with SMT-based model checking.

• RQ3 Assess the performance of STLts for real-world complex scenarios.

• RQ4 Assess the performance of STLts in parameter mining (Prob. 3.3).

We used three classes of benchmarks: rear-end near collision (RNC), navigation (NAV), and disturbance scenarios in ISO 34502 (ISO). In each class, we have multiple STL specs, resulting in benchmarks such as RNC1, RNC2, etc.

Rear-End Near Collision (RNC1–3). As discussed in Ex. 1.1, these automotive benchmarks are simplifications of the ISO benchmarks below. The spec $\texttt{RNC1}$ is presented in Ex. 1.1. The system model (2) (see also $\mathcal {M}_{\textrm{RNC}}$ in Ex. 2.4) is double integrator dynamics (§5.2) and is shared by the benchmarks RNC1–3.

The other two specs $\texttt{RNC2}, \texttt{RNC3}$ are defined as follows, using formulas in (1):

$$\begin{aligned} \begin{array}{rll} \texttt{RNC2} &{} \quad :\equiv \quad &{} \bigl (\Box (x_{\textrm{f}} - x_{\textrm{r}} \ge 0)\bigr ) \wedge \\ &{} &{} \quad \Diamond _{[0, 9]} \bigl ( (\Box _{[0,1]} \texttt{danger} ) \wedge (\Box _{[0,1]} a_{\textrm{r}} \ge 1 ) \wedge (\Diamond _{[1,5]} \lnot \texttt{danger}) \bigr ) \\ \texttt{trimming2} &{} \quad :\equiv \quad &{} (\Diamond \texttt{danger}) \Rightarrow \bigl ((\Box _{[0, 1]} a_{\textrm{r}} \ge 1) \mathbin {\mathcal {U}} \texttt{danger}\bigr ) \\ \texttt{RNC3} &{} \quad :\equiv \quad &{} \Box (\mathtt {dyn\_inv}\wedge \texttt{trimming2} ) \wedge \Diamond _{[0, 9]}\Box _{[0,1]}\texttt{danger} \end{array} \end{aligned}$$

(12)

Fig. 6.

The RHA $\mathcal {M}_{\textrm{NAV}}$ for NAV1–2

Navigation (NAV1–2). Here we use a system model that adapts NAV-2 from [15]. The latter is a standard example of an RHA [32, Appendix C], used e.g. in [10].

Our system model $\mathcal {M}_{\textrm{NAV}}$ is an RHA that describes the motion of a point robot in a $2 \times 2$ grid where each region has a rectangular vector field, with a time horizon $T = 40$. See Fig. 6. We have 4 regions $\ell _1,\dotsc ,\ell _4$, each associated with rectangular bounds for $\dot{x},\dot{y}$ and invariants; besides, we set an unsafe region $\texttt{unsafeR}$ ($x\in [9,10]$) and a goal region $\texttt{goalR}$ ($x\in [4,6] \wedge y\in [2,5]$). The robot starts from an initial position $(x_0,y_0)$ where $x_0\in [0,3]\wedge y_0=0$.

We consider two specs: $\texttt{NAV1}\,:\equiv \,\Diamond (\Box _{[0,3]} ((x, y) \in \texttt{goalR})) \wedge \Box ( x \not \in \texttt{unsafeR})$ and $\texttt{NAV2}\,\equiv \,\Box ((x,y) \in \ell _3 \rightarrow \Diamond _{[0,3]} (x,y) \in \ell _4)$. $\texttt{NAV1}$ is almost a standard reach-avoid constraint, but it additionally requires the persistence to the goal region for three seconds. Such specifications are not accommodated in many control and model checking frameworks specialized in reach-avoid constraints (see e.g. [10]). $\texttt{NAV2}$ is a response specification—the trigger (being in $\ell _3$) must be responded by moving to $\ell _4$ within a three-second deadline. Such specs are common in manufacturing; see e.g. [36].

ISO 34502 Disturbance Scenarios for Automated Driving (ISO1, ISO3, $\dotsc$ , ISO8). These benchmarks motivated the current work. As discussed in §1 (see Ex. 1.1), we obtained in [30] complex STL specs as the formalization of the disturbance scenarios in the ISO 34502 standard, but in our illustration efforts by trace synthesis, we found that existing techniques such as optimization-based falsification struggle.

Table 1. Disturbance scenarios in the ISO 34502 standard. Table from [21]

In our experiments, the system model is similar to $\mathcal {M}_{\textrm{RNC}}$ (Ex. 1.1 and 2.4), while lateral dynamics is added and the time horizon is 10 time units here. As for specs, we use seven STL specs $\texttt{ISO1}, \texttt{ISO3},\dotsc ,\texttt{ISO8}$; these are obtained in [30] as the formalization of the disturbance scenarios No. 1,3,$\dotsc$,8 in the ISO 34502 standard for automated driving vehicles. See Table 1. Scenario No. 2 was omitted in [30] since it involves three vehicles; we omit Scenarios No. 9–24 since they are the same with No. 1–8 except in the road shape.

Specifically, the specs $\texttt{ISO}i$ follow the common format shown below [30]:

$$\begin{aligned} \begin{array}{rcl} \texttt{ISO}{i} &{} \;\equiv \; &{} \texttt{initSafe} \wedge \texttt{disturb}_i, \qquad \\ \texttt{disturb}_i &{} \;\equiv \; &{} \texttt{initialCondition}_i \wedge \texttt{behaviourSV}_i \wedge \texttt{behaviourPOV}_i \end{array} \end{aligned}$$

where SV refers to the subject (“ego”) vehicle and POV refers to the principal other vehicle. The component formulas $\texttt{initialCondition}_i$, $\texttt{behaviourSV}_i$ and $\texttt{behaviourPOV}_i$ vary for different scenarios (No. i). Going into their definitions are beyond the scope of this paper; we highlight $\texttt{ISO5}$ as an example to demonstrate the complexity of the specs $\texttt{ISO}{i}$.

$$\begin{aligned} \begin{array}{rcl} \texttt{initialCondition}_5 &{} \;\equiv \; &{} \top \qquad \qquad \qquad \texttt{behaviourSV}_5 \;\equiv \; \texttt{leavingLane}(\texttt{SV},L) \\ \texttt{behaviourPOV}_5 &{} \;\equiv \; &{} \texttt{cutIn}(\texttt{POV},\texttt{SV}) \\ \texttt{leavingLane}(a,L) &{} \;\equiv \; &{} \texttt{atLane}(a,L) \wedge \Diamond (\lnot \texttt{atLane}(a,L)) \\ \texttt{cutIn}(\texttt{POV},\texttt{SV},L) &{} \;\equiv \; &{} \lnot \texttt{sameLane}(\texttt{POV},\texttt{SV},L) \wedge \Diamond \bigl (\texttt{danger}(\texttt{SV},\texttt{POV}) \\ {} &{}&{} \!\!\!\!\!\! \wedge \Diamond _{[0,\texttt{minDanger}]}(\texttt{sameLane}(\texttt{SV},\texttt{POV},L) \wedge \texttt{aheadOf}(\texttt{SV},\texttt{POV}))\bigr ) \\ \texttt{danger}(\texttt{SV},\texttt{POV}) &{} \;\equiv \; &{} \Box _{[0,\texttt{minDanger}]} \texttt{rssViolation}(\texttt{SV},\texttt{POV}) \end{array} \end{aligned}$$

(13)

The formulas not defined here are suitably defined atomic propositions.

Experiment Settings. Our implementation STLts is compared with the following tools: 1) a widely used optimization-based falsification tool Breach [11]; 2) another falsification tool ForeSee [1, 37] that emphasizes optimized treatment of Boolean connectives in STL; 3) an MILP-based STL optimal control tool bluSTL [14]; and 4) STLmc, an SMT-based bounded STL model checker [34].

The experiments were conducted on an Amazon EC2 c4.4xlarge instance (2.9 GHz Intel Xeon E502666 v3, 30.0GB RAM) running Ubuntu Server 20.04.

RQ1: the Effect of the Variability Bound N .

Fig. 7.

Execution time of STLts for different var. bd. N, on $\texttt{ISO6}$

There is an obvious trade-off about the choice of a variability bound N (Prob. 3.2): bigger N means the search is more extensive, but it incurs greater computational cost.

This tendency is confirmed in our experiments; the result for the $\texttt{ISO6}$ benchmark is in Fig. 7 for illustration. Here, synthesis was successful for $N=4$ for the first time.

We also observe in the figure that computational cost is low when trace synthesis is unsuccessful. This suggests the following strategy: we start with small N and increment it if trace synthesis is unsuccessful. We might waste time by trying too small N’s; but the wasted time should be small.

Table 2. Experimental results for trace synthesis, showing execution time (seconds). (N) for STLts is the first successful bound. Timeout (t/o) is 600 sec.

Experimental Results, Overview. Our experimental results are in summarized in Table 2, where the best performers are highlighted by color.

We explain the missing entries. In $(*)$, the tool is not applicable due to the nondeterminism of the benchmark. In $(\dagger )$, we did not conduct experiments since the performance comparison with STLts is already clear with simpler $\texttt{RNC}$ benchmarks. In $(\ddagger )$, bluSTL does not support multiple control modes. $(\P )$ is because bluSTL (at least its implementation available to us) does not support the until $\mathcal {U}$ modality.

Overall, our STLts is clearly the best performer in all benchmarks but one. The other tools time out, or takes tens of seconds. For our motivation of illustrating STL specs by trace synthesis in close interaction with users, tens of seconds is prohibitively long. The results adequately demonstrate satisfactory performance of our algorithm, in trace synthesis for complex STL specs.

Table 3. Comparison of our approach (STLts) with baselines (Breach, ForeSee, bluSTL, STLmc). Highlited cells represent positive features.

RQ2: Comparison with Other Approaches. A summary of comparison is in Table 3. The comparison with optimization-based falsification tools is as we expected—their struggle with complex specs motivated this work (§1). Boolean connectives in STL specs have been found problematic in falsification: this is called the scale problem [36, 37]. The results in Table 2 show that our benchmark specs are even beyond the capability of ForeSee, a tool that incorporates Monte Carlo tree search to specifically handle the scale problem. After all, one can say that falsification tools are aimed at complex models, while our STLts is aimed at complex specs.

STLmc has a similar (“dual”) scope and utilizes a similar technique (stable partitioning) to our STLts; the main difference is that STLmc is SMT-based while STLts is MILP-based. Therefore STLts accommodates a smaller class of models, but it can be faster on them exploiting numeric optimization. Table 2 suggests the advantage of STLts for common STL specs in manufacturing.

RQ3: Performance in Real-World Scenarios. For this RQ, we refer to STLts’s performance on the ISO benchmarks. Illustrating the specs $\texttt{ISO}i$ by trace synthesis is a real-world problem about safety standards for automated driving (§1), and Table 2 shows that STLts has sufficient performance and scalability to handle complex specs there (see (13)).

Fig. 8.

STLts for parameter synthesis. Red is execution time (axis left, seconds); blue is the maximum p (axis right). (Color figure online)

RQ4: Performance in Parameter Mining. We conducted parameter mining experiments with the $\texttt{ISO8}$ benchmark. Its specification has a subformula $\texttt{fasterThan}(SV,POV,p)$ that requires that SV’s velocity is bigger than POV’s by at least a parameter p. We used STLts to solve Prob. 3.3, that is, to find the maximum p for which a satisfying trace exists.

Figure 8 shows the results with varying variability bound N. Parameter mining is generally more expensive than trace synthesis. This is because the former has a nontrivial objective function (namely p in this example), while the latter does not (it is thus a constraint satisfaction problem). We observe the optimization with $N \ge 10$ resulted in a timeout. The tendency, much like in trace synthesis, is that the result (max p) improves but execution time gets larger as N becomes bigger (there are some exceptions such as $N=8,9$ though). Taking the same strategy as above (incrementing N), it takes roughly 10 min to obtain a largely converged value ($\sim 14.9$ for the maximum p). Overall, we believe this is a realistic performance for practical usage.