# Quantifying conformance using the Skorokhod metric

## Abstract

The conformance testing problem for dynamical systems asks, given two dynamical models (e.g., as Simulink diagrams), whether their behaviors are “close” to each other. In the semi-formal approach to conformance testing, the two systems are simulated on a large set of tests, and a metric, defined on pairs of real-valued, real-timed trajectories, is used to determine a lower bound on the distance. We show how the Skorokhod metric on continuous dynamical systems can be used as the foundation for conformance testing of complex dynamical models. The Skorokhod metric allows for both state value mismatches and timing distortions, and is thus well suited for checking conformance between idealized models of dynamical systems and their implementations. We demonstrate the robustness of the metric by proving a *transference theorem*: trajectories close under the Skorokhod metric satisfy “close” logical properties in the timed linear time logic FLTL (Freeze LTL) containing a rich class of temporal and spatial constraint predicates involving time and value freeze variables. We provide efficient window-based streaming algorithms to compute the Skorokhod metric for both piecewise affine and piecewise constant traces, and use these as a basis for a conformance testing tool for Simulink. We experimentally demonstrate the effectiveness of our tool in finding discrepant behaviors on a set of control system benchmarks, including an industrial challenge problem.

### Keywords

Trace metrics Conformance testing Signal temporal logic (STL) Skorokhod metric Timing distortion## 1 Introduction

A fundamental question in model-based design is *conformance testing*: whether two models of a system display similar behavior. For discrete systems, this question is well-studied [21, 22, 31, 32], and there is a rich theory of process equivalences based, e.g., on bisimilarity. For continuous and hybrid systems, however, the state of the art is somewhat unsatisfactory. While there is a straightforward generalization of process equivalences to the continuous case, in practice, equivalence notions such as bisimilarity are always too strong and most systems are not bisimilar. Since equivalence is a Boolean notion, one gets no additional information about the systems other than they are “not bisimilar.” Further, even if two dynamical systems are bisimilar, they may still differ in many control-theoretic properties. Thus, classical notions for equivalence and conformance have been of limited use in industrial practice.

In recent years, the notion of bisimulation has therefore been generalized to *metrics* on systems, which quantify the distance between them. For example, one approach is that of \(\epsilon \)-bisimulation, which requires that the states of the two systems remain “close” forever (within an \(\epsilon \)-ball), rather than coincide exactly. Under suitable stability assumptions on the dynamics, one can construct \(\epsilon \)-bisimulations [19, 20]. Unfortunately, proving the pre-requisites for the existence of \(\epsilon \)-bisimulations for complex dynamical models, or coming up with suitable and practically tractable bisimulation functions is extremely difficult in practice. In addition, establishing \(\epsilon \)-bisimulation requires full knowledge of the system dynamics making the scheme inapplicable where one system is an actual physical component with unknown dynamics. So, these notions have also been of limited industrial use so far.

*good*metric, with the following properties:

*Transference*Closeness in the metric must translate to preserving interesting classes of logical and functional specifications between systems, and*Tractability*The metric should be efficiently computable.

*usability*: the metric should classify systems which the engineers consider close as being close, and conversely.

The simplest candidate metric is a *pointwise* metric that computes the maximum pointwise difference between two trajectories, sometimes generalized to apply a constant time-shift to one trajectory [16]. Unfortunately, for many practical models, two trajectories may be close only under variable time-shifts. This is the case, for example, for two dynamical models that may use different numerical integration techniques (e.g., fixed step versus adaptive step) or when some component in the implementation has some jitter. Thus, the pointwise metric spuriously reports large distances for “close” models. More nuanced hybrid distances have been proposed [1], but the transference properties of these metrics w.r.t. common temporal logics are not yet clear.

*Skorokhod*metric [14]. The Skorokhod metric allows for mismatches in both the trace values and in the timeline, and quantifies temporal and spatial variation of the system dynamics under a unifying framework. The distortion of the timeline is specified by a

*retiming*function \({{\mathrm{\mathsf {r}}}}\) which is a continuous bijective strictly increasing function from \({\mathbb {R}}_+\) to \({\mathbb {R}}_+\). Using the retiming function, we obtain the

*retimed trace*\(x\left( {{\mathrm{\mathsf {r}}}}(t)\right) \) from the original trace

*x*(

*t*). Intuitively, in the retimed trace \(x\left( {{\mathrm{\mathsf {r}}}}(t)\right) \), we see exactly the same values as before, in exactly the same order, but the time duration between two values might now be different than the corresponding duration in the original trace. The amount of distortion for the retiming \({{\mathrm{\mathsf {r}}}}\) is given by \(\sup _{t\ge 0} \left| {{\mathrm{\mathsf {r}}}}(t)-t\right| \). Using retiming functions, the Skorokhod distance between two traces

*x*and

*y*is defined to be the least value over all possible retimings \({{\mathrm{\mathsf {r}}}}\) of:

*timing discrepancy*of the timing distortion required to “match” two traces, and the second quantifies the

*value mismatch*(in the metric space \({\mathscr {O}}\)) of the values under the timing distortion. The Skorokhod metric was introduced as a theoretical basis for defining the semantics of hybrid systems by providing an appropriate hybrid topology [10, 11]. We now demonstrate its usefulness in the context of conformance testing.

*Transference* We show that the Skorokhod metric gives a robust quantification of system conformance by relating the metric to TLTL (timed LTL) in the boolean setting; and to FLTL (Freeze LTL) in the real-valued signal setting. The logics contain (i) predicates of the form \(f(x_1,\dots , x_n) \ge 0\), as in Signal Temporal Logic (STL) [16] (we however allow *f* to be non-linear), for specifying constraints on trace values; and (ii) *freeze quantifiers*, as in TPTL [4] and STL\(^*\) [8, 9]^{1} for specifying temporal and value constraints relating different parts of traces (freeze quantifiers can express more complex timing constraints than bounded timing constraints, e.g., of MTL, and can also specify complex value constraints such as signal tracking which cannot be expressed by STL). TLTL subsumes MTL in the boolean setting; and FLTL subsumes STL in the real-valued setting. We prove a *transference theorem*: flows (and propositional traces) which are close under the Skorokhod metric satisfy “close” FLTL (resp. TLTL) formulae for a rich class of temporal and spatial predicates, where the untimed structure of the formulae remains unchanged, only the predicates are enlarged.

*Tractability* We improve on recent polynomial-time algorithms for the Skorokhod metric [27] between polygonal (piecewise affine and continuous) traces by taking advantage of the fact that, in practice, only retimings that map the times in one trace to “close” times in the other are of interest. This enables us to obtain a streaming sliding-window based monitoring procedure which takes only *O*(*W*) time per sample, where *W* is the window size (assuming the dimension *n* of the system to be a constant). In this work, we also develop and implement a significantly faster sliding-window based Skorokhod metric computation procedure for *piecewise constant* traces, and experimentally compare the tradeoff between faster computation time of the piecewise constant routine and the discrepancy compared to the polygonal trace procedure.

*Usability* Using the Skorokhod distance checking procedure as a subroutine, we have implemented a Simulink toolbox for conformance testing. Our tool integrates with Simulink’s model-based design flow for control systems, and provides a stochastic search-based approach to find inputs which maximize the Skorokhod distance between systems under these inputs.

We present three case studies from the control domain, including industrial challenge problems; our empirical evaluation shows that our tool computes sharp estimates of the conformance distance reasonably fast on each of them. Our input models were complex enough that techniques such as \(\epsilon \)-bisimulation functions are inapplicable. We conclude that the Skorokhod metric can be an effective foundation for semi-formal conformance testing for complex dynamical models.

*Related work* The work of [1, 2] is closely related to ours. In it, robustness properties of hybrid state sequences are derived with respect to a trace metric which also quantifies temporal and spatial variations. Our work differs in the following ways. First, we guarantee robustness properties over *flows* rather than only over (discrete) sequences. Second, the Skorokhod metric is a stronger form of the \((T,J,(\tau , \epsilon ))\)-closeness degree^{2}\(^{,}\)^{3}(for systems which do not have hybrid time); and allows us to give stronger robustness transference guarantees. The Skorokhod metric requires order preservation of the timeline, which the \((T,J,(\tau , \epsilon ))\)-closeness function does not. Preservation of the timeline order allows us to (i) keep the untimed structure of the formulae the same (unlike in the transference theorem of [1]); (ii) show transference of a rich class of global timing constraints using freeze quantifiers (rather than only for the standard bounded time quantifiers of MTL/MITL). However, for implementations where the timeline order is not preserved, we have to settle for the less stronger guarantees provided by [1]. The work of [16] deals with spatial robustness of STL; the only temporal disturbances considered are constant time-shifts for the entire signal where the entire signal is moved to the past, or to the future by the same amount. In contrast, the Skorokhod metric incorporates variable time-shifts.

*Summary of Results*The present work contains the following results.

- 1.
We develop an efficient algorithm for computing the Skorokhod distance between piecewise constant traces (for a general metric space) that may have discontinuities.

- 2.
We implement the above piecewise constant trace algorithm for \({\mathbb {R}}^n\), and also the polygonal trace algorithm from [27], and compare the two implementations. Our piecewise constant trace algorithm implementation runs two orders of magnitude faster than the polygonal trace algorithm procedure. We also experimentally explore the tradeoff between faster computation time of the piecewise constant routine and the distance values obtained compared to the polygonal procedure.

- 3.
We introduce the expressive logic FLTL (based on the logic STL* ([8, 9]), which augments LTL with

*value*and*time*freeze variables, and general constraint predicates. We prove a logic transference result with respect to this logic: flows which are close under the Skorokhod metric satisfy “close” FLTL formulae. Value and time freeze variables allow expression of complex properties of traces, such as signal tracking ([8, 9]). - 4.
We present detailed examples explaining and demonstrating our logic transference guarantees.

- 5.
We present a new application of the Skorokhod metric: we develop an algorithm to quantify the

*timing distortion*between traces under some allowed value distortion. More precisely, given polygonal (or piecewise constant) traces*x*,*y*, and an \(\epsilon \ge 0\), we develop an algorithm to compute the minimal retiming \({{\mathrm{\mathsf {r}}}}\) required in order that \(x\left( r(t)\right) \) and*y*(*t*) match, modulo \(\epsilon \), for all*t*. More precisely, we compute the retiming which minimizes the quantity \(\sup _{t\in [0,T]}\left| {{\mathrm{\mathsf {r}}}}(t)-t\right| \) (this value quantifies the timing distortion under \({{\mathrm{\mathsf {r}}}}\)) such that \(\sup _{t\in [0, T]} \mathscr {D}\left( x\left( r(t)\right) , \, y(t)\right) \ \le \epsilon \). We also implement the algorithm and present experimental results.

## 2 Conformance testing with the Skorokhod metric

### 2.1 Systems and conformance testing

*Traces and systems* A (finite) *trace* or a *signal*\(\pi \,{:}\,[T_i,T_e] \rightarrow {\mathscr {O}} \) is a mapping from a finite closed interval \([T_i,T_e]\) of \({\mathbb {R}}_+\), with \(0 \le T_i < T_e\), to some topological space \({\mathscr {O}}\). If \(\mathscr {O}\) is a metric space, we refer to the associated metric on \(\mathscr {O}\) as \(\mathscr {D}_{\mathscr {O}}\). The time-domain of \(\pi \), denoted \({{\mathrm{\mathsf {tdom}}}}(\pi )\), is the time interval \([T_i,T_e]\) over which it is defined. The time-duration of \(\pi \), denoted \({{\mathrm{\mathsf {tlen}}}}(\pi )\), is \(T_e-T_i\). The *t*-suffix of \(\pi \) for \(t\in {{\mathrm{\mathsf {tdom}}}}(\pi )\), denoted \(\pi ^t\), is the trace \(\pi \) restricted to the interval \([t, \max {\mathsf {tdom}} (\pi )]\). We denote by \(\pi _{\downarrow T'_e}\) the prefix trace obtained from \(\pi \) by restricting the domain to \([T_i,T'_e]\subseteq {{\mathrm{\mathsf {tdom}}}}(\pi )\).

A (continuous-time) *system*\(\mathfrak {A}: \left( {\mathbb {R}}_+^{{[]}}\mapsto {\mathscr {O}}_{{{\mathrm{\mathsf {ip}}}}}\right) \, \rightarrow \, \left( {\mathbb {R}}_+^{{[ ]}}\rightarrow {\mathscr {O}}_{{{\mathrm{\mathsf {op}}}}}\right) \), where \( {\mathbb {R}}_+^{{[]}}\) is the set of finite closed intervals of \({\mathbb {R}}_+\), transforms input traces \(\pi _{{{\mathrm{\mathsf {ip}}}}}: [T_i,T_e] \rightarrow {\mathscr {O}}_{{{\mathrm{\mathsf {ip}}}}} \) into output traces \(\pi _{{{\mathrm{\mathsf {op}}}}}: [T_i,T_e] \rightarrow {\mathscr {O}}_{{{\mathrm{\mathsf {op}}}}} \) (over the same time domain). We require that the system is *causal*: if \(\mathfrak {A}(\pi _{{{\mathrm{\mathsf {ip}}}}}) \rightarrow \pi _{{{\mathrm{\mathsf {op}}}}}\), then for every \(\min {{\mathrm{\mathsf {tdom}}}}(\pi ) \le T_e' < \max {{\mathrm{\mathsf {tdom}}}}(\pi )\), the system \(\mathfrak {A}\) maps \({\pi _{{{\mathrm{\mathsf {ip}}}}}}_{\downarrow T'_e} \) to \({\pi _{{{\mathrm{\mathsf {op}}}}}}_{\downarrow T'_e} \). Common examples of such systems are (causal) dynamical and hybrid dynamical systems [7, 34].

*Conformance testing* Let \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\) be systems and let \(\mathscr {D}_{{{\mathrm{{\mathcal {T}\!\mathcal {R}}}}}}\) be a metric over output traces. For a set \(\Pi _{{{\mathrm{\mathsf {ip}}}}}\) of input traces, we define the *(quantitative) conformance* between \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\) w.r.t. \(\Pi _{{{\mathrm{\mathsf {ip}}}}}\) as \(\sup _{\pi _{{{\mathrm{\mathsf {ip}}}}}\in \Pi _{{{\mathrm{\mathsf {ip}}}}}} \mathscr {D}_{{{\mathrm{{\mathcal {T}\!\mathcal {R}}}}}}\left( \mathfrak {A}_1\left( \pi _{{{\mathrm{\mathsf {ip}}}}}\right) , \mathfrak {A}_2\left( \pi _{{{\mathrm{\mathsf {ip}}}}}\right) \right) \) The conformance between \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\) is their conformance w.r.t. the set of all input traces.

Algorithm 1 is a standard optimization-guided adaptive testing algorithm. To define the set \(\Pi _{ test }\) of test inputs, we use a fixed finite parameterization of the input space using a finite set *F* of *basis functions* and fix a time horizon *T*. We only generate inputs obtained as a linear combination \(\sum _{f \in F} p_f \cdot f\) of basis functions over the interval [0, *T*], where the coefficients \(\{ p_f \mid f\in F \}\) come from a closed convex subset of \({\mathbb {R}}^{|F|}\).

In each step, Algorithm 1 picks an input signal *u* and computes the distance between the corresponding outputs \(y_1 = \mathfrak {A}_1(u)\) and \(y_2 = \mathfrak {A}_2(u)\). Based on heuristics that rely on the current distance, and a possibly bounded history of costs, the procedure then picks a new value for *u* by choosing new values for the coefficients \(\{ p_f\mid f\in F \}\). For instance, in a gradient-ascent based procedure, the new value of *u* is chosen by estimating the local gradient in each direction in the input-parameter space, and then picking the direction that has the largest (positive) gradient. In our implementation, we use the Nelder-Mead (or nonlinear simplex) algorithm to pick new inputs.

On termination (e.g., when some maximum number of iterations is reached), the algorithm returns the conformance distance between \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\) w.r.t. the set of tests generated. One can compare the distance to some tolerance \(\delta \) chosen based on engineering requirements.

*Sampling schemes and resulting interpolated traces* In practice, the output behaviors of the systems are observed with a sampling process, thus \(y_1\) and \(y_2\) on line 4 are discrete time-sampled *sequences*. We go from these sequences to output traces either by linear interpolation between the sampled time points, or by assuming a constant value in between the sampled values in a sample-and-hold scheme.

In the first case, we get a polygonal (piecewise linear) trace. Formally, a *polygonal trace*\(\pi : I_{\pi } \rightarrow \mathscr {O}\) where \(\mathscr {O}\) is a vector space with the scalar field \({\mathbb {R}}\) is a continuous trace such that there exists a finite sequence \(\min I_{\pi }= t_0< t_1< \dots < t_m = \max I_{\pi }\) of time-points such that the trace segment between \(t_k\) and \(t_{k+1}\) is affine for all \(0\le k < m\), i.e., for \(t_k \le t \le t_{k+1}\) we have \(\pi (t) = \pi (t_k) + \frac{t- t_k}{t_{k+1}-t_k}\!\cdot \!\left( \pi ( t_{k+1}) - \pi (t_k)\right) \).

In the sample-and-hold case, we get a *piecewise constant* trace \(\pi : I_{\pi } \rightarrow \mathscr {O}\) for which there exists a finite sequence \(\min I_{\pi }= t_0< t_1< \cdots < t_m = \max I_{\pi }\) of time-points such that \(\pi \) is constant over the left-closed right-open intervals \([t_i, t_{i+1})\) for \(0\le i<m\), and constant over \([t_{m-1},\, \max I_{\pi }]\) (the last interval is also right closed). A piecewise constant trace is right continuous, but need not be left continuous.

### 2.2 The Skorokhod metric

We now define the Skorokhod metric, which we use as the metric in Algorithm 1.

A *retiming*\({{\mathrm{\mathsf {r}}}}: I \rightarrow I' \), for closed intervals \(I, I'\) of \({\mathbb {R}}_+\) is an order-preserving (i.e., monotone strictly-increasing) continuous bijective function from *I* to \(I'\); thus if \(t<t'\) then \({{\mathrm{\mathsf {r}}}}(t) < {{\mathrm{\mathsf {r}}}}(t')\). Let \({{\mathrm{\mathsf {R}}}}_{I \rightarrow I' }\) be the class of retiming functions from *I* to \( I' \) and let \({{\mathrm{\mathcal {I}}}}\) be the identity retiming. Intuitively, retiming can be thought of as follows: imagine a stretchable and compressible timeline; a retiming of the original timeline gives a new timeline where some parts have been stretched, and some compressed, without the timeline having been broken. Given a trace \(\pi : I_{\pi } \rightarrow {\mathscr {O}} \), and a retiming \({{\mathrm{\mathsf {r}}}}: I \rightarrow I_{\pi } \); the function \(\pi \circ {{\mathrm{\mathsf {r}}}}\) is another trace from *I* to \({\mathscr {O}}\).

### Definition 1

*Skorokhod metric*) Given a retiming \({{\mathrm{\mathsf {r}}}}: I \rightarrow I' \), let \(||{{\mathrm{\mathsf {r}}}}-{{\mathrm{\mathcal {I}}}}||_{\sup }\) be defined as \( ||{{\mathrm{\mathsf {r}}}}-{{\mathrm{\mathcal {I}}}}||_{\sup } =\sup _{t\in I }|{{\mathrm{\mathsf {r}}}}(t)-t|\). Given two traces \(\pi : I_{\pi }\rightarrow {\mathscr {O}} \) and \(\pi ': I_{\pi '} \rightarrow {\mathscr {O}} \), where \({\mathscr {O}}\) is a metric space with the associated metric \( \mathscr {D}_{{\mathscr {O}}}\), and a retiming \({{\mathrm{\mathsf {r}}}}: I_{\pi } \rightarrow I_{\pi '}\), let \(\left\| \pi \,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}\right\| _{\sup }\) be defined as:

*Skorokhod distance*

^{4}between the traces \(\pi ()\) and \(\pi '()\) is defined to be:

Intuitively, the Skorokhod distance incorporates two components: the first component quantifies the *timing discrepancy* of the timing distortion required to “match” two traces, and the second quantifies the *value mismatch* (in the metric space \({\mathscr {O}}\)) of the values under the timing distortion. In the retimed trace \( \pi \circ {{\mathrm{\mathsf {r}}}}\), we see exactly the same values as in \(\pi \), in exactly the same order, but the times at which the values are seen can be different.

### 2.3 Skorokhod metric computation: piecewise constant traces

In this subsection we derive a procedure for computing the Skorokhod distance between piecewise constant traces. The outline of the subsection is as follows. First in Lemma 1, we show that we can use non-decreasing functions as retimings (instead of allowing only strictly increasing functions). Then, in Lemmas 2 and 3, we show that for piecewise constant traces, the values of \(\sup \nolimits _{t\in I_{\pi }} \mathscr {D}_{{\mathscr {O}}}\big (\, \pi (t)\ ,\ \pi '\left( {{\mathrm{\mathsf {r}}}}(t)\right) \, \big )\) and \(\sup _{t\in I }|{{\mathrm{\mathsf {r}}}}(t)-t|\) used in the definition of the Skorokhod metric (Definition 1) can be determined by looking at a *finite* set of timepoints. Using the this result, we then obtain a class of \(\epsilon \)-optimal retimings in Lemma 4. Finally, using Lemmas 3 and 4, we devise a dynamic programming algorithm to compute the Skorokhod distance between piecewise constant traces. The main result of this subsection is stated in Theorem 1. The reader may skip directly to Theorem 1 without hampering readability of the rest of the paper.

We assume for simplicity of presentation that both \(\pi , \pi '\) contain *m* segments, are over the same intervals, and moreover that both traces are the result of sampling at the same time instants (the inter-sample time duration may be variable). As a first step, we prove the retimings can be *non-decreasing* and onto, rather than monotone strictly-increasing and bijective

### Lemma 1

*non-decreasing*and onto retimings are allowed. That is, for \(\pi , \pi '\) piecewise constant traces, we have:

### Proof

*I*intervals, and strictly increasing over the

*J*intervals (\(J_0\) or \(J_a\) may be empty, but other

*J*intervals are non-empty). Fix \(\epsilon > 0\). Consider \(I_0\). Since \(I_{\pi }\) and \(I_{\pi '}\) contain more than one time-point, and and \({{\mathrm{\mathsf {r}}}}\) is onto, at least one of \(J_0, J_1\) is non-empty. We can show that we can “wiggle” the retiming \({{\mathrm{\mathsf {r}}}}\) to get another retiming \({{\mathrm{\mathsf {r}}}}_0\) such that

- 1.
\({{\mathrm{\mathsf {r}}}}_0\) is monotone increasing over \(J_0, I_0, J_1\) and \(J_k\) for \(k\ge 2\).

- 2.
\({{\mathrm{\mathsf {r}}}}_0\) is equal to \({{\mathrm{\mathsf {r}}}}\) over \(I_1, J_2, I_2, \dots , J_a\).

- 3.
\(\left\| {{\mathrm{\mathsf {r}}}}_0-{{\mathrm{\mathsf {r}}}}\right\| _{\sup } < \epsilon \) (over \(J_0, I_0, J_1\)).

- 4.
\(\pi '\circ {{\mathrm{\mathsf {r}}}}(t) = \pi '\circ {{\mathrm{\mathsf {r}}}}_0(t)\) for all \(t\in I_{\pi }\), which implies that \(\left\| \pi \,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}\right\| _{\sup } = \left\| \pi \,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}_0\right\| _{\sup }\).

*r*a little bit so that it becomes monotone increasing over \(I_0\), and the perturbation does not affect the trace matchings between \(\pi \) and \(\pi '\) under the retimings.

- 1.
\({{\mathrm{\mathsf {r}}}}_e: I_{\pi }\rightarrow I_{\pi '}\) is monotone strictly increasing and bijective.

- 2.
\(\left\| {{\mathrm{\mathsf {r}}}}_e-{{\mathrm{\mathsf {r}}}}\right\| _{\sup } < \epsilon \).

- 3.
\(\pi '\circ {{\mathrm{\mathsf {r}}}}(t) = \pi '\circ {{\mathrm{\mathsf {r}}}}_e(t)\) for all \(t\in I_{\pi }\), which implies that \(\left\| \pi \,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}\right\| _{\sup } = \left\| \pi \,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}_e\right\| _{\sup }\).

We now show that when computing the value of Eq. (2), given a retiming \({{\mathrm{\mathsf {r}}}}\), we only need to look at the values of \({{\mathrm{\mathsf {r}}}}(t)-{{\mathrm{\mathcal {I}}}}(t)\) and \(\pi (t)\,-\, \pi '\circ {{\mathrm{\mathsf {r}}}}(t)\) at a finite set of timepoints \(t\in \{ t_0, t_1, \dots , t_{m} \}\). For this, we need the following lemma.

### Lemma 2

### Proof

- 1.
\({{\mathrm{\mathsf {r}}}}^{\dagger }(t) = {{\mathrm{\mathsf {r}}}}(t)\) for \(t\in \{ t_0, t_1, \dots , t_{m} \}\), and

- 2.
\(\sup _{t\in I_{\pi }}\left| {{\mathrm{\mathsf {r}}}}(t)-{{\mathrm{\mathcal {I}}}}(t)\right| \)\(\ge \sup _{t\in I_{\pi }}\left| {{\mathrm{\mathsf {r}}}}^{\dagger }(t)-{{\mathrm{\mathcal {I}}}}(t)\right| \), and

- 3.
\(\sup _{t\in I_{\pi }}\left| {{\mathrm{\mathsf {r}}}}^{\dagger }(t)-{{\mathrm{\mathcal {I}}}}(t)\right| \ = \ \max \sup _{t\in \{ t_0, t_1, \dots , t_{m} \}}\left| {{\mathrm{\mathsf {r}}}}^{\dagger }(t)-{{\mathrm{\mathcal {I}}}}(t)\right| \).

### Lemma 3

\(\epsilon \)-*optimal retiming functions* Finally, we construct \(\epsilon \)-optimal retiming functions in the minimization of Eq. (5) (in general, optimal retiming functions need not exist as Eq. (5) still has an “\(\inf \)” over retimings). Fix \(\epsilon \) such that \(\min _{0\le k< m} (t_{k+1}-t_k)> \epsilon > 0\). Consider a class \(C_{\epsilon }\) of non-decreasing retiming functions such that for \({{\mathrm{\mathsf {r}}}}\in C_{\epsilon }\), for all *k*, there is some *j* such that \({{\mathrm{\mathsf {r}}}}(t_k) = t_j\), or \(t_j-\epsilon \). That it, the retiming \({{\mathrm{\mathsf {r}}}}\) maps the trace change timepoints \(t_k\) to other change time-points, or to just before other change timepoints. The following lemma shows that we can restrict retimings to belong to this class (the proof can be found in the appendix).

### Lemma 4

Lemma 4 shows that we can search over a finite space of retiming functions to compute \( \mathscr {D}_{\mathscr {S}}(\pi ,\pi ')\). The range of the retiming functions, over the set \(\{ t_0, t_1, \dots , t_m \}\) can be restricted to \(\{ t_0, t^-_1, t_1, t_2^-, t_2, \dots , t_{m-1}^-, t_{m-1}, t_m \}\), where \(t_k^-\) is the timepoint arbitrarily close to (and less than) \(t_k\). Using this fact, we design a dynamic programming algorithm to compute \( \mathscr {D}_{\mathscr {S}}(\pi ,\pi ')\) below.

*time-explicit*trace Open image in new window defined by Open image in new window, and let Open image in new window. It can be checked that Eq. (7) can be expressed asAnd thus, Eq. (6) as:Since the retimings can be restricted to \(\{ t_0, t^-_1, t_1, t_2^-, t_2, \dots , t_{m-1}^-, t_{m-1}, t_m \}\) (as mentioned previously), we solve the above problem (8) as follows. In order to simplify notation, let \( x_0, x_1, x_2, \dots , x_{2m}\) be defined as

*M*(

*i*,

*j*) for \(j>i\) denote the fact that point \(y_j\) is mapped to \(x_i\) (and possibly earlier

*x*points);

*M*(

*i*,

*j*) for \(i>j\) denote the fact that point \(y_j\) is mapped to \(x_i\) (and possibly later

*x*points); and

*M*(

*i*,

*i*) denote the case when point \(y_i\) is mapped to \(x_i\). For example,

*M*(2, 2

*m*) denotes the fact that point \(y_{2m}\) has been moved “left” via retiming to match \(x_2\) (and perhaps an earlier point). Note that this retiming has the effect of “stretching” \(y_{2m}\) – the single point \(y_{2m}\) now matches the

*x*segment \([x_2, x_{2m}]\). The recurrence relation for

*M*is set up as follows.(in case \(i-1\) or \(j-1\) is negative, the \(\min \) omits those lines). The base condition is given by Open image in new window. The correctness of the recurrence relation follows from the fact that either

The portion \([y_0,\, y_{j-1}]\) is mapped to \([x_0, x_{i-1}]\), and the portion \(( y_{j-1}, \, y_j]\) mapped to \(( x_{i-1}, \, x_i]\); or

The portion \([y_0,\, y_{j-1}]\) is mapped to \([x_0, x_{i}]\), and the portion \(( y_{j-1}, \, y_j]\) mapped to the single point \(x_i\); or

The portion \([y_0,\, y_j]\) is mapped to \([x_0, x_{i-1}]\), and the portion \(( x_{i-1}, \, x_i]\) mapped to the single point \(y_j\).

*M*(2

*m*, 2

*m*) computes \(\mathscr {D}_{\mathscr {S}}(\pi ,\pi ')\). A similar algorithm with more bookkeeping applies when \(\pi , \pi '\) sample points are at different time instants. Thus, we get the following theorem.

### Theorem 1

- 1.
We can compute \(\mathscr {D}_{\mathscr {S}}^L (\pi , \pi ') \) in time \(O\left( m_{\pi }\!\cdot \!m_{\pi '}\!\cdot \!n\right) \).

- 2.
Suppose we restrict retimings to be such that the

*i*-th constant segment of \(\pi \) can only be matched to \(\pi '\) constant segments \(i-W\) through \(i+W\) for all*i*, where \(W\ge 1\). Under this retiming restriction, we can compute \(\mathscr {D}_{\mathscr {S}}^L (\pi , \pi ')\) with a streaming algorithm in time \(O\left( \left( m_{\pi }+ m_{\pi '}\right) \!\cdot \!n\!\cdot \!W\right) \).\(\square \)

### 2.4 Skorokhod metric computation: polygonal traces

We devised an algorithm to compute the Skorokhod distance between polygonal traces in an earlier work [26]. Note after retiming, the retimed version \(\pi \circ {{\mathrm{\mathsf {r}}}}\) of a polygonal trace \(\pi \) need not be polygonal (see [26]), in spite of this, the algorithm is polynomial time.

### Theorem 2

- 1.
Given \(\delta \ge 0\), it can be checked whether \(\mathscr {D}_{\mathscr {S}} (\pi , \pi ') \le \delta \) in time \(O\left( m_{\pi }\!\cdot \!m_{\pi '}\!\cdot \!n\right) \).

- 2.
Suppose we restrict retimings to be such that the

*i*-th affine segment of \(\pi \) can only be matched to \(\pi '\) affine segments \(i-W\) through \(i+W\) for all*i*, where \(W\ge 1\). Under this retiming restriction, we can determine, with a streaming algorithm, whether \(\mathscr {D}_{\mathscr {S}} (\pi , \pi ') \le \delta \) in time \(O\left( \left( m_{\pi }+ m_{\pi '}\right) \!\cdot \!n\!\cdot \!W\right) \).\(\square \)

Let us denote by \(\mathscr {D}_{\mathscr {S}}^W (\pi , \pi ') \) the Skorokhod difference between \(\pi , \pi '\) under the retiming restriction of the second part of Theorem 2, i.e., the value obtained by restricting the retimings in Eq. (1).^{5} The value \( \mathscr {D}_{\mathscr {S}}^W (\pi , \pi ') \) is an upper bound on \( \mathscr {D}_{\mathscr {S}} (\pi , \pi ') \). In addition, for \(W' < W\), we have \( \mathscr {D}_{\mathscr {S}}^W (\pi , \pi ') \le \mathscr {D}_{\mathscr {S}}^{W'} (\pi , \pi ') \). These results on the distances computed under the window restriction *W* also apply to piecewise constant traces (Theorem 1).

*Computing the distance value between polygonal traces* A polynomial time algorithm was presented in [27] to *compute* the distance value using the distance monitoring routine. However, even though polynomial time, and even under a sliding-window restriction, the time complexity of the algorithm is unsatisfactory for use in conformance testing—a careful analysis shows that that the time complexity of the algorithm for determining the distance value using the procedure of [27] for a window size *W* is \(O\left( \left( m_{\pi }+ m_{\pi '}\right) \cdot n\cdot W^2\cdot \log (W)\right) \). In practice, it turned out to be more efficient to do a binary search employing the monitoring routine to obtain the distance value; we observed around 7 binary search calls on average (the typical window size was 100).

*k*. It can be checked that the pointwise distance between \(\pi \), and the corresponding piecewise constant trace \( \pi _{{{\mathrm{\mathsf {pwc}}}}}\), defined as

### Proposition 1

*W*retiming restriction lies in the interval \([\alpha _{\min }, \alpha _{\max }]\), where

Note that \( \mathscr {D}_{\sup }\!\left( \pi , \pi _{{{\mathrm{\mathsf {pwc}}}}}\right) \) and (similarly for \(\pi '\)) can be computed, for \(\mathscr {O}={\mathbb {R}}^n\) with norm \(L_1, L_2\) or \(L_{\infty }\), in time \(O(m\cdot n)\) (where *m* is the number of affine segments in \(\pi \)).

## 3 Transference of logical properties

In this section, we demonstrate a transference result involving the Skorokhod metric for the linear time logic FLTL (Freeze LTL) — a logic which augments LTL with *freeze* quantifiers [4] over both time and trace values. The logic we consider generalizes MTL, STL, TLTL [4], and STL\(^*\) [8, 9]. We show that if the Skorokhod distance between two traces is small, they satisfy close FLTL formulae. Given a formula \(\phi \) of FLTL satisfied by trace \(\pi _1\), we can compute a “relaxation” of \(\phi \) that will be satisfied by the “close” trace \(\pi _2\).

We first present the results in a propositional framework for the logic TLTL (obtained by augmenting LTL with freeze quantifiers over time), and then extend to \({\mathbb {R}}^n\)-valued spaces for the logic FLTL which also has freeze quantifiers over \({\mathbb {R}}^n\)-valued variables.

### 3.1 The logic TLTL

Let \(\mathcal {P}\) be a set of propositions. A *propositional trace*\(\pi \) over \(\mathcal {P}\) is a trace where the topological space is \(2^{\mathcal {P}}\), with the associated metric \(\mathscr {D}_{\mathcal {P}}(\sigma , \sigma ') = 0\) if \(\sigma = \sigma '\), and \(\infty \) otherwise, for \(\sigma ,\sigma '\in 2^{\mathcal {P}}\). The set of all timed propositional traces over \(\mathcal {P}\) is denoted by \(\Pi (\mathcal {P})\). Note that if a trace has finite variability, i.e., if there exists a finite partition of \({{\mathrm{\mathsf {tdom}}}}(\pi )\) into disjoint subintervals \(I_0, I_1, \dots , I_m\) such that \(\pi \) is constant on each subinterval, and in addition if \(I_0, I_1, \dots , I_{m-1}\) are left-closed and right open (with \(I_m\) closed on both sides), then the propositional trace \(\pi \) can be viewed as a trace obtained under a sample-and-hold scheme from a finite set of sample points.

### Definition 2

*TLTL*\((\mathcal {F}_{\mathsf {T}})\)

*Syntax*) Given a set of propositions \(\mathcal {P}\), a set of (time) variables \(V_{\mathsf {T}}\), and a set \(\mathcal {F}_{\mathsf {T}}\) of functions from \({\mathbb {R}}_+^l\) to \({\mathbb {R}}\), the formulae of TLTL(\(\mathcal {F}_{\mathsf {T}}\)) are defined by the following grammar.

\(p\in \mathcal {P}\) and \(x\in V_{\mathsf {T}}\), and \(\overline{x} = (x_1, \dots , x_l)\) with \(x_i\in V_{\mathsf {T}}\) for all \(1\le i \le l\);

\(f_{\mathsf {T}} \in \mathcal {F}_{\mathsf {T}}\) is a real-valued function, and \(\sim \) is one of \( \{ \le , <, \ge , > \}\). \(\square \)

The quantifier “*x*.” is known as the *freeze quantifier*, and binds variable *x* to the current time. A variable *x* is defined to be *free* in \(\phi \) as follows. The variable *x* is *not* free in \( x.\Psi \), or in *p* (a proposition), or in \({\textsc {true}}\), or in \( f_{\mathsf {T}}(x_1, \dots , x_l) \sim 0\) where \(x_i\ne x\) for all *i*. It is also not free in \(\phi \) if \(\phi \) does not contain an occurrence of *x*. It is free in \(\lnot \psi \) iff *x* is free in \(\psi \); and it is free in \( \phi _1 \mathrel {\begin{array}{c} \wedge \\ \vee \end{array}}\phi _2\), or in \( \phi _1 {\,\mathcal{U}\,}\phi _2\), iff *x* is free in either \(\phi _1\) or in \(\phi _2\). Finally, variable *x* is free in \( f_{\mathsf {T}}(x_1, \dots , x_l) \sim 0\) if some \(x_i\) is *x*. A formula is *closed* if it has no free variables.

### Definition 3

*TLTL*\((\mathcal {F}_{\mathsf {T}})\)

*Semantics*) Let \(\pi : I \rightarrow 2^{\mathcal {P}}\) be a timed propositional trace, and let \(\mathcal {E}: V_{\mathsf {T}} \mapsto I\) be the time environment mapping the variables in \(V_{\mathsf {T}}\) to time values in

*I*. The satisfaction of the trace \(\pi \) with respect to the TLTL(\(\mathcal {F}_{\mathsf {T}})\) formula \(\phi \) in the time environment \(\mathcal {E}\) is written as \(\pi \models _{\mathcal {E}} \phi \), and is defined inductively as follows (denoting \(t_0 = \min {{\mathrm{\mathsf {tdom}}}}(\pi )\)).A timed trace \(\pi \) is said to satisfy the closed formula \(\phi \) (written as \( \pi \models \phi \)) if there is some environment \(\mathcal {E}\) such that \( \pi \models _{\mathcal {E}} \phi \). \(\square \)

We define additional temporal operators in the standard way: the “eventually” operator \(\Diamond \phi \) stands for \({\textsc {true}}{\,\mathcal{U}\,}\phi \); and the “always” operator \(\Box \phi \) stands for \(\lnot \Diamond \lnot \phi \). TLTL\((\mathcal {F}_{\mathsf {T}})\) provides a richer framework than MTL [25] for expressing timing constraints as: (i) freeze quantifiers allow specification of constraints between distant contexts, which the bounded temporal operators in MTL cannot do; and (ii) the predicates \(f_{\mathsf {T}}() \sim 0\) for \(f_{\mathsf {T}}\in \mathcal {F}_{\mathsf {T}}\) allow the specification of complex timing requirements not expressible in MTL. Note that even if the predicates \(f_{\mathsf {T}}() \sim 0\) are restricted to be of the form \(x_1-x_2 +c \sim 0\), where \(x_1, x_2\) are freeze variables, and *c* is a constant, TLTL\((\mathcal {F}_{\mathsf {T}})\) is more expressive than MTL [6] (and hence more expressive than MITL on which STL is based).

### Example 1

*Freeze quantification*) Suppose we want to express that whenever the event

*Q*occurs, it is followed later by

*R*, and then by

*S*, such that the time difference between occurrences of

*Q*and

*R*is at most 5, and also the time difference between occurrences of

*Q*and

*S*is at most 10. This can be expressed in TLTL\((\mathcal {F}_{\mathsf {T}})\) as

### Example 2

(*Freeze quantification functions*) Suppose we want to express that whenever the event *Q* occurs, it must be followed by a response *R* within time \(\lambda ^{t_Q}\) for some \(\lambda > 1\) where \(t_Q\) is the time at which *Q* occurred; thus, the later *Q* occurs the more delay we can tolerate in the response time. The requirement can be expressed as \(x. \left( Q \rightarrow \Diamond \big (y. \left( R \wedge \left( y-x\, \le \lambda ^x\right) \right) \big )\right) \). \(\square \)

### Example 3

*TLTL*\((\mathcal {F}_{\mathsf {T}})\)

*subsumes MTL*) Let \(\mathcal {F}_{\mathsf {T}}\) be the set of two variable functions of the form \(f(x,y) = x-y+c\) where

*c*is a rational constant. Then TLTL\((\mathcal {F}_{\mathsf {T}})\) subsumes MTL. The MTL formula \(p {\,\mathcal{U}\,}_{[a,b]} q\) can be written as

*x*, and some future time \(t_y\) to the variable

*y*. The values \(t_x\) and \(t_y\) are such that at time \(t_y\), we have

*q*to be true, and moreover, at all times between \(t_x\) and \(t_y\), we have \(p\vee q\) to be true. Furthermore, \(t_y\) must be such that \(t_y\in [t_x+a, t_x+b]\), which is specified by the term \((y\le x+b) \wedge (y\ge x+a) \).

\(\square \)

### Example 4

*Temporal constraints*) Suppose we want to express that whenever the event

*p*occurs, it must be followed by a response

*q*, and then by

*r*. In addition, we have the following timing requirement: if \(\varepsilon _{pq}, \varepsilon _{qr}, \varepsilon _{pr}\) are the time delays between

*p*and

*q*, between

*q*and

*r*, and between

*p*and

*r*, respectively, then: we must have \(\varepsilon ^2_{pq}+ \varepsilon ^2_{qr}+\varepsilon ^2_{pr} \le d\) for a given positive constant

*d*. This can be written using freeze quantifiers as the TLTL formula \(\phi \):

### 3.2 Transference of TLTL properties for propositional traces

We now show that if a timed propositional trace \(\pi \) satisfies a TLTL\((\mathcal {F}_{\mathsf {T}})\) formula \(\phi \), then any timed trace \(\pi '\) that is at most \(\delta \) distance away from \(\pi \) satisfies a slightly relaxed version of the formula \(\phi \), the degree of relaxation being governed by \(\delta \); and the variance of the functions in \(\mathcal {F}_{\mathsf {T}}\) over the time interval containing the time domains of \(\pi \) and \(\pi '\).

We define the distance \(\mathscr {D}_{\mathscr {S}}\) between two propositional traces as the Skorokhod distance, where we use \(\mathscr {D}_{\mathcal {P}}\) as the distance between two sets of propositions.

*Removing negation using the*\({\,\mathcal{W}\,}\)

*operator*The following identities hold relating the \({\,\mathcal{W}\,}\) operator to the \({\,\mathcal{U}\,}\) operator

- 1.
\( \phi _1 {\,\mathcal{U}\,}\phi _2\, \equiv \, \lnot \left( \lnot (\phi _2) {\,\mathcal{W}\,}(\lnot \phi _1 \wedge \lnot \phi _2)\, \right) \); and

- 2.
\( \phi _1 {\,\mathcal{W}\,}\phi _2 \, \equiv \, \lnot \left( \lnot (\phi _2) {\,\mathcal{U}\,}(\lnot \phi _1 \wedge \lnot \phi _2)\, \right) \).

### Definition 4

*relaxation of TLTL*\((\mathcal {F}_{\mathsf {T}})\)

*formulae*) Let \(\phi \) be a TLTL\((\mathcal {F}_{\mathsf {T}})\) formula in which negations appear only on the propositional symbols. The \(\delta \) relaxation of \(\phi \) (for \(\delta \! \ge \! 0\)) over a closed interval

*J*, denoted \({{\mathrm{\mathsf {rx}}}}_{J}^{\delta }(\phi )\), is defined as:

Thus, instead of comparing the \(f_{\mathsf {T}}()\) values to 0, we relax by comparing instead to \(\pm K^{f_{\mathsf {T}}}_J(\delta )\). The other cases recursively relax the subformulae. The functions \(K^{f_{\mathsf {T}}}_J(\delta )\) define the maximal change in the value of \(f_{\mathsf {T}}\) that can occur when the input variables can vary by \(\delta \). The role of *J* is to restrict the domain of the freeze quantifier variables to the time interval *J* (from \({\mathbb {R}}_+\)) in order to obtain the least possible relaxation on a given trace \(\pi \) (e.g., we do not care about the values of a function in \({\mathcal {F}_{\mathsf {T}}}\) outside of the domain \({{\mathrm{\mathsf {tdom}}}}(\pi )\) of the trace).

### Proposition 2

The function \({{\mathrm{\mathsf {rx}}}}\) is a relaxation on TLTL\((\mathcal {F}_{\mathsf {T}})\) formulae, i.e. if a timed propositional trace \(\pi \models \phi \) for a TLTL\((\mathcal {F}_{\mathsf {T}})\) formula \(\phi \), then \(\pi \models {{\mathrm{\mathsf {rx}}}}_{ J}^{\delta }(\phi )\) for all \(\delta > 0\) and non-empty intervals *J*.

### Proof

Observe that, over the predicates \(f_{\mathsf {T}}(x_1, \dots , x_l) \sim 0\), the function \({{\mathrm{\mathsf {rx}}}}\) is indeed a relaxation, *i.e*, if \(f_{\mathsf {T}}(t_1, \dots , t_l) \sim 0\) for values \(t_1, \dots , t_l\), then \({{\mathrm{\mathsf {rx}}}}_{ J}^{\delta }\left( f_{\mathsf {T}}(t_1, \dots , t_l) \right) \sim 0) \) also holds. The result follows by a straightforward induction argument. \(\square \)

### Example 5

*relaxation for bounded temporal operators*—

*MTL*) We demonstrate how \(\delta \)-relaxation operates on bounded time constraints through an example. Consider the MTL formula \(\phi = Q {\,\mathcal{U}\,}_{[a,b]} R\). The \(\delta \)-relaxation of this formula over the interval \({\mathbb {R}}_+\) is \(Q {\,\mathcal{U}\,}_{[a-2\cdot \delta \,,\, b+2\cdot \delta ]} R\). This can be seen as follows. The formula \(\phi \) can be written in TLTL syntax as:

*Q*can be “pulled back” by \(\delta \), and the time of

*R*can be postponed by \(\delta \); thus, the time duration in between

*Q*and

*R*can increase (and similarly can decrease) by \(2\!\cdot \!\delta \). \(\square \)

### Theorem 3

(Transference for propositional traces) Let \(\pi , \pi '\) be two timed propositional traces such that \(\mathscr {D}_{\mathscr {S}}(\pi , \pi ') < \delta \) for some finite \(\delta \). Let \(\phi \) be a closed TLTL\((\mathcal {F}_{\mathsf {T}})\) formula in negation-normal form. If \(\pi \models \phi \), then \(\pi '\models {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi '}}^{\delta }(\phi )\) where \({I_{\pi ,\pi '}}\) is the convex hull of \({{\mathrm{\mathsf {tdom}}}}(\pi ) \cup {{\mathrm{\mathsf {tdom}}}}(\pi ')\).

### Proof

*e.g.*\({{\mathrm{\mathsf {untime}}}}(x. (Q \wedge \ x<5)) \) is \(x. (Q\wedge {\textsc {true}})\)). Since \(\mathscr {D}(\pi , \pi _2) < \delta \), we have that there exists a retiming \(r: {{\mathrm{\mathsf {tdom}}}}(\pi ) \rightarrow {{\mathrm{\mathsf {tdom}}}}(\pi _2)\) such that

We now prove the theorem statement claims concerning the freeze variable constraints. We assume WLOG that \(\phi \) does not freeze the same variable twice. Intuitively, if \(\mathcal {E}\) is an assignment to freeze variables to show \(\pi \models \phi \), then to show \(\pi _2\models {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi _2}}^{\delta }(\phi )\), we consider the environment \(\mathcal {E}_{r}\) is defined by \(\mathcal {E}_{r}(x) = \mathcal {E}(r(x))\). We show that under \(\mathcal {E}_{r}(x)\), the base constraints of \(\phi \) involving the freeze variables are satisfied provided the base constraints are relaxed according to \( {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi _2}}^{\delta }\). The satisfaction of non-base subformulae can then be shown by an induction argument.

- 1.
If a node is the tuple \(\left( \pi ^t, \mathcal {E}, \psi \right) \), then \(\pi ^t\models _{\mathcal {E}} \psi \).

- 2.A node \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _p\right) \) has the following children (mirroring the proof obligations for showing \(\pi ^{t_p}\models _{\mathcal {E}_p} \psi _p\) according to Definition 3:
- (a)
if \(\psi _p = \psi _1\wedge \psi _2\), then two children nodes labelled with \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _1 \right) \) and \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _2 \right) \) respectively.

- (b)
if \(\psi _p = \psi _1\vee \psi _2\), then one child node labelled with either \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _1 \right) \), or \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _2 \right) \).

- (c)If \(\psi _p = \psi _1{\,\mathcal{U}\,}\psi _2\), then, for a single
*t*with \(\max {{\mathrm{\mathsf {tdom}}}}(\pi ) \ge t\ge t_p\) such that \( \pi ^t \models _{\mathcal {E}} \psi _2\) and \(\pi ^{t^*} \models _{\mathcal {E}} \psi _1\vee \psi _2\) for all \(t_p\le t^* < t\), the following (possibly uncountably many) children:- i.
\(\left( \pi ^{t^*}, \mathcal {E}_p, \psi _1\vee \psi _2 \right) \)

- ii.
\(\left( \pi ^{t}, \mathcal {E}_p, \psi _2 \right) \)

- i.
- (d)
If \(\psi _p = \psi _1{\,\mathcal{W}\,}\psi _2\), then children based on how how \(\pi ^{t_p} \models _{\mathcal {E}_p} \psi _1{\,\mathcal{W}\,}\psi _2\) holds (similar to the \({\,\mathcal{U}\,}\) case).

- (e)
If \(\psi _p = x.\psi \), then one child labelled with \(\left( \pi ^{t_p}, \mathcal {E}_p[x:=t_p], \psi \right) \).

- (f)
If \(\psi _p = p\) for

*p*a proposition, or \(\psi _p= f_{\mathsf {T}}(x_1, \dots , x_l) \sim 0 \), then the node is a leaf node such that the proposition or \(\psi _p\) holds at \(\pi ^{t_p}(0)\).

- (a)
- 3.
The root node is \((\pi , \mathcal {E}^*, \phi )\) where \(\mathcal {E}^*\) is any environment, \(\pi \) is the whole trace, and \(\phi \) is the original formula.

*t*is not smaller than any value \(\mathcal {E}(x)\) in that tree for

*x*free in \(\psi \) (intuitively, the variable

*x*was bound to a value at a time earlier than, or equal to time

*t*).

For every node \((\pi ^t, \mathcal {E}, \psi )\) of \(\mathcal {T}\), the tree \(\mathcal {T}'\) has a corresponding node \(\left( \pi _2^{r(t)}, \mathcal {E}_{r}, {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi _2}}^{\delta }(\psi )\ \right) \), where the environment \(\mathcal {E}_{r}\) is defined by \(\mathcal {E}_{r}(x) = \mathcal {E}(r(x))\).

Moreover, if \(\left( \pi ^{t_p}, \mathcal {E}_p, \psi _p\right) \) is a parent to \(\left( \pi ^{t}, \mathcal {E}, \psi \right) \) in tree \(\mathcal {T}\), then in the tree \(\mathcal {T}'\) the node \(\left( \ (\pi _2^{r(t_p)}, {\left( \mathcal {E}_p\right) }_{r}, {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi _2}}^{\delta }(\psi _p)\ \right) \) is a parent to \(\left( \pi _2^{r(t)}, \mathcal {E}_{r}, {{\mathrm{\mathsf {rx}}}}_{I_{\pi ,\pi _2}}^{\delta }(\psi )\ \right) \).

*r*is strictly increasing and bijective, it can be checked that \(\mathcal {T}'\) can be proved to be a proof tree by induction on the height of nodes, if we can show that for leaf nodes \(\left( \pi _2^{r(t)}, \mathcal {E}_{r}, \psi \right) \), we have \( \pi _2^{r(t)}\models _{\mathcal {E}_{r}} \psi \). We prove this next.

If a leaf node is \(\left( \pi _2^{r(t)}, \mathcal {E}_{r}, p \right) \) for *p* a proposition, then \(\pi _2^{r(t)} \models p\) since (a) *r* maps the time *t* in \(\pi \) to time *r*(*t*) in \(\pi _2\); and (b) \(\mathscr {D}_{\mathscr {S}}(\pi , \pi _2) < \delta \) and \(\delta \) is finite; thus \(\pi _2(r(t)) = \pi (t)\); and (c) \(\mathcal {T}\) must contain the corresponding leaf node \(\left( \pi ^t, \mathcal {E}, p \right) \) and so \(\pi (t) = p\). The three previous facts imply \(\pi _2(r(t)) = \pi (t)= p\); and hence \(\pi _2^{r(t)} \models p\).

Thus, the leaves of the tree \(\mathcal {T}'\) satisfy property (1) of proof trees Property (1) can be seen to hold for non-leaf nodes using a bottom up argument following the structure of the proof tree \(\mathcal {T}\). This completes the proof. \(\square \)

Theorem 3 relaxes the freeze variables over the entire signal time-range \({I_{\pi ,\pi '}}\); it can be strengthened by relaxing over a smaller range: if \(\pi \models \phi \), and \(t_1, \dots , t_k\) are time-stamp assignments to the freeze variables \(x_1, \dots , x_k\) which witness \(\pi \) satisfying \(\phi \), then \(x_i\) only needs to be relaxed over \([t_i-\delta , t_i+\delta ]\) rather than the larger interval \({I_{\pi ,\pi '}}\). These smaller relaxation intervals for the freeze variables can be incorporated in Eq. 11. We omit this optimization for ease of presentation.

### Example 6

This can be seen as follows. Since \(\pi \) satisfies \(\phi \), we must have time-stamps \(t_x, t_y, t_z\) bound to *x*, *y*, *z* respectively so that with these assignments, the formula \(\phi \) is satisfied. Since \(\pi '\) is \(\delta \) close to \(\pi \), there is a retiming from \(\pi \) to \(\pi '\) such that the times \(t_x, t_y, t_z\) in \(\pi \) are mapped to corresponding time \(t_x', t_y', t_z'\) in \(\pi '\) such that (a) \(|t_x-t_x'| \le \delta \); and (b) \(|t_y-t_y'| \le \delta \); and (c) \(|t_z-t_z'| \le \delta \).

### 3.3 Transference of FLTL properties for \({\mathbb {R}}^n\)-valued traces

In the previous two subsections, we considered propositional traces. We now generalize to \({\mathbb {R}}^n\)-valued traces, and freeze quantification over both time and \({\mathbb {R}}^n\) trace values in the logic FLTL (as compared to the propositional case where the freeze quantifiers were only over the time domain).

A *timed *\({\mathbb {R}}^n\)-*valued trace*\(\pi \) is a function from a closed interval *I* of \({\mathbb {R}}_+\) to \({\mathbb {R}}^n\). To define the semantics of FLTL formulae over timed \({\mathbb {R}}^n\)-valued sequences, we use booleanizing predicates \(\mu : {\mathbb {R}}^m \rightarrow {\mathbb {B}}\), to transform \({\mathbb {R}}^n\)-valued sequences into timed propositional sequences. These predicates – with the help of freeze variables – denote relationships between different times and values in the trace.

Since we also have freeze variables, FLTL with predicates is strictly more expressive than STL^{6} (as in the propositional case [6]).

For ease of presentation, we assume *n* is fixed, i.e., we consider FLTL for \({\mathbb {R}}^n\)-valued traces

### Definition 5

*FLTL*\((\mathcal {F})\)

*Syntax*) Given a set of vectors

*V*(the freeze vectors, each vector having \(n+1\) variables), and a set \(\mathcal {F}\) of functions \(f: {\mathbb {R}}^{m(n+1)} \rightarrow {\mathbb {R}}\) for \(1\le m\le M\) for some \(M\ge 1\), the formulae of FLTL\((\mathcal {F})\) are defined by the grammar:

*Closed*FLTL\((\mathcal {F})\) formulae are defined analogously to closed TLTL\((\mathcal {F}_{\mathsf {T}})\) formulae. The semantics of FLTL\((\mathcal {F})\) are similar to that of TLTL\((\mathcal {F}_{\mathsf {T}})\) (Definition 3). The only new ingredient is that now the freeze variables are \(n+1\)-tuple freeze vectors. For a freeze vector \(\overline{x} = (x^0, x^1, \dots , x^n)\), we denote \(x^i\) by \(\overline{x}[i]\). The component \(\overline{x}[0]\) refers to the time dimension. The components \(\overline{x}[i]\) for \(i>1\) refer to the *i*-th value dimension. The freeze quantifier “\(\overline{x}.\)” now binds time, and the trace values in \({\mathbb {R}}^n\) to \(\overline{x}[0]\) and to \(\overline{x}[1], \dots , \overline{x}[n]\) respectively (the *i*-th dimension of the trace value is mapped to \(\overline{x}[i]\)).

### Example 7

FLTL\((\mathcal {F})\)*compared to the logic*STL\(^*\)*of* [8, 9]: The logic FLTL\((\mathcal {F})\) extends STL\(^*\) as follows: (1) it allows capture of time-stamps in freeze variables; (2) the functions in \(\mathcal {F}\) need not be linear; and (3) the temporal operators are not attached to closed non-singular intervals.

*relaxation of*FLTL\((\mathcal {F})\) Let \({\mathbb {R}}^n\) have the norm

*L*. Let \( \mathbf {J}\) be a mapping from \(\{ 0,1,\dots , n \}\) to subsets of of \({\mathbb {R}}\) (with \(\mathbf {J}(0)\) being a closed interval of \({\mathbb {R}}_+\)). The interval \(\mathbf {J}(i)\) for \(i>0\) denotes the range of the

*i*-th trace dimension. The time-range is the interval \(\mathbf {J}(0)\). The relaxation function \({{\mathrm{\mathsf {rx}}}}_{\mathbf {J}}^{\delta }\), for the \({\mathbb {R}}^n\) norm

*L*, which operates on FLTL\((\mathcal {F})\) formulae is defined analogous to the relaxation function \({{\mathrm{\mathsf {rx}}}}_{J}^{\delta }\) in Definition 4. We omit the similar cases, and only present the new case for the predicates formed from \( \mathcal {F}\) (the full definition can be found in Appendix).

*f*that can occur under the following constraints:

the input \(n+1\)-ary values \(\overline{u}_i\) can vary by at most \(\delta \) in the \(L^{\max }\) norm; and

the time-domain is restricted to \(\mathbf {J}(0)\); and

the

*k*-th value-domain is restricted to intervals in \(\mathbf {J}(k)\).

### Proposition 3

The function \({{\mathrm{\mathsf {rx}}}}_{\mathbf {J}}^{\delta }\) is a relaxation on FLTL\((\mathcal {F})\) formulae, i.e. if a timed \({\mathbb {R}}^n\)-valued trace \(\pi \models \phi \) for a FLTL\((\mathcal {F})\) formula \(\phi \), then \(\pi \models {{\mathrm{\mathsf {rx}}}}_{\mathbf {J}}^{\delta }(\phi )\).

### Proof

The proof is similar to the proof of Proposition 2. \(\square \)

### Theorem 4

*L*) is less than \(\delta \) for some finite \(\delta \). Let \(\phi \) be a closed FLTL\((\mathcal {F})\) formula in negation-normal form. If \(\pi \models \phi \), then \(\pi '\models {{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta }(\phi )\), where

\(\mathbf {J}\) be a mapping from \(\{ 0,1,\dots , n \}\) to subsets of \({\mathbb {R}}\);

\(\mathbf {J}(0)\) is the convex hull of \({{\mathrm{\mathsf {tdom}}}}(\pi ) \cup {{\mathrm{\mathsf {tdom}}}}(\pi ')\); and

\(\mathbf {J}(k)\) for \(n\ge k>0\) is \(\{ \pi (t)[k] \mid t\in {{\mathrm{\mathsf {tdom}}}}(\pi ) \} \cup \{ \pi '(t)[k] \mid t\in {{\mathrm{\mathsf {tdom}}}}(\pi ') \} \); where \(\pi (t)[k]\) denotes the

*k*-th dimensional value in \(\pi (t)\) (and similarly for \(\pi '(t)\); and\({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta }\) is defined with respect to the norm \(L^{\max }\).

### Proof

The theorem can be proved along the same lines as the proof of Theorem 3. The witnessing proof tree is constructed using the time-bindings as previously. In the propositional case, a retiming *r* in effect maps the proposition \(\pi (t)\) to \(\pi '(r(t))\); in the present case, it maps the \({\mathbb {R}}^n\) value \(\pi (t)\) to the \({\mathbb {R}}^n\) value \(\pi '(r(t))\) such that \(\left\| \pi '(r(t)) - \pi (t)\right\| _L < \delta \). This requires us to incorporate the value distortions in \({\mathbb {R}}^n\) (in addition to the time distortions) when relaxing the formulae \(f(\overline{x}_1, \dots , \overline{x}_l) \sim 0 \). This is handled by \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta }\) as shown previously. The rest of the proof is as in the propositional case. \(\square \)

Theorem 4 can be strengthened similar to the strengthening mentioned for Theorem 3 by relaxing the variables over smaller intervals obtained from assignments to variables which witness \(\pi \models \phi \).

### Example 8

*(Transference)* Recall Example 7. Suppose we have two traces \(\pi , \pi '\) over the time interval [0, 100] with trace values in \({\mathbb {R}}^2\). Let the range of the first value dimension be \([-8, 8]\) and range of the second value dimension be [0, 64]. Let the Skorokhod distance with respect to the \({\mathbb {R}}^2\) norm \(L_{\infty }\) be \(\delta \).

\( f_1(\overline{x}_1)> 0\), for \(f_1(\overline{x})=\overline{x}_1[0] -96 \). We have \(K^{f_1}_{ \mathbf {J}}(\delta )\) to be \(\delta \) (since \(\left\| \overline{u}_1- \overline{u}_1'\right\| _{L_{\infty }^{\max }} \le \delta \) implies that \(\left| f_1(\overline{u}_1) - f_1(\overline{u}'_1) \right| \le \delta \). Thus, \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta } (f_1(\overline{x}_1)> 0) \), which is defined to be \(f_1(\overline{x}_1) + K^{f_1}_{ \mathbf {J}}(\delta ) > 0\), is equal to \(\overline{x}_1[0] + \delta -96 >0\).

- \( f_2(\overline{x}_1, \overline{x}_2) \le 0\), for \(f_2(\overline{x}_1, \overline{x}_2) = \overline{x}_2[0]- \overline{x}_1[0]-4\). Since if \(\left\| \overline{u}_1- \overline{u}_1'\right\| _{L_{\infty }^{\max }} \le \delta \) and if \(\left\| \overline{u}_2- \overline{u}_2'\right\| _{L_{\infty }^{\max }} \le \delta \) we havewe get that \(K^{f_2}_{ \mathbf {J}}(\delta )\) to be \(2\delta \). Thus, \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta } (f_2(\overline{x}_1, \overline{x}_2) \le 0)\), which is defined to be \(f_2(\overline{x}_1, \overline{x}_2) -K^{f_2}_{ \mathbf {J}}(\delta ) \le 0\), is equal to \(\overline{x}_2[0]- \overline{x}_1[0]-4-2\delta \le 0\).$$\begin{aligned} \sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \big (\overline{u}_2[0]- \overline{u}_1[0]-4 \big ) - \big (\overline{u}'_2[0]- \overline{u}'_1[0]-4\big ) \right| \ =\ 2\delta , \end{aligned}$$
\( f_2(\overline{x}_1, \overline{x}_2) \ge 0\), for \(f_2(\overline{x}_1, \overline{x}_2) = \overline{x}_2[0]- \overline{x}_1[0]-4\).

Using the analysis from the previous case, we get \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta } (f_2(\overline{x}_1, \overline{x}_2) \ge 0)\) to be \(\overline{x}_2[0]- \overline{x}_1[0]-4+2\delta \ge 0\).

- \( f_3(\overline{x}_1, \overline{x}_2) \le 0\), for \(f_3(\overline{x}_1, \overline{x}_2) = \overline{x}_2[2] - \overline{x}_1[1] \cdot \overline{x}_1[1]\). For vectors \(\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2\) in \({\mathbb {R}}_+\times {\mathbb {R}}^{n}\) such thatwe have
- 1.
\(\left| \overline{u}_k[1]\right| \le 8\) and \(\left| \overline{u}'_k[1]\right| \le 8\) for \(k\in \{ 1,2 \}\), and

- 2.
\(0\le \overline{u}_k[2] \le 64\) and \(0\le \overline{u}'_k[2] \le 64\) for \(k\in \{ 1,2 \}\), and

- 3.
\(\left\| \overline{u}_1- \overline{u}_1'\right\| _{L_{\infty }^{\max }} \le \delta \) and

- 4.
\(\left\| \overline{u}_2- \overline{u}_2'\right\| _{L_{\infty }^{\max }} \le \delta \),

Denote \(\overline{u}_1[1]\) as$$\begin{aligned}&\sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \big (\overline{u}_2[2] - \overline{u}_1[1] \cdot \overline{u}_1[1]\big ) - \big (\overline{u}'_2[2] - \overline{u}'_1[1] \cdot \overline{u}'_1[1]\big )\right| \nonumber \\&\quad \le \sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \overline{u}_2[2] - \overline{u}'_2[2]\right| \ +\ \sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \overline{u}_1[1] \cdot \overline{u}_1[1]- \overline{u}'_1[1] \cdot \overline{u}'_1[1]\right| \nonumber \\&\quad \le \delta \ +\ \sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \overline{u}_1[1] \cdot \overline{u}_1[1]- \overline{u}'_1[1] \cdot \overline{u}'_1[1]\right| \end{aligned}$$(15)*a*, and \(\overline{u}'_1[1]\) as*b*. To compute the supremum in the last expression, we need to obtain the following:We have \( \left| a^2-b^2\right| = \left| a-b\right| \cdot \left| a+b\right| \le \delta \cdot \left| a+b\right| \). Using the ranges of$$\begin{aligned} \text {maximize }&\ \left| a^2-b^2\right| \\ \text {subject to }&-8\le a \le 8\\&-8\le b \le 8\\&\left| a-b\right| \le \delta \end{aligned}$$*a*,*b*, we get \( \left| a+b\right| \le 16\). Thus, \( \left| a^2-b^2\right| \le 16\cdot \delta \). Substituting back in Eq. (15), we getThus, \(K^{f_3}_{ \mathbf {J}}(\delta )\) is \(17\cdot \delta \). Hence, \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta } (f_3(\overline{x}_1, \overline{x}_2) \le 0)\), which is defined to be \(f_3(\overline{x}_1, \overline{x}_2) -K^{f_3}_{ \mathbf {J}}(\delta ) \le 0\), is equal to$$\begin{aligned} \sup _{\overline{u}_1, \overline{u}_2, \overline{u}'_1, \overline{u}'_2} \left| \big (\overline{u}_2[2] - \overline{u}_1[1] \cdot \overline{u}_1[1]\big ) - \big (\overline{u}'_2[2] - \overline{u}'_1[1] \cdot \overline{u}'_1[1]\big ) \right| \ \le \ \delta \ +16\cdot \delta = 17\cdot \delta \end{aligned}$$$$\begin{aligned} \overline{x}_2[2] - \overline{x}_1[1] \cdot \overline{x}_1[1] - 17\cdot \delta \le 0. \end{aligned}$$ - 1.
- \( f_3(\overline{x}_1, \overline{x}_2) \ge 0\), for \(f_3(\overline{x}_1, \overline{x}_2) = \overline{x}_2[2] - \overline{x}_1[1] \cdot \overline{x}_1[1]\). Using the analysis of the previous case, we get \({{\mathrm{\mathsf {rx}}}}_{ \mathbf {J}}^{\delta } (f_3(\overline{x}_1, \overline{x}_2) \ge 0)\) to be$$\begin{aligned} \overline{x}_2[2] - \overline{x}_1[1] \cdot \overline{x}_1[1] + 17\cdot \delta \ge 0. \end{aligned}$$

- 1.
The tracking requirements are only till time \(96-\delta \) (the original formula in (13) has a tracking requirement till time 96).

- 2.
The tracking delay is variable, and falls in the range \([\!4\!-\!2\delta , \!4\!+\!2\delta ]\) (the original formula in (13) has a tracking delay of exactly 4 time units).

- 3.
The tracked trace dimension value

*v*is output as the second dimension value in range \([v^2\!-\!17\delta , \ v^2\!+\!17\delta ]\) (the original formula in (13) output the second dimension as exactly \(v^2\)).

## 4 Quantifying timing distortion using the Skorokhod metric

\(\left\| {{\mathrm{\mathsf {r}}}}-{{\mathrm{\mathcal {I}}}}\right\| _{\sup } < \epsilon \); and

\(\mathscr {D}_{\sup }(\pi _{\alpha },\, \pi _{\alpha }'\circ {{\mathrm{\mathsf {r}}}}) < \epsilon \).

\(\left\| {{\mathrm{\mathsf {r}}}}_{\frac{1}{\alpha }}-{{\mathrm{\mathcal {I}}}}\right\| _{\sup } < \frac{\epsilon }{\alpha }\); and

\(\mathscr {D}_{\sup }(\pi ,\, \pi '\circ {{\mathrm{\mathsf {r}}}}_{\frac{1}{\alpha }}) < \epsilon \).

*T*on \(\lambda ^\star (\pi , \pi ', \epsilon )\) is given, then this can be done by binary search over the interval [0,

*T*]. Note that if \(\epsilon \ge \mathscr {D}_{\sup }(\pi , \pi ')\), then \(\lambda ^\star (\pi , \pi ', \epsilon ) =0\) as no retiming will be required. If an upper bound is not given, then we set

*T*as \((\max I_{\pi '} - \min I_{\pi })\), as this is the maximum value that \(\lambda ^\star (\pi , \pi ', \epsilon ) \) can take, corresponding to the case where the starting time \(\min I_{\pi }\) of \(\pi \) is mapped to the ending time \(\max I_{\pi '}\) of \(\pi '\). The algorithm for computing \(\lambda ^\star (\pi , \pi ', \epsilon ) \) is given below.

## 5 Experimental evaluation

In this section we provide experimental evidence on the efficacy of the Skorokhod metrics, and their use in a conformance testing framework. We begin the section by a brief remark on the difference between the precise Skorokhod distance between signals in a mathematical sense and the signals we obtain from a simulation framework.

### Remark 1

In what follows, we rely on simulation tools that internally perform numerical integration of ordinary differential equations (ODEs) to numerically approximate the solution to the dynamical equations of a given system. A rich set of algorithms to perform such numerical integration is supported by the tools Matlab\(^{\textregistered }\)and Simulink\(^{\textregistered }\). Essentially, these algorithms provide a discrete approximation of the mathematical solution to the ODEs, at a sequence of equally spaced time-points (for fixed-step solvers) or a set of variably spaced time-points (for variable-step solvers). For certain classes of variable step-solvers, the user has the ability to specify an absolute tolerance and a relative tolerance [28], which are used to bound the local truncation error or the error caused in a single iterative step in the numerical integration [33]. However, these tolerances do not control the global truncation error, or the error accumulated by several integration steps. If we had access to absolute bounds on the global truncation error, to find precise bounds on the Skorokhod distance between two dynamical systems, we could find the Skorokhod distance between the discrete solution signals obtained from numerical integration, and add the global truncation error bound to the result. However, precise bounds on the global truncation error require well-behavedness conditions on the numerical integration procedure used, as well as the functions being integrated. Such conditions are nearly impossible to satisfy in practice (for example when we have hybrid systems, systems with discontinuous dynamics, black-box systems, *etc.*). Hence, in the rest of the paper, we assume the results obtained from the numerical simulation tools as the precise time-varying behavior of the systems to be analyzed.

*Roadmap of the experiments* In Sect. 5.1, we present the running times of our sliding window Skorokhod metric computation procedure for polygonal traces. In Sect. 5.2, we present the running times of our sliding window Skorokhod metric computation procedure when traces are completed from the sample points under a sample-and-hold semantics, and compare the distance values obtained with those given by the polygonal trace routine. This routine for piecewise constant (PWC) traces runs two orders of magnitude faster than the polygonal trace metric routine as a result of the simpler dynamic programming algorithm for PWC traces. We note that difference between the distance given by the polygonal routine, and that given by the PWC routine can be made to decrease by “up-sampling” (where we generate new intermediate sample points using linear interpolation); however, as we increase the sampling rate, the number of samples in the trace *and* the window sizes required for the same allowed time distortion *both* increase—thus up-sampling by a factor of \(\gamma \) results in the computation time increasing by a factor of \(\gamma ^2\) for PWC traces.

In Sect. 5.3, we present the results on our implementation which computes the least retiming required in order to have the \(\mathscr {D}_{\sup }\) distance between two given traces after retiming be at most \(\epsilon \) (corresponding to the algorithm presented in Sect. 4).

In Sect. 5.4, we present three case studies for finding the distance between *systems* using the Skorokhod *trace* metric in a conformace testing framework (as sketched in Algorithm 1). The first case study involves two related LQR-based aircraft pitch controllers—the first controller is continuous time, and the second controller is in discrete time with added sensor delays. The second case study involves two air-fuel ratio controllers—the first one has highly nonlinear dynamics, and the second one is a hypothesized polynomial approximation of the first one. The third case study involves a Simulink model, under two different numerical integrators, of a four-cylinder spark ignition internal combustion engine.

### 5.1 Skorokhod metric computation: polygonal traces

We implemented a streaming, sliding window-based monitoring routine which checks, given a fixed \(\delta \), whether the linear interpolations of two time-sampled traces are at Skorokhod distance at most \(\delta \) away from each other (the distance between two points \(p_1, p_2\) in \({\mathbb {R}}^n\) is taken to be with respect to the \(L_2\) norm, i.e., \(\left\| p_1-p_2\right\| _{L_2}\)). The algortithm has a time complexity of \(O\left( m \cdot n\cdot W\right) \) where *m* is the number of sample points, *n* is the dimension of a sample point, and *W* is the window size. Our implementation uses only *O*(*W*) space. The least \(\delta \) value, *i.e.* the Skorokhod distance value, is then computed by binary search over the monitoring routine. The upper limit of the search range is set to the pointwise metric (i.e., assuming the identity retiming) between the two traces.

*Time and value scaling* The traces to the monitoring routine are pre-scaled, each dimension (and the time-stamp) is scaled by a different constant. The constants are chosen so that after scaling, one unit of deviation in one dimension is as undesirable as one unit of jitter in other dimensions.

*Skorokhod distance computation benchmark: I*We first show that the window-based implementation is efficient using the following benchmark. Figure 1 shows a hybrid dynamical system \(\mathfrak {A}_1\) consisting of two water tanks, each with an outlet from which water drains at a constant rate \(d_j\). Both tanks share a single inlet pipe that is switched between the tanks, filling only one tank at any given time at a constant inflow rate of

*i*. When the water-level in tank

*j*falls below level \(\ell _j\), the pipe switches to fill it. The drain and inflow rates \(d_1\), \(d_2\) and

*i*are assumed to be inputs to the system. Now consider a version \(\mathfrak {A}_2\) that incorporates an actuation delay that is a function of the inflow rate. This means that after the level drops to \(\ell _j\) for tank

*j*, the inlet pipe starts filling it only after a finite time. \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\) have the same initial water level. We perform a fixed number of simulations by systematically choosing drain and inflow rates \(d_1\), \(d_2\),

*i*to generate traces (water-level vs. time) of both systems and compute their Skorokhod distance. We summarize the results in Table 1.

Computation of \(\mathscr {D}_{\mathscr {S}}(\pi _1,\pi _2)\), where \(\pi _1\) is a trace of system \(\mathfrak {A}_1\) described in Fig. 1, and \(\pi _2\) is a trace of system \(\mathfrak {A}_2\), which is \(\mathfrak {A}_1\) with an actuation delay

Window size | Avg. \(\mathscr {D}_{\mathscr {S}}\) | Avg. time taken (s) | \(\frac{\mathscr {D}_{\sup } - \mathscr {D}_\mathscr {S}}{\mathscr {D}_{\sup }}\) | |||
---|---|---|---|---|---|---|

Computation | Monitoring | Max. | Avg. | SD | ||

20 | 8.58 | 0.81 | 0.13 | 0.11 | 0.03 | 0.03 |

40 | 8.35 | 1.55 | 0.26 | 0.23 | 0.06 | 0.06 |

60 | 8.09 | 2.31 | 0.39 | 0.34 | 0.1 | 0.09 |

80 | 7.88 | 3.05 | 0.52 | 0.38 | 0.1 | 0.11 |

100 | 7.72 | 3.77 | 0.64 | 0.38 | 0.1 | 0.11 |

Recall that the Skorokhod distance computation involves a sequence of monitoring calls with different \(\delta \) values picked by a binary-search procedure. Thus, the total time to compute \(\mathscr {D}_\mathscr {S}\) is the sum over the computation times for individual monitoring calls plus some bookkeeping. In Table 1, we make a distinction between the average time to monitor traces (given a \(\delta \) value), and the average time to compute \(\mathscr {D}_\mathscr {S}\). There are an average of 6 monitoring calls per \(\mathscr {D}_\mathscr {S}\) computation. We ran 64 simulations by choosing different input values, and then computing \(\mathscr {D}_\mathscr {S}\) for increasing window sizes. As the window size increases, the average \(\mathscr {D}_\mathscr {S}\) decreases and the computation time increases linearly, as expected from Theorem 2. Finally, the Skorokhod distance can be significantly smaller than the simpler metric \(\mathscr {D}_{\sup }\) (defined as the maximum of the pointwise \(L_2\) norm). This discrepancy becomes more prominent with increased window size. With a window size of 100, the variation between \(\mathscr {D}_\mathscr {S}\) and \(\mathscr {D}_{\sup }\) was up to \(38\%\) (mean difference of \(10\%\) with std. deviation of \(11\%\)).

### 5.2 Skorokhod metric computation: piecewise constant traces

A numerical solver for simulations typically returns a sequence of time-value pairs for the signals of interest, which are then interpreted as a signal over dense time by using an interpolation scheme. The most commonly used scheme is linear interpolation, which results in polygonal, i.e., continuous piecewise linear (PWL) traces. It is also of interest to consider constant interpolation; here, the interpolated value at a time point is simply the sample-value of the largest preceding time. Recall that the Skorokhod distance computation for piecewise constant (PWC) traces can be achieved by an algorithm that uses dynamic programming, and that this algorithm is computationally more efficient than the algorithm to compute Skorokhod distance between polygonal traces. This raises a natural question: are there cases where the simpler algorithm for computing distance between PWC traces can be used? We experimentally evaluate this question by comparing the results of the polygonal routine with the results from the PWC routine.

We implemented a streaming, sliding window-based dynamic programming algorithm to compute the Skorokhod distance between two piecewise constant traces based on the results in Sect. 2.3. The algortithm has a time complexity of \(O\left( m \cdot n\cdot W\right) \) where *m* is the number of sample points, *n* is the dimension of a sample point, and *W* is the window size. Our implementation uses only *O*(*W*) space. The time-stamps and the sample values are pre-scaled as in the polygonal case.

*Skorokhod distance computation benchmark: II* In the following experiment, we asssume that we are given two models of a bang-bang controller for a water boiler system. The controller operates at a fixed frequency, and turns on heat or turns off heat depending on whether the water temperature is below or above a user-specified reference temperature. In the first model, we assume that there is a fixed actuation delay in turning the boiler on or off. In the second model, the actuation delay in turning the boiler on is different than that for turning it off. The output signal of interest is the water temperature, and we compute the distance between the outputs of the two models, for 50 randomly chosen reference temperatures in the range 40–70 \(^\circ \)C.

We use three distance metrics to compute the distances: the \(\mathscr {D}_{\sup }\) metric defined in the previous benchmark, the Skorokhod distance between the output traces obtained using PWL interpolation of the samples returned by the simulator, and the Skorokhod distance between the output traces obtained using PWC interpolation. Further, we assume that the models are simulated with a variable-step solver (\(\mathtt {ode23}\) in Simulink\(^{\textregistered }\)).

Variation in Skorokhod Distance between PWC traces with sampling rate

Sampling rate | Avg. trace length | Avg. \(\frac{\mathscr {D}_{\mathscr {S}}^{\text { pwl}}-\mathscr {D}_{\mathscr {S}}^{\text { pwc}}}{\mathscr {D}_{\mathscr {S}}^{\text { pwl}}}\) | Avg. \(\frac{\mathscr {D}_{\sup } - \mathscr {D}_{\mathscr {S}}^{\text { pwc}}}{\mathscr {D}_{\sup }}\) | Avg. time (s) |
---|---|---|---|---|

1 | 381.5 | \(-0.149\) | 0.098 | 0.0034 |

2 | 761.9 | \(-0.064\) | 0.159 | 0.0104 |

5 | 1903.3 | \(-0.033\) | 0.181 | 0.0639 |

10 | 3805.6 | \(-0.019\) | 0.193 | 0.2553 |

15 | 5707.8 | \(-0.012\) | 0.198 | 0.5787 |

20 | 7610.2 | \(-0.009\) | 0.199 | 1.0365 |

50 | 19024.0 | \(-0.003\) | 0.204 | 6.5343 |

75 | 28535.5 | \(-0.002\) | 0.205 | 15.3230 |

Table 2 shows that the difference between the Skorokhod distance computed by the routine for PWL traces is about \(15\%\) lower than that computed by the routine for PWC traces for the same sequence of time-value pairs (i.e. sampling rate 1). This discrepancy improves as we sample the PWL traces at higher rates. However, as we increase the sampling rate, the number of samples in the trace *and* the window sizes required for the same allowed time distortion *both* increase – thus upsampling by a factor of \(\gamma \) results in the computation time increasing by a factor of \(\gamma ^2\) for PWC traces.

At around a sampling rate of 15, the computation time for the PWC routine is comparable to that of the PWL routine, while giving a discrepancy of about \(1.2\%\). Another observation of interest is that the discrepancy between the Skorokhod distance computed by the PWC and the \(\mathscr {D}_{\sup }\) metric steadily increases with sampling rate, approaching the discrepancy between the Skorokhod distance computed by the PWL routine and the \(\mathscr {D}_{\sup }\) metric.

*Skorokhod distance computation benchmark: III*We now examine the effect on accuracy of the distance computation for the algorithm to compute Skorokhod distance on polygonal traces with sampling rate of the trace. The thesis is that as we downsample the given pair of outputs, we can compute Skorokhod distance faster, but with loss of accuracy. The experiment below (using the same setting as the previous benchmarking experiment) explores this tradeoff curve. The results are presented in Table 3.

Variation in Skorokhod Distance between polygonal traces with sampling rate

Down-sampling rate ( | Avg. \(\frac{\mathscr {D}^1_{\mathscr {S}} - \mathscr {D}^{r}_{\mathscr {S}}}{\mathscr {D}^1_\mathscr {S}} \ \) | Avg. \(\frac{\mathscr {D}_{\sup } - \mathscr {D}^r_{\mathscr {S}}}{\mathscr {D}_{\sup }}\) | Avg. time (s) |
---|---|---|---|

1 | 0 | 0.207 | 0.53 |

2 | −0.159 | 0.092 | 0.28 |

3 | −0.259 | 0.018 | 0.20 |

4 | −0.265 | 0.014 | 0.16 |

5 | −0.267 | 0.012 | 0.13 |

10 | −0.274 | 0.007 | 0.08 |

25 | −0.278 | 0.003 | 0.05 |

50 | −0.279 | 0.002 | 0.03 |

Table 3 shows that as we down-sample the traces, the computation time improves. However, as expected, the accuracy of computation degrades. Down-sampling by a factor of 2 (i.e. dropping every other sample point) leads to an error of \(15\%\) in the distance computation.

### 5.3 Quantifying timing distortion using the Skorokhod metric

We implemented Algorithm 2 to compute the least retiming required in order to have the \(\mathscr {D}_{\sup }\) distance between the traces after retiming be at most \(\epsilon \).

### 5.4 Skorokhod distance between systems: case studies

We integrated the Skorokhod metric monitoring routine in an adaptive testing procedure for Simulink blocks based on Algorithm 1. The output of Algorithm 1 is compared against tolerance levels (e.g., maximum allowed jitter) given by the engineering requirements. In the following, we evaluate the effectiveness of the Skorokhod metric in conformance testing of Simulink applications. The subsequent case studies used the polygonal Skorokhod distance computation routine.

*Case study I: LQR-based controller*The first case study for conformance testing is an aircraft pitch control application taken from the openly accessible control tutorials for Matlab and Simulink [30]. The authors describe a linear dynamical system of the form: \(\dot{\mathbf {x}} = (A-BK)\mathbf {x} + B\theta _{des}\). Here, \(\mathbf {x}\) describes the vector of continuous state variables \(\mathbf {x}\) = \([\alpha \ q\ \theta ]\), where \(\alpha \) is the angle of attack,

*q*is the pitch rate, and \(\theta \) is the pitch angle. The system has a single input \(\delta \) (the elevator deflection angle); and \(\theta _{des}\) is the desired reference provided as an external input. In deriving the control law, the designers use the state feedback law to substitute \(\delta \) = \(\theta _{des}-K\mathbf {x}\). The resulting dynamical equations of the system are of the form \(\dot{\mathbf {x}} = (A-BK)\mathbf {x} + B\theta _{des}\), and the output of the system is the state variable \(\theta \). Note that the

*K*matrix is the gain matrix resulting from the LQR control design technique. The values of the

*A*,

*B*and

*K*matrices are as given below:

Control engineers are typically interested in the step response of a system. In particular, quantities such as the overshoot/undershoot of the output signal (maximum positive/negative deviation from a reference value) and the settling time (time it takes for transient behaviors to converge to some small region around the reference value) are of interest. Given a settling time and overshoot for the first system, we would like the second system to display similar characteristics. We remark that both of these properties can be expressed in STL (and hence in FLTL), see [23] for details. We quantify system conformance (and thereby adherence to requirements) in terms of the Skorokhod distance, or, in other words, maximum permitted time/space-jitter value \(\delta \). For this system, we know that at nominal conditions, the settling time is approximately 2.5 s, and that we can tolerate an increase in settling time of about 0.5 s. Thus, we chose a time-scaling factor of \(2 = \frac{1}{0.5}\). We observe that the range of \(\theta \) is about 0.4 radians, and specify an overshoot of \(20\%\) of this range as being permissible. Thus, we pick a scaling factor of \(\frac{1}{0.08}\) for the signal domain. In other words, Skorokhod distance \(\delta = 1\) corresponds to either a time-jitter of 0.5 s, or a space-discrepancy of 0.08 radians.

Variation in Skorokhod Distance with changing sampling time for an aircraft pitch control system with an LQR-based controller

Controller Sample-Time (s) | Skorokhod distance | Time taken (s) to compute \(\mathscr {D}_{\mathscr {S}}\) | Number of simulations |
---|---|---|---|

0.01 | 0.012 | 232 | 104 |

0.05 | 0.049 | 96 | 104 |

0.1 | 0.11 | 70 | 106 |

0.3 | 0.39 | 45 | 104 |

0.5 | 1.51 | 40 | 101 |

*Case study II: air-fuel ratio controller* In [23], the authors present three systems representing an air-fuel ratio (\(\lambda \)) controller for a gasoline engine, that regulate \(\lambda \) to a given reference value of \(\lambda _{\text {ref}} = 14.7\). These systems are simplified versions of industrial-scale models. Of interest to us are the second and the third systems. Both versions have 2 exogenous inputs, and states in both versions consist of 4 components taking values in \({\mathbb {R}}\) (thus, both systems have a continuous state space). The inputs are engine speed (measured in rpm) and the throttle angle (in degrees). The throttle angle is a user input, and it is common to assume a series of pulses or steps as throttle angle inputs. The engine speed is considered an input to avoid modeling parts of the powertrain dynamics. In our experiments, we typically hold the engine speed constant. This is to mimic a common engine testing scenario involving a dynamometer, which is a device to provide external torque to the engine to maintain it at a constant speed. Of the 4 state components, we assume that 2 of these are from the plant model (that encapsulates physical processes within the engine), while the other 2 belong to the controller. The plant state components *p* and \(\lambda \) denote intake manifold pressure and the A/F ratio respectively. The controller state component \(p_e\) denotes the estimated manifold pressure (with the use of an observer) used in the feed-forward control, and the state component *i* denotes the integrator state in the P+I feedback control. We check conformance with respect to the system output \(\lambda \). For the dynamical system equations, please refer to [23, 24].

The second sysrem has a continuous-time plant model with highly nonlinear dynamics, and a discrete-time controller model. In [24], the authors present a version of this system where the controller is also continuous. We take this to be \(\mathfrak {A}_1\). The third system in [23] is a continuous-time closed-loop system where all the system differential equations have right-hand-sides that are polynomial approximations of the nonlinear dynamics in \(\mathfrak {A}_1\). We call this polynomial dynamical system \(\mathfrak {A}_2\). The rationale for these system versions is as follows: existing formal methods tools cannot reason about highly nonlinear dynamical systems, but tools such as Flow* [12], C2E2 [18], and CORA [3] demonstrate good capabilities for polynomial dynamical systems. Thus, the hope is to analyze the simpler systems instead. In [23], the authors comment that the system transformations are not accompanied by formal guarantees.

We check for conformance using the Skorokhod metric. We pick a scaling factor of 2 for the time domain, as a time-jitter of 0.5 s is the maximum deviation we wish to tolerate in the settling time, and pick \(0.68 = \frac{1}{0.1*\lambda _{\text {ref}}}\) as the scaling factor for \(\lambda \) (which corresponds to the worst case tolerated discrepancy in the overshoot). The scaling factors transform the problem into one where a Skorokhod distance of greater than 1 is equivalent to time distortion and/or overshoot being unacceptable in \(\mathfrak {A}_2\).

Conformance testing for closed-loop A/F ratio controller at different engine speeds

Engine speed (rpm) | Skorokhod distance | Computation time (s) | Total time taken (secs) | Number of simulations |
---|---|---|---|---|

1000 | 0.47 | 218 | 544 | 700 |

1500 | 0.20 | 240 | 553 | 700 |

2000 | 0.27 | 223 | 532 | 700 |

As shown in Table 5, our conformance testing tool found the worst-case Skorokhod distance between the two models to be 0.469 at the speed of 1000 rpm.^{7} Even under the worst-case assumption that the dominant contribution to the Skorokhod distance is from the value-domain, this means that the worst-case discrepancy between the output signals for the two models is less than \(0.469\frac{1}{0.1\lambda _{\text {ref}}}\) = 0.31 (which is less than the 0.588 limit postulated above). This implies that the model with nonpolynomial dynamics indeed satisfies Requirement (18).

The polynomial dynamical model was obtained by approximating the nonlinear dynamics in the second model by a polynomial corresponding to minimize the error between the dynamics functions at the operating point of 1000 rpm. Hence, the largest Skorokhod distance between the models being at the input condition of 1000 rpm was surprising to the developers of these models. The designers’ expectation was thus to see the two models be most conformant at 1000 rpm, and less conformant at other speeds. The root cause was this discrepancy was determined to be the increased sensitivity of the system dynamics at 1000 rpm, which led to pronounced differences in the transient behavior upon small perturbations to the dynamics. However, this was not obvious *a priori*, and was investigated only due to the unexpected results from conformance testing.

*Case study III: engine timing model* The Simulink demo palette from Mathworks [29] contains a system representing a four-cylinder spark ignition internal combustion engine based on a model by Crossley and Cook [13]. This system is then enhanced by adding a proportional plus integral (P + I) control law. The integrator is used to adjust the steady-state throttle as the desired engine speed set-point changes, and the proportional term compensates for phase lag introduced by the integrator. In an actual implementation of such a system, such a P+I controller is implemented using a discrete-time integrator. Such integrator blocks are typically associated with a particular numerical integration technique, e.g., forward-Euler, backward-Euler, trapezoidal, etc. It is expected that different numerical techniques will produce slight variation in the results. We wish to quantify the effect of using different numerical integrators in a closed-loop setting. We checked if the user-provided tolerance of \(\delta = 1.0\) is satisfied by systems \(\mathfrak {A}_1\) and \(\mathfrak {A}_2\), where \(\mathfrak {A}_1\) is the original system provided in [29] and \(\mathfrak {A}_2\) is a modified system that uses the backward Euler method to compute the discrete-time integral in the controller. We scale the outputs in such a way that a value discrepancy of \(1\%\) of the the output range (\(\sim \)1000) is equivalent to a time discrepancy of 0.1 s. These values are chosen to bias the search towards finding signals that have a small time jitter. This is an interesting scenario for this case study where the two systems are equivalent except for the underlying numerical integration solver. We find the signal shown in Fig. 4, for which we find output traces with Skorokhod distance 1.04. The experiment uses 296 simulations and the total time taken to find the counterexample is 677 s.

## 6 Conclusion

We argue that the Skorokhod metric provides a robust basis for checking conformance between dynamical systems. We showed that it provides transference of a rich class of temporal logic properties and that it can be computed efficiently, both in theory and in practice. Our experiments indicate that conformance checking using the Skorokhod metric can be integrated into a testing flow for Simulink models and can find non-conformant behaviors effectively, allowing for independent weighing of time and value distortions.

## Footnotes

- 1.
TPTL has only time freeze variables, and STL\(^*\) has only value freeze variables. FLTL has both. Moreover, the predicates over the freeze variables can be non-linear in FLTL, unlike STL\(^*\) which only allows linear predicates.

- 2.
Instead of having only two parameters \(\tau \) and \(\epsilon \) for time and state variation, we generalize to \(n+1\) parameters: we pre-scale time and the

*n*state components with \(n+1\) constants, and have a single value quantifying closeness of the scaled traces. - 3.
Informally, two signals

*x*,*y*are \((T,J,(\tau , \epsilon ))\)-close if for each point*x*(*t*), there is a point \(y(t')\) with \(|t-t'| < \tau \) such that \(\mathscr {D}(x(t), y(t')) <\epsilon \); and similarly for*y*(*t*). - 4.
The two components of the Skorokhod distance (the retiming, and the value difference components) can be weighed with different weights—this simply corresponds to a change of scale.

- 5.
\(\mathscr {D}_{\mathscr {S}}^W\) is not a metric over traces (the triangle inequality fails).

- 6.
STL is MITL enriched with booleanzing predicates without freeze variables.

- 7.
In the version of this paper published previously [15], this number was reported as 0.31. This discrepancy can be attributed to the random seed selected by the optimizer used to maximize the Skorokhod distance. We report the higher number in this paper, as it was the maximum value obtained after trying a number of different random seeds.

## Notes

### Acknowledgements

Open access funding provided by Max Planck Society.

### References

- 1.Abbas H, Fainekos GE (2014) Formal property verification in a conformance testing framework. In: MEMOCODE, to appearGoogle Scholar
- 2.Abbas H, Hoxha B, Fainekos GE, Deshmukh JV, Kapinski J, Ueda K (2014) Conformance testing as falsification for cyber-physical systems. CoRR, arXiv:1401.5200
- 3.Althoff M (2013) Reachability analysis of nonlinear systems using conservative polynomialization and non-convex sets. In: HSCC 13, pp 173–182Google Scholar
- 4.Alur R, Henzinger TA (1994) A really temporal logic. J ACM 41(1):181–204MathSciNetCrossRefMATHGoogle Scholar
- 5.Annpureddy Y, Liu C, Fainekos GE, Sankaranarayanan S (2011) S-TaLiRo: a tool for temporal logic falsification for hybrid systems. In: Proc TACAS, pp 254–257Google Scholar
- 6.Bouyer P, Chevalier F, Markey N (2005) On the expressiveness of TPTL and MTL. In: FSTTCS 05, LNCS. vol 3821, pp 432–443. Springer, BerlinGoogle Scholar
- 7.Branicky MS (1995) Studies in hybrid systems: modeling, analysis, and control. PhD thesis, Massachusetts Institute of Technology, Cambridge, MAGoogle Scholar
- 8.Brim L, Dluhos P, Safránek D, Vejpustek T (2014) STL\(^*\): extending signal temporal logic with signal-value freezing operator. Inf Comput 236:52–67MathSciNetCrossRefMATHGoogle Scholar
- 9.Brim L, Vejpustek T, Safránek D, Fabriková J (2013) Robustness analysis for value-freezing signal temporal logic. In: Proceedings second international workshop on hybrid systems and biology, HSB 2013, EPTCS. vol 125, pp 20–36Google Scholar
- 10.Broucke M (1998) Regularity of solutions and homotopic equivalence for hybrid systems. IEEE Conf Decis Control 4:4283–4288Google Scholar
- 11.Caspi P, Benveniste A (2002) Toward an approximation theory for computerised control. In: EMSOFT, pp 294–304. Springer, BerlinGoogle Scholar
- 12.Chen X, Ábrahám E, Sankaranarayanan S (2013) Flow*: an analyzer for non-linear hybrid systems. CAV 13:258–263MathSciNetGoogle Scholar
- 13.Crossley PR, Cook JA (1991) A nonlinear engine model for drivetrain system development. In: International conference on control, pp 921–925. IETGoogle Scholar
- 14.Davoren JM (2009) Epsilon-tubes and generalized Skorokhod metrics for hybrid paths spaces. In: HSCC, LNCS. vol 5469, pp 135–149. Springer, BerlinGoogle Scholar
- 15.Deshmukh JV, Majumdar R, Prabhu VS (2015) Quantifying conformance using the skorokhod metric. In: Computer aided verification, CAV 2015, Part II, LNCS. vol 9207, pp 234–250. Springer, BerlinGoogle Scholar
- 16.Donzé A, Maler O (2010) Robust satisfaction of temporal logic over real-valued signals. In: FORMATS, LNCS. vol 6246, pp 92–106. Springer, BerlinGoogle Scholar
- 17.Donzé Alexandre (2010) Breach, a toolbox for verification and parameter synthesis of hybrid systems. In: CAV, pp 167–170Google Scholar
- 18.Duggirala PS, Mitra S, Viswanathan M (2013) Verification of annotated models from executions. In: EMSOFT 13, pp 26Google Scholar
- 19.Girard A, Pola G, Tabuada P (2010) Approximately bisimilar symbolic models for incrementally stable switched systems. IEEE Trans Autom Control 55(1):116–126MathSciNetCrossRefMATHGoogle Scholar
- 20.Haghverdi E, Tabuada P, Pappas GJ (2005) Bisimulation relations for dynamical, control, and hybrid systems. Theor Comput Sci 342(2–3):229–261MathSciNetCrossRefMATHGoogle Scholar
- 21.Hennessy M, Milner R (1985) Algebraic laws for nondeterminism and concurrency. J ACM 32(1):137–161MathSciNetCrossRefMATHGoogle Scholar
- 22.Henzinger MR, Henzinger TA, Kopke PW (1995) Computing simulations on finite and infinite graphs. In: FOCS: Foundations of Computer Science, pp 453–462. IEEE Computer SocietyGoogle Scholar
- 23.Jin X, Deshmukh JV, Kapinski J, Ueda K, Butts K (2014) Powertrain control verification benchmark. In: HSCC 14, pp 253–262Google Scholar
- 24.Kapinski J, Deshmukh JV, Sankaranarayanan S, Arechiga N (2014) Simulation-guided lyapunov analysis for hybrid dynamical systems. In: HSCC 14, pp 133–142. ACM, New YorkGoogle Scholar
- 25.Koymans R (1990) Specifying real-time properties with metric temporal logic. Real-Time Syst 2(4):255–299CrossRefGoogle Scholar
- 26.Majumdar R, Prabhu VS (214) Computing the Skorokhod distance between polygonal traces (full paper). CoRR, arXiv:1410.6075
- 27.Majumdar R, Prabhu VS (2015) Computing the Skorokhod distance between polygonal traces. In: HSCC. ACM, New YorkGoogle Scholar
- 28.The MathWorks. https://www.mathworks.com/
- 29.The Mathworks. Engine timing model with closed loop control. https://www.mathworks.com/help/simulink/examples/engine-timing-model- with-closed-loop-control.html
- 30.Messner W, Tilbury D. Control tutorials for matlab and simulink. https://www.mathworks.com/academia/courseware/control-tutorials.html
- 31.Milner R (1980) A calculus of communicating systems, LNCS. vol 92, Springer, BerlinGoogle Scholar
- 32.Sangiorgi D, Rutten J (2011) Advanced topics in bisimulation and coinduction. cambridge University Press, CambridgeCrossRefMATHGoogle Scholar
- 33.Süli E, Mayers DF (2003) An introduction to numerical analysis. Cambridge University Press, CambridgeCrossRefMATHGoogle Scholar
- 34.Tabuada P (2009) Verification and control of hybrid systems: a symbolic approach. Springer, BerlinCrossRefMATHGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.