Quantitative monitoring of STL with edit distance
 672 Downloads
 2 Citations
Abstract
In cyberphysical systems (CPS), physical behaviors are typically controlled by digital hardware. As a consequence, continuous behaviors are discretized by sampling and quantization prior to their processing. Quantifying the similarity between CPS behaviors and their specification is an important ingredient in evaluating correctness and quality of such systems. We propose a novel procedure for measuring robustness between digitized CPS signals and signal temporal logic (STL) specifications. We first equip STL with quantitative semantics based on the weighted edit distance, a metric that quantifies both space and time mismatches between digitized CPS behaviors. We then develop a dynamic programming algorithm for computing the robustness degree between digitized signals and STL specifications. In order to promote hardwarebased monitors we implemented our approach in FPGA. We evaluated it on automotive benchmarks defined by research community, and also on realistic data obtained from magnetic sensor used in modern cars.
Keywords
Weighted edit distance Robustness Hardware monitors Runtime verification Dynamic programming1 Introduction
Cyberphysical systems (\(\textsc {CPS}\)) integrate heterogeneous collaborative components that are interconnected between themselves and their physical environment. They exhibit complex behaviors that often combine discrete and continuous dynamics. The sophistication, complexity and heterogeneity of \(\textsc {CPS}\) makes their verification a difficult task. Runtime monitoring addresses this problem by providing a formal, yet scalable, verification method. It achieves both rigor and efficiency by enabling evaluation of systems according to the properties of their individual behaviors.
In the recent past, propertybased runtime monitoring of \(\textsc {CPS}\) centered around signal temporal logic (\(\textsc {STL}\)) [29] and its variants have received considerable attention [2, 6, 7, 14, 15, 18, 31]. \(\textsc {STL}\) is a formal specification language for describing properties of continuous and hybrid behaviors. In its original form, \(\textsc {STL}\) allows to distinguish correct from incorrect behaviors. However, the binary true/false classification may not be sufficient for realvalued behaviors. The classical satisfaction relation can be replaced by a more quantitative robustness degree [14, 15, 18] of a behavior with respect to a temporal specification. The robustness degree provides a finer measure of how far is the behavior from satisfying or violating of the specification.
Here we propose a novel quantitative semantics for \(\textsc {STL}\) that measures the behavior mismatches in both space and time. We consider applications in which continuous \(\textsc {CPS}\) behaviors are observed by a digital device. In this scenario, continuous behaviors are typically discretized, both in time and space, by an analogtodigital converter (ADC). As a consequence, we interpret \(\textsc {STL}\) over discretetime digitized behaviors.
 1.
It is cumulative, hence it can differentiate between a single and multiple deviations from a reference behavior;
 2.
It combines spatial and temporal aspects, which are both important when reasoning about CPS behaviors; and
 3.
It is defined in discrete time, which is an important aspect for the applications that we consider.

we provide extensive proofs of the theoretical results in [25]

we test our approach on an industrial case study with data taken from a real magnetic sensor and verify timing requirements of Single Edge Nibble Transmission Protocol [24], which are crucial for the integrity of information transferred

we further benchmark our approach with faulttolerant fuel control system [5] model, taken from the automotive domain
2 Related work
The Levenshtein (edit) distance [28] has been extensively used in information theory, computer science and bioinformatics for many applications, including approximate string matching, spell checking and fuzzy string searching. Levenshtein automata [37] were introduced to reason about the edit distance from a reference string. A Levenshtein automaton of degree n for a string w recognizes the set of all words whose edit distance from w is at most n. A dynamic programming procedure for computing the edit distance between a string and a regular language has been proposed in [42]. The problem of computing the smallest edit distance between any pair of distinct strings in a regular language has been studied in [26]. In contrast to our work, these classical approaches to edit distance consider only operations with simple weights on unordered alphabets and are not applied to dynamic reactive behaviors.
The edit distance for weighted automata was studied in [30], where the authors propose a procedure for computing the edit distance between weighted transducers. A space efficient algorithm for computing the edit distance between a string and a weighted automaton over a tropical semiring was developed in [3]. The resulting approach is generic and allows for instance to assign an arbitrary cost to each substitution pair. However, all substitution pairs must be enumerated by separate transitions. In contrast, we consider signals with naturally ordered alphabets as input strings and hence can efficiently handle substitution over large alphabets by treating allowed input values with symbolic constraints. In addition, we use the edit distance to define the semantics of a temporal specification formalism.
The weighted Hamming and edit distances between behaviors are also proposed in [36], where the authors use it to develop procedures for reasoning about the Lipshitzrobustness of Mealy machines and string transducers. The notion of robustness is different from ours, and in contrast to our work it is not computed against a specification.
The quantitative semantics for temporal logics were first proposed in [18, 35], with the focus on the spatial similarity of behaviors, given by their pointwise comparison. The spatial quantitative semantics is sensitive to phase shifts and temporal inaccuracies in behaviors—a small temporal shift in the behavior may result in a large robustness degree change. This problem was addressed in [15], in which \(\textsc {STL}\) with spatial quantitative semantics is extended with time robustness. In [2], the authors propose another approach of combining space and time robustness, by extending \(\textsc {STL}\) with averaged temporal operators. Another approach to determining robustness of hybrid systems using selfvalidated arithmetics is shown in [19]. Monitoring of different quantitative semantics is implemented in tools such as STaLiRo [4] and Breach [13].
The problem of online monitoring robustness was studied more recently in [9, 12]. The authors of [12] propose an online monitoring approach that uses a predictor, which requires for the future fragment of the logic the access to a model of the system. This is in contrast to our blackbox view of monitoring. In [9], the authors propose an intervalbased approach of online evaluation that allows estimating the minimum and the maximum robustness with respect to both the observed prefix and unobserved suffix of the trace. In our work, we do not provide such estimation about the future. Instead, our robustness value at every point in time gives the distance of the observed prefix from the satisfaction/violation of the specification.
The recent results on using Skorokhod metric [39] to compute the distance between piecewiselinear or piecewiseconstant continuous behaviors [10] partially inspired our work. Skorokhod metric quantifies both space and time mismatches between continuous behaviors by allowing application of time distortions in behaviors in order to minimize their pointwise distance. The distortion of the timeline is achieved by applying a retiming function—a continuous bijective strictly increasing function from time domain to time domain. Given a behavior x(t), the resulting retimed behavior r(x(t)) preserves the values and their order but not the duration between two values. This informationpreserving distance relies on continuous time and is not applicable to the discrete time domain—stretching and compressing the discrete time axis results inevitably in an information loss. Finally, the computation of the Skorokhod distance was extended to the flowpipes in [11] and to the epsilontubes in [8],where the authors consider computing the distance between hybrid (continuous and discretetime) signals. We are not aware of any work addressing the problem of computing the Skorokhod distance between a behavior and a temporal specification.
Our work is also related with the notions of \((\upvarepsilon , \uptau )\)closeness in [1] and \((\upvarepsilon , \uptau )\)similarity (requires the retiming to be orderpreserving) introduced in [34] to compare two mixedanalog signals and in conformance testing [1]. The parameters \(\uptau \) and \(\upvarepsilon \) are used to specify how much it is allowed to wiggle in both time and space in order to transform one trace into another. The main difference with this work is that our distance provides a cumulative measure, while the other notions try to find the max possible discrepancy.
Recently published industrial case study [38] shows an application of STL monitoring for verifying the sensor which uses SENT [24] protocol. We regard that work as completely orthogonal to this paper. The case study focuses on qualitative monitors able to recover upon violation detection and that are able to detect and collect multiple violations in one go. The framework in that paper is limited to a particular class of asynchronous communication protocols. In contrast, this paper is about quantitative monitoring for arbitrary STL properties.
3 Preliminaries
In this section, we provide the necessary definitions to develop the algorithm presented in subsequent sections of the paper. We first shortly recall the notion of metric spaces and distances. We then define signals and signal temporal logic. Finally, we introduce a variant of symbolic and weighted symbolic automata.
3.1 Metric spaces and distances
A metric space is a set for which distances between all elements in the set are defined.
Definition 1
(Metric space and distance) Suppose that \({\mathcal {M}}\) is a set and \(d~:~{\mathcal {M}}\times {\mathcal {M}} \rightarrow {\mathbb {R}}\) is a function that maps pairs of elements in \({\mathcal {M}}\) into the real numbers. Then \({\mathcal {M}}\) is a metric space with the distance measure d, if (1) \(d(m_{1},m_{2}) \ge 0\) for all \(m_{1},m_{2}\) in \({\mathcal {M}}\); (2) \(d(m_{1},m_{2}) = 0\) if and only if \(m_{1} = m_{2}\); (3) \(d(m_{1},m_{2}) = d(m_{2},m_{1})\) for all \(m_{1},m_{2}\) in \({\mathcal {M}}\); and (4) \(d(m_{1},m_{2}) \le d(m_{1},m) + d(m,m_{2})\) for all \(m,m_{1},m_{2}\) in \({\mathcal {M}}\).
3.2 Signals
Let X be a finite set of variables defined over some domain \({\mathbb {D}}\). Then, a signal s is a function \(s~:~{\mathbb {T}} \times X \rightarrow {\mathbb {D}}\), where \({\mathbb {T}}\) is the time domain^{1}. We distinguish between analog, discrete and digital signals. Analog signals have continuous value and time domains. The time domain of discrete signals is the set of integers, while digital signals have in addition their value domain restricted to a finite set. Digital signals can be obtained by sampling and quantization of analog signals. The conversion of analog to digital signals is at the core of the signal processing field and is in practice done by an analogtodigital converter (ADC).
Sampling is the process of reducing the continuous time in analog signals to the discrete time in the resulting discrete signal. The ideal theoretical sampling function periodically measures the value of the analog signal every T time units, where T denotes the sampling interval. Similarly, we denote by f the sampling frequency, that is the average number of measurements obtained by sampling in one second, where \(f = 1/T\). Given an analog signal \(s_a~:~{\mathbb {R}}_{\ge 0} \times X \rightarrow {\mathbb {R}}^{n}\) and a sampling interval T, applying the ideal sampling function to \(s_a\) results in a discrete signal \(s_{disc }~:~{\mathbb {N}} \times X \rightarrow {\mathbb {R}}\) such that \(s_{disc }(i, x) = s_a(iT, x)\) for all \(i \ge 0\) and \(x \in X\).
3.3 Signal temporal logic
In this paper, we study signal temporal logic (\(\textsc {STL}\)) with both past and future operators interpreted over digital signals of finite length.^{2}
3.4 Automata and weighted automata
In this section, we define a variant of symbolic automata [41] and also present its weighted extension. The notion of weighted automata and its wellestablished theory is provided in [16] while symbolic weighted automata accepting input string over not necessarily finite set have been investigated in [21].
Similarly to the definition of STL, we consider \({\mathbb {D}}= [v_{min },v_{max }]\) to be the finite interval of integers equipped with the distance d and let X to be a finite set of variables defined over \({\mathbb {D}}\). The variable valuation v(x) is a function \(v~:~X \rightarrow {\mathbb {D}}\), which we naturally extend to the valuation v(X) of the set X. A variable constraint \(\upgamma \) over X is defined by the grammar in negation normal form \(\upgamma := x \le c~~\lnot (x \le c)~~\upgamma _{1} \vee \upgamma _{2}~~\upgamma _{1} \wedge \upgamma _{2}\), where \(x \in X\) and \(c \in {\mathbb {D}}\). We denote by \(\Gamma (X)\) the set of all constraints definable over X. Given the valuation v(X) and a constraint \(\upgamma \) over X, we write \(v(X) \models \upgamma \) when v(X) satisfies \(\upgamma \).
Definition 2
(Symbolic automata) We define a symbolic automaton \({\mathcal {A}}\) as the tuple \({\mathcal {A}}= ({\mathbb {D}}, X, Q, I, F, \Delta )\), where \({\mathbb {D}}\) is the finite alphabet, X is a finite set of variables over \({\mathbb {D}}\), Q is a finite set of states, \(I \subseteq Q\) is the set of initial states, \(F \subseteq Q\) is the set of final states and \(\Delta = \Delta _{X} \cup \Delta _{\upvarepsilon }\) is the transition relation, where \(\Delta _{X} \subseteq Q \times \Gamma (X) \times Q\) and \(\Delta _{\upvarepsilon } \subseteq Q \times \{ \upvarepsilon \} \times Q\) are sets of transitions that consume an input letter and silent transitions, respectively.
Given a \(q \in Q\), let \({\mathcal {E}}(q)\) denote the set of states reachable from q by following \(\upvarepsilon \)transitions in \(\Delta \) only. Formally, we say that \(p \in {\mathcal {E}}(q)\) iff there exists a sequence of states \(q_{1}, \ldots , q_{k}\) such that \(q = q_{1}\), \((q_{i}, \upvarepsilon , q_{i+1}) \in \Delta \) for all \(0 \le i < k\), and \(p = q_{k}\). Let \(s~:~[0,l) \times X \rightarrow {\mathbb {D}}\) be a signal. We say that s is a trace of \({\mathcal {A}}\) if there exists a sequence of states \(q_{0}, \ldots , q_{l}\) in Q such that \(q_{0} \in {\mathcal {E}}(q)\) for some \(q \in I\), for all \(0 \le i < l\), there exists \((q_{i}, \upgamma , q_{i+1}) \in \Delta \) for some \(\upgamma \) such that \(s(i, X) \models \upgamma \) and \(q_{i+1} \in {\mathcal {E}}(q)\) and \(q_{l} \in F\). We denote by \(L({\mathcal {A}})\) the set of all traces of \({\mathcal {A}}\). A path \(\uppi \) in \({\mathcal {A}}\) is a sequence Open image in new window such that \(q_{0} \in I\) and for all \(0 \le i < n\), \(\updelta _{i}\) is either of the form \((q_{i}, \upgamma , q_{i+1})\) or \((q_{i}, \upvarepsilon , q_{i+1})\). We say that \(\uppi \) is accepting if \(q_{n} \in F\). Given a trace \(s~:~[0,l) \times X \rightarrow {\mathbb {D}}\) and a path \(\uppi = q_{0} \cdot \updelta _{0} \cdot q_{1} \cdot \updelta _{1} \cdots \updelta _{n1} \cdot q_{n}\), we say that s induces \(\uppi \) in \({\mathcal {A}}\) if \(\uppi \) is an accepting path in \({\mathcal {A}}\) and its projection to observable alphabet letters gives s. We denote by \(\Pi ({\mathcal {A}},s) = \{ \uppi ~~s \text { induces } \uppi \text { in } {\mathcal {A}}\}\) the set of all paths in \({\mathcal {A}}\) induced by s.
We now introduce weighted symbolic automata, by adding a weight function to the transitions of the symbolic automaton, relative to the consumed input letter.
Definition 3
(Weighted symbolic automata) A weighted symbolic automaton \({\mathcal {W}}\) is the tuple \({\mathcal {W}}= ({\mathbb {D}}, X, Q, I, F, \Delta , \uplambda )\), where \({\mathcal {A}}= ({\mathbb {D}}, X, Q, I, F, \Delta )\) is a symbolic automaton and \(\uplambda ~:~ \Delta \times ({\mathbb {D}}^{X} \cup \{ \upvarepsilon \}) \rightarrow {\mathbb {Q}}^{+}\) is the weight function.
Let s be a signal of size l and \(\uppi = q_{0} \cdot \updelta _{0} \cdots \updelta _{n1} \cdot q_{n}\) a path in \({\mathcal {W}}\) induced by s. The value of \(\uppi \) in \({\mathcal {W}}\) subject to s, denoted by \(v_{ \uppi } (s, {\mathcal {W}})\), is the sum of weights associated to the transitions in the path \(\uppi \) and subject to the signal s. We define the value \(v(s,{\mathcal {W}})\) of s as the minimum value from all the paths in \({\mathcal {W}}\) induced by s, i.e. \(v(s,{\mathcal {W}}) = \min _{\uppi \in \Pi ({\mathcal {W}},s)} v_{ \uppi } (s, {\mathcal {W}})\).
4 Weighted edit distance
Measuring the similarity of sequences is important in many application areas, such as information theory, spell checking and bioinformatics. The Hamming distance \(d_{H}\) is the most basic and common string measure arising from the information theory. It measures the minimum number of substitution operations needed to match equal length sequences. The edit distance \(d_{E}\) extends the Hamming distance with two additional operations, insertion and deletion and is defined as the minimum accumulation of edit operation costs used to transform one sequence into the other.
Neither of these metrics provide satisfactory solution for comparing digitized signals. They are defined over unordered alphabets and associate fixed costs to different kinds of operations. In contrast, the value domain of digital signals admits a natural notion of a distance representing the difference between two signal valuations. In addition, the Hamming distance provides only pointwise comparisons between sequences and consequently does not account for potential timing discrepancies in the sampled signals. Two discrete signals that differ only in a constant time delay will typically have a large Hamming distance. The edit distance addresses this problem by allowing us to bridge the time shifts using insertion and deletion operations.
Inspired by [30, 36], we propose the weighted edit distance as the measure for comparing the similarity of two discrete signals. It adopts the insertion and deletion operations from the edit distance and adapts the substitution operation to the ordered alphabets. Since we consider multidimensional signals, we extend the cost of the substitution operation to take into account different variable valuations.
Let X be a finite set of variables defined over some interval domain \({\mathbb {D}}= [v_{min },v_{max }]\). Given two valuation vectors \(a,b \in {\mathbb {D}}^{X}\) of X, we denote by \(d_{M}(a, b)\) the Manhattan distance [27] between a and b, where \(d_{M}(a,b) = \Sigma _{i=0}^{X1} a_{i}  b_{i}\). Let \(w_{i},w_{d} \in {\mathbb {Q}}\) be weight constants for the insertion and deletion operations. We then define the costs of the substitution \(c_{s}\), insertion \(c_{i}\) and deletion \(c_{d}\) operations as follows: \((1) c_{s}(a,b) = d_{M}(a,b); (2) c_{i} = w_{i}; (3) c_{d} = w_{d}\). The definition of the WED adapts the classical edit distance recursive definition with the new costs.
Definition 4
Proposition 1
The weighted edit distance is a distance.
Remark We chose the Manhattan distance for the substitution cost because it combines the absolute difference of several signal components.
We now further motivate the use of the weighted edit distance and discuss in more depth its characteristics. We do this by comparing the weighted edit distance (\(d_{W}\)) to the Hamming distance (\(d_{H}\)) and to the distance based on the infinity norm (\(d_{max }\)). In order to compare these three distances, we record the data from a device implementing an automotive communication protocol. We manually manipulate the data to illustrate specific distance properties. We note that we normalize the two cumulative distances with the total number of data samples, in order to have comparable results.
4.1 Sampling, quantization and weighted edit distance
We compute the WED between digital signals resulting from physical behavior observations after sampling and quantization. In this section, we discuss the effect of inaccuracies introduced by these operations on the WED.
Let s be an analog signal, T a sampling period and \({\mathsf {Q}}\) a quantization step. We assume that s has a band limit \(f_{M}\) and \( T \le 1/(2f_{M})\). We denote by s[T] the discrete signal obtained from s by sampling with the period T, and by \(s[T][{\mathsf {Q}}]\) the digital signal obtained from s[T] by quantization with the step \({\mathsf {Q}}\).
We cannot directly relate the WED to the analog signals, because it is not defined in continuous time. However, this distance allows tackling phase shifts in the sampled signals. Consider two analog signals \(s_{1}(t)\) and \(s_{2}(t  \uptau )\) such that \(\uptau = iT\) for some \(i \ge 0\) and their sampled variants \(s_{1}[T](t)\) and \(s_{2}[T](t)\). It is clear that with \(2\cdot i\) insertion and deletion operations, \(s_{2}[T]\) can be transformed into \(s_{1}[T]\) such that their remaining substitution cost equals to 0. This situation is illustrated in Fig. 4 (see signals \(s_{1}\) and \(s_{2}\)). We see that the distance between the two signals initially grows due to the insertion and deletion operations, but that eventually it becomes perfectly stable.
It is obvious that the actual distance between two behaviors is affected by the sampling frequency. We refer to [40] for the survey on the sampling theory, a field that studies the effects of sampling continuous behaviors.
4.2 Normalized weighted edit distance
The weighted edit distance is an accumulative distance. It follows that the distance between two behaviors depends on several factors, including: 1) the size of the value domains; 2) the frequency at which the two signals are sampled; and 3) the total duration of the trace. For instance, the comparison of two analog behaviors sampled at two different frequencies can result in completely different absolute distance values. In order to have a more uniform robustness valuation that is less affected by the above factors, we propose normalizing the robustness values as follows.
5 Weighted edit robustness for signal temporal logic
In this section, we propose a novel procedure for computing the robustness degree of a discrete signal with respect to an \(\textsc {STL}\) property. In our approach, we set \(c_{i}\) and \(c_{d}\) to be equal to \(X(v_{max } v_{min })\). In other words, the deletion and insertion costs are equal to the largest substitution cost. The rationale behind this choice is that by inserting/deleting a data point, we can add/remove the maximum value from the domain in the worst case.
5.1 From STL to weighted edit automata
Let X be a set of finite variables defined over the domain \({\mathbb {D}}= [v_{min },v_{max }] \subseteq {\mathbb {N}}\). We consider an STL formula \(\upvarphi \) defined over X. Let \(s~:~[0,l) \times X \rightarrow {\mathbb {D}}\) be a digital signal.
5.1.1 From \(\upvarphi \) to \({\mathcal {A}}_{\upvarphi }\)
In the first step, we translate the STL specification \(\upvarphi \) into the automaton \({\mathcal {A}}_{\upvarphi }\) such that \(L(\upvarphi ) = L({\mathcal {A}}_{\upvarphi })\). The translation from \(\textsc {STL}\) interpreted over discrete time and finite valued domains to finite automata is standard, and can be achieved by using for instance onthefly tableau construction [20] or the temporal testers approach [33]. We note that we need to accommodate these classic constructions to the finitary semantics of the temporal logic by adapting accordingly the acceptance conditions (see for instance [17] for the interpretation of LTL over finite traces).
Example 1
5.1.2 From \({\mathcal {A}}_{\upvarphi }\) to \({\mathcal {W}}_{\upvarphi }\)
In this step, we translate the automaton \({\mathcal {A}}_{\upvarphi }\) to the weighted edit automaton \({\mathcal {W}}_{\upvarphi }\). The automaton \({\mathcal {W}}_{\upvarphi }\) reads an input signal and mimics the weighted edit operations. In essence, \({\mathcal {W}}_{\upvarphi }\) accepts every signal along multiple paths. Each accepting path induced by the signal corresponds to a sequence of weighted edit operations needed to transform the input signal into another one allowed by the specification. The value of the least expensive path corresponds to the weighted edit distance between the input signal and the specification. The weighted automaton \({\mathcal {W}}_{\upvarphi }\) explicitly treats substitution, insertion and deletion operations, by augmenting \({\mathcal {A}}_{\upvarphi }\) with additional transitions and associating to them the appropriate weight function. We now provide details of the translation and describe the handling of weighted edit operations. Let \({\mathcal {A}}_{\upvarphi } = ({\mathbb {D}}, X, Q, I, F, \Delta )\) be the symbolic automaton accepting the language of the specification \(\upvarphi \).

\((q, \textsf {true}, q') \in \Delta _{s}\) if there exists \((q, \upgamma , q') \in \Delta \) for some \(\upgamma \); and

\(\uplambda _{s}((q,\textsf {true},q'),v) = d_{M}(v, \upgamma (q,q'))\), for all \(v \in {\mathbb {D}}^{X}\).
Intuitively, we replace all the transitions in \({\mathcal {A}}_{\upvarphi }\) with new ones that have the same source and target states. We relax the guards in the new transitions and make them enabled for any input. On the other hand, we control the cost of making a transition with the weight function \(\uplambda _{s}\), which computes the substitution cost needed to take the transition with a specific input. This cost is the Manhattan distance between the input value and the guard associated to the original transition.

\((q, \textsf {true}, q) \in \Delta _{d}\) if \((q, \textsf {true}, q) \not \in \Delta _{s}\); and

\(\uplambda _{d}(\updelta , v) = c_{d}\) for all \(\updelta \in \Delta _{d}\) and \(v \in {\mathbb {D}}^{X}\).

\((q, \upvarepsilon , q') \in \Delta _{i}\) if \((q, \upgamma , q') \in \Delta \) for some \(\upgamma \); and

\(\uplambda _{i}(\updelta , \{\upvarepsilon \}) = c_{i}\) for all \(\updelta \in \Delta _{i}\).
Example 2
The weighted edit automaton \({\mathcal {W}}_{\upvarphi }\) obtained from \({\mathcal {A}}_{\upvarphi }\) is illustrated in Fig. 6b. Both automatons from Fig. 6b use the same input alphabet \({\mathbb {D}}= \{ 0,1,2,3,4,5 \}\). The blue transitions, such as (A, 0, A) with weight 5, correspond to the deletion transitions. The red transitions, such as \((A,\upvarepsilon , B)\), correspond to the insertion transitions.
The resulting weighted automaton \({\mathcal {W}}_{\upvarphi }\) allows determining the weighted edit distance between a signal w and the formula \(\upvarphi \), by computing the value of s in \({\mathcal {W}}_{\upvarphi }\).
Theorem 1
\(d_{W}(s,\upvarphi ) = v(s, {\mathcal {W}}_{\upvarphi })\).
The consequence of this Theorem is that two symbolic automata that accept the same language will always give the same distance from the same input.
5.2 Computing the value of a signal in a weighted edit automaton
We now present an onthefly algorithm Val, shown in Algorithm 1, that computes the value of a signal s in a weighted automaton \({\mathcal {W}}\). In every step i, the algorithm computes the minimum cost of reaching the state q with the prefix of s consisting of its first i values. After reading a prefix of s, we may reach a state \(q \in Q\) in different ways with different costs. Note that it is sufficient to keep the state with the minimum value in each iteration. It follows that the algorithm requires book keeping Q state value fields in every iteration. We now explain the details of the algorithm. The procedure first initializes the costs of all the states in \({\mathcal {W}}\) (see Algorithm 2). The initial states are set to 0 and the noninitial ones to \(\infty \). Then, we compute the effect of taking the \(\upvarepsilon \) transitions without reading any signal value. It is sufficient to iterate this step Q times, since within Q iterations, one is guaranteed to reach a state q that was already visited with a smaller value v. In every subsequent iteration i, we first update the state values by applying the cost of taking all transitions labeled by s(i, X) and then update the effect of taking \(\upvarepsilon \) transitions Q times. The weight function of a substitution cost is computed as follows: \(\uplambda (v,x \le k)\) gives 0 if \(v \le k\), and \(vk\) otherwise; \(\uplambda (v,\lnot (x \le k))\) is symmetric; \(\uplambda (v, \upvarphi _{1} \wedge \upvarphi _{2}) = \max (\uplambda (v,\upvarphi _{1}), \uplambda (v, \upvarphi _{2}))\) and \(\uplambda (v, \upvarphi _{1} \vee \upvarphi _{2}) = \min (\uplambda (v,\upvarphi _{1}), \uplambda (v, \upvarphi _{2}))\).
Upon termination, the algorithm returns the minimum cost of reaching an accepting state in the automaton.
Theorem 2
Val\((s, {\mathcal {W}}) = v(s, {\mathcal {W}})\).
Theorem 3
Given a signal s of length l defined over X and a weighted automaton \({\mathcal {W}}\) with n states and m transitions, \(\textsf {Val}(s,{\mathcal {W}})\) takes in the order of \({\mathcal {O}}(lnm))\) iterations to compute the value of s in \({\mathcal {W}}\), and requires in the order of \({\mathcal {O}}(n(\lceil log (l(v_{max }v_{min }))\rceil ))\) memory.
Example 3
6 Implementation and case study
We now describe our implementation of quantitative monitors for STL. In order to evaluate our approach, we conducted two case studies. The first case study takes specification from automotive benchmarks published in [5]. In second case study we applied our quantitative monitors on Single Edge Nibble Transmission (SENT) protocol, a standard for sensor to controller communication in the automotive industry [24].
In both cases, parser for STL formulas is developed in Java using ANTLR [32]. In order to translate STL properties into temporal testers, we take basic temporal testers for STL operators and create their product. Then, we convert such top level temporal tester into an acceptor automaton. We use JAutomata [22] library to represent the testers and the acceptors. We then generate quantitative monitor code in Verilog HDL. The resulting monitor is a hardware implementation of the weighted automata and the underlying algorithm for computing the weighted edit distance. The monitor operates at the frequency limited by the maximum achievable frequency of the FPGA.
6.1 Benchmarks for automotive systems
For the evaluation of our approach, we apply it to two benchmarks implemented in Matlab/Simulink and published in [5].
6.1.1 Automatic transmission system
Automatic transmission properties [5]
\(\upvarphi \)  

\(\upvarphi _{1}\)  
\(\upvarphi _{2}\)  
\(\upvarphi _{3}\)  
\(\upvarphi _{4}\)  
\(\upvarphi _{5}\)  
\(\upvarphi _{6}\)  
\(\upvarphi _{7}\)  
\(\upvarphi _{8}\)  
Figure 8 illustrates the monitoring results for \(\upvarphi _{6}\) on a specific gear input. In the depicted scenario, the speed does not reach 120 mph in 4 s, a sufficient condition for the satisfaction of the formula. In order to violate the formula, we need to alter both v and \(\upomega \) signals such that 1) v reaches 120 mph at any moment within the first 4 s; and (2) \(\upomega \) remains continuously below 4500 rpm. These alterations result in (1) a single substitution happening within the first 4 s which is necessary to bring v to 120 mph; and (2) the accumulation of substitution costs in the interval between 7 and 8 s of the simulation where \(\upomega \) actually exceeds 4500 rpm. Note that the robustness degree decreases in the first 4 s. This happens because the actual v increases and the substitution cost needed for v to reach 120 mph is continuously being improved.
Evaluation results for automatic transmission benchmark
\(\uprho \)  \({\mathcal {W}}_{\upvarphi }\)  \({\mathcal {W}}_{\lnot \upvarphi }\)  

Q  \(\Delta \)  #FF  #LUT  Q  \(\Delta \)  #FF  #LUT  
\(\upvarphi _{1}\)  \(\) 2528  2  2  62  260  4  8  94  657 
\(\upvarphi _{2}\)  \(\) 11,423  2  2  75  306  4  11  107  799 
\(\upvarphi _{3}\)  1000  496  1374  4106  53,033  992  2878  8127  106,937 
\(\upvarphi _{4}\)  1000  496  692  3061  22,777  992  1445  6025  44,968 
\(\upvarphi _{5}\)  n/a  n/a  n/a  n/a  n/a  n/a  n/a  n/a  n/a 
\(\upvarphi _{6}\)  5337  405  813  6540  66,085  409  903  6504  73,657 
\(\upvarphi _{7}\)  \(\) 5336  403  903  6504  73,766  405  813  6545  66,116 
\(\upvarphi _{8}\)  n/a  n/a  n/a  n/a  n/a  n/a  n/a  n/a  n/a 
6.1.2 Faulttolerant fuel control system
The second automotive benchmark is based on faulttolerant fuel control system model [5, 23]. This system ensures proper airtofuel ratio in modern car engines. It must be adaptive to any kind of external failures, such as sensor failures. Since the occurrence of failures is modeled by Poisson stochastic processes, this benchmark will evaluate our quantitative monitors with a model of a Stochastic Cyber Physical System.
The system has throttle as an input which affect failure arrival rates. The change in detected fuel level can be caused either by throttle or a sensor failure. Such change directly affects airtofuel ratio \(\uplambda \) which is the output of the model. We sample this variable over time in order to create stimulus for our monitors. We collected 10,000 samples from the model output. We rounded double precision output to 2 decimals, and multiplied it by 100 for easier representation in hardware testbench.
In Fig. 9 we can observe change of \(\uplambda \) and robustness values w.r.t. the formula. We see several \(\uplambda \) pulses caused by the disturbance in the system. Due to the initial conditions, negative robustness is greater than zero. The first pulse is satisfying the requirement since it stabilizes to required \(V_{limit}\) within 1 second time window. Since it satisfies the bounded stabilization property and does not add any WED cost, the negative robustness value remains the same before and after the pulse.
The next disturbance in the system generates more impact on airtofuel ratio. In this case the signal does not stabilize fast enough. Therefore the WED algorithm suggests to substitute problematic parts of the trace with correct values. Since the substitution costs accumulate, the negative robustness keeps increasing. Positive robustness equals zero throughout the simulation due to the fact that the trace is violating the formula from the start.
Evaluation results for faulttolerant fuel control system properties [5]
\(\uprho \)  \({\mathcal {W}}_{\upvarphi }\)  \({\mathcal {W}}_{\lnot \upvarphi }\)  

Q  \(\Delta \)  #FF  #LUT  Q  \(\Delta \)  #FF  #LUT  
\(\upvarphi _{9}\)  \(\) 43,878  882  1493  13,203  119,989  1574  2648  23,624  212,341 
6.2 SENT protocol case study
6.2.1 Formalized SENT requirements
6.2.2 Evaluation results
In order to test the monitors with realistic data, we recorded output from a real magnetic sensor which implements the SENT protocol. We used the Halleffect sensor with SENT interface from Infineon Technologies. The Halleffect cell in this sensor measures the magnetic flux. Such information can be used for linear and angular position sensing. In the automotive domain, this sensor is used to sense steering torque and pedal and throttle position.
According to the SENT standard, devices are configured prior to operation. Therefore, we are allowed to assume that the configuration of SENT frame is static and its structure cannot change during runtime.
In Fig. 12 we can see the first SENT requirement monitored on a trace which represents a correct SENT pulse falling edge. For this pulse we compute both positive and negative robustness degree. In the beginning of the trace, the left hand side of the implication is not satisfied, therefore the entire formula is trivially satisfied and the negative robustness is zero. In contrast, the positive robustness is equal to the WED cost of creating a violating trace—which can be done simply by substituting \(high \) sample with \(mid \), thus making \(\downarrow high \) condition true and the entire formula false. We note that the positive robustness decreases in the course of the execution—this happens because the robustness algorithm dynamically discovers a cheaper way to transform the trace into a violating one.
In Fig. 13 we can see the rising edge timing requirement monitored on a trace which represents a violating SENT pulse. The violating pulse was artificially created from a correct trace which was recorded from the actual sensor. The violation was created by replaying the correct recorded values at a slower speed, which prolonged the rising edge length.
In this case the evolution of positive and negative robustness degree over time is converse to the previous case. The obvious difference is that the final value of the positive robustness is zero and the negative robustness degree is nonnegative. This is valid result because the trace is violating the rising edge timing requirement \( T_{rise} \le 18~\upmu \hbox {s}\). The negative robustness is larger than the positive robustness degree of the previous example in Fig. 12, due to the larger cost of compensating for timing violation of the rising edge.
Evaluation results for SENT protocol properties
\(\uprho \)  \({\mathcal {W}}_{\upvarphi }\)  \({\mathcal {W}}_{\lnot \upvarphi }\)  SAT  

Q  \(\Delta \)  #FF  #LUT  Q  \(\Delta \)  #FF  #LUT  trace  
\(\upvarphi _{10}\)  11  208  627  3272  50,046  498  1945  7745  148,852  Yes 
\(\upvarphi _{11}\)  \(\) 41  558  1677  8865  136,321  1338  5224  21,191  405,604  No 
7 Conclusions and future work
In this paper, we proposed a new procedure for measuring robustness of STL properties based on the weighted edit distance. The distance is cumulative by definition which allows robustness degree to be sensitive on the number of violations of the formula. It is also sensitive to the length of the signal, but also to the sampling rate and the number of components in the signal. Distance normalization would help to obtain a uniform measure of “goodness” of a behavior. Although in this paper we focus on the quantitative semantics of STL, the weighted edit distance can be applied to other specification languages over finite signals.
Our FPGA implementation provides the possibility to quantify the distance to the violation of safety requirements in realtime on actual or emulated hardware. We have successfully demonstrated our approach to check relevant safety properties in the automotive domain, i.e. by monitoring the behavior of the engine through the observation of essential signals such as airtofuel ratio. Furthermore, we show that our method is also suitable to verify wellestablished industrial standard such as the SENT protocol.
Future work Treating the value domain symbolically is natural and we exploit this fact in the paper. On the other hand, combining quantitative semantics with symbolic time is not straightforward. In the qualitative case, representing the time symbolically can be done because there is a certain equivalence between states that have the same discrete location and different clock valuations, and such states can be grouped together. In the quantitative setting, this is not the case—two states with the same discrete location and different clock valuations will in general have different values and hence cannot be grouped together. Such a symbolic representation of quantitative states might be possible if some accuracy can be dropped. We will consider extending our algorithm to automata with discrete clocks.
We plan to exploit the quantitative robustness degree of our framework to gain predictive ability and extend our monitors for the system health and failaware applications.
Footnotes
 1.
We use s(t) to denote the valuation vector of the variables in X at time t.
 2.
Although this segment of \(\textsc {STL}\) is expressively equivalent to \(\textsc {LTL}\), use the \(\textsc {STL}\) name to highlight the explicit notions of realtime and quantitative values in the language.
 3.
The time in \({\mathcal {A}}_{\upvarphi }\) cannot be treated symbolically with digital clocks since every pair of states and clock valuation may behave differently with respect to the WED.
 4.
Since s has only one component, we skip the variable name.
Notes
Acknowledgements
Open access funding provided by Austrian Science Fund (FWF).
Supplementary material
References
 1.Abbas H, Mittelmann HD, Fainekos GE (2014) Formal property verification in a conformance testing framework. In: Proceedings of MEMOCODE 2014: the twelfth ACM/IEEE international conference on formal methods and models for codesign, pp 155–164. IEEE. https://doi.org/10.1109/MEMCOD.2014.6961854
 2.Akazaki T, Tasuo I (2015) Time robustness in MTL and expressivity in hybrid system falsification. In: Proceedings of CAV 2015: the 27th international conference on computer aided verification, LNCS, vol 9207. Springer. https://doi.org/10.1007/9783319216683
 3.Allauzen C, Mohri M (2009) Linearspace computation of the editdistance between a string and a finite automaton. CoRR arXiv:0904.4686
 4.Annpureddy Y, Liu C, Fainekos GE, Sankaranarayanan S (2011) STaLiRo: a tool for temporal logic falsification for hybrid systems. In: Proceedings of TACAS 2011: the 17th international conference on tools and algorithms for the construction and analysis of systems, LNCS, vol 6605, pp 254–257. Springer. https://doi.org/10.1007/9783642198359_21
 5.Bardh Hoxha HA, Fainekos G (2015) Benchmarks for temporal logic requirements for automotive systems. In: Proceedings of ARCH@CPSWeek 2014 and ARCH@CPSWeek 2015: the 1st and 2nd international workshop on applied verification for continuous and hybrid systems, vol 34Google Scholar
 6.Bartocci E, Bortolussi L, Sanguinetti G (2014) Datadriven statistical learning of temporal logic properties. In: Proceedings of FORMATS 2014: the 12th international conference on formal modeling and analysis of timed systems, LNCS, vol 8711, pp 23–37. Springer. https://doi.org/10.1007/9783319105123_3
 7.Brim L, Dluhos P, Safránek D, Vejpustek T (2014) \({STL}^*\): extending signal temporal logic with signalvalue freezing operator. Inf Comput 236:52–67. https://doi.org/10.1016/j.ic.2014.01.012 MathSciNetCrossRefzbMATHGoogle Scholar
 8.Davoren JM (2009) Epsilontubes and generalized Skorokhod metrics for hybrid paths spaces. In: Proceedings of HSCC 2009: the 12th international conference on hybrid systems: computation and control, LNCS, vol 5469, pp 135–149. Springer. https://doi.org/10.1007/9783642006029_10
 9.Deshmukh JV, Donzé A, Ghosh S, Jin X, Juniwal G, Seshia SA (2017) Robust online monitoring of signal temporal logic. Form Methods Syst Des 51(1):5–30. https://doi.org/10.1007/s1070301702867 CrossRefzbMATHGoogle Scholar
 10.Deshmukh JV, Majumdar R, Prabhu VS (2015) Quantifying conformance using the Skorokhod metric (full version). CoRR arXiv:1505.05832
 11.Deshmukh JV, Majumdar R, Prabhu VS (2017) Quantifying conformance using the Skorokhod metric. Form Methods Syst Des 50(2–3):168–206. https://doi.org/10.1007/s1070301602618 CrossRefzbMATHGoogle Scholar
 12.Dokhanchi A, Hoxha B, Fainekos GE (2014) Online monitoring for temporal logic robustness. In: Proceedings RV 2014: the 5th international conference on runtime verification, LNCS, vol 8734, pp 231–246. Springer. https://doi.org/10.1007/9783319111643_19
 13.Donzé A (2010) Breach, a toolbox for verification and parameter synthesis of hybrid systems. In: Proceedings of CAV 2010: the 22nd international conference on computer aided verification, LNCS, vol 6174, pp 167–170. Springer. https://doi.org/10.1007/9783642142956_17
 14.Donzé A, Ferrère T, Maler O (2013) Efficient robust monitoring for STL. In: Proceedings of CAV 2013: the 25th international conference on computer aided verification, LNCS, vol 8044, pp 264–279. Springer. https://doi.org/10.1007/9783642397998
 15.Donzé A, Maler O (2010) Robust satisfaction of temporal logic over realvalued signals. In: Proceedings of FORMATS 2010: the 8th international conference on formal modeling and analysis of timed systems, LNCS, vol 6246, pp 92–106. Springer. https://doi.org/10.1007/9783642152979
 16.Droste M, Kuich W, Vogler H (2009) Handbook of weighted automata. Springer, Berlin (2009). https://doi.org/10.1007/9783642014925
 17.Eisner C, Fisman D, Havlicek J, Lustig Y, McIsaac A, Campenhout DV (2003) Reasoning with temporal logic on truncated paths. In: Proceedings of the computer aided verification, 15th international conference, CAV 2003, Boulder, CO, USA, July 8–12, 2003, pp 27–39Google Scholar
 18.Fainekos GE, Pappas GJ (2009) Robustness of temporal logic specifications for continuoustime signals. Theor Comput Sci 410(42):4262–4291. https://doi.org/10.1016/j.tcs.2009.06.021 MathSciNetCrossRefzbMATHGoogle Scholar
 19.Fainekos GE, Sankaranarayanan S, Ivancic F, Gupta A (2009) Robustness of modelbased simulations. In: Proceedings of RTSS 2009: the 30th IEEE realtime systems symposium, pp 345–354. IEEE Computer Society. https://doi.org/10.1109/RTSS.2009.26
 20.Gerth R, Peled D, Vardi MY, Wolper P (1996) Simple onthefly automatic verification of linear temporal logic. In: Proceedings of the fifteenth IFIP WG6.1 international symposium on protocol specification, testing and verification, IFIP conference proceedings, vol 38, pp 3–18. Chapman & HallGoogle Scholar
 21.Herrmann L, Vogler H (2016) Weighted symbolic automata with data storage. In: Proceedings of DLT 2016: the 20th international conference on developments in language theory, LNCS, vol 9840, pp 203–215. Springer. https://doi.org/10.1007/9783662531327
 22.http://jautomata.sourceforge.net/. Accessed 28 March 2017
 23.http://www.mathworks.com/products/demos/stateflow/fuelsys.html. Accessed 28 March 2017
 24.International S (2016) SENT—single edge nibble transmission for automotive applications, J2716, Standard. http://standards.sae.org/j2716_201001/. Accessed 21 Jan 2017
 25.Jaksic S, Bartocci E, Grosu R, Nickovic D (2016) Quantitative monitoring of STL with edit distance. In: Proceedings of RV 2016: the 16th international conference on runtime verification, LNCS, vol 10012, pp 201–218. Springer. https://doi.org/10.1007/9783319469829_13
 26.Konstantinidis S (2007) Computing the edit distance of a regular language. Inf Comput 205(9):1307–1316. https://doi.org/10.1016/j.ic.2007.06.001 MathSciNetCrossRefzbMATHGoogle Scholar
 27.Krause EF (2012) Taxicab geometry: an adventure in nonEuclidean geometry. Courier Corporation, North ChelmsfordGoogle Scholar
 28.Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Sov Phys Dokl 10:707MathSciNetGoogle Scholar
 29.Maler O, Nickovic D (2013) Monitoring properties of analog and mixedsignal circuits. STTT 15(3):247–268. https://doi.org/10.1007/s1000901202479 CrossRefGoogle Scholar
 30.Mohri M (2003) Editdistance of weighted automata: general definitions and algorithms. Int J Found Comput Sci 14(6):957–982. https://doi.org/10.1142/S0129054103002114 MathSciNetCrossRefzbMATHGoogle Scholar
 31.Nguyen T, Nickovic D (2014) Assertionbased monitoring in practice—checking correctness of an automotive sensor interface. In: Proceedings of FMICS 2014: the 19th international conference on formal methods for industrial critical systems, LNCS, vol 8718, pp 16–32. Springer. https://doi.org/10.1007/9783319107028
 32.Parr T (2013) The definitive ANTLR 4 reference, 2nd edn. Pragmatic Bookshelf, DallasGoogle Scholar
 33.Pnueli A, Zaks A (2008) On the merits of temporal testers. In: 25 years of model checking—history, achievements, perspectives, LNCS, vol 5000, pp 172–195. Springer. https://doi.org/10.1007/9783540698500
 34.Quesel J (2013) Similarity, logic, and games—bridging modeling layers of hybrid systems. Ph.D. thesis, Universität OldenburgGoogle Scholar
 35.Rizk A, Batt G, Fages F, Soliman S (2008) On a continuous degree of satisfaction of temporal logic formulae with applications to systems biology. In: Proceedings of CMSB 2008: the 6th international conference on computational methods in systems biology, LNCS, vol 5307, pp 251–268. Springer. https://doi.org/10.1007/9783540885627
 36.Samanta R, Deshmukh JV, Chaudhuri S (2013) Robustness analysis of string transducers. In: Proceedings of ATVA 2013: the 11th international symposium on automated technology for verification and analysis, LNCS, vol 8172, pp 427–441. Springer. https://doi.org/10.1007/9783319024448_30
 37.Schulz UK, Mihov S (2002) Fast string correction with Levenshtein automata. Int J Doc Anal Recognit 5(1):67–85. https://doi.org/10.1007/s1003200200828 CrossRefzbMATHGoogle Scholar
 38.Selyunin K, Jaksic S, Nguyen T, Reidl C, Hafner U, Bartocci E, Nickovic D, Grosu R (2017) Runtime monitoring with recovery of the SENT communication protocol. In: Proceedings of CAV 2017: the 29th international conference on computer aided verification, LNCS, vol 10426, pp 336–355. Springer. https://doi.org/10.1007/9783319633879
 39.Skorokhod AV (1956) Limit theorems for stochastic processes. Theory Probab Appl 1(3):261–290MathSciNetCrossRefzbMATHGoogle Scholar
 40.Unser M (2000) Sampling 50 years after Shannon. Proc IEEE 88(4):569–587CrossRefGoogle Scholar
 41.Veanes M, Bjørner N, de Moura LM (2010) Symbolic automata constraint solving. In: Proceedings of LPAR17: the 17th international conference on logic for programming, artificial intelligence, and reasoning, LNCS, vol 6397, pp 640–654. Springer. https://doi.org/10.1007/9783642162428
 42.Wagner RA (1974) Ordern correction for regular languages. Commun ACM 17(5):265–268. https://doi.org/10.1145/360980.360995 CrossRefzbMATHGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.