NuRV: A nuXmv Extension for Runtime Verification
 894 Downloads
Abstract
We present NuRV, an extension of the nuXmv model checker for assumptionbased LTL runtime verification with partial observability and resets. The tool provides some new commands for online/offline monitoring and code generations into standalone monitor code. Using the online/offline monitor, LTL properties can be verified incrementally on finite traces from the system under scrutiny. The code generation currently supports C, C++, Common Lisp and Java, and is extensible. Furthermore, from the same internal monitor automaton, the monitor can be generated into SMV modules, whose characteristics can be verified by Model Checking using nuXmv. We show the architecture, functionalities and some use scenarios of NuRV, and we compare the performance of generated monitor code (in Java) with those generated by a similar tool, RVMonitor. We show that, using a benchmark from Dwyer’s LTL patterns, besides the capacity of generating monitors for long LTL formulae, our Javabased monitors are about 200x faster than RVMonitor at generationtime and 2–5x faster at runtime.
1 Introduction
Symbolic Model Checking [16] is a powerful formal verification technique for proving temporal properties of transition systems (a.k.a. models) represented by logical formulae. In the case of Linear Temporal Logic (LTL) [15], the properties can be translated into symbolically represented \(\omega \)automata, which is then conjoined with the model and proved by searchbased techniques that exhaustively analyze the infinite traces of the system [7]. Runtime Verification (RV) [10, 13] on the other hand, is a lightweight verification technique for checking if a given property is satisfied (or violated) on a finite trace of the system under scrutiny (SUS). In general, LTLbased RV problems can be resolved by automatabased [1], rewritingbased [17], or rulebased [11] approaches.
In this paper, we present a new tool called NuRV, an extension of the nuXmv [4] model checker for LTLbased RV. To the best of our knowledge, this is the first time that a model checker is directly modified (or extended) into a runtime monitor (or monitor generator). It is natural to do so, as nuXmv has already provided the needed infrastructure, such as a symbolic translation from LTL to \(\omega \)automata, an algorithm for computing the “fair states” (those leading to infinite paths), together with an interface to BDD library [3] based on CUDD 2.4.1 [18].

conclusive true (\(\top \)), if \(r_\varphi \ne \emptyset \) and \(r_{\lnot \varphi } = \emptyset \). \(\varphi \) is verified for all future inputs;

conclusive false (\(\bot \)), if \(r_\varphi = \emptyset \) and \(r_{\lnot \varphi } \ne \emptyset \). \(\varphi \) is violated for all future inputs;

inconclusive (?), if \(r_\varphi \ne \emptyset \) and \(r_{\lnot \varphi } \ne \emptyset \). In this case, the knowledge of the monitor is limited by the finiteness of u.
Besides the property \(\varphi \), the monitoring algorithm takes in input a model K of the SUS. This is used to declare the variables in which the properties are expressed, but more importantly to define some constraints on their temporal evolution, which represent assumptions on the behavior of the SUS. By considering only (infinite) traces of K, the above algorithm may give more precise outputs (turning ? into \(\top /\bot \)). This is obtained by using \(K \otimes T_\varphi \) (the synchronous product of K and \(T_\varphi \)) and \(K \otimes T_{\lnot \varphi }\) instead of \(T_\varphi \) and \(T_{\lnot \varphi }\), respectively. This coincides with [12], where the resulting monitor is called to be predictive.
The model is used by NuRV in different novel ways. First of all, there is the possibility that \(u \notin L(K)\), because the model may be wrong, or it only captures a partial knowledge of the SUS, or due to unexpected faults. In this case we have \(r_\varphi = r_{\lnot \varphi } = \emptyset \) in above algorithm, and we naturally let the monitor returns a fourth verdict called outofmodel (\(\times \)). This is why we call K an assumption, and the two verdicts \(\top /\bot \) are only conclusive under assumptions, thus renamed to \(\top ^\mathrm {a}/\bot ^\mathrm {a}\). This extended RV approach may be called assumptionbased. In particular, if one only cares whether the SUS always follows its model, we can use a dummy LTL property \(\mathrm {true}\) in above procedure, so that \(K \otimes T_{\lnot \varphi }\) is always empty, and the monitor will output either \(\top ^\mathrm {a}\) or \(\times \), indicating whether \(u \in L(K)\). This application coincides with modelbased RV [19].
Second, the above monitoring algorithm directly supports partially observable traces, i.e. variables appeared in the monitoring property are not (always) known in each state of the input trace. This is because the symbolic forwardimage computations do not require full observability—less restrictive inputs result to coarser belief states. Partial observability becomes more useful under assumptions, as an assumption may express a relation between observable and unobservable variables of the SUS.
Third, NuRV supports resettable monitors, i.e. it can evaluate an LTL property at arbitrary positions of the input trace. This idea was inspired by the observation that, in \(r_\varphi \) and \(r_{\lnot \varphi }\), all variables (some are generated by the LTL translations) related to the present and the past have the same values, while all variables related to the future have opposite values. There is no easy way to distinguish these two groups of variables. However, by taking \(r_\varphi \cup r_{\lnot \varphi }\) we smartly get a new belief state which represents the history of the system after a run given by the input trace seen so far. If we restart the monitor algorithm at state i using this history as the new initial condition of K (also with a reduced version of initial conditions of \(T_\varphi \) and \(T_{\lnot \varphi }\)), the new monitor is essentially evaluating Open image in new window for \(u > i\), with the underlying assumptions taken into account. This is again an orthogonal feature, but having an assumption makes resetting of the monitor more interesting as the assumption evolves to take into consideration the history of the system.
Furthermore, NuRV can synthesize the symbolic monitors into explicitstate monitor automata and then generate them into standalone monitor code in various programming languages (currently we support C, C++, Java, and Common Lisp). Besides, it is possible to dump the monitor automata into SMV modules, which can be further analyzed in nuXmv for their correctness and other properties.
The rest of this paper is organized as follows: In Sect. 2 we describe its architecture and functionalities. Some use case scenarios (as running examples) are given in Sect. 3. Section 4 shows some experimental evaluation results. Finally, we conclude the paper in Sect. 5 with some directions for future work.
2 Architecture and Functionalities
2.1 Architecture of NuRV
 1.
build_monitor: build the symbolic monitor for a given LTL property;
 2.
verify_property: verify a currently loaded trace in the symbolic monitor;
 3.
heartbeat: verify one input state in the symbolic monitor (online monitoring);
 4.
generate_monitor: generate standalone monitors in a target language.
The commands build_monitor and verify_property together implemented the offline monitoring algorithm described in [6]. The command generate_monitor further generates explicitstate monitors in various languages from the symbolic monitor built by the command build_monitor. These commands must work with other nuXmv commands [2] to be useful.
2.2 Structure of ExplicitState Monitors
 L1

The monitor synthesis stops at all conclusive states;
 L2

The monitor synthesis explores all states;
 L3

The monitor synthesis explores all states and reset states.
A sample explicitstate monitor for LTL property \(p\,\mathbf {U}\,q\) generated by NuRV is shown in Fig. 2. The monitor is generated under the assumption that either p or q is true in the input. The monitor starts at location 1, and returns ? if the input is \(p \wedge \lnot q\) until it received \(\lnot p \wedge q\) which has the output \(\top ^\mathrm {a}\) (Y). The L1 monitor has no further transition at locations associated with conclusive verdicts (\(\top ^\mathrm {a}\) or \(\bot ^\mathrm {a}\)), since it can be easily proved that ABRVLTL monitors are monotonic if the assumption is always respected by the input trace. The L2 monitor contains all locations and transitions, thus it may return \(\times \) even after the monitor reached conclusive verdicts. The L3 monitor additionally contains information for the resets: in case the monitor is reset, the current location will first jump to the location indicated in the bracket \(\texttt {[]}\), of current location, then goes to next location according to the input state. However, in the above monitor all reset locations are just the initial location (1), this is mostly because the assumption is an invariant property and the LTL property does not have any past operators.
Standalone monitor code are literally translated from these monitor automata (FSMs). The correctness of monitors in C, for instance, comes indirectly from the correctness of the symbolic algorithm and mode checking on SMVbased monitors.
2.3 API of Generated Code
NuRV currently supports monitor code generation into five languages: C, C++, Java, Common Lisp and SMV. The structure of monitor code is simple yet efficient: it simply mimics the simulations of deterministic FSMs.
 1.
Static partial observability: state denotes a full assignment of the observables, encoded in binary bits: 0 for false (\(\bot \)), 1 for true (\(\top \));
 2.
Dynamic partial observability: state denotes a ternary number, whose each ternary bit represents 3 possible values of an observable variable: 0 for unknown (?), 1 for true (\(\top \)) and 2 for false (\(\bot \)).
Note that the symbolic monitoring algorithm can take in general input states expressed in Boolean formulae (e.g., if the observables are p and q, our monitor may take an input state “\(p\;\mathrm {xor}\;q\)”, either p or q is true but not both), but this is not supported by the generated code.
BDD operations are implemented by the BDD manager. Their performance strongly depends on the variable ordering used in the BDD construction. This can be controlled by setting an input_order_file in nuXmv. The input of generated monitor code requires an encoding of BDDs into long integers according to this file. This encoding is done from the least to the most significant bit. For instance, if the observables are p and q with the same order, an binary encoding for the state \(\{p = \top , q = \bot \}\) would be \((01)_2 = 1\), and a ternary encoding for the same state would be \((21)_3 = 7\). The design purpose is to make sure that the comparison of two encoded states can be as fast as possible. The signatures of monitors in other languages are quite similar, except that the parameter current_loc can be put inside C++/Java classes as an member variable, and each monitor is an instance of the generated monitor class.
3 Use Case Scenario
The command go builds the model from the input file disjoint.smv which defines two Boolean variables p and q, together with the invariant \(p \ne q\).
The generated monitors M0.c and M1.c (together with their C headers) are under the full observability of p and q. The variable ordering is given by the file default.ord, in which each line denotes one variable in the model.
For offline monitoring, there is no need to call generate_monitor in above batch command. Suppose a trace \(u=p\,p\,p\,q\,q\,q\) has been loaded (by read_trace), the command verify_property verifies the trace against the symbolic monitor of \(\varphi _0\), shown in Fig. 5 (here “−n 0” denotes the first monitor, and 1 denotes the first loaded trace).
It is also possible to verify just one input state by heartbeat (online monitoring). It has a similar interface with verify_property, just the trace ID is replaced by a single state expressed by a logical formula (as a string), e.g. "p & !q".
4 Experimental Evaluation
Eight long formulae from Dwyer’s patterns
ID  Pattern  LTL 

13  Trans to p occur at most twice (between q and r)  \(\mathbf {G}\,((q \wedge \mathbf {F}\,r) \rightarrow ((\lnot {}p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee ((p \wedge \lnot {}r)\,\mathbf {U}\, (r \,\vee ((\lnot {}p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee ((p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee (\lnot {}p \,\mathbf {U}\, r))))))))))\) 
14  Trans to p occur at most twice (after q until r)  \(\mathbf {G}\,(q \rightarrow ((\lnot {}p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee ((p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee ((\lnot {}p \wedge \lnot {}r) \mathbf {U}\, (r \vee ((p \wedge \lnot {}r) \,\mathbf {U}\, (r \vee (\lnot {}p \,\mathbf {W}\, r) \vee \mathbf {G}\,p)))))))))\) 
39  p precedes s, t (after q until r)  \(\mathbf {G}\,(q \rightarrow (\lnot {}(s \wedge (\lnot {}r) \wedge \mathbf {X}\,(\lnot {}r \,\mathbf {U}\, (t \wedge \lnot {}r))) \,\mathbf {U}\, (r \vee p) \vee \mathbf {G}\,(\lnot {}(s \wedge \mathbf {X}\,\mathbf {F}\,t))))\) 
43  p responds to s, t (between q and r)  \(\mathbf {G}\, ((q \wedge \mathbf {F}\,r) \rightarrow (s \wedge \mathbf {X}\, (\lnot {}{r} \,\mathbf {U}\, t) \rightarrow \mathbf {X}\, (\lnot {}{r}\,\mathbf {U}\, (t\wedge \mathbf {F}\, p))) \,\mathbf {U}\, r)\) 
44  p responds to s, t (after q until r)  \(\mathbf {G}\, (q \rightarrow (s\wedge \mathbf {X}\, (\lnot {}{r}\,\mathbf {U}\, t) \rightarrow \mathbf {X}\,(\lnot {}{r} \,\mathbf {U}\,(t \wedge \mathbf {F}\, p))) \mathbf {U}\, (r\vee \mathbf {G}\, (s\wedge \mathbf {X}\,(\lnot {}{r} \,\mathbf {U}\,t) \rightarrow \mathbf {X}\, (\lnot {}{r} \,\mathbf {U}\, (t \wedge \mathbf {F}\, p)))))\) 
49  s, t responds to p (after q until r)  \(\mathbf {G}\, (q \rightarrow (p \rightarrow (\lnot {}r \,\mathbf {U}\, (s \wedge \lnot {}r \wedge \mathbf {X}\,(\lnot {}r \,\mathbf {U}\, t)))) \,\mathbf {U}\, (r \vee \mathbf {G}\, (p \rightarrow (s \wedge \mathbf {X}\,\mathbf {F}\, t))))\) 
53  s, t without z responds to p (between q and r)  \(\mathbf {G}\, ((q \wedge \mathbf {F}\,r) \rightarrow (p \rightarrow (\lnot {}r \,\mathbf {U}\, (s \wedge \lnot {}r \wedge \lnot {}z \wedge \mathbf {X}\,((\lnot {}r \wedge \lnot {}z) \,\mathbf {U}\, t)))) \,\mathbf {U}\, r)\) 
54  s, t without z responds to p (after q until r)  \(\mathbf {G}\, (q \rightarrow (p \rightarrow (\lnot {}r \,\mathbf {U}\, (s \wedge \lnot {}r \wedge \lnot {}z \wedge \mathbf {X}\,((\lnot {}r \wedge \lnot {}z) \,\mathbf {U}\, t)))) \mathbf {U}\, (r \vee \mathbf {G}\, (p \rightarrow (s \wedge \lnot {}z \wedge \mathbf {X}\,(\lnot {}z \,\mathbf {U}\, t)))))\) 
We observed that rvm monitors does not report further violations once the first violation happens, and goes into terminal states. To get visible performance metrics we chose to reset all monitors once a violation is reported. Also, to prevent extra performance loss in rvm monitors by creating multiple monitor instances [5], we have used a single trace (stored in a vector) with \(10^7\) random states. For each of the 37 LTL patterns, we recorded the time (in ms) spent by both monitors (running in the same Java process), the result is shown in Fig. 6. Our monitors (in Java) have shown a constantlike time complexity (approx. 250 ms), i.e. the time needed for processing one input trace is almost the same for all patterns. This reflects the spirit of automatabased approaches. Rvm monitors vary from 500 ms to more than 6 s, depending on the number of resets.
5 Conclusions and Future Work
We presented NuRV, a nuXmv extension for Runtime Verification. It supports assumptionbased RV for propositional LTL with both future and past operators, with the supports of partial observability and resets. It has functionalities for offline and online monitoring, and code generation of the monitors in various programming languages. The experimental evaluation on standard LTL patterns shows that NuRV is quite efficient in both generation and running time. In the future, we plan to participate in the RV competition to broaden the tool comparison and to extend the monitor specification language beyond the propositional case.
Footnotes
 1.
 2.
All test data and materials for reproducing these experiments are available at https://es.fbk.eu/people/ctian/papers/rv2019/rv2019data.tar.gz.
 3.
The latest version (55 in total) is available at http://patterns.projects.cs.ksu.edu/documentation/patterns/ltl.shtml. We call them Pattern \(0,1, \ldots , 54\) in the same order.
 4.
The error message is “violation is not a supported state in this logic, ltl.”.
Notes
Acknowledgment
We thank the anonymous reviewers for their helpful comments.
References
 1.Bauer, A., Leucker, M., Schallhart, C.: Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. 20(4), 14–64 (2011). https://doi.org/10.1145/2000799.2000800CrossRefGoogle Scholar
 2.Bozzano, M., et al.: nuXmv 1.1.1 User Manual (2016). https://es.fbk.eu/tools/nuxmv/downloads/nuxmvusermanual.pdf
 3.Bryant, R.E.: Binary decision diagrams. In: Clarke, E.M., Henzinger, T.A., Veith, H., Bloem, R. (eds.) Handbook of Model Checking, pp. 191–217. Springer, Cham (2018). https://doi.org/10.1007/9783319105758_7CrossRefGoogle Scholar
 4.Cavada, R., et al.: The nuXmv symbolic model checker. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 334–342. Springer, Cham (2014). https://doi.org/10.1007/9783319088679_22CrossRefGoogle Scholar
 5.Chen, F., Roşu, G.: Parametric trace slicing and monitoring. In: Kowalewski, S., Philippou, A. (eds.) TACAS 2009. LNCS, vol. 5505, pp. 246–261. Springer, Heidelberg (2009). https://doi.org/10.1007/9783642007682_23CrossRefzbMATHGoogle Scholar
 6.Cimatti, A., Tian, C., Tonetta, S.: Assumptionbased runtime verification with partial observability and resets. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, 165–184. Springer, Cham (2019). https://doi.org/10.1007/9783030320799_10
 7.Clarke, E.M., Grumberg, O., Hamaguchi, K.: Another look at LTL model checking. Formal Methods Syst. Des. 10(1), 47–71 (1997). https://doi.org/10.1023/A:1008615614281CrossRefGoogle Scholar
 8.Dwyer, M.B., Avrunin, G.S., Corbett, J.C.: Patterns in property specifications for finitestate verification. In: Proceedings of the 21st International Conference on Software Engineering, pp. 411–420. ACM Press, New York (1999). https://doi.org/10.1145/302405.302672
 9.Allen Emerson, E., Lei, C.L.: Temporal reasoning under generalized fairness constraints. In: Monien, B., VidalNaquet, G. (eds.) STACS 1986. LNCS, vol. 210, pp. 21–36. Springer, Heidelberg (1986). https://doi.org/10.1007/3540160787_62CrossRefGoogle Scholar
 10.Falcone, Y., Havelund, K., Reger, G.: A tutorial on runtime verification. Eng. Dependable Softw. Syst. 34, 141–175 (2013). https://doi.org/10.3233/9781614992073141CrossRefGoogle Scholar
 11.Havelund, K.: Rulebased runtime verification revisited. Int. J. Softw. Tools Technol. Transfer 17(2), 143–170 (2014). https://doi.org/10.1007/s1000901403092CrossRefGoogle Scholar
 12.Leucker, M.: Sliding between model checking and runtime verification. In: Qadeer, S., Tasiran, S. (eds.) RV 2012. LNCS, vol. 7687, pp. 82–87. Springer, Heidelberg (2013). https://doi.org/10.1007/9783642356322_10CrossRefGoogle Scholar
 13.Leucker, M., Schallhart, C.: A brief account of runtime verification. J. Logic Algebraic Program. 78(5), 293–303 (2009). https://doi.org/10.1016/j.jlap.2008.08.004CrossRefzbMATHGoogle Scholar
 14.Luo, Q., et al.: RVMonitor: efficient parametric runtime verification with simultaneous properties. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 285–300. Springer, Cham (2014). https://doi.org/10.1007/9783319111643_24CrossRefGoogle Scholar
 15.Manna, Z., Pnueli, A.: The Temporal Logic of Reactive and Concurrent Systems: Specification. SpringerVerlag, New York (1992). https://doi.org/10.1007/9781461209317CrossRefzbMATHGoogle Scholar
 16.McMillan, K.L.: Symbolic Model Checking. Springer, Heidelberg (1993). https://doi.org/10.1007/9781461531906CrossRefzbMATHGoogle Scholar
 17.Roşu, G., Havelund, K.: Rewritingbased techniques for runtime verification. Autom. Softw. Eng. 12(2), 151–197 (2005). https://doi.org/10.1007/s105150056205yCrossRefGoogle Scholar
 18.Somenzi, F.: CUDD: CU Decision Diagram Package, Release 2.4.1. University of Colorado at Boulder (2005)Google Scholar
 19.Zhao, Y., Rammig, F.: Modelbased runtime verification framework. Electron. Notes Theoret. Comput. Sci. 253(1), 179–193 (2009). https://doi.org/10.1016/j.entcs.2009.09.035CrossRefGoogle Scholar
Copyright information
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.