ModelPlex: verified runtime validation of verified cyberphysical system models
 1.9k Downloads
 7 Citations
Abstract
Formal verification and validation play a crucial role in making cyberphysical systems (CPS) safe. Formal methods make strong guarantees about the system behavior if accurate models of the system can be obtained, including models of the controller and of the physical dynamics. In CPS, models are essential; but any model we could possibly build necessarily deviates from the real world. If the real system fits to the model, its behavior is guaranteed to satisfy the correctness properties verified with respect to the model. Otherwise, all bets are off. This article introduces ModelPlex, a method ensuring that verification results about models apply to CPS implementations. ModelPlex provides correctness guarantees for CPS executions at runtime: it combines offline verification of CPS models with runtime validation of system executions for compliance with the model. ModelPlex ensures in a provably correct way that the verification results obtained for the model apply to the actual system runs by monitoring the behavior of the world for compliance with the model. If, at some point, the observed behavior no longer complies with the model so that offline verification results no longer apply, ModelPlex initiates provably safe fallback actions, assuming the system dynamics deviation is bounded. This article, furthermore, develops a systematic technique to synthesize provably correct monitors automatically from CPS proofs in differential dynamic logic by a correctbyconstruction approach, leading to verifiably correct runtime model validation. Overall, ModelPlex generates provably correct monitor conditions that, if checked to hold at runtime, are provably guaranteed to imply that the offline safety verification results about the CPS model apply to the present run of the actual CPS implementation.
Keywords
Runtime verification Static verification Cyberphysical systems Hybrid systems Differential dynamic logic1 Introduction
Cyberphysical systems (CPS) involve controllers and the relevant dynamics of the environment. Since safety is crucial for CPS, their models (e. g., hybrid system models [31]) need to be verified formally. Formal verification guarantees that a model is safe with respect to a safety property. The remaining task is to validate whether the model is adequate, so that the verification results for the model transfer to the actual system implementation [18, 42]. This article introduces ModelPlex [24], a method to synthesize correctbyconstruction monitors for CPS by theorem proving automatically: it uses sound axioms and proof rules of differential dynamic logic [33] to formally verify that a model is safe and to synthesize provably correct monitors that validate compliance of system executions with that model. The difficult question answered by ModelPlex is what exact conditions need to be monitored at runtime to guarantee compliance with the models and thus safety.
System execution, however, provides many opportunities for surprising deviations from the model: faults may cause the system to function improperly [43], sensors may deliver uncertain values, actuators may suffer from disturbance, or the formal verification may have assumed simpler idealworld dynamics for tractability reasons or made unrealistically strong assumptions about the behavior of other agents in the environment. Simpler models are often better for timecritical decisions and optimizations, because they make it possible to compute predictions at the rate required for realtime decisions. The same phenomenon of simplicity for predictability is often exploited for the models in formal verification and validation, where formal verification results are often easier to obtain for simpler models. It is more helpful to obtain a verification or prediction result about a simpler model than to fail on a more complex one. The flipside is that the verification results obtained about models of a CPS only apply to the actual CPS at runtime to the extent that the system fits to the model. ModelPlex enables tradeoffs between analytic power and accuracy of models while retaining strong safety guarantees.
Validation, i. e., checking whether a CPS implementation fits to a model, is an interesting but difficult problem. Even more so, since CPS models are more difficult to analyze than ordinary (discrete) programs because of the continuous physical plant, the environment, sensor inaccuracies, and actuator disturbance, making full model validation quite elusive.
In this article, we, thus, settle for the question of runtime model validation, i. e. validating whether the model assumed for verification purposes is adequate for a particular system execution to ensure that the offline safety verification results apply to the current execution.^{1} But we focus on verifiably correct runtime validation to ensure that verified properties of models provably apply to the CPS implementation, which is important for safety and certification [5]. Only with such a way of validating model compliance is there an unbroken chain of evidence of safety claims that apply to the actual system, rather than merely to its model. ModelPlex provides a chain of formal proofs as a strong form of such evidence.
Checking whether a system execution fits to a verified model includes checking that the actions chosen by the (unverified) controller implementation fit to one of the choices and requirements that the verified controller model allows. It also includes checking that the observed states can be explained by the plant model. The crucial questions are: What are the right conditions to monitor? Which monitor conditions guarantee safety without being overly restrictive? How can the correctness of such executable monitor conditions be proved formally? How can a compliance monitor be synthesized that provably represents all important aspects of complying with the verified model correctly? How much safety margin does a system need to ensure that failsafe actions are always initiated early enough for the system to remain safe, even if its behavior ceases to comply with the model?
The last question is related to feedback control and can only be answered when assuming some constraints on the maximum deviation of the real system dynamics from the plant model [36]. Otherwise, i. e., if the real system might be infinitely far off from the model, safety guarantees are impossible. By the sampling theorem in signal processing [40], such constraints further enable compliance monitoring solely on the basis of sample points instead of the unobservable intermediate states about which no sensor data exists.^{2}
Extension In addition to providing proofs for the results, this article extends the short version [24] with support for a correctbyconstruction approach to synthesize ModelPlex monitors by a systematic transformation in the differential dynamic logic axiomatization [33]. We leverage an implementation of this axiomatization in our entirely new theorem prover KeYmaera X [14] by performing the ModelPlex monitor proof construction in place, as opposed to splitting it over the branches of its classical sequent calculus [29]. Sequent calculi are usually preferred for proving properties, because they induce a sequent normal form that simplifies proof construction by narrowing proof search to proof rules for toplevel operators and splitting the proof over independent branches as needed. Proofs cannot close during the ModelPlex monitor construction, however, because the proof represents the conditions on system executions that the verified model imposes. That is why proof branching in our previous ModelPlex implementation [24] led to sizeable monitors with nontrivial redundancy which were simplified with (unverified) external optimization tools and, thus, had to be reverified for correctness.
Our new ModelPlex monitor synthesis presented here exploits the flexibility of differential dynamic logic axioms [33] more liberally to significantly improve locality of the construction, which leads to reductions of the resulting monitors compared to our previous approach [24]. The axiomatic ModelPlex construction also preserves the structure in the model better. The ModelPlex construction now remains entirely under the auspices of the theorem prover without external simplification, thereby eliminating the need to reverify correctness of the resulting monitor. Efficiency during the ModelPlex monitor construction in the prover is retained using contextual rewriting in the uniform substitution calculus for differential dynamic logic [35]. We now also implemented optimizations of the ModelPlex monitor constructions as proof tactics that were previously performed manually. This leads to a fully automatic synthesis procedure for correctbyconstruction ModelPlex monitors that produces proofs of correctness for the monitors it synthesizes.
2 Differential dynamic logic by example
This section recalls differential dynamic logic Open image in new window [29, 31, 33], which we use to syntactically characterize the semantic conditions required for correctness of the ModelPlex approach. Its proof calculus [29, 31, 33, 35] is also exploited to guarantee correctness of the specific ModelPlex monitors produced for concrete CPS models. A tactic for the proof calculus implements the correctbyconstruction ModelPlex monitor synthesis algorithm.
The water level in the tank is controlled by a digital controller that can periodically adjust flow into and from the tank by adjusting two valves. Every time the controller decides on adjusting the flow, it measures the water level through a sensor (i. e., it samples the water level). As a safety condition, we want the water tank to never overflow: any control decision of the controller must be such that the water level stays within 0 and a maximum water level m at all times. We will use this example to introduce Open image in new window and its syntax for modeling hybrid programs step by step. The final example is repeated in Appendix 1 for easy reference.
2.1 Syntax and informal semantics
Differential dynamic logic has a notation for modeling hybrid systems as hybrid programs. Table 1 summarizes the relevant syntax fragment of hybrid programs together with an informal semantics. The formal semantics \(\rho (\alpha )\) of hybrid program \(\alpha \) is a relation on initial and final states of running \(\alpha \) (recalled in Sect. 2.2 below).
Hybrid program representations of hybrid systems
Statement  Effect 

\(\alpha ;\beta \)  Sequential composition, first run hybrid program \(\alpha \), then hybrid program \(\beta \) 
Nondeterministic choice, following either hybrid program \(\alpha \) or \(\beta \)  
Nondeterministic repetition, repeats hybrid program \(\alpha \) \(n\ge 0\) times  
Assign value of term \(\theta \) to variable x (discrete jump)  
Assign arbitrary real number to variable x  
Check that a particular condition F holds, and abort if it does not  
Evolve \(x_i\) along differential equation system Open image in new window  
Restricted to maximum evolution domain Open image in new window 
Syntax summary Sequential composition \(\alpha ;\beta \) says that \(\beta \) starts after \(\alpha \) finishes. The nondeterministic choice \(\alpha ~\cup ~\beta \) follows either \(\alpha \) or \(\beta \). The nondeterministic repetition operator Open image in new window repeats \(\alpha \) zero or more times. Assignment Open image in new window instantaneously assigns the value of term \(\theta \) to the variable x, while Open image in new window assigns an arbitrary value to x. The test ?F checks that a condition F holds, and aborts if it does not. Open image in new window describes a continuous evolution of x within the evolution domain F.
2.2 Formal semantics of Open image in new window
ModelPlex is based on a transition semantics instead of trace semantics [31], since it is easier to handle and fits to checking monitors at sample points.
The semantics of Open image in new window , as defined in [29], is a Kripke semantics in which states of the Kripke model are states of the hybrid system. Let \(\mathbb {R} \) denote the set of real numbers, and \(V\) denote the set of variables. A state is a map Open image in new window ; the set of all states is denoted by Open image in new window . We write Open image in new window if formula \(\phi \) is true at Open image in new window Open image in new window (Definition 2). Likewise, Open image in new window denotes the real value of term \(\theta \) at Open image in new window Open image in new window , while \(\nu (x)\) denotes the real value of variable x at state \(\nu \). The semantics of HP \(\alpha \) is captured by the state transitions that are possible by running \(\alpha \). For continuous evolutions, the transition relation holds for pairs of states that can be interconnected by a continuous flow respecting the differential equation and invariant region. That is, there is a continuous transition along Open image in new window from state Open image in new window to state \(\omega \), if there is a solution of the differential equation Open image in new window that starts in state Open image in new window and ends in \(\omega \) and that always remains within the region \(H\) during its evolution.
Definition 1
 1.
Open image in new window iff Open image in new window and \(\nu (z) = \omega (z)\) for all state variables \(z \ne x\).
 2.
Open image in new window iff \(\nu (z) = \omega (z)\) for all state variables \(z \ne x\).
 3.
 4.
Open image in new window iff for some \({r\ge 0}\), there is a (flow) function Open image in new window with Open image in new window , such that for each time \(\zeta \in [0,r]\) the differential equation holds and the evolution domain is respected Open image in new window , see [29, 34] for details
 5.
 6.
 7.
Open image in new window where \(\alpha ^{i+1} \,\hat{=}\, (\alpha ; \alpha ^i)\) and \(\alpha ^0 \,\hat{=}\, ?true\).
Definition 2
 1.
Open image in new window iff Open image in new window for \({\sim } \in \{=,\le ,<,\ge ,>\}\)
 2.
Open image in new window iff Open image in new window and Open image in new window , accordingly for Open image in new window
 3.
Open image in new window iff Open image in new window for all Open image in new window that agree with Open image in new window except for the value of x
 4.
Open image in new window iff Open image in new window for some Open image in new window that agrees with Open image in new window except for the value of x
 5.
 6.
2.3 Notation and supporting lemmas
Open image in new window denotes the bound variables [35] in \(\alpha \), i. e., those written to in \(\alpha \), Open image in new window are free variables [35] in \(\psi \), \(\varSigma \) is the set of all variables, and \(A \backslash B\) denotes the set of variables being in some set A but not in some other set B. Furthermore, \(\nu _A\) denotes the state \(\nu \) projected to just the variables in A, whereas \(\nu _x^y\) denotes the state \(\nu \) in which x is interpreted as y.
In the proofs throughout this article, we will use the following lemmas specialized from [35, Lemmas 12, 14, and 15]. Hybrid programs only change their bound variables:
Lemma 1
(Bound effect lemma) If \((\nu ,\omega ) \in \rho (\alpha )\), then \(\nu =\omega \) on Open image in new window .
The truth of formulas only depends on their free variables:
Lemma 2
(Coincidence lemma) If \(\nu =\tilde{\nu }\) on Open image in new window then \(\nu \models \phi \) iff \(\tilde{\nu } \models \phi \).
Similar states (that agree on the free variables) have similar transitions:
Lemma 3
(Coincidence lemma) If \(\nu =\tilde{\nu }\) on Open image in new window and \((\nu ,\omega ) \in \rho (\alpha )\), then there is an \(\tilde{\omega }\) such that \((\tilde{\nu },\tilde{\omega }) \in \rho (\alpha )\) and \(\omega =\tilde{\omega }\) on \(V\).
The notation \(\nu _V=\tilde{\nu }_V\) is used interchangeably with \(\nu =\tilde{\nu }\) agree on \(V\).
3 ModelPlex approach for verified runtime validation
However, since we usually made approximations when modeling the controller and the physics, and since failures and other deviations may occur in reality (e. g., a valve could fail), we cannot simply transfer this safety proof to the real system. The safety guarantees that we obtain by proving formula (2) about the model Open image in new window transfer to the real system, if the actual CPS execution fits to Open image in new window .
Example 1
(What to monitor) Let us recall the water tank example. First, since failures may occur we need to monitor actual evolution, such as that the actual water level corresponds to the level expected by the chosen valve positions and the actual time passed between controller executions does not exceed the modeled sampling period. The monitor needs to allow some slack around the expected water level to compensate for the neglected physical phenomena. Sections 3.2 and 3.5 describe how to synthesize such model monitors automatically. Second, the controller implementation differs from the model, e.g., it might follow different filling strategies, so we need to check that the implemented controller only chooses flows f that satisfy \(1 \le f \le \tfrac{mx}{\varepsilon }\). Section 3.4 describes how to synthesize such controller monitors automatically. Finally, we can monitor controller decisions for the expected realworld effect, since the hybrid system model contains a model of the physics of the water tank. Section 3.6 describes how to synthesize such prediction monitors automatically. The controller in the model, which is verified to be safe, gives us a failsafe action that we can execute instead of the unverified controller implementation when one of the monitors is not satisfied.
Since we want to preserve safety properties, a CPS \(\gamma \) fits to a model Open image in new window , if the CPS reaches at most those states that are reachable by the model, i. e., Open image in new window [27], because all states reachable by Open image in new window from states satisfying \(\phi \) are safe by (2). For example, a controller that chooses inflow more cautiously, such as only half the maximum inflow from the model, i. e., \(f\le \tfrac{mx}{2\varepsilon }\), would also be safe. So would be running the controller more frequently than every \(\varepsilon \) time, but not less frequently.
However, we do not know the true CPS \(\gamma \) precisely,^{4} so we cannot use refinementbased techniques (e. g., [27]) to prove that the true CPS \(\gamma \) refines the model Open image in new window . Therefore, we need to find a condition based on Open image in new window that we can check at runtime to see if concrete runs of the true CPS \(\gamma \) behave like the model Open image in new window .
Example 2
(Canonical monitor candidates) A monitor condition that would be easy to check is to monitor the postcondition \(\psi \) (e. g., monitor the safety condition of the water tank \(0 \le x \le m\)). But that monitor is unsafe, because if \(\psi \) is violated at runtime, the system is already unsafe and it is too late to do anything about it (e. g., the water tank did already overflow). Another monitor that would be easy to check is the invariant \(\varphi \) used to prove Formula (2). But that monitor is also unsafe, because once \(\varphi \) is violated at runtime, the controller is no longer guaranteed to be safe, since Formula (3) only proves it to be safe when maintaining invariant \(\varphi \) (e. g., in the water tank example, the invariant \(\varphi \equiv 0 \le x \le m\) is not even stronger than the safety condition). But if we detect when a CPS is about to deviate from \(\alpha \) before leaving \(\varphi \), we can still switch to a failsafe controller to avoid \(\lnot \psi \) from ever happening (see Fig. 3). Yet even so, the invariant \(\varphi \) will not even contain all conditions that need to be monitored, since \(\varphi \) only reflects what will not change when running the particular model \(\alpha \), which says nothing about the behavior of the true CPS \(\gamma \).
The basic idea behind ModelPlex is based on online monitoring: we periodically sample \(\gamma \) to obtain actual system states \(\nu _i\). A state \(\nu _i\) includes values for each of the bound variables (i. e., those that are written) from the model Open image in new window . For example, for our water tank we need to sample flow f (written to in Open image in new window ), water level x (written to in Open image in new window ), and time t (written to in Open image in new window and Open image in new window ). We then check pairs of such states for being included in the reachability relation of the model, which is expressed in Open image in new window semantics as Open image in new window . We will refer to the first state in such a pair by prior state and to the second one by posterior state. This is the right semantic condition to check, but not computationally represented. The important question answered by ModelPlex through automatic synthesis is how that check can be represented in a monitor condition in an easily and efficiently computable form.
Example 3
This formula describes that (i) the flow \(\nu _i(f)\) in the posterior state has to obey certain bounds, depending on the prior water level \(\nu _{i1}(x)\), resulting from the nondeterministic assignment and the test; (ii) the posterior water level \(\nu _i(x)\) is given by the solution of the differential equation \(x + \int f dt = x+ft\), i. e., the posterior water level should be equal to the prior water level \(\nu _{i1}(x)\) plus the amount resulting from flow \(\nu _i(f)\) in time \(\nu _i(t)\); finally, (iii) the evolution domain constraints must be true, meaning the posterior water level must be nonnegative and the time \(\nu _i(t)\) must be between 0 and \(\varepsilon \). Note, that it is tempting to just read off a wrong condition \(\nu _i(t)=0\) from hybrid program Open image in new window . Since t is not constant in the ODE following the assignment ( Open image in new window ), this condition must be phrased \(0 \le \nu _i(t)\). Also note, that it is very easy to get the evolution domain wrong: evolution domain constraints have to hold throughout the ODE, which includes the beginning and the end, so the check must include both \(\nu _{i1}(x)\ge 0\) and \(\nu _i(x)\ge 0\). The sound proof calculus of Open image in new window prevents such mistakes when deriving monitor conditions.
 Model monitor

The model monitor checks the previous state \(\nu _{i1}\) and current state \(\nu _{i}\) for compliance with the model, i. e.. whether the observed transition from \(\nu _{i1}\) to \(\nu _{i}\) is compatible with the model. In each state \(\nu _{i}\) we test the sample point \(\nu _{i1}\) from the previous execution \(\gamma _{i1}\) for deviation from Open image in new window , i. e., test Open image in new window . If violated, other verified properties may no longer hold for the system so a failsafe action is initiated. The system itself, however, still satisfies safety condition \(\psi \) if the prediction monitor was satisfied at \(\nu _{i1}\). Frequent violations indicate an inadequate model that should be revised to better reflect reality.
 Controller monitor

The controller monitor checks the output of a controller implementation against the correct controller model. If the controller implementation performs an action that the controller model allows in the present state, then it has been verified offline to be safe by Formula (2). Otherwise, the action is discarded and replaced by a default action that has been proved safe. In intermediate state \(\tilde{\nu }_{i}\) we test the current controller decisions of the controller implementation \(\gamma _{\text {ctrl}}\) for compliance with the model, i. e., test \((\nu _{i}, \tilde{\nu }_{i}) \in \rho (\alpha _{\text {ctrl}})\). The controller \(\alpha _\text {ctrl}\) will be obtained from the model Open image in new window through proof steps. Controller monitors have some similarities with Simplex [39], which is designed for switching between verified and unverified controllers. The controller monitor, instead, corresponds to the more general idea of testing contracts dynamically at runtime while defaulting to a specified default action choice if the contract fails. If a controller monitor is violated, commands from a failsafe controller replace the current controller’s decisions to ensure that no unsafe commands are ever actuated.
 Prediction monitor

The model monitor detects deviations from the model as soon as possible on the measured data, but that may already have made the system unsafe. The role of the prediction monitor is to check the impact of bounded deviations from the model to predict whether the next state could possibly become unsafe upon deviation from the model so that a corrective action is advised. If the actual execution stays far enough away from unsafe states, the prediction monitor will not intervene because no disturbance within the bound could make it unsafe. In intermediate state \(\tilde{\nu }_{i}\) we test the safety impact of the current controller decision w.r.t. the predictions of a bounded deviation plant model \(\alpha _{\delta \text {plant}}\), which has a tolerance around the model plant \(\alpha _\text {plant}\), i. e., check \(\nu _{i+1} \models \varphi \) for all \(\nu _{i+1}\) such that \((\tilde{\nu }_{i},\nu _{i+1}) \in \rho (\alpha _{\delta \text {plant}})\). Note, that we simultaneously check all \(\nu _{i+1}\) by checking a characterizing condition of \(\alpha _{\delta \text {plant}}\) at \(\tilde{\nu }_{i}\). If violated, the current control choice is not guaranteed to keep the system safe under all disturbances until the next control cycle and, thus, a failsafe controller takes over.
A simulation illustrating the effect of these monitors on the water tank running example will be discussed in Fig. 11, where an unsafe controller and small deviation from the idealistic model would result in violation of the safety property, if not corrected by the monitors synthesized in this article.
The assumption for the prediction monitor is that the real execution is not arbitrarily far off the plant models used for safety verification, because otherwise safety guarantees can be neither made on unobservable intermediate states nor on safety of the future system evolution [36]. We propose separation of disturbance causes in the models: ideal plant models \(\alpha _{\text {plant}}\) for correctness verification purposes, implementation deviation plant models \(\alpha _{\delta \text {plant}}\) for monitoring purposes. We support any deviation model (e. g., piecewise constant disturbance, differential inclusion models of disturbance), as long as the deviation is bounded and differential invariants can be found. We further assume that monitor evaluations are at most some \(\varepsilon \) time units apart (e. g., along with a recurring controller execution). Note that disturbance in \(\alpha _{\delta \text {plant}}\) is more manageable compared to a model of the form Open image in new window , because we can focus on single runs \(\alpha \) instead of repetitions for guaranteed monitoring purposes.
3.1 Characterizing semantic relations between states in logic
All ModelPlex monitors relate states, albeit for different purposes to safeguard different parts of the CPS execution (Fig. 4). States are semantic objects and as such cannot be related, manipulated, or even just represented precisely in a program. This section develops a systematic logical characterization as syntactic expressions for such state relations, which will ultimately lead to computable programs for the corresponding monitor conditions. We systematically derive a check that inspects states of the actual CPS to detect deviation from the model \(\alpha \). We first establish a notion of state recall and show that compliance of an execution from state \(\nu \) to \(\omega \) with \(\alpha \) can be characterized syntactically in Open image in new window .
The ModelPlex monitoring principle illustrated in Fig. 4 is intuitive, but its sequence of states \(\nu _i\) is inherently semantic and, thus, inaccessible in syntactic programs. Our first step is to introduce a vector of logical variables x and \(x^+\) for the symbolic prior and posterior state variables. The basic idea is that ModelPlex monitors identify conditions on the relationships between the values of prior and posterior state expressed as a logical formula involving the variables x and \(x^+\). Concrete states \(\nu _{i1}\) and \(\nu _i\) can then be fed into the monitor formula as the real values for the variables x and \(x^+\) to check whether the monitor is satisfied along the actual system execution.
Definition 3 and Lemma 4 below describe central ingredients for online monitoring in this article and are true for models \(\beta \) of arbitrary form (not just for models Open image in new window with a loop).
Definition 3
(State recall) Let \(V\) denote the set of variables whose state we want to recall. We use the formula Open image in new window to express a characterization of the values of variables x in a state posterior to a run of \(\beta \), where we always assume the fresh variables \(x^+\) to occur solely in \(\varUpsilon ^+\). The variables in \(x^+\) can be used to recall this state. We define the satisfaction relation \((\nu , \omega ) \models \phi \) of Open image in new window formula \(\phi \) for a pair of states \((\nu ,\omega )\) as \(\phi \) evaluated in the state resulting from \(\nu \) by interpreting \(x^+\) as \(\omega (x)\) for all \(x \in V\), i. e., \((\nu , \omega ) \models \phi \) iff \(\nu _{x^+}^{\omega (x)} \models \phi \).
This enables a key ingredient for ModelPlex: establishing a direct correspondence of a semantic reachability of states with a syntactic logical formula internalizing that semantic relationship by exploiting the Open image in new window modality of Open image in new window .
Lemma 4
(Logical state relation) Let Open image in new window . Two states \(\nu ,\omega \) that agree on \(\varSigma \setminus V\), i. e., \(\nu _{\varSigma \setminus V} = \omega _{\varSigma \setminus V}\), i. e., \(\nu (z)=\omega (z)\) for all \(z\in \varSigma \setminus V\), satisfy \((\nu ,\omega ) \in \rho (\beta )\) iff Open image in new window .
Proof
 “ Open image in new window ”

Let \((\nu ,\omega ) \in \rho (\beta )\). Since \(\nu \) and \(\nu _{x^+}^{\omega (x)}\) agree except on \(x^+\), which are not free variables of \(\beta \), \((\nu ,\omega ) \in \rho (\beta )\) also implies by coincidence Lemma 3 that there is a \(\tilde{\omega }\) such that \((\nu _{x^+}^{\omega (x)},\tilde{\omega }) \in \rho (\beta )\) and \(\omega =\tilde{\omega }\) except on \(x^+\). Now \((\nu _{x^+}^{\omega (x)},\tilde{\omega }) \in \rho (\beta )\) implies that \(\nu _{x^+}^{\omega (x)}=\tilde{\omega }\) agree except on Open image in new window by bound effect Lemma 1. Hence, \(\nu _{x^+}^{\omega (x)}=\tilde{\omega }\) agree on \(x^+\) since Open image in new window and, thus, also \(\omega _{x^+}^{\omega (x)}=\tilde{\omega }\) on \(x^+\). Since \(\omega =\tilde{\omega }\) agree except on \(x^+\) and \(\omega _{x^+}^{\omega (x)}=\tilde{\omega }\) agree on \(x^+\), also \(\omega _{x^+}^{\omega (x)}=\tilde{\omega }\) agree everywhere, which implies, \((\nu _{x^+}^{\omega (x)},\omega _{x^+}^{\omega (x)}) \in \rho (\beta )\), because \((\nu _{x^+}^{\omega (x)},\tilde{\omega }) \in \rho (\beta )\). As \(\omega _{x^+}^{\omega (x)} \models x=x^+\) for all x, so \(\omega _{x^+}^{\omega (x)} \models \varUpsilon ^+\). Consequently, Open image in new window , which is Open image in new window .
 “ Open image in new window ”

Let Open image in new window , that is, Open image in new window . So there is a \(\tilde{\omega }\) such that \((\nu _{x^+}^{\omega (x)},\tilde{\omega }) \in \rho (\beta )\) and \(\tilde{\omega } \models \varUpsilon ^+\). Now \(\tilde{\omega } \models \varUpsilon ^+\) implies that \(\tilde{\omega }(x) = \tilde{\omega }(x^+)\). By the bound effect Lemma 1, \(\nu _{x^+}^{\omega (x)}=\tilde{\omega }\) agree except on Open image in new window . Thus, \(\tilde{\omega }(x^+)=\nu _{x^+}^{\omega (x)}(x^+)=\omega (x)\) for all Open image in new window as Open image in new window . Combining both yields that \(\tilde{\omega }=\omega \) agree on all Open image in new window . Since \((\nu _{x^+}^{\omega (x)},\tilde{\omega }) \in \rho (\beta )\) and \(\nu _{x^+}^{\omega (x)}=\nu \) agree except on Open image in new window , coincidence Lemma 3 implies there is a \(\mu \) such that \((\nu ,\mu ) \in \rho (\beta )\) and \(\mu =\tilde{\omega }\) agree except on \(x^+\). So, \(\mu =\tilde{\omega }=\omega \) agree on Open image in new window . And \(\mu =\nu \) agree except on Open image in new window by bound effect Lemma 1. From the assumption that \(\nu =\omega \) agree except on Open image in new window , it follows that \(\mu =\omega \) also on Open image in new window , so \(\mu =\omega \). Hence, \((\nu ,\mu ) \in \rho (\beta )\) implies \((\nu ,\omega ) \in \rho (\beta )\). \(\square \)
In principle, formula (4) would already be a perfect monitor for the question whether the state change to \(\varUpsilon ^+\) can be explained by model Open image in new window . But formula (4) is hard if not impossible to evaluate at runtime efficiently, because it refers to a hybrid system Open image in new window , which includes loops, nondeterminism, and differential equations and is, thus, difficult to execute without nontrivial backtracking and differential equation solving. Yet, any formula that is equivalent to or implies (4) but is easier to evaluate in a state is a correct monitor as well.
To simplify formula (4), we use theorem proving to find a quantifierfree firstorder real arithmetic form so that it can be evaluated efficiently at runtime. The resulting firstorder real arithmetic formula can be easily implemented in a runtime monitor that is evaluated by plugging the concrete values in for x and \(x^+\). A monitor is executable code that only returns true if the transition from the prior system state to the posterior state is compliant with the model. Thus, deviations from the model can be detected at runtime, so that appropriate fallback and mitigation strategies can be initiated.
3.2 Model monitor synthesis
Figure 5 gives an overview of the offline synthesis process for model monitors. Semantically, a monitor is a check that a pair of states \((\nu ,\omega )\) is contained in the transition relation Open image in new window of the monitored hybrid systems model Open image in new window (Fig. 6). This corresponds to our intuitive understanding of a monitor: through sensors, we observe states of a system, and want to know if those observations fit to the model Open image in new window of the system. By Lemma 4, the syntactic counterpart in the logic Open image in new window of this semantic condition Open image in new window is the logical formula Open image in new window from (4). The Open image in new window formula (4) syntactically characterizes the semantic statement that the hybrid system model Open image in new window can reach a posterior state^{6} characterized by \(x^+\) from the prior state characterized by x. The Open image in new window formula (4) is a perfect logical monitor but difficult to execute quickly, so we are looking for easier logical formulas \(F(x,x^+)\) that are equivalent to or imply formula (4). ModelPlex uses theorem proving to systematically synthesize a provably correct real arithmetic formula \(F(x,x^+)\) in a correctbyconstruction approach.^{7}
The intuition is that formula (4) holds because all conditions hold that are identified as implying formula (4) in its proof. Some of these conditions hold always (subgoals that can be proved to be valid always) while others will be checked at runtime whether they hold (subgoals that do not always hold but only during executions that fit to the particular hybrid system Open image in new window ). If the ModelPlex monitor is satisfied at runtime, then the proof implying formula (4) holds in the current CPS execution.
Intuitively, a model monitor \(\chi _{\text {m}}\) is correct when the monitor entails safety if it is satisfied on consecutive observations, which is formalized in Theorem 1 below. Note, that Theorem 1 for models \(\beta \) without loops follows immediately from Lemma 4 and the safety proof. Thanks to Lemma 4, correctness of model monitors is also easy to prove:
Theorem 1
Proof
Show Open image in new window by induction over n, such that Open image in new window and \(\nu _0 \models \phi \) imply \(\nu _n \models \psi \). If \(n = 0\) then Open image in new window trivially by Definition 1. For \(n+1 > 0\) assume Open image in new window and Open image in new window . By Lemma 4, Open image in new window implies that Open image in new window . Now Open image in new window and Open image in new window imply Open image in new window . Hence we conclude \(\nu _{n+1} \models \psi \) from \(\nu _0 \models \phi \) and Open image in new window . \(\square \)
By Theorem 1, any formula implying \(\chi _{\text {m}}\) is also a correct model monitor, such as Open image in new window , which more conservatively limits acceptable executions of the real \(\gamma \) to those that correspond to just one iteration of Open image in new window as opposed to arbitrarily many.
Example 4
This formula corresponds to the expected result from Example 3, since x corresponds to \(\nu _{i1}(x)\) and \(x^+\) corresponds to \(\nu (x)\), and so forth.
The formula in Example 4 contains checks for water level x, flow f, and time t, because these are the variables changed by the model. If we want to additionally monitor that the model does not change anything except these variables, we can use Corollary 1 to include frame constraints for specific variables into a monitor (e. g., the value of variable \(\varepsilon \) is not changed by the water tank model, and therefore not supposed to change in reality).
Corollary 1
Theorem 1 continues to apply when replacing \(V\) by any superset Open image in new window .
Proof
Any variable Open image in new window can be added to Theorem 1 by considering Open image in new window instead of Open image in new window , which has the same behavior but one more bound variable. \(\square \)
So far, Theorem 1 assumed that everything stays constant, except for the water level x, the flow f, and the time t. This assumption is stronger than absolutely necessary, and, strictly speaking, prevents us from using the monitor in an environment where values that are irrelevant to the model and its safety condition change (e. g., the water temperature). Corollary 2 ensures monitor correctness in environments where irrelevant variables change arbitrarily. Theorem 2 and 3 can be extended with corollaries similar to Corollaries 1 and 2.
Corollary 2
When replacing \(V\) by any superset Open image in new window , Theorem 1 continues to hold without the assumption that the \(\nu _k\) agree on \(\varSigma \backslash V\).
Proof
Assume the conditions of Theorem 1 with any sequence of states \(\nu _0, \nu _1, \nu _2, \nu _3 \ldots \in \mathbb {R}^n\), with \(\nu _0 \models \phi \). Consider a modified sequence of states \(\bar{\nu }_0, \bar{\nu }_1, \bar{\nu }_2, \bar{\nu }_3 \ldots \) such that for all k: \(\nu _k\) agrees with \(\bar{\nu }_k\) on \(V\) and \(\bar{\nu }_k\) agrees with \(\nu _0\) on \(\varSigma \backslash V\), which, thus, satisfies the assumptions of Theorem 1. Hence, \((\nu _i, \nu _{i+1}) \models \chi _{\text {m}}\) implies \((\bar{\nu }_i, \bar{\nu }_{i+1}) \models \chi _{\text {m}}\) by Lemma 2 using Open image in new window . Thus, \((\nu _i, \nu _{i+1}) \models \chi _{\text {m}}\) for all \(i<n\) implies \((\bar{\nu }_i, \bar{\nu }_{i+1}) \models \chi _{\text {m}}\) for all \(i<n\), so Theorem 1 implies \(\bar{\nu }_n \models \psi \). Since Open image in new window , Lemma 2 implies that \(\nu _n \models \psi \).
\(\square \)
Theorem 1 ensures that, when the monitor is satisfied, the monitored states are safe, i. e., \(\psi \) holds. We can get an even stronger result by Corollary 3, which says that a model monitor also ensures that inductive invariants \(\varphi \) of the model are preserved.
Corollary 3
Under the conditions of Theorem 1 it is also true that \(\nu _n \models \varphi \) for an invariant \(\varphi \) s.t. Open image in new window , Open image in new window , and Open image in new window .
Proof
From Open image in new window it follows that there exists a \(\varphi \) s.t. Open image in new window , Open image in new window , and Open image in new window . Hence Open image in new window and Theorem 1 applies with \(\varphi \) in place of \(\psi \). \(\square \)
Now that we know the correctness of the logical monitor representation, let us turn to synthesizing its arithmetical form.
3.3 Monitor synthesis algorithm
 1.
A Open image in new window formula (2) about a model Open image in new window of the form Open image in new window is turned into a specification conjecture (5) of the form Open image in new window .
 2.
Theorem proving according to the tactic choice is applied on the specification conjecture (5) until no further Open image in new window proof rules are applicable and only firstorder real arithmetic formulas remain open.
 3.
The monitor specification \(F(x,x^+)\) is the conjunction of the unprovable firstorder real arithmetic formulas from open subgoals. The intuition behind this is that goals, which remain open in the offline proof, are proved online through monitoring. Although this do not yield a proof for all imaginable runs, that way we obtain a proof for the current run of the real CPS.
Generate the specification conjecture We map Open image in new window formula (2) syntactically to a specification conjecture of the form (5), i. e., Open image in new window . By design, this conjecture will not be provable. But the unprovable branches of a proof attempt will reveal information that, had it been in the premises, would make (5) provable. Through \(\varUpsilon ^+\), those unprovable conditions collect the relations of the posterior state of model Open image in new window characterized by \(x^+\) to the prior state x, i. e., the conditions are a representation of (4) in quantifierfree firstorder real arithmetic.
Example 5
Use theorem proving to analyze the specification conjecture We use the axioms and proof rules of Open image in new window [29, 33, 35] to analyze the specification conjecture Open image in new window . These proof rules syntactically decompose a hybrid model into easiertohandle parts, which leads to sequents with firstorder real arithmetic formulas towards the leaves of a proof. Using real arithmetic quantifier elimination we close sequents with logical tautologies, which do not need to be checked at runtime since they always evaluate to Open image in new window for any input. The conjunction of the remaining open sequents is the monitor specification; it implies formula (4).
In the remainder of this article, we follow a synthesis style based on the axiomatization of Open image in new window . Axiomatizationstyle synthesis differs from the sequentstyle synthesis of the short version [24] in the mechanics of the simplification step of Algorithm 1. The axiomatization of Open image in new window allows working in place with fast contextual congruences. This leads to simpler monitors and simpler proofs since the synthesis proof does not branch and thus keeps working on the same goal (\(\tilde{g}=g\), so \(\left G\right =1\)), as opposed to the sequentstyle synthesis, which may create new goals (\(\left G\right \ge 1\)). For comparison, the corresponding sequentstyle synthesis techniques of the short version [24] of this article is elaborated in Appendix 3. The complete proof calculus is reported in the literature [29, 33, 35]. We explain the requisite proof rules onthefly while discussing their use in the running example.
Example 6
Let us look more closely into the first step of Example 6, i. e., Open image in new window . Usually, proving properties of the form Open image in new window about loops requires an inductive variant in order to prove arbitrarily many repetitions of the loop body. With monitoring in mind, though, we can unwind the loop and execute the resulting conditions repeatedly instead, as elaborated in Lemma 5.
Lemma 5
(Loop elimination) Let \(\alpha \) be a hybrid program and Open image in new window be the program that repeats \(\alpha \) arbitrarily many times. Then Open image in new window is valid.
Proof
Lemma 5 allows us to check compliance with the model Open image in new window by checking compliance on each execution of \(\alpha \) (i. e., online monitoring [18]), which is easier than for Open image in new window because the loop was eliminated.
We will continue Example 6 in subsequent examples. The complete sequence of proof rules applied to the specification conjecture of the water tank is described in Appendix 2. Most steps are simple when analyzing specification conjectures: sequential composition ( Open image in new window ), nondeterministic choice ( Open image in new window ), deterministic assignment ( Open image in new window ) replace current facts with simpler ones (or branch the proof as propositional rules do). Challenges arise from handling nondeterministic assignment and differential equations in hybrid programs.
Let us first consider nondeterministic assignment Open image in new window . The proof rule for nondeterministic assignment ( Open image in new window ) results in a new existentially quantified variable. Using axiomaticstyle synthesis, we can postpone instantiating the quantifier until enough information about what exact instance to use is discovered, see Example 7. The sequentstyle synthesis, in contrast, must instantiate the quantifier right away, in order to continue synthesis on the existentially quantified formula. Appendix 3 discusses ways on how to instantiate such quantifiers ahead of time.

we can postpone handling the quantifier until additional facts about a concrete instance are discovered, which is the preferred tactic in axiomaticstyle synthesis;

we can instantiate the existential quantifier, if we know that the duration will be \(t^+\);

we can introduce a new logical variable, which is the generic case in sequentstyle synthesis that always yields correct results, but may discover monitor specifications that are harder to evaluate;

we can use quantifier elimination (QE) to obtain an equivalent quantifierfree result (a possible optimization could inspect the size of the resulting formula).
Example 7
The analysis of the specification conjecture finishes with collecting the open sequents from the proof to create the monitor specification Open image in new window . The axiomaticstyle synthesis operates fully inplace, so there is only one open sequent to collect. In contrast, the sequentstyle synthesis usually splits into multiple branches. Moreover, the collected open sequents may include new logical variables and new (Skolem) function symbols that were introduced for nondeterministic assignments and differential equations when handling existential or universal quantifiers. These can be handled in a final step by reintroducing and instantiating quantifiers, see Appendix 3.
Remark 1
(Monitor evaluation at runtime) The complexity of evaluating an arithmetic formula over the reals for concrete numbers (such as a monitor for the concrete numbers corresponding to the current state) is linear in the formula size, as opposed to deciding the validity of such formulas, which is doubly exponential [10]. Evaluating the same formula on floating point numbers is inexpensive, but may yield incorrect results due to rounding errors; on exact rationals the bitcomplexity can be nonnegligible. We use interval arithmetic to obtain correct results while retaining the efficiency of floatingpoint computations. Interval arithmetic overapproximates a real value using an interval of two floatingpoint values that contains the real, which means the monitors become more conservative (e. g., to evaluate \(x \le m\) in interval arithmetic, consider \(x \in [x_l, x_u]\) and \(m \in [m_l,m_u]\), so \([x_l,x_u] \le [m_l, m_u]\) if \(x_u \le m_l\), which in turn implies \(x \le m\)). This leads to an intervalarithmetic formula \(\hat{F}(x,x^+)\) that implies \(F(x,x^+)\) and, thus, also implies the required monitor condition Formula (5).
3.4 Controller monitor synthesis
For a hybrid system Open image in new window of the canonical form Open image in new window , a controller monitor \(\chi _{\text {c}}\), cf. Fig. 8, checks that two consecutive states \(\nu \) and \(\tilde{\nu }\) are reachable with one controller execution \(\alpha _{\text {ctrl}}\), i. e., \((\nu ,\tilde{\nu }) \in \rho (\alpha _{\text {ctrl}})\) with Open image in new window . This controller monitor is to be executed before a control choice by the controller is sent to the actuators. The program \(\alpha _\text {ctrl}\) is derived from \(\alpha \) by skipping differential equations according to Lemma 6 below. Recall that a differential equation Open image in new window can be followed for a nondeterministic amount of time, including 0, which lets us skip it as long as its evolution domain constraint H is satisfied in the beginning, as captured by Open image in new window . That way, a controller monitor ensures that the states reachable by a controller enable subsequent runs of the plant, see Theorem 2. We systematically derive a controller monitor from the specification formula Open image in new window , see Fig. 7. A controller monitor can be used to initiate controller switching similar to Simplex [39], yet in provably correctbyconstruction ways.
Lemma 6
(Differential skip) Let Open image in new window denote a set of differential equations with evolution domain H. Then Open image in new window is valid.
Proof
We prove in Open image in new window using [\(^{\prime }\)] skip derived from DW [35].
\(\square \)
Theorem 2
Proof
By Lemma 4, Open image in new window implies \((\nu ,\tilde{\nu }) \in \rho (\alpha _\text {ctrl})\). The assumption Open image in new window furthermore implies \(\tilde{\nu } \models H\), Open image in new window by Lemma 6, hence Open image in new window and \((\nu ,\tilde{\nu }) \in \rho (\alpha )\) by Lemma 4. Since \(\nu \models \varphi \) by assumption, we get \(\tilde{\nu } \models \varphi \) from Open image in new window . Now \(\tilde{\nu } \models \varphi \wedge H\), so there exists \(\omega \) s.t. \((\tilde{\nu },\omega ) \in \rho (\alpha _\text {plant})\). \(\square \)
The corollaries to Theorem 1 carry over to Theorem 2 accordingly.
3.5 Monitoring in the presence of expected uncertainty and disturbance
Up to now we considered exact idealworld models. But realworld clocks drift, sensors measure with some uncertainty, and actuators are subject to disturbance. This makes the exact models safe but too conservative, which means that monitors for exact models are likely to fall back to a failsafe controller rather often. In this section we discuss how we find ModelPlex specifications in the sequentstyle synthesis techniques so that the safety property (2) and the monitor specification become more robust to expected uncertainty and disturbance. That way, only unexpected deviations beyond those captured in the normal operational uncertainty and disturbance of model Open image in new window cause the monitor to initiate failsafe actions.
In Open image in new window , we can, for example, use nondeterministic assignment from an interval to model sensor uncertainty and piecewise constant actuator disturbance (e. g., as in [26]), or differential inequalities for actuator disturbance (e. g., as in [38]). Such models include nondeterminism about sensed values in the controller model and often need more complex physics models than differential equations with polynomial solutions.
Example 8
We analyze Example 8 in the same way as the previous examples, with the crucial exception of the differential inequalities. We cannot use the proof rule Open image in new window to analyze this model, because differential inequalities do not have polynomial solutions. Instead, we use Open image in new window (cf. Lemma 7) and the DE proof rule of Open image in new window [31] to turn differential inequalities into a differentialalgebraic constraint form that lets us proceed with the proof. Rule DE turns a differential inequality Open image in new window into a quantified differential equation Open image in new window with an equivalent differentialalgebraic constraint. Rule Open image in new window turns a fluctuating disturbance Open image in new window into a mean disturbance Open image in new window , see Lemma 7.
Lemma 7
(Mean disturbance) Reachability with mean disturbance \(\bar{d}\) throughout approximates fluctuating disturbance Open image in new window .
Proof
Example 9
(Analyzing differential inequalities) Loops, assignments and tests are analyzed as in the previous examples. We continue with differential inequalities as follows. First, we eliminate the differential inequalities by rephrasing them as differentialalgebraic constraints in step (DE). Then, we refine by extracting the existential quantifiers for flow disturbance \(\tilde{d}\) and time drift \(\tilde{t}\), so that they become mean disturbance and mean time drift in step ( Open image in new window ). Note, that the existential quantifier moved from inside the modality Open image in new window to the outside Open image in new window , which captures that the states reachable with fluctuating disturbance could also have been reached by following a mean disturbance throughout. The resulting differential equation has polynomial solutions and, thus, we can use Open image in new window and proceed with the proof as before.
As expected, we get a more permissive monitor specification. Such a monitor specification says that there exists a mean disturbance \(\bar{d}\) and a mean clock drift \(\bar{c}\) within the allowed disturbance bounds, such that the measured flow \(f^+\), the clock \(t^+\), and the measured level \(x^+\) can be explained with the model. These existential quantifiers will be turned into equivalent quantifierfree form in subsequent steps by Open image in new window .
3.6 Monitoring compliance guarantees for unobservable intermediate states
With controller monitors, noncompliance of a controller implementation w.r.t. the modeled controller can be detected right away. With model monitors, noncompliance of the actual system dynamics w.r.t. the modeled dynamics can be detected when they first occur. We switch to a failsafe action, which is verified using standard techniques, in both noncompliance cases. The crucial question is: can such a method always guarantee safety? The answer is linked to the image computation problem in model checking (i. e., approximation of states reachable from a current state), which is known to be not semidecidable by numerical evaluation at points; approximation with uniform error is only possible if a bound is known for the continuous derivatives [36]. This implies that we need additional assumptions about the deviation between the actual and the modeled continuous dynamics to guarantee compliance for unobservable intermediate states. Unbounded deviation from the model between sample points just is unsafe, no matter how hard a controller tries. Hence, worstcase bounds capture how well reality is reflected in the model.
We derive a prediction monitor, cf. Figs. 9 and 10, to check whether a current control decision will be able to keep the system safe for time \(\varepsilon \) even if the actual continuous dynamics deviate from the model. A prediction monitor checks the current state, because all previous states are ensured by a model monitor and subsequent states are then safe by (2).
In order to derive a prediction monitor, we use Lemma 8 to introduce a plant with disturbance as additional predicate into our logical representation.
Lemma 8
(Introduce predicate) Formula Open image in new window is valid.
Proof
Follows from Open image in new window using the diamond variant of Open image in new window . \(\square \)
Definition 4
(\(\varepsilon \)bounded plant with disturbance \(\delta \)) Let \(\alpha _{\text {plant}}\) be a model of the form Open image in new window . An \(\varepsilon \)bounded plant with disturbance \(\delta \), written \(\alpha _{\delta \text {plant}}\), is a plant model of the form Open image in new window for some f, g with fresh variable \(\varepsilon > 0\) and with a clock Open image in new window . We say that disturbance \(\delta \) is constant if Open image in new window ; it is additive if \(f(\theta ,\delta ) = \theta  \delta \) and \(g(\theta , \delta ) = \theta + \delta \).
Theorem 3
Proof
Assume \((\nu , \tilde{\nu }) \models \chi _{\text {p}}\), i. e., \(\nu _{x^+}^{\tilde{\nu }(x)} \models \chi _{\text {p}}\). By Theorem 2, Open image in new window implies \(\tilde{\nu } \models \varphi \), since \(\nu \models \varphi \). Furthermore, then there exists \(\mu \) such that Open image in new window with \((\nu _{x^+}^{\tilde{\nu }(x)}, \mu ) \in \rho (\alpha _{\text {ctrl}})\) and the two states \(\nu \) and \(\mu \) agree on all variables except the ones modified by \(\alpha _{\text {ctrl}}\), i. e., Open image in new window . Now \(\mu \models \varUpsilon ^+\) implies \(\mu (x) = \mu (x^+) = \nu _{x^+}^{\tilde{\nu }(x)}(x^+) = \tilde{\nu }(x)\). (in other words, \(\mu _{V} = \tilde{\nu }_{V}\)). Also Open image in new window . Thus, by Lemma 2, Open image in new window since Open image in new window and hence we have \(\omega \models \varphi \) for all \((\tilde{\nu }, \omega ) \in \rho (\alpha _{\delta \text {plant}})\). \(\square \)
Observe that this is also true for all intermediate times \(\zeta \in \left[ 0,\omega (t)\right] \) by the transition semantics of differential equations, where \(\omega (t) \le \varepsilon \) because \(\alpha _{\delta \text {plant}}\) is bounded by \(\varepsilon \).
Remark 2
By adding a controller execution Open image in new window prior to the disturbed plant model, we synthesize prediction monitors that take the actual controller decisions into account. For safety purposes, we could just as well use a monitor definition without controller Open image in new window . But that would result in a rather conservative monitor, which has to keep the CPS safe without knowledge of the actual controller decision.
3.7 Decidability and computability
One useful characteristic of ModelPlex beyond soundness is that monitor synthesis is computable, which yields a synthesis algorithm, and that the correctness of those synthesized monitors w.r.t. their specification is decidable, cf. Theorems 4 and 5.
From Lemma 5 it follows that online monitoring [18] (i. e., monitoring the last two consecutive states) is permissible. So, ModelPlex turns questions Open image in new window into Open image in new window . For decidability, we first consider canonical hybrid programs \(\alpha \) of the form \(\alpha \equiv \alpha _\text {ctrl} ; \alpha _\text {plant}\) where \(\alpha _\text {ctrl}\) and \(\alpha _\text {plant}\) are free of further nested loops. To handle differential inequalities in Open image in new window formulas of the form Open image in new window , the subsequent proofs additionally use the rules for handling differentialalgebraic equations [31].
Theorem 4
(Monitor correctness is decidable) We assume canonical models of the form \(\alpha \equiv \alpha _{\text {ctrl}} ; \alpha _{\text {plant}}\) without nested loops, with solvable differential equations in \(\alpha _\text {plant}\) and disturbed plants \(\alpha _{\delta \text {plant}}\) with constant additive disturbance \(\delta \) (see Definition 4) and \(F(x,x^+),\varphi ,H\) to be firstorder formulas. Then, monitor correctness is decidable, i. e., the formulas Open image in new window , Open image in new window , and Open image in new window are decidable.
Proof

Model monitor Open image in new window : Follows from relative decidability of Open image in new window [33, Theorem 11], because Open image in new window contains no free variables.

Controller monitor Open image in new window : Follows from relative decidability of Open image in new window [33, Theorem 11], because Open image in new window contains no free variables.
 Prediction monitor Open image in new window : First assume that Open image in new window can be represented in a firstorder formula B such that Open image in new window . Then, by decidability splits into two cases:It remains to show that Open image in new window can be represented in a firstorder formula B such that Open image in new window . We know from Lemma 7 that any fluctuating disturbance can be approximated by its mean disturbance throughout. So for all fluctuating disturbances in \(\left[ \delta ,\delta \right] \) we have a corresponding constant additive mean disturbance from \(\left[ \delta ,\delta \right] \), which yields solvable differential equations. Hence, there exists a firstorder formula B such that Open image in new window is valid. For the constant additive case, there even is a firstorder formula B that is equivalent to Open image in new window , because every constant additive disturbance can be replaced equivalently by a mean disturbance using the meanvalue theorem for the disturbance as a (continuous!) function of time [30]. Consequently, the above cut to add B is possible if and only if the monitor \(\chi _{\text {p}}\) is correct, leading to a decision procedure. \(\square \)

Case Open image in new window : follows from case Open image in new window (controller monitor) above.
 Case Open image in new window : Since the disturbance \(\delta \) in \(\alpha _{\delta \text {plant}}\) is constant additive and the differential equations in \(\alpha _\text {plant}\) are solvable, we have the disturbance functions \(f(\theta , \delta )\) and \(g(\theta ,\delta )\) applied to the solution as an oracle^{8} for differential invariants (i. e., the differential invariant is a pipe around the solution without disturbance). Specifically, to show Open image in new window by Definition 4 we have to show Open image in new window . We proceed with only Open image in new window since the case Open image in new window follows in a similar manner. By definition of \(\alpha _{\delta \text {plant}}\) we know \(0 \le x_0\), and hence continue with Open image in new window by differential cut \(0 \le x_0\). Using the differential cut rule [31], we further supply the oracle \(\text {sol}_x + \delta x_0\), where \(\text {sol}_x\) denotes the solution of Open image in new window in \(\alpha _\text {plant}\) and \(\delta x_0\) the solution for the disturbance since \(\delta \) is constant additive. This leads to two proof obligations:We conclude that the oracle is proven correct and its usage is decidable.

Prove oracle Open image in new window , which by rule differential invariant [31] is valid if we can show Open image in new window where the primed variables are replaced with the respective righthand side of the differential equation system. From Definition 4 we know that Open image in new window and Open image in new window and since \(\text {sol}_x\) is the solution of Open image in new window in \(\alpha _\text {plant}\) we further know that Open image in new window ; hence we have to show Open image in new window , which is trivially true.
 Use oracle Open image in new window , which by rule differential weaken [31] is valid if we can show where \(\forall ^\alpha \) denotes the universal closure w.r.t. \(\alpha \), i. e., \(\forall x\). But since Open image in new window is valid, this is provable by quantifier elimination. Furthermore, we cannot get a better result than differential weaken, because the evolution domain constraint contains the oracle’s answer for the differential equation system, which characterizes exactly the reachable set of the differential equation system.


For computability, we start with a theoretical proof on the basis of decidability, before we give a constructive proof, which is more useful in practice.
Theorem 5
(Monitor synthesis is computable) We assume canonical models of the form \(\alpha \equiv \alpha _{\text {ctrl}} ; \alpha _{\text {plant}}\) without nested loops, with solvable differential equations in \(\alpha _\text {plant}\) and disturbed plants \(\alpha _{\delta \text {plant}}\) with constant additive disturbance \(\delta \) (see Definition 4). Then, monitor synthesis is computable, i. e., the functions Open image in new window , Open image in new window , and Open image in new window are computable.
Proof
Follows immediately from Theorem 4 with recursive enumeration of monitors. \(\square \)
We give a constructive proof of Theorem 5. The proof is based on the observation that, except for loop and differential invariants/variants, rule application in the Open image in new window calculus is deterministic: from [31, Theorem 2.4] we know that, relative to an oracle for firstorder invariants and variants, the Open image in new window calculus gives a semidecisionprocedure for Open image in new window formulas with differential equations having firstorder definable flows.
Proof

Model monitor synthesis \(\chi _{\text {m}}\): the solution rule Open image in new window would make progress, because the differential equations in \(\alpha _\text {plant}\) are solvable; and for

Prediction monitor synthesis \(\chi _{\text {p}}\): the disturbance functions \(f(\theta ,\delta )\) and \(g(\theta ,\delta )\) applied to the solution provide differential invariants (see proof of Theorem 4) so that the differential cut rule, the differential invariant rule, and the differential weakening rule [31] would make progress.
3.8 A proof tactic for automatic monitor synthesis
Based on the decidability and computability results above, this section explains how to implement ModelPlex monitor synthesis (Algorithm 1) as an automatic proof tactic for correctbyconstruction monitor synthesis. This proof tactic is formulated in the tactic language of our theorem prover Open image in new window X [14]. Open image in new window X features a small soundnesscritical core for axiomatic reasoning. On top of that core, tactics steer the proof search: axiomatic tactics constitute the most basic constructs of a proof, while tactic combinators (e. g., sequential tactic execution) are a language to combine tactics into more powerful proof procedures. The tactic language of Open image in new window X provides operators for sequential tactic composition (; ), tactic repetition (\(^*\)), optional execution (?), and alternatives ( Open image in new window ) to combines basic Open image in new window tactics, see [14].
The synthesize tactic operates on a specification conjecture Open image in new window . It combines tactic selection as in a regular expression with search, so that formulas are turned into axioms stepbystep (backwards search). Synthesize starts with \(\textit{prepare}\), which determines whether to synthesize a controller monitor (unwinds loops and skips differential equations) or a model monitor (unwinds loops). Then, it repeats rewriting hybrid programs until none of the hybrid program tactics is applicable anymore, indicated by \(\textit{locate}(\textit{rewriteHP})^*\). Note, that the synthesize tactic does not need to instantiate existential quantifiers at intermediate steps, since it can continue rewriting inside existential quantifiers. After rewriting hybrid programs is done, an optional local quantifier elimination step is made, cf. (6), in case that any universal quantifiers remained from the ODE in the innermost subformula, followed by instantiating the existential quantifiers using Open image in new window .
At the heart of the synthesize tactic is locate, which searches for the topmost formula that includes a hybrid program (i. e., a diamond modality) and chooses the appropriate tactic to reduce that program. The proof search itself is backward in sequentstyle, which starts from the monitor specification conjecture and searches for steps that transform the conjecture gradually into axioms. This tactic seems like a natural way of synthesizing monitors, since it starts from the conjecture and repeatedly applies proof steps until no more progress can be made (i. e., no more steps are applicable). However, repeated search for the topmost hybrid program operator incurs considerable computation time overhead, as we will see in Sect. 4.
To avoid repeated search, we provide another tactic using a forward chase. The forward chase uses proofascomputation and is based on unification and recursive forward application of axioms, which allows us to construct a proof computationally from axioms until we reach the monitor specification conjecture. Each step of the recursive computation knows the position where to apply the subsequent step, so that no search is necessary.
4 Evaluation
4.1 Monitor synthesis
We developed two software prototypes: A sequentstyle synthesis prototype uses Open image in new window 3 [37] and Mathematica. It uses Mathematica to simplify redundant monitor conditions after synthesizing the monitor in Open image in new window 3, and therefore must recheck the final monitor for correctness. An axiomaticstyle prototype is implemented as a tactic in Open image in new window X [14], which generates controller and model monitors fully automatically and avoids branching by operating on subformulas in a single sequent. The axiomaticstyle prototype synthesizes correctbyconstruction monitors and produces a proof of correctness during the synthesis without the need to recheck.
Case study overview
Case study  Characteristics  Dim.  Proof steps  Branches 

Water tank  1 control branch, solvable ODE  5  38  4 
Cruise control [20]  3 control branches,solvable ODE  11  969  124 
Speed limit [25]  6 control branches,solvable ODE  9  410  30 
ETCS safety [38]  8 control branches,solvable ODE  16  193  10 
Robot [26]  3 control branches,nonsolvable ODE  14  3350  225 
Monitor synthesis case studies

Table 3 summarizes the evaluation results. The main result is the completely automatic, correctbyconstruction synthesis in Open image in new window X with a single open branch on which the monitor is being synthesized. The monitor sizes in Open image in new window X are usually smaller than those of Open image in new window 3, because the structure is preserved better so no external simplification is needed, cf. last column “unsimplified”. Without external simplification, very similar conditions with only small deviations are repeated on each open branch, For example, the controller monitor sizes listed the sequentstyle synthesis need to be multiplied roughly by the number of open branches, in order to get the monitor size before external simplification.
A detailed analysis follows in subsequent paragraphs below. For the monitor, we list the dimension of the monitor specification in terms of the number of variables, compare the number of manual steps among all steps and branches left open among all branches when deriving the monitor with or without Opt. 1, as well as the number of steps when rechecking monitor correctness. Finally, we list the monitor size in terms of the number of arithmetic, comparison, and logical operators in the monitor formula. The number of proof steps of Open image in new window 3 and Open image in new window are not directly comparable, because both implement different calculi. Open image in new window X leads to more but simpler proof steps.
For the sequentstyle synthesis technique we list the time needed to perform the fully automated steps without Opt. 1 in Open image in new window . The raw synthesis times are comparable to those in the chasebased axiomaticstyle synthesis, because the sequentstyle technique always operates on the toplevel operator and does not need search. Recall, however, that in the sequentstyle synthesis technique the monitors are simplified with an unverified external procedure and, therefore, need to be rechecked for correctness in Open image in new window . This check needs considerable time, as listed in the last column of Table 4.
Open image in new window X The axiomaticstyle synthesis prototype supports proof search steering with finegrained tactics, and applies tactics inplace on subformulas, without branching on toplevel first. As a result, synthesis both with and without Opt. 1 is fully automatic and avoids redundancies in monitor conditions. The reasoning style of Open image in new window X, as illustrated in Proof 3, uses frequent cuts to collect all monitoring conditions in a single open branch, which results in a larger overall number of branches than in sequentstyle synthesis. The important characteristic is that these side branches all close, so that only a single branch remains open. This means that synthesis does not require untrusted procedures to simplify monitoring conditions that were duplicated over multiple branches, which also entails that the final rechecking of the monitor is not required, see column “proof steps (branches)”. Having only one branch and operating on subformulas also means that Opt. 1 does not need to be executed at intermediate stages in the synthesis process. Remaining existential quantifiers can be instantiated once at the end of the synthesis process, so that synthesis with and without Opt. 1 become identical.
KeYmaera X, however, is still in an early development stage and, so far, does not support differential inequalities and arbitrary differential equations in diamond modalities, so we cannot evaluate prediction monitor and model monitor synthesis fully. As development progress continues, these restrictions will diminish and we will analyze the model monitor and prediction monitor case studies with the axiomaticstyle synthesis prototype once available.
KeYmaera 3 In the sequentstyle synthesis prototype we support model monitor and prediction monitor synthesis for a wider range of systems, albeit at the cost of significantly increased manual interaction: for example, Opt. 1 has to be applied manually, since Open image in new window 3 does not provide sufficiently finegrained steering of its automated proof search. Since optimization occurs after nondeterministic assignments and differential equations (i. e., in the middle of a proof), most of the synthesis process is userguided as a consequence. For controller monitors, the sequentstyle synthesis prototype without Opt. 1 is fully automatic (see number of manual steps in column “without Opt. 1” in lines 47, marked \(\chi _{\text {c}}\)). In full automation, however, the proof search of Open image in new window 3 results in increased branching, since propositional steps during proofs are usually cheap (see number of branches in column “without Opt. 1”). As a consequence, even though the relative number of manual proof steps is reduced, the massive branching of the automatic proof search implies that in absolute terms more manual steps might be necessary than in the completely manual process (see number of manual steps in line 3, Speed limit case study, where local quantifier elimination after solving ODEs is performed manually). This can be avoided with finegrained tactic support, as is achieved in the axiomaticstyle synthesis prototype.
Although the number of steps and open branches differ significantly between manual interaction for Opt. 1 and automated synthesis, the synthesized monitors are logically equivalent. But applying Opt. 1 usually results in structurally simpler monitors, because the conjunction over a smaller number of open branches (cf. Table 3) can still be simplified automatically. The model monitors for cruise control and speed limit control are significantly larger than the other monitors, because their size already prevents automated simplification by Mathematica. Here, the axiomaticstyle synthesis approach is expected to provide significant advantage, since it does not duplicate conditions over many branches and, thus, computes small monitors even without further simplification.
4.2 Model simulation with monitoring
We tested the ModelPlex monitors with a simulation in Mathematica^{9} on hybrid system models of the water tank example used throughout this article.
Figure 11 shows a plot of the variable traces of one simulation run. In the simulation, we ran the controller every 2 s (\(\varepsilon = \text {2\,s}\), indicated by the grid for the abscissa and the marks on sensor and actuator plots). The controller was set to adjust flow to \(\frac{5(mx_0)}{2\varepsilon } = \frac{5}{2}\) for the first three controller cycles, which is unsafe on the third controller cycle. Monitor B immediately detects this violation at \(t=4\), because on the third controller cycle setting \(f=\frac{5}{2}\) violates \(f \le \frac{mx_1}{\varepsilon }\). The failsafe action at \(t=4\) drains the tank and, after that, normal operation continues until \(t=12\). Unexpected disturbance Open image in new window occurs throughout \(t=\left[ 12,14\right] \), which is detected by monitor \(\chi _{\text {m}}\). Note, that such a deviation would remain undetected with conventional approaches (monitor \(\chi _{\text {c}}\) is completely unaware of the deviation). In this simulation run, the disturbance is small enough to let the failsafe action at \(t=14\) keep the water tank in a safe state.
5 Related work
Runtime verification and monitoring for finite state discrete systems has received significant attention (e. g., [9, 16, 23]). Other approaches monitor continuoustime signals (e. g., [11, 28]). We focus on hybrid systems models of CPS to combine both.
Several tools for formal verification of hybrid systems are actively developed (e. g., SpaceEx [13], dReal [15], extended NuSMV/MathSat [6]). For monitor synthesis, however, ModelPlex crucially needs the rewriting capabilities and flexibility of (nested) Open image in new window and Open image in new window modalities in Open image in new window [31] and Open image in new window [37]; it is thus an interesting question for future work if other tools could be adapted to ModelPlex.
Runtime verification is the problem of checking whether or not a trace produced by a program satisfies a particular formula (cf. [18]). In [44], a method for runtime verification of LTL formulas on abstractions of concrete traces of a flight data recorder is presented. The RV system for Java programs [22] predicts execution traces from actual traces to find concurrency errors offline (e. g., race conditions) even if the actual trace did not exhibit the error. We, instead, use prediction on the basis of disturbed plant models for hybrid systems at runtime to ensure safety for future behavior of the system and switch to a failsafe fallback controller if necessary. Adaptive runtime verification [4] uses state estimation to reduce monitoring overhead by sampling while still maintaining accuracy with Hidden Markov Models, or more recently, particle filtering [17] to fill the sampling gaps. The authors present interesting ideas for managing the overhead of runtime monitoring, which could be beneficial to transfer into the hybrid systems world. The approach, however, focuses purely on the discrete part of CPS.
The Simplex architecture [39] (and related approaches, e. g., [1, 3, 19]) is a control system principle to switch between a highly reliable and an experimental controller at runtime. Highly reliable control modules are assumed to be verified with some other approach. Simplex focuses on switching when timing faults or violation of controller specification occur. Our method complements Simplex in that (i) it checks whether or not the current system execution fits the entire model, not just the controller; (ii) it systematically derives provably correct monitors for hybrid systems; (iii) it uses prediction to guarantee safety for future behavior of the system.
Further approaches with interesting insights on combined verification and monitor or controller synthesis for discrete systems include, for instance, [2, 12].

Unlike [39], who focus on timing faults and specification violations, we propose a systematic principle to derive monitors that react to any deviation from the model.

Unlike [4, 17, 19, 22], who focus on the discrete aspects of CPS, we use hybrid system models with differential equations to address controller and plant.

Unlike [19, 39], who assume that failsafe controllers have been verified with some other approach and do not synthesize code, we can use the same technical approach ( Open image in new window ) for verifying controllers and synthesizing provably correct monitors.

ModelPlex combines the leightweight monitors and runtime compliance of online runtime verification with the design time analysis of offline verification.

ModelPlex synthesizes provably correct monitors, certified by a theorem prover.

To the best of our knowledge, our approach is the first to guarantee that verification results about a hybrid systems model transfer to a particular execution of the system by verified runtime validation. We detect deviation from the verified model when it first occurs and, given bounds, can guarantee safety with failsafe fallback. Other approaches (e. g., [3, 19, 39]) assume the system perfectly complies with the model.
6 Conclusion
ModelPlex is a principle to build and verify highassurance controllers for safetycritical computerized systems that interact physically with their environment. It guarantees that verification results about CPS models transfer to the real system by safeguarding against deviations from the verified model. Monitors created by ModelPlex are provably correct and check at runtime whether or not the actual behavior of a CPS complies with the verified model and its assumptions. Upon noncompliance, ModelPlex initiates failsafe fallback strategies. In order to initiate those strategies early enough, ModelPlex uses prediction on the basis of disturbed plant models to check safety for the next control cycle. This way, ModelPlex ensures that verification results about a model of a CPS transfer to the actual system behavior at runtime.
The new axiomaticstyle monitor synthesis performs monitor construction in place, which enables correctbyconstruction synthesis entirely within the theorem prover, constructing a proof as evidence of the correctness of the monitor. The axiomaticstyle synthesis retains efficiency using contextual rewriting in a uniform substitution calculus for differential dynamic logic. It also preserves structure, leading to smaller monitor sizes.
Future research directions include extending ModelPlex with advanced Open image in new window proof rules for differential equations [33, 41], so that we not only synthesize prediction monitors from differential equations without polynomial solutions, but also model monitors. An interesting question for certification purposes is endtoend verification from the model to the final machine code, which this article reduces to the problem of a verified translation from the monitor formula to the monitor code. This last step is conceptually straightforward but technically nontrivial in languages like C.
Footnotes
 1.
ModelPlex checks system execution w.r.t. a monitor specification, and thus, belongs to the field of runtime verification [18]. In this article we use the term runtime validation in order to clearly convey the purpose of monitoring (i. e., runtime verification monitors properties without offline verification; ModelPlex validates adequacy of models to transfer offline verification results to the online situation).
 2.
When such constraints are not available, our method still generates verifiably correct runtime tests, which detect deviation from the model at the sampling points, just not between them. A failsafe action will then lead to earliest possible besteffort mitigation of safety risks (rather than guaranteed safety).
 3.
We use differential dynamic logic ( Open image in new window ) and Open image in new window and KeYmaera X as a theorem prover to illustrate our concepts throughout this article. The concept of ModelPlex is not predicated on the use of Open image in new window /KeYmaera X to prove (2). Other verification techniques could be used to establish validity of this formula. The flexibility of the underlying logic Open image in new window , its support for both Open image in new window and Open image in new window , and its proof calculus, however, are exploited for systematically constructing monitors from proofs in the rest of the article.
 4.
It is an annoying fact of physics that there will never quite be a perfect model of \(\gamma \).
 5.
Consecutive states for Open image in new window mean before and after executions of \(\alpha \) (i. e., in \(\alpha \overset{\downarrow }{;} \alpha \overset{\downarrow }{;} \alpha \overset{\downarrow }{;}\) at the positions indicated with an arrow, not within \(\alpha \)).
 6.
Recall that Open image in new window for variables \(V\).
 7.
The formula \(F(x,x^+)\) implies Open image in new window , because we will use nonequivalence proof steps to derive \(F(x,x^+)\) from Open image in new window .
 8.
By design, the disturbed plant \(\alpha _{\delta \text {plant}}\) also includes a clock \(x_0\), so the oracle additionally includes the trivial differential invariant \(x_0 \ge 0\).
 9.
 10.
We could run monitor \(\chi _{\text {m}}\) in place of \(\chi _{\text {c}}\) to achieve the same effect. But monitor \(\chi _{\text {m}}\) implements a more complicated formula, which is unnecessary when only the controller output should be validated.
 11.
In this example, this simplification is admittedly somewhat artificial but reminiscent of aspects of more general systems. In more complicated systems, such as adaptive cruise control, a controller needs to aim for passenger comfort, high fuel economy, and other secondary goals besides ensuring safety, so focusing on the safetyrelevant features greatly fosters safety verification.
Notes
Acknowledgments
Open access funding provided by Johannes Kepler University Linz. This material is based on research sponsored by DARPA under agreement number DARPA FA87501220291. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The research received funding from NSF, grant agreement CNS1054246. The research leading to these results has received funding from the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme (FP7/20072013) under REA grant agreement no. PIOFGA2012328378.
References
 1.Aiello AM, Berryman JF, Grohs JR, Schierman JD (2010) Runtime assurance for advanced flightcritical control systems. In: AIAA guidance, navigation and control conference, AIAA, doi: 10.2514/6.20108041
 2.Alur R, Bodík R, Juniwal G, Martin MMK, Raghothaman M, Seshia SA, Singh R, SolarLezama A, Torlak E, Udupa A (2013) Syntaxguided synthesis. In: FMCAD, IEEE, pp 1–17Google Scholar
 3.Bak S, Greer A, Mitra S (2010) Hybrid cyberphysical system verification with Simplex using discrete abstractions. In: Caccamo M (ed) IEEE Realtime and embedded technology and applications symposium, IEEE Computer Society, pp 143–152Google Scholar
 4.Bartocci E, Grosu R, Karmarkar A, Smolka SA, Stoller SD, Zadok E, Seyster J (2012) Adaptive runtime verification. In: Qadeer S, Tasiran S (eds) Runtime verification. LNCS, vol 7687. Springer, Berlin, pp 168–182Google Scholar
 5.Blech JO, Falcone Y, Becker K (2012) Towards certified runtime verification. In: Aoki T, Taguchi K (eds) International conference on formal engineering methods. LNCS, vol 7635. Springer, Berlin, pp 494–509Google Scholar
 6.Cimatti A, Mover S, Tonetta S (2013) SMTbased scenario verification for hybrid systems. Form Methods Syst Des 42(1):46–66CrossRefzbMATHGoogle Scholar
 7.Collins GE, Hong H (1991) Partial cylindrical algebraic decomposition for quantifier elimination. J Symb Comput 12(3):299–328MathSciNetCrossRefzbMATHGoogle Scholar
 8.Daigle MJ, Roychoudhury I, Biswas G, Koutsoukos XD, PattersonHine A, Poll S (2010) A comprehensive diagnosis methodology for complex hybrid systems: a case study on spacecraft power distribution systems. IEEE Trans Syst Man Cybern Part A 40(5):917–931CrossRefGoogle Scholar
 9.D’Angelo B, Sankaranarayanan S, Sánchez C, Robinson W, Finkbeiner B, Sipma HB, Mehrotra S, Manna Z (2005) LOLA: Runtime monitoring of synchronous systems. In: TIME, IEEE Computer Society, pp 166–174Google Scholar
 10.Davenport JH, Heintz J (1988) Real quantifier elimination is doubly exponential. J Symb Comput 5(1–2):29–35. doi: 10.1016/S07477171(88)80004X MathSciNetCrossRefzbMATHGoogle Scholar
 11.Donzé A, Ferrère T, Maler O (2013) Efficient robust monitoring for STL. In: Sharygina N, Veith H (eds) Computer aided verification. LNCS, vol 8044. Springer, Berlin, pp 264–279Google Scholar
 12.Ehlers R, Finkbeiner B (2011) Monitoring realizability. In: Khurshid S, Sen K (eds) Runtime verification. LNCS, vol 7186. Springer, Berlin, pp 427–441Google Scholar
 13.Frehse G, Guernic CL, Donzé A, Cotton S, Ray R, Lebeltel O, Ripado R, Girard A, Dang T, Maler O (2011) SpaceEx: scalable verification of hybrid systems. In: Gopalakrishnan G, Qadeer S (eds) Computer aided verification. LNCS, vol 6806. Springer, Berlin, pp 379–395Google Scholar
 14.Fulton N, Mitsch S, Quesel J, Völp M, Platzer A (2015) Keymaera X: an axiomatic tactical theorem prover for hybrid systems. In: Felty AP, Middeldorp A (eds) Automated deduction  CADE25  25th international conference on automated deduction, Berlin, 1–7 Aug 2015, Proceedings, Springer, Lecture Notes in Computer Science, vol 9195, pp 527–538, doi: 10.1007/9783319214016_36
 15.Gao S, Kong S, Clarke EM (2013) dReal: an SMT solver for nonlinear theories over the reals. In: Bonacina MP (ed) International conference on automated deduction. LNCS, vol 7898. Springer, Berlin, pp 208–214Google Scholar
 16.Havelund K, Rosu G (2004) Efficient monitoring of safety properties. STTT 6(2):158–173Google Scholar
 17.Kalajdzic K, Bartocci E, Smolka SA, Stoller SD, Grosu R (2013) Runtime verification with particle filtering. In: Legay A, Bensalem S (eds) Runtime verification. LNCS, vol 8174. Springer, BerlinGoogle Scholar
 18.Leucker M, Schallhart C (2009) A brief account of runtime verification. J Log Algebr Program 78(5):293–303CrossRefzbMATHGoogle Scholar
 19.Liu X, Wang Q, Gopalakrishnan S, He W, Sha L, Ding H, Lee K (2008) ORTEGA: An efficient and flexible online fault tolerance architecture for realtime control systems. IEEE Trans Ind Inform 4(4):213–224CrossRefGoogle Scholar
 20.Loos SM, Platzer A, Nistor L (2011) Adaptive cruise control: Hybrid, distributed, and now formally verified. In: Butler M, Schulte W (eds) Formal methods, Springer, LNCS, vol 6664, doi: 10.1007/9783642214370_6
 21.McIlraith SA, Biswas G, Clancy D, Gupta V (2000) Hybrid systems diagnosis. In: Lynch NA, Krogh BH (eds) Hybrid systems: computation and cpmtrol. LNCS, vol 1790. Springer, Berlin, pp 282–295Google Scholar
 22.Meredith PO, Rosu G (2010) Runtime verification with the RV system. In: Barringer H, Falcone Y, Finkbeiner B, Havelund K, Lee I, Pace GJ, Rosu G, Sokolsky O, Tillmann N (eds) Runtime verification. LNCS, vol 6418. Springer, Berlin, pp 136–152Google Scholar
 23.Meredith PO, Jin D, Griffith D, Chen F, Rosu G (2012) An overview of the MOP runtime verification framework. STTT 14(3):249–289CrossRefGoogle Scholar
 24.Mitsch S, Platzer A (2014) ModelPlex: Verified runtime validation of verified cyberphysical system models. In: Bonakdarpour B, Smolka SA (eds) Runtime Verification  5th International Conference, RV 2014, Toronto, ON, Canada, September 22–25, 2014. Proceedings, Springer, Lecture Notes in Computer Science, vol 8734, pp 199–214, doi: 10.1007/9783319111643_17
 25.Mitsch S, Loos SM, Platzer A (2012) Towards formal verification of freeway traffic control. In: Lu C (ed) ICCPS, IEEE, pp 171–180, doi: 10.1109/ICCPS.2012.25
 26.Mitsch S, Ghorbal K, Platzer A (2013) On provably safe obstacle avoidance for autonomous robotic ground vehicles. In: Fox D, Hsu D (eds) Newman P. Robotics, Science and Systems. Technische Univ., BerlinGoogle Scholar
 27.Mitsch S, Quesel JD, Platzer A (2014) Refactoring, refinement, and reasoning: A logical characterization for hybrid systems. In: Jones CB, Pihlajasaari P, Sun J (eds) Formal methods, vol 8442, Springer, pp 481–496, doi: 10.1007/9783319064109_33
 28.Nickovic D, Maler O (2007) AMT: A propertybased monitoring tool for analog systems. In: Raskin JF, Thiagarajan PS (eds) FORMATS. LNCSSpringer, Berlin, pp 304–319Google Scholar
 29.Platzer A (2008) Differential dynamic logic for hybrid systems. J Autom Reason 41(2):143–189. doi: 10.1007/s1081700891038 MathSciNetCrossRefzbMATHGoogle Scholar
 30.Platzer A (2010a) Differentialalgebraic dynamic logic for differentialalgebraic programs. J Log Comput 20(1):309–352, 2008, doi: 10.1093/logcom/exn070, advance access published on 18 Nov
 31.Platzer A (2010b) Logical analysis of hybrid systems. Springer, New York. doi: 10.1007/9783642145094 zbMATHGoogle Scholar
 32.Platzer A (2011) The structure of differential invariants and differential cut elimination. Log Methods Comput Sci 8(4):1MathSciNetGoogle Scholar
 33.Platzer A (2012a) The complete proof theory of hybrid systems. In: LICS, IEEE, doi: 10.1109/LICS.2012.64
 34.Platzer A (2012b) Logics of dynamical systems. In: LICS, IEEE, pp 13–24, doi: 10.1109/LICS.2012.13
 35.Platzer A (2015) A uniform substitution calculus for differential dynamic logic. In: Felty AP, Middeldorp A (eds) Conference on automated deduction. LNCS, vol 9195. Springer, pp 467–481, doi: 10.1007/9783319214016_32, 1503.01981
 36.Platzer A, Clarke EM (2007) The image computation problem in hybrid systems model checking. In: Bemporad A, Bicchi A, Buttazzo G (eds) Hybrid systems: computation control. LNCS, Springer. doi: 10.1007/9783540714934_37
 37.Platzer A, Quesel JD (2008) KeYmaera: A hybrid theorem prover for hybrid systems. In: Armando A, Baumgartner P, Dowek G (eds) International joint conference on automated reasoning. LNCS, vol 5195. Springer, Berlin. doi: 10.1007/9783540710707_15
 38.Platzer A, Quesel JD (2009) European Train Control System: A case study in formal verification. In: Breitman K, Cavalcanti A (eds) International conference on formal engineering methods. LNCS, vol 5885. Springer, Berlin. doi: 10.1007/9783642103735_13
 39.Seto D, Krogh B, Sha L, Chutinan A (1998) The Simplex architecture for safe online control system upgrades. In: American control conference, pp 3504–3508, doi: 10.1109/ACC.1998.703255
 40.Shannon C (1949) Communication in the presence of noise. Proc IRE 37(1):10–21. doi: 10.1109/JRPROC.1949.232969 MathSciNetCrossRefGoogle Scholar
 41.Sogokon A, Jackson PB (2015) Direct formal verification of liveness properties in continuous and hybrid dynamical systems. In: Bjørner N, de Boer FD (eds) FM 2015: formal methods– 20th international symposium, Oslo. 24–26 June 2015. Proceedings, Springer, Lecture Notes in Computer Science, vol 9109, pp 514–531, doi: 10.1007/9783319192499_32
 42.Srivastava AN, Schumann J (2013) Software health management: a necessity for safety critical systems. ISSE 9(4):219–233Google Scholar
 43.Wang D, Yu M, Low CB, Arogeti S (2013) Modelbased health monitoring of hybrid systems. Springer, New York. doi: 10.1007/9781461473695 Google Scholar
 44.Wang S, Ayoub A, Sokolsky O, Lee I (2011) Runtime verification of traces under recording uncertainty. In: Sen K, Khurshid S (eds) Runtime verification. LNCS, Springer, Berlin, pp 442–456Google Scholar
 45.Zhao F, Koutsoukos XD, Haussecker HW, Reich J, Cheung P (2005) Monitoring and fault diagnosis of hybrid systems. IEEE Trans Syst Man Cybern Part B 35(6):1225–1240CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.