ConstraintBased Monitoring of Hyperproperties
Abstract
Verifying hyperproperties at runtime is a challenging problem as hyperproperties, such as noninterference and observational determinism, relate multiple computation traces with each other. It is necessary to store previously seen traces, because every new incoming trace needs to be compatible with every run of the system observed so far. Furthermore, the new incoming trace poses requirements on future traces. In our monitoring approach, we focus on those requirements by rewriting a hyperproperty in the temporal logic HyperLTL to a Boolean constraint system. A hyperproperty is then violated by multiple runs of the system if the constraint system becomes unsatisfiable. We compare our implementation, which utilizes either BDDs or a SAT solver to store and evaluate constraints, to the automatabased monitoring tool RVHyper.
Keywords
Monitoring Rewriting Constraintbased Hyperproperties1 Introduction
As today’s complex and largescale systems are usually far beyond the scope of classic verification techniques like model checking or theorem proving, we are in the need of lightweight monitors for controlling the flow of information. By instrumenting efficient monitoring techniques in such systems that operate in an unpredictable privacycritical environment, countermeasures will be enacted before irreparable information leaks happen. Informationflow policies, however, cannot be monitored with standard runtime verification techniques as they relate multiple runs of a system. For example, observational determinism [19, 21, 24] is a policy stating that altering nonobservable input has no impact on the observable behavior. Hyperproperties [7] are a generalization of trace properties and are thus capable of expressing informationflow policies. HyperLTL [6] is a recently introduced temporal logic for hyperproperties, which extends Lineartime Temporal Logic (LTL) [20] with trace variables and explicit trace quantification. Observational determinism is expressed as the formula Open image in new window , stating that all traces \(\pi ,\pi '\) should agree on the output as long as they agree on the inputs.
To further improve the conciseness of the stored traces information, we use rewriting, which is a more finegrained monitoring approach. The basic idea is to track the requirements that future traces have to fulfill, instead of storing a set of traces. In the example above, we would track the requirement that b is not allowed to hold on the first three positions of every freshly incoming trace. Rewriting has been applied successfully to trace properties, namely LTL formulas [17]. The idea is to partially evaluate a given LTL specification \(\varphi \) on an incoming event by unrolling \(\varphi \) according to the expansion laws of the temporal operators. The result of a single rewrite is again an LTL formula representing the updated specification, which the continuing execution has to satisfy. We use rewriting techniques to reduce \(\forall ^2\)HyperLTL formulas to LTL constraints and check those constraints for inconsistencies corresponding to violations.
In this paper, we introduce a complete and provably correct rewritingbased monitoring approach for \(\forall ^2\)HyperLTL formulas. Our algorithm rewrites a HyperLTL formula and a single event into a constraint composed of plain LTL and HyperLTL. For example, assume the event \(\{ in , out \}\) while monitoring observational determinism formalized above. The first step of the rewriting applies the expansion laws for the temporal operators, which results in Open image in new window . The event \(\{in, out\}\) is rewritten for atomic propositions indexed by the trace variable \(\pi \). This means replacing each occurrence of in or out in the current expansion step, i.e., before the Open image in new window operator, with \(\top \). Additionally, we strip the \(\pi '\) trace quantifier in the current expansion step from all other atomic propositions. This leaves us with Open image in new window . After simplification we have Open image in new window as the new specification, which consists of a plain LTL part and a HyperLTL part. Based on this, we incrementally build a Boolean constraint system: we start by encoding the constraints corresponding to the LTL part and encode the HyperLTL part as variables. Those variables will then be incrementally defined when more elements of the trace become available. With this approach, we solely store the necessary information needed to detect violations of a given hyperproperty.
We evaluate two implementations of our approach, based on BDDs and SATsolving, against RVHyper [13], a highly optimized automatonbased monitoring tool for temporal hyperproperties. Our experiments show that the rewriting approach performs equally well in general and better on a class of formulas which we call guarded invariants, i.e., formulas that define a certain invariant relation between two traces.
Related Work. With the need to express temporal hyperproperties in a succinct and formal manner, the above mentioned temporal logics HyperLTL and HyperCTL* [6] have been proposed. The modelchecking [6, 14, 15], satisfiability [9], and realizability problem [10] of \(\text {HyperLTL}~\) has been studied before.
Runtime verification of HyperLTL formulas was first considered for (co)ksafety hyperproperties [1]. In the same paper, the notion of monitorability for HyperLTL was introduced. The authors have also identified syntactic classes of HyperLTL formulas that are monitorable and they proposed a monitoring algorithm based on a progression logic expressing trace interdependencies and the composition of an LTL\(_3\) monitor.
Another automatabased approach for monitoring HyperLTL formulas was proposed in [12]. Given a HyperLTL specification, the algorithm starts by creating a deterministic monitor automaton. For every incoming trace it is then checked that all combinations with the already seen traces are accepted by the automaton. In order to minimize the number of stored traces, a languageinclusionbased algorithm is proposed, which allows to prune traces with redundant information. Furthermore, a method to reduce the number of combination of traces which have to get checked by analyzing the specification for relations such as reflexivity, symmetry, and transitivity with a HyperLTLSAT solver [9, 11], is proposed. The algorithm is implemented in the tool RVHyper [13], which was used to monitor informationflow policies and to detect spurious dependencies in hardware designs.
Another rewritingbased monitoring approach for HyperLTL is outlined in [5]. The idea is to identify a set of propositions of interest and aggregate constraints such that inconsistencies in the constraints indicate a violation of the HyperLTL formula. While the paper describes the building blocks for such a monitoring approach with a number of examples, we have, unfortunately, not been successful in applying the algorithm to other hyperproperties of interest, such as observational determinism.
In [3], the authors study the complexity of monitoring hyperproperties. They show that the form and size of the input, as well as the formula have a significant impact on the feasibility of the monitoring process. They differentiate between several input forms and study their complexity: a set of linear traces, treeshaped Kripke structures, and acyclic Kripke structures. For acyclic structures and alternationfree HyperLTL formulas, the problems complexity gets as low as NC.
In [4], the authors discuss examples where static analysis can be combined with runtime verification techniques to monitor HyperLTL formulas beyond the alternationfree fragment. They discuss the challenges in monitoring formulas beyond this fragment and lay the foundations towards a general method.
2 Preliminaries
Let \( AP \) be a finite set of atomic propositions and let \(\varSigma = 2^ AP \) be the corresponding alphabet. An infinite trace \(t \in \varSigma ^\omega \) is an infinite sequence over the alphabet. A subset \(T \subseteq \varSigma ^\omega \) is called a trace property. A hyperproperty \(H \subseteq 2^{(\varSigma ^\omega )}\) is a generalization of a trace property. A finite trace \(t \in \varSigma ^+\) is a finite sequence over \(\varSigma \). In the case of finite traces, t denotes the length of a trace. We use the following notation to access and manipulate traces: Let t be a trace and i be a natural number. t[i] denotes the ith element of t. Therefore, t[0] represents the first element of the trace. Let j be natural number. If \(j \ge i\) and \(i\ge t\), then t[i, j] denotes the sequence \(t[i] t[i+1] \cdots t[min(j,t1)]\). Otherwise it denotes the empty trace \(\epsilon \). \(t[i\rangle \) denotes the suffix of t starting at position i. For two finite traces s and t, we denote their concatenation by \(s \cdot t\).
3 Rewriting HyperLTL
Lemma 1
Given the body \(\varphi \) of a \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \), and a finite trace \(t \in \varSigma ^+\), it holds that \( \mathcal {L}^{\pi }_{t}(\hat{\varphi }) = \mathcal {L}_{t}(\varphi )\).
Theorem 1
Given a \(\forall ^2\)HyperLTL formula \(\forall \pi , \pi ' \mathpunct {.}\varphi \) and any two finite traces \(t,t' \in \varSigma ^+\) it holds that Open image in new window .
Proof
By induction on the size of t. Induction Base (\(t=e\), where \(e \in \varSigma \)): Let \(t' \in \varSigma ^+\) be arbitrarily chosen. We distinguish by structural induction the following cases over the formula \(\varphi \). We begin with the base cases.

\(a_{\pi }\): we know by definition that \(a_{\pi }^{\pi }_{t}\) equals \(\top \) if \(a \in t[0]\) and \(\bot \) otherwise, so it follows that Open image in new window .

\(a_{\pi '}\): Open image in new window .

\(\lnot a_{\pi }\) and \(\lnot a_{\pi '}\) are proven analogously.
The structural induction hypothesis states that Open image in new window (SIH1), where \(\psi \) is a strict subformula of \(\varphi \).

Open image in new window are proven analogously.
Induction Step (\(t = e \cdot t^*\), where \(e \in \varSigma , t^* \in \varSigma ^+\)): The induction hypothesis states that Open image in new window (IH). We make use of structural induction over \(\varphi \). All cases without temporal operators are covered as their proofs above were independent of t. The structural induction hypothesis states for all strict subformulas \(\psi \) that Open image in new window (SIH2).

Open image in new window are proven analogously.
4 ConstraintBased Monitoring
4.1 Algorithm
A Boolean constraint system is then build incrementally: we start encoding the constraints corresponding to the LTL part (in front of the nextoperator) and encode the HyperLTL part (after the nextoperator) as variables that are defined when more events of the trace come in. We continue by explaining the algorithm in detail. In line 1, we construct \(\psi \) as the negation normal form of the symmetric closure of the original formula. We build two constraint systems: C containing constraints of previous traces and \(C_t\) (built incrementally) containing the constraints for the current trace t. Consequently, we initialize C with \(\top \) and \(C_t\) with \(v_{\psi ,0}\) (lines 2 and 4). If the trace ends, we define the remaining \(v\) variables according to their polarities and add \(C_t\) to C. For each new event \(e_i\) in the trace t, and each “open” constraint in \(C_t\) corresponding to step i, i.e., \(v_{\phi ,i} \in C_t\), we rewrite the formula \(\phi \) (line 9) and define \(v_{\phi ,i}\) with the rewriting result, which, potentially introduced new open constraints \(v_{\phi ',i+1}\) for the next step \(i+1\). The constraint encoding of the current trace is aggregated in constraint \(t_ enc \) (line 7). If the constraint system given the encoding of the current trace turns out to be unsatisfiable, a violation to the specification is detected, which is then returned.
In the following, we sketch two algorithmic improvements. First, instead of storing the constraints corresponding to traces individually, we use a new data structure, which is a tree maintaining nodes of formulas, their corresponding variables and also child nodes. Such a node corresponds to already seen rewrites. The initial node captures the (transformed) specification (similar to line 4) and it is also the root of the tree structure, representing all the generated constraints which replaces C in Fig. 1. Whenever a trace deviates in its rewrite result a new child or branch is added to the tree. If a rewrite result is already present in the node tree structure there is no need to create any new constraints nor new variables. This is crucial in case we observe many equal traces or traces behaving effectively the same. In case no new constraints were added to the constraint system, we omit a superfluous check for satisfiability.
Second, we use conjunct splitting to utilize the node tree optimization even more. We illustrate the basic idea on an example. Consider \(\forall \pi , \pi ' \mathpunct {.}\varphi \) with Open image in new window , which demands that on all executions on each position at least on of propositions a or b agree in its evaluation. Consider the two traces \(t_1 = \{a\}\{a\}\{a\}\), \(t_2 = \{a\}\{a,b\}\{a\}\) that satisfy the specification. As both traces feature the same first event, they also share the same rewrite result for the first position. Interestingly, on the second position, we get \((a \vee \lnot b) \wedge s_\varphi \) for \(t_1\) and \((a \vee b) \wedge s_\varphi \) for \(t_2\) as the rewrite results. While these constraints are no longer equal, by the nature of invariants, both feature the same subterm on the right hand side of the conjunction. We split the resulting constraint on its syntactic structure, such that we would no longer have to introduce a branch in the tree.
4.2 Correctness
In this technical subsection, we will formally prove correctness of our algorithm by showing that our incremental construction of the Boolean constraints is equisatisfiable to the HyperLTL rewriting presented in Sect. 3. We begin by showing that satisfiability is preserved when shifting the indices, as stated by the following lemma.
Lemma 2
For any \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) over atomic propositions \(\text {AP}\), any finite traces \(t,t' \in \varSigma ^+\) and \(n \ge 0\) it holds that \(enc_{\text {AP}}(t') \vDash constr(\varphi ,t) \Leftrightarrow enc^{n}_{\text {AP}}(t') \vDash constr^{n}(\varphi ,t)\).
Proof
By renaming of the positional indices.
In the following lemma and corollary, we show that the semantics of the next operators matches the finite LTL semantics.
Lemma 3
For any \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) over atomic propositions \(\text {AP}\) and any finite traces \(t,t' \in \varSigma ^+\) it holds that Open image in new window
Proof
Let \(\varphi \), \(t,t'\) be given. It holds that Open image in new window by definition. As \(constr(v^{}_{\varphi ,1},t[1\rangle )\) by construction does not contain any variables with positional index 0, we only need to check satisfiability with respect to \(enc(t'[1\rangle )\). Thus Open image in new window .
Corollary 1
For any \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) over atomic propositions \(\text {AP}\) and any finite traces \(t,t' \in \varSigma ^+\) it holds that Open image in new window
We will now state the correctness theorem, namely that our algorithm preserves the HyperLTL rewriting semantics.
Theorem 2
For every \(\forall ^2 \)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) in negation normal form over atomic propositions \(\text {AP}\) and any finite trace \(t \in \varSigma ^+\) it holds that Open image in new window .
Proof

\(a_{\pi }\): \(constr(a_{\pi },e) = (a_\pi )[\pi ,e,0] = \top \text { if, and only if, } a \in e\). Thus Open image in new window .

\(a_{\pi '}\): \(constr(a_{\pi '},e) = (a_{\pi '})[\pi ,e,0] = a_0\) Thus Open image in new window .

\(\lnot a_{\pi }\) and \(\lnot a_{\pi '}\) are proven analogously.
The structural induction hypothesis states that Open image in new window (SIH1), where \(\psi \) is a strict subformula of \(\varphi \).

\(\varphi \wedge \psi \), Open image in new window are proven analogously.
Induction Step (\(t = e \cdot t^*\), where \(e \in \varSigma \) and \(t^* \in \varSigma ^+\)): The induction hypothesis states that Open image in new window (IH). We make use of structural induction over \(\varphi \). All base cases are covered as their proofs above are independent of t. The structural induction hypothesis states for all strict subformulas \(\psi \) that Open image in new window .

\(\varphi \vee \psi \):
Open image in new window Open image in new window \(\dag \): \(\Leftarrow \): trivial, \(\Rightarrow \): Assume a model \(M_\varphi \) for \(enc(t') \vDash \varphi [\pi ,e,0]\,\wedge \,A\). By construction, constraints by \(\varphi \) do not share variable with constraints by \(\psi \). We extend the model by assigning \(v_{\psi ',1}\) with \(\bot \), for all \(v_{\psi ',1} \in \psi [\pi ,e,0]\) and assigning the rest of the variables in \(\psi [\pi ,e,0]\) arbitrarily.

\(\varphi \wedge \psi \), Open image in new window are proven analogously.
Corollary 2
For any \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) in negation normal form over atomic propositions \(\text {AP}\) and any finite traces \(t,t' \in \varSigma ^+\) it holds that \(t' \in \mathcal {L}_{t}(\varphi ) \Leftrightarrow enc_{\text {AP}}(t') \vDash constr(\hat{\varphi },t)\).
Proof
Lemma 4
For any \(\forall ^2\)HyperLTL formula \(\forall \pi ,\pi ' \mathpunct {.}\varphi \) in negation normal form over atomic propositions \(\text {AP}\) and any finite traces \(t,t' \in \varSigma ^+\) it holds that \(enc_{\text {AP}}(t') \nvDash constr(\varphi ,t) \Rightarrow \forall t'' \in \varSigma ^+ \mathpunct {.}t' \le t'' \mathbin {\rightarrow }~enc_{\text {AP}}(t'') \nvDash constr(\varphi ,t)\).
Proof
We proof this via contradiction. We choose \(t,t'\) as well as \(\varphi \) arbitrarily, but in a way such that \(enc(t') \nvDash constr(\varphi ,t)\) holds. Assume that there exists a continuation of \(t'\), that we call \(t''\), for which \(enc(t'') \vDash constr(\varphi ,t)\) holds. So there has to exist a model assigning truth values to the variables in \(constr(\varphi ,t)\), such that the constraint system is consistent. From this model we extract all assigned truths values for positional variables for position \(t'\) to \(t''1\). As \(t'\) is a prefix of \(t''\), we can use these truth values to construct a valid model for \(enc(t') \vDash constr(\varphi ,t)\), which is a contradiction.
Corollary 3
Proof
It holds that \(\forall t,t' \in \varSigma ^+ \mathpunct {.}t \ne t' \mathbin {\rightarrow }~constr(\varphi ,t) \ne constr(\varphi ,t')\). Follows with same reasoning as in earlier proofs combined with Corollary 2.
5 Experimental Evaluation
Average results of our implementation compared to RVHyper on traces generated from circuit instances. Every instance was run 10 times.
instance  # traces  length  time RVHyper  time SAT  time BDD 

xor1  19  5  12 ms  47 ms  49 ms 
xor2  1000  5  16913 ms  996 ms  1666 ms 
counter1  961  20  9610 ms  8274 ms  303 ms 
counter2  1353  20  19041 ms  13772 ms  437 ms 
mux1  1000  5  14924 ms  693 ms  647 ms 
mux2  80  5  121 ms  79 ms  81 ms 
The biggest difference can be seen between the runtimes for counter2. This is explained by the fact that this benchmark demands the highest number of observed traces, and therefore the impact of the quadratic runtime costs in the number of traces dominates the result. We can, in fact, clearly observe this correlation between the number of traces and the runtime on RVHyper’s performance over all benchmarks. On the other hand our constraintbased implementations do not show this behavior.
Guarded Invariants. We consider a new class of benchmarks, called guarded invariants, which express a certain invariant relation between two traces, which are, additionally, guarded by a precondition. Figure 3 shows the results of monitoring an arbitrary invariant \(P : \varSigma \rightarrow \mathbb {B}\) of the following form: Open image in new window . Our approach significantly outperforms RVHyper on this benchmark class, as the conjunct splitting optimization, described in Sect. 4.1, synergizes well with SATsolver implementations.
Atomic Proposition Scalability. While RVHyper is inherently limited in its scalability concerning formula size as the construction of the deterministic monitor automaton gets increasingly hard, the rewritebased solution is not affected by this limitation. To put it to the test we have ran the SATbased implementation on guarded invariant formulas with up to 100 different atomic propositions. Formulas have the form: Open image in new window where \(n_{in}, n_{out}\) represents the number of input and output atomic propositions, respectively. Results can be seen in Fig. 4. Note that RVHyper already fails to build monitor automata for \(n_{in}+n_{out} > 10\).
6 Conclusion
We pursued the success story of rewritebased monitors for trace properties by applying the technique to the runtime verification problem of Hyperproperties. We presented an algorithm that, given a \(\forall ^2\)HyperLTL formula, incrementally constructs constraints that represent requirements on future traces, instead of storing traces during runtime. Our evaluation shows that our approach scales in parameters where existing automatabased approaches reach their limits.
Notes
Acknowledgments
We thank Bernd Finkbeiner for his valuable feedback on earlier versions of this paper.
References
 1.Agrawal, S., Bonakdarpour, B.: Runtime verification of ksafety hyperproperties in HyperLTL. In: Proceedings of CSF, pp. 239–252. IEEE Computer Society (2016). https://doi.org/10.1109/CSF.2016.24
 2.Bonakdarpour, B., Finkbeiner, B.: Runtime verification for HyperLTL. In: Falcone, Y., Sánchez, C. (eds.) RV 2016. LNCS, vol. 10012, pp. 41–45. Springer, Cham (2016). https://doi.org/10.1007/9783319469829_4CrossRefGoogle Scholar
 3.Bonakdarpour, B., Finkbeiner, B.: The complexity of monitoring hyperproperties. In: Proceedings of CSF, pp. 162–174. IEEE Computer Society (2018). https://doi.org/10.1109/CSF.2018.00019
 4.Bonakdarpour, B., Sánchez, C., Schneider, G.: Monitoring hyperproperties by combining static analysis and runtime verification. In: Margaria, T., Steffen, B. (eds.) ISoLA 2018. LNCS, vol. 11245, pp. 8–27. Springer, Cham (2018). https://doi.org/10.1007/9783030034214_2CrossRefGoogle Scholar
 5.Brett, N., Siddique, U., Bonakdarpour, B.: Rewritingbased runtime verification for alternationfree HyperLTL. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10206, pp. 77–93. Springer, Heidelberg (2017). https://doi.org/10.1007/9783662545805_5CrossRefGoogle Scholar
 6.Clarkson, M.R., Finkbeiner, B., Koleini, M., Micinski, K.K., Rabe, M.N., Sánchez, C.: Temporal logics for hyperproperties. In: Abadi, M., Kremer, S. (eds.) POST 2014. LNCS, vol. 8414, pp. 265–284. Springer, Heidelberg (2014). https://doi.org/10.1007/9783642547928_15CrossRefGoogle Scholar
 7.Clarkson, M.R., Schneider, F.B.: Hyperproperties. J. Comput. Secur. 18(6), 1157–1210 (2010). https://doi.org/10.3233/JCS20090393CrossRefGoogle Scholar
 8.Clavel, M., et al.: The Maude 2.0 system. In: Nieuwenhuis, R. (ed.) RTA 2003. LNCS, vol. 2706, pp. 76–87. Springer, Heidelberg (2003). https://doi.org/10.1007/3540448810_7CrossRefGoogle Scholar
 9.Finkbeiner, B., Hahn, C.: Deciding hyperproperties. In: Proceedings of CONCUR. LIPIcs, vol. 59, pp. 13:1–13:14. Schloss Dagstuhl  LeibnizZentrum fuer Informatik (2016). https://doi.org/10.4230/LIPIcs.CONCUR.2016.13
 10.Finkbeiner, B., Hahn, C., Lukert, P., Stenger, M., Tentrup, L.: Synthesizing reactive systems from hyperproperties. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 289–306. Springer, Cham (2018). https://doi.org/10.1007/9783319961453_16CrossRefGoogle Scholar
 11.Finkbeiner, B., Hahn, C., Stenger, M.: EAHyper: satisfiability, implication, and equivalence checking of hyperproperties. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10427, pp. 564–570. Springer, Cham (2017). https://doi.org/10.1007/9783319633909_29CrossRefGoogle Scholar
 12.Finkbeiner, B., Hahn, C., Stenger, M., Tentrup, L.: Monitoring hyperproperties. In: Lahiri, S., Reger, G. (eds.) RV 2017. LNCS, vol. 10548, pp. 190–207. Springer, Cham (2017). https://doi.org/10.1007/9783319675312_12CrossRefGoogle Scholar
 13.Finkbeiner, B., Hahn, C., Stenger, M., Tentrup, L.: RVHyper: a runtime verification tool for temporal hyperproperties. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 194–200. Springer, Cham (2018). https://doi.org/10.1007/9783319899633_11CrossRefGoogle Scholar
 14.Finkbeiner, B., Hahn, C., Torfah, H.: Model checking quantitative hyperproperties. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 144–163. Springer, Cham (2018). https://doi.org/10.1007/9783319961453_8CrossRefGoogle Scholar
 15.Finkbeiner, B., Rabe, M.N., Sánchez, C.: Algorithms for model checking HyperLTL and HyperCTL\(^*\). In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 30–48. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_3CrossRefGoogle Scholar
 16.Goguen, J.A., Meseguer, J.: Security policies and security models. In: Proceedings of S&P, pp. 11–20. IEEE Computer Society (1982). https://doi.org/10.1109/SP.1982.10014
 17.Havelund, K., Rosu, G.: Monitoring programs using rewriting. In: Proceedings of ASE, pp. 135–143. IEEE Computer Society (2001). https://doi.org/10.1109/ASE.2001.989799
 18.Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems  Safety. Springer, New York (1995). https://doi.org/10.1007/9781461242222CrossRefzbMATHGoogle Scholar
 19.McLean, J.: Proving noninterference and functional correctness using traces. J. Comput. Secur. 1(1), 37–58 (1992). https://doi.org/10.3233/JCS19921103CrossRefGoogle Scholar
 20.Pnueli, A.: The temporal logic of programs. In: Proceedings of FOCS, pp. 46–57. IEEE Computer Society (1977). https://doi.org/10.1109/SFCS.1977.32
 21.Roscoe, A.W.: CSP and determinism in security modelling. In: Proceedings of S&P, pp. 114–127. IEEE Computer Society (1995). https://doi.org/10.1109/SECPRI.1995.398927
 22.Somenzi, F.: Cudd: Cu decision diagram packagerelease 2.4.0. University of Colorado at Boulder (2009)Google Scholar
 23.Soos, M., Nohl, K., Castelluccia, C.: Extending SAT solvers to cryptographic problems. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 244–257. Springer, Heidelberg (2009). https://doi.org/10.1007/9783642027772_24CrossRefGoogle Scholar
 24.Zdancewic, S., Myers, A.C.: Observational determinism for concurrent program security. In: Proceedings of CSFW, p. 29. IEEE Computer Society (2003). https://doi.org/10.1109/CSFW.2003.1212703
Copyright information
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.