Coupled Relational Symbolic Execution for Differential Privacy

Differential privacy is a de facto standard in data privacy with applications in the private and public sectors. Most of the techniques that achieve differential privacy are based on a judicious use of randomness. However, reasoning about randomized programs is difficult and error prone. For this reason, several techniques have been recently proposed to support designer in proving programs differentially private or in finding violations to it. In this work we propose a technique based on symbolic execution for reasoning about differential privacy. Symbolic execution is a classic technique used for testing, counterexample generation and to prove absence of bugs. Here we use symbolic execution to support these tasks specifically for differential privacy. To achieve this goal, we leverage two ideas that have been already proven useful in formal reasoning about differential privacy: relational reasoning and probabilistic coupling. Our technique integrates these two ideas and shows how such a combination can be used to both verify and find violations to differential privacy.


Introduction
Differential Privacy [17] has become a de facto gold standard definition of privacy for statistical analysis. This success is mostly due to the generality of the definition, its robustness and compositionality. These valuable properties helped researchers from many different communities -e.g. machine learning, data analysis, and security -in coming up with differentially private algorithms for specific goals. However, it was quickly understood that getting differential privacy right in practice is a hard task. Even privacy experts have released fragile code subject to attacks [2,11,18,22,30] and published incorrect algorithms [28]. This challenge has motivated the programming language community to develop techniques to support programmer to show their algorithms differentially private. Among the techniques that have been proposed there are type systems [4,7,20,31,33,37,38], methods based on model checking and Markov chains [3, 12-14, 27, 35], and program logics [5,6,8,9,34]. More recently, the formal methods community have also focused on developing techniques to find violations to differential privacy [3,10,16].
Most of these works focus on either verifying a program differentially private or finding violations to differential privacy and they do not consider techniques supporting both kind of reasoning. An exception is the recent work by Barthe et al. [3] which proposes a method based on a decidable logic for a simple while language over finite input and output domains, that can be used for both verifying and finding violation to differential privacy.
Motivated by this picture, we propose a new technique based on relational symbolic execution, named Coupled Relational Symbolic Execution (CRSE), which supports proving and finding violation to differential privacy for programs. Our technique is based on two essential ingredients: the use of a recently introduced notion of relational symbolic execution [19] and the use of approximate probabilistic couplings [8] to reason about differential privacy a relational way. This approach allow us also to support reasoning over countable input and output domains.
Relational Symbolic Execution. Symbolic execution is a classic technique used for bug finding, testing and proving. In symbolic execution an evaluator executes the program which consumes symbolic inputs instead of concrete ones. The evaluator follows, potentially, all the execution paths the program could take and collects constraints over the symbolic values, corresponding to these paths. The evaluator collects in this way a description of the traces in terms of constraints on symbolic values or expressions involving them. Every trace is associated with a set of constraints and every input satisfiying these constraints will lead the actual concrete execution along that trace.
Similarly, in relational symbolic execution [19] (RSE) one is concerned with bug finding, testing, or proving for relational properties. These are properties about two executions of two potentially different programs. RSE executes two potentially different programs in a symbolic fashion. RSE exploits relational assumptions about the two inputs to the two programs in order to reduce the number of states to analyze. This can be particularly effective when the codes of the two programs share some similarities, and when the property under consideration is relational in nature, as in the case of differential privacy. Approximate Probabilistic Couplings. Probabilistic coupling [26] is a proof technique useful to relate two random variables through a common joint probability distribution. Probabilistic coupling has been used in formal verification [24] to lift a relation over the joint support of two probability distribution to a relation over the two probability distributions themselves. This allows one to reason about relations between probability distributions by reasoning about relations on their support, which can be usually done in a symbolic way. In this approach the actual probabilistic reasoning is confined to the soundness of the verification system, rather than being spread everywhere. A relaxation of the notion of coupling, called approximate probabilistic coupling [8,9], has been designed to reason about differential privacy. This can be seen as a regular probabilistic coupling with some additional parameter describing how close the two probability distribution are.
In this work, we combine these two approaches in a framework called Coupled Relational Symbolic Execution (CRSE). In this framework, a program is executed in a relational and symbolic way. When some probabilistic primitive is executed, CRSE introduces constraints corresponding to the existence of an approximate probabilistic coupling on the output. These constraints are combined with the constraints on the execution traces generated by symbolically and relationally executing other non-probabilistic commands. These combined constraints can be exploited to reduce the number of states to analyze. When the execution is concluded CRSE checks whether there is a coupling between the two outputs, or whether there is some violation to the coupling. We show the soundness of this approach for both proving and refuting differential privacy. However, for finding violations, one cannot reason only symbolically, and since checking directly a coupling can be computationally expensive, we devise several heuristic which can be used to facilitate this task. Using these techniques, CRSE allows one to verifying differential privacy for an interesting class of programs, including programs working on countable input and output domains, and to find violations to programs that are not differentially private.
As we discussed at the begin of this section, other techniques have been devised to achieve similar goals. CRSE is not a replacement for them but it should be seen as an additional method to put in the set of tools of the privacy developer which provides an high level of generality. Indeed, by being a totally symbolic technique, it can leverage on a pletora of current technologies such as SMT solvers, e.g. [15], algebraic solvers, e.g. [23], and numeric solvers, e.g. [29].
Summarizing, the contribution of our work are: -We combine relational symbolic execution and approximate probabilistic coupling in a new technique, named Coupled Relational Symbolic Execution (CRSE). -We show CRSE sound for both proving programs differentially private and for refuting differential privacy. -We devise a set of heuristic that can help a programmer in finding violations to differential privacy. -We show how CRSE can help in proving and refuting differential privacy for an interesting class of programs

CRSE Informally
In this section, we will motivate in an informal way CRSE through three examples of programs showing potential errors in implementations of (supposedly) differentially private algorithms. In doing this we will also presenting the notation that will use in the rest of the paper.

Single query with wrong noise parameter.
Differential Privacy. Informally, a randomized function A is -differential privacy if it maps two databases d 1 and d 2 that differ for the data of one single individual (denoted d 1 ∼ d 2 ) to output distributions that are indistinguishable up to some value -usually referred to as the privacy budget -this is formalized by requiring that the log-ratio of the two probability distributions is bounded pointwise by , i.e. for every u, log Pr x←A (d 2 ) [x=u] ≤ -we will give the precise definition in Section 3. The smaller the , the more privacy is guaranteed. A standard way to achieve differential privacy when we are interested in a numeric query over a dataset is to add to the query result some noise sampled from the Laplace distribution with mean 0 and scale proportional to the sensitivity of the function (how far the function maps two databases differing for the data of one single individual) over [17]. Algorithm 1 is a wrong implementation of this principlemore in general it is a simple example of a program that is implemented with the wrong noise parameters. Specifically, it takes in input a numeric query q with type D → Z a database d ∈ D, and the privacy budget we want to guarantee ∈ R + . It then computes the query on the database, adds Laplace noise with scale equal to 1 to the result of the query 4 , and releases the result. This program is not -differentially private, because it doesn't calibrate the Laplace noise to the sensitivity of the query. In fact, as a precondition we assert that the query q is r-sensitive by the requirement d 1 ∼ d 2 ⇒ |q(d 1 ) − q(d 2 )| ≤ r, asserting that given two databases d 1 and d 2 differing for the data of one individual the query q returns two results that are at most at distance r. The program implementing algorithm 1 would be -differentially private if we added noise proportional to 1 r instead of 1 , that is using the assignment ρ $ ← lap r (0) instead of ρ $ ← lap (0), in line 2 the algorithm. To show formally that we have a privacy violation, accordingly to the definition of differential privacy, we need to witness a query q, two databases d 1 and d 2 in the relation d 1 ∼ d 2 , and a possible output u making the two probability distributions distinguishable for more than . Approching this task directly is intractable [21].

Algorithm 1 A buggy Laplace mechanism
Instead, in order to do this, CRSE will execute the program in a relational symbolic fashion and it will try to prove that in two runs of the program the output variable has the same value and the privacy budget spent is at most . Technically, this is implemented by considering the postcondition , where c is a distinguished variable recording the privacy budget spent. If CRSE succeed, then the program is -differentially private. If there is an execution that invalidates this post-condition, then we will have a candidate for a witness of the violation.
To avoid resorting to sampling, when CRSE executes the command for Laplace (as in line 2), following the approximate probabilistic coupling idea from [8], it couples the samples (ρ 1 , ρ 2 ) in the two runs, and adds the constraint ρ 1 +k = ρ 2 , for some k. It also tracks the budget spent with the constraint c = |k|· . The intuition behind this constraint is that we can ensure the two samples to be at some distance if we pay enough budget. From this we can see that if o 1 is to be equal to o 2 then k needs to be necessarily equal to v 1 − v 2 . Since, q(d 1 ) = v 1 , q(d 2 ) = v 2 , the difference v 1 − v 2 is bounded above by r, and we get that, in the worst case c = r . This means that in order to achieve equality of the output variables and hence, differential privacy, we need to spend at least r times the budget . So, if we are trying to use less budget, the constraints will give us a candidate for a witness of the violation.

Algorithm 3 Another buggy Above Threshold
The next two examples are variations of the same algorithm: above threshold, a component of the sparse vector technique [28]. Given a numeric threshold, an array of numeric queries of length n, and a dataset, this algorithm returns the index of the first query whose result exceeds the threshold -and potentially it should also return the value of that query. This should be done in a way that preserves differential privacy. To do this in the right way, a program should add noise to the threshold (even if it is not a sensitive data), add noise to each query, compare the values, and return the index of the first query for which this comparison succeed. The analysis of this algorithm is rather complex: it uses the noise on the threshold as a way to pay only once for all the queries that are below the threshold, and the noise on the queries to pay for the first and only query that is above the threshold, if any. Due to this complex analysis, this algorithm has been a benchmark for tools for reasoning about differential privacy [3,8,38]. Algorithm 2 has a bug making the whole algorithm not differentially private, for values of n greater than 4. The program takes in input an array of queries of type D → Z, a privacy budget and a thresold t. The program initializes an array of outputs o to all bottoms values, and a variable r to n + 1 which will be used as guard in the main loop. It then adds noise to the threshold, and iterates over all the queries adding noise to their results. If one of the noised-results is above the noisy threshold it saves the value in the array of outputs and updates the value of the guard variable, causing it to exit the main loop. Otherwise it keeps iterating. The bug is returning the value of the noisy query that is above the threshold and not only its index, as done by the instruction in red in line 6 -this is indeed not enough for guaranteeing differential privacy. For n < 5 this program can be shown -differentially private by using the composition property of differential privacy that says that the k-fold composition of -DP programs is k -differentially private(Section 3). However, for n ≥ 5 the more sophisticated analysis we described above fails. The proof principle CRSE will use to try to show this program -differentially private is to prove the assertion o 1 = ι =⇒ o 2 = ι ∧ c ≤ , for every ι ≤ n -the soundness of this principle has been proved in [8]. That is, CRSE will try to prove the following assertions (which would prove the program without bug -differentially private): While proving the first assertion, CRSE will first couple at line 3 the threshold ast 1 + k 0 =t 2 , for k 0 > 1 where 1 is the sensitivity of the queries, which is needed to guarantee that all the query results below the threshold in one run stay below the threshold in the other run, then, it will increase appropriately the privacy budget by k 0 2 . As a second step it will coupleŝ 1 + k 1 =ŝ 2 in line 4. Now, the only way for the assertion o 1 = [ŝ 1 , ⊥, ⊥] =⇒ o 2 = [ŝ 1 , ⊥, ⊥] to hold, is guaranting that bothŝ 1 =ŝ 2 andŝ 1 ≥ t 1 =⇒ŝ 2 ≥ t 2 hold. But these two assertions are not consistent with each other because k 0 ≥ 1. That is, the only way, using these coupling rules, to guarantee that the run on the right follows the same branches of the run on the left (this being necessary for proving the postcondion) is to couple the samplesŝ 1 andŝ 2 so that they are different, this necessarily implying the negation of the postcondition. This would not the the case, if we were returning only the index of the query, since we can have that both the queries are above the threshold but return different values. Indeed, by substituting line 7 with o[i] $ ← − the program can be proven -differentially private. So the refuting principle CRSE will use here is the one that finds a trace on the left run such that the only way the right run can be forced to follow it is by making the output variables different.
A second example with bug of the above threshold algorithm is shown in Figure 3. In this example, in the body of the loop, the test is performed between the noisy threshold and the actual value of the query on the database -that is, we don't add noise to the query. CRSE will use for this example another refuting principle based on reachability. In particular, it will vacuously couple the two thresholds at line 1. That is it will not introduce any relation betweent 1 , andt 2 . CRSE will then search for a trace which is satisfiable in the first run but not in the second one. This translates in an output event which has positive probability on the first run but 0 probability in the second one leading to an unbounded privacy loss, and making the algorithm not -differentially private for all finite . Interestingly this unbounded privacy loss can be achieved with just 2 iterations.

Preliminaries
Discrete Probability Distributions Let A be a denumerable set, a subdistribution over A is a function µ : A → [0, 1] with weight a∈A µ(a) less or equal than 1. We can think abour subdistributions as functions assigning to each subset of A a probability mass. We denote the set of subdistributions over A as sdistr(A). When a subdistribution has weight equal to 1, then we call it a distribution. We denote the set of distributions over A by distr(A). An example of a subdistribution that we will use in the sequel is the null subdistribution µ 0 : A → [0, 1], assigning to every element of A mass 0. Another example is the Dirac's distribution unit(a) : This is a distribution assigning all the mass to the element a ∈ A. The set of subprobability distributions can be given the structure of a monad, with unit the function unit associating with each element its Dirac's distribution -this is why we chose this notation. We have also a function bind : sdistr(A) → (A → sdistr(B)) → sdistr(B) allowing us to compose subdistributions (as we compose monads). This is defined as bind ≡ λµ.λf.λa. b∈O µ(b) · f (b)(a). We will use these constructions to give a semantics to our language in Section 4.3. We will also use the following notion of -divergence to define a notion of approximate coupling at the end of this section. Definition 1. Let ≥ 0. The -divergence between two subdistributions µ 1 , µ 2 ∈ sdistr(A), denoted by ∆ (µ 1 , µ 2 ), is defined as: Differential Privacy Differential Privacy intuitively guarantees that computation over any two inputs differing for the data of one individual result in close distributions over outputs. Formally, it is defined as follows.
The relation ∼ models which pairs of input databases should be considered sensitive, i.e., what data should be nearly indistinguishable for an adversary. In this work we will mostly consider the vanilla definition of differential privacy where δ = 0. Differential privacy implies a number of interesting properties. Here we will describe the most interesting ones for this work.
Lemma 1 (Sequential Composition [17]). Given an A 1 and A 2 , respectively ( 1 , δ 1 )-dp and ( 2 , δ 2 )-dp, their sequential composition In the specific case of A 2 being 0-d.p, for instance when A 2 ignores or does not depend on d, the property of sequential composition is called post-processing. It intuitively means that an ( , δ) differentially private answer remains such when arbitrarly post processed, as long as the post processing does not depend on the data. Any differentially private version of a numeric query has necessarily to hide the difference in output of two adjacent inputs [36]. This difference in output is captured by the following notion of sensitivity of a function.
The following lemma provides the first differentially private primitive.
Lemma 2 (Laplace Mechanism [17]). Let > 0, and assume that f : D → Z is a k sensitive function with repsect to ∼⊆ D × D. Then the randomized algorithm mapping D to f (D) + ν, where ν is sampled from the Laplace distribution with scale 1 , is k -differentially private w.r.t to ∼.
Approximate Probabilistic Liftings In this section we will give the formal and precise defintion of probabilistic approximate liftings and we will make explicit their connection with differential privacy. Definition 4. Given two sub-distributions µ 1 ∈ sdistr(A), µ 2 ∈ sdistr(B), a relation Ψ ⊆ A × B, and ∈ R, δ ∈ [0, 1], we say that µ 1 , µ 2 are related by the ( , δ) approximate lifting of Ψ iff there exists µ L , µ R ∈ distr(A × B) such that: From Lemma 3 we can derive that an algorithm A is ( , δ)-dp w.r.t to an dajcency The following lemma states another useful proof principle.
Lemma 4 (Pointwise Differential Privacy [8]). An algorithm A : The next lemma, finally, casts the Laplace mechanisms in terms of couplings.
Lemma 5. Let L v1,b , L v2,b two random variables with law Laplace distribution with mean v 1 , and v 2 respectively, and b as scale.

Concrete languages
In this section we will describe the syntax and the semantics of the concrete languages PFOR and RPFOR. We call them concrete languages, as opposed to symbolic, as it is standard in the symbolic execution literature( [19,25]). The first language, PFOR, is the language in which our programs will be written in and is a simple imperative language with for loops and random assignment. In order to prove relational properties about them we will define RPFOR that, with a relational semantics, will capture pairs of PFOR programs and their paired semantics. We start off with PFOR.  PFOR is a basic FOR-like language with array and probabilistic sampling from the Laplace distribution. The syntax ispretty standard and we present it in Figure 2. We let n, n 1 , n 2 range over Z. We let e 1 , e 2 , e range over the set of arithmetic expressions E which is inductively defined. Expressions are basic values v ∈ V ≡ Z ∪ X p , where the set X p contains values denoting random expressions and will be explained more at the semantic level in the next section. Program variables are also expressions x ∈ V as well as arithmetic operations e 1 ⊕ e 2 where ⊕ ∈ {+, −, * , /}. Finally, array accesses a[e], and len(a) when a is an array name in A. We assume V, A, and X p to be pairwise disjoint. The set of commands C includes assignments, array assignments, the skip command, sequencing, branching, and a looping construct. Finally, we also include a primitive instruction x $ ← −lap e2 (e 1 ) to model random sampling from the laplace distribution.

PFOR Semantics
As mentioned above, the set X p contains values denoting random expressions. We call values in X p distribution values. We will use capital letters such as X, Y, . . . to denote arbitrary elements in X p In Figure 3, we introduce a grammar of random expressions, where X ranges over X p and n, n 1 , n 2 ∈ Z. The simple constraints in the syntactic categories ra and re record that a random value is either associated with a specific distribution, or that the computation is conditioned on some random expression being greater than 0 or less than or equal than 0. The former constraints, as we will see, come from branching instructions. We treat constraint lists p, p , in Figure 3 as lists of simple constraints and hence, from now on, we will use the infix operators :: and @, respectively, for appending a simple constraint to a constraint and for concatenating two constraints. The symbol [] denotes the empty list of probabilistic constraints. Environments in the set M, or probabilistic memories, map program variables to values in V, and array names to elements in Array ≡ i V i , so the type of a memory m ∈ M is V → V ∪ A → Array. We will distinguish between probabilistic concrete memories in M and concrete memories in the set M c ≡ V → Z ∪ A → i Z i . Probabilistic concrete memories are meant to denote subdistributions over the set of concrete memories M c , more about this connection in Section 4.3. Expressions in PFOR are given meaning through a big-step evaluation semantics specified by a judgment of the form: m, e, p ↓ c v, p , where m ∈ M, e ∈ E, p, p ∈ P, v ∈ V. The judgments reads as: expression e reduces to the value v and probabilistic constraints p in an enviroment m with probabilistic concrete constraints p. Commands are given meaning through a small-step evaluation semantics specified by a judgment of the form: m, c, p → c m , c , p , where m, m ∈ M, c, c ∈ C, p, p ∈ P . The judgment reads as: the probabilistic concrete configuration m, c, p steps in to the probabilistic concrete configuration m , c , p . We call a probabilistic concrete configuration of the form m, skip, p final. A set of concrete configurations D is called final and we denote it by Final(D) if all its concrete configurations are final. We will use this predicate even for sets of sets of concrete configurations with the obvious lifted meaning. Figure (4) shows a selection of the rules defining these judgments. re ::= n | X | re ⊕ re P p : Fig. 3: Concrete probabilistic constraints Most of the rules are self-explanatory so we only describe the ones which are non standard. Rule lap-ass handles the random assignment. It evaluates the mean e 1 and the scale e 2 of the distribution and checks that e 2 actually denotes a positive number. The semantic predicate · fresh(·) asserts that the first argument is drawn non deterministically from the second argument and that it was never used before in the computation. Notice that if one of these two expressions reduces to a probabilistic symbolic value the computation halts. Rule if-true-prob (and if-false-prob) reduces the guard of a branching instruction to a value. If the value is a probabilistic symbolic constraint then it will nondeterministically choose one of the two branches recording the choice made in the list of probabilistic constraints. If instead the value of the guard is a numerical constant it will choose the right branch deterministically using the rules if-false and if-true (not showed). As clear from the rules a run of a PFOR program can generate many different final concrete configurations. A different judgment of the form D ⇒ c D , where D, D ∈ P(M × C × P ), and in particular its transitive and reflexive closure ( ⇒ * c ), will help us in collecting all the possible final configurations stemmign from a computation. The only rule that defines the judgment, Sub-distr-step, is presented in Figure 5. Rule Sub-distr-step selects non deterministically one configuration s = m, c, p from D, removes s from it, and adds to D all the configurations s that are reachable from s.

From configurations to subdistribution
In section 3 we defined the notions of lifting, coupling and differential privacy using subdistributions in the form of functions from a set of atomic events to the interval [0, 1]. The semantics of the languages proposed so far though only deal with subidstributions represented as set of concrete probabilistic configurations. In this section we will map the latter to the former. We start by giving two operators used to compose and define new subdistridbutions in the functional form, that is: unit(·), and bind(·, ·). In the following, we use lambda notation and the denumerable sets O, O are uni-versally quantified. The first one is defined as: The second one, is defined as In particular unit(·) takes an arbitrary element a in a set O and returns a delta distribution centered in a. bind(·, ·) builds a new subdistribution starting from a family an initial subdistribution and a family of conditional distributions. Using unit(·), bind(·, ·) it is possible to give a monadic structure to the semantics of the language as it is done in ( [9]) for the language pWhile. In Figure 6 we define a translation function ( ·; · mp ) and, auxiliary functions as well, between a single probabilistic concrete configuration and a subdistribution defined using the unit(·)/bind(·, ·) constructs. We make use of the constant subdistribution µ 0 which maps every element to mass 0, and is usually referred to as the null subdistribution, also by lap n2 (n 1 )(z) we denote the mass of (discrete version of) the Laplace distribution centered in n 1 with scale n 2 at the point z. The so , λzo.unit(X = zo :: so))) re > 0 :: p p = bind( p p , λso.bind( re re so , λzo.if (zo > 0) then unit(zo) else µ0)) re ≤ 0 :: Fig. 6: Translation from configuration to unit(·)/bind(·, ·) representation of subdistribution idea of the translation is that we can transform a probabilistic concrete memory m s ∈ M into a distribution over fully concrete memories in M c by sampling from the distributions of the probabilistic variables defined in m s in the order they were decleared which is specified by the probabilistic path constraints. To do this we first build a substitution for the probabilistic variable which maps them into integers and then we perform the substitution on m s . Given a set of probabilistic concrete memories we can then turn them in a subdistribution by summing up all the translations of the single probabilistic configurations. Indeed, given two subdistributions µ 1 , µ 2 defined over the same set we can always define the subdistribution µ 1 + µ 2 by the mapping (µ 1 + µ 2 )(a) = µ 1 (a) + µ 2 (a).
The following Lemma states an equivalence between these two representations of probability subdistributions. The hypothesis of the theorem involve a judgment, m p, which has not been specified for lack of space but can be found in the appendix, it deals with well-formedness of the probabilistic path constraint p with respect to the concrete probabilistic memory m.
We can now hence take, in this work, the following as definition of full denotational semantics of a program executed in a memory:

RPFOR syntax
PFOR's semantics is unary, syntactically meaning that the configurations it deals with are characterized by memories mapping variables and arrays names to single objects. Semantically, it means that it captures only the computation of a program over a single memory. In order to be able to reason about a relational property, such as differential privacy, we will build on top of it a relational language called RPFOR with a relational semantics dealing with pair of traces. Intuitively, an execution of a single RPFOR program represents the execution of two PFOR programs. Inspired by the approach of [32], we extend the grammar of PFOR with a pair constructor ·|· which can be used at the level of values v 1 |v 2 , expressions e 1 |e 2 , or commands c 1 |c 2 . Notice that c i , e i , v i for i ∈ {1, 2} are commands, expressions, and values in PFOR, hence nested pairing is not allowed. This syntactic invariant is preserved by the rules handling the branching instruction. Pair constructs are used to indicate where commands, values, or expressions might be different in the two unary executions represented by a single RPFOR execution. To define the semantics for RPFOR, we first extend memories to allow program variables to map to pairs of integers, and array variables to map to pairs of arrays. The set of expressions and commands in RPFOR, E r , C r are generated by the grammars: Values can now be also pairs of unary values, that is V r ≡ V ∪ V 2 .

RPFOR semantics
In the following we will use the following projection functions · i for i ∈ {1, 2}, which project, respectively, the first (left) and second (right) elements of a pair construct (i.e., c 1 |c 2 i = c i , e 1 |e 2 i = e i with v i = v when v ∈ V), and are homomorphic for other constructs. The semantics of expressions in RPFOR is specified through the following judgment m 1 , m 2 , e, p 1 , p 2 ↓ rc v, p 1 , p 2 , where m 1 , m 2 ∈ M, p 1 , p 2 , p 1 , p 2 ∈ P, e ∈ E r , v ∈ V r . Similarly, for commands, we have the following judgment m 1 , m 2 , c, p 1 , p 2 → rc m 1 , m 2 , c , p 1 , p 2 . Again, we use the predicate Final(·) for configurations m 1 , m 2 , c, p 1 , p 2 such that c = skip, and lift the predicate to sets of configurations as well. Intuitively a relational probabilistic concrete configuration m 1 , m 2 , c, p 1 , p 2 denotes a pair of probabilistic concrete states, that is a pair of subdistributions over the space of concrete memories. In Figure 7 a selection of the rules defining the the judgements is presented. Most of the rules are quite natural. Notice how branching instructions combine both probabilistic and relational nondeterminism. So, as in the case of , m2, c1|c2 , p1, p2 →rc m 1 , m 2 , c 1 |c 2 , p 1 , p 2 Fig. 7: RPFOR selected rules PFOR, we collect sets of relational configurations using the judgment R ⇒ rc R with R, R ∈ P(M × M × C r × P × P ), defined by only one rule presented in Figure 8: SUB-PDISTR-STEP. The rule picks and remove non deterministically one relational configuration from a set and adds to it all those configurations that are reachable from it. As mentioned before a run of a program in RPFOR corresponds to the execution of two runs the program in PFOR. Before making this precise we extend projection functions to relational configurations in the following way: m 1 , m 2 , c, p 1 , p 2 i = m i , c, p i , for i ∈ {1, 2}. Projection functions extend in the obvious way also to sets of relational configurations. We are now ready to state the following lemma:

Symbolic languages
In this section we proceed to lift the concrete languages, metioned in Section (4), to their symbolic versions (respectively, SPFOR and SRPFOR). As it is standard in symbolic execution literature, the first step is to extend, with symbolic values X ∈ X , the set of values of the concrete languages, after that rules of semantics execution will be defined. We start off by doing this for the language PFOR. We now extend PFOR expressions with symbolic values X ∈ X . Syntax of SPFOR is presented in Figure 9. As we can see the syntax is similar to that of PFOR except that the set of expressions has been increased with symbolic values from X denoting integers. We assume X p ∩ X = ∅. This assumption reflects the idea that symbolic values in X do not denote unknown or sets of probability distributions but only unknown sets of integers. The set of values is now V s ≡ V ∪ X , we will also need the set V is ≡ Z ∪ X . Notice how symbolic values can very well appear in probabilistic expressions.

SPFOR: Semantics of expressions
In order to collect constraints on symbolic values we extend configurations with set of constraints over integer values, drawn from the set S, not to be confused with probabilistic path constraints. The former express constraints over integer values, for instance parameters of the distributions. The grammar of constraints over integers and array values is presented in Figure 10a. In particular con-Se e ::= n | X | i | e ⊕ e | store(e, e, e) | select(e, e) | |e| S s ::= | e • e | s ∧ s | ¬s | ∀i.s (a) Grammar of constraints. X ∈ X , n ∈ V.
Grammar of symbolic probabilistic constraints. ce ∈ S, X ∈ X , Y ∈ Xp straint expressions include standard arithmetic expressions with values being symbolic or integer constants, and array selection. Actual constraints include first order logic formulas over arithmetic expressions. Finally, probabilistic path constraints now can also contain symbolic integer values and hence the grammar gets updated to what is shown in Figure 10b. What changes with respect to Figure 3 is that now arithmetic expressions can also include symbolic integer values. Indeed, also probabilistic path constraints now can be symbolic. Since values have been extended, also memories, now properly symbolic, change type in: In particular, we represent arrays in memory as pairs (X, v), where v is a (concrete or symbolic) integer value representing the length of the array, and X is a symbolic value representing the array contents. The content of the arrays is kept and refined in the set of constraints by means of select(·, ·) and store(·, ·, ·) relation symbols. Evaluation judgments, both for expressions and commands, will also include a set of constraints over integers. This is because constraints can also be generated during evaluation of expressions. We can no proceed with the semantics of expressions. Figure 11 shows the judgment form. The judgment is then inductively defined. We only show few rules for this judgment. We briefly describe the rules presented. Rule S-P-Op-2 applies when an arithmetic operation has both of its operands that reduce respectively to elements X p . Appropriately it updates the set of probabilistic constraints. Rules S-P-Op-5 instead fires when one of them is an integer and the other is a symbolic value. In this case only the list of symbolic constraints needs to be updated. Finally, in rule S-P-Op-6 one of the operands reduces to an element in X p and the other to an element in X . We only update the list of probabilistic constraints appropriately, as integer constraints cannot contain symbols in X p .

SPFOR: Semantics of commands
We can now formalize the semantics of commands of SPFOR. Again, we provide a selection of the rules of the small step semantics in Figure 12. (m, e, p, s) ↓SP (v, p , s ) Rule S-P-If-sym-true fires when a branching instruction is to be executed and the guard is reduced to either an integer or a value in X . In this case we can very well proceed with the true branch recording in the set of integer constraints the fact that the guard is greater than 0. Notice that if the guard is an integer actually less than or equal than 0 then there will never be a ground substitution for that set of constraints and hence this is not unsound.
In that case the rule, showed in the appendix, S-P-If-sym-false would instead lead to a satisfiable constraint. Rule S-P-If-prob-false handles a branching instruction which has a guard reducing to a value in X p . In this case we can proceed in both branches, even though here we only show one of the two rules, by recording the conditioning fact on the list of probabilistic constraints. Finally, rule S-P-Lap-Ass handles probabilistic assignment. After having reduced both the expression for the mean and the expression for the scale to values we check that those are both either integers or symbolic integers, if that's the case we make sure that the scale is greater than 0 and we add a probabilistic constraints recording the fact that the modified variable now points to a probabilistic symbolic value related to a Laplace distribution. Notice that again we don't handle situations where the expression for the mean or the expression for the scale reduces to a probabilistic symbolic value.

SPFOR: Collecting semantics
Semantics of SPFOR introduces two levels of nondeterminism, the first one is given by branching instructions whose guard reduces to a symbolic value, the second one comes from branching instructions whose guard reduces to a probabilistic symbolic value. The collecting semantics of SPFOR, specified by the judg-   Again, we extend the notation and use H D [s] = = ⇒ sp H when we want to make explicit the set of symbolic configurations, D [s] , that we are using to make the step. So s-p-collect, starts from a set of configurations and reaches all of those that are reachable from it. By reachable we mean that they have a satisfiable set of configurations and are reachable from one of the original configurations with only one step of the symbolic semantics. Similarly to deterministic case, the following lemma of coverage connects the PFOR with SPFOR. The difference is in the use of sets of configurations instead of single configurations. Notice that in a set of constraints can appear constraints involving probabilistic symbols. For instance if the i-th element of an array is associated with a random expression. The predicate SAT(·) does not take in consideration relations involving probabilistic symbolic constraints but only relations involving symbolic values denoting integers. Intuitively, Lemma 8 ensures us that a concrete execution is covered by a symbolic one.

SRPFOR
We finally arrived to the last rung on this ladder of languages. The language presented in this section is the the symbolic extension of the concrete language RPFOR. It can also be seen as the relational extension of SPFOR. The key part of this language's semantics will be the handling of the probabilistic assignment. For that construct we will provide 2 rules instead of one. The first one is the obvious one which carries on a standard symbolic probabilistic assignment. The second one will implement a coupling semantics on the basis of Section 3. We start off by providing the syntax of the language.   Figure 14 shows the semantics of the language SRPFOR. We extended the language SPFOR with the pairing construct, both at the level of expressions and commands. Importantly, only unary symbolic expressions and commands are admitted in the pairing construct. This invariant is maintained during branching by projection functions. In fact, projection function · i for i ∈ {1, 2}, extend to also relational symbolic expressions and commands in the following way: e 1 |e 2 i = e i , c 1 |c 2 i = c i . With v i = v, for v ∈ V s . Also, the projection functions behave homomorphically on the other constructs.

SRPFOR: Semantics of expressions
(m 1 , m 2 , e, p 1 , p 2 , s) ↓ SRP (v, p 1 , p 2 , s ) As usual we provide a bigstep evaluation semantics for expressions. The judgment form and a selection of the rules defining the judgment are provided in Figure 15. The set of val- The only rule defining the judgment ↓ SRP is S-R-P-Lift. It project the symbolic relational expression first on the left and evaluates it to a unary symbolic value, potentially updating the probabilistic symbolic constraints and the symbolic constraints. It then does the same projecting the expression on the right but starting from the potentially previously updated constraints. Now, the only case when the value returned is unary is when both the previous evaluation returned equal integers, in all the other cases an element in V s 2 is returned. So, the relational symbolic semantics leverages on the unary semantics. Before proceeding with the semantics of commands, we need to introduce the following grammars of contexts. We use evaluation contexts to simplify the exposition of the rules since for specific constructs such as relational probabilistic assignment we will have more than one rule we could fire. The two grammars for the evaluation contexts are shown in Figure 16. Notice how P gets saturated by pairs of commands. Before proceeding we want to make a syntactical distinction between commands. In particular, we call synchronizing all the commands in C rs with the following shapes x $ ← −lap e2 (e 1 ), x $ ← −lap e2 (e 1 )|x $ ← −lap e 2 (e 1 ) . We call commands with this structure: synchronizing because they allow synchronization of two runs as we will see later on. In particular, synchronizing commands are the ones that allow the use of coupling semantics and coupling rules. We call non synchronizing all the other commands in C rs The semantics of commands is again provided Non synchronizing commands In this section we provide the semantics for non synchronizing commands. In particular we are defining a judgment for a small-step semantics with the form in Figure 17. A selection of the rules inductively defining the judgment is specified in Figure 18.
An explanation of the rules follows. Rule s-r-if-prob-prob-true-false fires when evaluating a branching instruction. In particular, it fires when the guard  s-r-pair-ctxt-1 m2, P(c1, c2), p1, p2, s) →SRP (m 1 , m 2 , P(c 1 , c 2 ), p 1 , p 2 , s ) evaluates on both side to a probabilistic symbolic value. In this case the semantics can continue with the true branch on the left run and with the false branch on the right one. Notice that commands are projected to avoid pairing commands appearing in a nested form. In the case where the guard of a branching instruction evaluates to a probabilistic symbolic value on the left run and a symbolic integer value on the right one, rule s-r-if-prob-sym-true-false can apply. The rule allows to continue on the true branch on the left run and on the false branch on the right one. Notice that in one case the probabilistic list of constraints is updated, while on the other the symbolic set of constraints. Rule s-r-pair-lap-skip handles the pairing command where on the left hand side we have a probabilistic assignment and on the right a skip instruction. In this case, there is no hope for synchronization between the two runs and hence we can just unarily perform the left probabilistic assignment relying on the unary symbolic semantics. Rule s-r-pair-lapleft-sync instead applies when on the left we have a probabilistic assignment and on the right we have another arbitrary command. In this case we can hope to reach a situation where on the right run another probabilistic assignment appears. Hence, it makes sense to continue the computation unarily on the right side. Rule s-r-pair-ctxt-1 applies when a pairing command is built out of two different unary commands neither of which is a probabilistic assignment, or a sequence of commands. In this case we can just rely on the unary semantics and execute one step on side. Rule s-r-pairctxt-2 instead applies in all the other cases by recursively relying on the → SRP semantics.
SRPFOR: Collecting semantics for non synchronizing commands Again → SRP is a non deterministic semantics. The non determinism comes from the use of probabilistic symbols as guards in branching instructions, as well as from symbolic values used as guards. Finally the layer of non determinism given by the relational approach which allows runs to take different branches in a branching instruction. So, in order to collect all the possible traces stemming from such non determinism we define a collecting semantics relating set of configurations to set of configurations. The semantics is specified through a judgment with the form: SR ⇒ srp SR , with SR, SR ∈ P(M SP × M SP × C rs × SP × SP × S). The only rule defining the judgment is given in Figure 19. The rule, and the auxiliary Fig. 19: Rule for ⇒ srp notation R [s] , is pretty similar to the one in Figure 13. The only difference is that here sets of symbolic relational probabilistic configurations are considered instead of symbolic (unary) probabilistic configurations.
Synchronizing commands So far all the semantics rules for SRPFOR we presented are uniquely determined by the syntactic construct they are associated to. Also, the rules of the → SRP semantics and, hence, also of the ⇒ srp semantics, don't deal with synchronizing commands. For those we want to be able to apply different rules. In order to consider all the possibilities we define a new judgment with form G G , with G, G ∈ P(P( In Figures 20 and 21, a selected collection of the rules defining the judgment is presented. One of the rules defining this last judgment relies on the previously defined collecting semantics. Indeed, rule Proof-Step-No-Sync applies when no synchronizing commands are involved, and hence there is no possible Fig. 21: SRPFOR: Proof collecting semantics, other rules coupling rule to be applied. Before proceeding with the explanation of the other rules there is the need to explain the variable c which is used in the latter rules. The variable c is a variable that symbolically counts the current level of privacy in the current relational execution. The variable gets increased when the rule Proof-Step-Lap-Gen fires. We have chosen to omit a similar ghost counter variable for δ so rules would be more readable. The variable gets increased also when the rules handling the pairing command x . This latter rule is shown in Appendix. This symbolic counter variable is useful when trying to prove equality of certain variables without spending more than a specific budget. In the set of sets of configurations G, a set of configurations, SR, is nondeterministically chosen. Among elements in SR a configuration is also nondeterministically chosen. Using contexts we check that in the selected configuration the next command to execute is the probabilistic assignment. After reducing to values both the mean and scale expression, and verified (that is, assumed in the set of constraints) that in the two runs the scales have the same value, the rule adds to the set of constraints a new element, that is, where K, K , E are fresh symbols denoting integers and E is the symbolic integer to which the budget variable c maps to. Notice that c needs to point to the same symbol in both memories. This because it's a shared variable tracking the privacy budget spent so far in both runs. This new constraint increases the budget spent. The other constraint added is the real coupling relation, that is X 1 + K = X 2 . Where X 1 , X 2 are fresh in X . Later, K will be existentially quantified in order to search for a proof ofindistinguishability. Rule Proof-Step-Avoc does not use any coupling rule but treats the samples in a purely in a symbolic manner. It intuitively asserts that the two samples are drawn from the distributions and assigns to them arbitrary integers free to vary on the all domain of Thai Laplace distribution. Finally, Proof-Step-No-Coup applies to synchronizing commands as well. It does not add any relational constraints to the samples. This rules intuitively means that we are not correlating in any way the two samples. Notice that since we are not using any coupling rule we don't need to check that the scale value is the same in the two runs as it is requested in the previous rule. We could think of this as a way to encode the relational semantics of the program in an expression which later can be fed in input to other tools. The main difference with the previous rule is that here we treat the sampling instruction symbolically and that's why the fresh symbols are in X p denoting full distributions, while in the previous rule the fresh symbols are in X denoting sampled integers, even though in no particular relation. When the program involves a synchronizing command we basically fork the execution when it's time to execute it. In particular the set of configurations gets to continue the computation in different ways, one for every rule applicable.

Coverage
The coverage lemma can be extended also to the relational setting. To do that though we need to consider only the fragment of the semantics that only uses the rules Proof-Step-No-Sync, and Proof-Step-No-Coupl. We denote this fragment with the notation = ⇒ ∼ . While the semantics that uses only the following rules will be denoted by

Derivations
The language of relational assertions is defined using first order predicate logic formulas involving relational program expressions and logical variables in an unspecified set LogVar. The interpretation of a relational assertions is naturally defined as a subset of M c × M c , that is the set of pairs of memories modelling the assertion. We will let capital greek letters such as Φ, Ψ . . . range of the set of relational assertions. We will also need an additional substituting function · · taking an assertion and a memory in input and returning the assertion where all the program variables have been substituted with the values to which the memory maps variables to. That is, given a memory, relational or unary, m and an assertion, relational or unary, Φ, Φ m is a constraint. More details can be found in [19].
Definition 6. Let Φ, Ψ be relational assertions, c ∈ C r , I : LogVar → R defined on , δ. We say that, Φ yields Ψ through c within and δ under I (and we write , and m I2 are fully symbolic memories, and k = k 1 , k 2 , . . . are the symbols generated by the rules for synchronizing commands.
The idea of this definition is to automatize the proof search. When proving differential privacy we will usually consider Ψ as being equality of the output variables in the two runs and Φ as being our preconditions.

Soundness of proofs and refutations
In this section we will connect the material presented in Section 6 with the one presented in Section 3.

Soundness
Lemma 10. Let c ∈ C r with o its output variable taking values over the set O.
Let c ∈ C r . If I c :

Strategies for counterexample finding
Lemma 11 is hard to use to find counterexamples because given two arbitrary probability distributions computing their -divergence is hard in general. For this reasons we will now describe three strategies that might help in reducing the effort in counterexample finding. This strategies help in isolating traces that could potentially lead to violations. For this we need first some notation. Given a set of constraints s we define the triple Ω = Ω 1 , Ω 2 , C(k) ≡ s 1 , s 2 , s \ ( s 1 ∪ s 2 ) . Given a relational symbolic configuration we can always split its symbolic set of constraints along the Ω pattern. Given a set of constraint s of a relational trace its Ω 1 projection is the set of constraints generated by the branching instruction performed by the left run, simlarly for Ω 2 . Finally, C(k) is the set of relational constraints coming from either preconditions or invariants or, from the rule PROOF-STEP-LAP-GEN. The, potentially empty, vector k = K 1 , . . . K n is the set of fresh symbols K generated by that rule. We sometimes abuse notation and consider Ω also as a set of constraints given by the union of its first, second and third projection. Given a set of constraints we will also consider it as a single proposition given by the conjunction of its elements.

A simplifying assumption on traces and events
In this section we will formalize the main assumption that we will use in order to apply the strategies for counterexample finding presented in the following two subsections.
The idea of this assumption is to consider only programs for which it is necessary, for the output variable on both runs to assume the same values, that the two runs follow the same branches. That is, if the two output variables are different then the two executions must have, at some point, taken different branches.
The following definition will be used to distinguish relational traces which are reachable on one run but not on the other. We call this traces orthogonal.
Definition 7. A final relational symbolic trace is orthogonal when its set of constraints is such that ∃σ.σ |= Ω 2 and σ |= Ω 1 ∧ C(k). That is a trace for which the following formula is satisiable: The next definition, instead, will be used to isolate relational traces for which it's not possible that the left one is executed but the right one is not. We call this traces specular.
Definition 8. A final relational symbolic trace is specular when its set of constraints is such that ∃k.
The constraint Ω 1 ∧ C(k) includes all the constraints coming from the left projection's branching of the symbolic execution and all the relational assumptions such as the adjacency condition, and all constraints added by the potentially fired PROOF-STEP-LAP-GEN rule. A specular trace is such that its left projection constraints plus the relational assumptions imply the right projection constraints. We will now describe three strategies that will be used to isolate relational symbolic traces potentially leading to counterexamples.

Strategy A
In this strategy CRSE uses only the rule PROOF-STEP-AVOC for sampling instructions, also this strategy searches for orthogonal relational traces. Under assumption 1, if this happens for a program then it must be the case that the progam can output one value on one run with some probability but the same value has 0 probability of being output on the second run. This very fact implies that for some input the program has an unbounded privacy loss. To implement this strategy CRSE looks for orthogonal relational traces m 1 , m 2 , skip, p 1 , p 2 , Ω such that: ∃σ.σ |= Ω 1 ∧ C(k) but σ |= Ω 2 . Notice that using this strategy k will always be empty, as the rule used for samplings does not introduce any coupling between the two samples. Hence, we don't need to quantify over those symbols, and we can just write C. The set C though might very well be non empty, because it will potentially include relational assumptions, e.g. adjacency of the inputs.

Strategy B
This strategy symbolically executes the program in order to find a specular trace for which no matter how we relate, within the budget, the various pairs of samples X i 1 , X i 2 in the two runs -using the relational schema X i 1 + K i = X i 2 -the postcondition is always false. That is CRSE looks for specular relational traces m 1 , m 2 , skip, p 1 , p 2 , Ω such that:

Strategy C
This strategy looks for relational traces for which the output variable takes the same value on the two runs but too much of the budget was spent. That is CRSE looks for traces m 1 , m 2 , skip, p 1 , p 2 , Ω such that:

Examples
In this section we will review the examples presented in Section 2 and variations thereof to show how CRSE works.

Unsafe sparse vector implementation: Algorithm 2
In this section we will describe in more detail how CRSE deals with Algorithm 2.
The algorithm is not -differentially private. An easy fix would be to add noise the output too, that is, substitute line 7 with o[i]←lap 2 (q[i](D)), giving us an 2 -differentially private algorithm. Algorithm 2 satisfies assumption 1 because it outputs the whole array o which takes values of the form ⊥ i , t or ⊥ n for 1 ≤ i ≤ n and t ∈ R. The array, hence, encodes the whole trace. So if two runs of the algorithm output the same value it must be the case that they followed they same branching instructions. Let's first notice that the algorithm is trivially differentially private, for any , when the number of iterations n is less than or equal to 4. Indeed it's enough to apply the sequential composition theorem and get the obvious bound 4 · n. CRSE can prove this by applying the rule Proof-Step-Lap-Gen n times, and then choosing K 1 , . . . , K n all equal to 0. This would imply the statement of equality of the output variables spending less than . Hence, it is obvious that if there is a potential counterexample it can only be found after 4 iterations. In fact, a potential counterexample can be found in 5 iterations. If we apply strategy B to this algorithm and follow the relational symbolic trace that applies the rule Proof-Step-Lap-Gen for all the samplings we can isolate the relational specular trace showed in Figure 2, which corresponds to the left execution following the false branch for the first four iterations and then following the true branch and setting the fifth element of the array to the sampled value. Let's denote the respective final relational configuration by m 1 , m 2 , skip, p 1 , p 2 , s . The set of constraints is as follows: , m 1 (t) = T 1 , m 2 (t) = T 2 . We can see that strategy B applies, because holds. The probability associated with these two traces can be expressed as: where j ∈ {1, 2} denotes which run (left or right) we are considering, q is the vector of queries such that | q[i](D 1 ) − q[i](D 2 ) |≤ 1 for adjacent databases D 1 , D 2 and 1 ≤ i ≤ 5. Also, pdflap T 2 (ρ) denotes the probability density at the point ρ of a random variable with Laplace distribution with mean T and scale 2 , and, finally, cdflap . This pair of traces is, in fact, the same that has been found in [28] for a sligthly more general version of Algorithm (2). Strategy B selects this relational trace because, as already noticed in [8] for a different version of the algorithm, in order to make sure that the traces follow the same branches, the coupling rules enforce necessarily that the two samples released are different, preventing the CRSE to prove equality of the output variables in the two runs.

Unsafe sparse vector implementation: Algorithm 3
Algorithm 3 also satisfies assumption 1: that is, the output encodes univocally the whole history of the trace, and hence every trace corresponds injectively to an event. The algorithm is trivially differentially private for one iteration. This because, intuitively, adding noise to the threshold protects the result of the query as well at the branching instruction, but only for one iteration, after that there is no resampling. Indeed, the algorithm is not -differentially private, for any finite already at the second iteration, and a witness for this can be found using CRSE. We can see this using strategy B. Thanks to this strategy we will isolate a relational orthogonal trace, similarly to what has been found in [28] for the same algorithm. CRSE will unfold the loop twice, and it will scan all relational traces to see if there is an orthogonal trace. In particular, the relational trace that corresponds to the output o 1 = o 2 = [⊥, ], that is the the trace with set of constraints Ω 1 , C(k), Ω 2 = Since the vector k is empty we can omit it and just write C. It is easy to see now that the following sigma: σ ≡ [q 1d1 → 0, q 2d1 → 1, q 1d2 → 1, q 2d2 → 0], proves that this relational trace is orthogonal: that is σ |= Ω 1 ∧ C, but σ |= Ω 2 . Indeed if we consider two inputs D 1 , D 2 and two queries q 1 , q 2 such that: q 1 (D 1 ) = q 2 (D 2 ) = 0, q 2 (D 1 ) = q 1 (D 2 ) = 1 we get that the probability of outputting the value o = [⊥, ] is positive in the first run, but it is 0 on the second. This implies that the algorithm can merely be proven to be ∞-differentially private. The former version can be proven 2 -dp, while the latter: -dp. We will explain a proof of this last statement for a constant n, for example 5. The proof presented is based on that in [8], but will have a relational symbolic execution style instead of an apRHL + one. CRSE will try to prove the following postconditions: When trying to prove the i-th one the only interesting iteration will be the i-th one. This because all the others the postcondition will be vacuously true, and also the budget spent will be k 0 2 , the one spent for the threshold, and for all the other sampling instruction we can spend 0 by just setting k j = q[j](D 2 )−q[j](D 1 ) for all j = i, that is by couplin in this way the samples:ŝ 1 + k j =ŝ 2 , with k j = q[j](D 2 ) − q[j](D 1 ), spending |k j + q[j](D 2 ) − q[j](D 1 )| = 0. So, at the i-th iteration the samples are coupledŝ 1 + k i =ŝ 2 , with k i = 1. So ifŝ 1 ≥t 1 then alsoŝ 2 ≥t 2 , and also, ifŝ 1 <t 1 then alsoŝ 2 <t 2 . This implies that at th i-th iteration we enter on the right run the true branch iff we enter the true branch on the left one. This by spending |k i + q[i](D 2 ) − q[i](D 1 )| 4 ≤ 2 4 . For a total of .
Unsafe Laplace mechanism: Algorithm 1 Algorithm 1 is not differentially private for any finite . The intuition is that not enough noise is added to hide the difference of the result of a query applied to two adjacent databases. This translates in any possible potential proof based on the coupling rules in using too much of the budget. The program of Algorithm 1 has only one possible final relational trace: m 1 , m 2 , skip, p 1 , p 2 , Ω 1 , C(k, Ω 2 ) . Since there are no branching instructions Ω 1 = { 2E 1 > 0} and Ω 2 = ∅, where m 1 ( ) = m 2 ( ) = E. Since there is one sampling instruction C(k) will include the following set of constraints {|Q d1 − Q d2 | ≤ 1, Intuitively we can see that, given this set of constraints, if it has to be the case that O 1 = O 2 then, Q d1 − Q d2 = K. But Q d1 − Q d2 can be 1 and hence, E c is at least 2. This tells us that if we want to equate the two output variables we need to spen at least twice the budget. Any relational input satisfying the precondition will give us a counterexample, provided the two projections are different.

Related Works
The closest work to ours is [3] where authors devise a decision logic for differential privacy. The logic allows to soundly prove or disprove and ( , δ) differential privacy by encoding the semantics of the progam into a decidable fragment of the first-order theory of the reals with exponentiation. The programs considered don't allow assignemnts to real and integer variables inside the body of while loops. Other two works very related to ours are [16] and [10]. Their approach to finding counterexamples to differential privacy differs from ours in two main ways. First of all they use a statistical approach by approximating the output distributions of the program on two related inputs and then smartly checking whether for some events these output distributions provide an out of bound ratio. Secondly they are by nature numerical methods providing results stating that an algorithm is not differential private for some actual concrete . These kind of results obviously imply that the algorithm is not differentially private for all other concrete such that > , but they can only suggest that the algorithm is not private for also the , such that < , if that is indeed the case. Our work instead is a purely symbolic technique which provides results stating that an algorithm is not -differentially private for any finite . An advantage of our approach is obviously the speed of the analysis which does not require any sampling. In [27] authors add model checking to the tools for counterexample finding to differential privacy. The main difference with our work is that, as usual, model checking analyzes a model of the code and not directly the code.
Also, in the specific case of the sparse vector algorithm family, their work seem to be able to handle only a finite number of iterations.
This work can be seen as a non trivial probabilistic extension of the framework presented in [19], where sampling instructions in the relational symbolic semantics are handled through an adaptation, in a symbolic execution framework, of the apRHL + rules first presented in [8]. Their logic proves judgments implying differential privacy but does not help in finding counterexamples when the program is not private. This work is also close to [1] where authors devised a framework to automatically discover proofs of privacy using coupling rules, but again thier work does not help in refuting privacy of buggy programs.

Conclusion and Future Work
In this work we presented CRSE: a symbolic execution engine framework which integrates relational reasoning and probabilistic couplings. The framework allows both proving differential privacy of the most known differentially private algorithms, and refutation of buggy versions thereof which are particularly trick to distinguish from the correct ones. While refuting it is also able isolate traces and events which leads to counterexamples to differnetial privacy. When proving CRSE uses a similar approach to apRHL + but follows a strong postcondion approach instead of the more standard weak precondition style of proof, as it is common in symbolic execution style of proofs. CRSE uses refuting principles, or strategies, to isolate potentially dangerous traces. Of the presented strategies only one is sound with respect to counterexample finding, while the other two apply when the algorithm cannot be proven differentially private by any combination of the rules. In this second case though CRSE provides counterexamples which agree with other refutation oriented results in literature. Future work includes interfacing more efficiently CRSE with numeric solvers to find maximums of ratios of probabilities of traces.