1 Introduction

The Volkswagen exhaust emissions scandal [43] has put software doping in the spotlight: Proprietary embedded control software does not always exploit functionality offered by a device in the best interest of the device owner. Instead the software may be tweaked in various manners, driven by interests different from those of the owner or of society. This is indeed a common characteristics for the manner how different manufacturers circumvented [12, 25] the diesel emission regulations around the world. The exhaust software was manufactured in such a way that it heavily polluted the environment, unless the software detected the car to be (likely) fixed on a particular test setup used to determine the \(\mathrm {NO}_x\) footprint data officially published. Phenomena resembling the emission scandal have also been reported in the context of smart phone designs [2], where software was tailored to perform better when detecting it was running a certain benchmark, and otherwise running in lower clock speed. Another smart phone case, disabling the phone [11] via a software update after “non-authorised” repair, has later been undone [36].

Usually, it is the software manufacturer who employs verification or testing to ensure that the software embedded in a device meets its main objectives. However, these days we are confronted with the situation that economical or technological reasons might make a manufacturer become interested in the software slightly deviating from its main objective for dubious reasons. This phenomenon is what we call software doping. It is turning more widespread as software is embedded in ever more devices of daily use.

The simplest and likely most common example of software doping (effectuating a customer lock-in strategy [3]) is that of ink printers [42] refusing to work when supplied with a toner or ink cartridge of a third party manufacturer [41], albeit being technically compatible. Similarly, cases are known where laptops refuse to charge [40] the battery if connected to a third-party charger. More subtle variations of this kind of doping just issue a warning message about the risk of using a “foreign” cartridge [20]. In the same vein, it is known that printers emit “low toner” warnings [33] earlier than needed, so as to drive or force the customer into replacing cartridges prematurely. Moreover, there are allegations that software doping has occurred in the context of electronic-voting so as to manipulate the outcome [1]. Tampering with voting machines has been proved a relatively easy task [21]. Common to all these examples is that the software user has little or no control over its execution, and that the functionality in question is against the interests of user or of society.

Despite the apparently pervasive presence of software doping, a systematic investigation or formalisation from the software engineering perspective is not existing. Fragmentary attention has been payed in the security domain with respect to cryptographic protections being sabotaged by insiders [37]. Typical examples are the many known backdoors, including the prominent dual EC deterministic random bit generator standardised by NIST [14]. Software doping however goes far beyond inclusion of backdoors.

Despite the many examples, it is not at all easy to provide a crisp characterisation of what constitutes software doping. This paper explores this issue, and proposes a hierarchy of formal characterisations of software doping. We aim at formulating and enforcing rigid requirements on embedded software driven by public interest, so as to effectively ban software doping. In order to sharpen our intuition, we offer the following initial characterisation attempt [5].

$$\begin{aligned} \begin{array}{ll} &{}\hbox {A software system is doped if the manufacturer has included a} \\ &{}\hbox {hidden functionality in such a way that the resulting behaviour }\\ &{}\hbox {intentionally favors a designated party, against the interest of }\\ &{}\hbox {society or of the software licensee.} \end{array} \end{aligned}$$
(1)

So, a doped software induces behaviour that can not be justified by the interest of the licensee or of society, but instead serves another usually hidden interest. It thereby favors a certain brand, vendor, manufacturer, or other market participant. This happens intentionally, and not by accident. However, the question whether a certain behaviour is intentional or not is very difficult to decide. To illustrate this, we recall that the above mentioned smart phone case, to be specific the iPhone-6, where “non-authorised” repair rendered the phone unusable [11] after an iOS update, seemed to be intentional when it surfaced, but was actually tracked down to a software glitch of the update and fixed later. Notably, if the iOS designers would have had the particular intention to mistreat licensees who went elsewhere for repair, the same behaviour could well have qualified as software doping in the above sense (1). As a result, we will look at software doping according to the above characterisation, keeping in mind the possibility of intentionality but not aiming to capture it in a precise manner.

In our work, we use concise examples that are directly inspired by the real cases reviewed above. They motivate our hierarchy of formal characterisations of clean or doping-free software.

A core observation will be that software doping can be characterised by considering the program if started from two different but compatible initial states. If the obtained outputs are not compatible, then this implies that the software is doped. Thinking in terms of the printer, one would expect that printing with different but compatible cartridges would yield the same printout without any alteration in the observed alerts. As a consequence, the essence of the property of being clean can be cast as a hyperproperty [16, 17].

We first explore characterisations on sequential software (Sect. 2). We introduce a characterisation that ensures the proper functioning of the system whenever it is confined to standard parameters and inputs. Afterwards, we give two other characterisations that limit the behaviour of the system whenever it goes beyond such standard framework. We then revise these characterisations so as to apply to reactive non-deterministic systems (Sect. 3).

Traditionally hyperproperties require to be analysed in an ad-hoc manner depending on the particular property. However, a general framework is provided by techniques based on, e.g., self-composition techniques [6] or specific logic such as HyperLTL [15]. Indeed, we show (Sect. 4) how these properties can be analysed using self-composition on deterministic programs, particularly using weakest precondition reasoning [18], and we do the same (Sect. 5) for reactive systems using HyperLTL. In both settings we demonstrate principal feasibility by presenting verification studies of simple but representative examples.

2 Software Doping on Sequential Programs

Think of a program as a function that accepts some initial parameters and, given some inputs, produces some outputs, maybe in a non-deterministic manner. Thus, a parameterised sequential non-deterministic program is a function \(S:{\mathsf {Param}}\rightarrow {\mathsf {In}}\rightarrow 2^{\mathsf {Out}}\), where \({\mathsf {Param}}\) is a set of parameters, each one of them fixing a particular instance of the program S, and \({\mathsf {In}}\) and \({\mathsf {Out}}\) being respectively the sets of inputs accepted by S and outputs produced by S. Notice that for a fixed parameter \(\mathsf {p}\) and input \(\mathsf {i}\in {\mathsf {In}}\), the run of program \(S(\mathsf {p})(\mathsf {i})\) may give a set of possible outputs.

Fig. 1.
figure 1

A simple printer.

Full size image
Fig. 2.
figure 2

A doped printer.

Full size image

To understand a first possible definition, consider the program embedded in a printer (a simple abstraction is given in Fig. 1). This program may check compatibility of the ink or toner cartridge and print whenever the cartridge is compatible. In this case, we can think of the program Printer as a function parameterised with the information on the cartridge, that receives a document as input and produces a sequence of pages as outputs whenever the cartridge is compatible, otherwise it turns on an alert led. In this setting, we expect that the printer shows the same input-output behaviour for any compatible cartridge.

A printer manufacturer may manipulate this program in order to favour its own cartridge brand. An obvious way is displayed in Fig. 2. This is a sort of discrimination based on parameter values. Therefore, a first approach to characterising a program as clean (or doping-free) is that it should behave in a similar way for all parameters of interest. By “similar behaviour” we mean that the visible output should be the same for any given input in two different instances of the same (parameterised) program. Also, by “all parameters of interest”, we refer to all parameter values we are interested in. In the case of the printer, we expect that it works with any compatible cartridge, but not with every cartridge. Such a compatibility domain defines a first scope within which a software is evaluated to be clean or doped.

Formally, if \({\mathsf {PIntrs}}\subseteq {\mathsf {Param}}\), we could say that a parameterised program S is clean (or doping-free) if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and input \(\mathsf {i}\in {\mathsf {In}}\), \(S(\mathsf {p})(\mathsf {i}) = S(\mathsf {p}')(\mathsf {i})\). Thus, the program of Fig. 1 satisfies this constraint whenever \(\textsf {Compatible}\) is the set of parameters of interest (i.e. \(\textsf {Compatible}={\mathsf {PIntrs}}\)). Instead, the program of Fig. 2 would be rejected as doped by the previous definition.

We could imagine, nonetheless, that the printer manufacturer may like to provide extra functionalities for its own product which is outside of the standard for compatibility. For instance (and for the sake of this discussion) suppose the printer manufacturer develops a new file format that is more efficient or versatile at the time of printing, but this requires some new technology on the cartridge (we could compare this to the introduction of the postscript language when standard printing was based on dots or ASCII code). The manufacturer still wants to provide the usual functionality for standard file formats that work with standard compatible cartridges and comes up with the program of Fig. 3. Notice that this program does not conform to the specification of a clean program as given above since it behaves differently when a document of the new (non-standard) type is given. This is clearly not in the spirit of the program in Fig. 3 which is actually conforming to the expected requirements.

Fig. 3.
figure 3

A clean printer.

Full size image

Thus, our first definition states that a program is clean if, for any possible instance from the set of parameters of interest, it exhibits the same visible outputs when supplied with the same input, provided this input complies with a given standard. Formally, we assume a set \({\mathsf {PIntrs}}\subseteq {\mathsf {Param}}\) of parameters of interest and a set \({\mathsf {StdIn}}\subseteq {\mathsf {In}}\) of standard inputs and propose the following definition.

Definition 1

A parameterised program S is clean (or doping-free) if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and input \(\mathsf {i}\in {\mathsf {In}}\), if \(\mathsf {i}\in {\mathsf {StdIn}}\) then \(S(\mathsf {p})(\mathsf {i}) = S(\mathsf {p}')(\mathsf {i})\). If the program is not clean we will say that it is doped.

The characterisation given above is based on a comparison of the behaviour of two instances of a program, each of them responding to different parameter values within \({\mathsf {PIntrs}}\). A second, different characterisation may instead require to compare a reference specification capturing the essence of clean behaviour against any possible instance of the program. The first approach seems more general than the second one in the sense that the specification could be considered as one of the possible instances of the (parameterised) program. However, we can consider a distinguished parameter \(\hat{\mathsf {p}}\) so that the instance \(S(\hat{\mathsf {p}})\) is actually the specification of the program, in which case, both definitions turn out to be equivalent. In any case, it is important to observe that the specification may not be available since it is also made by the software manufacturer, and only the expected requirements may be known.

We remark that Definition 1 entails the existence of a contract which defines the set of parameters of interest and the set of standard inputs. In fact, Definition 1 only asserts doping-freedom if the program is well-behaved within such a contract, namely, as long as the parameters are within \({\mathsf {PIntrs}}\) and inputs are within \({\mathsf {StdIn}}\). A behaviour outside this realm is deemed immediately correct since it is of no interest. This view results too mild in some cases where the change of behaviour of a program between a standard input and a non-standard but yet not-so-different input is extreme.

Fig. 4.
figure 4

A simple emission control.

Full size image

Consider the electronic control unit (ECU) of a diesel vehicle, in particular its exhaust emission control module. For diesel engines, the controller injects a certain amount of a specific fluid (an aqueous urea solution) into the exhaust pipeline in order to lower mono-nitrogen oxides (\(\mathrm {NO}_x\)) emissions. We simplify this control problem to a minimal toy example. In Fig. 4 we display a function that reads the \(\textit{throttle}\) position and calculates which is the dose of diesel exhaust fluid (DEF) (stored in \(\textit{def}\)_\(\textit{dose}\)) that should be injected to reduce the \(\mathrm {NO}_x\) emission. The last line of the program precisely models the \(\mathrm {NO}_x\) emission by storing it in the output variable \(\textit{NOx}\) after a (made up) calculation directly depending on the \(\textit{throttle}\) value and inversely depending on the \(\textit{def}\)_\(\textit{dose}\).

Fig. 5.
figure 5

A doped emission control.

Full size image

The Volkswagen emission scandal arose precisely because their software was instrumented so that it works as expected only if operating in or very close to the lab testing conditions [19]. For our simplified example, this behaviour is exemplified by the algorithm of Fig. 5. Of course, the real case was less simplistic. Precisely, in this setting, the lab conditions define the set of standard inputs, i.e., the set \({\mathsf {StdIn}}\) is actually \(\textsf {ThrottleTestValues}\) and, as a consequence, a software like this one trivially meets the characterisation of clean given in Definition 1. However, this unit is intentionally programmed to defy the regulations when being unobserved and hence it falls directly within our intuition of what a doped software is (see (1)).

The spirit of the emission tests is to verify that the amount of \(\mathrm {NO}_x\) in the car exhaust gas does not exceed a given threshold in general. Thus, one would expect that if the input values of the EmissionControl function deviates within “reasonable distance” from the standard input values provided during the lab emission test, the amount of \(\mathrm {NO}_x\) found in the exhaust gas is still within the regulated threshold, or at least it does not exceed it more than a “reasonable amount”. A similar rationale could be applied for regulation of other systems such as speed limit controllers in scooters and electric bikes.

Therefore, we need to introduce two notions of distance \(d_{\mathsf {In}}:({\mathsf {In}}\times {\mathsf {In}})\rightarrow \mathbb {R}_{\ge 0}\) and \(d_{\mathsf {Out}}:({\mathsf {Out}}\times {\mathsf {Out}})\rightarrow \mathbb {R}_{\ge 0}\) on inputs and outputs respectively. In principle, we do not require them to be metrics, but they need to be commutative and satisfy that \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}) = d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}) = 0\) for all \(\mathsf {i}\in {\mathsf {In}}\) and \(\mathsf {o}\in {\mathsf {Out}}\). Since programs are non-deterministic, we need to lift the output distance to sets of outputs and for that we will use the Hausdorff lifting which, as we will see, is exactly what we need. Given a distance d, the Hausdorff lifting \(\mathcal {H}(d)\) is defined by

$$\begin{aligned} \textstyle \mathcal {H}(d)(A,B) = \max \big \{ \sup _{a\in A}\inf _{b\in B} d(a,b), \sup _{b\in B}\inf _{a\in A} d(a,b) \big \} \end{aligned}$$
(2)

Based on this, we provide a new definition that considers two parameters: parameter \(\kappa _\mathsf {i}\) refers to the acceptable distance an input may deviate from the norm to be still considered, and parameter \(\kappa _\mathsf {o}\) that tells how far apart outputs are allowed to be in case their respective inputs are within \(\kappa _\mathsf {i}\) distance.

Definition 2

A parameterised program S is robustly clean if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and inputs \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}\), if \(\mathsf {i}\in {\mathsf {StdIn}}\) is a standard input and \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}')\le \kappa _\mathsf {i}\) then \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}')) \le \kappa _\mathsf {o}\).

Requiring that \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}')) \le \kappa _\mathsf {o}\) is equivalent to demand that

  1. 1.

    for all \(\mathsf {o}\in S(\mathsf {p})(\mathsf {i})\) there exists \(\mathsf {o}'\in S(\mathsf {p}')(\mathsf {i}')\) such that \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}') \le \kappa _\mathsf {o}\), and

  2. 2.

    for all \(\mathsf {o}'\in S(\mathsf {p}')(\mathsf {i}')\) there exists \(\mathsf {o}\in S(\mathsf {p})(\mathsf {i})\) such that \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}') \le \kappa _\mathsf {o}\).

Notice that this is what we actually need for the non-deterministic case: each output of one of the program instances should be matched within “reasonable distance” by some output of the other program instance.

Notice that \(\mathsf {i}'\) does not need to satisfy \({\mathsf {StdIn}}\), but it will be considered as long as it is within \(\kappa _\mathsf {i}\) distance of any input satisfying \({\mathsf {StdIn}}\). In such a case, outputs generated by \(S(\mathsf {p}')(\mathsf {i}')\) will be requested to be within \(\kappa _\mathsf {o}\) distance of some output generated by the respective execution induced by a standard input. In addition, notice that if the program S is deterministic and terminating we could simply write that \(d_{\mathsf {Out}}(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}'))\le \kappa _\mathsf {o}\).

The concept of robustly clean programs generalises that of clean programs. Indeed, by taking \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i})=0\) and \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}')>\kappa _\mathsf {i}\) for all \(\mathsf {i}\ne \mathsf {i}'\), and \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o})=0\) and \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}')>\kappa _\mathsf {o}\) for all \(\mathsf {o}\ne \mathsf {o}'\), we see that Definition 1 is subsumed by Definition 2. Also, notice that the tolerance parameters \(\kappa _\mathsf {i}\) and \(\kappa _\mathsf {o}\) are values that should be provided as well as the notions of distance \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\), and, together with the set \({\mathsf {PIntrs}}\) of parameters of interest and the set \({\mathsf {StdIn}}\) of standard inputs, are part of the contract that ensures that the software is robustly clean. Moreover, the limitation to these tolerance values has to do with the fact that, beyond it, particular requirements (e.g. safety) may arise. For instance, a smart battery may stop accepting charge if the current emitted by a standardised but foreign charger is higher than “reasonable” (i.e. than the tolerance values); however, it may still proceed in case it is dealing with a charger of the same brand for which it may know that it can resort to a customised protocol allowing ultra-fast charging in a safe manner.

Example 3

We remark that Definition 2 will actually detect as doped the program of Fig. 5 for appropriate distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\) and tolerance parameters \(\kappa _\mathsf {i}\) and \(\kappa _\mathsf {o}\). Indeed, suppose that \(\textsc {SCRModel}(x)= x^2\), \(\textsc {altSCRModel}(x)= x\), and \(\textsf {k}=2\). To check if the programs are robustly clean, take \({\mathsf {In}}=(0,2]\) (these are the values that variable \(\textit{throttle}\) takes), \({\mathsf {StdIn}}=(0,1]\), let the distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\) be the absolute values of the differences of the values that take \(\textit{throttle}\) and \(\textit{NOx}\), respectively, and let \(\kappa _\mathsf {i}=2\) and \(\kappa _\mathsf {o}=1\). With this setting, the program of Fig. 4 is robustly clean while the program of Fig. 5 is not.

Definition 2 can be further generalised by adjusting to a precise desired granularity given by a function \(f:\mathbb {R}\rightarrow \mathbb {R}\cup \{\infty \}\) that relates the distances of the input with the distances of the outputs as follows.

Definition 4

A parameterised program S is f-clean if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and inputs \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}\), if \(i\in {\mathsf {StdIn}}\) is a standard input then \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}')) \le f(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}'))\).

Like for Definition 2, the definition of f-clean does not require \(\mathsf {i}'\) to satisfy \({\mathsf {StdIn}}\). Moreover, notice that it is important that f can map into \(\infty \), in which case it means that input \(\mathsf {i}'\) becomes irrelevant to the property. Also here the Hausdorff distance is elegantly encoding the requirement that

  1. 1.

    for all \(\mathsf {o}\in S(\mathsf {p})(\mathsf {i})\) there exists \(\mathsf {o}'\in S(\mathsf {p}')(\mathsf {i}')\) s.t. \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}') \le f(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}'))\), and

  2. 2.

    for all \(\mathsf {o}'\in S(\mathsf {p}')(\mathsf {i}')\) there exists \(\mathsf {o}\in S(\mathsf {p})(\mathsf {i})\) s.t. \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}') \le f(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}'))\).

This definition is strictly more general than Definition 2, which can be seen by taking f defined by \(f(x)=\kappa _\mathsf {o}\) whenever \(x\le \kappa _\mathsf {i}\) and \(f(x)=\infty \) otherwise. (Notice here the use of \(\infty \).) Also, if the program S is deterministic, we could simply require that \(d_{\mathsf {Out}}(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}'))\le f(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}'))\).

In this new definition, the bounding function f, together with the distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\), the set \({\mathsf {PIntrs}}\) of parameters of interest and the set \({\mathsf {StdIn}}\) of standard inputs, are part of the contract that ensures that the software is f-clean.

Example 5

For the example of the emission control take the setting as in Example 3 and let \(f(x)=x/2\). Then the program of Fig. 4 is f-clean while the program of Fig. 5 is not.

We remark that the notion of f-clean strictly relates the distance of the input values with the distance of the output values. Thus, e.g., the accepted distance on the outputs may grow according the distance of the input grows. Compare it to the notion of robustly clean in which the accepted distance on the outputs is only bounded by a constant (\(\kappa _\mathsf {o}\)), regardless of the proximity of the inputs (which is only observed w.r.t. to constant \(\kappa _\mathsf {i}\)).

3 Software Doping on Reactive Programs

Though we use the Volkswagen ECU case study as motivation for introducing Definitions 2 and 4, this program is inherently reactive: the DEF dosage depends not only of the current inputs but also on the current state (which in turn is set according to previous inputs). Therefore, in this section, we revise the definitions given in the previous section within the framework of reactive programs.

We consider a parameterised reactive program as a function \(S:{\mathsf {Param}}\rightarrow {\mathsf {In}}^\omega \rightarrow 2^{({\mathsf {Out}}^\omega )}\) so that any instance of the program reacts to the k-th input in the input sequence producing the k-th output in each respective output sequence. Thus each instance of the program can be seen, for instance, as a (non-deterministic) Mealy or Moore machine. In this setting, we require that \({\mathsf {StdIn}}\subseteq {\mathsf {In}}^\omega \). Thus, the definition of a clean reactive program strongly resembles Definition 1.

Definition 6

A parameterised reactive program S is clean if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and input \(\mathsf {i}\in {\mathsf {In}}^\omega \), if \(\mathsf {i}\in {\mathsf {StdIn}}\) then \(S(\mathsf {p})(\mathsf {i}) = S(\mathsf {p}')(\mathsf {i})\).

Naively, we may think that the definition of robustly clean may be also reused as given in Definition 2 by considering metrics on \(\omega \)-traces. Unfortunately this definition does not work as expected: suppose two input sequences in \({\mathsf {In}}^\omega \) that only differ by a single input in some late k-th position but originates a distance larger than \(\kappa _\mathsf {i}\). Now the program under study may become clean even if the respective outputs differ enormously at an early \(k'\)-th position (\(k'<k\)). Notice that there is no justification for such early difference on the output, since the input sequences are the same up to position \(k'\).

In fact, we notice that the property of being clean is of a safety nature: if there is a point in a pair of executions in which the program is detected to be doped, there is no extension of such executions that can correct it and make the program clean. In the observation above, the \(k'\)-th prefix of the trace should be considered the bad prefix and the program deemed as doped.

Therefore, we consider distances on finite traces: \(d_{\mathsf {In}}:({\mathsf {In}}^*\times {\mathsf {In}}^*)\rightarrow \mathbb {R}_{\ge 0}\) and \(d_{\mathsf {Out}}:({\mathsf {Out}}^*\times {\mathsf {Out}}^*)\rightarrow \mathbb {R}_{\ge 0}\). Now, we provide a definition of robustly clean on reactive programs that ensures that, as long as all j-th prefix of a given input sequence, with \(j\le k\), are within \(\kappa _\mathsf {i}\) distance, the k-th prefix of the output sequence are within \(\kappa _\mathsf {o}\) distance, for any \(k\ge 0\). In the following definition, we denote with \(\mathsf {i}[..k]\) the k-th prefix of the input sequence \(\mathsf {i}\) (and similarly for output sequences).

Definition 7

A parameterised reactive program S is robustly clean if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and input sequences \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}^\omega \), if \(\mathsf {i}\in {\mathsf {StdIn}}\) then, for all \(k\ge 0\) the following must hold

$$ \left( \forall {j\le k} : d_{\mathsf {In}}(\mathsf {i}[..j],\mathsf {i}'[..j])\le \kappa _\mathsf {i}\right) \rightarrow {\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i})[..k],S(\mathsf {p}')(\mathsf {i}')[..k]) \le \kappa _\mathsf {o}}, $$

where \(S(\mathsf {p})(\mathsf {i})[..k]=\{\mathsf {o}[..k] \mid \mathsf {o}\in S(\mathsf {p})(\mathsf {i})\}\) and similarly for \(S(\mathsf {p}')(\mathsf {i}')[..k]\).

By having as precondition that \(d_{\mathsf {In}}(\mathsf {i}[..j],\mathsf {i}'[..j])\le \kappa _\mathsf {i}\) for all \(j\le k\), this definition considers the fact that once one instance of the program deviates too much from the normal behaviour (i.e. beyond \(\kappa _\mathsf {i}\) distance at the input), this instance is not obliged any longer to meet (within \(\kappa _\mathsf {o}\) distance) the output, even if later inputs get closer again. This enables robustly clean programs to stop if an input outside the standard domain may result harmful for the system. Also, notice that, by considering the conditions through all k-th prefixes the definition encompasses the safety nature of the robustly cleanness property.

Example 8

A slightly more realistic version of the emission control system on the ECU is given in Fig. 6. It is a closed loop where the calculation of the DEF dosage also depends on the previous reading of \(\mathrm {NO}_x\). Moreover, the DEF dosage does not affect deterministically in the \(\mathrm {NO}_x\) emission. Instead, there is a margin of error on the \(\mathrm {NO}_x\) emission which is represented by the factor \(\lambda \) and the non-deterministic assignment of variable \(\textit{NOx}\) in the penultimate line within the loop.

Fig. 6.
figure 6

An emission control (reactive).

Full size image

This non-deterministic assignment is an (admittedly unrealistic) abstraction of the chemical reaction between the exhaust gases and the DEF dosage. Figure 7 gives the version of the emission control system instrumenting the cheating hack. We define the selective catalytic reduction (SCR) models as follows:

$$ \textsc {SCRModel}(x,n)= {\left\{ \begin{array}{ll} x^2 &{} \text {if } \textsf {k}\cdot n\le x \\ (1+\lambda )\cdot x^2 &{} \text {otherwise} \\ \end{array}\right. } $$

where \(\lambda =0.1\) and \(\textsf {k}=2\), and \(\textsc {altSCRModel}(x,n)= x\) (i.e., it ignores the feedback of the \(\mathrm {NO}_x\) emission resulting in the same \(\textsc {altSCRModel}\) as in Example 3). We also take \({\mathsf {In}}=(0,2]\) (recall that these are the values that variable \(\textit{throttle}\) takes). The idea of the feedback in \(\textsc {SCRModel}\) is that if the previous emission was higher than expected with the planned current dosage, then the actual current dosage is an extra \(\lambda \) portion above the planned dosage.

Fig. 7.
figure 7

A doped emission control (reactive).

Full size image

For the contract required by robustly cleanness, we let \({\mathsf {StdIn}}=(0,1]^\omega \) and define \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}')=|{\mathop {\mathrm {last}}(\mathsf {i})-\mathop {\mathrm {last}}(\mathsf {i}')} |\) and similarly \(d_{\mathsf {Out}}(\mathsf {o},\mathsf {o}')=|{\mathop {\mathrm {last}}(\mathsf {o})-\mathop {\mathrm {last}}(\mathsf {o}')} |\), where \(\mathop {\mathrm {last}}(t)\) is the last element of the finite trace t. We take \(\kappa _\mathsf {i}=2\) and \(\kappa _\mathsf {o}=1.1\). (\(\kappa _\mathsf {o}\) needs to be a little larger than in Example 3 due to the non-deterministic assignment to \(\textit{NOx}\).)

In Sect. 6 we will use a model checking tool to prove that the algorithm in Fig. 6 is robustly clean, while the algorithm of Fig. 7 is not.

As before, Definition 7 can be further generalised by adjusting to a precise desired granularity given by a function \(f:\mathbb {R}\rightarrow \mathbb {R}\cup \{\infty \}\) that relates the distances of the input with the distances of the outputs as follows.

Definition 9

A parameterised reactive program S is f-clean if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and input sequences \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}^\omega \), if \(\mathsf {i}\in {\mathsf {StdIn}}\) then for all \(k\ge 0\), \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i})[..k],S(\mathsf {p}')(\mathsf {i}')[..k]) \le f(d_{\mathsf {In}}(\mathsf {i}[..k],\mathsf {i}'[..k]))\).

Like for Definition 7, the definition of f-cleanness also considers distance on prefixes to ensure that major differences in late inputs do not impact on differences of early outputs, capturing also the safety nature of the property.

We observe that Definition 9 is more general than Definition 7. As before, define f by \(f(x)=\kappa _\mathsf {o}\) whenever \(x\le 1\) and \(f(x)=\infty \) otherwise, but also redefine the metric on the input domain as follows:

$$ d_{\mathsf {In}}^{\text {new}}(\mathsf {i}[..k],\mathsf {i}'[..k])= {\left\{ \begin{array}{ll} 0 &{} \text {if } \mathsf {i}[..k]=\mathsf {i}'[..k] \\ 1 &{} \text {if either } \mathsf {i}\in {\mathsf {StdIn}}\text { or } \mathsf {i}'\in {\mathsf {StdIn}}, \mathsf {i}[..k]\ne \mathsf {i}'[..k] \\ &{} \text {and } d_{\mathsf {In}}(\mathsf {i}[..j],\mathsf {i}'[..j])\le \kappa _\mathsf {i}\text { for all } 0\le j\le k \\ 2 &{} \text {otherwise} \end{array}\right. } $$

for all \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}\) and \(k\ge 0\).

Example 10

For the example of the emission control take the setting as in Example 8 and let \(f(x)=x/2+0.3\). The variation of f w.r.t. Example 5 is necessary to cope with the non-determinism introduced in these models. With this setting, in Sect. 6 we will check that the program of Fig. 6 is f-clean while the program of Fig. 7 is not.

4 Analysis Through Self-composition

In this section we will focus on sequential deterministic programs and we will see them in the usual way: as state transformers. Thus, if \(\mu ,\mu ':\mathsf {Var}\rightarrow \mathsf {Val}\) are states mapping the variables of a program into values within their domain, we denote with \((S,\mu )\Downarrow \mu '\) that a program S, initially taking values according to \(\mu \), executes and terminates in state \(\mu '\). We indicate with \((S,\mu )\Downarrow \bot \) that the program S starting at state \(\mu \) does not terminate. As usual, we denote by \(\mu \models \phi \) that a predicate \(\phi \) holds on a state \(\mu \).

In this new setting, and restricting to deterministic programs, Definition 1 could be alternatively formulated as in Proposition 11. For this, we will assume that S contains sets of variables \(\vec {x}_\mathsf {p}\), \(\vec {x}_\mathsf {i}\), and \(\vec {x}_\mathsf {o}\) which are respectively parameter variables, input variables and output variables. Moreover, let \({\mathsf {PIntrs}}\) and \({\mathsf {StdIn}}\) be predicates on states containing only program variables in \(\vec {x}_\mathsf {p}\) and \(\vec {x}_\mathsf {i}\), respectively. They characterise the set of parameters of interest and the set of standard inputs. Now, we can state,

Proposition 11

A sequential and deterministic program S is clean if and only if for all states \(\mu _1\), \(\mu _2\) and \(\mu '_1\) such that \(\mu _1\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\), \(\mu _2\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\), \(\mu _1(\vec {x}_\mathsf {i})=\mu _2(\vec {x}_\mathsf {i})\) and \((S,\mu _1)\Downarrow \mu '_1\), it holds that \((S,\mu _2)\Downarrow \mu '_2\) and \(\mu '_1(\vec {x}_\mathsf {o})=\mu '_2(\vec {x}_\mathsf {o})\) for some \(\mu '_2\).

The proof of the proposition is straightforward since it is basically a notation change, hence we omit it. Also, notice that we omit any explicit reference to non-terminating programs. This is not necessary due to the symmetric nature of the predicates.

In the nomenclature of [7] relations

are called indistinguishable criteriaFootnote 1, and if \((\mu _1,\mu _2)\in \mathcal {I}\) then we say that \(\mu _1\) and \(\mu _2\) are \(\mathcal {I}\)-indistinguishableFootnote 2. Similarly, for \(\mathcal {I}'\). Thus, Proposition 11 characterises what in [7] is called termination-sensitive \((\mathcal {I},\mathcal {I}')\)-security and, by [7, Proposition 3], the property of cleanness can be analysed using the weakest (conservative) precondition (\(\mathop {\mathrm {wp}}\)) [18] through self-composition.

Proposition 12

Let \([\vec {x}/\vec {x}']\) indicate the substitution of each variable x by variable \(x'\). Then a deterministic program S is clean if and only if

$$\begin{aligned} \left( \begin{array}{l} {({\mathsf {PIntrs}}\wedge {\mathsf {StdIn}})} \wedge {({\mathsf {PIntrs}}\wedge {\mathsf {StdIn}})[\vec {x}/\vec {x}']} \\ {} \wedge {\vec {x}_\mathsf {i}=\vec {x}'_\mathsf {i}} \wedge {\mathop {\mathrm {wp}}(S,\textsf {true})} \end{array}\right) \ \Rightarrow \ \mathop {\mathrm {wp}}(S ; S[\vec {x}/\vec {x}'],\vec {x}_\mathsf {o}=\vec {x}'_\mathsf {o}). \end{aligned}$$

The term \(\mathop {\mathrm {wp}}(S,\textsf {true})\) in the antecedent of the implication is the weakest precondition that ensures that program S terminates. It is necessary in the predicate, otherwise it could become false only because program S does not terminate.

With the same setting as before, and taking \(d_{\mathsf {In}}\), \(d_{\mathsf {Out}}\), \(\kappa _\mathsf {i}\) and \(\kappa _\mathsf {o}\) as for Definition 2, we obtain an alternative definition of robustly cleanness for deterministic programs.

Proposition 13

A sequential and deterministic program S is robustly clean if and only if for all states \(\mu _1\), \(\mu _2\), and \(\mu '\) such that \(\mu _1\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\), \(\mu _2\models {\mathsf {PIntrs}}\), and \(d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i}))\le \kappa _\mathsf {i}\), the following two conditions hold:

  1. 1.

    if \((S,\mu _1)\Downarrow \mu '\), then \((S,\mu _2)\Downarrow \mu '_2\) and \(d_{\mathsf {Out}}(\mu '(\vec {x}_\mathsf {o}),\mu '_2(\vec {x}_\mathsf {o}))\le \kappa _\mathsf {o}\) for some \(\mu '_2\); and

  2. 2.

    if \((S,\mu _2)\Downarrow \mu '\), then \((S,\mu _1)\Downarrow \mu '_1\) and \(d_{\mathsf {Out}}(\mu '_1(\vec {x}_\mathsf {o}),\mu '(\vec {x}_\mathsf {o}))\le \kappa _\mathsf {o}\) for some \(\mu '_1\).

In this case, the indistinguishability criteria are

$$\begin{aligned} \!\mathcal {I}&=\{(\mu _1,\mu _2)\mid {\mu _1\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}, \mu _2\models {\mathsf {PIntrs}}, \text { and } d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i}))\le \kappa _\mathsf {i}}\} \\ \!\!\!\mathcal {I}'&=\{(\mu _1,\mu _2)\mid {d_{\mathsf {Out}}(\mu _1(\vec {x}_\mathsf {o}),\mu _2(\vec {x}_\mathsf {o}))\le \kappa _\mathsf {o}}\} \end{aligned}$$

Notice that \(\mathcal {I}\) is not symmetric. Then the first item of Proposition 13 characterises termination-sensitive \((\mathcal {I},\mathcal {I}')\)-security while the second item characterises termination-sensitive \((\mathcal {I}^{-1},\mathcal {I}')\)-security. Using again [7, Proposition 3], the property of robustly cleanness can be analysed using \(\mathop {\mathrm {wp}}\) through self-composition.

Proposition 14

A deterministic program S is robustly clean if and only if

$$\begin{aligned}&{\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\wedge {{\mathsf {PIntrs}}[\vec {x}/\vec {x}']} \wedge {d_{\mathsf {In}}(\vec {x}_\mathsf {i},\vec {x}'_\mathsf {i})\le \kappa _\mathsf {i}} \\&\qquad \Rightarrow \ \left( \begin{array}{ll} &{} \mathop {\mathrm {wp}}(S,\textsf {true}) \Rightarrow \mathop {\mathrm {wp}}(S ; S[\vec {x}/\vec {x}'], d_{\mathsf {Out}}(\vec {x}_\mathsf {o},\vec {x}'_\mathsf {o})\le \kappa _\mathsf {o}) \\ \wedge \ \ &{} \mathop {\mathrm {wp}}(S[\vec {x}/\vec {x}'],\textsf {true}) \Rightarrow \mathop {\mathrm {wp}}(S[\vec {x}/\vec {x}'] ; S, d_{\mathsf {Out}}(\vec {x}_\mathsf {o},\vec {x}'_\mathsf {o})\le \kappa _\mathsf {o}) \end{array}\right) \end{aligned}$$

Proceeding in a similar manner, we can also obtain an alternative definition of f-cleanness for deterministic programs.

Proposition 15

A sequential and deterministic program S is f-clean if and only if for all states \(\mu _1\), \(\mu _2\), and \(\mu '\) such that \(\mu _1\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\), and \(\mu _2\models {\mathsf {PIntrs}}\), the following two conditions hold:

  1. 1.

    if \((S,\mu _1){\Downarrow }\mu '\), then \((S,\mu _2){\Downarrow }\mu '_2\) and \(d_{\mathsf {Out}}(\mu '(\vec {x}_\mathsf {o}),\mu '_2(\vec {x}_\mathsf {o}))\le f(d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i}))\) for some \(\mu '_2\); and

  2. 2.

    if \((S,\mu _2){\Downarrow }\mu '\), then \((S,\mu _1){\Downarrow }\mu '_1\) and \(d_{\mathsf {Out}}(\mu '_1(\vec {x}_\mathsf {o}),\mu '(\vec {x}_\mathsf {o}))\le f(d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i}))\) for some \(\mu '_1\).

Notice that the term \(f(d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i}))\) appears in the conclusion of the implications of both items. This may look unexpected since it seems to be related to the input requirements rather than the output requirements, in particular because it refers to the input states. This makes this case a little less obvious than the previous one. To overcome this situation, we introduce a constant \(Y\in \mathbb {R}_{\ge 0}\) which we assume universally quantified. Using this, we define the following indistinguishability criteria

$$\begin{aligned}&\mathcal {I}_Y =\{(\mu _1,\mu _2)\mid {\mu _1\models {\mathsf {PIntrs}}\wedge {\mathsf {StdIn}},}\\&\quad \quad \quad \quad \quad \quad \quad \mu _2\models {\mathsf {PIntrs}}, \text { and } f(d_{\mathsf {In}}(\mu _1(\vec {x}_\mathsf {i}),\mu _2(\vec {x}_\mathsf {i})))=Y\}\qquad \\&\mathcal {I}'_Y =\{(\mu _1,\mu _2)\mid {d_{\mathsf {Out}}(\mu _1(\vec {x}_\mathsf {o}),\mu _2(\vec {x}_\mathsf {o}))\le Y}\} \end{aligned}$$

By using this, by Proposition 15, we have that S is f-clean if and only if for every \(Y\in \mathbb {R}_{\ge 0}\), and for all states \(\mu _1\), \(\mu _2\), and \(\mu '\) such that \((\mu _1,\mu _2)\in \mathcal {I}_Y\)

  1. 1.

    if \((S,\mu _1)\Downarrow \mu '\), then \((S,\mu _2)\Downarrow \mu '_2\) and \((\mu ',\mu '_2)\in \mathcal {I}'_Y\) for some \(\mu '_2\); and

  2. 2.

    if \((S,\mu _2)\Downarrow \mu '\), then \((S,\mu _1)\Downarrow \mu '_1\) and \((\mu '_1,\mu ')\in \mathcal {I}'_Y\) for some \(\mu '_1\).

With this new definition, and taking into account again the asymmetry of \(\mathcal {I}_Y\), the first item characterises termination-sensitive \((\mathcal {I}_Y,\mathcal {I}'_Y)\)-security while the second one characterises termination-sensitive \((\mathcal {I}_Y^{-1},\mathcal {I}'_Y)\)-security. From this and [7, Prop. 3], the property of f-cleanness can be analysed using \(\mathop {\mathrm {wp}}\) and self-composition.

Proposition 16

A deterministic program S is f-clean if and only if for all \(Y\in \mathbb {R}_{\ge 0}\)

$$\begin{aligned}&{\mathsf {PIntrs}}\wedge {\mathsf {StdIn}}\wedge {{\mathsf {PIntrs}}[\vec {x}/\vec {x}']} \wedge {f(d_\mathsf {i}(\vec {x}_\mathsf {i},\vec {x}'_\mathsf {i}))=Y} \\&\qquad \Rightarrow \ \left( \begin{array}{ll} &{} \mathop {\mathrm {wp}}(S,\textsf {true}) \Rightarrow \mathop {\mathrm {wp}}(S ; S[\vec {x}/\vec {x}'], d_{\mathsf {Out}}(\vec {x}_\mathsf {o},\vec {x}'_\mathsf {o})\le Y \\ \wedge \ \ &{} \mathop {\mathrm {wp}}(S[\vec {x}/\vec {x}'],\textsf {true}) \Rightarrow \mathop {\mathrm {wp}}(S[\vec {x}/\vec {x}'] ; S, d_{\mathsf {Out}}(\vec {x}_\mathsf {o},\vec {x}'_\mathsf {o})\le Y \end{array}\right) \end{aligned}$$
Fig. 8.
figure 8

Equations for the \(\mathop {\mathrm {wp}}\) calculus

Full size image

Example 17

In this example, we use Proposition 16 to prove correct our statements in Example 3. First, we recall the definition of \(\mathop {\mathrm {wp}}\) in Fig. 8, and rewrite the programs in Figs. 4 and 5 with all functions and values properly instantiated in the way we need it here (see Figs. 9 and 10).

Fig. 9.
figure 9

Program EC.

Full size image
Fig. 10.
figure 10

Program AEC.

Full size image

On the one hand, none of the programs have parameters, then \({\mathsf {PIntrs}}= \textsf {true}\). On the other hand, \({\mathsf {StdIn}}= (\textit{thrtl}\in (0,1])\). Since \(\mathop {\mathrm {wp}}(\textsc {ec},\textsf {true})=\textsf {true}\) we have to prove that

$$\begin{aligned}&{\textit{thrtl}\in (0,1]} \wedge {\left( {\textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'} |}{2}}=Y\right) }\nonumber \\&\qquad \Rightarrow \left( \begin{array}{ll} &{} \mathop {\mathrm {wp}}(\textsc {ec};\textsc {ec}',|{\textit{NOx}-\textit{NOx}\,'} |\le Y)\\ \wedge &{} \mathop {\mathrm {wp}}(\textsc {ec}';\textsc {ec},|{\textit{NOx}-\textit{NOx}\,'} |\le Y) \end{array} \right) \end{aligned}$$
(3)

where \(\textsc {ec}'\) is another instance of ec with every program variable x renamed by \(x'\). Moreover, function f and distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\) are already instantiated. It is not difficult to verify that \(\mathop {\mathrm {wp}}(\textsc {ec};\textsc {ec}',{|{\textit{NOx}{-}\textit{NOx}\,'} |}\le Y) \equiv \left( \frac{|{\textit{thrtl}{-}\textit{thrtl}\,'} |}{2}\le Y\right) \) and \(\mathop {\mathrm {wp}}(\textsc {ec}';\textsc {ec},{|{\textit{NOx}{-}\textit{NOx}\,'} |}\le Y) \equiv \left( \frac{|{\textit{thrtl}\,'{-}\textit{thrtl}} |}{2}\le Y\right) \) from which the implication follows and hence ec is f-clean.

For \(\textsc {aec}\) we also have that \(\mathop {\mathrm {wp}}(\textsc {aec},\textsf {true})=\textsf {true}\) and hence we have to prove a formula similar to 3. In this case, \(\mathop {\mathrm {wp}}(\textsc {aec};\textsc {aec}',{|{\textit{NOx}-\textit{NOx}\,'} |}\le Y)\) is

$$\begin{aligned}&\ ({\textit{thrtl}\in (0,1]} \wedge {\textit{thrtl}\,'\in (0,1]}) \ \Rightarrow \ \textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'} |}{2}\le Y \quad \\ \wedge&\ ({\textit{thrtl}\in (0,1]} \wedge {\textit{thrtl}\,'\notin (0,1]}) \ \Rightarrow \ \textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'^2} |}{2}\le Y \quad \\ \wedge&\ ({\textit{thrtl}\notin (0,1]} \wedge {\textit{thrtl}\,'\in (0,1]}) \ \Rightarrow \ \textstyle \frac{|{\textit{thrtl}^2-\textit{thrtl}\,'} |}{2}\le Y \\ \wedge&\ ({\textit{thrtl}\notin (0,1]} \wedge {\textit{thrtl}\,'\notin (0,1]}) \ \Rightarrow \ \textstyle \frac{|{\textit{thrtl}^2-\textit{thrtl}\,'^2} |}{2}\le Y \end{aligned}$$

The predicate is the same for \(\mathop {\mathrm {wp}}(\textsc {aec}';\textsc {aec},{|{\textit{NOx}-\textit{NOx}\,'} |}\le Y)\), since \(|{a-b} |=|{b-a} |\). Then, the predicate

$$\begin{aligned} \!\!\!\left( {\textit{thrtl}\in (0,1]} \wedge {{\textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'} |}{2}}=Y}\right) \Rightarrow \left( \begin{array}{ll} &{} \mathop {\mathrm {wp}}(\textsc {aec};\textsc {aec}',|{\textit{NOx}-\textit{NOx}\,'} |\le Y)\\ \wedge &{} \mathop {\mathrm {wp}}(\textsc {aec}';\textsc {aec},|{\textit{NOx}-\textit{NOx}\,'} |\le Y) \end{array} \right) \end{aligned}$$

is equivalent to

$$\begin{aligned} \!\!\!\left( {\textit{thrtl}\in (0,1]} \wedge {{\textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'} |}{2}}=Y}\right) \Rightarrow \left( \begin{array}{ll} &{} {\textit{thrtl}\,'\in (0,1]} \Rightarrow \textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'} |}{2}\le Y \\ \wedge \ &{} {\textit{thrtl}\,'\notin (0,1]} \Rightarrow \textstyle \frac{|{\textit{thrtl}-\textit{thrtl}\,'^2} |}{2}\le Y \end{array} \right) \end{aligned}$$

which can be proved false if, e.g., \(\textit{thrtl}=1\) and \(\textit{thrtl}\,'=1.5\).

Notwithstanding the simplicity of the previous example, the technique can be applied to complex programs including loops. We decided to keep it simple as it is not our intention to show the power of \(\mathop {\mathrm {wp}}\), but the applicability of our definition.

We could profit from [7] for the use of other verification techniques, including separation logic and model checking where the properties can be expressed in terms of LTL and CTL. Particularly, CTL permits the encoding of the full non-deterministic properties given in Sect. 2. We will not dwell on this since in the next section we explore the encoding of the reactive properties through a more general setting.

5 Analysis of Reactive Programs with HyperLTL

HyperLTL [15] is a temporal logic for the specification of hyperproperties of reactive systems. HyperLTL extends linear-time temporal logic (LTL) with trace quantifiers and trace variables, which allow the logic to refer to multiple traces at the same time. The problem of model checking a HyperLTL formula over a finite-state model is decidable [24]. In this section, we focus on reactive non-deterministic programs and use HyperLTL to encode the different definitions of clean reactive programs given in Sect. 3. In the following, we interpret a program as a set \(S\subseteq (2^{\mathsf {AP}})^\omega \) of infinite traces over a set \(\mathsf {AP}\) of atomic propositions.

Let \(\pi \) be a trace variable from a set \(\mathcal {V}\) of trace variables. A HyperLTL formula is defined by the following grammar:

(4)

The quantifiers \(\exists \) and \(\forall \) quantify existentially and universally, respectively, over the set of traces. For example, the formula means that for every trace \(\pi \) there exists another trace \(\pi '\) such that \(\phi \) holds over the pair of traces. If no universal quantifier occurs in the scope of an existential quantifier, and no existential quantifiers occurs in the scope of a universal quantifier, we call the formula alternation-free. In order to refer to the values of the atomic propositions in the different traces, the atomic propositions are indexed with trace variables: for some atomic proposition \(a\in \mathsf {AP}\) and some trace variable \(\pi \in \mathcal {V}\), \(a_\pi \) states that a holds in the initial position of trace \(\pi \). The temporal operators and Boolean connectives are interpreted as usual. In particular, \(\mathop {\mathsf {X}}\phi \) means that \(\phi \) holds in the next state of every trace under consideration. Likewise, \(\phi \mathbin {\mathsf {U}}\phi '\) means that \(\phi '\) eventually holds in every trace under consideration at the same point in time, provided \(\phi \) holds in every previous instant in all such traces. We also use the standard derived operators: , , and \({\phi \mathbin {\mathsf {W}}\phi '}\equiv {\lnot (\lnot \phi '\mathbin {\mathsf {U}}(\lnot \phi \wedge \lnot \phi '))}\).

A trace assignment is a partial function \(\varPi :\mathcal {V}\rightarrow (2^\mathsf {AP})^\omega \) that assigns traces to variables. Let \(\varPi [\pi \mapsto t]\) denote the same function as \(\varPi \) except that \(\pi \) is mapped to the trace \(t\). For \(k\in \mathbb {N}\), let t[k], t[k..], and t[..k] denote respectively the k-th element of t, the k-th suffix of t, and the k-th prefix of t. The trace assignment suffix \(\varPi [k..]\) is defined by \(\varPi [k..](\pi )=\varPi (\pi )[k..]\). By \(\varPi \mathrel {\models _{S}}\psi \) we mean that formula \(\phi \) is satisfied by the program S under the trace assignment \(\varPi \). Satisfaction is recursively defined as follows.

We say that a program S satisfies a HyperLTL formula \(\psi \) if it is satisfied under the empty trace assignment, that is, if \(\varnothing \mathrel {\models _{S}}\psi \).

In the following, we give the different characterisations of cleanness for reactive programs in terms of HyperLTL. For this, let \(\mathsf {AP}= {\mathsf {AP}_\mathsf {p}\cup \mathsf {AP}_\mathsf {i}\cup \mathsf {AP}_\mathsf {o}}\) where \(\mathsf {AP}_\mathsf {p}\), \(\mathsf {AP}_\mathsf {i}\), and \(\mathsf {AP}_\mathsf {o}\) are the atomic propositions that define the parameter values, the input values, and the output values respectively. Thus, we take \({\mathsf {Param}}=2^{\mathsf {AP}_\mathsf {p}}\), \({\mathsf {In}}=2^{\mathsf {AP}_\mathsf {i}}\) and \({\mathsf {Out}}=2^{\mathsf {AP}_\mathsf {o}}\). Therefore, a program \(S\subseteq (2^\mathsf {AP})^\omega \) can be seen as a function \(\hat{S}:{\mathsf {Param}}\rightarrow {\mathsf {In}}^\omega \rightarrow 2^{({\mathsf {Out}}^\omega )}\) where

$$\begin{aligned} t\in S \quad \text { if and only if } \quad (t\downarrow \mathsf {AP}_\mathsf {o})\ \in \ \hat{S}(t[0]\cap \mathsf {AP}_\mathsf {p})(t\downarrow \mathsf {AP}_\mathsf {i}), \end{aligned}$$
(5)

with \(t\downarrow A\) defined by \((t\downarrow A)[k]=t[k]\cap A\) for all \(k\in \mathbb {N}\).

For the propositions appearing in the rest of this sections, we will assume that distances between traces are defined only according to its last element. That is, for the distance \(d_{\mathsf {In}}:({\mathsf {In}}^*\times {\mathsf {In}}^*)\rightarrow \mathbb {R}_{\ge 0}\) there exists a distance \(\hat{d}_{\mathsf {In}}:({\mathsf {In}}\times {\mathsf {In}})\rightarrow \mathbb {R}_{\ge 0}\) such that \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}')=\hat{d}_{\mathsf {In}}(\mathop {\mathrm {last}}(\mathsf {i}),\mathop {\mathrm {last}}(\mathsf {i}'))\) for every \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}^*\), and similarly for \(d_{\mathsf {Out}}:({\mathsf {Out}}^*\times {\mathsf {Out}}^*)\rightarrow \mathbb {R}_{\ge 0}\). Let us call these type of distances past-forgetful. Moreover, we will need the abbreviations given in Table 1 for a clear presentation of the formulas.

Table 1. Syntactic sugar for comparisons between traces
Full size table

The set of parameters of interest \({\mathsf {PIntrs}}\subseteq {\mathsf {Param}}\) defines a Boolean formula which we ambiguously call \({\mathsf {PIntrs}}\). Also, we let \({\mathsf {StdIn}}\) be an LTL formula with atomic propositions in \(\mathsf {AP}_\mathsf {i}\), that is, a formula obtained with the grammar in the second line of (4) where atomic propositions have the form \(a\in \mathsf {AP}_\mathsf {i}\) (instead of \(a_\pi \)). Thus \({\mathsf {StdIn}}\) characterises the set of all input sequences through an LTL formula. With \({\mathsf {StdIn}}_\pi \) we represent the HyperLTL formula that is exactly like \({\mathsf {StdIn}}\) but where each occurrence of \(a\in \mathsf {AP}_\mathsf {i}\) has been replaced by \(a_\pi \). Likewise, we let \({\mathsf {PIntrs}}_\pi \) represent the Boolean formula that is exactly like \({\mathsf {PIntrs}}\) with each occurrence of \(a\in \mathsf {AP}_\mathsf {p}\) replaced by \(a_\pi \). We are now in conditions to state the characterisation of a clean program in terms of HyperLTL.

Proposition 18

A reactive program S is clean if and only if it satisfies the HyperLTL formula

(6)

As it is given, the formula actually states that

$$\begin{aligned} \forall \mathsf {p}_1 : \forall \mathsf {p}_2 : \forall \mathsf {i}: \mathsf {p}_1,\mathsf {p}_2\in {\mathsf {PIntrs}}\wedge \mathsf {i}\in {\mathsf {StdIn}}: \hat{S}(\mathsf {p}_1)(\mathsf {i}) \subseteq \hat{S}(\mathsf {p}_2)(\mathsf {i}) \end{aligned}$$

Because of the symmetry of this definition (namely, interchanging \(\mathsf {p}_1\) and \(\mathsf {p}_2\)), this is indeed equivalent to Definition 6. Notice that in (6), \(\pi _2\) quantifies universally the parameter of the second instance, while \(\pi '_2\) represents the existence of the output sequence in such instance. The proofs of Propositions 18 to 20 follow the same structures. So we only provide the proof of Proposition 19 which is the most involved.

In fact, Proposition 19 below states the characterisation of a robustly clean program in terms of two HyperLTL formulas (or as a single HyperLTL formula by taking the conjunction).

Proposition 19

A reactive program S is robustly clean under past-forgetful distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\) if and only if S satisfies the following two HyperLTL formulas

(7)

The difference between the first and second formula is subtle, but reflects the fact that, while the first formula has the universal quantification on the outputs of the program that takes standard input and the existential quantification on the program that may deviate, the second one works in the other way around. Thus each of the formulas capture each of the \(\sup \)-\(\inf \) terms in the definition of Hausdorff distance (see (2)). To notice this, follow the existentially quantified variable (\(\pi '_2\) for the first formula, and \(\pi '_1\) for the second one). Also, the weak until operator \(\mathbin {\mathsf {W}}\) has exactly the behaviour that we need to represent the interaction between the distances of inputs and the distances of outputs. The semantics of \(\phi \mathbin {\mathsf {W}}\psi \) is defined by

$$\begin{aligned} t\models \phi \mathbin {\mathsf {W}}\psi \, \text {iff}\quad \forall k\ge 0: (\forall j\le k: t[j..]\models \lnot \psi ) \rightarrow t[k..]\models \phi \end{aligned}$$
(8)

Next, we prove Proposition 19.

Proof

We only prove that the first formula captures the bound on the left sup-inf term of the definition of Hausdorff distance (see eq.  (2)) in Definition 7. The other condition is proved in the same way and corresponds to the other \(\sup \)-\(\inf \) term of the Hausdorff distance. Taking into account the semantics of the weak until operator given in Eq. 8, the semantics of HyperLTL in general and using abbreviations in Table 1, formula 7 is equivalent to the following statement

$$\begin{aligned}&\forall t_1\in S : \forall t_2\in S : \exists t'_2\in S : \\&\ (t_1\models {\mathsf {PIntrs}}\wedge t_2\models {\mathsf {PIntrs}}\wedge t_1\models {\mathsf {StdIn}})\\&\rightarrow \Big ( {(t_2[0]\cap \mathsf {AP}_\mathsf {p})=(t'_2[0]\cap \mathsf {AP}_\mathsf {p})} \wedge (\forall j\ge 0 : t_2[j]\cap \mathsf {AP}_\mathsf {i}= t'_2[j]\cap \mathsf {AP}_\mathsf {i}) \\&\quad \quad \,\,\, \wedge \forall k\ge 0: (\forall j\le k:{\hat{d}_{\mathsf {In}}(t_1[j]\cap \mathsf {AP}_\mathsf {i},t'_2[j]\cap \mathsf {AP}_\mathsf {i})}\le \kappa _\mathsf {i}) \phantom {\ }\\&\quad \quad \quad \quad \quad \quad \quad \quad \rightarrow {\hat{d}_{\mathsf {Out}}(t_1[k]\cap \mathsf {AP}_\mathsf {o},t'_2[k]\cap \mathsf {AP}_\mathsf {o})}\le \kappa _\mathsf {o}\Big ) \end{aligned}$$

By applying some definitions and notation changes, this is equivalent to

which, by logic manipulation, is equivalent to

$$\begin{aligned}&\forall \mathsf {p}_1 : \forall \mathsf {p}_2 : \forall \mathsf {i}_1 : \forall \mathsf {i}_2 : \forall \mathsf {o}_1 : \\&\Big (\exists t_1\in S : \exists t_2\in S : \\&\qquad \qquad (\mathsf {p}_1=(t_1[0]\cap \mathsf {AP}_\mathsf {p})\in {\mathsf {PIntrs}}) \wedge (\mathsf {p}_2=(t_2[0]\cap \mathsf {AP}_\mathsf {p})\in {\mathsf {PIntrs}}) \\&\qquad \qquad {} \wedge \mathsf {i}_1 = (t_1\downarrow \mathsf {AP}_\mathsf {i}) \wedge \mathsf {i}_2 = (t_2\downarrow \mathsf {AP}_\mathsf {i}) \wedge \mathsf {o}_1 = (t_1\downarrow \mathsf {AP}_\mathsf {o}) \wedge \mathsf {i}_1 \in {\mathsf {StdIn}}\Big )\\&\ {} \rightarrow \exists \mathsf {o}_2 : \exists t'_2\in S : \\&\qquad \qquad \Big ( {\mathsf {p}_2=(t'_2[0]\cap \mathsf {AP}_\mathsf {p})} \wedge {\mathsf {i}_2 = (t'_2\downarrow \mathsf {AP}_\mathsf {i})} \wedge {\mathsf {o}_2=(t'_2\downarrow \mathsf {AP}_\mathsf {o})} \\&\qquad \qquad {} \wedge \forall k\ge 0: (\forall j\le k:{\hat{d}_{\mathsf {In}}(\mathsf {i}_1[j],\mathsf {i}_2[j])}\le \kappa _\mathsf {i}) \rightarrow {\hat{d}_{\mathsf {Out}}(\mathsf {o}_1[k],\mathsf {o}_2[k])}\le \kappa _\mathsf {o}\Big ) \end{aligned}$$

By (5) and the fact that distances are past-forgetful, the previous equation is equivalent to

$$\begin{aligned}&\forall \mathsf {p}_1 : \forall \mathsf {p}_2 : \forall \mathsf {i}_1 : \forall \mathsf {i}_2 : \forall \mathsf {o}_1 : \\&\Big ( \mathsf {p}_1,\mathsf {p}_2\in {\mathsf {PIntrs}}\wedge \mathsf {i}_1 \in {\mathsf {StdIn}}\wedge \forall k\ge 0: (\forall j\le k:{d_{\mathsf {In}}(\mathsf {i}_1[..j],\mathsf {i}_2[..j])}\le \kappa _\mathsf {i}) \\&\qquad \qquad \ {} \wedge \mathsf {o}_1\in \hat{S}(\mathsf {p}_1)(\mathsf {i}_1) \Big ) \rightarrow \big (\exists \mathsf {o}_2\in \hat{S}(\mathsf {p}_2)(\mathsf {i}_2): {d_{\mathsf {Out}}(\mathsf {o}_1[..k],\mathsf {o}_2[..k])}\le \kappa _\mathsf {o}\big ) \end{aligned}$$

which in turn corresponds to bounding the left \(\sup \)-\(\inf \) term of the Hausdorff distance (see  (2)) in Definition 7,

$$\begin{aligned}&\forall \mathsf {p}_1 : \forall \mathsf {p}_2 : \forall \mathsf {i}_1 : \forall \mathsf {i}_2 : \\&\big ( \mathsf {p}_1,\mathsf {p}_2\in {\mathsf {PIntrs}}\wedge \mathsf {i}_1 \in {\mathsf {StdIn}}\wedge \forall k\ge 0: (\forall j\le k:{d_{\mathsf {In}}(\mathsf {i}_1[..j],\mathsf {i}_2[..j])}\le \kappa _\mathsf {i})\big ) \\&\qquad \qquad \qquad \quad \ {} \rightarrow \textstyle \big (\sup _{\mathsf {o}_1\in \hat{S}(\mathsf {p}_1)(\mathsf {i}_1)} \inf _{\mathsf {o}_2\in \hat{S}(\mathsf {p}_2)(\mathsf {i}_2)} {d_{\mathsf {Out}}(\mathsf {o}_1[..k],\mathsf {o}_2[..k])}\big )\le \kappa _\mathsf {o}\end{aligned}$$

thus proving this part of the proposition.    \(\square \)

Finally, we also give the characterisation of an f-clean program in terms of HyperLTL.

Proposition 20

A reactive program S is f-clean under past-forgetful distances \(d_{\mathsf {In}}\) and \(d_{\mathsf {Out}}\) if and only if S satisfies the following two HyperLTL formulas

(9)

As before, the difference between the first and second formula is subtle and can be noticed again by following the existentially quantified variables in each of the formulas.

We remark that the HyperLTL characterisations presented in Propositions 19 and 20 can be extended to any distance of bounded memory, that is, distances such that \(d(t,t')=d(t[k..],t'[k..])\) for every finite traces t and \(t'\) and a fixed bound \(k\in \mathbb {N}\). The solution proceeds by basically using the same formulas on an expanded and annotated model (with the expected exponential blow up w.r.t. to the original one).

Example 21

In our running example of the emission control system (see Examples 8 and 10), the property of robustly cleanness reduces to checking formula

(10)

and the obvious symmetric formula. For readability reasons, we shorthandedly write \(\textit{t}\) for \(\textit{thrtl}\) and \(\textit{n}\) for \(\textit{NOx}\). Notice that any reference to parameters disappears since the emission control system does not have parameters, and the set of standard inputs is characterised by the LTL formula . Likewise, we can verify that the model of the emission control system is f-clean through the formula

(11)

and the symmetric formula.

6 Experimental Results

We verified the cleanness of the emission control system using the HyperLTL model checker MCHyper [24]. The input to the model checker is a description of the system as an Aiger circuit and a hyperproperty specified as an alternation-free HyperLTL formula. Since the HyperLTL formulas from the previous section are of the form \(\forall \pi _1 \forall \pi _2 \exists \pi _2' \ldots \), and are, hence, not alternation-free, MCHyper cannot check these formulas directly. However, it is possible to prove or disprove such formulas by strengthening the formulas and their negations manually into alternation-free formulas that are accepted by MCHyper.

In order to prove that program ec in Fig. 9 is robustly clean, we strengthen formula (10) by substituting \(\pi _2\) for the existentially quantified variable \(\pi _2'\). The resulting formula is alternation-free:

(12)

MCHyper confirms that program ec satisfies (12). The program thus also satisfies (10). Notice that we had obtained the same formula if we would have started from the formula symmetric to (10).

To prove that program aec in Fig. 10 is doped with respect to (10), we negate (10) and obtain

This formula is of the form and, hence, again not alternation-free. We replace the two existential quantifiers with universal quantifiers and restrict the quantification to two specific throttle values, a for \(\pi _1\) and b for \(\pi _2\):

(13a)

This transformation is sound as long as there actually exist traces with throttle values a and b. We establish this by checking, separately, that the following existential formula is satisfied:

(14)

MCHyper confirms the satisfaction of both formulas, which proves that (10) is violated by program aec. Precisely, the counterexample that shows the violation of (10) is any pair of traces \(\pi _1\) and \(\pi _2\) that makes true in (14). We proceed similarly for the formula symmetric to (10) obtaining two formulas just as before which are also satisfied by aec and hence the original formula is not. Also, we follow a similar process to prove that ec is f-clean but aec is not.

Table 2. Experimental results from the verification of robust cleanness of ec and aec
Full size table

Table 2 shows experimental results obtained with MCHyperFootnote 3 version 0.91 for the verification of robustly cleanness. The Aiger models were constructed by discretizing the values of the throttle and the \(\mathrm {NO}_x\). We show results from two different models, where the values of the throttle was discretised in steps of 0.1 units in both models and the values of the \(\mathrm {NO}_x\) in steps of 0.05 and 0.00625. All experiments were run under OS X “El Capitan” (10.11.6) on a MacBook Air with a 1.7GHz Intel Core i5 and 4GB 1333MHz DDR3. In Table 2, the model size is given in terms of the number of transitions, while the size of the Aiger circuit encoding the model prepared for the property is given in terms of the number of latches and gates. The specification checked by MCHyper is the formula indicated in the property column. Formula  (13b) is the formula symmetric to (13a). For the throttle values a and b in formulas (13a) and (13b), we chose \(b=2\) and let a vary as specified in the property column. Table 3 shows similar experimental results for the verification of f-cleanness. With (12\('\)), (13a\('\)), and (13b\('\)) we indicate the similar variations to (12), (13a), and (13b) required to verify (11). Model checking takes less than two seconds for the coarse discretisation and about two minutes for the fine discretisation.

Table 3. Experimental results from the verification of f-cleanness of ec and aec
Full size table

7 A Comprehensive Characterisation

If we concretely focus on the contract between the society or the licensee, and the software manufacturer, we can think in a more general but precise definition. It emerges by noticing that there is a partition on the set of inputs in three sets, each one of them fulfilling a different role within the contract:

  1. 1.

    The set \({\mathsf {StdIn}}\) of standard inputs. For these inputs, the program is expected to work exactly as regulated. It is the case, e.g., of the inputs defining the tests for the \(\mathrm {NO}_x\) emission. Thus, it is expected that the program complies to Definition 1 when provided only with inputs in \({\mathsf {StdIn}}\).

  2. 2.

    The set \({\mathsf {Comm}}\) of committed inputs such that \({\mathsf {Comm}}\cap {\mathsf {StdIn}}=\varnothing \). These inputs are expected to be close according to a distance to \({\mathsf {StdIn}}\) and are not strictly regulated. However, it is expected that the manufacturer commits to respect certain bounds on the outputs. This would correspond to the inputs that do not behave exactly like the tests for the \(\mathrm {NO}_x\) emission, but yet define “reasonable behaviour” of the car on the road. The behaviour of the program under this set of inputs can be characterised either by Definition 2 or Definition 4.

  3. 3.

    All other inputs are supposed to be anomalous and expected to be significantly distant from the standard inputs. In our emission control example, this can occur, e.g., if the car is climbing a steep mountain or speeding up in a highway. In this realm the only expectation is that the behaviour of the output is continuous with respect to the input.

Bearing this partition in mind, we propose the following general definition.

Definition 22

A parameterised program S is clean (or doping-free) if for all pairs of parameters of interest \(\mathsf {p},\mathsf {p}'\in {\mathsf {PIntrs}}\) and inputs \(\mathsf {i},\mathsf {i}'\in {\mathsf {In}}\),

  1. 1.

    if \(\mathsf {i}\in {\mathsf {StdIn}}\) then \(S(\mathsf {p})(\mathsf {i}) = S(\mathsf {p}')(\mathsf {i})\);

  2. 2.

    if \(\mathsf {i}\in {\mathsf {StdIn}}\) and \(\mathsf {i}'\in {\mathsf {Comm}}\) then \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}')) \le f(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}'))\).

  3. 3.

    for every \(\epsilon >0\) there exists \(\delta >0\) such that for all \(\mathsf {i}'\notin {\mathsf {StdIn}}\cup {\mathsf {Comm}}\) and \(\mathsf {i}\in {\mathsf {In}}\), \(d_{\mathsf {In}}(\mathsf {i},\mathsf {i}')<\delta \) implies \(\mathcal {H}(d_{\mathsf {Out}})(S(\mathsf {p})(\mathsf {i}),S(\mathsf {p}')(\mathsf {i}'))<\epsilon \).

Notice that, while \({\mathsf {PIntrs}}\), \({\mathsf {StdIn}}\), \({\mathsf {Comm}}\), \(d_{\mathsf {In}}\), \(d_{\mathsf {Out}}\), and f are part of the contract entailed by the definition, \(\epsilon \) and \(\delta \) in item 3 are not since they are quantified (universally and existentially, resp.) in the definition. In this case, we choose for item 3 to require that the program S is uniformly continuous in \({\mathsf {In}}\setminus ({\mathsf {StdIn}}\cup {\mathsf {Comm}})\). However, we could have opted for stronger requirements such as Lipschitz continuity. The chosen type of continuity would also be part of the contract. Notice that this is the only case in which we require continuity. Instead, discontinuities are allowed in cases 1 and 2 as long as the conditions are respected since they may be part of the specification. In particular, notice that f could be any function. Obviously, a similar definition can be obtained for reactive systems.

We remark that cases 1 and 2 can be verified, as we showed in the paper. We have not yet explored the verification of case 3.

8 Related Work

The term “software doping” has being coined by the press about a year ago and, after the Volkswagen exhaust emissions scandal, the elephant in the room became unavoidable: software developers introduce code intended to deceive [28]. Recently, a special session at ISOLA 2016 was devoted to this topic [34]. In [9], Baum attacks the problem from a philosophical point of view and elaborates on the ethics of it. In [5], we provided a first discussion of the problem and some informal characterisations hinting at the formal proposal of this paper. Though all these works point out the need for a technical attack on the problem, none of them provide a formal proposal.

Similar to software doping, backdoored software is a class of software that does not act in the best interest of users; see for instance the recent analysis in [37]. The primary emphasis of backdoored software is on leaking confidential information while guaranteeing functionality.

Dope-freedom in sequential programs is strongly related to abstract non-intereference [6, 26] as already disussed in Sect. 4. More generally, our notions of dope-freedom are hyperproperties [16], a general class that encompasses notions across different domains, in particular non-interference in security [39], robustness (a.k.a.  stability) in cyber-physical systems [13], and truthfulness in algorithmic game theory [8]. There exist several methods for verifying hyperproperties, including relational and Cartesian Hoare logics [10, 38, 44], self-composition and product programs constructions [4, 7], temporal logics [15, 23, 24], or games [35]. These techniques greatly vary in their completeness, efficiency, and scalability.

Another worthwhile direction to study is the use of program equivalence analysis [22, 27] for the analysis of cleanness.

9 Concluding Remarks

This article has focused on a serious and yet long overlooked problem, arising if software developers intentionally and silently deviate from the intended objective of the developed software. A notorious reason behind such deviations are simple and blunt lock-in strategies, so as to bind the software licensee to a certain product or product family. However, the motivations can be more diverse and obscure. As the software manufacturer has full control over the development process, the deviation can be subtle and surreptitiously introduced in a way that the fact that the program does not quite conform to the expected requirements may go well unnoticed.

We have pioneered the formalisation of this problem domain by offering several formal characterisations of software doping. These can serve as a framework for establishing a contract between the interested parties, namely the society or the licensee, and the software manufacturer, so as to avoid and eventually ban the development of doped programs.

We have also reported on the use of existing theories and tools at hand to demonstrate that the formal characterisation can indeed be analysed in various ways. In particular, the application of the self-composition technique opens many research directions for further analysis of software doping as it has been widely studied in the area of security [29, 31], semantical differences [32] and cross or relative verification [30].

As we have demonstrated, the use of HyperLTL enables the automatic analysis of reactive models with respect to software doping. However, the complexity of this technique imposes some serious limits on its applicability. Thus, further studies in this direction are needed in order to enable analysis of reactive models of relatively large size, or alternatively to analyse the program code directly.

We believe our characterisations provide a first solid step to understand software doping and that our result opens a large umbrella of new possibilities, both in the direction of more dedicated characterisations as well as specifically tailored analysis techniques. For instance, the idea of dealing with distances and thresholds already rises the question of whether such distances could be quantified by probabilities. Also, the \(\mathrm {NO}_x\) emission example would immediately suggest that the technique should also be addressed with testing. Moreover, the fact that the characterisations are hyperproperties also invites us to investigate for static analysis of source code based on type systems, abstraction techniques, etc.