Optimising Spectrum Based Fault Localisation for Single Fault Programs Using Specifications.

. Spectrum based fault localisation determines how suspicious a line of code is with respect to being faulty as a function of a given test suite. Outstanding problems include identifying properties that the test suite should satisfy in order to improve fault localisation eﬀectiveness subject to a given measure, and developing methods that generate these test suites eﬃciently. We address these problems as follows. First, when single bug optimal measures are being used with a single-fault program, we identify a formal property that the test suite should satisfy in order to optimise fault localisation. Second, we introduce a new method which generates test data that satisﬁes this property. Finally, we empirically demonstrate the utility of our implementation at fault localisation on sv-comp benchmarks and the tcas program, demonstrating that test suites can be generated in almost a second with a fault identiﬁed after inspecting under 1% of the program.


Introduction
Faulty software is estimated to cost 60 billion dollars to the US economy per year [1] and has been single-handedly responsible for major newsworthy catastrophes 1 .This problem is exacerbated by the fact that debugging (defined as the process of finding and rectifying a fault) is complex and time consumingestimated to consume 50-60% of the time a programmer spends in the maintenance and development cycle [2].Consequently, the development of effective and efficient methods for software fault localisation has the potential to greatly reduce costs, wasted programmer time and the possibility of catastrophe.
In this paper, we advance the state of the art in lightweight fault localisation by building on research in spectrum-based fault localisation (sbfl).sbfl is one of the most prominent areas of software fault localisation research, estimated to make up 35% of published work in the field to date [3], and has been demonstrated to be efficient and effective at finding faults [4][5][6][7][8][9][10][11][12].The effectiveness relies on two factors, (1) the quality of the measure used to identify the lines of code that are suspected to be faulty, and (2) the quality of the test suite used.Most research in the field has been focussed on finding improved measures [4][5][6][7][8][9][10][11][12], but there is a growing literature on how to improve the quality of test suites [13][14][15][16][17][18][19][20].An outstanding problem in this field is to identify the properties that test suites should satisfy to improve fault localisation.
To address this problem, we focus our attention on improving the quality of test suites for the purposes of fault localisation on single-fault programs.Programs with a single fault are of special interest, as a recent study demonstrates that 82% of faulty programs could be repaired with a "single fix" [21], and that "when software is being developed, bugs arise one-at-a-time and therefore can be considered as single-faulted scenarios", suggesting that methods optimised for use with single-fault programs would be most helpful in practice.Accordingly, the contributions of this paper are as follows.
1. We identify a formal property that a test suite must satisfy in order to be optimal for fault localisation on a single-fault program when a single-fault optimal sbfl measure is being used.2. We provide a novel algorithm which generates data that is formally shown to satisfy this property.3. We integrate this algorithm into an implementation which leverages model checkers to generate small test suites, and empirically demonstrate its practical utility at fault localisation on our benchmarks.
The rest of this paper is organized as follows.In Sect.2, we present the formal preliminaries for sbfl and our approach.In Sect.3, we motivate and describe a property of single-fault optimality.In Sect.4, we present an algorithm which generates data for a given faulty program, and prove that the data generated satisfies the property of single fault optimality, and in Sect. 5 discuss implementation details.In Sect.6 we present our experimental results where we demonstrate the utility of an implementation of our algorithm on our benchmarks, and in Sect.7 we present related work.

Preliminaries
In this section we formally present the preliminaries for understanding our fault localisation approach.In particular, we describe probands, proband models, and sbfl.

Probands
Following the terminology in Steimann et al. [22], a proband is a faulty program together with its test suite, and can be used for evaluating the performance of a given fault localization method.A faulty program is a program that fails to always satisfy a specification, which is a property expressible in some formal language and describes the intended behaviour of some part of the program under test (put).When a specification fails to be satisfied for a given execution (i.e., an error occurs), it is assumed there exists some (incorrectly written) lines of code in the program which was the cause of the error, identified as a fault (aka bug).Example 1.An example of a faulty c program is given in Fig. 1 (minmax.c,taken from Groce et al. [23]), and we shall use it as our running example throughout this paper.There are some executions of the program in which the assertion statement least <= most is violated, and thus the program fails to always satisfy the specification.The fault in this example is labelled C4, which should be an assignment to least instead of most.
A test suite is a collection of test cases whose result is independent of the order of their execution, where a test case is an execution of some part of a program.Each test case is associated with an input vector, where the n-th value of the vector is assigned to the n-th input of the given program for the purposes of a test (according to some given method of assigning values in the vector to inputs in the program).Each test suite is associated with a set of input vectors which can be used to generate the test cases.A test case fails (or is failing) if it violates a given specification, and passes (or is passing) otherwise.
Example 2. We give an example of a test case for the running example.The test case with associated input vector 0, 1, 2 is an execution in which input1 is assigned 0, input2 is assigned 1, and input3 is assigned 2, the statements labeled C1, C2 and C3 are executed, but C4 and C5 are not executed, and the assertion is not violated at termination, as least and most assume values of 0 and 2 respectively.Accordingly, we may associate a collection of test cases (a test suite) with a set of input vectors.For the running example the following ten input vectors are associated with a test suite of ten test cases: 1, 0, 2 , 2, 0, 1 , 2, 0, 2 , 0, 1, 0 , 0, 0, 1 , 1, 1, 0 , 2, 0, 0 , 2, 2, 2 , 1, 2, 0 , and 0, 1, 2 .Here, the first three input vectors result in error (and thus their associated test cases are failing), and the last seven do not (and thus their associated test cases are passing).
A unit under test uut is a concrete artifact in a program which is a candidate for being at fault.Many types of uuts have been defined and used in the literature, including methods [24], blocks [25,26], branches [16], and statements [27][28][29].A uut is said to be covered by a test case just in case that test case executes the uut.For convenience, it will help to always think of uuts as being labeled C1, C2, ... etc. in the program itself (as they are in the running example).Assertion statements are not considered to be uuts, and we assume that each fault in the program has a corresponding uut.
Example 3. To illustrate some uuts for the running example (Fig. 1), we have chosen the units under test to be the statements labeled in comments marked C1, . . ., C5.The assertion is labeled E, which is violated when an error occurs.To illustrate a proband, the faulty program minmax.c(described in Example 1), and the test suite associated with the input vectors described in Example 2, together form a proband.

Proband Models
In this section we define proband models, which are the principle formal objects used in sbfl.Informally, a proband model is a mathematical abstraction of a proband.We assume the existence of a given proband in which the uuts have already been identified for the faulty program and appropriately labeled C1, . . ., Cn, and assume a total of n uuts.We begin as follows.We also call a set of coverage vectors T the fault localisation data or a dataset.Intuitively, each coverage vector can be thought of as a mathematical abstraction of an associated test case which describes which uuts were executed/covered in that test case.We also use the following additional notation.If the last argument of a coverage vector in T is the number k it is denoted t k where k uniquely identifies a coverage vector in T and the corresponding test case in the associated test suite.In general, for each t k ∈ T, c k i is a coverage variable and gives the value of the i-th argument in t k .If c k n+1 = 1, then t k is called a failing coverage vector, and passing otherwise.The set of failing coverage vectors/the event of an error is denoted E (such that the set of passing vectors is then E).Element c k n+1 is also denoted e k (as it describes whether the error occurred).For convenience, we may represent the set of coverage vectors T with a coverage matrix, where for all 0 < i n and t k ∈ T the cell intersecting the i-th column and k-th row is c k i and represents whether the i-th uut was covered in the test case corresponding to t k .The cell intersecting the last column and k-th row is e k and represents whether t k is a failing or passing test case.Fig. 2 is an example coverage matrix.In practice, given a program and an input vector, one can extract coverage information from an associated test case using established tools2 .

Definition 2.
Let T be a non-empty set of coverage vectors, then T's program model PM is defined as the sequence C 1 , . . ., C |PM| , where for each We often use the notation PM T to denote the program model PM associated with T. The final component C |PM| is also denoted E (denoting the event of the error).Each member of a program model is called a program component or event, and if c k i = 1 we say C i occurred in t k , that t k covers C i , and say that C i is faulty just in case its corresponding uut is faulty.Following the definition above, each component C i is the set of vectors in which C i is covered, and obey set theoretic relationships.For instance, for all components In general, we assume that E contains at least one coverage vector and each coverage vector covers at least one component.Members of E and E are called failing/passing vectors, respectively.
Example 5. We use the running example to illustrate a program model.For the set of coverage vectors T = {t 1 , . . ., t 10 }, we may define a program model Here, we may think of C 1 , . . ., C 5 as events which occur just in case a corresponding uut (lines of code labeled C1, . . .,C5 respectively) is executed, and E as an event which occurs just in case the assertion least <= most is violated.C 4 is identified as the faulty component.Finally, we extend our setup to distinguish between samples and populations.The population test suite for a given program is a test suite consisting of all possible test cases for the program, a sample test suite is a test suite consisting of some (but not necessarily all) possible test cases for the program.All test suites are sample test suites drawn from a given population.Let PM, T be a given proband model for a given faulty program and sample test suite, we denote the population vectors, corresponding to the population test suite of the given faulty program, as T * (and E * and E * as the population failing and passing vectors in T * respectively).The population program model associated with the population test suite is denoted PM * (aka PM * T * ).PM * , T * is called the population proband model.Finally, we extend the use of asterisks to make clear that the asterisked variable is associated with a given population.Accordingly, each component in the population program model is also superscripted with a * to denote that it is a member of PM * (e.g.C * 1 ).Each vector in the population set of vectors T * (e.g., t * 1 ), and each coverage variable in each vector t * k ∈ T * (e.g., c k * 1 ).It is assumed that for a given sample proband model PM, T and its population proband model PM * , T * , we have T ⊆ T * .Intuitively, this is because a sample test suite is drawn from the population.In addition, for each i ∈ N if Intuitively, this is because if the i-th uut is executed by a test case in the sample then it is executed by that test case in the population.

Spectrum Based Fault Localisation
We first define what a program spectrum is, as it serves as the principle formal object used in spectrum based fault localization (sbfl).The higher/lower the degree of suspiciousness the more/less suspicious C i is assumed to be with respect to being a fault.A property of some sbfl measures is single-fault optimality [7,30].Using our notation we can express this property as follows: Definition 6.A suspiciousness measure w is single-fault optimal if it satisfies the following.For every program model PM and every C i ∈ PM:

Definition 4. For each proband model PM, T , and each component
Under the assumption that there is a single fault in the program, Naish et al. argue that a measure must have this property to be optimal [7].Informally, the first condition demands that uuts covered by all failing test cases are more suspicious than anything else.The rationale here is that if there is only one faulty uut in the program, then it must be executed by all failing test cases (otherwise there would be some failing test case which executes no fault -which is impossible given it is assumed that all errors are caused by the execution of some faulty uut) [7,30].The second demands that of two uuts covered by all failing test cases, the one which is executed by fewer passing test cases is more suspicious.
An example of a single fault optimal measure is the Naish-I measure w(C i ) = [31].A framework that optimises any given sbfl measure to being single fault optimal was first given by Naish [31].For any suspiciousness measure w scaled from 0 to 1, we can construct the single fault optimised version for w (written Opt w ) as follows (here, we use the equivalent formulation of Landsberg et al. [4]): and w(C i ) otherwise.We now describe the established sbfl algorithm [4,[7][8][9][10][11][12].The method produces a list of program component indices ordered by suspiciousness, as a function of set of coverage vectors T (taken from a proband model PM, T ) and suspiciousness measure w.As the algorithm is simple, we informally describe the algorithm in three stages, as follows.First, the program spectrum for each program component is constructed as a function of T. Second, the indices of program components are ordered in a suspiciousness list according to decreasing order of suspiciousness.Third, the suspiciousness list is returned to the user, who will inspect each uut corresponding to each index in the suspiciousness list in decreasing order of suspiciousness until a fault is found.We assume that in the case of ties of suspiciousness, the uut that comes earlier in the code is investigated first, and assume effectiveness of a sbfl measure on a proband is measured by the number of non-faulty uuts a user has to investigate before a fault is found.
Example 7. We illustrate an instance of sbfl using our running minmax.cexample of Fig. 1, and the Naish-I measure as an example suspiciousness measure.First, the program spectra (given in Example 6) are constructed as a function of the given coverage vectors (represented by the coverage matrix of Fig. 2).Second, the suspiciousness of each program component is computed (here, the suspiciousness of the five components are 2.125, −0.375, 0.75, 2.875, 0.625 respectively), and the indices of components are ordered according to decreasing order of suspiciousness.Thus we get the list 4, 1, 3, 5, 2 .Finally, the list is returned to the user, and the uuts in the program are inspected according to this list in descending order of suspiciousness until a fault is found.In our running example, C4 (the fault) is investigated first.

A Property of Single-Fault Optimal Data
In this section, we identify a new property for the optimality of a given dataset T for use in fault localisation.Throughout we make two assumptions: Firstly that a single bug optimal measure w is being used and secondly that there is a single bug in a given faulty program (henceforth our two assumptions).Let PM, T be a given sample proband model, then we have the following: If this condition holds, then we say the dataset T (and its associated test suite) satisfies this property of single fault optimality.Informally, the condition demands that if a uut is covered by all failing test cases in the sample test suite then it is covered by all failing test cases in the population.If our two assumptions hold, we argue it is a desirable that a test suite satisfies this property.This is because the fault is assumed to be covered by all failing test cases in the population (similar to the rationale of Naish et al. [7]), and as uuts executed by all failing test cases in the sample are investigated first when a single fault optimal measure is being used, it is desirable that uuts not covered by all failing test cases in the population are less suspicious in order to guarantee the fault is found earlier.An additional desirable feature of knowing one's data satisfies this property, is that we do not have to add any more failing test cases to a test suite, given it is then impossible to improve fault localization effectiveness by adding more failing test cases under our two assumptions.

Algorithm
In this section we present an algorithm which outputs single fault optimal data for a given faulty program.We assume several preconditions for our algorithm.
-For the given faulty program, at least one uut is executed by all failing test cases (for C programs this could be a variable initialization in the main function).-The population proband model is available (but as we shall see in the next section, practical implementations will not require this).-We also assume that E is a mutable set, and shall make use of a choose(X) subroutine which non-deterministically returns the set of a single a member of X (if one exists, otherwise it returns the empty set).
The algorithm is formally presented as Algorithm 1.We assume that an associated sample test suite will also be available as a by-product of the algorithm in addition to producing the data E. The intuition behind the algorithm is that failing vectors are iteratively accumulated in a set E one by one, where the next failing vector added does not cover some component which is covered by all vectors already in E (the algorithm terminates if no such vector exists).The resulting set is observed to be single-fault optimal.To illustrate the algorithm we give the example below.We then give a proof of partial correctness.
Example 8. We assume some population set of failing coverage vectors E * , which we may identify with the set {t * 1 , t * 2 , t * 3 } = { 1, 0, 1, 1, 0, 1, 1 , 1, 0, 0, 1, 1, 1, 2 , 1, 0, 0, 1, 0, 1, 3 } described in the coverage matrix of Fig. 2. In reality, the population set of failing coverage vectors for this faulty program is much larger than this, but this will suffice for our example.The algorithm proceeds as follows.First, we assume E is a non-empty subset of E * , and thus may assume E = { 1, 0, 1, 1, 0, 1, 1 }.Now, to evaluate step 2, we first evaluate the set . Intuitively, this is the set of failing vectors in the population which do not cover some component which is covered by all vectors in E. We may find a member of this set as follows.First, we must evaluate the condition for when E * = {t * 1 , t * 2 , t * 3 }.Given c 1 3 = 1 holds of t 1 , and t 1 is the only member of E, and given c 2 * 3 = 0, we have the conclusion that t * 2 is a member of the set.Thus, for our example we may assume that choose returns t * 2 from this set such that T = {t * 2 }.So at step 3 the new version of E is E = { 1, 0, 1, 1, 0, 1, 1 , 1, 0, 0, 1, 1, 1, 2 }.Consequently, on the next iteration of the loop the set condition will be unsatisfiable -this is because there is no index to a component i such that both ∀t j ∈ E.c j i = 1 holds (i.e., E ⊆ C i ), and also c k * i = 0 holds for some vector t * k in the population (i.e., not E * ⊆ C i ).Thus, choose will return the empty set, and the algorithm will terminate returning the dataset E to the user to be used in sbfl.Using the Naish-I measure with this dataset, we have the result that C1 and C4 are associated with the largest suspicious score of 2.0.Thus, with single-fault optimal data alone we can find a fault C4 reasonably effectively in our running example.

Proposition 1. All datasets returned by Algorithm 1 are single-fault optimal.
Proof.We show partial correctness as follows.Let PM * , T * be a given population proband model, where E * ⊆ T * is the population set of failing vectors, and let E be returned by the algorithm.We must show that for all i (by def. of single fault optimality).We prove this by contradiction.Assume there is some C i ∈ PM E (without loss of generality we may assume i = 1), such that E ⊆ C 1 but not E * ⊆ C * 1 .Given we assume E has been returned by the algorithm, we may assume T = ∅ (step 4), and thus choose returned ∅ at step 2 (by def. of choose).Accordingly, there is no 1 (by substitution of equivalents).This contradicts the initial assumption.
Finally, we informally observe that the maximum size of the E returned is the number of uuts.In this case E is input to the algorithm with a failing vector that covers all components, and choose always returns a failing vector that covers 1 fewer uuts than the failing vector covering the fewest uuts already in E (noting that we assume at least one component will always be covered).The minimum is one.In this case E is input to the algorithm with a failing vector which covers some components and the post-condition is already fulfilled.In general, E can potentially be much smaller than E * .

Implementation
We now discuss our implementation of the algorithm.In practice, we can leverage model checkers to compute members of E * (the population set of failing vectors) on the fly, where computing E * as a pre-condition would usually be intractable.This can be done by appeal to a SMT solving subroutine, which we describe as follows.Given a formal model of some code F code , a formal specification φ, set of Booleans which are true just in case a corresponding uut is executed in a given execution {C1, . . ., Cn}, and a set E ⊆ E * , we can use a SMT solver to return a satisfying assignment by calling SMT(F code ∧ ¬φ ∧ (∀t k ∈E)c k i =1 Ci = 0), and then extracting a coverage vector from that assignment.A subroutine which returns this coverage vector (or the empty set if one does not exist) can act as a substitute for the choose subroutine in Algorithm 1, and the generation of a static object E * is no longer required as an input to the algorithm.Our implementation of this is called sfo (single fault optimal data generation tool).
We now discuss extensions of sfo.It is known that adding passing executions help in sbfl [4,5,[7][8][9][10][11][12], thus to develop a more effective fault localisation procedure we developed a second implementation sfo p (sfo with passing traces) that runs sfo and then adds passing test cases.To do this, after running sfo we call a SMT solver 20 times to find up to 20 new passing execution, where on each call if the vector found has new coverage properties (does not cover all the same uuts as some passing vector already computed) it is added to a set of passing vectors.
Our implementations of sfo and sfo p are integrated into a branch of the model checker cbmc [32].Our branch of the tool is available for download at the URL given in the footnote3 .Our implementations, along with generating fault localisation data, rank uuts by degree of suspiciousness according to the Naish-I measure and report this fault localisation data to the user.

Experimentation
In this section we provide details of evaluation results for the use of sfo and sfo p in fault localisation.The purpose of the experiment is to demonstrate that implementations of Algorithm 1 can be used to facilitate efficient and effective fault localisation in practice on small programs (≤2.5kloc).We think generation of fault localisation information in a few seconds (≤2) is sufficient to demonstrate practical efficiency, and ranking the fault in the top handful of the most suspicious lines of code (≤5) on average is sufficient to demonstrate practical effectiveness.In the remainder of this section we present our experimental setup (where we describe our scoring system and benchmarks), and our results.

Setup
For the purposes of comparison, we tested the fault localisation potential of sfo and sfo p against a method named 1f , which performes sbfl when only a single failing test case was generated by cbmc (and thus uuts covered by the test case were equally suspicious).We used the following scoring method to evaluate the effectiveness of each of the methods for each benchmark.We envisage an engineer who is inspecting each loc in descending order of suspiciousness using a given strategy (inspecting lines that appear earlier in the code first in the case of ties).We rank alternative techniques by the number of non-faulty loc that are investigated until the engineer finds a fault.Finally, we report the average of these scores for the benchmarks to give us an overall measure of fault localisation effectiveness.
We now discuss the benchmarks used in our experiments.In order to perform an unbiased experiment to test our techniques on, we imposed that our benchmarks needed to satisfy the following three properties (aside from being a C program which cbmc could be used on): 1. Programs must have been created by an independent source, to prevent any implicit bias caused by creating benchmarks ourselves.2. Programs must have an explicit, formally stated specification that can be given as an assertion statement in order to apply a model checker.3.In each program, the faulty code must be clearly identifiable, in order to be able to measure the quality of fault localisation.
Unfortunately, benchmarks satisfying these conditions are rare.In practice, benchmarks exist in verification research that satisfy either the second or third criterion, but rarely both.For instance, the available sir benchmarks satisfy the third criterion, but not the second4 .The software verification competition (sv-comp) benchmarks satisfy the second criterion, but almost never satisfy the third 5 .Furthermore, it is often difficult to obtain benchmarks from authors even when usable benchmarks do in fact exist.Finally, we have been unable to find an instance of a C program that was not artificially developed for the purposes of testing.
The benchmarks are described in Table 1, where we give the benchmark name, the number of faults in the program, and lines of code (loc).The modified versions of tcas were made available by Alex Groce via personal correspondence and were used with the Explain tool in [33] 6 .The remaining benchmarks were identified as usable by manual investigation and testing in the repositories of sv-comp 2013 and 2017.We have made our benchmarks available for download directly from the link on footnote 4. Faults in sv-comp programs were identified by comparing them to an associated fault-free version (in tcas the fault was already identified).A series of continuous lines of code that differed from the fault free version (usually one line, and rarely up to 5 loc for larger programs) constituted one fault.loc were counted using the cloc utility.
We give further details about our application of cbmc in this experiment.For all our benchmarks, we used the smallest unwinding number that enables the bounded model checker to find a counterexample.These counterexamples were sliced, which usually results in a large improvement in fault localisation.For details about unwindings and slicing see the cbmc documentation [34].In each benchmark each executable statement (variable initialisations, assignments, or condition statements) was determined as a uut.

Results and Discussion
In this section we discuss our experimental results.In Table 1, columns 1f /sfo p /sfo give the scores for when the respective method is used.Column t gives the runtime for cbmc and sfo p respectively (we ignore the runtime for sfo due to negligible difference).|E| and |E| give the number of failing and passing test cases generated by sfo p .The AVG row gives averages column values.We are primarily interested in comparing the scores of sfo p and 1f .We now discuss the results of the three techniques 1f , sfo and sfo p .On average, 1f located a fault after investigating 17.23 lines of code (4.09% of the program on average).The results here are perhaps better than expected.We observed that the single failing test case consistently returned good fault localisation potential given the use of slicing by the technique.
We now discuss sfo.On average, sfo located a fault after investigating 16 lines of code (3.8% of the program on average).Thus, the improvement over 1f is very small.When only one failing test case was available for sfo (i.e.|E| = 1) we emphasise that the SMT solver could not find any other failing traces which covered different parts of the program.In such cases, sfo performed the same as 1f (as expected).However, when there was more than one failing test case available (i.e.|E| > 1), sfo always made a small improvement.Accordingly, for benchmarks 1, 2, 3, 5, 9, and 12 the improvements in terms fewer loc examined are 2, 6, 3, 1, 2, and 3, respectively.An improvement in benchmarks where sfo generated more than one test case is to be expected, given there was always a fault covered by all failing test cases in each program (even in programs with multiple faults), thus taking advantage of the property of single fault optimal data.Finally, we conjecture that on programs with more failing test cases available in the population, and on longer faulty programs, that this improvement will be larger.
We now discuss sfo p .On average, sfo p located a fault after investigating 4.08 loc (0.97% of each program on average).Thus, the improvement over the other techniques is quite large (four times as effective as 1f ).Moreover, this effectiveness came at very little expense to runtime -sfo p had an average runtime of 1.06 s, which is comparable to the runtime of 1f of 0.78 s.This is despite the fact that sfo p generated over 7 executions on average.We consequently conclude that implementations of Algorithm 1 can be used to facilitate efficient and effective fault localisation in practice on small programs.

Related Work
The techniques discussed in this paper improve the quality of data usable for sbfl.We divide the research in this field into the following areas; many other methods can be potentially combined with our technique.[36].Aside from coverage criteria, methods have been studied which generate test cases with a minimal distance from a given failed test case [18].Baudry et al. use a bacteriological approach in order to generate test suites that simultaneously facilitate both testing and fault localisation [19].Concolic execution methods have been developed to add test cases to a test suite based on their similarity to an initial failing run [20].
Prominent approaches which leverage model checkers for fault localisation are as follows.Groce [33] uses integer linear programming to find a passing test case most similar to a failing one and then compare the difference.Schupman and Bierre [37] generate short counterexamples for use in fault localisation, where a short counterexample will usually mean fewer uuts for the user to inspect.Griesmayer [38] and Birch et al. [39] use model checkers to find failing executions and then look for whether a given number of changes to values of variables can be made to make the counterexample disappear.Gopinath et al. [40] compute minimal unsatisfiable cores in a given failing test case, where statements in the core will be given a higher suspiciousness level in the spectra ranking.Additionally, when generating a new test, they generate an input whose test case is most similar to the initial run in terms of its coverage of the statements.Fey et al. [41] use SAT solvers to localise faults on hardware with LTL specifications.In general, experimental scale is limited to a small number of programs in these studies, and we think our experimental component provides an improvement in terms of experimental scale (13 programs).
Test Suite Reduction.An alternative approach to expanding a test suite is to use reduction methods.Recently, many approaches have demonstrated that it is not necessary for all test cases in a test suite to be used.Rather, one can select a handful of test cases in order to minimise the number of test cases required for fault localisation [42,43].Most approaches are based on a strategy of eliminating redundant test cases relative to some coverage criterion.The effectiveness of applying various coverage criteria in test suite reduction is traditionally based on empirical comparison of two metrics: one which measures the size of the reduction, and the other which measures how much fault detection is preserved.
Slicing.A prominent approach to improving the quality of test suites involves the process of slicing test cases.Here, sbfl proceeds as usual except the program and/or the test cases composing the test suite are sliced (with irrelevant lines of code/parts of the execution removed).For example, Alves et al. [44] combine Tarantula along with dynamic slices, Ju et al. [45] use sbfl in combination with both dynamic and execution slices.Syntactic dynamic slicing is built-in in all our tested approaches by appeal to the functionalities of cbmc.
To our knowledge, no previous methods generate data which exhibit our property of single fault optimality.

Conclusion
In this paper, we have presented a method to generate single fault optimal data for use with sbfl.Experimental results on our implementation sfo p , which integrates single fault optimal data along with passing test cases, demonstrate that small optimized fault localisation data can be generated efficiently in practice (1.06 s on average), and that subsequent fault localization can be performed effectively using this data (investigating 4.06 loc until a fault is found).We envisage that implementations of the algorithm can be used in two different scenarios.In the first, the test suite generated can be used in standalone fault localisation, providing a small and low cost test suite useful for repeating iterations of simultaneous testing and fault localisation during program development.In the second, the data generated can be added to any pre-existing data associated with a test suite, which may be useful at the final testing stage where we may wish to optimise single fault localisation.
Future work involves finding larger benchmarks to use our implementation on and developing further properties, and methods for use with programs with multiple faults.We would also like to combine our technique with existing test suite generation algorithms in order to experiment how much test suites can be additionally improved for the purposes of fault localization.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/),which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material.If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Definition 3 .
For a given proband we define a proband model PM, T , consisting of the given faulty program's program model PM, and an associated test suite's set of coverage vectors T.
is the number of failing coverage vectors in T that do not cover C i , |C i ∩ E| is the number of passing coverage vectors in T that cover C i , and |C i ∩ E| is the number of passing coverage vectors in T that do not cover C Example 6.For the proband model of the running example PM, T (where PM = C 1 , ..., C 5 , E and T is represented by the coverage matrix in Fig.2), the spectra for C 1 , . . .C 5 , and E are 3, 0, 7, 0 , 0, 3, 3, 4 , 1, 2, 2, 5 , 3, 0, 1, 6 , 1, 2, 3, 4 , and 3, 0, 0, 7 respectively.
i .|C i ∩ E|, |C i ∩ E|, |C i ∩ E| and |C i ∩ E| are often denoted a i ef , a i nf , a i ep , and a i np respectively in the literature [4,7-12].Definition 5. A suspiciousness measure w is a function with signature w : PM → R, and maps each C i ∈ PM to a real number as a function of C i 's program spectrum |C i ∩ E|, |C i ∩ E|, |C i ∩ E|, |C i ∩ E| , where this number is called the component's degree of suspiciousness.

Table 1 .
Experimental results Test Suite Expansion.One approach to improving test suites is to add more test cases which satisfy a given criterion.A prominent criterion is that the test suite has sufficient program coverage, where studies suggest that test suites with high coverage improve fault localisation Other ways to improve test suites for sbfl are as follows.Li et al. generate suites for sbfl, considering failing to passing test case ratio to be more important than number [35].Zhang et al. consider cloning failed test cases to improve sbfl [13].Perez et al. develop a metric for diagnosing whether a test suite is of sufficient quality for sbfl to take place [14].Li et al. consider weighing different test cases differently