CoVeriTest: Cooperative Veriﬁer-Based Testing

. Testing is a widely used method to assess software quality. Coverage criteria and coverage measurements are used to ensure that the constructed test suites adequately test the given software. Since manually developing such test suites is too expensive in practice, various automatic test-generation approaches were proposed. Since all approaches come with diﬀerent strengths, combinations are necessary in order to achieve stronger tools. We study cooperative combinations of veriﬁcation approaches for test generation, with high-level information exchange. We present CoVeriTest , a hybrid approach for test-case generation, which iteratively applies diﬀerent conditional model checkers. Thereby, it allows to adjust the level of cooperation and to assign individual time budgets per veriﬁer. In our experiments, we combine explicit-state model checking and predicate abstraction (from CPAchecker ) to systematically study different CoVeriTest conﬁgurations. Moreover, CoVeriTest achieves higher coverage than state-of-the-art test-generation tools for some programs.


Introduction
Testing is a commonly used technique to measure the quality of software.Since manually creating such test suites is laborious, automatic techniques are used: e.g., model-based techniques for black-box testing and techniques based on control-flow coverage for white-box testing.Many automatic techniques have been proposed, ranging from random testing [36,57] and fuzzing [26,52,53], over search-based testing [55] to symbolic execution [23,24,58] and reachability analyses [5,12,45,46].The latter are well-suited to find bugs and derive test suites that achieve high coverage, and several verification tools support test generation (e.g., Blast [5], PathFinder [61], CPAchecker [12]).The reachability checks for all test goals seem too expensive, but in practice, those approaches can be made pretty efficient.
Encouraged by tremendous advances in software verification [3] and a recent case study that compared model checkers with test tools w.r.t.bug finding [17], we study a new kind of combination of reachability analyses for test generation.Combinations are necessary because different analysis techniques have different strength and weaknesses.For example, consider function foo in Listing 1. of the statements in the outermost if branch (lines 3-6), while it has difficulties with the complex condition in the else-branch (line 8).
Inspired by abstraction-driven concolic testing [32], which interleaves concolic execution and predicate abstraction, we propose CoVeriTest, which stands for cooperative verifier-based testing.CoVeriTest iteratively executes a given sequence of reachability analyses.In each iteration, the analyses are run in sequence and each analysis is limited by its individual, but configurable time limit.Furthermore, CoVeriTest allows the analysis to share various types of analysis information, e.g., which paths are infeasible, have already been explored, or which abstraction level to use.To get access to a large set of reachability analyses, we implemented CoVeriTest in the configurable software-analysis framework CPAchecker [15].We used our implementation to evaluate different CoVeriTest configurations on a large set of well-established benchmark programs and to compare CoVeriTest with existing state-of-the-art test-generation techniques.Our experiments confirm that reachability analyses are valuable for test generation.Contributions.In summary, we make the following contributions: • We introduce CoVeriTest, a flexible approach for high-level interleaving of reachability analyses with information exchange for test generation.

Testing with Verifiers
The basic idea behind testing with verifiers is to derive test cases from counterexamples [5,61].Thus, meeting a test goal during verification has to trigger a specification violation.First, we remind the reader of some basic notations.Programs.Following literature [9], we represent programs by control-flow automata (CFAs).A CFA P = (L, 0 , G) consists of a set L of program locations (the program-counter values), an initial program location 0 ∈ L, and a set of control-flow edges G ⊆ L × Ops × L. The set Ops describes all possible operations, e.g., assume statements (resulting from conditions in if or while statements) and assignments.For the program semantics, we rely on an operational semantics, which we do not further specify.Abstract Reachability Graph (ARG).ARGs record the work done by reachability analyses.An ARG is constructed for a program P = (L, 0 , G) and stores (a) the abstract state space that has been explored so far, (b) which abstract states must still be explored, and (c) what abstraction level (tracked variables, considered predicates, etc.) is used.Technically, an ARG is a five-tuple (N, succ, root, F, π) that consists of a set N of abstract states, a special node root ∈ N that represents the initial states of program P , a relation succ ⊆ N × G × N that records already explored successor relations, a set F ⊆ N of frontier nodes, which remembers all nodes that have not been fully explored, and a precision π describing the abstraction level.Every ARG must ensure that a node n is either contained in F or completely explored, i.e., all abstract successors have been explored.We use ARGs for information exchange between reachability analyses.q 0 q e g ∈ goals g / ∈ goals Test Goals.In this paper, we are interested in structural coverage, e.g., branch coverage.Transferred to our notion of programs, this means that our test goals are a subset of the program's control-flow edges.For using a verifier to generate tests, we have to encode the test goals as a specification violation.Figure 2 shows a possible encoding, which uses a protocol automaton.Whenever a test goal is executed, the automaton transits from the initial, safe state q 0 to the accepting state q e , which marks a property violation.Note that reachability analyses, which we consider for test generation, can easily monitor such specifications during exploration.Now, we have everything at hand to describe how reachability analyses generate tests.Algorithm 1 shows the test-generation process.The algorithm gets as input a program, a set of test goals, and a time limit for test generation.For cooperative test generation, we need to guide state-space explorations.To this end, we also provide an initial ARG and a condition.A condition is a concept known from conditional model checking [10] and describes which parts of the state space have already been explored by other verifiers.A verifier, e.g., a reachability analysis, can use a condition to ignore the already explored parts of the state space.Verifiers that do not understand conditions can safely ignore them.
At the beginning, Alg. 1 sets up the data structures for the test suite and the set of covered goals.To set up the specification, it follows the idea of Fig. 2. As long as not all test goals are covered, there exist abstract states that must be explored, and the time limit has not elapsed, the algorithm tries to generate new tests.Therefore, it resumes the exploration of the current ARG [5] taking into account program prog, specification ϕ, and (if understood) the condition ψ.If the exploration stops, then it returns an updated ARG.Exploration stops due to one of three reasons: (1) the state space is explored completely (F = ∅), (2) the Algorithm 1 Generating tests with a (conditional) reachability analysis time limit is reached, or (3) a counterexample has been found. 3In the latter case, a new test is generated.First, a counterexample trace is extracted from the ARG.The trace describes a path through the ARG that starts at the root and its last edge is a test goal (the reason for the specification violation).Next, a test is constructed from the path and added to the test suite.Basically, the path is converted into a formula and a satisfying assignment4 is used as the test case.For the details, we refer the reader to the work that defined the method [5].Additionally, the covered goal (last edge on the counterexample path) is removed from the set of open test goals and added to the set of covered goals.Finally, the specification is updated to no longer consider the covered goal.When the algorithm finishes, it returns the generated test suite, the set of covered goals and the last ARG considered.The ARG is returned to enable cooperation.

CoVeriTest
The previous section described how to use a single reachability analysis to produce tests for covering a set of test goals.Due to different strengths and weaknesses, some test goals are harder to cover for one analysis than for another.To maximize the number of covered goals, different analyses should be combined.In CoVeri-Test, we rotate analyses for test generation.Thus, we avoid that analyses try to cover the same goal in parallel and we do not need to know in advance which analysis can cover which goals.Moreover, analyses that get stuck trying to cover goals that other analyses handle later, get a chance to recover.Additionally, CoVeriTest supports cooperation among analyses.More concrete: analyses may extract and use information from ARGs constructed by previous analysis runs.
Algorithm 2 describes the CoVeriTest workflow.It gets four inputs.Program, test goals, and time limit are already known from Alg. 1 (test generation with a single analysis).Additionally, CoVeriTest gets a sequence of configurations, namely pairs of reachability analysis and time limit.The time limit accompanied with the analysis restricts the runtime of the respective analysis per call (see line 5).In contrast to Alg. 1, CoVeriTest does not get an ARG or condition.To enable cooperation between analyses, CoVeriTest constructs these two elements individually for each analysis run.During construction, it may extract and use information from results of previous analysis runs.
After initializing the test suite and the data structure to store analysis results (args), CoVeriTest repeatedly iterates over the configurations.It starts with the first pair in the sequence and finishes iterating when its time limit exceeded or all goals are covered.In each iteration, CoVeriTest first extracts the analysis to execute and its accompanied time limit (line 3).Then, it constructs the remaining inputs of the analysis: ARG and condition.Details regarding the construction are explained later in Alg. 3. Next, CoVeriTest executes the current analysis with the given program, the remaining test goals, the accompanied time limit, and the constructed ARG and condition.When the analysis has finished, CoVeriTest adds the returned tests to its test suite, removes all test goals covered by the analysis run from the set of goals, and stores the analysis result for cooperation (concatenates arg to the sequence of ARGs).If the analysis finished its exploration (arg.F=∅), any remaining test goal should be unreachable and CoVeriTest returns its test suite.Otherwise, CoVeriTest determines how to continue in the next iteration (i.e., which configuration to consider).At the end of all iterations, CoVeriTest returns its generated test suite.

Algorithm 3 cooperateAndInit: set up start point for analysis exploration, possibly transferring knowledge from previous analysis runs
Input: prog = (L, 0, G), args ∈ (arg) + , numAnalyses ∈ N Output: ARG for program prog, condition describing explored state space 1: ψ=false; π = ∅; root = ( 0, ); 2: if (length(args)≥numAnalyses) then 3: if (reuse-arg) then 4: return (last_arg_of_analysis(numAnalyses, args), ψ); 5: if (reuse-precision) then 6: π = last_arg_of_analysis(numAnalyses, args).π; Next, we explain how to construct the ARG and the condition input for an analysis.The ARG describes the level of abstraction and where to continue exploration while the condition describes which parts of the state space have already been explored.Both guide the exploration of an analysis, which makes them well-suited for cooperation.While there are plenty of possibilities for cooperation, we currently only support three basic options: continue exploration of the previous ARG of the analysis (reuse-arg), reuse the analysis' abstraction level (reuse-precision), and restrict the exploration to the state space left out by the previous analysis (use-condition).The first two options only ensure that an analysis does not loose too much information due to switching.The last option, which is inspired by abstraction-driven concolic execution [32], indeed realizes cooperation between different analyses.Note that the last two options can also be combined. 5If all options are turned off, no information will be exchanged.
Algorithm 3 shows the cooperative initialization of ARG and condition discussed above.It gets three inputs: the program, a sequence of args needed to realize cooperation, and the number of analyses used.At the beginning, it initializes the ARG components and the condition assuming no cooperation should be done.The condition states that nothing has been explored, the abstraction level becomes the coarsest available and the ARG root considers the start of all program executions (initial program location and arbitrary variable values).If no cooperation is configured or the ARG required for cooperation is not available (e.g., in the first round), the returned ARG and condition tell the analysis to explore the complete state space from scratch.In all other cases, the analysis will be guided by information obtained in previous iterations.Option reuse-arg looks up the last ARG of the analysis stored in args.Reuse-precision considers the same ARG as reuse-arg, but only provides the ARG's precision π.For use-condition, a condition is constructed from the last ARG in args.For the details of the condition construction, we refer to conditional model checking [10].
Next, we study the effectiveness of different CoVeriTest configurations and compare CoVeriTest with existing test-generation tools.

Evaluation
We systematically evaluate CoVeriTest along the following claims: Claim 1.For analyses that discard their own results from previous iterations (i.e., reuse-arg and reuse-precision turned off), CoVeriTest achieves higher coverage if switches between analyses happen rarely.Evaluation Plan: We look at CoVeriTest configurations in which analyses discard their own, previous results and compare the number of covered test goals reported by configurations that only differ in the analyses' time limits.Claim 2. For analyses that reuse knowledge from their own, previous execution (i.e., reuse-arg or reuse-precision turned on), CoVeriTest achieves higher coverage if favoring more powerful analyses.Evaluation Plan: We look at CoVeri-Test configurations in which analyses reuse their own, previous knowledge and compare the number of covered test goals reported by configurations that only differ in the analyses' time limits.Claim 3. CoVeriTest performs better if analyses reuse knowledge from their own, previous execution (i.e., reuse-arg or reuse-precision turned on).Evaluation Plan: From all sets of CoVeriTest configurations that only differ in the analyses' time limits, we select the best and compare these.Claim 4. Interleaving multiple analyses with CoVeriTest often achieves better results than using only one of the analyses for test generation.Evaluation Plan: We compare the number of covered goals reported by the best CoVeriTest configuration with those numbers achieved when running only one analysis of the CoVeriTest configuration for the total time limit.Claim 5. Interleaving verifiers for test generation is often better than running them in parallel.
Value analysis.CPAchecker's value analysis [18] tracks the values of variables stored in its current precision explicitly while assuming that the remaining variables may have any possible value.It iteratively increases its precision, i.e., the variables to track, combining counterexample-guided abstraction [28] with path-prefix slicing [22], and refinement selection [21].Value analysis is efficient if few variable values need to be tracked, but it may get stuck in loops or suffers from a large state space in case variables are assigned many different values.
Predicate analysis.CPAchecker's predicate analysis uses predicate abstraction with adjustable-block encoding (ABE) [16].ABE is configured to abstract at loop heads and uses the strongest postcondition at all remaining locations.To compute the set of predicates-its precision-, it uses counterexample-guided abstraction refinement [28] combined with lazy refinement [43] and interpolation [41].While the predicate analysis is powerful and often summarizes loops easily, successor computation may require expensive SMT solver calls.
For both analyses, a CoVeriTest configuration specifies how Alg. 3 reuses the ARGs returned by previous analysis runs to set up the initial ARG and condition.In our experiments, we consider the following types of reuses.
plain Ignores all ARGs returned by previous analysis runs, i.e., reuse-arg, reuse-prec, and use-condition are turned off.cond v The value analysis does not obtain information from previous ARGs and the predicate analysis is only steered by the condition extracted from the ARG returned by the previous value analysis.cond p The value analysis is steered by the condition extracted from the ARG returned by the previous run of the predicate analysis and the predicate analysis ignores all previous ARGs.cond v,p Value and predicate analysis are steered by the condition extracted from the last ARG returned, i.e., only use-condition turned on.reuse-prec In each round, each analysis resumes its precision from the previous round, but restarts exploration, i.e., only reuse-prec is turned on.reuse-arg In each round, each analysis continues to explore the ARG it returned in the previous round, i.e., only reuse-arg is turned on.cond v +r Similar to cond v , but additionally the value analysis continues to explore the ARG it returned in the previous round and the predicate analysis restarts exploration with its precision from the previous round.cond p +r Similar to cond p , but additionally the value analysis restarts exploration with its precision from the previous round and the predicate analysis continues to explore the ARG it returned in the previous round.cond v,p +r Like cond v,p , but additionally the value and predicate analysis reuse their previous precision, i.e., reuse-prec and use-condition are turned on.
Finally, we need to fix the time limit for each analysis.We want to find out whether switches between analyses are important to the CoVeriTest approach.Therefore, we chose four limits (10 s, 50 s, 100 s, 250 s) that are applied to both analyses and trigger switches often, sometimes, or rarely.Additionally, we want to study whether it is advantageous if the time CoVeriTest spends in a round is not equally spread among the analyses.Thus, we come up with two additional time limit pairs: (20 s, 80 s) and (80 s, 20 s).
We combine all nine reuse types with the six time limit pairs, which results in 54 CoVeriTest configurations.All 54 configurations aim at generating tests to cover the assume edges of a program.
Tools.For CoVeriTest, we used the implementation in CPAchecker version 29 347.Moreover, we compare CoVeriTest against the two best tools VeriFuzz [26] and Klee [23]  Programs.CoVeriTest, Klee, and VeriFuzz produce tests for C programs.All three tools participated in TestComp'19.Thus, for comparison of the three tools, we consider all 1 720 tasks of the TestComp'19 benchmark set 10 that support the branch-coverage property.Since we do not need to execute tests for the comparison of the different CoVeriTest configurations, we evaluated them on a larger benchmark set, which contains all 6 703 C programs from the well-established SV-benchmark set 11 in the version tagged svcomp18.

Experiments
Claim 1 (Reduce switching when discarding own results).Four types of reuse (namely, plain, cond v , cond p , and cond v,p ) let the analyses discard their own knowledge from their previous executions.For each of these types, we compare the coverage achieved by all six CoVeriTest configurations that use this type 13 .More concrete, for all six CoVeriTest configurations applying the same reuse type, we first compute for each program the maximum over the number of covered goals achieved by each of these six configurations for that program.Then, for each of the six CoVeriTest configurations that use that reuse type, we divide the number of covered goals achieved for a program by the respective maximum computed.We call this measure relative coverage because the value is relative to the maximum and not the total number of goals.Figure 3 shows box plots per reuse type.The box plots show the distribution of the relative coverage.The closer the bottom border of a box is to value one, the higher coverage is achieved.For all four reuse types, the fourth box plot has the bottom border closest to value one.Since the fourth box plot is a configuration that grants each analysis 250 s per round (highest limit considered, only three switches), the claim holds.Claim 2 (Favor powerful analysis when reusing own results).Five types of reuse (namely, reuse-prec, reuse-arg, cond v +r, cond p +r, and cond v,p +r) let analyses reuse knowledge from their own, previous execution.Similar to the previous claim, we compute for each of these types the relative coverage of all six configurations using this particular type of reuse.For each reuse type, Fig.  shows box plots of the distributions of the relative coverage.As before, a bottom border closer to value one reflects higher coverage.In all five cases, the last box plot has the bottom border closest to value one.The last box plots represent CoVeriTest configurations that grant the value analysis 20 s and the predicate analysis 80 s in each round.Since the predicate analysis, which gets more time per round, is more powerful than the value analysis, our claim is valid. 14laim 3 (Better reuse own results).So far, we know how to configure time limits.Now, we want to find out how to reuse information from previous analysis runs.For each reuse type, we select from the six available configurations the configuration that performed best.Again, we use the relative coverage to compare the resulting nine configurations.Figure 5 shows box plots of the distributions of the relative coverage.The first four box plots show configurations in which analyses discard their own results, while the last five box plots refer to configurations in which analyses reuse knowledge from their own, previous executions.Since the last five boxes are smaller than the first four and their bottom borders are closer to one, the last five configurations achieve higher coverage.Hence, our claim holds.Moreover, from Fig. 5 we conclude that it is best to reuse the ARG (although cond v +r and cond p +r are close by).Claim 4 (Interleave multiple analyses rather than use one of them).To evaluate whether CoVeriTest benefits from interleaving, we compare CoVeri-Test against the analyses used by it.CoVeriTest interleaves value and predicate analysis.Figure 6(a) and 6(b) show scatter plots that compare for each program the coverage, i.e., number of covered goals divided by number of total goals, achieved by the best CoVeriTest configuration (x-axis) with the coverage achieved when only using either value or predicate analysis for test generation.(c) CoVeriTest (x-axis) vs. value and predicate analysis in parallel Fig. 6.Compares the coverage achieved by CoVeriTest (best configuration) with the coverage achieved when running CoVeriTest's analyses alone or in parallel Note that we excluded those programs from the scatter plots, for which we miss the number of covered goals for at least one test generator, e.g., due to timeout of the analysis.Figure 6(a) compares CoVeriTest and value analysis; we see that almost all points are in the lower right half.Thus, CoVeriTest typically achieves higher coverage than value analysis alone.Figure 6 Looking at the two scatter plots, we observe that there exist programs for which CoVeriTest performs better and vice versa.Generally, we observed that CoVeriTest has problems with array tasks and ECA tasks.We already know from verification that CPAchecker sometimes lacks refinement support for array tasks.Moreover, the problem with the ECA tasks is that CPAchecker splits conditions with conjunctions or disjunctions -which ECA tasks contain a lotinto multiple assume edges.Thus, the number of test goals is much larger than the actual branches to be covered.However, CoVeriTest seems to benefit from splitting for some of the float tasks.Additionally, CoVeriTest is often better on tasks of the sequentialized subcategory.We think that CoVeriTest benefits from the value analysis since the tasks of the sequentialized subcategory contain lots of branch conditions checking for a specific value or interpreting variable values as booleans.All in all, CoVeriTest is not always best, but is also not dominated.Thus, CoVeriTest complements the existing approaches.

Threats to Validity
All our CoVeriTest configurations consider the same two analyses.Our results might not apply if using CoVeriTest with a different set of analyses.In our experiments, we used benchmark programs instead of real-world applications.
Although the benchmark set is diverse and well-established, our results may not carry over into practice.
The validator TBF Test-Suite Validator might contain bugs that result in wrong coverage numbers.However, the validator was used in Test-Comp'19 already, and is based on the well-established coverage-measurement tool gcov.
For the comparison of the CoVeriTest configurations as well as the comparison of CoVeriTest with the single analyses and the parallel approach, we relied on the number of covered goals reported by CoVeriTest.Invalid counterexamples could be used to cover test goals.The analyses used by CoVeriTest apply CE-GAR approaches and should detect spurious counterexamples.Moreover, these analyses run in the SV-COMP configuration of CPAchecker and are tuned to not report false results.Another problem is that whenever CPAchecker does not output statistics (due to timeout, out of memory, etc.), we use the last number of covered goals reported in the log.However, this might be an underapproximation of the number of covered goals.All these problems do not occur in the comparison of CoVeriTest with Klee and VeriFuzz, in which the coverage is measured by the validator.Thus, this comparison still supports the value of CoVeriTest.

Related Work
CoVeriTest interleaves reachability analyses to construct tests for C programs.To enable cooperation, CoVeriTest extracts information from ARGs constructed by previous analysis runs.
A few tools use reachability analyses for test generation.Blast [5] considers a target predicate p and generates a test for each program location that can be reached with a state fulfilling the predicate p.For test generation, Blast uses predicate abstraction.FShell [44,45,46] and CPA/Tiger [12] generate tests for a coverage criterion specified in the FShell query language (FQL) [46].Both transform the FQL specification into a set of test-goal automata and check for each automaton whether its final state can be reached.FShell uses CBMC to answer those reachability queries and CPA/Tiger uses predicate abstraction.
Various combinations have been proposed for verification [2,10,11,14,25,27,29,30,31,35,37,40,50,64] and test-suite generation [1,32,34,36,38,47,51,54,56,59,60,63].We focus on combinations that interleave approaches.SYNERGY [40] and DASH [2] alternate test generation and proof construction to (dis)prove a property.Similarly, SMASH [37] combines underapproximation with overapproximation.Interleaving is also used in test generation.Hybrid concolic testing [54] interleaves random testing with symbolic execution.When random testing gets stuck, symbolic execution is started from the current state.As soon as a new goal is covered, symbolic execution hands over to random testing providing the values used to cover the goal.Similarly, Driller [60] and Badger [56] combine fuzzing with concolic execution.However, they only exchange inputs.Xu et al. [51,63] interleave different approaches to augment test suites.The approach closest to CoVeriTest is abstraction-driven concolic testing [32].Abstraction-driven concolic testing interleaves concolic execution and predicate analysis.Furthermore, it uses conditions extracted from the ARGs generated by the predicate analysis to direct the concolic execution towards feasible paths.Abstraction-driven concolic testing can be seen as one particular configuration of CoVeriTest.
Also, ARG information has been reused in different contexts.Precision reuse [19] uses the precision determined in a previous analysis run to reverify a modified program.Similarly, extreme model checking [42] adapts an ARG constructed in a previous analysis to fit to the modified program.CPA/Tiger [12] transforms an ARG that was constructed for one test goal such that it fits to a new test goal.Lazy abstraction refinement [43] adapts an ARG to continue exploration after abstraction refinement.Configurable program certification [48,49] constructs a certificate from an ARG, which can be used to reverify a program.Similarly, reachability tools like CPAchecker construct witnesses [6,7] from ARGs.Conditional model checking [10,14] constructs a condition from an ARG when a verifier gives up.The condition describes the remaining verification task and is used by a subsequent verifier to restrict its exploration.

Conclusion
Testing is a standard technique for software quality assurance.But state-ofthe-art techniques still miss many bugs that involve sophisticated branching conditions [17].It turns out that techniques performing abstract reachability analyses are well-suited for this task.They simply need to check the reachability of every branch and generate a test for each positive check.However, in practice, for every such technique there exist reachability queries on which the technique is inefficient or fails [8].We propose CoVeriTest to overcome these practical limitations.CoVeriTest interleaves different reachability analyses for test generation.We experimented with various configurations of CoVeri-Test, which vary in the time limits of the analyses and the type of information exchanged between different analysis runs.CoVeriTest works best when each analysis resumes its exploration, different analyses only share test goals, and more powerful analyses get larger time budgets.Moreover, a comparison of CoVeriTest with (a) the reachability analyses used by CoVeriTest and (b) state-of-the-art test generation tools witness the benefits of the new CoVeriTest approach.
CoVeriTest participated in Test-Comp 2019 [4] and achieved rank 3 (out of 9) in both categories, bug finding and branch coverage. 17n future, we plan to integrate further analyses, e.g., bounded model checking or symbolic execution, into CoVeriTest and to evaluate CoVeriTest on realworld applications.

Fig. 3 .
Fig. 3. Comparing relative coverage (number of covered goals divided by maximal of covered goals) achieved by CoVeriTest configurations with different time limits.All configurations let analyses discard their own knowledge gained in previous executions.

Fig. 4 .
Fig.4.Comparing relative coverage (number of covered goals divided by maximal number of covered goals) achieved by CoVeriTest configurations when using different time limits and a fixed reuse type.All considered configurations let analyses reuse knowledge from their own, previous execution.

Fig. 5 .
Fig. 5. Comparing relative coverage achieved by CoVeriTest configurations applying different strategies to reuse information gained by previous verifier runs.
Algorithm 2 CoVeriTest: alternating reachability analyses to generate tests Evaluation Plan: We compare the number of covered goals reported by the best CoVeriTest configuration with the number achieved when running all analyses of the CoVeriTest configuration in parallel.Claim 6. CoVeriTest complements existing test generation tools.Evaluation Plan: We use the same infrastructure and resources as used by the International Competition on Software Testing (Test-Comp'19) 6 and let the best CoVeriTest configuration construct test suites.These test suites are executed by the Test-Comp'19 validator to measure the achieved branch coverage.Then, we compare the coverage achieved by CoVeriTest with the coverage of the best two test generation tools from Test-Comp'19.
from Test-Comp'19 (in the versions submitted to Test-Comp'19 7 ).The tool VeriFuzz is based on the evolutionary fuzzer AFL and uses verification techniques to compute initial input values and parameters for AFL.Klee applies symbolic execution.To compare CoVeriTest against Klee and VeriFuzz, we use the validator TBF Test-Suite Validator v1.2 8 to measure branch coverage.TBF Test-Suite Validator is based on gcov 9 .
[4], comparing CoVeriTest with predicate analysis, is more diverse.About 54 % of the points are on the diagonal, i.e., CoVeriTest and predicate analysis cover the same number of goals.The upper left half contains 19 % of the points, i.e., predicate analysis alone achieves higher coverage.These points for example reflect float programs and ECA programs without arithmetic computations.In contrast, CoVeriTest achieves higher coverage in 27 % of the programs.CoVeriTest is beneficial for programs that only need few variable values to trigger the branches, like ssh programs or programs from the product-lines subcategory.CoVeriTest also profits from the value analysis when considering ECA programs with arithmetic computations, since the variables have a fixed value in each loop iteration.All in all, CoVeriTest performs slightly better than predicate analysis alone.Claim 5 (Interleave rather than parallelize).Figure6(c) shows a scatter plot that compares for each program the coverage achieved by CoVeriTest (x-axis) and a test generator that runs the value analysis and the predicate analysis in parallel15.As before, we exclude programs for which we could not CoVeriTest vs. Klee Fig.7.Compares the branch coverage achieved by CoVeriTest (best configuration) with the branch coverage achieved by existing state-of-the-art test generation tools get the number of covered goals for at least one of the analyses.Looking at Fig.6(c), we observe that many points (60 %) are on the diagonal, i.e., the achieved coverage is identical.Moreover, CoVeriTest performs better for 30 % (lower right half), while approximately 10 % of the points are in the upper left half.Since CoVeriTest achieves the same or better coverage results in about 90 % of the cases, it should be preferred over parallelization.This is no surprise since we showed that a test generator should favor the more powerful analysis (which CoVeriTest does, but parallelization evenly distributes CPU time).Claim 6 (CoVeriTest complementary).Our goal is to compare CoVeri-Test and the two best tools of Test-Comp'19[4]: VeriFuzz and Klee.All three tools aim at constructing test suites with high branch coverage.Thus, we use branch coverage as comparison criterion.We measure branch coverage with TBF Test-Suite Validator.Figure7shows two scatter plots.Each plot compares branch coverage achieved by CoVeriTest and by one of the other techniques.16Points in the lower right half indicate that CoVeriTest achieved higher coverage.