ByMC: Byzantine Model Checker
 3 Citations
 667 Downloads
Abstract
In recent work [10, 12], we have introduced a technique for automatic verification of thresholdguarded distributed algorithms that have the following features: (1) up to t of processes may crash or behave Byzantine; (2) the correct processes count messages and progress when they receive sufficiently many messages, e.g., at least \(t+1\); (3) the number n of processes in the system is a parameter, as well as t; (4) and the parameters are restricted by a resilience condition, e.g., \(n > 3t\).
In this paper, we present Byzantine Model Checker that implements the abovementioned technique. It takes two kinds of inputs, namely, (i) threshold automata (the framework of our verification techniques) or (ii) Parametric Promela (which is similar to the way in which the distributed algorithms were described in the literature).
We introduce a parallel extension of the tool, which exploits the parallelism enabled by our technique on an MPI cluster. We compare performance of the original technique and of the extensions by verifying 10 benchmarks that model faulttolerant distributed algorithms from the literature. For each benchmark algorithm we check two encodings: a manual encoding in threshold automata vs. a Promela encoding.
1 Introduction
In recent work [10, 11, 12] we applied bounded model checking to verify reachability properties of thresholdbased faulttolerant distributed algorithms (FTDA), which are parameterized in the number of processes n and the fraction of faults t. FTDAs typically work only under arithmetic resilience conditions such as \(n>3t\). Our methods allow us to do parameterized verification of sophisticated FTDAs [3, 5, 6, 18, 20, 21] that have not been automatically verified before. Our bounded model checking technique produces a number of queries to a Satisfiability Modulo Theories solver (SMT). These queries correspond to different execution patterns.
In [12], we conjectured that, by design, this technique allows many SMT queries to be checked in parallel. In this paper, we present a parallel extension of ByMC that executes SMT queries in a computer cluster.
 1.
We present the tool ByMC 2.4.1 that implements sequential and parallel verification [10, 12]. The parallel verification is implemented with MPI (Message Passing Interface).
 2.
We introduce the details of the parallel extension of the technique and perform experimental evaluation, both for the sequential and parallel versions of the tool.
 3.
We report the experimental results both for the abstractions that are automatically constructed from Promela code (as in [10, 12]) and for manual abstractions in terms of threshold automata, which we use as a direct input for the first time. Our experiments show that explicit modeling of faulttolerant distributed algorithms with threshold automata leads to a dramatic speed up in most cases.
2 Distributed Algorithm Example: Naïve Voting

Agreement. No two correct processes decide on different values (0 and 1).

Validity. If a correct process decides on a value \(v \in \{0, 1\}\), then there is a process i, whose initial value \(u_i\) equals v.

Termination. All correct processes eventually decide.
Figure 1 shows a naïve attempt to solve this problem by majority voting. As usual in the distributed algorithms literature, we give a solution in pseudocode, which is supposed to work as follows. Each process starts with a binary value \(u_i \in \{0, 1\}\) and sends \(u_i\) to all processes, including itself. When a process receives a value \(v \in \{0, 1\}\) from a majority of processes, it decides on v.

Asynchronous computations. Every correct process is scheduled infinitely often, and there are no assumptions on the relative processor speeds. The process steps are interleaved.

Reliable communication. The processes communicate via message passing. Every message sent by a correct process is eventually delivered, although there are no timing or ordering assumptions about message delivery.

Faults. A fraction f of processes may fail. For instance, they can crash or behave Byzantine — the faulty processes do not follow the algorithm. There is an upper bound \(t \ge f\) on the number of faults. We assume \(n > 3t\) for the Byzantine faults, and \(n > 2t\) for the crash faults.

Validity. We consider the Byzantine case here, which is more complicated. In order to decide on a value v in line 6, a correct process has to receive \(\lceil \frac{n+1}{2}\rceil \) messages carrying v. By the assumption on the number of faults (\(n > 3t\) and \(t \ge f\)), we have \(f < \lceil \frac{n+1}{2}\rceil \), and if a process decides on v in 6, there is at least one correct process that has sent the value v in line 3. Thus, the algorithm satisfies “Validity”.
 Agreement. Whether the algorithm satisfies “Agreement” depends on the considered fault model:

No faults or crash faults. By line 4, a process has to receive the same value from \(\lceil \frac{n+1}{2}\rceil \) distinct processes. Since \(2 \cdot \lceil \frac{n+1}{2}\rceil > n\), and each process sends only one value (line 3), no two processes \(i, j:\ 1 \le i < j \le n\) can reach line 6 with different values \(v_i \ne v_j\). Thus, the processes cannot decide differently, and agreement is satisfied.

Byzantine faults. When \(f > 0\), the Byzantine processes can send value 0 to a process i and value 1 to a process \(j: j \ne i\). If the initial states of the correct processes are split into two equal sets, that is, \(n  f = 2 \cdot \{k \in \{1 \dots n\} :k \text { is correct} \text { and } u_k = 0 \}\), then the processes i and j reach line 6 with the values \(v_i=0\) and \(v_j=1\). As a result, agreement can be violated, and a verification tool must produce a counterexample.


Termination. Assume that there are no faults (\(f=0\)) and the initial states are equally partitioned, that is, \(n = 2 \cdot \{k \in \{1 \dots n\} :k \text { is correct} \text { and } u_k = 0 \}\). No process can pass beyond line 4, as none of the initial value sets form a majority. Therefore, Naïve Voting violates liveness, namely, “Termination”. This subtle bug renders the algorithm useless! A tool should thus not only check invariants, but also find counterexamples to liveness specifications.
The manual proofs are tricky, as they combine several kinds of reasoning: temporal reasoning, local reasoning about process code, global reasoning about the number of messages, correct and faulty processes, etc. Our tool ByMC automatically proves temporal properties (or finds counterexamples) of distributed algorithms that (i) communicate by sending to all, and (ii) contain actions that are guarded by comparison of the number of received messages against a linear combination of parameter values (e.g., for a majority).
3 Inputs: Parametric Promela and Threshold Automata
3.1 Parametric Promela
Promela is the input language of the Spin model checker [7]. As it is designed to specify concurrent systems, several features are suitable for capturing distributed algorithms. However, Spin is a finite state model checker, and so Promela only allows us to specify finite state systems. We have thus extended Promela in order to have a parametric number of processes and faults, etc. In the following we will discuss some of our extensions.
Figure 3 shows a model of the Naïve Voting algorithm from Fig. 1. This example contains all the essential features of parametric Promela. In line 2, we declare parameters: the number of processes n, the number of Byzantine processes f, and the minimal size of a majority set, that is, \(\lceil \frac{n+1}{2} \rceil \). In line 3, we declare two shared integer variables nsnt0 and nsnt1 that store the number of zeroes and ones sent by the correct processes. The expressions assume(...) in lines 4–5 restrict the choice of parameter values.

pc to store the algorithm’s control state, that is, whether a process is initialized with values 0 and 1 (i.e., pc=V0 and pc=V1 resp.), sent a message (pc=SE), decided on values 0 and 1 (i.e., pc=D0 and pc=D1 resp.)

nrcvd0 and nrcvd1 to store the number of zeroes and ones received from the correct and Byzantine processes; and

nextstate variables next_pc, next_nrcvd0, and next_nrcvd1 that are used to perform a process step.
An initial process state is chosen nondeterministically in lines 12–13.
A single process step is encoded as an atomic block in lines 14–31, which corresponds to an indivisible receivecomputesend step. In lines 15–18, a process possibly receives new messages: by invoking havoc(x), we forget the contents of a variable x, and by writing assume(e), we restrict the variable values to those that satisfy a logical expression e. Note that the statements havoc and assume do not belong to the standard Promela; they belong to parametric Promela and are inspired by the similar statements in Boogie [2]. Lines 20–27 encode the computations that can be found in pseudocode in Fig. 1. Like in Promela, a process nondeterministically picks an option of the form “:: guard > actions”, if guard evaluates to true, and executes actions.
To specify temporal properties, we first define atomic propositions in lines 34–38. The keywords some and all correspond to existential and universal quantification over the processes; they belong to parametric Promela. In lines 40–43, define LTL formulas that capture the properties of consensus (cf. Sect. 2).
Promela code in Fig. 3 models the informal pseudo code of Naïve Voting. Note that the manual translation from pseudo code is straightforward, except for one thing: It may seem more honest to maintain sets of sent and received messages, instead of storing only integer message counters such as nrcvd0 and nsnt0. It has been proven that modeling with sets is equivalent (bisimilar) to modeling with message counters [14]. Obviously, modeling with message counters produces smaller transition systems (cf. [9]).
3.2 Threshold Automata
We model Naïve Voting with the threshold automaton shown in Fig. 2. Its code in the .ta input format of ByMC is shown in Fig. 4. We are running \(nf\) instances of the threshold automaton; each instance is modelling a correct process. The automata operate on shared variables such as \(\mathsf {nsnt_0}\) and \(\mathsf {nsnt_1}\), which can be only incremented. A threshold automaton resides in a local state from a finite set \({\mathcal L}\), e.g., in our example, \({\mathcal L}=\{\textsf {V0}, \textsf {V1}, \textsf {SE}, \textsf {D0}, \textsf {D1}\}\). A rule (corresponding to an edge in Fig. 2) can move an automaton from one local state to another, provided that the shared variables in the current global state satisfy the rule’s threshold guard, e.g., \(2 \cdot (\mathsf {nsnt_0}+ f) \ge n + 1\). If a rule is labeled with an increment of a shared variable, e.g., \(\mathsf {nsnt_0}\small {\texttt {++}}\), then the shared variable is updated accordingly.
4 Theoretical Background
4.1 System
We assume fixed three finite sets: the set \({\mathcal L}\) contains the local states, the set \(\varGamma \) contains the shared variables that range over nonnegative integers, and the set \(\varPi \) contains the parameters that range over nonnegative integers.
Configurations \(\varSigma \) and \(I\). A configuration is a vector \(\sigma =({\mathbf {\varvec{\kappa }}},\mathbf {g},\mathbf {p})\), where \(\sigma .{\mathbf {\varvec{\kappa }}}\) is a vector of counter values, \(\sigma .\mathbf {g}\) is a vector of shared variable values, and \(\sigma .\mathbf {p}= \mathbf {p}\) is a vector of parameter values. In \(\sigma .{\mathbf {\varvec{\kappa }}}\) we store for each local state \(\ell \), how many processes are in this state. All values are nonnegative integers. In every initial configuration global variables have value zero, and all “modelled” processes are in initial locations. If specifications do not limit the behavior of faulty processes (which is typically the case with Byzantine faults), we only model the correct processes explicitly, while the impact of faulty processes is modelled as nondeterminism in the environment.
Transition relation \(R\). A transition is a pair \(t=({ rule },{ factor })\) of a rule of the \({\textsf {TA}}\) and a nonnegative integer called the acceleration factor, or just factor for short. If the factor is always 1, this corresponds that at each step exactly one processes takes a step, that is, interleaving semantics. Having factors greater than 1 permits a specific form of acceleration where an arbitrary number of processes that are ready to execute a rule can do that at the same time.

\(\sigma '.\mathbf {g}= \sigma .\mathbf {g}+ t.{ factor }\cdot t.\mathbf {u}\) and \(\sigma '.\mathbf {p}= \sigma .\mathbf {p}\)
 if \(t.{ from }\ne t.{ to }\) then

\(\sigma '.{\mathbf {\varvec{\kappa }}}[t.{ from }]=\sigma .{\mathbf {\varvec{\kappa }}}[t.{ from }]  t.{ factor }\),

\(\sigma '.{\mathbf {\varvec{\kappa }}}[t.{ to }]=\sigma .{\mathbf {\varvec{\kappa }}}[t.{ to }] + t.{ factor }\), and

\(\forall \ell \in {\mathcal L}\setminus \{t.{ from }, t.{ to }\}\) it holds that \(\sigma '.{\mathbf {\varvec{\kappa }}}[\ell ]=\sigma .{\mathbf {\varvec{\kappa }}}[\ell ]\)


if \(t.{ from }= t.{ to }\) then \(\sigma '.{\mathbf {\varvec{\kappa }}}= \sigma .{\mathbf {\varvec{\kappa }}}\)
Finally, the transition relation \(R\subseteq \varSigma \times \varSigma \) of the counter system is defined as follows: \((\sigma , \sigma ') \in R\) iff there is a rule \(r\in {\mathcal R}\) and a factor \(k\in {\mathbb N}_0\) such that \(\sigma ' = t(\sigma )\) for \(t=(r,k)\).
Observe that configurations, transitions, guard, etc. can be encoded in linear integer arithmetic.
4.2 Safety and Liveness Specifications
The syntax of \({\textsf {ELTL}}_{\textsf {FT}}\)formulas [10]: \( pform \) defines propositional formulas, and \(\psi \) defines temporal formulas. We assume that \( Locs \subseteq {\mathcal L}\) and \( guard \in \varPhi ^{\mathrm {rise}}\cup \varPhi ^{\mathrm {fall}}\).

In [12], we have introduced a bounded model checking technique with SMT that checks reachability in counter systems of threshold automata for all combinations of the parameters. We proved that if a configuration is reachable, then there is a short schedule that reaches this configuration. As a result, bounded model checking is a complete method for reachability checking in our case. In [10], this technique was extended to \({\textsf {ELTL}}_{\textsf {FT}}\) — a fragment of \({\textsf {ELTL}}({\textsf {F}}, {\textsf {G}})\), which allows us to verify safety and liveness of counter systems of threshold automata. The syntax of \({\textsf {ELTL}}_{\textsf {FT}}\) is given in Table 1. We use this logic to express counterexamples, that is, negations of the safety and liveness specifications from above.
5 Parameterized Model Checking by Schema Enumeration
Our verification technique consists of the following steps: From the \({\textsf {ELTL}}_{\textsf {FT}}\) specifications, our tool enumerates all shapes counterexamples can have. Each of these shapes is encoded as an SMT query, and using SMT solvers, our tool checks for each shape, whether there exists a run of the system that has this shape. Such a run would then be a witness to the violation of a specification.
A different schema can be obtained by changing the order in which the two threshold guards become true. In general each possible order generates a different schema. The number of different schemas to be checked is factorial in the number of guards [12]. As our benchmarks have only a small number of guards, the number of calls to the SMT solver is still practical.
5.1 Checking a Single Lasso Schema with SMT
Our tool encodes each schema in SMT and then calls a backend solver in order to check whether the schema generates a counterexample. In [10], we explained the SMT encoding. As the schemas are independent, these checks can be done in parallel. We have implemented and exploited this feature in [15]. As [15] was concerned with synthesis, we did not discuss the effects of parallelization there. In the following we discuss and compare the sequential and the parallel approaches.
The experiments with the sequential (SEQ) and parallel (MPI) techniques on two kinds of inputs: Promela (white rows) and threshold automata (gray rows). The sequential experiments were run with GNU parallel [23] at AMD Opteron^{®} 6272, 32 cores, 192 GB. The MPI benchmarks were run at Vienna Scientific Cluster 3 using 16 nodes \(\times \) 16 cores (256 processes). The symbol “ Open image in new window ” indicates timeout of 24 h.

6 Benchmarks and Experiments
Byzantine model checker is written in OCaml. Its source code and the virtual machines are available from the tool web page^{2}. For the experiments conducted in this paper, we used Z3 4.6.0 [4] as a backend SMT solver, which was linked to ByMC via Z3 OCaml bindings.
In earlier work [9], we encoded our benchmarks in Parametric Promela, using a shared variable to record the number of processes that have sent a message, and using for each process a local variable that records how many messages a process received. For this modeling we presented a data abstraction and counter abstraction in [8]. To compare later verification techniques with these initial results, we kept that encoding, although the newer techniques rest on a more abstract model of threshold automata, which have finitely many local states.
The threshold automata constructed by data abstraction are significantly larger than threshold automata constructed by a human expert. To see the influence of these modeling decisions on the verification results, we manually encoded our benchmarks as threshold automata. These benchmarks are available from our benchmark repository ^{3}. Table 2 compares the size of the threshold automata that are: (1) produced automatically by abstraction and (2) handcoded. The essential features of the automata are: the number of local states \({\mathcal L}\), the number of rules \({\mathcal R}\), and the numbers of the guards \(\varPhi ^{\mathrm {rise}}\) and \(\varPhi ^{\mathrm {fall}}\), that is, the guards of the form \(x \ge \dots \) and \(x < \dots \) respectively. Moreover, due to data abstraction, we had to consider several cases that differ in the order between the thresholds. They are mentioned in the column “Case”.
Table 2 shows the verification results for benchmarks in Promela as well as threshold automata. We ran the sequential schema enumeration (SEQ, [10]) and the parallel schema checking technique (MPI) that is presented in this paper. The parallel experiments were run at Vienna Scientific Cluster using 256 CPU cores. For each benchmark, we picked the most challenging specifications —many of them are liveness properties —and show experimental results for them. (Needless to say, we did not run the MPI technique on the benchmarks that could be enumerated with the sequential technique in seconds.) Two columns show the essential features of the enumerated schemas: “number” displays the total number of explored schemas, and “length avg” displays the average length of schemas. For both techniques, we report the computation times and maximal memory usage during a run. For the MPI experiments, we report the average time per CPU core (column “MPI avg”) as well as the maximum time per CPU core (column “MPI max”). The deviation from the average case is negligible.
As expected, the handcoded benchmarks are usually verified much faster. Interestingly, the manually constructed threshold automaton for onestep consensus (c1cs [3]) has more threshold guards than the abstract one: We had to more accurately encode algorithm’s decisions, crash faults, and fairness. The sequential technique times out on this benchmark. The parallel technique takes about seven times longer than with the automatic abstraction.
The parallel technique benefits from running on multiple cores, though the actual gains from parallelism depend on the benchmark. As in our experiments the verification times of a single schema negligibly deviate from the average case, the uniform distribution of schemas among the nodes seems sufficient. However, one can construct threshold automata that produce schemas whose verification times significantly vary from each other. We conjecture that an implementation with a dynamic balancer would make better use of cluster resources.
7 Conclusions
We presented our tool ByMC, and compared its sequential verification implementation to its parallel one. Moreover, by experimental evaluation we showed that manual abstractions give us threshold automata that can be verified significantly faster than those that result from automatic abstraction.
We observe that the sizes of the manually constructed threshold automata are not significantly larger than the (manually crafted) models of roundbased distributed consensus presented in [17]. In their theory, thresholdguarded expressions also play a central role. Our gains in efficiency in this paper—due to manual encodings—show that the discrepancy was a result of automatic abstraction and not of the technique that uses threshold automata as its input.
We needed from one to three hours per benchmark to specify and debug a threshold automaton, while it usually took us less than 30 min to specify the same benchmark in Parametric Promela. The most difficult part of the encoding with threshold automata was to faithfully express fairness constraints over shared variables and process counters. In case of Parametric Promela, fairness constraints were much easier to write, as one could refer to the shared and local variables, which count the number of sent and received messages respectively.
Footnotes
Notes
Acknowledgments
We are grateful to our past and present collaborators Annu Gmeiner, Marijana Lazić, Ulrich Schmid, and Helmut Veith, who contributed to many of the described ideas that are now implemented in ByMC.
References
 1.Attiya, H., Welch, J.: Distributed Computing, 2nd edn. Wiley, Chichester (2004)CrossRefGoogle Scholar
 2.Barnett, M., Chang, B.Y.E., DeLine, R., Jacobs, B., Leino, K.R.M.: Boogie: a modular reusable verifier for objectoriented programs. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.P. (eds.) FMCO 2005. LNCS, vol. 4111, pp. 364–387. Springer, Heidelberg (2006). https://doi.org/10.1007/11804192_17CrossRefGoogle Scholar
 3.Brasileiro, F., Greve, F., Mostefaoui, A., Raynal, M.: Consensus in one communication step. In: Malyshkin, V. (ed.) PaCT 2001. LNCS, vol. 2127, pp. 42–50. Springer, Heidelberg (2001). https://doi.org/10.1007/3540447431_4CrossRefGoogle Scholar
 4.de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/9783540788003_24CrossRefGoogle Scholar
 5.Dobre, D., Suri, N.: Onestep consensus with zerodegradation. In: DSN, pp. 137–146 (2006)Google Scholar
 6.Guerraoui, R.: Nonblocking atomic commit in asynchronous distributed systems with failure detectors. Distrib. Comput. 15(1), 17–25 (2002)CrossRefGoogle Scholar
 7.Holzmann, G.: The SPIN Model Checker. AddisonWesley, Reading (2003)Google Scholar
 8.John, A., Konnov, I., Schmid, U., Veith, H., Widder, J.: Parameterized model checking of faulttolerant distributed algorithms by abstraction. In: FMCAD, pp. 201–209 (2013)Google Scholar
 9.John, A., Konnov, I., Schmid, U., Veith, H., Widder, J.: Towards modeling and model checking faulttolerant distributed algorithms. In: Bartocci, E., Ramakrishnan, C.R. (eds.) SPIN 2013. LNCS, vol. 7976, pp. 209–226. Springer, Heidelberg (2013). https://doi.org/10.1007/9783642391767_14CrossRefGoogle Scholar
 10.Konnov, I., Lazić, M., Veith, H., Widder, J.: A short counterexample property for safety and liveness verification of faulttolerant distributed algorithms. In: POPL, pp. 719–734 (2017)CrossRefGoogle Scholar
 11.Konnov, I., Veith, H., Widder, J.: On the completeness of bounded model checking for thresholdbased distributed algorithms: reachability. In: Baldan, P., Gorla, D. (eds.) CONCUR 2014. LNCS, vol. 8704, pp. 125–140. Springer, Heidelberg (2014). https://doi.org/10.1007/9783662445846_10CrossRefGoogle Scholar
 12.Konnov, I., Veith, H., Widder, J.: SMT and POR beat counter abstraction: parameterized model checking of thresholdbased distributed algorithms. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 85–102. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_6CrossRefGoogle Scholar
 13.Konnov, I., Veith, H., Widder, J.: What you always wanted to know about model checking of faulttolerant distributed algorithms. In: Mazzara, M., Voronkov, A. (eds.) PSI 2015. LNCS, vol. 9609, pp. 6–21. Springer, Cham (2016). https://doi.org/10.1007/9783319415796_2CrossRefzbMATHGoogle Scholar
 14.Konnov, I., Widder, J., Spegni, F., Spalazzi, L.: Accuracy of message counting abstraction in faulttolerant distributed algorithms. In: Bouajjani, A., Monniaux, D. (eds.) VMCAI 2017. LNCS, vol. 10145, pp. 347–366. Springer, Cham (2017). https://doi.org/10.1007/9783319522340_19CrossRefGoogle Scholar
 15.Lazić, M., Konnov, I., Widder, J., Bloem, R.: Synthesis of distributed algorithms with parameterized threshold guards. In: OPODIS. LIPIcs, vol. 95, pp. 32:1–32:20 (2017)Google Scholar
 16.Lynch, N.: Distributed Algorithms. Morgan Kaufman, Burlington (1996)zbMATHGoogle Scholar
 17.Marić, O., Sprenger, C., Basin, D.: Cutoff bounds for consensus algorithms. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10427, pp. 217–237. Springer, Cham (2017). https://doi.org/10.1007/9783319633909_12CrossRefGoogle Scholar
 18.Mostéfaoui, A., Mourgaya, E., Parvédy, P.R., Raynal, M.: Evaluating the conditionbased approach to solve consensus. In: DSN, pp. 541–550 (2003)Google Scholar
 19.Pease, M., Shostak, R., Lamport, L.: Reaching agreement in the presence of faults. J. ACM 27(2), 228–234 (1980)MathSciNetCrossRefGoogle Scholar
 20.Raynal, M.: A case study of agreement problems in distributed systems: nonblocking atomic commitment. In: HASE, pp. 209–214 (1997)Google Scholar
 21.Song, Y.J., van Renesse, R.: Bosco: onestep byzantine asynchronous consensus. In: Taubenfeld, G. (ed.) DISC 2008. LNCS, vol. 5218, pp. 438–450. Springer, Heidelberg (2008). https://doi.org/10.1007/9783540877790_30CrossRefGoogle Scholar
 22.Srikanth, T., Toueg, S.: Simulating authenticated broadcasts to derive simple faulttolerant algorithms. Distrib. Comput. 2, 80–94 (1987)CrossRefGoogle Scholar
 23.Tange, O., et al.: GNU parallelthe commandline power tool. USENIX Mag. 36(1), 42–47 (2011)Google Scholar
 24.Tseng, L.: Voting in the presence of byzantine faults. In: Dependable Computing (PRDC), pp. 1–10. IEEE (2017)Google Scholar