FnF-BFT: A BFT Protocol with Provable Performance Under Attack

Abstract

We introduce FnF-BFT, the first partially synchronous BFT protocol with performance guarantees under truly byzantine attacks during stable networking conditions. At its core, FnF-BFT parallelizes the execution of requests by allowing all replicas to act as leaders independently. Leader parallelization distributes the load over all replicas. Consequently, FnF-BFT fully utilizes all correct replicas’ processing power and increases throughput by overcoming the single-leader bottleneck.

We prove lower bounds on FnF-BFT ’s efficiency and performance in synchrony: the amortized communication complexity is linear in the number of replicas and thus competitive with state-of-the-art protocols; FnF-BFT ’s amortized throughput with less than $\frac{1}{3}$ byzantine replicas is at least $\frac{16}{27}$th of its best-case throughput. We also provide a proof-of-concept implementation and preliminary evaluation of FnF-BFT.

Appendices

A Analysis

We show that FnF-BFT satisfies the properties specified in Sect. 2. In particular, we prove the safety and liveness of FnF-BFT, argue that it is efficient, and evaluate its resilience to byzantine attacks in stable network conditions.

1.1 A.1 Safety

FnF-BFT generalizes Linear-PBFT [18], which is an adaptation of PBFT [9] that reduces its authenticator complexity during epoch operation. We thus rely on similar arguments to prove FnF-BFT ’s safety in Theorem 1.

Theorem 1

If any two correct replicas commit a request with the same sequence number, they both commit the same request.

Proof

We start by showing that if ${\langle \text {prepared-certificate}, sn , e, \sigma (d) \rangle }$ exists, then ${\langle \text {prepared-certificate}, sn , e, \sigma (d') \rangle }$ cannot exist for $d'\ne d$. Here, $d= h(sn \Vert e\Vert r)$ and $d'= h(sn \Vert e\Vert r')$. Further, we assume the probability of $r\ne r'$ and $d=d'$ to be negligible. The existence of ${\langle \text {prepared-certificate}, sn , e, \sigma (d) \rangle }$ implies that at least $f+1$ correct replicas sent a pre-prepare message or a prepare message for the request r with digest d in epoch e with sequence number sn. For ${\langle \text {prepared-certificate}, sn , e, \sigma (d') \rangle }$ to exist, at least one of these correct replicas needs to have sent two conflicting prepare messages (pre-prepare messages in case it leads sn). This is a contradiction.

Through the epoch-change protocol we further ensure that correct replicas agree on the sequence of requests that are committed locally in different epochs. The existence of ${\langle \text {prepared-certificate}, sn , e, \sigma (d) \rangle }$ implies that there cannot exist ${\langle \text {prepared-certificate}, sn , e', \sigma (d') \rangle }$ for $d'\ne d$ and $e'>e$. Any correct replica only commits a request with sequence number sn in epoch e if it saw the corresponding commit-certificate. For a commit-certificate for request r with digest d and sequence number sn to exist a set $R_1$ of at least $f+1$ correct replicas needs to have seen ${\langle \text {prepared-certificate}, sn , e, \sigma (d) \rangle }$. A correct replica will only accept a pre-prepare message for epoch $e'>e$ after having received a new-epoch message for epoch $e'$. Any correct new-epoch message for epoch $e'>e$ must contain epoch-change messages from a set $R_2$ of at least $f+1$ correct replicas. As there are $2f+1$ correct replicas, $R_1$ and $R_2$ intersect in at least one correct replica u. Replica u’s epoch-change message ensures that information about request r being prepared in epoch e is propagated to subsequent epochs, unless sn is already included in the stable checkpoint of its leader. In case the prepared-certificate is propagated to the subsequent epoch, a commit-certificate will potentially be propagated as well. If the new-epoch message only includes the prepared-certificate for sn, the protocol is redone for request r with the same sequence number sn. In the two other cases, the replicas commit sn locally upon seeing the new-epoch message and a correct replica will never accept a request with sequence number sn again.   $\square$

1.2 A.2 Liveness

One cannot guarantee safety and liveness for deterministic BFT protocols in asynchrony [15]. We will, therefore, show that FnF-BFT eventually makes progress after GST. In other words, we consider a stable network when discussing liveness. Furthermore, we assume that after an extended period without progress, the time required for local computation in an epoch-change is negligible. Thus, we focus on analyzing the network delays for liveness.

Definition 1

Two replicas are called synchronized, if they start their epoch-change timer for an epoch e within at most $2 \varDelta$.

Similar to PBFT [9], FnF-BFT ’s epoch-change employs three techniques to ensure that correct replicas become synchronized (Definition 1) after GST:

• 1. A replica in epoch e observing epoch-change messages from $f+1$ other replicas calling for any epoch(s) greater than e issues an epoch-change message for the smallest such epoch $e'$.

• 2. A replica only starts its epoch-change timer after receiving 2f other epoch-change messages, thus ensuring that at least $f + 1$ correct replicas have broadcasted an epoch-change message for the epoch (or higher). Hence, all correct replicas start their epoch-change timer for an epoch $e'$ within at most 2 message delay. After GST, this amounts to at most $2 \varDelta$.

• 3. Byzantine replicas are unable to impede progress by calling frequent epoch-changes, as an epoch-change will only happen if at least $f+1$ replicas call it. A byzantine primary can hinder the epoch-change from being successful. However, there can only be f byzantine primaries in a row.

Lemma 1

After GST, correct replicas eventually become synchronized.

Proof

Let u be the first correct replica to start its epoch-change timer for epoch e at time $t_0$. Following (2), this implies that u received at least 2f other epoch-change messages for epoch e (or higher). Of these 2f messages, at least f originate from other correct replicas. Thus, together with its own epoch-change message, at least $f+1$ correct replicas broadcasted epoch-change messages by time $t_0$. These $f+1$ epoch-change messages are seen by all correct replicas at the latest by time $t_0 + \varDelta$. Thus, according to (1), at time $t_0 + \varDelta$ all correct replicas broadcast an epoch-change message for epoch e. Consequently, at time $t_0 + 2\varDelta$ all correct replicas have received at least 2f other epoch-change messages and will start the timer for epoch e.   $\square$

Lemma 2

After GST, all correct replicas will be in the same epoch long enough for a correct leader to make progress.

Proof

From Lemma 1, we conclude that after GST, all correct replicas will eventually enter the same epoch if the epoch-change timer is sufficiently large. Once the correct replicas are synchronized in their epoch, the duration needed for a correct leader to commit a request is bounded. Note that all correct replicas will be in the same epoch for a sufficiently long time as the timers are configured accordingly. Additionally, byzantine replicas are unable to impede progress by calling frequent epoch-changes, according to (3).   $\square$

Theorem 2

If a correct client c broadcasts request r, then every correct replica eventually commits r.

Proof

Following Lemmas 1 and 2, we know that all correct replicas will eventually be in the same epoch after GST. Hence, in any epoch with a correct primary, the system will make progress. Note that a correct client will not issue invalid requests. It remains to show that an epoch with a correct primary and a correct leader assigned to hash bucket h(c) will occur. We note that this is given by the bucket rotation, which ensures that a correct leader repeatedly serves each bucket in a correct primary epoch.   $\square$

1.3 A.3 Efficiency

To demonstrate that FnF-BFT is efficient, we first analyze the authenticator complexity for reaching consensus during an epoch. Like Linear-PBFT [18], using each leader as a collector for partial signatures in the backup prepare and commit phase allows FnF-BFT to achieve linear complexity during epoch operation.

Lemma 3

The authenticator complexity for committing a request during an epoch is $\varTheta (n)$.

Proof

During the leader prepare phase, the authenticator complexity is at most n. The primary computes its signature to attach it to the pre-prepare message. This signature is verified by no more than $n-1$ replicas.

Furthermore, the backup prepare and commit phase’s authenticator complexity is less than 3n each. Initially, at most $n-1$ backups, compute their partial signature and send it to the leader, who, in turn, verifies 2f of these signatures. The leader then computes its partial signature, as well as computing the combined signature. Upon receiving the combined signature, the $n-1$ backups need to verify the signature.

Overall, the authenticator complexity committing a request during an epoch is thus at most $7n + o(n)\in \varTheta (n)$.   $\square$

Next, we analyze the authenticator complexity of an epoch-change. Intuitively speaking, we reduce PBFT’s view-change complexity from $\varTheta (n^3)$ to $\varTheta (n^2)$ by employing threshold signatures. However, as FnF-BFT allows for n simultaneous leaders, we obtain an authenticator complexity of $\varTheta (n^3)$ as a consequence of sharing the same information for n leaders during the epoch-change.

Lemma 4

The authenticator complexity of an epoch-change is $\varTheta (n^3)$.

Proof

The epoch-change for epoch $e+1$ is initiated by replicas sending epoch-change messages to the primary of epoch $e+1$. Each epoch-change message holds n authenticators for each leader’s last checkpoint-certificates. As there are at most 2k hanging requests per leader, further $\mathcal {O}(n)$ authenticators for prepared- and commit-certificates of the open requests per leader are included in the message. Additionally, the sending replica also includes its signature. Each replica newly computes its signature to sign the epoch-change message, the remaining authenticators are already available and do not need to be created by the replicas. Thus, a total of no more than n authenticators are computed for the epoch-change messages. Note that epoch-change messages contain $\varTheta (n)$ authenticators. Thus, the number of authenticators received by each replica is $\varTheta (n^2)$.

After the collection of $2f+1$ epoch-change messages, the primary performs a classical 3-phase reliable broadcast. The primary broadcasts the same signed message to start the classical 3-phase reliable broadcast. While the primary computes 1 signature, at most $n-1$ replicas verify this signature. In the two subsequent rounds of all-to-all communication, each participating replica computes 1 and verifies 2f signatures. Thereby, the authenticator complexity of each round of all-to-all communication is at most $(2f+1)\cdot n$. Thus, the authenticator complexity of the 3-phase reliable broadcast is bounded by $(4f+3)\cdot n\in \varTheta (n^2)$.

After successfully performing the reliable broadcast, the primary sends out a new-epoch message to every replica in the network. The new-epoch message contains the epoch-change messages held by the primary and the required pre-prepare messages for open requests. There are $\mathcal {O}(n)$ such pre-prepare messages, all signed by the primary. Finally, each new-epoch message is signed by the primary. Thus, the authenticator complexity of the new-epoch message is $\varTheta (n^2)$. However, suppose a replica has previously received and verified an epoch-change from replica u whose epoch-change message is included in the new-epoch message. In that case, the replica no longer has to check the authenticators in u’s epoch-change message again. For the complexity analysis, it does not matter when the replicas verify the signature. We assume that all replicas verify the signatures contained in the epoch-change messages before receiving the new-epoch messages. Thus, the replicas only need to verify the $\mathcal {O}(n)$ new authenticators contained in the new-epoch message.

Overall, the authenticator complexity of an epoch-change is $\varTheta ( n^3)$.    $\square$

Finally, we argue that after GST, there is sufficient progress by correct replicas to compensate for the high epoch-change cost.

Theorem 3

After GST, the amortized authenticator complexity of committing a request is $\varTheta (n)$.

Proof

To find the amortized authenticator complexity of committing a request, we consider an epoch and the following epoch-change. After GST, the authenticator complexity of committing a request for a correct leader is $\varTheta (n)$. The timeout value is set such that a correct worst-case leader creates at least $C_{\min }$ requests in each epoch initiated by a correct primary. Thus, there are $\varTheta (n)$ correct replicas, each committing $C_{\min }$ requests. By setting $C_{\min } \in \varOmega (n^2)$, we guarantee that at least $\varOmega (n^3)$ requests are created during an epoch given a correct primary.

Byzantine primaries can ensure that no progress is made in epochs they initiate, simply by withholding the new-epoch messages. However, at most a constant fraction of epochs lies in the responsibility of byzantine primaries. We conclude that, on average, $\varOmega (n^3)$ requests are created during an epoch.

Following Lemma 4, the authenticator complexity of an epoch-change is $\varTheta (n^3)$. Note that the epoch-change timeout $T_e$ is set so that correct primaries can successfully finish the epoch-change after GST. Not every epoch-change will be successful immediately, as byzantine primaries might cause unsuccessful epoch-changes. Specifically, byzantine primaries can purposefully summon an unsuccessful epoch-change to decrease efficiency.

In case of an unsuccessful epoch-change, replicas initiate another epoch-change – and continue doing so – until a successful epoch-change occurs. However, we only need to start $\mathcal {O}(1)$ epoch-changes on average to be successful after GST, as the primary rotation ensures that at least a constant fraction of primaries is correct. Hence, the average cost required to reach a successful epoch-change is $\varTheta (n^3)$.

We find the amortized request creation cost by adding the request creation cost to the ratio between the cost of a successful epoch-change and the number of requests created in an epoch, that is, $\varTheta (n) + \frac{\varTheta (n^3)}{\varOmega (n^3)}= \varTheta (n).$   $\square$

1.4 A.4 Optimistic Performance

Throughout this section, we make the following optimistic assumptions: all replicas are considered correct, and the network is stable and synchronous. We employ this model to assess the optimistic performance of FnF-BFT, i.e., theoretically evaluating its best-case throughput. Note that this scenario is motivated by practical applications, as one would hope to have functioning hardware at hand, at least initially. Additionally, we assume that the best-case throughput is limited by the available computing power of each replica – predominantly required for the computation and verification of cryptographic signatures. We further assume that the available computing power of each correct replica is the same, which we believe is realistic as the same hardware will often be employed for each replica in practice. Without loss of generality, each leader can compute/verify one authenticator per time unit. As throughput, we define the number of requests committed by the system per time unit. Finally, we assume that replicas only verify the authenticators of relevant messages. For example, a leader receiving 3f prepare messages for a request will only verify 2f authenticators. Similarly, pre-prepare messages outside the leaders’ watermarks will not be processed by backups. Note that we will carry all assumptions into the next section. There they will, however, only apply to correct replicas.

Sequential-Leader Protocols. We claim that FnF-BFT achieves higher throughput than sequential-leader protocols due to its leader parallelization. To support this claim, we compare FnF-BFT ’s throughput to that of a generic sequential-leader protocol. The generic sequential-leader protocol serves as an asymptotic characterization of several sequential-leader protocols, e.g., [9, 18, 38].

A sequential-leader protocol characteristically relies on a unique leader at any point in time (Fig. 4). During its reign, the leader is responsible for serving all client requests. The leader can be rotated repeatedly or only upon failure.

Lemma 5

A sequential-leader protocol requires at least $\varOmega (n)$ time units to process a client request.

Proof

In sequential-leader protocols, a unique replica is responsible for serving all client requests at any point in time. This replica must verify $\varOmega (n)$ signatures to commit a request while no other replica leads requests simultaneously. Thus, a sequential-leader protocol requires $\varOmega (n)$ time units to process a request.    $\square$

Fig. 4.

Sequential leader example with four leaders taking turns in serving client requests. Leader changes are indicated by vertical lines.

FnF-BFT Epoch. With FnF-BFT, we propose a parallel-leader protocol that divides client requests into $m\cdot n$ independent hash buckets. Each hash bucket is assigned to a unique leader at any time (Fig. 5). The hash buckets are rotated between leaders across epochs to ensure liveness (cf. Sect. 3.3). Within an epoch, a leader is only responsible for committing client requests from its assigned hash bucket(s). Overall, this parallelization leads to a significant speed-up.

Fig. 5.

Parallel leader example with four leaders and four hash buckets. In each epoch, leaders are only responsible for serving client requests in their hash bucket. Epoch-changes are indicated by vertical lines.

To show the speed-up gained through parallelization, we first analyze the optimistic epoch throughput of FnF-BFT, i.e., the throughput of the system during stable networking conditions in a best-case scenario with $3f + 1$ correct replicas. Furthermore, we assume the number of requests included in a checkpoint to be sufficiently large, such that no leader must ever stall when waiting for a checkpoint to be created. Finally, we analyze the effects of epoch-changes and compute the overall best-case throughput of FnF-BFT in the aforementioned optimistic setting.

Lemma 6

After GST, the best-case epoch throughput with $3f+1$ correct replicas is $\dfrac{k\cdot (3f+1)}{k\cdot (19f+3)+(8f+2)}.$

Proof

In the optimistic setting, all epochs are initiated by correct primaries, and thus all replicas will be synchronized after GST.

In FnF-BFT, n leaders work on client requests simultaneously. Similar to sequential-leader protocols, each leader needs to verify at least $\mathcal {O}(n)$ signatures to commit a request. A leader needs to compute 3 and verify 4f authenticators precisely to commit a request it proposes during epoch operation. Thus, leaders need to process a total of $4f+3\in \varTheta (n)$ signatures to commit a request. With the help of threshold signatures, backups involved in committing a request only need to compute 2 and verify 3 authenticators. We follow that a total of $4f+3 +5\cdot 3f=19f+3$ authenticators are computed/verified by a replica for one of its own requests and 3f requests of other leaders.

After GST, each correct leader v will quickly converge to a $C_v$ such that it will make progress for the entire epoch-time, hence, working at its full potential. We achieve this by rapidly increasing the number of requests assigned to each leader outperforming its assignment and never decreasing the assignment below what the replica recently managed.

Checkpoints are created every k requests and add to the computational load. A leader verifies and computes a total of $2f+2$ messages to create a checkpoint, and the backups are required to compute 1 partial signature and verify 1 threshold signature. The authenticator cost of creating $3f+1$ checkpoints, one for each leader, is, therefore, $8f+2$ per replica.

Thus, the best-case throughput of the system is

$$\dfrac{ k\cdot (3f+1)}{k\cdot (19f+3)+(8f+2)}.$$

   $\square$

Note that it would have been sufficient to show that the epoch throughput is $\varOmega (1)$ per time unit, but this more precise formula will be required in the next section. Additionally, we would like to point out that the choice of k does not influence the best-case throughput asymptotically.

FnF-BFT Epoch-Change. As FnF-BFT employs bounded-length epochs, repeated epoch-changes have to be considered. In the following, we will show that FnF-BFT ’s throughput is dominated by its authenticator complexity during the epochs. To that end, observe that for $C_{\min } \in \varOmega (n^2)$, every epoch will incur an authenticator complexity of $\varOmega (n^3)$ per replica and thus require $\varOmega (n^3)$ time units.

Lemma 7

After GST, an epoch-change under a correct primary requires $\varTheta (n^2)$ time units.

Proof

Following Lemma 4, the number of authenticators computed and verified by each replica for all epoch-change messages is $\varTheta (n^2)$. Each replica also processes $\varTheta (n)$ signatures during the reliable broadcast, and $\mathcal {O}(n)$ signatures for the new-epoch messages. Overall, each replica thus processes $\varTheta (n^2)$ authenticators during the epoch-change. Subsequently, this implies that the epoch-change requires $\varTheta (n^2)$ time units, as we require only a constant number of message delays to initiate and complete the epoch-change protocol. Recall that we assume the throughput to be limited by the available computing power of each replica.    $\square$

Theoretically, one could set $C_{\min }$ even higher such that the time the system spends with epoch-changes becomes negligible. However, there is a trade-off for practical reasons: increasing $C_{\min }$ increases the minimal epoch-length, allowing a byzantine primary to slow down the system for a longer time. Note that the guarantee for byzantine-resilient performance (cf. Sect. A.5) would still hold.

FnF-BFT Optimistic Performance. Ultimately, it remains to quantify FnF-BFT ’s overall best-case throughput.

Lemma 8

After GST, and assuming all replicas are correct, FnF-BFT requires $\mathcal {O}(n)$ time units to process n client requests on average.

Proof

Under a correct primary, each correct leader will commit at least $C_{\min } \in \varOmega (n^2)$ requests after GST. Hence, FnF-BFT will spend at least $\varOmega (n^3)$ time units in an epoch, while only requiring $\varTheta (n^2)$ time units for an epoch-change (Lemma 7). Thus, following Lemma 6, FnF-BFT requires an average of $\mathcal {O}(n)$ time units to process n client requests.   $\square$

Following Lemmas 5 and 8, the speed-up of a parallel-leader protocol over a sequential-leader protocol is proportional to the number of leaders.

Theorem 4

If the throughput is limited by the (equally) available computing power at each replica, the speed-up for equally splitting requests between n parallel leaders over a sequential-leader protocol is at least $\varOmega (n)$.

1.5 A.5 Byzantine-Resilient Performance

While many BFT protocols present practical evaluations of their performance that neglect truly byzantine adversarial behavior [9, 18, 35, 38], we provide a novel, theory-based byzantine-resilience guarantee. We first analyze the impact of byzantine replicas in an epoch under a correct primary. Next, we discuss the potential strategies of a byzantine primary trying to stall the system. And finally, we conflate our observations into a concise statement.

Correct Primary Throughput. To gain insight into the byzantine-resilient performance, we analyze the optimal byzantine strategy. In epochs led by correct primaries, we will consider their roles as backups and leaders separately. On the one hand, for a byzantine leader, the optimal strategy is to leave as many requests hanging, while not making any progress (Lemma 9).

Lemma 9

After GST and under a correct primary, the optimal strategy for a byzantine leader is to leave 2k client requests hanging and commit no request.

Proof

Correct replicas will be synchronized as a correct primary initiates the epoch. Thus, byzantine replicas’ participation is not required to make progress.

Byzantine leaders can follow the protocol accurately (at any chosen speed), send messages that do not comply with the protocol, or remain unresponsive.

Hanging requests reduce the throughput as they increase the number of authenticators shared during the epoch and the epoch-change. Hence, byzantine leaders leave the maximum number of requests hanging, i.e., 2k requests as all further prepare messages would be discarded by correct replicas.

While byzantine replicas cannot hinder correct leaders from committing requests, committing any request can only benefit the throughput of FnF-BFT. To that end, note that after GST, each correct leader v will converge to a $C_v$ such that it will make progress during the entire epoch-time; hence, prolonging the epoch-time is impossible. The optimal strategy for byzantine leaders is thus to stall progress on their assigned hash buckets.

Finally, note that we assume the threshold signature scheme to be robust and can, therefore, discard any irrelevant message efficiently.   $\square$

On the other hand, as a backup, the optimal byzantine strategy is not helping other leaders to make progress (Lemma 10).

Lemma 10

Under a correct primary, the optimal strategy for a byzantine backup is to remain unresponsive.

Proof

Byzantine participation in the protocol can only benefit the correct leaders’ throughput. Invalid messages can simply be ignored, while additional authenticators are not verified and thus do not reduce the system throughput.    $\square$

In conclusion, we observe that byzantine replicas have little opportunity to reduce the throughput in epochs under a correct primary.

Theorem 5

After GST, the effective utilization under a correct primary is at least $\frac{8}{9}$ for $n \rightarrow \infty$.

Proof

Moving from the best-case scenario with $3f+1$ correct leaders to only $2f+1$ correct leaders, each correct leader still processes $4f+3$ authenticators per request, and 5 authenticators for each request of other leaders. We know from Lemma 9 that only the $2f+1$ correct replicas are committing requests and creating checkpoints throughout the epoch. The authenticator cost of creating $2f+1$ checkpoints, one for each correct leader, becomes $6f+2$ per replica.

Byzantine leaders can open at most 2k new requests in an epoch. Each hanging request is seen at most twice by correct replicas without becoming committed. Thus, each correct replica processes no more than 8k authenticators for requests purposefully left hanging by a byzantine replica in an epoch. Thus, the utilization is reduced at most by a factor $\left( 1-\frac{8 k f}{T}\right)$, where T is the maximal epoch length. While epochs can finish earlier, this will not happen after GST as soon as each correct leader v works at its capacity $C_v$.

Hence, the byzantine-resilient epoch throughput becomes

$$\frac{k\cdot (2f+1)}{k\cdot (14f+3)+(6f+2)} \cdot \left( 1-\frac{8 kf}{T}\right) .$$

By comparing this to the best-case epoch throughput from Lemma 6, we obtain a maximal throughput reduction of

$$\begin{aligned} \frac{ (2f+1)(k \cdot (19f+3)+(8f+2))}{(3f+1)(k \cdot (14f+3)+(6f+2))}\cdot \left( 1-\frac{8 k f}{ T}\right) . \end{aligned}$$

Observe that the first term decreases and approaches $\frac{8}{9}$ for $n \rightarrow \infty$:

$$\begin{aligned} \frac{(2f+1)(k \cdot (19f+3)+(8f+2))}{(3f+1)(k \cdot (14f+3)+(6f+2))}{\mathop {=}\limits ^{n\rightarrow \infty }}\frac{16 + 38 k}{18 + 42 k }\ge \frac{8}{9}. \end{aligned}$$

We follow that the epoch time is $T \in \varOmega (n^3)$, as we set $C_{\min }\in \varOmega (n^2)$ and each leader requires $\varOmega (n)$ time units to commit one of its requests. Additionally, we know that $8 k f\in \mathcal {O}(n)$, and thus: $\left( 1-\dfrac{8 k f}{ T}\right) {\mathop {=}\limits ^{n\rightarrow \infty }}1.$

For $n\rightarrow \infty$, the throughput reduction byzantine replicas can impose on the system during a synchronized epoch is therefore bounded by a factor $\dfrac{8}{9}$.   $\square$

Byzantine Primary Throughput. A byzantine primary, evidently, aims to perform the epoch-change as slow as possible. Furthermore, a byzantine primary can impede progress in its assigned epoch entirely, e.g., by remaining unresponsive. We observe that there are two main byzantine strategies to be considered.

Lemma 11

Under a byzantine primary, an epoch is either aborted quickly or $\varOmega (n^3)$ new requests become committed.

Proof

A byzantine adversary controlling the primary of an epoch has three options. Following the protocol and initiating the epoch for all $2f+1$ correct replicas will ensure high throughput and is thus not optimal. Alternatively, initiating the epoch for $s \in [f+1, 2f]$ correct replicas will allow the byzantine adversary to control the progress made in the epoch, as no correct leader can make progress without a response from at least one byzantine replica. However, slow progress can only be maintained as long as at least $2f + 1$ leaders continuously make progress. By setting the no-progress timeout $T_p\in \varTheta (T/C_{\min })$, $\varOmega (n^3)$ new requests per epoch can be guaranteed. In all other scenarios, the epoch will be aborted after at most one epoch-change timeout $T_e$, the initial message transmission time $5\varDelta$, and one no-progress timeout $T_p$.

Note that we do not increase the epoch-change timer $T_e$ for f unsuccessful epoch-changes in a row. In doing so, we prevent f consecutive byzantine primaries from increasing the epoch-change timer exponentially; thus potentially reducing the system throughput significantly.   $\square$

FnF-BFT Primaries. Primaries rotate across epochs based on their performance history to reduce the control of the byzantine adversary on the system.

Lemma 12

After a sufficiently long stable time period, the performance of a byzantine primary can only drop below the performance of the worst correct primary once throughout the sliding window.

Proof

The network is considered stable for a sufficiently long time when all leaders work at their capacity limit, i.e., the number of requests they are assigned in an epoch matches their capacity, and primaries have subsequently been explored once. As soon as all leaders are working at their capacity limit, we observe the representative performance of all correct primaries, at least.

FnF-BFT repeatedly cycles through the $2f+1$ best primaries. A primary’s performance is based on its last turn as primary. Consequently, a primary is removed from the rotation as soon as its performance drops below one of the f remaining primaries. We conclude that a byzantine primary will only be nominated beyond its single exploration throughout the sliding window if its performance matches at least the performance of the worst correct primary.   $\square$

As its successor determines a primary’s performance, the successor can influence the performance slightly. However, this is bounded by the number of open requests – $\mathcal {O}(n)$ many – which we consider being well within natural performance variations, as $\varOmega (n^3)$ requests are created in an epoch under a correct primary. Thus, we will disregard possible performance degradation originating at the succeeding primary.

From Lemma 12, we easily see that the optimal strategy for a byzantine primary is to act according to Lemma 11 – performing better would only help the system. In a stable network, byzantine primaries will thus only have one turn as primary throughout any sliding window. In the following, we consider a primary to be behaving byzantine if it performs worse than all correct primaries.

Theorem 6

After the system has been in stability for a sufficiently long time period, the fraction of byzantine behaving primaries is $\frac{f}{g}$.

Proof

Following from Lemma 12, we know that a primary can only behave byzantine once in the sliding window. There are a total of g epochs in a sliding window, and the f byzantine replicas in the network can only act byzantine in one epoch included in the sliding window. We see that the fraction of byzantine behaving primaries is $\frac{f}{g}$.   $\square$

The configuration parameter g determines the fraction of byzantine primaries in the system’s stable state, while simultaneously dictating how long it takes to get there after GST. Setting g to a small value ensures that the system quickly recovers from asynchrony. On the other hand, setting g to larger values provides near-optimal behavior once the system is operating at its optimum.

FnF-BFT Byzantine-Resilient Performance. Combining the byzantine strategies from Theorem 5, Lemma 11 and Theorem 6, we obtain the following.

Theorem 7

After GST, the effective utilization is asymptotically $\frac{8}{9} \cdot \frac{g-f}{g}$ for $n \rightarrow \infty$.

Proof

To estimate the effective utilization, we only consider the throughput within epochs. That is because the time spent in correct epochs dominates the time for epoch-changes, as well as the time for failed epoch-changes under byzantine primaries, as the number of replicas increases (Lemma 7). Without loss of generality, we consider no progress to be made in byzantine primary epochs. We make this assumption, as we cannot guarantee asymptotically significant throughput. From Theorem 5, we know that in an epoch initiated by a correct primary, the byzantine-resilient effective utilization is at least $\frac{8}{9}$ for $n \rightarrow \infty$. Further, at least $\frac{g-f}{g}$ of the epochs are led by correct primaries after a sufficiently long time period in stability and thus obey this bound (Theorem 6). In the limit for $n \rightarrow \infty$ the effective utilization is $\frac{8}{9} \cdot \frac{g-f}{g}$.   $\square$

B Implementation and Preliminary Evaluation

Features. FnF-BFT ’s proof-of-concept implementation is directly based on the code of HotStuff’s open-source prototype implementation libhotstuff.³ We implement the basic functionality of FnF-BFT including the epoch-change and watermarks, while only changing $\approx 2000$ lines of code and maintaining the general framework and experiment setup. In addition, we extend both implementations to support BLS threshold signatures.⁴

Threshold Signatures. Note that while HotStuff is designed with threshold signatures in mind and relies on them for its theoretic performance analysis [38], libhotstuff uses sets of $2f + 1$ signatures instead of real threshold signatures. While this workaround maintains a complexity of $\mathcal {O}(n)$ for creation of such a “threshold signature”, it comes at the expense of a verification complexity of $\mathcal {O}(n)$ as well. In HotStuff, this additional overhead affects mainly the non-primary replicas, which would otherwise be idle in the HotStuff protocol. However, the design of FnF-BFT ensures that all replicas’ computational resources are utilized at all times. Since the originally used secp256k1 cryptographic signatures appear to be more optimized than the BLS threshold signatures, and to ensure a fair comparison, we thus compare FnF-BFT ’s throughput and latency to HotStuff using the identical BLS threshold signature implementation.

Limitations. As in the theoretical analysis (see Sect. 3.1), we did not implement a batching process for client requests. Hence, each block contains only a single client request. For this reason, the expected throughput (and typically reported number for BFT protocols) in practical deployments is much higher.

While FnF-BFT ’s design inherently allows to utilize concurrent threads during epoch operation, e.g., to interact with all parallel leaders, our proof-of-concept implementation currently only supports single-threaded operation.

Setup and Methodology. We compare FnF-BFT ’s single-threaded implementation to both single- and multi-threaded HotStuff (with 12 threads per replica) with respect to best-case performance, i.e., throughput and latency when all n replicas operate correctly. Experiments are repeated for $n \in \{4, 7, 10, 16, 31, 61\}$ replicas. We deploy both protocols on Amazon EC2 using c5.4xlarge AWS cloud instances. Each replica is assigned to a dedicated VM instance with 16 CPU cores powered by Intel Xeon Platinum 8000 processors clocked at up to 3.6 GHz, 32 GB of RAM, and a maximum network bandwidth of 10 Gigabits per second.

We measure the average throughput and latency over multiple epochs to include the expected drop in performance for FnF-BFT during the epoch-change. For each experiment, we run both protocols for at least three minutes and measure their average performance accordingly. We divide the hash space into n buckets, resulting in one bucket per replica. For generating requests, we run the libhotstuff client with its default settings, meaning that the payload of each request is empty. Clients generate and broadcast four requests in parallel, and issue a new request whenever one of their requests is answered. For the throughput measurements, we launch sufficiently many clients until we observe that no buckets are idle. For the latency measurement, we run a single client instance such that the system does not operate at its throughput limit. In general, we use the same settings for both protocols wherever applicable (Table 1).

Table 1. Experiment parameters used for FnF-BFT and HotStuff, if applicable.

Performance. Figure 6 depicts a best-case operation of FnF-BFT over five epochs and demonstrates its consistently high throughput.⁵ As expected, the throughput of our protocol stalls during an epoch-change. However, in comparison to HotStuff (Fig. 7), FnF-BFT ’s average throughput remains on top over multiple epochs (i.e., including the epoch-change gap). As HotStuff’s throughput decreases for increasing number of replicas, FnF-BFT showcases its superior scalability. Specifically, FnF-BFT handles large amounts of requests up to 4.7$\times$ faster than multi-threaded and 16$\times$ faster than single-threaded HotStuff.

Figure 8 depicts the average latency of both FnF-BFT and HotStuff, showing that they scale similarly with the number of replicas. As latency expresses the time between a request being issued and committed, both protocols exhibit very fast finality for requests on average, even with many replicas. In combination, Fig. 7 and Fig. 8 demonstrate the high performance and competitiveness of FnF-BFT with HotStuff, especially when scaling to many replicas.

Fig. 6.

Throughput of FnF-BFT with $n = 4$ replicas over 5 epochs.

Fig. 7.

Average Throughput Comparison

Fig. 8.

Average Latency Comparison