Skip to main content
Log in

Liveness and latency of Byzantine state-machine replication

  • Published:
Distributed Computing Aims and scope Submit manuscript

Abstract

Byzantine state-machine replication (SMR) ensures the consistency of replicated state in the presence of malicious replicas and lies at the heart of the modern blockchain technology. Byzantine SMR protocols often guarantee safety under all circumstances and liveness only under synchrony. However, guaranteeing liveness even under this assumption is nontrivial. So far we have lacked systematic ways of incorporating liveness mechanisms into Byzantine SMR protocols, which often led to subtle bugs. To close this gap, we introduce a modular framework to facilitate the design of provably live and efficient Byzantine SMR protocols. Our framework relies on a view abstraction generated by a special SMR synchronizer primitive to drive the agreement on command ordering. We present a simple formal specification of an SMR synchronizer and its bounded-space implementation under partial synchrony. We also apply our specification to prove liveness and analyze the latency of three Byzantine SMR protocols via a uniform methodology. In particular, one of these results yields what we believe is the first rigorous liveness proof for the algorithmic core of the seminal PBFT protocol.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Notes

  1. In PBFT this information is sent in VIEW-CHANGE messages, which also play a role similar to \( \texttt {WISH}\) messages in our synchronizer (Fig. 3). In PBFT-light we opted to eschew VIEW-CHANGE messages to maintain a clear separation between view synchronization internals and the SMR protocol.

  2. Recall that even a single \( \texttt {advance}\) call at a correct process may lead to a view switch (§3). For example, with the synchronizer in Fig. 3 this may happen as follows. The \( \texttt {advance}\) call at a process generates \( \texttt {WISH}(v+1)\) (line 2) and the Byzantine processes produce another f copies of the same message. Correct processes receiving the resulting \(f+1\) copies of the message relay it via line 8, which yields \(2f+1\) copies in total. The synchronizer then triggers \(\texttt {new\_view}(v+1)\) notifications at correct processes that receive all these copies (line 15), causing them to enter \(v+1\) without increasing their timeouts.

  3. The bug was acknowledged by the textbook authors [18].

References

  1. Abraham, I., Devadas, S., Dolev, D., Nayak, K., Ren, L.: Synchronous Byzantine agreement with expected \({O}(1)\) rounds, expected \({O}(n^2)\) communication, and optimal resilience. In: Conference on Financial Cryptography and Data Security (FC) (2019)

  2. Abraham, I., Gueta, G., Malkhi, D., Alvisi, L., Kotla, R., Martin, J.-P.: Revisiting fast practical Byzantine fault tolerance (2017). arXiv:1712.01367

  3. Abraham, I., Nayak, K., Ren, L., Xiang, Z.: Good-case latency of Byzantine broadcast: a complete categorization. In: Symposium on Principles of Distributed Computing (PODC) (2021)

  4. Aştefănoaei, L., Chambart, P., Del Pozzo, A., Rieutord, T., Tucci-Piergiovanni, S., Zălinescu, E.: Tenderbake-a solution to dynamic repeated consensus for blockchains. In: Symposium on Foundations and Applications of Blockchain (FAB) (2021)

  5. Alistarh, D., Gilbert, S., Guerraoui, R., Travers, C.: Generating fast indulgent algorithms. In: International Conference on Distributed Computing and Networking (ICDCN) (2011)

  6. Amoussou-Guenou, Y., Del Pozzo, A., Potop-Butucaru, M., Tucci-Piergiovanni, S.: Correctness of Tendermint-core blockchains. In: Conference on Principles of Distributed Systems (OPODIS) (2018)

  7. Amoussou-Guenou, Y., Del Pozzo, A., Potop-Butucaru, M., Tucci-Piergiovanni, S.: Dissecting tendermint. In: Conference on Networked Systems (NETYS) (2019)

  8. Awerbuch, Baruch: Complexity of network synchronization. J. ACM 32(4), 804–823 (1985)

    Article  MathSciNet  Google Scholar 

  9. Bazzi, R.A., Ding, Y.: Non-skipping timestamps for Byzantine data storage systems. In: Symposium on Distributed Computing (DISC) (2004)

  10. Berger, C., Reiser, H.P., Bessani, A.: Making reads in BFT state machine replication fast, linearizable, and live. In: Symposium on Reliable Distributed Systems (SRDS) (2021)

  11. Bessani, A.N., Sousa, J., Adílio Pelinson, R.M., Alchieri, E.: State machine replication for the masses with BFT-SMART. In: Conference on Dependable Systems and Networks (DSN) (2014)

  12. Biely, M., Widder, J., Charron-Bost, B., Gaillard, A., Hutle, M., Schiper, A.: Tolerating corrupted communication. In: Symposium on Principles of Distributed Computing (PODC) (2007)

  13. Bracha, G.: Asynchronous Byzantine agreement protocols. Inf. Comput. 75(2), 130–143 (1987)

    Article  MathSciNet  Google Scholar 

  14. Bravo, M., Chockler, G., Gotsman, A.: Making Byzantine consensus live. In: Symposium on Distributed Computing (DISC) (2020)

  15. Bravo, M., Chockler, G., Gotsman, A.: Liveness and latency of Byzantine state-machine replication (extended version) (2022). arXiv:2202.06679. https://arxiv.org/abs/2202.06679

  16. Bravo, M., Chockler, G., Gotsman, A.: Making Byzantine consensus live. Distrib. Comput. 35(6), 503–532 (2022)

    Article  MathSciNet  Google Scholar 

  17. Buchman, E., Kwon, J., Milosevic, Z.: The latest gossip on BFT consensus (2018). arXiv:1807.04938

  18. Cachin, C.: Personal communication (2022)

  19. Cachin, C., Guerraoui, R., Rodrigues, L.: Introduction to Reliable and Secure Distributed Programming, 2nd edn. Springer, Berlin (2011)

    Book  Google Scholar 

  20. Cachin, C., Kursawe, K., Petzold, F., Shoup, V.: Secure and efficient asynchronous broadcast protocols. In: International Cryptology Conference (CRYPTO) (2001)

  21. Cachin, C., Vukolić, M.: Blockchain consensus protocols in the wild (keynote talk). In: Symposium on Distributed Computing (DISC) (2017)

  22. Castro, M.: Practical Byzantine Fault Tolerance. PhD thesis, Massachusetts Institute of Technology (2001)

  23. Castro, M., Liskov, B.: Practical Byzantine fault tolerance. In: Symposium on Operating Systems Design and Implementation (OSDI) (1999)

  24. Castro, M., Liskov, B.: Practical Byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20(4), 398–461 (2002)

    Article  Google Scholar 

  25. Chandra, T.D., Toueg, S.: Unreliable failure detectors for reliable distributed systems. J. ACM 43(2), 225–267 (1996)

    Article  MathSciNet  Google Scholar 

  26. Charron-Bost, B., Schiper, A.: The Heard-Of model: computing in distributed systems with benign faults. Distrib. Comput. 22(1), 49–71 (2009)

    Article  Google Scholar 

  27. Civit, P., Dzulfikar, M.A., Gilbert, S., Gramoli, V., Guerraoui, R., Komatovic, J., Vidigueira, M.: Byzantine consensus is \(\theta (n^2)\): the Dolev-Reischuk bound is tight even in partial synchrony! In: Symposium on Distributed Computing (DISC) (2022)

  28. Clement, A., Wong, E., Alvisi, L., Dahlin, M., Marchetti, M.: Making Byzantine fault tolerant systems tolerate Byzantine faults. In: Symposium on Networked Systems Design and Implementation (NSDI) (2009)

  29. DiemBFT v4: state machine replication in the Diem blockchain. https://developers.diem.com/papers/diem-consensus-state-machine-replication-in-the-diem-blockchain/2021-08-17.pdf

  30. Dolev, D., Halpern, J.Y., Simons, B., Strong, R.: Dynamic fault-tolerant clock synchronization. J. ACM 42(1), 143–185 (1995)

    Article  Google Scholar 

  31. Doudou, A., Garbinato, B., Guerraoui, R.: Abstractions for devising Byzantine-resilient state machine replication. In: Symposium on Reliable Distributed Systems (SRDS) (2000)

  32. Dragoi, C., Widder, J., Zufferey, D.: Programming at the edge of synchrony. Proc. ACM Program. Lang. 4(OOPSLA), 1–30 (2020)

    Article  Google Scholar 

  33. Dwork, C., Lynch, N.A., Stockmeyer, L.J.: Consensus in the presence of partial synchrony. J. ACM 35(2), 288–323 (1988)

    Article  MathSciNet  Google Scholar 

  34. Fischer, M.J., Lynch, N.A., Paterson, M.: Impossibility of distributed consensus with one faulty process. J. ACM 32(2), 374–382 (1985)

    Article  MathSciNet  Google Scholar 

  35. Freiling, F.C., Guerraoui, R., Kuznetsov, P.: The failure detector abstraction. ACM Comput. Surv. 43(2), 9:1-9:40 (2011)

    Article  Google Scholar 

  36. Gafni, E.: Round-by-round fault detectors: unifying synchrony and asynchrony. In: Symposium on Principles of Distributed Computing (PODC) (1998)

  37. Gilbert, S., Guerraoui, R., Kowalski, D.R.: On the message complexity of indulgent consensus. In: Symposium on Distributed Computing (DISC) (2007)

  38. Golan-Gueta, G., Abraham, I., Grossman, S., Malkhi, D., Pinkas, B., Reiter, M.K., Seredinschi, Dragos-Adrian, Tamir, Orr, Tomescu, Alin: SBFT: A scalable and decentralized trust infrastructure. In: Conference on Dependable Systems and Networks (DSN) (2019)

  39. Guerraoui, R.: Indulgent algorithms (preliminary version). In: Symposium on Principles of Distributed Computing (PODC) (2000)

  40. Guerraoui, R., Raynal, M.: The information structure of indulgent consensus. IEEE Trans. Comput. 53(4), 453–466 (2004)

    Article  Google Scholar 

  41. Haeberlen, A., Kuznetsov, P.: The fault detection problem. In: Conference on Principles of Distributed Systems (OPODIS) (2009)

  42. Herzberg, A., Kutten, S.: Fast isolation of arbitrary forwarding faults. In: Symposium on Principles of Distributed Computing (PODC) (1989)

  43. Incorrect by construction-CBC Casper isn’t live. https://derekhsorensen.com/docs/CBC_Casper_Flaw.pdf

  44. Keidar, I., Shraer, A.: Timeliness, failure-detectors, and consensus performance. In: Symposium on Principles of Distributed Computing (PODC) (2006)

  45. Kihlstrom, K.P., Moser, L.E., Melliar-Smith, P.M.: Byzantine fault detectors for solving consensus. Comput. J. 46(1), 16–35 (2003)

    Article  Google Scholar 

  46. Kotla, R., Alvisi, L., Dahlin, M., Clement, A., Wong, E.: Zyzzyva: speculative byzantine fault tolerance. ACM Trans. Comput. Syst. 27(4), 7:1-7:39 (2010)

    Google Scholar 

  47. Lamport, L.: The part-time parliament. ACM Trans. Comput. Syst. 16(2), 133–169 (1998)

    Article  Google Scholar 

  48. Malkhi, D., Reiter, M.: Unreliable intrusion detection in distributed computations. In: Workshop on Computer Security Foundations (CSFW) (1997)

  49. Mostéfaoui, A., Raynal, M.: Solving consensus using Chandra-Toueg’s unreliable failure detectors: A general quorum-based approach. In: Symposium on Distributed Computing (DISC) (1999)

  50. Naor, O., Baudet, M., Malkhi, D., Spiegelman, A.: Cogsworth: Byzantine view synchronization. In: Cryptoeconomics Systems Conference (CES) (2020)

  51. Naor, O., Keidar, I.: Expected linear round synchronization: the missing link for linear Byzantine SMR. In: Symposium on Distributed Computing (DISC) (2020)

  52. Pass, R., Shi, E.: Hybrid consensus: efficient consensus in the permissionless model. In: Symposium on Distributed Computing (DISC) (2017)

  53. Pass, R., Shi, E.: Thunderella: blockchains with optimistic instant confirmation. In: Conference on the Theory and Applications of Cryptographic Techniques (EUROCRYPT) (2018)

  54. Schneider, F.B.: Implementing fault-tolerant services using the state machine approach: a tutorial. ACM Comput. Surv. 22(4), 299–319 (1990)

    Article  Google Scholar 

  55. Simons, B., Welch, J., Lynch, N.: An overview of clock synchronization. In: Fault-Tolerant Distributed Computing (1986)

  56. Sousa, J.: Byzantine State Machine Replication for the Masses. PhD thesis, University of Lisbon (2017)

  57. Stathakopoulou, C., David, T., Vukolić, M.: Mir-BFT: high-throughput BFT for blockchains (2019). arXiv:1906.05552

  58. Veronese, G.S., Correia, M., Bessani, A.N., Lung, L.C.: Spin one’s wheels? Byzantine fault tolerance with a spinning primary. In: Symposium on Reliable Distributed Systems (SRDS) (2009)

  59. Yin, M., Malkhi, D., Reiter, M.K., Golan-Gueta, G., Abraham, I.: HotStuff: BFT consensus with linearity and responsiveness. In: Symposium on Principles of Distributed Computing (PODC), (2019)

Download references

Funding

This research was partially supported by the European Research Council (Starting Grant RACCOON), Nomadic Labs/Tezos Foundation, the Spanish Ministry of Science and Innovation (projects PRODIGY, DECO and BYZANTIUM), and the CHIST-ERA network (project REDONDA).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alexey Gotsman.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Constructing a consensus synchronizer from an SMR synchronizer

We now show that we can use an SMR synchronizer presented in this paper to implement a consensus synchronizer [16, 51] without extra overhead, thereby demonstrating the generality of the former abstraction. We first recap the interface and specification of a consensus synchronizer. This synchronizer produces notifications \(\texttt {new\_consensus\_view} (v)\) at at each correct process, telling it to enter view v. A process can ensure that the consensus synchronizer has started operating by calling a special \(\texttt {start}()\) function. We assume that each correct process eventually calls \(\texttt {start}()\), unless it gets a \(\texttt {new\_consensus\_view}\) notification first.

For a consensus protocol to terminate, its processes need to stay in the same view for long enough to complete the message exchange leading to a decision. Since the message delay \(\delta \) after \( \textsf {GST}\) is unknown to the consensus protocol, we need to increase the view duration until it is long enough for the protocol to terminate. To this end, the synchronizer is parameterized by a function defining this duration – \(F: \textsf {View}\cup \{0\} \rightarrow \textsf {Time}\), which is monotone, satisfies \(F(0) = 0\), and increases unboundedly. The latter is formalized as follows:

$$\begin{aligned} \forall \theta .\,\exists v.\,\forall v'.\, v'\ge v \implies F(v')>\theta . \end{aligned}$$
(9)

Figure 8 presents the specification of a consensus synchronizer proposed in [16]. This relies on the following notation, analogous to the one used for SMR synchronizers. Given a view v for which a correct process \(p_i\) received a \(\texttt {new\_consensus\_view}(v)\) notification, we denote by \({\mathbb {E}}_{i}(v)\) the time when this happens; we let \({\mathbb {E}}_{{\textrm{first}}}(v)\) and \({\mathbb {E}}_{\textrm{last}}(v)\) denote respectively the earliest and the latest time when some correct process receives a \(\texttt {new\_consensus\_view}(v)\) notification. Like an SMR synchronizer, a consensus synchronizer must guarantee that views only increase at a given process (Property I). A consensus synchronizer ensures view synchronization only starting from some view \({\mathcal {V}}\), entered after \( \textsf {GST}\) (Property II). Starting from \({\mathcal {V}}\), correct processes do not skip any views (Property III), enter each view \(v \ge {\mathcal {V}}\) within at most d of each other (Property IV) and stay there until at least \(F(v)\) after the first process enters v (Property V).

Fig. 8
figure 8

Consensus synchronizer specification [16], holding for some \({\mathcal {V}}\in \textsf {View}\)

Fig. 9
figure 9

A consensus synchronizer from an SMR synchronizer

Figure 9 shows how we can construct a consensus synchronizer from an SMR synchronizer by generalizing the simple client in Fig. 2. Upon a \(\texttt {start}()\) call, the consensus synchronizer just tells the underlying SMR synchronizer to advance (line 1). When the SMR synchronizer produces a \(\texttt {new\_view}(v)\) notification (line 3), the consensus synchronizer immediately produces the corresponding \(\texttt {new\_consensus\_view}(v)\) notification (thus, we always have \({\mathbb {E}}_{i}(v) = E_{i}(v)\) and use the two interchangeably in the proofs below). The synchronizer also sets a timer \(\textsf {timer\_view}\) for the duration F(v). When the timer expires (line 7), the consensus synchronizer tells the SMR synchronizer to advance. We now prove that this construction is correct.

Theorem 6

The consensus synchronizer in Fig. 9 satisfies the properties in Fig. 8, provided the SMR synchronizer it uses satisfies the properties in Fig. 1.

First, analogously to Proposition 2 we can prove that the consensus synchronizer keeps switching processes through views forever.

Proposition 7

\(\forall v.\, \exists v'.\, v'>v \wedge {{\mathbb {E}}_{{\textrm{first}}}(v')\mathpunct {\downarrow }}\).

The following lemma is used to prove Property V in Fig. 8.

Lemma 12

If a correct process enters a view \(v>0\) and \(E_{{\textrm{first}}}(v) \ge \textsf {GST}\), then for all \(v' > v\), no correct process attempts to advance from \(v'-1\) before \(E_{{\textrm{first}}}(v) + F(v)\).

Proof

Suppose by contradiction that there exists a time \(t' < E_{{\textrm{first}}}(v) + F(v)\) and a correct process \(p_i\) such that \(p_i\) attempts to advance from \(v'-1 > v-1\) at \(t'\). Since \(v' \ge v + 1 > 1\), at \(t'\) the process \(p_i\) executes the handler at line 7 and the last view it entered is \(v' - 1\). Since \(p_i.\textsf {timer\_view}\) is not enabled at \(t'\), \(p_i\) must have entered \(v' - 1\) at least \(F(v)\) before \(t'\) according to its local clock. Since \(v' - 1 \ge v\), by Proposition 1, we have \(E_{{\textrm{first}}}(v' - 1) \ge E_{{\textrm{first}}}(v) \ge \textsf {GST}\). Therefore, given that the clocks of all correct processes progress at the same rate as real time after \( \textsf {GST}\), we get

$$\begin{aligned} E_{{\textrm{first}}}(v) \le E_{{\textrm{first}}}(v' - 1) \le t' - F(v' - 1). \end{aligned}$$

Hence,

$$\begin{aligned} t' \ge E_{{\textrm{first}}}(v) + F(v' - 1). \end{aligned}$$

Since \(F\) is non-decreasing and \(v' - 1 \ge v\), we have \(F(v' - 1) \ge F(v)\), so that

$$\begin{aligned} t' \ge E_{{\textrm{first}}}(v) + F(v), \end{aligned}$$

which contradicts our assumption that \(t' < E_{{\textrm{first}}}(v) + F(v)\). This contradiction shows the required. \(\square \)

Proof of Theorem 6

Property I follows from Monotonicity of the SMR synchronizer. Let d and \({\mathcal {V}}\) be the witnesses for the existential from Bounded Entry and let \({\mathcal {V}}'\) be the minimal view such that \({\mathcal {V}}' \ge {\mathcal {V}}\), \(E_{{\textrm{first}}}({\mathcal {V}}') \ge \textsf {GST}\) and \(F({\mathcal {V}}') \ge d\). Such a view exists by (9) and Proposition 7. Then Property II holds for \({\mathcal {V}}= {\mathcal {V}}'\). By Propositions 1 and 7, a correct process enters every view \(v \ge {\mathcal {V}}'\). By Proposition 1, \(v \ge {\mathcal {V}}'\) implies

$$\begin{aligned} E_{{\textrm{first}}}(v) \ge E_{{\textrm{first}}}({\mathcal {V}}') \ge \textsf {GST}. \end{aligned}$$
(10)

Since \(F\) is a non-decreasing function, \(F(v) \ge d\). Thus, by Lemma 12 and Bounded Entry, all correct processes enter v, and \(E_{{\textrm{last}}}(v) \le E_{{\textrm{first}}}(v) + d\), which validates Properties III and IV for \({\mathcal {V}}= {\mathcal {V}}'\). To prove Property V, fix a view \(v\ge {\mathcal {V}}'\). Since a correct process enters view \(v+1\), by Validity, there exist a time \(t < E_{{\textrm{first}}}(v+1)\) at which some correct process attempts to advance from v. By (10), \(E_{{\textrm{first}}}(v) \ge \textsf {GST}\). Then by Lemma 12 we get \(t \ge E_{{\textrm{first}}}(v) + F(v)\), so that \(E_{{\textrm{first}}}(v+1) > t \ge E_{{\textrm{first}}}(v) + F(v)\), as required. \(\square \)

Proof of the synchronizer correctness (Theorem 1)

The local view of a process \(p_i\) at time t, denoted \(\textsf {LV}_{i}(t)\), is the latest view entered by \(p_i\) at or before t, or 0 if \(p_i\) has not entered any views by then. We prove Validity using the following lemma, which shows that a \( \texttt {WISH}(v+1)\) can only be sent by a correct process if some correct process has already attempted to advance from the view v.

Lemma 13

For all t and \(v\ge 0\), if a correct process sends \( \texttt {WISH}(v+1)\) at t, then there exists a time \(t' \le t\) such that some correct process attempts to advance from v at \(t'\).

Proof

We first prove the following auxiliary proposition:

$$\begin{aligned}&\forall p_i.\, \forall v.\, p_i \text { is correct} \wedge p_i \text { sends } \texttt {WISH}(v+1) \text { at } t \implies \nonumber \\&\quad \exists t' \le t.\, \exists v' \ge v.\, \exists p_j.\, \nonumber \\&\quad p_j \text { is correct} \wedge p_j \text { attempts to advance from } v' \text {~at~} t'. \end{aligned}$$
(11)

By contradiction, assume that a correct process \(p_i\) sends \( \texttt {WISH}(v+1)\) at t, but for all \(t' \le t\) and all \(v' \ge v\), no correct process attempts to advance from \(v'\) at \(t'\). Consider the earliest time \(t_k\) when some correct process \(p_k\) sends a \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\), so that \(t_k \le t\).

Since \(p_k\) sends \( \texttt {WISH}(v_k)\) at \(t_k\), either \(v_k = p_k. \textsf {view}^+(t_k)\) or \(p_k. \textsf {view}(t_k) = p_k. \textsf {view}^+(t_k) = v_k - 1\), and in the latter case \(p_k\) executes either line 2 or line 6. If \(p_k. \textsf {view}^+(t_k) = v_k \ge v+1\), then \(p_k.\textsf {max\_views}(t_k)\) includes \(f+1\) entries \(\ge v_k \ge v+1\), and therefore, there exists a correct process \(p_l\) that sent \( \texttt {WISH}(v')\) with \(v' \ge v+1\) at \(t_l < t_k\), contradicting the assumption that \(t_k\) is the earliest time when this can happen. Suppose that \(p_k. \textsf {view}(t_k) = p_k. \textsf {view}^+(t_k) = v_k - 1\) and at \(t_k\), \(p_k\) executes either line 2 or line 6. Then \(\textsf {LV}_{k}(t_k) = v_k - 1\). If \(p_k\) executes line 2 at \(t_k\), then since \(\textsf {LV}_{k}(t_k) = v_k - 1\), \(p_k\) attempts to advance from \(v_k-1 \ge v\) at \(t_k\le t\), contradicting our assumption that no such attempt can occur. Suppose now that \(p_k\) executes the code in line 6 at \(t_k\). If \(v_k > 1\), then since \(p_k. \textsf {view}(t_k) = p_k. \textsf {view}^+(t_k) = v_k - 1\), we know that \(E_{k}(v_k-1)\) is defined and satisfies \(E_{k}(v_k-1) < t_k\). Let \(t'_k = E_{k}(v_k-1)\) if \(v_k > 1\), and \(t'_k = 0\) otherwise. Then \(p_k. \textsf {view}(t'_k) = p_k. \textsf {view}^+(t'_k) = v_k - 1\) and \(p_k. \textsf {advanced}(t'_k) = \text {false}\). Since \(p_k. \textsf {advanced}(t_k) = \text {true}\), there exists a time \(t''_k\) such that \(t'_k < t''_k \le t_k\) and \(p_k\) calls \( \texttt {advance}()\) at \(t''_k\). Since both \(p_k. \textsf {view}\) and \(p_k. \textsf {view}^+\) are non-decreasing, and both are equal to \(v_k - 1\) at \(t''_k\) as well as \(t_k\), \(p_k. \textsf {view}(t''_k) = p_k. \textsf {view}^+(t''_k) = v_k - 1\). Thus, \(\textsf {LV}_{k}(t''_k) = v_k-1\), which implies that at \(t''_k < t_k \le t\), \(p_k\) attempts to advance from \(v_k-1 \ge v\), contradicting our assumption that no such attempt can happen. Thus, (11) holds.

We now prove the lemma. Let t and v be such that some correct process sends \( \texttt {WISH}(v+1)\) at t. By (11), there exists a correct process that attempts to advance from a view \(\ge v\) at or before t. Let \(t'\) be the earliest time when some correct process attempts to advance from a view \(\ge v\), and let \(p_j\) be this process and \(v' \ge v\) be the view from which \(p_j\) attempts to advance at \(t'\). Thus, at \(t'\), \(p_j\) executes the code in line 2 and \(\textsf {LV}_{j}(t') = v' \ge v\). Hence, there exists an earlier time at which \(p_j. \textsf {view}^+= p_j. \textsf {view}= v'\). Since \(p_j. \textsf {view}^+\) is non-decreasing, \(p_j. \textsf {view}^+(t') \ge v'\). If \(p_j. \textsf {view}^+(t') > v'\), then given that \(v' \ge v\), \(p_j. \textsf {view}^+(t') \ge v+1\). Thus, there exists a correct process \(p_k\) and time \(t'' < t'\) such that \(p_k\) sent \( \texttt {WISH}(v'')\) with \(v'' \ge v+1\) to \(p_j\) at \(t''\). By (11), there exists a time \(\le t'' < t'\) at which some correct process attempts to advance from a view \(\ge v''-1 \ge v\), which is impossible. Thus, \(p_j. \textsf {view}^+(t') = v'\). Since \(\textsf {LV}_{j}(t') = v'\), we have \(p_j. \textsf {view}(t') = p_j. \textsf {view}^+(t') = v' \ge v\). By the definitions of \( \textsf {view}\) and \( \textsf {view}^+\), \(v'\) is both the lowest view among the highest \(2f+1\) views in \(p_j.\textsf {max\_views}(t')\), and the lowest view among the highest \(f+1\) views in \(p_j.\textsf {max\_views}(t')\). Hence, \(p_j.\textsf {max\_views}(t')\) includes \(f+1\) entries equal to \(v'\), and therefore, there exists a correct process \(p_k\) such that

$$\begin{aligned}&p_j. \textsf {view}(t') = p_j. \textsf {view}^+(t') ={}\nonumber \\&\quad p_j.\textsf {max\_views}[k](t') = v' \ge v-1. \end{aligned}$$
(12)

Also, for all correct processes \(p_l\), \(p_j.\textsf {max\_views}[l](t') < v+1\): otherwise, some correct process sent \( \texttt {WISH}(v'')\) with \(v'' \ge v+1\) at \(t'' < t'\), and therefore, by (11), some correct process attempted to advance from a view \(\ge v\) earlier than \(t'\), which is impossible. Thus,

$$\begin{aligned}{} & {} p_j. \textsf {view}(t') = p_j. \textsf {view}^+(t') \\{} & {} \quad = p_j.\textsf {max\_views}[k](t') < v+1. \end{aligned}$$

Together with (12), this implies

$$\begin{aligned} p_j. \textsf {view}(t') = p_j. \textsf {view}^+(t') = v. \end{aligned}$$

Hence, \(\textsf {LV}_{j}(t') = v\), and therefore, \(p_j\) attempts to advance from v at \(t'\). Thus, \(v' = v\) and \(t' \le t\), as required. \(\square \)

Lemma 14

Validity holds: \(\forall i, v.\, {E_{i}(v+1)\mathpunct {\downarrow }} {\implies } {A_{\textrm{first}}(v)}\mathpunct {\downarrow }\wedge A_{\textrm{first}}(v) < E_{i}(v+1)\).

Proof

Since \(p_i\) enters a view \(v+1\), we have \(p_i. \textsf {view}(E_{i}(v+1)) = p_i. \textsf {view}^+(E_{i}(v+1))=v+1\). By the definitions of \( \textsf {view}\) and \( \textsf {view}^+\), \(v+1\) is both the lowest view among the highest \(2f+1\) views in \(p_i.\textsf {max\_views}(E_{i}(v+1))\), and the lowest view among the highest \(f+1\) views in \(p_i.\textsf {max\_views}(E_{i}(v+1))\). Hence, \(p_i.\textsf {max\_views}(E_{i}(v+1))\) includes \(f+1\) entries equal to \(v+1\). Then there exists a time \(t' < E_{i}(v+1)\) at which some correct process sends \( \texttt {WISH}(v+1)\). Hence, by Lemma 13, there exists a time \(t \le t' < E_{i}(v+1)\) at which some correct process attempts to advance from v. \(\square \)

In the following we use the next technical lemma, following from Lemmas 13 and 14.

Lemma 15

For all times t and views \(v>0\), if a correct process sends \( \texttt {WISH}(v)\) at t, then there exists a time \(t' \le t\) such that some correct process attempts to advance from view 0 at \(t'\).

Proof

Consider the earliest time \(t_k \le t\) at which some correct process \(p_k\) sends \( \texttt {WISH}(v_k)\) for some view \(v_k\). By Lemma 13, there exists a time \(t_j \le t_k\) at which some correct process attempts to advance from \(v_k-1 \ge 0\), and therefore, sends \( \texttt {WISH}(v_k)\) at \(t_j\). Since \(t_k\) is the earliest time when this could happen, we have \(t_j = t_k\). Also, if \(v_k - 1 > 0\), then \(E_{k}(v_k-1)\) is defined, and hence, by Lemma 14, some correct process attempts to advance from \(v_k - 2\) by sending \( \texttt {WISH}(v_k - 1)\) earlier than \(t_j = t_k\), which cannot happen. Thus, \(v_k = 1\) and at \(t_k\), \(p_k\) attempts to advance from 0, as required. \(\square \)

Lemma 16 below establishes that the views sent in the \( \texttt {WISH}\) messages by the same process can only increase. Its proof relies on the next proposition, stating some simple invariants that follow immediately from the structure of the code.

Proposition 8

Let \(p_i\) be a correct process. Then:

  1. 1.

    \(\forall v.\forall t.\, p_i \text { sends } \texttt {WISH}(v) \text {~at~} t {\implies } \)

    \(v \in \{p_i. \textsf {view}^+(t), p_i. \textsf {view}^+(t)+1\}\).

  2. 2.

    \(\forall v.\forall t.\, p_i \text { sends } \texttt {WISH}(v) \text { at } t \wedge v = p_i. \textsf {view}^+(t)+1 {\implies }\)

    \(p_i. \textsf {view}^+(t)=p_i. \textsf {view}(t) \wedge p_i. \textsf {advanced}(t) = \text {true}\).

Lemma 16

For all views \(v, v' > 0\), if a correct process sends \( \texttt {WISH}(v)\) before sending \( \texttt {WISH}(v')\), then \(v \le v'\).

Proof

Let s and \(s'\) such that \(s < s'\) be the times at which a correct process \(p_i\) sends \( \texttt {WISH}(v)\) and \( \texttt {WISH}(v')\) messages, respectively. We show that \(v' \ge v\). By Proposition 8(1), \(v \in \{p_i. \textsf {view}^+(s), p_i. \textsf {view}^+(s)+1\}\) and \(v' \in \{p_i. \textsf {view}^+(s'), p_i. \textsf {view}^+(s')+1\}\). Hence, if \(v = p_i. \textsf {view}^+(s)\) or \(v' = p_i. \textsf {view}^+(s')+1\), then we get \(v \le v'\) from the fact that \(p_i. \textsf {view}^+\) is non-decreasing. It thus remains to consider the case when \(v = p_i. \textsf {view}^+(s)+1\) and \(v' = p_i. \textsf {view}^+(s')\). In this case by Proposition 8(2), \(p_i. \textsf {view}^+(s)=p_i. \textsf {view}(s)\) and \(p_i. \textsf {advanced}(s) = \text {true}\). We now consider several cases depending on the line at which \( \texttt {WISH}(v')\) is sent.

  • \( \texttt {WISH}(v')\) is sent at lines 2 or 6. Then \(v'=p_i. \textsf {view}^+(s')=\max (p_i. \textsf {view}(s')+1, p_i. \textsf {view}^+(s'))\). Since \(p_i. \textsf {view}\) is non-decreasing, we get \(p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}(s')+1 > p_i. \textsf {view}(s') \ge p_i. \textsf {view}(s) = p_i. \textsf {view}^+(s)\). Hence, \(p_i. \textsf {view}^+(s') > p_i. \textsf {view}^+(s)\), and therefore, \(v'=p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}^+(s)+1=v\), as required.

  • \( \texttt {WISH}(v')\) is sent at line 8. Then \(p_i. \textsf {advanced}(s') = \text {false}\). Since \(p_i. \textsf {advanced}(s) = \text {true}\), there exists a time \(s''\) such that \(s< s'' < s'\) and \(p_i\) enters a view at \(s''\). By the view entry condition \(p_i. \textsf {view}(s'') > p_i.\textit{prev\_v}(s'')\). Since \(p_i. \textsf {view}\) is non-decreasing, we get \(p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}(s') \ge p_i. \textsf {view}(s'') > p_i. \textsf {view}(s) = p_i. \textsf {view}^+(s)\). Thus, \(p_i. \textsf {view}^+(s') > p_i. \textsf {view}^+(s)\) and therefore, \(v' = p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}^+(s)+1 = v\), as required.

  • \( \texttt {WISH}(v')\) is sent at line 18. Then \(p_i. \textsf {view}^+(s') > p_i.\textit{prev\_v}^+(s') \ge p_i. \textsf {view}^+(s)\), and therefore, \(v' = p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}^+(s) + 1 = v\), as required.

\(\square \)

In order to cope with message loss before \( \textsf {GST}\), every correct process retransmits the highest \( \texttt {WISH}\) it sent every \(\rho \) time units, according to its local clock (lines 4-8). Eventually, one of these retransmissions will occur after \( \textsf {GST}\), and therefore, there exists a time by which all correct processes are guaranteed to send their highest \( \texttt {WISH}\)es at least once after \( \textsf {GST}\). The earliest such time, \( \overline{\textsf {GST}}\), is defined as follows:

$$\begin{aligned} \overline{\textsf {GST}}= {\left\{ \begin{array}{ll} \textsf {GST}+ \rho ,&{} \quad \text {if } A_{\textrm{first}}(0) < \textsf {GST};\\ A_{\textrm{first}}(0),&{} \quad \text {otherwise}. \end{array}\right. } \end{aligned}$$

From this definition it follows that

$$\begin{aligned} \overline{\textsf {GST}}\ge \textsf {GST}. \end{aligned}$$
(13)

The following lemma formalizes the key property of \( \overline{\textsf {GST}}\).

Lemma 17

For all correct processes \(p_i\), times \(t \ge \overline{\textsf {GST}}\), and views v, if \(p_i\) sends \( \texttt {WISH}(v)\) at a time \(\le t\), then there exists a view \(v' \ge v\) and a time \(t'\) such that \( \textsf {GST}\le t' \le t\) and \(p_i\) sends \( \texttt {WISH}(v')\) at \(t'\).

Proof

Let \(s \le t\) be the time at which \(p_i\) sends \( \texttt {WISH}(v)\). We consider two cases. Suppose first that \(A_{\textrm{first}}(0) \ge \textsf {GST}\). By Lemma 15, \(s \ge A_{\textrm{first}}(0)\), and therefore, \( \textsf {GST}\le s \le t\). Thus, choosing \(t'=s\) and \(v' = v\) validates the lemma. Suppose next that \(A_{\textrm{first}}(0) < \textsf {GST}\). Then by the definition of \( \overline{\textsf {GST}}\), \(t \ge \textsf {GST}+ \rho \). If \(s \ge \textsf {GST}\), then \( \textsf {GST}\le s \le t\), and therefore, choosing \(t'=s\) and \(v' = v\) validates the lemma. Assume now that \(s < \textsf {GST}\). Since after \( \textsf {GST}\) the \(p_i\)’s local clock advances at the same rate as real time, there exists a time \(s'\) satisfying \( \textsf {GST}\le s' \le t\) such that \(p_i\) executes the periodic retransmission code in lines 4-8 at \(s'\). We now show that

$$\begin{aligned} p_i. \textsf {advanced}(s') \vee p_i. \textsf {view}^+(s') > 0. \end{aligned}$$
(14)

Since \(p_i\) already sent a \( \texttt {WISH}\) message at \(s < \textsf {GST}\le s'\), by the structure of the code,

$$\begin{aligned} p_i. \textsf {advanced}(s) \vee p_i. \textsf {view}^+(s) > 0. \end{aligned}$$

If \(p_i. \textsf {view}^+(s) > 0\), then since \(p_i. \textsf {view}^+\) is non-decreasing, \(p_i. \textsf {view}^+(s') > 0\), and therefore, (14) holds. Assume now that \(p_i. \textsf {advanced}(s)\). If \(p_i. \textsf {advanced}(s')\), then (14) holds too. We therefore consider the case when \(\lnot p_i. \textsf {advanced}(s')\). Then there exists a time \(s \le s'' \le s'\) at which \(p_i\) enters the view \(p_i. \textsf {view}(s'')>0\). Hence, \(p_i. \textsf {view}^+(s') \ge p_i. \textsf {view}^+(s'') \ge p_i. \textsf {view}(s'') > 0\), validating (14). Thus, (14) holds in all cases. Therefore, at \(s'\) the process \(p_i\) sends \( \texttt {WISH}(v')\) for some view \(v'\). By Lemma 16, \(v' \ge v\), and above we established \( \textsf {GST}\le s' \le t\), as required. \(\square \)

The following lemma is used to prove Bounded Entry.

Lemma 18

Consider a view \(v>0\) and assume that v is entered by a correct process. If \(E_{{\textrm{first}}}(v) \ge \overline{\textsf {GST}}\), and no correct process attempts to advance from v before \(E_{{\textrm{first}}}(v) + 2\delta \), then all correct processes enter v and \(E_{{\textrm{last}}}(v) \le E_{{\textrm{first}}}(v) + 2\delta \).

Proof

If some correct process attempts to advance from a view \(v' > v\) before \(E_{{\textrm{first}}}(v) + 2\delta \), then by Proposition 1, some correct process must also enter the view \(v+1\). By Lemma 14, this implies that some correct process attempts to advance from v before \(E_{{\textrm{first}}}(v) + 2\delta \), contradicting the lemma’s premise. Thus, no correct process attempts to advance from any view \(v' \ge v\) before \(E_{{\textrm{first}}}(v) + 2\delta \), and therefore, by Lemma 13, no correct process can send \( \texttt {WISH}(v')\) with \(v' > v\) earlier than \(E_{{\textrm{first}}}(v) + 2\delta \). Once any such \( \texttt {WISH}(v')\) is sent, it will take a non-zero time until it is received by any correct process. Thus, we have:

  1. (*)

    no correct process receives \( \texttt {WISH}(v')\) with \(v' > v\) from a correct process until after \(E_{{\textrm{first}}}(v) + 2\delta \).

Let \(p_i\) be a correct process that enters v at \(E_{{\textrm{first}}}(v)\). By the view entry condition, \(p_i. \textsf {view}(E_{{\textrm{first}}}(v)) = v\), and therefore \(p_i.\textsf {max\_views}(E_{{\textrm{first}}}(v))\) includes \(2f+1\) entries \(\ge v\). At least \(f+1\) of these entries belong to correct processes, and by (*), none of them can be \(> v\). Hence, there exists a set C of \(f+1\) correct processes, each of which sends \( \texttt {WISH}(v)\) to all processes before \(E_{{\textrm{first}}}(v)\).

Since \(E_{{\textrm{first}}}(v) \ge \overline{\textsf {GST}}\), by Lemma 17, every \(p_j \in C\) also sends \( \texttt {WISH}(v')\) with \(v'\ge v\) at some time \(s_j\) such that \( \textsf {GST}\le s_j \le E_{{\textrm{first}}}(v)\). Then by (*) we have \(v'=v\). It follows that each \(p_j \in C\) is guaranteed to send \( \texttt {WISH}(v)\) to all correct processes between \( \textsf {GST}\) and \(E_{{\textrm{first}}}(v)\). Since all messages sent by correct processes after \( \textsf {GST}\) are guaranteed to be received by all correct processes within \(\delta \) of their transmission, by \(E_{{\textrm{first}}}(v) + \delta \) all correct processes will receive \( \texttt {WISH}(v)\) from at least \(f+1\) correct processes.

Consider an arbitrary correct process \(p_j\) and let \(t_j \le E_{{\textrm{first}}}(v) + \delta \) be the earliest time by which \(p_j\) receives \( \texttt {WISH}(v)\) from \(f+1\) correct processes. By (*), no correct process sends \( \texttt {WISH}(v')\) with \(v' > v\) before \(t_j < E_{{\textrm{first}}}(v) + 2\delta \). Thus, \(p_j.\textsf {max\_views}(t_j)\) includes at least \(f+1\) entries equal to v and at most f entries \(>v\), so that \(p_j. \textsf {view}^+(t_j) = v\). Then \(p_j\) sends \( \texttt {WISH}(v)\) to all processes no later than \(t_j \le E_{{\textrm{first}}}(v) + \delta \). Since \(E_{{\textrm{first}}}(v) \ge \overline{\textsf {GST}}\), by Lemma 17, \(p_j\) also sends \( \texttt {WISH}(v')\) with \(v'\ge v\) in-between \( \textsf {GST}\) and \(E_{{\textrm{first}}}(v) + \delta \). By (*), \(v'=v\), and therefore, \(p_j\) must have sent \( \texttt {WISH}(v)\) to all processes sometime between \( \textsf {GST}\) and \(E_{{\textrm{first}}}(v) + \delta \). Hence, all correct processes are guaranteed to send \( \texttt {WISH}(v)\) to all correct processes between \( \textsf {GST}\) and \(E_{{\textrm{first}}}(v) + \delta \).

Consider an arbitrary correct process \(p_k\) and let \(t_k \le E_{{\textrm{first}}}(v) + 2\delta \) be the earliest time by which \(p_k\) receives \( \texttt {WISH}(v)\) from all correct processes. Then by (*), all entries of correct processes in \(p_k.\textsf {max\_views}(t_k)\) are equal to v. Since there are at least \(2f+1\) correct processes: (i) at least \(2f+1\) entries in \(p_k.\textsf {max\_views}(t_k)\) are equal to v, and (ii) one of the \(f+1\) highest entries in \(p_k.\textsf {max\_views}(t_k)\) is equal to v. From (i), \(p_k. \textsf {view}^+(t_k) \ge p_k. \textsf {view}(t_k) \ge v\), and from (ii), \(p_k. \textsf {view}(t_k) \le p_k. \textsf {view}^+(t_k) \le v\). Therefore, \(p_k. \textsf {view}(t_k) = p_k. \textsf {view}^+(t_k) = v\), so that \(p_k\) enters v no later than \(t_k \le E_{{\textrm{first}}}(v)+2\delta \). We have thus shown that by \(E_{{\textrm{first}}}(v)+2\delta \), all correct processes enter v, as required. \(\square \)

Lemma 19

Bounded Entry holds for \(d=2\delta \) and the \({\mathcal {V}}\) defined in Theorem 3. Namely, consider a view \(v\ge {\mathcal {V}}\) and assume that v is entered by a correct process. If no correct process attempts to advance from v before \(E_{{\textrm{first}}}(v) + 2\delta \), then all correct processes enter v and \(E_{{\textrm{last}}}(v) \le E_{{\textrm{first}}}(v) + 2\delta \).

Proof

Consider the \({\mathcal {V}}\) defined in Theorem 3. It is easy to see that

$$\begin{aligned} {\mathcal {V}}=\max \{v \mid ({E_{{\textrm{first}}}(v)\mathpunct {\downarrow }} \wedge E_{{\textrm{first}}}(v) < \overline{\textsf {GST}}) \vee v = 0\} + 1.\nonumber \\ \end{aligned}$$
(15)

Then \(\forall v \ge {\mathcal {V}}.\, {E_{{\textrm{first}}}(v)\mathpunct {\downarrow }} {\implies } E_{{\textrm{first}}}(v) \ge \overline{\textsf {GST}}\). Bounded Entry now follows from Lemma 18. \(\square \)

Lemma 20

Startup holds: suppose there exists a set P of \(f+1\) correct processes such that \(\forall p_i \in P.\, {A_{i}(0)\mathpunct {\downarrow }}\); then eventually some correct process enters view 1.

Proof

Assume by contradiction that there exists a set P of \(f+1\) correct processes such that \(\forall p_i \in P.\, {A_{i}(0)\mathpunct {\downarrow }}\), and no correct process enters the view 1. By Proposition 1, the latter implies

$$\begin{aligned} \forall v' > 0.\, E_{{\textrm{first}}}(v')\mathpunct {\uparrow }. \end{aligned}$$
(16)

Then by Lemma 13 we have

$$\begin{aligned} \forall t.\, \forall v' > 1.\, \forall p_i.\, \lnot (p_i \text { sends } \texttt {WISH}(v') \text { at } t \wedge p_i \text { is correct}).\nonumber \\ \end{aligned}$$
(17)

Let \(T_1 = \max ( \overline{\textsf {GST}}, A_{\textrm{last}}(0))\). Since there exists a set P of \(f+1\) correct processes that attempt to advance from view 0, each \(p_i\in P\) sends \( \texttt {WISH}(v_i)\) with \(v_i > 0\) before \(T_1\). Since \(T_1 \ge \overline{\textsf {GST}}\), by Lemma 17, there exists a view \(v_i'\ge 1\) and a time \(s_i\) such that \( \textsf {GST}\le s_i \le T_1\) and \(p_i\) sends \( \texttt {WISH}(v_i')\) at \(s_i\). By (17), \(v_i'=1\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(1)\) sent by \(p_i\) at \(s_i\) will be received by all correct processes.

Thus, there exists a time \(T_2 \ge T_1 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(1)\) from all processes in P. Fix an arbitrary correct process \(p_j\). Since all process in P are correct, all entries in \(p_j.\textsf {max\_views}(T_2)\) associated with the processes in P are equal to 1. Since \(|P|=f+1\), \(p_j.\textsf {max\_views}(T_2)\) includes at least \(f+1\) entries \(\ge 1\), and therefore, \(p_j. \textsf {view}^+(T_2)\ge 1\). Hence, \(p_j\) sends \( \texttt {WISH}(v_j)\) with \(v_j \ge 1\) no later than \(T_2\). Since \(T_2 \ge \overline{\textsf {GST}}\), by Lemma 17 there exists a view \(v'_j\ge 1\) and a time \(s_j\) such that \( \textsf {GST}\le s_j \le T_2\) and \(p_j\) sends \( \texttt {WISH}(v'_j)\) at \(s_j\). By (17), \(v'_j = 1\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(1)\) sent by \(p_j\) will be received by all correct processes.

Thus, there exists a time \(T_3 \ge T_2 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(1)\) from all correct processes. Fix an arbitrary correct process \(p_k\). By (17), all entries of correct processes in \(p_k.\textsf {max\_views}(T_3)\) are equal to 1. Since there are at least \(2f+1\) correct processes: (i) at least \(2f+1\) entries in \(p_k.\textsf {max\_views}(T_3)\) are equal to 1, and (ii) one of the \(f+1\) highest entries in \(p_k.\textsf {max\_views}(T_2)\) is equal to 1. From (i), \(p_k. \textsf {view}^+(T_2) \ge p_k. \textsf {view}(T_2) \ge 1\), and from (ii), \(p_k. \textsf {view}(T_2) \le p_k. \textsf {view}^+(T_2) \le 1\). Hence, \(p_k. \textsf {view}(T_2) = p_k. \textsf {view}^+(T_2) = 1\), and therefore, \(p_k\) enters view 1 by \(T_2\), contradicting (16). \(\square \)

Lemma 21

Progress holds: consider a view \(v>0\) that is entered by a correct process, and suppose there exists a set P of \(f+1\) correct processes such that

$$\begin{aligned} \forall p_i\in P.\, {E_{i}(v)\mathpunct {\downarrow }} \implies {A_{i}(v)\mathpunct {\downarrow }}; \end{aligned}$$
(18)

then eventually some correct process enters \(v+1\).

Proof

Assume by contradiction that the required does not hold. Then, there exists a view \(v > 0\) such that some correct process enters v, (18) holds, and no correct process enters the view \(v+1\). By Proposition 1, the latter implies that

$$\begin{aligned} \forall v' > v.\, E_{{\textrm{first}}}(v')\mathpunct {\uparrow }. \end{aligned}$$
(19)

Thus, by Lemma 13, we have

$$\begin{aligned}&\forall t.\, \forall v' > v + 1.\, \forall p_i.\, \lnot (p_i \text {~sends~} \texttt {WISH}(v') \text {~at~} t \wedge {} \nonumber \\&p_i \text { is correct}). \end{aligned}$$
(20)

Let \(T_1 = \max ( \overline{\textsf {GST}}, E_{{\textrm{first}}}(v))\). Since some correct process entered v by \(T_1\), there exists a set C consisting of \(f+1\) correct processes all of which sent \( \texttt {WISH}(v')\) with \(v'\ge v\) before \(T_1\). Consider \(p_i\in C\) and let \(t_i \le T_1\) be a time such that at \(t_i\) the process \(p_i\) sends \( \texttt {WISH}(v_i)\) with \(v_i \ge v\). Since \(T_1 \ge \overline{\textsf {GST}}\), by Lemma 17, there exists a view \(v_i'\ge v_i\) and a time \(s_i\) such that \( \textsf {GST}\le s_i \le T_1\) and \(p_i\) sends \( \texttt {WISH}(v_i')\) at \(s_i\). By (20), we have \(v_i' \in \{v, v+1\}\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(v_i')\) sent by \(p_i\) at \(s_i\) will be received by all correct processes.

Thus, there exists a time \(T_2 \ge T_1 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(v')\) with \(v' \in \{v, v+1\}\) from all processes in C. Consider an arbitrary correct process \(p_j\). By (20), the entry of every process in C in \(p_j.\textsf {max\_views}(T_2)\) is equal to either v or \(v+1\). Since \(|C|\ge f+1\) and all processes in C are correct, \(p_j.\textsf {max\_views}(T_2)\) includes at least \(f+1\) entries \(\ge v\). Thus, \(p_j. \textsf {view}^+(T_2) \ge v\), and therefore, \(p_j\) sends \( \texttt {WISH}(v_j)\) with \(v_j \ge v\) no later than at \(T_2\). By (20), \(v_j \in \{v, v+1\}\). Since \(T_2 \ge \overline{\textsf {GST}}\), by Lemma 17, there exists a view \(v_j'\ge v\) and a time \(s_j\) such that \( \textsf {GST}\le s_j \le t_j\) and \(p_j\) sends \( \texttt {WISH}(v_j')\) at \(s_j\). By (20), \(v_j' \in \{v, v+1\}\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(v_j')\) sent by \(p_j\) at \(s_j\) will be received by all correct processes.

Thus, there exists a time \(T_3 \ge T_2 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(v')\) such that \(v' \in \{v, v+1\}\) from all correct processes. Consider an arbitrary correct process \(p_k\), and suppose that \(p_k\) is a member of the set P stipulated by the lemma’s premise. Then at \(T_3\), all entries of correct processes in \(p_k.\textsf {max\_views}\) are \(\ge v\). By (20), each of these entries is equal to either v or \(v+1\). Since at least \(2f+1\) processes are correct: (i) at least \(2f+1\) entries in \(p_k.\textsf {max\_views}(T_3)\) are \(\ge v\), and (ii) one of the \(f+1\) highest entries in \(p_k.\textsf {max\_views}(T_3)\) is \(\le v+1\). From (i), \(p_k. \textsf {view}^+(T_3) \ge p_k. \textsf {view}(T_3) \ge v\), and from (ii), \(p_k. \textsf {view}(T_3) \le p_k. \textsf {view}^+(T_3) \le v+1\). Hence, \(p_k. \textsf {view}(T_3), p_k. \textsf {view}^+(T_3) \in \{v, v+1\}\). Since no correct process enters \(v+1\), \(p_k. \textsf {view}(T_3)\) and \(p_k. \textsf {view}^+(T_3)\) cannot be both simultaneously equal to \(v+1\). Thus, \(p_k. \textsf {view}(T_3) = v\), and either \(p_k. \textsf {view}^+(T_3) = v\) or \(p_k. \textsf {view}^+(T_3)=v+1\). If \(p_k. \textsf {view}^+(T_3) = v+1\), then \(p_k\) has sent \( \texttt {WISH}(v_k)\) with \(v_k=v+1\) when \(p_k. \textsf {view}^+\) has first become equal to \(v+1\) sometime before \(T_3\). On the other hand, if \(p_k. \textsf {view}(T_3) = p_k. \textsf {view}^+(T_3) = v\), then \(p_k\) has entered v at some time \(t\le T_3\). Since \(p_k\in P\), by (18), there exists a time \(t' \ge t\) such that \(p_k\) attempts to advance from v at \(t'\), and therefore, sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) at \(t'\). By (20), \(v_k \le v + 1\), and therefore, \(v_k = v+1\). Thus, there exists a time \(t_k \ge T_3\) by which \(p_k\) sends \( \texttt {WISH}(v+1)\) to all processes. Since \(t_k \ge T_3 \ge \overline{\textsf {GST}}\), by Lemma 17, there exists a view \(v'_k\ge v+1\) and a time \(s_k\) such that \( \textsf {GST}\le s_k \le t_k\) and \(p_k\) sends \( \texttt {WISH}(v'_k)\) at \(s_k\). By (20), \(v'_k=v+1\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(v+1)\) sent by \(p_k\) will be received by all correct processes.

Thus, there exists a time \(T_4 \ge T_3 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(v+1)\) from all processes in P. Fix an arbitrary correct process \(p_l\). Since all process in P are correct, by (20), all entries in \(p_l.\textsf {max\_views}(T_4)\) associated with the processes in P are equal to \(v+1\). Since \(|P|=f+1\), \(p_l.\textsf {max\_views}(T_4)\) includes at least \(f+1\) entries equal to \(v+1\), and therefore, \(p_l. \textsf {view}^+(T_4)\ge v+1\). Hence, \(p_l\) sends \( \texttt {WISH}(v_l)\) with \(v_l \ge v+1\) no later than \(T_4\). Since \(T_4 \ge \overline{\textsf {GST}}\), by Lemma 17 there exists a view \(v'_l\ge v+1\) and a time \(s_l\) such that \( \textsf {GST}\le s_l \le T_4\) and \(p_l\) sends \( \texttt {WISH}(v'_l)\) at \(s_l\). By (20), \(v_l' = v+1\). Since the links are reliable after \( \textsf {GST}\), the \( \texttt {WISH}(v+1)\) sent by \(p_l\) will be received by all correct processes.

Thus, there exists a time \(T_5 \ge T_4 \ge \overline{\textsf {GST}}\) by which all correct processes have received \( \texttt {WISH}(v + 1)\) from all correct processes. Fix an arbitrary correct process \(p_m\). By (20), all entries of correct processes in \(p_m.\textsf {max\_views}(T_5)\) are equal to \(v+1\). Since there are at least \(2f+1\) correct processes: (i) at least \(2f+1\) entries in \(p_m.\textsf {max\_views}(T_5)\) are equal to \(v+1\), and (ii) one of the \(f+1\) highest entries in \(p_m.\textsf {max\_views}(T_5)\) is equal to \(v+1\). From (i), \(p_m. \textsf {view}^+(T_5) \ge p_m. \textsf {view}(T_5) \ge v+1\), and from (ii), \(p_m. \textsf {view}(T_5) \le p_l. \textsf {view}^+(T_5) \le v+1\). Hence, \(p_m. \textsf {view}(T_5) = p_m. \textsf {view}^+(T_5) = v+1\), and therefore, \(p_m\) enters \(v+1\) by \(T_5\), contradicting (19). \(\square \)

Proof of Theorem 1

Monotonicity is satisfied trivially. Validity, Bounded Entry, Startup, and Progress are established by Lemmas 14, 19, 20, and 21, respectively. \(\square \)

Proof of the synchronizer performance properties (Theorem 3)

The following lemma bounds the latency of entering v as a function of the time by which all correct processes have sent such \( \texttt {WISH}\)es.

Lemma 22

For all views \(v>0\) and times s, if all correct processes \(p_i\) send \( \texttt {WISH}(v_i)\) with \(v_i \ge v\) no later than at s, and some correct process enters v, then \(E_{{\textrm{last}}}(v) \le \max (s, \overline{\textsf {GST}}) + \delta \).

Proof

Fix an arbitrary correct process \(p_i\) that sends \( \texttt {WISH}(v_i)\) with \(v_i \ge v\) to all processes at time \(t_i \le s \le \max (s, \overline{\textsf {GST}})\). Since \(\max (s, \overline{\textsf {GST}}) \ge \overline{\textsf {GST}}\), by Lemma 17 there exists a time \(t_i'\) such that \( \textsf {GST}\le t_i' \le \max (s, \overline{\textsf {GST}})\) and at \(t_i'\), \(p_i\) sends \( \texttt {WISH}(v_i')\) with \(v_i' \ge v_i \ge v\) to all processes. Since \(t_i' \ge \textsf {GST}\), all correct processes receive \( \texttt {WISH}(v_i')\) from \(p_i\) no later than at \(t_i' + \delta \le \max (s, \overline{\textsf {GST}}) + \delta \).

Consider an arbitrary correct process \(p_j\) and let \(t_j \le \max (s, \overline{\textsf {GST}}) + \delta \) be the earliest time by which \(p_j\) receives \( \texttt {WISH}(v_i')\) with with \(v_i' \ge v\) from each correct processes \(p_i\). Thus, at \(t_j\), the entries of all correct processes in \(p_j.\textsf {max\_views}\) are occupied by views \(\ge v\). Since at least \(2f+1\) entries in \(p_j.\textsf {max\_views}\) belong to correct processes, the \((2f+1)\)th highest entry is \(\ge v\). Thus, \(p_j. \textsf {view}(t_j) \ge v\). Since \(p_j. \textsf {view}\) is non-decreasing, there exists a time \(t_j' \le t_j\) at which \(p_j. \textsf {view}\) first became \(\ge v\). If \(p_j. \textsf {view}(t_j') = p_j. \textsf {view}^+(t_j') = v\), then \(p_j\) enters v at \(t_j'\). Otherwise, either \(p_j. \textsf {view}(t_j') > v\) or \(p_j. \textsf {view}^+(t_j') > v\). Since both \(p_j. \textsf {view}\) and \(p_j. \textsf {view}^+\) are non-decreasing, \(p_j\) will never enter v after \(t_j'\). Thus, a correct process cannot enter v after \(\max (s, \overline{\textsf {GST}}) + \delta \). Since by the lemma’s premise, some correct process does enter v, \(E_{{\textrm{last}}}(v) \le \max (s, \overline{\textsf {GST}}) + \delta \), as needed. \(\square \)

The next lemma gives an upper bound on the duration of time a correct process may spend in a view before sending a \( \texttt {WISH}\) for a higher view.

Lemma 23

Let \(p_k\) be a correct process that enters a view v. Then \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) no later than at \(T_{\textrm{last}}(v)\).

Proof

Suppose that \(p_k\) enters a view \(v > 0\) at time \( \textsf {GST}\le s_k \le E_{{\textrm{last}}}(v)\). Then

$$\begin{aligned} p_k. \textsf {view}(s_k) = p_k. \textsf {view}^+(s_k) = v. \end{aligned}$$

By the definition of \(T_{\textrm{last}}(v)\), there exists a time \(s_k'\) such that

$$\begin{aligned} s_k \le s_k' \le T_{\textrm{last}}(v), \end{aligned}$$

and at \(s_k'\), \(p_k\) either attempts to advance from v or enters a view \(v' > v\). If \(p_k\) attempts to advance from v at \(s_k'\), then \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k = \max (p_k. \textsf {view}(s_k')+1, p_k. \textsf {view}^+(s_k'))\). Since both \(p_k. \textsf {view}\) and \(p_k. \textsf {view}^+\) are non-decreasing, we have \(p_k. \textsf {view}(s_k') \ge v\) and \(p_k. \textsf {view}^+(s_k') \ge v\). Thus, \(v_k \ge v + 1\), as required. On the other hand, if \(p_k\) enters a view \(v' > v\) at \(s_k'\), then \(v' = p_k. \textsf {view}(s_k') > p_k. \textsf {view}(s_k) = v\) and therefore, \(p_k. \textsf {view}^+(s_k') \ge p_k. \textsf {view}(s_k') \ge v+1\). Since \(p_k. \textsf {view}^+\) is non-decreasing and \(p_k. \textsf {view}^+(s_k)=v\), \(p_k. \textsf {view}^+\) must have changed its value from v to \(v_k'' \ge v+1\) at some time \(s_k''\) such that \(s_k < s_k'' \le s_k'\). Thus, the condition in line 17 holds at \(s_k''\), which means that \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) at \(s_k''\). Thus, in all cases, \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) no later than at \(T_{\textrm{last}}(v)\), as required. \(\square \)

The next lemma bounds the time by which every correct process either enters a view \(v > 0\), or sends a \( \texttt {WISH}\) messages with a view \(> v\).

Lemma 24

Consider a view \(v > 0\) such that some correct process enters v. Then, for all times t, if \(t \ge \max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})\), then \(E_{{\textrm{last}}}(v) \le t+2\delta \) and for all correct processes \(p_k\), if \(p_k\) never enters v, then, by \(E_{{\textrm{last}}}(v)\), \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) to all processes.

Proof

Since \(v>0\), \(E_{{\textrm{first}}}(v)\mathpunct {\downarrow }\), and \(t \ge E_{{\textrm{first}}}(v)\), there exists a correct process \(p_l\) such that \(p_l\) entered v and \(E_{l}(v) \le t\). By the view entry condition, \(p_l. \textsf {view}(E_{l}(v)) = v\), and therefore \(p_l.\textsf {max\_views}(E_{l}(v))\) includes \(2f+1\) entries \(\ge v\). Since \(f+1\) of these entries belong to correct processes, there exists a set C of \(f+1\) correct processes \(p_i\), each of which sent \( \texttt {WISH}(v_i)\) with \(v_i\ge v\) to all processes before \(E_{l}(v) \le t\). Since \(t \ge \overline{\textsf {GST}}\), by Lemma 17, \(p_i\) sends \( \texttt {WISH}(v_i')\) with \(v_i' \ge v_i \ge v\) sometime between \( \textsf {GST}\) and t. Since after \( \textsf {GST}\) every message sent by a correct process is received by all correct processes within \(\delta \) of its transmission, the above implies that by \(t + \delta \) every correct process receives a \( \texttt {WISH}(v_i')\) with \(v_i' \ge v\) from each process \(p_i \in C\).

Consider an arbitrary correct process \(p_j\) and let \(t_j \le t + \delta \) be the earliest time by which \(p_j\) receives \( \texttt {WISH}(v_i)\) with \(v_i \ge v\) from each process \(p_i \in C\). Thus, for all processes \(p_i\in C\), \(p_j.\textsf {max\_views}[i](t_j) \ge v\). Since \(|C|=f+1\), the \((f+1)\)th highest entry in \(p_j.\textsf {max\_views}[i](t_j)\) is \(\ge v\), and therefore, \(p_j. \textsf {view}^+(t_j) \ge v\). Then each correct process \(p_j\) sends \( \texttt {WISH}(v_j)\) with \(v_j \ge v\) to all correct processes no later than \(t_j \le t + \delta \). Since \(t+\delta > t \ge \overline{\textsf {GST}}\) and, and some correct process entered v, by Lemma 22,

$$\begin{aligned} E_{{\textrm{last}}}(v) \le t + 2\delta . \end{aligned}$$
(21)

In addition, by Lemma 17, there exists a time \(t_j'\) such that \( \textsf {GST}\le t_j' \le t+\delta \) and \(p_j\) sends \( \texttt {WISH}(v_j')\) with \(v_j' \ge v_j \ge v\) at \(t_j'\). Since a message sent by a correct process after \( \textsf {GST}\) is received by all correct processes within \(\delta \) of its transmission, all correct processes must have received \( \texttt {WISH}(v_j')\) with \(v_j' \ge v\) from each correct process \(p_j\) in-between \( \textsf {GST}\) and \(t + 2\delta \).

Suppose that \(p_k\) never enters v, and let \(t_k\) be the earliest time \(\ge \textsf {GST}\) by which \(p_k\) receives \( \texttt {WISH}(v_j')\) from each correct process \(p_j\); we have \(t_k \le t+2\delta \). Since \(v_j' \ge v\), and there are \(2f+1\) correct processes, \(p_k.\textsf {max\_views}(t_k)\) includes at least \(2f+1\) entries \(\ge v\). Thus, \(p_k. \textsf {view}(t_k) \ge v\). Since \(p_k\) never enters v, we have either \(p_k. \textsf {view}^+(t_k) \ge p_k. \textsf {view}(t_k) \ge v+1\) or \(p_k. \textsf {view}(t_k) = v \wedge p_k. \textsf {view}^+(t_k) \ge v+1\). Thus, \(p_k. \textsf {view}^+(t_k) \ge v + 1\) and therefore, \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k\ge v+1\) by \(t_k \le t+2\delta \), which combined with (21) validates the lemma. \(\square \)

We are now ready to prove the SMR synchronizer performance bounds.

Theorem 7

The SMR synchronizer in Fig. 3 satisfies Property A.

Proof

Consider a view v such that \(E_{{\textrm{first}}}(v)\mathpunct {\downarrow }\), and let \(t = \max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})\). Since \(A_{\textrm{first}}(0) < \textsf {GST}\), by the definition of \( \overline{\textsf {GST}}\), \( \overline{\textsf {GST}}= \textsf {GST}+ \rho \). Thus, \(t = \max (E_{{\textrm{first}}}(v), \textsf {GST}+ \rho )\). By Lemma 24, \(E_{{\textrm{last}}}(v) \le t + 2\delta = \max (E_{{\textrm{first}}}(v), \textsf {GST}+ \rho ) + 2\delta \), as needed. \(\square \)

Theorem 8

The SMR synchronizer in Fig. 3 satisfies Property B.

Proof

Consider a view \(v\ge 0\) such that \(E_{{\textrm{first}}}(v+1)\mathpunct {\downarrow }\). If \(v=0\), then since we assume for all correct processes \(p_i\), \(T_{i}(0)\mathpunct {\downarrow }\), by Lemma 23, all correct processes send \( \texttt {WISH}(v')\) with \(v' \ge 0\) to all processes no later than at \(T_{\textrm{last}}(0)\). Thus, by Lemma 22, \(E_{{\textrm{last}}}(1) \le \max (T_{\textrm{last}}(0), \overline{\textsf {GST}}) + \delta \). If \(A_{\textrm{first}}(0) < \textsf {GST}\), then \( \overline{\textsf {GST}}= \textsf {GST}+ \rho \), and therefore, \(E_{{\textrm{last}}}(1) \le \max (T_{\textrm{last}}(0), \textsf {GST}+ \rho ) + \delta \). Otherwise, \( \overline{\textsf {GST}}= A_{\textrm{first}}(0) \le T_{\textrm{last}}(0)\), so that \(E_{{\textrm{last}}}(1) \le T_{\textrm{last}}(0) + \delta \). Thus, the theorem holds for \(v=0\).

Suppose that \(v>0\). Since some correct process enters \(v+1\), by Proposition 1, some correct process enters view v as well. Consider a correct process \(p_k\). If \(p_k\) enters v, then by Lemma 24, \(E_{k}(v) \le \max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})+2\delta \), and therefore, by Lemma 23, \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) no later than at

$$\begin{aligned} T_{\textrm{last}}(v) > \max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})+2\delta . \end{aligned}$$
(22)

On the other hand, if \(p_k\) never enters v, then by Lemma 24, \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) than at \(\max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})+2\delta \). Thus, every correct process \(p_k\) sends \( \texttt {WISH}(v_k)\) with \(v_k \ge v+1\) no later than

$$\begin{aligned} \max (T_{\textrm{last}}(v), \max (E_{{\textrm{first}}}(v), \overline{\textsf {GST}})+2\delta ), \end{aligned}$$

which by (22), implies that all correct processes send a \( \texttt {WISH}\) message with a view \(\ge v+1\) no later than \(T_{\textrm{last}}(v)\). Thus, by Lemma 22, we have

$$\begin{aligned} E_{{\textrm{last}}}(v+1) \le \max (T_{\textrm{last}}(v), \overline{\textsf {GST}}) + \delta . \end{aligned}$$
(23)

If \(A_{\textrm{first}}(0) < \textsf {GST}\), then \( \overline{\textsf {GST}}= \textsf {GST}+ \rho \), and therefore, (23) implies that \(E_{{\textrm{last}}}(v+1) \le \max (T_{\textrm{last}}(v), \textsf {GST}+ \rho ) + \delta \), as required. Otherwise, \( \overline{\textsf {GST}}= A_{\textrm{first}}(v) \le T_{\textrm{last}}(v)\), which by (23) implies that \(E_{{\textrm{last}}}(v+1) \le T_{\textrm{last}}(v) + \delta \), validating the theorem. \(\square \)

Proof of Theorem 3

Follows from Theorems 7 and 8. \(\square \)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bravo, M., Chockler, G. & Gotsman, A. Liveness and latency of Byzantine state-machine replication. Distrib. Comput. (2024). https://doi.org/10.1007/s00446-024-00466-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00446-024-00466-4

Navigation