Skip to main content
SpringerLink
Log in

A History of the Virtual Synchrony Replication Model

  • Chapter
Replication

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 5959))

Abstract

In this chapter, we discuss a widely used fault-tolerant data replication model called virtual synchrony. The model responds to two kinds of needs. First, there is the practical question of how best to embed replication into distributed systems. Virtual synchrony defines dynamic process groups that have self-managed membership. Applications can join or leave groups at will: a process group is almost like a replicated variable that lives in the network. The second need relates to performance. Although state machine replication is relatively easy to understand, protocols that implement state machine replication in the standard manner are too slow to be useful in demanding settings, and are hard to deploy in very large data centers of the sort seen in today’s cloud-computing environments. Virtual synchrony implementations, in contrast, are able to deliver updates at the same data rates (and with the same low latencies) as IP multicast: the fast (but unreliable) Internet multicast protocol, often supported directly by hardware. The trick that makes it possible to achieve these very high levels of performance is to hide overheads by piggybacking extra information on regular messages that carry updates. The virtual synchrony replication model has been very widely adopted, and was used in everything from air traffic control and stock market systems to data center management platforms marketed by companies like IBM and Microsoft. Moreover, in recent years, state machine protocols such as those used in support of Paxos have begun to include elements of the virtual synchrony model, such as self-managed and very dynamic membership. Our exploration of the model takes the form of a history. We start by exploring the background, and then follow evolution of the model over time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aguilera, M., Merchant, A., Shah, M., Veitch, A., Karamanolis, C.: Sinfonia: a new paradigm for building scalable distributed systems. In: 21st ACM SOSP, Nov. 2007, pp. 159–174 (2007)

    Google Scholar 

  2. Amir, Y., Nita-Rotaru, C., Stanton, J., Tsudik, G.: Secure Spread: An Integrated Architecture for Secure Group Communication. IEEE TDSC 2(3) (2005)

    Google Scholar 

  3. Babaoglu, Ö., Bartoli, A., Dini, G.: Enriched view synchrony: A programming paradigm for partitionable asynchronous distributed systems. IEEE Transactions on Computers 46(6), 642–658 (1997)

    Article  Google Scholar 

  4. Bernstein, P., Goodman, N.: Concurrency Control in Distributed Database Systems. ACM Computing Surveys 13(2) (1981)

    Google Scholar 

  5. Bernstein, P., Goodman, N.: An algorithm for concurrency control and recovery in replicated distributed databases. ACM Transactions on Database Systems 9(4), 596–615 (1984)

    Article  MathSciNet  Google Scholar 

  6. Bhatti, N., Hiltunen, M., Schlichting, R., Chiu, W.: Coyote: A System for Constructing Fine-Grain Configurable Communication Services. ACM Transactions on Computer Systems 16(4), 321–366 (1998)

    Article  Google Scholar 

  7. Biagioni, E., Harper, R., Lee, P.: A Network Protocol Stack in Standard ML. Journal of Higher-Order and Symbolic Computation 14(4) (2001)

    Google Scholar 

  8. Birman, K.: Replication and Fault-Tolerance in the ISIS System. In: 10th ACM Symposium on Operating Systems Principles, Dec. 1985, pp. 79–86 (1985)

    Google Scholar 

  9. Birman, K.: Responses to Cheriton and Skeen’s SOSP paper on Understanding the Limitations of Causal and Total Event Ordering. SIGOPS Operating Systems Review 28(1) (1994)

    Google Scholar 

  10. Birman, K.: A review of experiences with reliable multicast. Software Practice and Experience 29(9) (1999)

    Google Scholar 

  11. Birman, K.: Reliable Distributed Systems. Springer, New York (2004)

    Google Scholar 

  12. Birman, K., Chandersekaran, C., Dolev, D., Van Renesse, R.: How the Hidden Hand Shapes the Market for Software Reliability. In: Proceedings IEEE Workshop on Applied Software Reliability, Philadelphia, PA (June 2006)

    Google Scholar 

  13. Birman, K., Joseph, T.: Reliable communication in the presence of failures. Tech. Rep. TR85-694 (August 1985)

    Google Scholar 

  14. Birman, K., Joseph, T.: Exploiting Virtual Synchrony in Distributed Systems. In: 11th ACM Symposium on Operating Systems Principles (Dec. 1987)

    Google Scholar 

  15. Birman, K., Joseph, T.: Reliable communication in the presence of failures. ACM Transactions on Computer Systems 5(1) (1987)

    Google Scholar 

  16. Birman, K., Joseph, T., Raeuchle, T., El Abbadi, A.: Implementing Fault-Tolerant Distributed Objects. IEEE Transactions on Software Engineering 11(6) (1985)

    Google Scholar 

  17. Birman, K., Schiper, A., Stephenson, P.: Lightweight causal and atomic group multicast. ACM Transactions on Computer Systems 9(3), 272–314 (1991)

    Article  Google Scholar 

  18. Briot, J., Guerraoui, R., Lohr, K.: Concurrency and Distribution in Object-Oriented Programming. ACM Comput. Surv. 30(3), 291–329 (1998)

    Article  Google Scholar 

  19. Burrows, M.: The Chubby Lock Service for Loosely-Coupled Distributed Systems. In: OSDI, pp. 335–350 (2006)

    Google Scholar 

  20. Chandra, T., Hadzilacos, V., Toueg, S., Charron-Bost, B.: On the impossibility of group membership. In: Proc. 15th PODC, May 23-26, 1996, pp. 322–330 (1996)

    Google Scholar 

  21. Chandra, T.D., Griesemer, R., Redstone, J.: Paxos Made Live — An Engineering Perspective (based on Chandra’s 2006 invited talk). In: Proc. 26th PODC, Aug. 2007, pp. 398–407 (2007)

    Google Scholar 

  22. Chang, J., Maxemchuk, N.: Reliable broadcast protocols. ACM Trans. on. Computer Systems 2(3), 251–273 (1984)

    Article  Google Scholar 

  23. Cheriton, D., Skeen, D.: Understanding the Limitations of Causally and Totally Ordered Communication. In: SOSP, pp. 44–57 (1993)

    Google Scholar 

  24. Cheriton, D., Zwaenepoel, W.: Distributed process groups in the V Kernel. ACM Transactions on Computer Systems (TOCS) 3(2), 77–107 (1985)

    Article  Google Scholar 

  25. Chockler, G., Keidar, I., Vitenberg, R.: Group Communication Specifications: A Comprehensive Study. ACM Computing Surveys 33(4) (2001)

    Google Scholar 

  26. Chrysanthis, P.K., Ramamritham, K.: ACTA: the SAGA Continues. In: Elmagarmid, A. (ed.) Database transaction models for advanced applications, Morgan Kaufmann, San Francisco (1992)

    Google Scholar 

  27. Cristian, F., Aghili, H., Strong, R., Volev, D.: Atomic Broadcast: From Simple Message Diffusion to Byzantine Agreement. In: Proc. 15th Int. Symp. on Fault-Tolerant Computing (FTCS-15), Ann Arbor, MI, USA, June 1985, pp. 200–206. IEEE Computer Society Press, Los Alamitos (1985)

    Google Scholar 

  28. Damm, C., Eugster, P., Guerraoui, R.: Linguistic Support for Distributed Programming Abstractions. In: CDCS, pp. 244–251 (2004)

    Google Scholar 

  29. Dekel, E., et al.: Distribution and Consistency Services (DCS), http://www.haifa.ibm.com/projects/systems/dcs/index.html

  30. Dekel, E., Frenkel, O., Goft, G., Moatti, Y.: Easy: engineering high availability QoS in wservices. In: Proc. 22nd Reliable Distributed Systems, pp. 157–166 (2003)

    Google Scholar 

  31. Dolev, D., Malkhi, D.: The Transis Approach to High Availability Cluster Communication. Comm. ACM 39(4), 87–92 (1996)

    Article  Google Scholar 

  32. Eugster, P.: Type-based Publish/Subscribe: Concepts and Experiences. ACM Transactions on Programming Languages and Systems (TOPLAS) 29(1) (2007)

    Google Scholar 

  33. Eugster, P., Damm, C., Guerraoui, R.: Towards Safe Distributed Application Development. In: ICSE, pp. 347–356 (2004)

    Google Scholar 

  34. Eugster, P., Guerraoui, R., Damm, C.: On Objects and Events. In: OOPSLA, pp. 254–269 (2001)

    Google Scholar 

  35. Fischer, M., Lynch, N.A., Paterson, M.: Impossibility of distributed consensus with one faulty process (initially published in ACM PODS, August 1983). Journal of the ACM (JACM) 32(2) (1985)

    Google Scholar 

  36. Guerraoui, R., Schiper, A.: Consensus Service: A Modular Approach for Building Agreement Protocols in Distributed Systems. In: Proc. 26th FTCS, Japan, June 1996, pp. 168–177 (1996)

    Google Scholar 

  37. Hayden, M.: The Ensemble System. Ph.D. thesis, Cornell University, available as TR 98-1662 (May 1998)

    Google Scholar 

  38. Herlihy, M., Wing, J.: Linearizability: A Correctness Condition for Concurrent Objects. ACM TOPLAS 12(3), 463–492 (1990)

    Article  Google Scholar 

  39. Hutchinson, N.C., Peterson, L.L.: The x-Kernel: An architecture for implementing network protocols. IEEE Trans. Software Eng. 17(1) (1991)

    Google Scholar 

  40. Joseph, T.A., Birman, K.: Low Cost Management of Replicated Data in Fault-Tolerant Distributed Systems. ACM Trans. Comput. Syst. 4(1), 54–70 (1986)

    Article  Google Scholar 

  41. Kaashoek, M.F., Tanenbaum, A.S., Verstoep, K.: Group Communication in Amoeba and its Applications. Distributed Systems Engineering 1(1), 48–58 (1993)

    Article  Google Scholar 

  42. Keidar, I., Dolev, D.: Increasing the Resilience of Atomic Commit at no Additional Cost. In: ACM PODS, May 1995, pp. 245–254 (1995)

    Google Scholar 

  43. Keidar, I., Khazan, R., Lynch, N., Shvartsman, A.: An Inheritance-Based Technique for Building Simulation Proofs Incrementally. ACM TOSEM 11(1) (2002)

    Google Scholar 

  44. Krumvieda, C.: Distributed ML: Abstractions for Efficient and Fault-Tolerant Programming. Ph.D. thesis, Cornell University, available as TR 93-1376 (1993)

    Google Scholar 

  45. Lamport, L.: Time, Clocks, and the Ordering of Events in a Distributed System. Comm. ACM 21(7) (1978)

    Google Scholar 

  46. Lamport, L.: Using Time Instead of Timeout for Fault-Tolerant Distributed Systems. ACM TOPLAS 6(2) (1984)

    Google Scholar 

  47. Lamport, L.: The temporal logic of actions. ACM TOPLAS 16(3), 872–923 (1994)

    Article  Google Scholar 

  48. Lamport, L.: The Part-Time Parliament (technical report version: 1990). ACM Transactions on Computer Systems 16(2), 133–169 (1998)

    Article  Google Scholar 

  49. Liskov, B., Scheifler, R.: Guardians and Actions: Linguistic Support for Robust, Distributed Programs. ACM TOPLAS 5(3) (1983)

    Google Scholar 

  50. Liu, X., Kreitz, C., van Renesse, R., Hickey, J., Hayden, M., Birman, K., Constable, R.: Building reliable, high-performance communication systems from components. In: 17th ACM SOSP (Dec. 1999)

    Google Scholar 

  51. Lynch, N., Tuttle, M.: An Introduction to Input/Output automata (also Technical Memo MIT/LCS/TM-373, Laboratory for Computer Science, Massachusetts Institute of Technology). CWI Quarterly 2(3), 219–246 (1989)

    MathSciNet  MATH  Google Scholar 

  52. Malloth, C.P., Felber, P., Schiper, A., Wilhelm, U.: Phoenix: A Toolkit for Building Fault-Tolerant Distributed Applications in Large Scale. In: Proc. of IEEE Workshop on Parallel and Distributed Platforms in Industrial Products, San Antonio, TX (Oct. 1995)

    Google Scholar 

  53. Malloth, C.P., Schiper, A.: View Synchronous Communication in the Internet. Tech. Rep. 94/84, EPFL (Oct. 1994)

    Google Scholar 

  54. Manferdelli, J.: Microsoft Corporation. Unpublished correspondence (Oct. 2007)

    Google Scholar 

  55. McKendry, M.S.: Clouds: A fault-tolerant distributed operating system. IEEE Tech. Com. Distributed Processing Newsletter 2(6) (1984)

    Google Scholar 

  56. Mishra, S., Peterson, L.L., Schlichting, R.D.: Experience with modularity in Consul. Software—Practice and Experience 23(10) (1993)

    Google Scholar 

  57. Moser, L.E., Amir, Y., Melliar-Smith, P.M., Agarwal, D.A.: Extended virtual synchrony. In: Proceedings of the 14th IEEE International Conference on Distributed Computing Systems, Poznan, Poland, June 1994, pp. 56–65 (1994)

    Google Scholar 

  58. Moser, L.E., Melliar-Smith, P.M., Agarwal, D., Budhia, R.K., Lingley-Papadopoulos, C.A., Archambault, T.: The Totem system. In: Proceedings of the 25th Annual International Symposium on Fault-Tolerant Computing, Pasadena, CA (June 1995)

    Google Scholar 

  59. Moser, L.E., Melliar-Smith, P.M., Narasimhan, P.: The Eternal System. In: Workshop on Dependable Distributed Object Systems, OOPSLA’97, Atlanta, Georgia (October 1997)

    Google Scholar 

  60. Ostrowski, K., Birman, K., Dolev, D.: Live Distributed Objects: Enabling the Active Web. IEEE Internet Computing (Nov./Dec. 2007)

    Google Scholar 

  61. Ostrowski, K., Birman, K., Dolev, D.: QuickSilver Scalable Multicast. In: Network Computing and Applications (NCA), Cambridge, MA (2008)

    Google Scholar 

  62. Ostrowski, K., Birman, K., Dolev, D., Ahnn, J.H.: Programming with live distributed objects. In: Vitek, J. (ed.) ECOOP 2008. LNCS, vol. 5142, pp. 463–489. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  63. Randell, B., Xu, J.: The Evolution of the Recovery Block Concept. In: Lyu, M.R. (ed.) Software Fault Tolerance, pp. 1–21. John Wiley & Sons, Chichester (1995)

    Google Scholar 

  64. Reed, B., Junqueira, F., Konar, M.: Zookeeper: Because Building Distributed Systems is a Zoo. Submitted for publication (Oct. 2007)

    Google Scholar 

  65. Ricciardi, A., Birman, K.: Using Process Groups to Implement Failure Detection in Asynchronous Environments. In: PODC, pp. 341–353 (1991)

    Google Scholar 

  66. Schiper, A., Sandoz, A.: Uniform reliable multicast in a Virtually Synchronous Environment. In: Proc. 13th ICDCS, Pittsburgh (May 1993)

    Google Scholar 

  67. Schiper, A., Sandoz, A.: Primary Partition “Virtually-Synchronous Communication” Harder than Consensus. In: Tel, G., Vitányi, P.M.B. (eds.) WDAG 1994. LNCS, vol. 857, pp. 39–52. Springer, Heidelberg (1994)

    Chapter  Google Scholar 

  68. Schneider, F.: Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Computing Surveys 22(4), 299–319 (1990)

    Article  Google Scholar 

  69. Schneider, F., Schlichting, R.: Fail-stop processors: An approach to designing fault-tolerant computing systems. TOCS 1(3), 222–238 (1983)

    Article  Google Scholar 

  70. Skeen, D.: Nonblocking Commit Protocols. In: Proc. ACM SIGMOD, pp. 133–142 (1981)

    Google Scholar 

  71. Skeen, D.: Determining the Last Process to Fail. In: ACM PODS, pp. 16–24 (1983)

    Google Scholar 

  72. Van Renesse, R., Birman, K.: Reliable Distributed Computing with the Isis Toolkit. IEEE Computer Society Press, Los Alamitos (1994)

    Google Scholar 

  73. Van Renesse, R., Birman, K., Maffeis, S.: Horus: A Flexible Group Communication System. Communications of the ACM 39(4), special issue on Group Communication Systems (1996)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cornell University, USA

    Ken Birman

Authors
  1. Ken Birman
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. CNRS, École Polytechnique, 91128, Palaiseau Cedex, France

    Bernadette Charron-Bost

  2. Facoltà di scienze informatiche, Università della Svizzera italiana (USI), Via Giuseppe Buffi 6, 6900, Lugano, Switzerland

    Fernando Pedone

  3. School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Station 14, 1015, Lausanne, Switzerland

    André Schiper

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Birman, K. (2010). A History of the Virtual Synchrony Replication Model. In: Charron-Bost, B., Pedone, F., Schiper, A. (eds) Replication. Lecture Notes in Computer Science, vol 5959. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11294-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-11294-2_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-11293-5

  • Online ISBN: 978-3-642-11294-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation