Totally Ordered Replication for Massive Scale Key-Value Stores

  • José Ribeiro
  • Nuno MachadoEmail author
  • Francisco Maia
  • Miguel Matos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10853)


Scalability is one of the most relevant features of today’s data management systems. In order to achieve high scalability and availability, recent distributed key-value stores refrain from costly replica coordination when processing requests. However, these systems typically do not perform well under churn. In this paper, we propose DataFlagons, a large-scale key-value store that integrates epidemic dissemination with a probabilistic total order broadcast algorithm. By ensuring that all replicas process requests in the same order, DataFlagons provides probabilistic strong data consistency while achieving high scalability and robustness under churn.



The authors would like to thank the anonymous reviewers for their valuable feedback. This work was partially supported by Project “TEC4Growth - Pervasive Intelligence, Enhancers and Proofs of Concept with Industrial Impact” (NORTE-01-0145-FEDER-000020), financed by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within project “POCI-01-0145-FEDER-006961”, and by National Funds through the Portuguese funding agency, FCT – Fundação para a Ciência as part of project “UID/EEA/50014/2013”.


  1. 1.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. ACM Trans. Comput. Syst. 26(2), 4:1–4:26 (2008)CrossRefGoogle Scholar
  2. 2.
    Cooper, B.F., Ramakrishnan, R., Srivastava, U., Silberstein, A., Bohannon, P., Jacobsen, H.A., Puz, N., Weaver, D., Yerneni, R.: Pnuts: Yahoo!’s hosted data serving platform. Proc. VLDB Endow. 1(2), 1277–1288 (2008)CrossRefGoogle Scholar
  3. 3.
    Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J.J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., Hsieh, W., Kanthak, S., Kogan, E., Li, H., Lloyd, A., Melnik, S., Mwaura, D., Nagle, D., Quinlan, S., Rao, R., Rolig, L., Saito, Y., Szymaniak, M., Taylor, C., Wang, R., Woodford, D.: Spanner: Google’s globally distributed database. ACM Trans. Comput. Syst. 31(3), 251–264 (2013)CrossRefGoogle Scholar
  4. 4.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: amazon’s highly available key-value store. ACM SIGOPS Oper. Syst. Rev. 41(6), 205–220 (2007)CrossRefGoogle Scholar
  5. 5.
    Eyal, I., Gencer, A.E., Sirer, E.G., Van Renesse, R.: Bitcoin-NG: a scalable blockchain protocol. In: NSDI 2016. USENIX Association (2016)Google Scholar
  6. 6.
    Koldehofe, B.: Simple gossiping with balls and bins. Stud. Inform. Univ. 3(1), 43–60 (2004)Google Scholar
  7. 7.
    Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. ACM SIGOPS Oper. Syst. Rev. 44(2), 35–40 (2010)CrossRefGoogle Scholar
  8. 8.
    Lourenço, J.R., Cabral, B., Carreiro, P., Vieira, M., Bernardino, J.: Choosing the right NoSQL database for the job: a quality attribute evaluation. J. Big Data 2(1), 18 (2015)CrossRefGoogle Scholar
  9. 9.
    Maia, F., Matos, M., Vilaça, R., Pereira, J., Oliveira, R., Riviere, E.: Dataflasks: epidemic store for massive scale systems. In: SRDS 2014. IEEE (2014)Google Scholar
  10. 10.
    Matos, M., Mercier, H., Felber, P., Oliveira, R., Pereira, J.: EpTO: an epidemic total order algorithm for large-scale distributed systems. In: Middleware 2015. ACM (2015)Google Scholar
  11. 11.
    Rhea, S., Geels, D., Roscoe, T., Kubiatowicz, J.: Handling churn in a DHT. In: Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATC 2004, p. 10. USENIX Association, Berkeley (2004)Google Scholar
  12. 12.
    Vogels, W.: Eventually consistent. Commun. ACM 52(1), 40–44 (2009)CrossRefGoogle Scholar
  13. 13.
    Voulgaris, S., Gavidia, D., van Steen, M.: CYCLON: inexpensive membership management for unstructured P2P overlays. J. Netw. Syst. Manag. 13(2), 197–217 (2005)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2018

Authors and Affiliations

  • José Ribeiro
    • 1
  • Nuno Machado
    • 1
    Email author
  • Francisco Maia
    • 1
  • Miguel Matos
    • 2
  1. 1.HASLab – INESC TECUniversidade do MinhoBragaPortugal
  2. 2.INESC-IDInstituto Superior Técnico, Universidade de LisboaLisbonPortugal

Personalised recommendations