Towards Comprehensive Measurement of Consistency Guarantees for Cloud-Hosted Data Storage Services

  • David Bermbach
  • Liang Zhao
  • Sherif Sakr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8391)


The CAP theorem and the PACELC model have described the existence of direct trade-offs between consistency and availability as well as consistency and latency in distributed systems. Cloud storage services and NoSQL systems, both optimized for the web with high availability and low latency requirements, hence, typically opt to relax consistency guarantees. In particular, these systems usually offer eventual consistency which guarantees that all replicas will, in the absence of failures and further updates, eventually converge towards a consistent state where all replicas are identical. This, obviously, is a very imprecise description of actual guarantees.

Motivated by the popularity of eventually consistent storage systems, we take the position that a standard consistency benchmark is of great practical value. This paper is intended as a call for action; its goal is to motivate further research on building a standard comprehensive benchmark for quantifying the consistency guarantees of eventually consistent storage systems. We discuss the main challenges and requirements of such a benchmark, and present first steps towards a comprehensive consistency benchmark for cloud-hosted data storage systems. We evaluate our approach using experiments on both Cassandra and MongoDB.


Virtual Machine Storage System Cloud Storage Storage Service Cloud Storage Service 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, D.: Consistency tradeoffs in modern distributed database system design: Cap is only part of the story. Computer 45(2) (2012)Google Scholar
  2. 2.
    Anderson, E., Li, X., Shah, M.A., Tucek, J., Wylie, J.J.: What consistency does your key-value store actually provide? In: HotDep (2010)Google Scholar
  3. 3.
    Bailis, P., Venkataraman, S., Franklin, M., Hellerstein, J., Stoica, I.: Probabilistically bounded staleness for practical partial quorums. PVLDB 5(8) (2012)Google Scholar
  4. 4.
    Baker, J., Bond, C., Corbett, J., Furman, J., Khorlin, A., Larson, J., Léon, J.M., Li, Y., Lloyd, A., Yushprakh, V.: Megastore: Providing scalable, highly available storage for interactive services. In: Proc. of CIDR, pp. 223–234 (2011)Google Scholar
  5. 5.
    Bermbach, D., Tai, S.: Eventual consistency: How soon is eventual? an evaluation of amazon s3’s consistency behavior. In: Proceedings of the 6th Workshop on Middleware for Service Oriented Computing (2011)Google Scholar
  6. 6.
    Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems: An overview of models, metrics and measurement approaches. In: Gramoli, V., Guerraoui, R. (eds.) NETYS 2013. LNCS, vol. 7853, pp. 175–189. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  7. 7.
    Bermbach, D., Kuhlenkamp, J., Derre, B., Klems, M., Tai, S.: A middleware guaranteeing client-centric consistency on top of eventually consistent datastores. In: Proceedings of the 1st International Conference on Cloud Engineering (IC2E). IEEE (2013)Google Scholar
  8. 8.
    Binnig, C., Kossmann, D., Kraska, T., Loesing, S.: How is the weather tomorrow?: towards a benchmark for the cloud. In: Proceedings of the Second International Workshop on Testing Database Systems (2009)Google Scholar
  9. 9.
    Bodík, P., Fox, A., Franklin, M.J., Jordan, M.I., Patterson, D.A.: Characterizing, modeling, and generating workload spikes for stateful services. In: SoCC (2010)Google Scholar
  10. 10.
    Brewer, E.A.: Towards robust distributed systems (abstract). In: PODC (2000)Google Scholar
  11. 11.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst. 26(2) (2008)Google Scholar
  12. 12.
    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing, pp. 143–154. ACM (2010)Google Scholar
  13. 13.
    Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al.: Spanner: Google’s globally-distributed database. To appear in Proceedings of OSDI, p. 1 (2012)Google Scholar
  14. 14.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: amazon’s highly available key-value store. In: SOSP (2007)Google Scholar
  15. 15.
    Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the Cloud: What It Should, Can, and Cannot Be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Golab, W., Li, X., Shah, M.: Analyzing consistency properties for fun and profit. In: Proceedings of the 30th Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, pp. 197–206. ACM (2011)Google Scholar
  17. 17.
    Gray, J. (ed.): The Benchmark Handbook for Database and Transaction Systems, 1st edn. Morgan Kaufmann (1991)Google Scholar
  18. 18.
    Klems, M., Bermbach, D., Weinert, R.: A runtime quality measurement framework for cloud database service systems. In: Proceedings of the 8th International Conference on the Quality of Information and Communications Technology. Springer (2012)Google Scholar
  19. 19.
    Lakshman, A., Malik, P.: Cassandra: A structured storage system on a p2p network. In: Proceedings of the Twenty-First Annual Symposium on Parallelism in Algorithms and Architectures, pp. 47–47. ACM (2009)Google Scholar
  20. 20.
    Patil, S., Polte, M., Ren, K., Tantisiriroj, W., Xiao, L., López, J., Gibson, G., Fuchs, A., Rinaldi, B.: Ycsb++: benchmarking and performance debugging advanced features in scalable table stores. In: Proceedings of the 2nd ACM Symposium on Cloud Computing, p. 9. ACM (2011)Google Scholar
  21. 21.
    Rahman, M.R., Golab, W.M., AuYoung, A., Keeton, K., Wylie, J.J.: Toward a Principled Framework for Benchmarking Consistency. In: HotDep (2012)Google Scholar
  22. 22.
    Sakr, S., Liu, A., Batista, D.M., Alomari, M.: A Survey of Large Scale Data Management Approaches in Cloud Environments. IEEE Communications Surveys and Tutorials 13(3), 311–336 (2011)CrossRefGoogle Scholar
  23. 23.
    Silberstein, A., Chen, J., Lomax, D., McMillan, B., Mortazavi, M., Narayan, P.P.S., Ramakrishnan, R., Sears, R.: PNUTS in Flight: Web-Scale Data Serving at Yahoo. IEEE Internet Computing 16(1) (2012)Google Scholar
  24. 24.
    Tanenbaum, A.S., van Steen, M.: Distributed systems: principles and paradigms, 2nd edn. Pearson, Prentice Hall, Upper Saddle River, NJ (2007)Google Scholar
  25. 25.
    Vogels, W.: Eventually Consistent. Queue 6 (October 2008),
  26. 26.
    Wada, H., Fekete, A., Zhao, L., Lee, K., Liu, A.: Data Consistency Properties and the Trade-offs in Commercial Cloud Storage: the Consumers’ Perspective. In: CIDR (2011)Google Scholar
  27. 27.
    Zellag, K., Kemme, B.: How Consistent is your Cloud Application? In: SoCC (2012)Google Scholar
  28. 28.
    Zhao, L., Sakr, S., Fekete, A., Wada, H., Liu, A.: Application-Managed Database Replication on Virtualized Cloud Environments. In: ICDE Workshops on Data Management in the Cloud (DMC) (2012)Google Scholar
  29. 29.
    Zhao, L., Sakr, S., Liu, A.: Application-Managed Replication Controller for Cloud-Hosted Databases. In: IEEE CLOUD (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • David Bermbach
    • 1
  • Liang Zhao
    • 2
  • Sherif Sakr
    • 2
  1. 1.Karlsruhe Institute of TechnologyKarlsruheGermany
  2. 2.NICTAUniversity of New South WalesSydneyAustralia

Personalised recommendations