Abstract
The CAP theorem and the PACELC model have described the existence of direct trade-offs between consistency and availability as well as consistency and latency in distributed systems. Cloud storage services and NoSQL systems, both optimized for the web with high availability and low latency requirements, hence, typically opt to relax consistency guarantees. In particular, these systems usually offer eventual consistency which guarantees that all replicas will, in the absence of failures and further updates, eventually converge towards a consistent state where all replicas are identical. This, obviously, is a very imprecise description of actual guarantees.
Motivated by the popularity of eventually consistent storage systems, we take the position that a standard consistency benchmark is of great practical value. This paper is intended as a call for action; its goal is to motivate further research on building a standard comprehensive benchmark for quantifying the consistency guarantees of eventually consistent storage systems. We discuss the main challenges and requirements of such a benchmark, and present first steps towards a comprehensive consistency benchmark for cloud-hosted data storage systems. We evaluate our approach using experiments on both Cassandra and MongoDB.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Abadi, D.: Consistency tradeoffs in modern distributed database system design: Cap is only part of the story. Computer 45(2) (2012)
Anderson, E., Li, X., Shah, M.A., Tucek, J., Wylie, J.J.: What consistency does your key-value store actually provide? In: HotDep (2010)
Bailis, P., Venkataraman, S., Franklin, M., Hellerstein, J., Stoica, I.: Probabilistically bounded staleness for practical partial quorums. PVLDBÂ 5(8) (2012)
Baker, J., Bond, C., Corbett, J., Furman, J., Khorlin, A., Larson, J., Léon, J.M., Li, Y., Lloyd, A., Yushprakh, V.: Megastore: Providing scalable, highly available storage for interactive services. In: Proc. of CIDR, pp. 223–234 (2011)
Bermbach, D., Tai, S.: Eventual consistency: How soon is eventual? an evaluation of amazon s3’s consistency behavior. In: Proceedings of the 6th Workshop on Middleware for Service Oriented Computing (2011)
Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems: An overview of models, metrics and measurement approaches. In: Gramoli, V., Guerraoui, R. (eds.) NETYS 2013. LNCS, vol. 7853, pp. 175–189. Springer, Heidelberg (2013)
Bermbach, D., Kuhlenkamp, J., Derre, B., Klems, M., Tai, S.: A middleware guaranteeing client-centric consistency on top of eventually consistent datastores. In: Proceedings of the 1st International Conference on Cloud Engineering (IC2E). IEEE (2013)
Binnig, C., Kossmann, D., Kraska, T., Loesing, S.: How is the weather tomorrow?: towards a benchmark for the cloud. In: Proceedings of the Second International Workshop on Testing Database Systems (2009)
BodÃk, P., Fox, A., Franklin, M.J., Jordan, M.I., Patterson, D.A.: Characterizing, modeling, and generating workload spikes for stateful services. In: SoCC (2010)
Brewer, E.A.: Towards robust distributed systems (abstract). In: PODC (2000)
Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst. 26(2) (2008)
Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing, pp. 143–154. ACM (2010)
Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al.: Spanner: Google’s globally-distributed database. To appear in Proceedings of OSDI, p. 1 (2012)
DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: amazon’s highly available key-value store. In: SOSP (2007)
Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the Cloud: What It Should, Can, and Cannot Be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013)
Golab, W., Li, X., Shah, M.: Analyzing consistency properties for fun and profit. In: Proceedings of the 30th Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, pp. 197–206. ACM (2011)
Gray, J. (ed.): The Benchmark Handbook for Database and Transaction Systems, 1st edn. Morgan Kaufmann (1991)
Klems, M., Bermbach, D., Weinert, R.: A runtime quality measurement framework for cloud database service systems. In: Proceedings of the 8th International Conference on the Quality of Information and Communications Technology. Springer (2012)
Lakshman, A., Malik, P.: Cassandra: A structured storage system on a p2p network. In: Proceedings of the Twenty-First Annual Symposium on Parallelism in Algorithms and Architectures, pp. 47–47. ACM (2009)
Patil, S., Polte, M., Ren, K., Tantisiriroj, W., Xiao, L., López, J., Gibson, G., Fuchs, A., Rinaldi, B.: Ycsb++: benchmarking and performance debugging advanced features in scalable table stores. In: Proceedings of the 2nd ACM Symposium on Cloud Computing, p. 9. ACM (2011)
Rahman, M.R., Golab, W.M., AuYoung, A., Keeton, K., Wylie, J.J.: Toward a Principled Framework for Benchmarking Consistency. In: HotDep (2012)
Sakr, S., Liu, A., Batista, D.M., Alomari, M.: A Survey of Large Scale Data Management Approaches in Cloud Environments. IEEE Communications Surveys and Tutorials 13(3), 311–336 (2011)
Silberstein, A., Chen, J., Lomax, D., McMillan, B., Mortazavi, M., Narayan, P.P.S., Ramakrishnan, R., Sears, R.: PNUTS in Flight: Web-Scale Data Serving at Yahoo. IEEE Internet Computing 16(1) (2012)
Tanenbaum, A.S., van Steen, M.: Distributed systems: principles and paradigms, 2nd edn. Pearson, Prentice Hall, Upper Saddle River, NJ (2007)
Vogels, W.: Eventually Consistent. Queue 6 (October 2008), http://doi.acm.org/10.1145/1466443.1466448
Wada, H., Fekete, A., Zhao, L., Lee, K., Liu, A.: Data Consistency Properties and the Trade-offs in Commercial Cloud Storage: the Consumers’ Perspective. In: CIDR (2011)
Zellag, K., Kemme, B.: How Consistent is your Cloud Application? In: SoCC (2012)
Zhao, L., Sakr, S., Fekete, A., Wada, H., Liu, A.: Application-Managed Database Replication on Virtualized Cloud Environments. In: ICDE Workshops on Data Management in the Cloud (DMC) (2012)
Zhao, L., Sakr, S., Liu, A.: Application-Managed Replication Controller for Cloud-Hosted Databases. In: IEEE CLOUD (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Bermbach, D., Zhao, L., Sakr, S. (2014). Towards Comprehensive Measurement of Consistency Guarantees for Cloud-Hosted Data Storage Services. In: Nambiar, R., Poess, M. (eds) Performance Characterization and Benchmarking. TPCTC 2013. Lecture Notes in Computer Science, vol 8391. Springer, Cham. https://doi.org/10.1007/978-3-319-04936-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-04936-6_3
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-04935-9
Online ISBN: 978-3-319-04936-6
eBook Packages: Computer ScienceComputer Science (R0)