Evaluating the Price of Consistency in Distributed File Storage Services

  • José Valerio
  • Pierre Sutra
  • Étienne Rivière
  • Pascal Felber
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7891)


Distributed file storage services (DFSS) such as Dropbox, iCloud, SkyDrive, or Google Drive, offer a filesystem interface to a distributed data store. DFSS usually differ in the consistency level they provide for concurrent accesses: a client might access a cached version of a file, see the immediate results of all prior operations, or temporarily observe an inconsistent state. The selection of a consistency level has a strong impact on performance. It is the result of an inherent tradeoff between three properties: consistency, availability, and partition-tolerance. Isolating and identifying the exact impact on performance is a difficult task, because DFSS are complex designs with multiple components and dependencies. Furthermore, each system has a different range of features, its own design and implementation, and various optimizations that do not allow for a fair comparison. In this paper, we make a step towards a principled comparison of DFSS components, focusing on the evaluation of consistency mechanisms. We propose a novel modular DFSS testbed named FlexiFS, which implements a range of state-of-the-art techniques for the distribution, replication, routing, and indexing of data. Using FlexiFS, we survey six consistency levels: linearizability, sequential consistency, and eventual consistency, each operating with and without close-to-open semantics. Our evaluation shows that: (i) as expected, POSIX semantics (i.e., linearizability without close-to-open semantics) harm performance; and (ii) when close-to-open semantics is in use, linearizability delivers performance similar to sequential or eventual consistency.


Storage System Eventual Consistency Storage Node Replication Factor Consistency Level 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Gilbert, S., Lynch, N.: Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33(2), 51–59 (2002)CrossRefGoogle Scholar
  2. 2.
    IEEE, The Open Group: Standard for Information Technology-Portable Operating System Interface (POSIX) System Interfaces (2004)Google Scholar
  3. 3.
    Howard, J.H., Kazar, M.L., Menees, S.G., Nichols, D.A., Satyanarayanan, M., Sidebotham, R.N., West, M.J.: Scale and performance in a distributed file system. ACM Trans. Comput. Syst. 6(1) (February 1988)Google Scholar
  4. 4.
    The Apache Software Foundation: The Hadoop Distributed File System (2012)Google Scholar
  5. 5.
    Ghemawat, S., Gobioff, H., Leung, S.T.: The google file system. In: SOSP (2003)Google Scholar
  6. 6.
    Shvachko, K.V.: HDFS Scalability: The Limits to Growth. USENIX login 35(2) (April 2010)Google Scholar
  7. 7.
    Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44(2) (April 2010)Google Scholar
  8. 8.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: amazon’s highly available key-value store. In: SOSP (2007)Google Scholar
  9. 9.
    Dabek, F., Kaashoek, M.F., Karger, D., Morris, R., Stoica, I.: Wide-area cooperative storage with CFS. In: SOSP (2001)Google Scholar
  10. 10.
    Muthitacharoen, A., Morris, R., Gil, T.M., Chen, B.: Ivy: a read/write peer-to-peer file system. In: OSDI (2002)Google Scholar
  11. 11.
    Busca, J.-M., Picconi, F., Sens, P.: Pastis: A highly-scalable multi-user peer-to-peer file system. In: Cunha, J.C., Medeiros, P.D. (eds.) Euro-Par 2005. LNCS, vol. 3648, pp. 1173–1182. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    File System in User Space,
  13. 13.
    Karger, D., Lehman, E., Leighton, T., Panigrahy, R., Levine, M., Lewin, D.: Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the world wide web. In: STOC (1997)Google Scholar
  14. 14.
    Nelson, M.N., Welch, B.B., Ousterhout, J.K.: Caching in the sprite network file system. ACM Trans. Comput. Syst. 6(1) (February 1988)Google Scholar
  15. 15.
    Sun Microsystems, Inc.: NFS: Network file system protocol specification. RFC 1094, Network Information Center, SRI International (March 1989)Google Scholar
  16. 16.
    Aguilera, M.K., Merchant, A., Shah, M., Veitch, A., Karamanolis, C.: Sinfonia: a new paradigm for building scalable distributed systems. In: SOSP (2007)Google Scholar
  17. 17.
    Satyanarayanan, M., Kistler, J.J., Kumar, P., Okasaki, M.E., Siegel, E.H., Steere, D.C.: Coda: A highly available file system for a distributed workstation environment. IEEE Trans. Comput. 39(4) (April 1990)Google Scholar
  18. 18.
    Herlihy, M., Wing, J.: Linearizability: a correcteness condition for concurrent objects. ACM Trans. Prog. Lang. and Sys. 12(3) (July 1990)Google Scholar
  19. 19.
    Lamport, L.: The part-time parliament. ACM Trans. Comp. Sys. 16(2) (1998)Google Scholar
  20. 20.
    Lamport, L.: How to make a correct multiprocess program execute correctly on a multiprocessor. IEEE Trans. Comput. 46(7), 779–782 (1997)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Chandra, T.D., Toueg, S.: Unreliable failure detectors for reliable distributed systems. Journal of the ACM 43(2), 225–267 (1996)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Fekete, A., Gupta, D., Luchangco, V., Lynch, N., Shvartsman, A.: Eventually-serializable data services. Theoretical Computer Science 220 (1999)Google Scholar
  23. 23.
    Saito, Y., Shapiro, M.: Optimistic replication. Computing Surveys 37(1) (2005)Google Scholar
  24. 24.
    Saltzer, J.H., Kaashoek, M.F.: Principles of Computer System Design: An Introduction. Morgan Kaufmann Publishers Inc. (2009)Google Scholar
  25. 25.
    Levy, E., Silberschatz, A.: Distributed file systems: concepts and examples. ACM Comput. Surv. 22(4) (December 1990)Google Scholar
  26. 26.
    Leonini, L., Rivière, E., Felber, P.: SPLAY: Distributed systems evaluation made simple. In: NSDI (2009)Google Scholar
  27. 27.
    Walker, B., Popek, G., English, R., Kline, C., Thiel, G.: The LOCUS distributed operating system. In: SOSP (1983)Google Scholar
  28. 28.
    Kubiatowicz, J., Bindel, D., Chen, Y., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., Wells, C., Zhao, B.: Oceanstore: An architecture for global-scale persistent storage. In: ASPLOS (2000)Google Scholar
  29. 29.
    Stoica, I., Morris, R., Liben-Nowell, D., Karger, D.R., Kaashoek, M.F., Dabek, F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Trans. Net (1) (February 2003)Google Scholar
  30. 30.
    Stamatakis, D., Tsikoudis, N., Smyrnaki, O., Magoutis, K.: Scalability of replicated metadata services in distributed file systems. In: Göschka, K.M., Haridi, S. (eds.) DAIS 2012. LNCS, vol. 7272, pp. 31–44. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  31. 31.
    Geoffray, N., Thomas, G., Lawall, J., Muller, G., Folliot, B.: VMKit: a substrate for managed runtime environments. In: VEE (2010)Google Scholar
  32. 32.
    PolyORB Middleware Technology,

Copyright information

© IFIP International Federation for Information Processing 2013

Authors and Affiliations

  • José Valerio
    • 1
  • Pierre Sutra
    • 1
  • Étienne Rivière
    • 1
  • Pascal Felber
    • 1
  1. 1.University of NeuchâtelSwitzerland

Personalised recommendations