A Load-Aware Data Placement Policy on Cluster File System

  • Yu Wang
  • Jing Xing
  • Jin Xiong
  • Dan Meng
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6985)


In a large-scale cluster system with many applications running on it, cluster-wide I/O access workload disparity and disk saturation on only some storage servers have been the severe performance bottleneck that deteriorates the system I/O performance. As a result, the system response time will increase and the throughput of the system will decrease drastically. In this paper, we present a load-aware data placement policy that will distribute data across the storage servers based on the load of each server and automatically migrate data from heavily-loaded servers to lightly-loaded servers. This policy is adaptive and self-managing. It operates without any prior knowledge of application access workload characteristics or the capabilities of storage servers. It can make full use of the aggregate disk bandwidth of all storage servers efficiently. Performance evaluation shows that our policy will improve the aggregate I/O bandwidth by 10%-20% compared with random data placement policy especially under mixed workloads.


Cluster File System Data Placement 


  1. 1.
  2. 2.
    DOE National Nuclear Security Administration and the DOE National Security Agency. SGS file system (April 2001)Google Scholar
  3. 3.
    Kramer, W.T.C., Shoshani, A., Agarwal, D.A., Draney, B.R., Jin, G., Butler, G.F., Hules, J.A.: Deep scientific computing requires deep data. IBM J. Res. Dev. 48(2), 209–232 (2004)CrossRefGoogle Scholar
  4. 4.
  5. 5.
    Wang, F., Xin, Q., Hong, B., Brandt, S.A., Miller, E.L., Long, D.D.E., McLarty, T.T.: File system workload analysis for large scale scientific computing applications. In: Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, College Park, MD (April 2004)Google Scholar
  6. 6.
    Leung, A.W., Pasupathy, S., Goodson, G., Miller, E.L.: Measurement and analysis of large scale network file system workloads. In: Proceedings of the 2008 USENIX Annual Technical Conference (June 2008)Google Scholar
  7. 7.
    Evans, K.M., Kuenning, G.H.: Irregularities in file-size distributions. In: Proceedings of the 2nd International Symposiusm on Performance Evaluation of Computer and Telecommunication Systems, SPECTS (July 2002)Google Scholar
  8. 8.
    McKusick, M., Joy, W., Leffler, S., Fabry, R.: A Fast File System for UNIX. ACM Trans. on Computer Systems 2(3), 181–197 (1984)CrossRefGoogle Scholar
  9. 9.
    Lawrence Livermore National Laboratory. IOR software,
  10. 10.
    Zhu, Y., Jiang, H., Qin, X., Swanson, D.: A Case Study of Parallel I/O for Biological Sequence Analysis on Linux Clusters. In: Proceedings of Cluster 2003, Hong Kong, December 1-4 (2003)Google Scholar
  11. 11.
    Karger, D., Lehman, E., Leighton, T., Levine, M., Levin, D., Panigraphy, R.: Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web. In: Proceedings of ACM Symposium on Theory of Computing (STOC 1997), pp. 654–663 (1997)Google Scholar
  12. 12.
    Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup service for internet applications. In: Proc. of the 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM 2001), San Diego, CA (August 2001)Google Scholar
  13. 13.
    Choy, D.M., Fagin, R., Stockmeyer, L.: Efficiently extendible mappings for balanced data distribution. Algorithmica 16, 215–232 (1996)MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Honicky, R.J., Miller, E.L.: Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution. In: Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), Santa Fe, NM (April 2004)Google Scholar
  15. 15.
    Weil, S.A., Brandt, S.A., Miller, E.L., Maltzahn, C.: CRUSH: Controlled, scalable, decentralized placement of replicated data. In: Proc. of the 2006 ACM/IEEE Conference on Supercomputing, Tampa, FL (November 2006)Google Scholar
  16. 16.
    Brinkmann, A., Salzwedel, K., Scheideler, C.: Efficient, distributed data placement strategies for storage area networks. In: In Proceedings of the 12th ACM Symposium on Parallel Algorithms and Architectures SPAA (2000)Google Scholar
  17. 17.
    Wu, C., Lau, F.: Load Balancing in Parallel Computers: Theory and Practice. Kluwer Academic Publishers, Boston (1997)Google Scholar
  18. 18.
    Zhou, S., Wang, J., Zheng, X., Delisle, P.: Utopia: A load-sharing facility for large heterogeneous distributed computing systems. Software Practice and Experience 23(12) (1993)Google Scholar
  19. 19.
    Zhu, H., Yang, T., Zheng, Q., Watson, D., Ibarra, O.H.: Adaptive load sharing for clustered digital library servers. International Journal on Digital Libraries 2(4) (2000)Google Scholar
  20. 20.
    Lee, L., Scheauermann, P., Vingralek, R.: File Assignment in Parallel I/O Systems with Minimal Variance of Service time. IEEE Trans. on Computers 49Google Scholar
  21. 21.
    Xiao, Q., Jiang, H., Zhu, Y., Swanson, D.: Toward load balancing support for I/O intensive parallel jobs in a cluster of workstation. In: Proc. of the 5th IEEE International Conference Cluster Computing, Hong Kong (December 14, 2003)Google Scholar
  22. 22.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: “Dynamo: Amazon’s Highly Available Key-Value Store”. In: ACM SOSP (2007)Google Scholar
  23. 23.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W., Wallach, D., Burrows, M., Chandra, T.: Bigtable: A Distributed Storage System for Structured Data. In: Proc. of OSDI 2006 (2006)Google Scholar
  24. 24.
    Tang, H., Gulbeden, A., Chu, L.: A Self-Organizing Storage Cluster for Parallel Data-Intensive Applications. In: Proc. of SC 2004, PA (November 2004)Google Scholar
  25. 25.
  26. 26.
    Pedretti, K.T., Casavant, T.L., Roberts, C.A.: Three complementary approaches to parallelization of local BLAST service on workstation clusters. In: Malyshkin, V.E. (ed.) PaCT 1999. LNCS, vol. 1662, Springer, Heidelberg (1999)CrossRefGoogle Scholar
  27. 27.
    Xing, J., Xiong, J., Ma, J., Sun, N.: Main Memory Metadata Server for Large Distributed File Systems. In: GCC 2008 (October 2008)Google Scholar
  28. 28.
    Gulati, A., Kumar, C., Ahmad, I., Kumar, K.: BASIL: automated IO load balancing across storage devices. In: Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST 2010), pp. 13–13. USENIX Association, Berkeley, CA, USA (2010)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Yu Wang
    • 1
    • 2
  • Jing Xing
    • 1
  • Jin Xiong
    • 1
  • Dan Meng
    • 1
  1. 1.National Research Center for Intelligent Computing Systems, Institute of Computing TechnologyChinese Academy of SciencesChina
  2. 2.Graduate University of Chinese Academy of SciencesChina

Personalised recommendations