Scalable Load Balancing in Cluster Storage Systems

  • Gae-won You
  • Seung-won Hwang
  • Navendu Jain
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7049)


Enterprise and cloud data centers are comprised of tens of thousands of servers providing petabytes of storage to a large number of users and applications. At such a scale, these storage systems face two key challenges: (a) hot-spots due to the dynamic popularity of stored objects and (b) high reconfiguration costs of data migration due to bandwidth oversubscription in the data center network. Existing storage solutions, however, are unsuitable to address these challenges because of the large number of servers and data objects. This paper describes the design, implementation, and evaluation of Ursa, which scales to a large number of storage nodes and objects and aims to minimize latency and bandwidth costs during system reconfiguration. Toward this goal, Ursa formulates an optimization problem that selects a subset of objects from hot-spot servers and performs topology-aware migration to minimize reconfiguration costs. As exact optimization is computationally expensive, we devise scalable approximation techniques for node selection and efficient divide-and-conquer computation. Our evaluation shows Ursa achieves cost-effective load balancing while scaling to large systems and is time-responsive in computing placement decisions, e.g., about two minutes for 10K nodes and 10M objects.


Load balancing storage optimization linear programming 


  1. 1.
  2. 2.
  3. 3.
    Abd-El-Malek, M., Courtright II, W.V., Cranor, C., Ganger, G.R., Hendricks, J., Klosterman, A.J., Mesnier, M., Prasad, M., Salmon, B., Sambasivan, R.R., Sinnamohideen, S., Strunk, J.D., Thereska, E., Wachs, M., Wylie, J.J.: Ursa Minor: Versatile Cluster-based Storage. In: Proc. of FAST (2005)Google Scholar
  4. 4.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: Proc. of OSDI (2006)Google Scholar
  5. 5.
    Curino, C., Jones, E., Zhang, Y., Madden, S.: Schism: a Workload-Driven Approach to Database Replication and Partitioning. In: Proc. of VLDB (2010)Google Scholar
  6. 6.
    Curino, C., Jones, E., Zhang, Y., Wu, E., Madden, S.: Relational Cloud: The Case for a Database Service. Technical Report MIT-CSAIL-TR-2010-014, MIT (2010)Google Scholar
  7. 7.
    Das, S., Nishimura, S., Agrawal, D., Abbadi, A.E.: Live Database Migration for Elasticity in a Multitenant Database for Cloud Platforms. Technical report, UCSB (2010)Google Scholar
  8. 8.
    Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: Proc. of OSDI (2004)Google Scholar
  9. 9.
    Elmore, A., Das, S., Agrawal, D., Abbadi, A.E.: Who’s Driving this Cloud? Towards Efficient Migration for Elastic and Autonomic Multitenant Databases. Technical report, UCSB (2010)Google Scholar
  10. 10.
    Elmore, A., Das, S., Agrawal, D., Abbadi, A.E.: Zephyr: Live Migration in Shared Nothing Databases for Elastic Cloud Platforms. In: Proc. of SIGMOD (2011)Google Scholar
  11. 11.
    Eric, E.A., Spence, S., Swaminathan, R., Kallahalla, M., Wang, Q.: Quickly Finding Near-optimal Storage Designs. ACM Transactions on Computer Systems 23, 337–374 (2005)CrossRefGoogle Scholar
  12. 12.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google File System. SIGOPS Operating System Review 37, 29–43 (2003)CrossRefGoogle Scholar
  13. 13.
    Greenberg, A.G., Hamilton, J.R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D.A., Patel, P., Sengupta, S.: VL2: A Scalable and Flexible Data Center Network. In: Proc. of SIGCOMM (2009)Google Scholar
  14. 14.
    Gulati, A., Kumar, C., Ahmad, I., Kumar, K.: BASIL: Automated IO Load Balancing Across Storage Devices. In: Proc. of FAST (2010)Google Scholar
  15. 15.
    Hiller, F.S., Lieberman, G.J.: Introduction to Operations Research, 8th edn. McGraw-Hill (2005)Google Scholar
  16. 16.
    Kunkle, D., Schindler, J.: A Load Balancing Framework for Clustered Storage Systems. In: Sadayappan, P., Parashar, M., Badrinath, R., Prasanna, V.K. (eds.) HiPC 2008. LNCS, vol. 5374, pp. 57–72. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Lang, W., Patel, J.M., Naughton, J.F.: On Energy Management, Load Balancing and Replication. SIGMOD Record 38, 35–42 (2010)CrossRefGoogle Scholar
  18. 18.
    Litwin, W.: Linear Hashing: A New Tool for File and Table Addressing. In: Proc. of VLDB (1980)Google Scholar
  19. 19.
    Narayanan, D., Donnelly, A., Thereska, E., Elnikety, S., Rowstron, A.: Everest: Scaling Down Peak Loads through I/O Off-loading. In: Proc. of OSDI (2008)Google Scholar
  20. 20.
    Savinov, S., Daudjee, K.: Dynamic Database Replica Provisioning through Virtualization. In: Proc. of CloudDB (2010)Google Scholar
  21. 21.
    Tam, H.V., Chen, C., Ooi, B.C.: Towards Elastic Transactional Cloud Storage with Range Query Support. In: Proc. of VLDB (2010)Google Scholar
  22. 22.
    Thereska, E., Donnelly, A., Narayanan, C.: Sierra: A Power-proportional, Distributed Storage System. In: Technical Report MSR-TR-2009-153 (2009)Google Scholar
  23. 23.
    Venkataramani, A., Kokku, R., Dahlin, M.: TCP Nice: A Mechanism for Background Transfers. In: Proc. of OSDI (2002)Google Scholar
  24. 24.
    Verma, A., Ahuja, P., Neogi, A.: pMapper: Power and Migration Cost Aware Application Placement in Virtualized Systems. In: Issarny, V., Schantz, R. (eds.) Middleware 2008. LNCS, vol. 5346, pp. 243–264. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  25. 25.
    Weil, S.A., Brand, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: A Scalable, High-performance Distributed File System. In: Proc. of OSDI (2006)Google Scholar
  26. 26.
    Yin, Q., Schüpbach, A., Cappos, J., Baumann, A., Roscoe, T.: Rhizoma: A Runtime for Self-deploying, Self-managing Overlays. In: Bacon, J.M., Cooper, B.F. (eds.) Middleware 2009. LNCS, vol. 5896, pp. 184–204. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  27. 27.
    Zeng, L., Feng, D., Wang, F., Zhou, K.: A Strategy of Load Balancing in Objects Storage System. In: Proc. of CIT (2005)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Gae-won You
    • 1
  • Seung-won Hwang
    • 1
  • Navendu Jain
    • 2
  1. 1.Pohang University of Science and TechnologyRepublic of Korea
  2. 2.Microsoft ResearchRedmondUSA

Personalised recommendations