Skip to main content
Log in

Tight bounds for parallel randomized load balancing

  • Published:
Distributed Computing Aims and scope Submit manuscript

Abstract

Given a distributed system of \(n\) balls and \(n\) bins, how evenly can we distribute the balls to the bins, minimizing communication? The fastest non-adaptive and symmetric algorithm achieving a constant maximum bin load requires \(\varTheta (\log \log n)\) rounds, and any such algorithm running for \(r\in {\mathcal {O}}(1)\) rounds incurs a bin load of \(\varOmega ((\log n/\log \log n)^{1/r})\). In this work, we explore the fundamental limits of the general problem. We present a simple adaptive symmetric algorithm that achieves a bin load of 2 in \(\log ^* n+{\mathcal {O}}(1)\) communication rounds using \({\mathcal {O}}(n)\) messages in total. Our main result, however, is a matching lower bound of \((1-o(1))\log ^* n\) on the time complexity of symmetric algorithms that guarantee small bin loads. The essential preconditions of the proof are (i) a limit of \({\mathcal {O}}(n)\) on the total number of messages sent by the algorithm and (ii) anonymity of bins, i.e., the port numberings of balls need not be globally consistent. In order to show that our technique yields indeed tight bounds, we provide for each assumption an algorithm violating it, in turn achieving a constant maximum bin load in constant time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Strictly speaking, the bound is shown for constant round numbers only, but a generalization seems feasible.

  2. We refer to [24] for a number of additional upper bounds; for the sake of a streamlined presentation, we focus on the main techniques and results here.

  3. There is no common agreement on the notion of w.h.p. Frequently it refers to probabilities of at least \(1-1/n\) or \(1-o(1)\), as so in the work of Azar et al.; however, their proof also provides their result w.h.p. in the sense we use throughout this paper.

  4. Roughly speaking, this means that any other algorithm is as least as likely to produce bad load vectors as the greedy algorithm. An \(n\)-dimensional load vector is worse than another, if after reordering the components of both vectors descendingly, any partial sum of the first \(i\in \{1,\ldots ,n\}\) entries of the one vector is greater or equal to the corresponding partial sum of the other.

  5. If balls cannot be allocated, they get an additional random choice. However, one could also give all balls this additional choice and let some of them ignore it, i.e., this kind of adaptivity cannot circumvent the lower bound.

  6. Since each ball sends \({\mathcal {O}}(t \log n)\) messages and the algorithm sends \({\mathcal {O}}(n)\) messages in expectation, induction shows that w.h.p. no bin is contacted by more than \({\mathcal {O}}(t\log n)\) balls.

  7. Note that there is no need to execute \({\mathcal {A}}_{i+1}\) in a distributed fashion, as we only require it to prove statements about \({\mathcal {A}}\). Hence, \({\mathcal {A}}_{i+1}\) can gather all information to make the appropriate decision; for the purpose of the proof, we simply ignore the related communication.

References

  1. Adler, M., Chakrabarti, S., Mitzenmacher, M., Rasmussen, L.: Parallel randomized load balancing. In: Proc. 27th Symposium on Theory of Computing (STOC), pp. 238–247 (1995)

  2. auf der Heide, F.M., Scheideler, C., Stemann, V.: Exploiting storage redundancy to speed up randomized shared memory simulations. Theor. Comput. Sci. 162(2), 245–281 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  3. Awerbuch, B., Azar, Y., Grove, E.F., Kao, M.Y., Krishnan, P., Vitter, J.S.: Load balancing in the \(L_p\) norm. In: Proc. 36th Symposium on Foundations of Computer Science (FOCS), pp. 383–391 (1995)

  4. Azar, Y., Broder, A.Z., Karlin, A.R., Upfal, E.: Balanced allocations. SIAM J. Comput. 29(1), 180–200 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bast, H., Hagerup, T.: Fast and reliable parallel hashing. In: Proc. 3rd Symposium on Parallel Algorithms and Architectures (SPAA), pp. 50–61 (1991)

  6. Berenbrink, P., Friedetzky, T., Goldberg, L.A., Goldberg, P.W., Hu, Z., Martin, R.: Distributed selfish load balancing. SIAM J. Comput. 37(4), 1163–1181 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  7. Berenbrink, P., Friedetzky, T., Hu, Z., Martin, R.: On weighted balls-into-bins games. Theor. Comput. Sci. 409(3), 511–520 (2008)

  8. Berenbrink, P., auf der Heide, F.M., Schröder, K.: Allocating weighted jobs in parallel. In: Proc. 9th Symposium on Parallel Algorithms and Architectures (SPAA), pp. 302–310 (1997)

  9. Czumaj, A., Stemann, V.: Randomized allocation processes. Random Struct. Algorithms 18(4), 297–331 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  10. Sarma, A.D., Holzer, S., Kor, L., Korman, A., Nanongkai, D., Pandurangan, G., Peleg, D., Wattenhofer, R.: Distributed verification and hardness of distributed approximation. In: Proc. 43rd Symposium on Theory of Computing (STOC) (2011)

  11. Dubhashi, D., Ranjan, D.: Balls and bins: a study in negative dependence. Random Struct. Algorithms 13, 99–124 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  12. Even, G., Medina, M.: Revisiting randomized parallel load balancing algorithms. In: Proc. 16th Colloquium on Structural Information and Communication Complexity (SIROCCO), pp. 209–221 (2009)

  13. Even, G., Medina, M.: Parallel randomized load balancing: a lower bound for a more general model. In: Proc. 36th Conference on Theory and Practice of Computer Science (SOFSEM), pp. 358–369 (2010)

  14. Fich, F., Ruppert, E.: Hundreds of impossibility results for distributed computing. Distrib. Comput. 16(2–3), 121–163 (2003)

    Article  Google Scholar 

  15. Gil, J., auf der Heide, F.M., Wigderson, A.: The tree model for hashing: lower and upper bounds. SIAM J. Comput. 25(5), 936–955 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gonnet, G.H.: Expected length of the longest probe sequence in hash code searching. J. ACM 28(2), 289–304 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  17. Hagerup, T.: The log-star revolution. In: Proc. 9th Symposium on Theoretical Aspects of Computer Science (STACS), pp. 259–278 (1992)

  18. Karp, R.M., Luby, M., auf der Heide, F.M.: Efficient PRAM simulation on a distributed memory machine. Algorithmica 16, 517–542 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  19. Kenthapadi, K., Panigrahy, R.: Balanced allocation on graphs. In: Proc. 7th Symposium on Discrete Algorithms (SODA), pp. 434–443 (2006)

  20. Kleinberg, R., Piliouras, G., Tardos, E.: Load balancing without regret in the bulletin board model. In: Proc. 28th Symposium on Principles of Distributed Computing (PODC), pp. 56–62 (2009)

  21. Koutsoupias, E., Mavronicolas, M., Spirakis, P.G.: Approximate equilibria and ball fusion. Theory Comput. Syst. 36(6), 683–693 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  22. Kuhn, F., Moscibroda, T., Wattenhofer, R.: What cannot Be computed locally! In: Proc. 23rd Symposium on the Principles of Distributed Computing (PODC) (2004)

  23. Lenzen, C.: Optimal deterministic routing and sorting on the congested clique. In: Proc. 32nd Symposium on Principles of Distributed Computing (PODC), pp. 42–50 (2013)

  24. Lenzen, C., Wattenhofer, R.: Tight bounds for parallel randomized load balancing. Comput. Res. Repos. arXiv:1102.5425 (2011)

  25. Lenzen, C., Wattenhofer, R.: Tight bounds for parallel randomized load balancing: extended abstract. In: Proc. 43rd Symposium on Theory of Computing (STOC), pp. 11–20 (2011)

  26. Linial, N.: Locality in distributed graph algorithms. SIAM J. Comput. 21(1), 193–201 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  27. Lotker, Z., Patt-Shamir, B., Peleg, D.: Distributed MST for constant diameter graphs. Distrib. Comput. 18(6), 453–460 (2006)

    Article  MATH  Google Scholar 

  28. Lynch, N.: A hundred impossibility proofs for distributed computing. In: Proc. 8th Symposium on Principles of distributed computing (PODC), pp. 1–28 (1989)

  29. Matias, Y., Vishkin, U.: Converting high probability into nearly-constant time—with applications to parallel hashing. In: Proc. 23rd Symposium on Theory of Computing (STOC), pp. 307–316 (1991)

  30. Mitzenmacher, M.: The Power of Two Choices in Randomized Load Balancing. Ph.D. thesis, University of California, Berkeley (1996)

  31. Mitzenmacher, M.: How Useful is Old Information? Tech. rep., Systems Research Center, Digital Equipment Corporation (1998)

  32. Mitzenmacher, M.: On the analysis of randomized load balancing schemes. In: Proc. 10th Symposium on Parallel Algorithms and Architectures (SPAA), pp. 292–301 (1998)

  33. Mitzenmacher, M., Prabhakar, B., Shah, D.: Load balancing with memory. In: Proc. 43th Symposium on Foundations of Computer Science (FOCS), pp. 799–808 (2002)

  34. Mitzenmacher, M., Richa, A., Sitaraman, R.: Handbook of Randomized Computing, vol. 1, chap. The Power of Two Random Choices: A Survey of the Techniques and Results, pp. 255–312. Kluwer Academic Publishers, Dordrecht (2001)

  35. Patt-Shamir, B., Teplitsky, M.: The round complexity of distributed sorting: extended abstract. In: Proc. 30th Symposium on Principles of Distributed Computing (PODC), pp. 249–256 (2011)

  36. Peleg, D., Rubinovich, V.: A near-tight lower bound on the time complexity of distributed mst construction. In: Proc. 40th Symposium on Foundations of Computer Science (FOCS), pp. 253–261 (1999)

  37. Peres, Y., Talwar, K., Wieder, U.: The \((1+\beta )\)-choice process and weighted balls-into-bins. In: Proc. 21th Symposium on Discrete Algorithms (SODA), pp. 1613–1619 (2010)

  38. Raman, R.: The power of collision: randomized parallel algorithms for chaining and integer sorting. In: Proc. 10th Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), pp. 161–175 (1990)

  39. Schneider, J., Wattenhofer, R.: A new technique for distributed symmetry breaking. In: 29th Symposium on Principles of Distributed Computing (PODC) (2010)

  40. Stemann, V.: Parallel balanced allocations. In: Proc. 8th Symposium on Parallel Algorithms and Architectures (SPAA), pp. 261–269 (1996)

  41. Talwar, K., Wieder, U.: Balanced allocations: the weighted case. In: Proc. 39th Symposium on Theory of Computing (STOC), pp. 256–265 (2007)

  42. Vöcking, B.: How asymmetry helps load balancing. J. ACM 50(4), 568–589 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  43. Wieder, U.: Balanced allocations with heterogenous bins. In: Proc. 19th Symposium on Parallel Algorithms and Architectures (SPAA), pp. 188–193 (2007)

Download references

Acknowledgments

We would like to thank Mohsen Ghaffari, Fabian Kuhn, Thomas Locher, and Reto Spöhel. This work was partly supported by the Swiss National Fund (SNF), the Society of Swiss Friends of the Weizmann Institute of Science, and the German Research Foundation (DFG, reference number Le 3107/1-1).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Christoph Lenzen.

Additional information

An extended abstract of preliminary work appeared at STOC 2011 [25] and the corresponding article has been published on arxiv [24].

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lenzen, C., Wattenhofer, R. Tight bounds for parallel randomized load balancing. Distrib. Comput. 29, 127–142 (2016). https://doi.org/10.1007/s00446-014-0225-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00446-014-0225-4

Keywords

Navigation