Load Balancing for Parallel Query Execution on NUMA Multiprocessors

  • Luc Bouganim
  • Daniela Florescu
  • Patrick Valduriez


To scale up to high-end configurations, shared-memory multiprocessors are evolving towards Non Uniform Memory Access (NUMA) architectures. In this paper, we address the central problem of load balancing during parallel query execution in NUMA multiprocessors. We first show that an execution model for NUMA should not use data partitioning (as shared-nothing systems do) but should strive to exploit efficient shared-memory strategies like Synchronous Pipelining (SP). However, SP has problems in NUMA, especially with skewed data. Thus, we propose a new execution strategy which solves these problems. The basic idea is to allow partial materialization of intermediate results and to make them progressivly public, i.e., able to be processed by any processor, as needed to avoid processor idle times. Hence, we call this strategy Progressive Sharing (PS). We conducted a performance comparison using an implementation of SP and PS on a 72-processor KSR1 computer, with many queries and large relations. With no skew, SP and PS have both linear speed-up. However, the impact of skew is very severe on SP performance while it is insignificant on PS. Finally, we show that, in NUMA, PS can also be beneficial in executing several pipeline chains concurrently.


parallel databases query execution load balancing NUMA synchronous pipeline execution engines 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. Agarwal, R. Bianchini, D. Chaiken, K.L. Johnson, D. Kranz, J. Kubiatowicz, B.-H. Lim, K. Mackenzie, and D. Yeung, “The MIT alewife machine: Architecture and performance,” Int. Symp. on Computer Architecture, June 1995.Google Scholar
  2. 2.
    P.M.G. Apers, C.A. van den Berg, J. Flokstra, P.W.P.J. Grefen, M.L. Kersten, and A.N. Wilschut, “PRISMA/DB: A parallel main memory relational DBMS,” IEEE Trans. Knowledge and Data Engineering, vol. 4, no. 6, December 1992.Google Scholar
  3. 3.
    A. Bhide, “An analysis of three transaction processing architectures,” Int. Conf on VLDB, Los Angeles, August 1988.Google Scholar
  4. 4.
    M. Blasgen, J. Gray, M. Mitoma, and T. Price, “The convoy phenomenon,” Operating Systems Review, vol. 13, no. 2, April 1979.Google Scholar
  5. 5.
    H. Boral, W. Alexander, L. Clay, G. Copeland, S. Danforth, M. Franklin, B. Hart, M. Smith, and P. Valduriez, “Prototyping Bubba: A highly parallel database system,” IEEE Trans. Knowledge and Data Engineering, vol. 2, no. 1, March 1990.Google Scholar
  6. 6.
    L. Bouganim, B. Dageville, and P. Valduriez, “Adaptative parallel query execution in DBS3,” Industrial Paper, Int. Conf. on EDBT Avignon, March 1996.Google Scholar
  7. 7.
    L. Bouganim, D. Florescu, and P. Valduriez, “Dynamic load balancing in hierarchical parallel database systems,” Int. Conf. on VLDB, Bombay, September 1996. Can be retrieved at Google Scholar
  8. 8.
    Data General Corporation, “Data general and oracle to optimize oracle universal server for ccNUMA system,” can be retrieved at
  9. 9.
    Data General Corporation, “The NUMA invasion,” can be retrieved at
  10. 10.
    Data General Corporation, “Standard high volume servers: The new building block,” can be retrieved at
  11. 11.
    D.J. DeWitt, S. Ghandeharizadeh, D. Schneider, A. Bricker, H. Hsiao, and R. Rasmussen, “The gamma database machine project,” IEEE Trans. on Knowledge and Data Engineering, vol. 2, no. 1, March 1990.Google Scholar
  12. 12.
    D.J. DeWitt and J. Gray, “Parallel database systems: The future of high performance database processing,” Communications of the ACM, vol. 35, no. 6, June 1992.Google Scholar
  13. 13.
    D.J. DeWitt, J.F. Naughton, D.A. Schneider, and S. Seshadri, “Practical skew handling in parallel joins,” Int. Conf. on VLDB, Vancouver, August 1992.Google Scholar
  14. 14.
    S. Frank, H. Burkhardt, and J. Rothnie, “The KSR1: Bridging the gap between shared-memory and MPPs,” Compcon’93, San Francisco, February 1993.Google Scholar
  15. 15.
    M.N. Garofalakis and Y.E. Yoannidis, “Multi-dimensional resource scheduling for parallel queries,” ACMSIGMOD Int. Conf., Montreal, June 1996.Google Scholar
  16. 16.
    J.R. Goodman and P.J. Woest, “The Wisconsin multicube: A new large-scale cache-coherent multiprocessor,” University of Wisconsin-Madison, TR 766, April 1988.Google Scholar
  17. 17.
    G. Graefe, “Volcano: An extensible and parallel dataflow query evaluation system,” IEEE Trans. on Knowledge and Data Engineering, vol. 6, no. 1, February 1994.Google Scholar
  18. 18.
    E. Hagersten, E. Landin, and S. Haridi, “Ddm-A cache-only memory architecture,” IEEE Computer, vol. 25, no. 9, September 1992.Google Scholar
  19. 19.
    W. Hasan and R. Motwani, “Optimization algorithms for exploiting the parallel communication tradeoff in pipelined parallelism,” Int. Conf on VLDB, Santiago, 1994.Google Scholar
  20. 20.
    Y. Hirano, T. Satoh, A.U. Inoue, and K. Teranaka, “Load balancing algorithms for parallel database processing on shared memory multiprocessors,” Int. Conf. on Parallel and Distributed Information Systems, Miami Beach, December 1991.Google Scholar
  21. 21.
    W. Hong, “Exploiting inter-operation parallelism in XPRS,” ACM-SIGMOD Int. Conf., San Diego, June 1992.Google Scholar
  22. 22.
    H. Hsiao, M.S. Chen, and P.S. Yu, “On parallel execution of multiple pipelined hash joins,” ACM-SIGMOD Int. Conf., Minneapolis, May 1994.Google Scholar
  23. 23.
    IEEE Computer Society, “IEEE standard for scalable coherent interface (SCI),” IEEE Std 1596, New York, August 1992.Google Scholar
  24. 24.
    Intel Corporation, “Standard high volume servers: Changing the rules for buiseness computing,” can be retrieved at
  25. 25.
    M. Kitsuregawa and Y. Ogawa, “Bucket spreading parallel hash: A new, robust, parallel hash join method for data skew in the super database computer,” Int. Conf on VLDB, Brisbane, 1990.Google Scholar
  26. 26.
    J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy, “The Stanford FLASH multiprocessor,” Int. Symp. on Computer Architecture, April 1994.Google Scholar
  27. 27.
    R. Lanzelotte, P. Valduriez, and M. Zait, “On the effectiveness of optimization search strategies for parallel execution spaces,” Tnt. Conf. on VLDB, Dublin, August 1993.Google Scholar
  28. 28.
    D. Lenoski, J. Laudon, K. Gharachorloo, W.D. Weber, A. Gupta, J. Henessy, M. Horowitz, and M.S. Lam, “The Stanford dash multiprocessor,” IEEE Computer, vol. 25, no. 3, March 1992.Google Scholar
  29. 29.
    D. Lenoski, J. Laudon, T. Joe, D. Nakahira, L. Stevens, A. Gupta, and J. Hennessy, “The DASH prototype: Logic overhead and performance,” IEEE Transactions of Parallel and Distributed Systems, vol. 4, no. I, January 1993.Google Scholar
  30. 30.
    M.L. Lo, M-S. Chen, C. V. Ravishankar, and P.S. Yu, “On optimal processor allocation to support pipelined hash joins,” ACM-SIGMOD Int. Conf., Washington, May 1993.Google Scholar
  31. 31.
    T. Lovett and R. Clapp, “STiNG: A CC-NUMA computer system for the commercial marketplace,” Int. Symp. on Computer Architecture, May 1996.Google Scholar
  32. 32.
    H. Lu, M.-C. Shan, and K.-L. Tan, “Optimization of multi-way join queries for parallel execution,” Int. Conf. on VLDB, Barcelona, September 1991.Google Scholar
  33. 33.
    M. Metha and D. DeWitt, “Managing intra-operator parallelism in parallel database systems,” Int. Conf. on VLDB, Zurich, September 1995.Google Scholar
  34. 34.
    C. Morin, A. Gefflaut, M. Banâtre, and A.M. Kermarrec, “COMA: An opportunity for building fault-tolerant scalable shared memory multiprocessors,” Tnt. Symp. on Computer Architectures, 1996.Google Scholar
  35. 35.
    M.C. Murphy and M.-C. Shan, “Execution plan balancing,” IEEE Int. Conf. on Data Engineering, Kobe, April 1991.Google Scholar
  36. 36.
    E. Omiecinski, “Performance analysis of a load balancing hash-join algorithm for a shared-memory multiprocessor,” Int. Conf on VLDB, Barcelona, September 1991.Google Scholar
  37. 37.
    H. Pirahesh, C. Mohan, J. Cheng, T.S. Liu, and R. Selinger, “Parallelism in relational database systems: Architectural issues and design approaches,” Int. Symp. on Databases in Parallel and Distributed Systems, Dublin, July 1990.Google Scholar
  38. 38.
    E. Rahm and R. Marek, “Dynamic multi-resource load balancing in parallel database systems,” Int. Conf. on VLDB, Zurich, Switzerland, September 1993.Google Scholar
  39. 39.
    D. Schneider and D. DeWitt, “A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment,” ACM-SIGMOD Int. Conf., Portland, May-June 1989.Google Scholar
  40. 40.
    A. Shatdal and J.F. Naughton, “Using shared virtual memory for parallel join processing,” ACM-SIGMOD Int. Conf., Washington, May 1993.Google Scholar
  41. 41.
    E.J. Shekita and H.C. Young, “Multi-join optimization for symmetric multiprocessor,” Int. Conf. on VLDB, Dublin, August 1993.Google Scholar
  42. 42.
    A.J. Smith, “Cache memories,” ACM Computing Surveys, vol. 14, no. 3, September 1982.Google Scholar
  43. 43.
    J. Srivastava and G. Elsesser, “Optimizing multi-join queries in parallel relational databases,” Int. Conf. on Parallel and Distributed Information Systems, San Diego, January 1993.Google Scholar
  44. 44.
    R. Stenstrom, T. Joe, and A. Gupta, “Comparative performance evaluation of cache-coherent NUMA and COMA architectures,” Int. Symp. on Computer Architecture, May 1992.Google Scholar
  45. 45.
    P. Valduriez, “Parallel database systems: Open problems and new issues,” Int. Journal on Distributed and Parallel Databases, vol. I, no. 2, 1993.Google Scholar
  46. 46.
    R. Valduriez and G. Gardarin, “Join and semi-join algorithms for a multiprocessor database machine,” ACM Trans. on Database Systems, vol. 9, no. 1, March 1984.Google Scholar
  47. 47.
    C.A. van den Berg and M.L. Kersten, “Analysis of a dynamic query optimization technique for multi join queries,” Int. Conf. on Information and Knowledge Engineering, Washington, 1992.Google Scholar
  48. 48.
    C.B. Walton, A.G. Dale, and R.M. Jenevin, “A taxonomy and performance model of data skew effects in parallel joins,” Int. Conf. on VLDB, Barcelona, September 1991.Google Scholar
  49. 49.
    A.N. Wilshut, J. Flokstra, and P.G. Apers, “Parallel evaluation of multi-join queries,” ACM-SIGMOD Int. Conf., San Jose, 1995.Google Scholar

Copyright information

© Springer Science+Business Media New York 1999

Authors and Affiliations

  • Luc Bouganim
    • 1
  • Daniela Florescu
    • 1
  • Patrick Valduriez
    • 1
  1. 1.INRIA RocquencourtFrance

Personalised recommendations