An Optimal Skew-insensitive Join and Multi-join Algorithm for Distributed Architectures
The development of scalable parallel database systems requires the design of efficient algorithms for the join operation which is the most frequent and expensive operation in relational database systems. The join is also the most vulnerable operation to data skew and to the high cost of communication in distributed architectures.
In this paper, we present a new parallel algorithm for join and multi-join operations on distributed architectures based on an efficient semi-join computation technique. This algorithm is proved to have optimal complexity and deterministic perfect load balancing. Its tradeoff between balancing overhead and speedup is analyzed using the BSP cost model which predicts a negligible join product skew and a linear speed-up. This algorithm improves our fa_join and sfa_join algorithms by reducing their communication and synchronization cost to a minimum while offering the same load balancing properties even for highly skewed data.
Unable to display preview. Download preview PDF.
- 3.Bamha, M., Hains, G.: A self-balancing join algorithm for SN machines. In: Proceedings of International Conference on Parallel and Distributed Computing and Systems (PDCS), Las Vegas, Nevada, USA, October 1998, pp. 285–290 (1998)Google Scholar
- 6.Bamha, M., Hains, G.: A frequency adaptive join algorithm for Shared Nothing machines. Journal of Parallel and Distributed Computing Practices (PDCP) 3(3), 333–345 (1999)Google Scholar
- 8.DeWitt, D.J., Naughton, J.F., Schneider, D.A., Seshadri, S.: Practical Skew Handling in Parallel Joins. In: Proceedings of the 18th VLDB Conference, Vancouver, British Columbia, Canada, pp. 27–40 (1992)Google Scholar
- 9.Hua, K.A., Lee, C.: Handling data skew in multiprocessor database computers using partition tuning. In. In: Proc. of the 17th International Conference on Very Large Data Bases, Barcelona, Catalonia, Spain, pp. 525–535. Morgan Kaufmann, San Francisco (1991)Google Scholar
- 10.Kitsuregawa, M., Ogawa, Y.: Bucket spreading parallel hash: A new, robust, parallel hash join method for skew in the super database computer (SDC). In: Very Large Data Bases: 16th International Conference on Very Large Data Bases, Brisbane, Australia, August 13–16, pp. 210–221 (1990)Google Scholar
- 11.Poosala, V., Ioannidis, Y.E.: Estimation of query-result distribution and its application in parallel-join load balancing. In: Proc. 22th Int. Conference on Very Large Database Systems, VLDB 1996, Bombay, India, September 1996, pp. 448–459 (1996)Google Scholar
- 12.Schneider, D., DeWitt, D.: A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment. In: Proceedings of the 1989 ACM SIGMOD International Conference on the Management of Data, Portland, Oregon, pp. 110–121. ACM Press, New York (1989)CrossRefGoogle Scholar
- 14.Skillicorn, D.B., Hill, J.M.D., McColl, W.F.: Questions and Answers about BSP. Scientific Programming 6(3), 249–274 (1997)Google Scholar