Design Strategies for Handling Data Skew in MapReduce Framework

  • Avinash PotluriEmail author
  • S. Nagesh Bhattu
  • N. V. Narendra Kumar
  • R. B. V. Subramanyam
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 98)


Multiway spatial join has drawn significant interest in research community because of its wide range of applications. Multiway spatial join further enjoys lots of applications in location based services. The analysis of communication cost is vital in the performance analysis of computing distributed multiway spatial join due to the skew observed in real world data. We analyze the performance of multiway spatial join using two strategies for addressing skew (a) whether to have a constraint on the number of reducers or (b) to have a constraint on the size of the input to the reducer (reducer is a computing facility). Our study gives a solution to address the issue of skew and to minimize the cost for communication in a network. We propose two algorithms, which study the trade-offs between the two strategies. We conducted experiments on real world datasets shows the performance in various scenarios. Based on the learning we provide insights into the selection of appropriate strategies for a given task.


Distributed computing Skew Communication cost 


  1. 1.
    Afrati, F.N., Stasinopoulos, N., Ullman, J.D., Vassilakopoulos, A.: SharesSkew: an algorithm to handle skew for joins in mapreduce. Inform. Syst. 77, 129–150 (2018)CrossRefGoogle Scholar
  2. 2.
    Afrati, F.N., Ullman, J.D.: Optimizing joins in a map-reduce environment. In: Proceedings of the 13th International Conference on Extending Database Technology, pp. 99–110. ACM (2010)Google Scholar
  3. 3.
    Beame, P., Koutris, P., Suciu, D.: Communication steps for parallel query processing. In: Proceedings of the 32nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pp. 273–284. ACM (2013)Google Scholar
  4. 4.
    Beame, P., Koutris, P., Suciu, D.: Skew in parallel query processing. In: Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 212–223. ACM (2014)Google Scholar
  5. 5.
    Cheng, L., Kotoulas, S., Liu, Q., Wang, Y.: Load-balancing distributed outer joins through operator decomposition. J. Parallel Distrib. Comput. (2019)Google Scholar
  6. 6.
    Cho, E., Myers, S.A., Leskovec, J.: Friendship and mobility: user movement in location-based social networks. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1082–1090. ACM (2011)Google Scholar
  7. 7.
    Chu, S., Balazinska, M., Suciu, D.: From theory to practice: efficient join query evaluation in a parallel database system. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 63–78. ACM (2015)Google Scholar
  8. 8.
    Gavagsaz, E., Rezaee, A., Javadi, H.H.S.: Load balancing in join algorithms for skewed data in mapreduce systems. J. Supercomput. 75(1), 228–254 (2019) CrossRefGoogle Scholar
  9. 9.
    Irandoost, M.A., Rahmani, A.M., Setayeshi, S.: MapReduce data skewness handling: a systematic literature review. Int. J. Parallel Program. 1–44 (2019)Google Scholar
  10. 10.
    Joglekar, M., Re, C.: It’s all a matter of degree: using degree information to optimize multiway joins. arXiv preprint arXiv:1508.01239 (2015)
  11. 11.
    Koutris, P., Beame, P., Suciu, D.: Worst-case optimal algorithms for parallel query processing. In: LIPIcs-Leibniz International Proceedings in Informatics, vol. 48. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)Google Scholar
  12. 12.
    Kwon, Y., Balazinska, M., Howe, B., Rolia, J.: Skewtune in action: mitigating skew in mapreduce applications. Proc. VLDB Endow. 5(12), 1934–1937 (2012) CrossRefGoogle Scholar
  13. 13.
    Ngo, H.Q., Ré, C., Rudra, A.: Skew strikes back: new developments in the theory of join algorithms. arXiv preprint arXiv:1310.3314 (2013)
  14. 14.
    Shi, Y., Qian, K.: LBMM: a load balancing based task scheduling algorithm for cloud. In: Future of Information and Communication Conference, pp. 706–712. Springer (2019)Google Scholar
  15. 15.
    Wang, Z., Chen, Q., Suo, B., Pan, W., Li, Z.: Reducing partition skew on mapreduce: an incremental allocation approach. Front. Comput. Sci. 13(5), 960–975 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Avinash Potluri
    • 1
    • 2
    Email author
  • S. Nagesh Bhattu
    • 3
  • N. V. Narendra Kumar
    • 2
  • R. B. V. Subramanyam
    • 1
  1. 1.National Institute of Technology WarangalWarangalIndia
  2. 2.Institute for Development and Research in Banking TechnologyHyderabadIndia
  3. 3.National Institute of Technology AndhraPradeshTadepalligudemIndia

Personalised recommendations