Peer-to-Peer Networking and Applications

, Volume 6, Issue 4, pp 409–424 | Cite as

Handling partitioning skew in MapReduce using LEEN

  • Shadi IbrahimEmail author
  • Hai Jin
  • Lu Lu
  • Bingsheng He
  • Gabriel Antoniu
  • Song Wu


MapReduce is emerging as a prominent tool for big data processing. Data locality is a key feature in MapReduce that is extensively leveraged in data-intensive cloud systems: it avoids network saturation when processing large amounts of data by co-allocating computation and data storage, particularly for the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew (Partitioning skew refers to the case when a variation in either the intermediate keys’ frequencies or their distributions or both among different data nodes) causes a huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications severe performance degradation due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. In this paper, we develop a novel algorithm named LEEN for locality-aware and fairness-aware key partitioning in MapReduce. LEEN embraces an asynchronous map and reduce scheme. All buffered intermediate keys are partitioned according to their frequencies and the fairness of the expected data distribution after the shuffle phase. We have integrated LEEN into Hadoop. Our experiments demonstrate that LEEN can efficiently achieve higher locality and reduce the amount of shuffled data. More importantly, LEEN guarantees fair distribution of the reduce inputs. As a result, LEEN achieves a performance improvement of up to 45 % on different workloads.


MapReduce Hadoop Cloud computing Skew partitioning Intermediate data 



This work is supported by NSF of China under grant No.61232008, 61133008 and 61073024, the National 863 Hi-Tech Research and Development Program under grant 2013AA01A213, the Outstanding Youth Foundation of Hubei Province under Grant No.2011CDA086, the National Science & Technology Pillar Program of China under grant No.2012BAH14F02, the Inter-disciplinary Strategic Competitive Fund of Nanyang Technological University 2011 No.M58020029, and the ANR MapReduce grant (ANR-10-SEGI-001). This work was done in the context of the Héméra INRIA Large Wingspan-Project (see


  1. 1.
    Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51:107–113CrossRefGoogle Scholar
  2. 2.
    Jin H, Ibrahim S, Bell T, Qi L, Cao H, Wu S, Shi X (2010) Tools and technologies for building the clouds. Cloud computing: principles systems and applications, Computer Communications and Networks. Springer-Verlag, Berlin, pp 3–20Google Scholar
  3. 3.
    Jin H, Ibrahim S, Qi L, Cao H, Wu S, Shi X (2011) The mapreduce programming model and implementations. In: Buyya R, Broberg J, Goscinski A (eds) Cloud computing: principles and paradigms. Wiley, Hoboken, pp 373–390CrossRefGoogle Scholar
  4. 4.
    Isard M, Budiu M, Yu Y, Birrell A, Fetterly D (2007) Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2nd ACM European conference on computer systems (EuroSys ’07), Lisbon, pp 59–72Google Scholar
  5. 5.
    He B, Fang W, Luo Q, Govindaraju NK, Wang T (2008) Mars: a mapreduce framework on graphics processors. In: Proceedings of the 17th international conference on parallel architectures and compilation techniques, Toronto, pp 260–269Google Scholar
  6. 6.
    Ranger C, Raghuraman R, Penmetsa A, Bradski G, Kozyrakis C (2007) Evaluating mapreduce for multi-core and multiprocessor systems. In: Proceedings of the 2007 IEEE 13th international symposium on high performance computer architecture (HPCA-13), Phoenix, pp 13–24Google Scholar
  7. 7.
    Hadoop project (2011)
  8. 8.
    Yahoo! (2011) Yahoo! developer network, Scholar
  9. 9.
    Hadoop (2011) Applications powered by hadoop:
  10. 10.
    Ghemawat S, Gobioff H, Leung S-T (2003) The Google file system. SIGOPS - Oper Syst Rev 37(5):29–43CrossRefGoogle Scholar
  11. 11.
    Condie T, Conway N, Alvaro P, Hellerstein JM, Elmeleegy K, Sears R (2010) Mapreduce online. In: Proceedings of the 7th USENIX conference on networked systems design and implementation (NSDI’10), San JoseGoogle Scholar
  12. 12.
    Kwon Y, Balazinska M, Howe B, Rolia J (2011) A study of skew in mapreduce applications,
  13. 13.
    Qiu X, Ekanayake J, Beason S, Gunarathne T, Fox G, Barga R, Gannon D (2009) Cloud technologies for bioinformatics applications. In: Proceedings of the 2nd workshop on many-task computing on grids and supercomputers (MTAGS ’09)Google Scholar
  14. 14.
    Lin J (2009) The curse of zipf and limits to parallelization: a look at the stragglers problem in mapreduce. In: Proceedings of the 7th workshop on large-scale distributed systems for information retrieval (LSDS-IR’09)Google Scholar
  15. 15.
    DeWitt DJ, Stonebraker M (2008) Mapreduce: a major step backwards,
  16. 16.
    Wiley K, Connolly A, Gardner JP, Krughof S, Balazinska M, Howe B, Kwon Y, Bu Y (2011) Astronomy in the cloud: using MapReduce for image coaddition, CoRR abs/1010.1015Google Scholar
  17. 17.
    Chen R, Yang M, Weng X, Choi B, He B, Li X (2012) Improving large graph processing on partitioned graphs in the cloud. In: Proceedings of the third ACM symposium on cloud computing, SoCC ’12, ACM, New York, pp 3:1–3:13. doi: 10.1145/2391229.2391232.
  18. 18.
    Schatz MC (2009) Cloudburst: highly sensitive read mapping with mapreduce. Bioinformatics 25:1363–1369CrossRefGoogle Scholar
  19. 19.
    Verma A, Llorà X, Goldberg DE, Campbell RH (2009) Scaling genetic algorithms using MapReduce. In: Proceedings of the 2009 9th international conference on intelligent systems design and applications, pp 13–18Google Scholar
  20. 20.
    Ng AY, Bradski G, Chu C-T, Olukotun K, Kim SK, Lin Y-A, Yu Y (2006) MapReduce for machine learning on multicore. In: Proceedings of the twentieth annual conference on neural information processing systems (NIPS ’06), Vancouver, pp 281–288Google Scholar
  21. 21.
    Lin J, Schatz M (2010) Design patterns for efficient graph algorithms in mapreduce. In: Proceedings of the eighth workshop on mining and learning with graphs, Washington, pp 78–85Google Scholar
  22. 22.
    Xen hypervisor homepage (2011)
  23. 23.
    Amazon elastic compute cloud (2011)
  24. 24.
    Ibrahim S, Jin H, Lu L, Wu S, He B, Qi L (2010) Leen: locality/fairness-aware key partitioning for mapreduce in the cloud. In: Proceedings of the 2010 IEEE second international conference on cloud computing technology and science (CLOUDCOM’10), Indianapolis, pp 17–24Google Scholar
  25. 25.
    Jain R, Chiu D-M, Hawe W (1984) A quantitative measure of fairness and discrimination for resource allocation in shared computer systems, DEC Research Report TR-301Google Scholar
  26. 26.
    Ibrahim S, Jin H, Lu L, Qi L, Wu S, Shi X (2009) Evaluating mapreduce on virtual machines: the hadoop case. In: Proceedings of the 1st international conference on cloud computing (CLOUDCOM’09), Beijing, pp 519–528Google Scholar
  27. 27.
    Ibrahim S, Jin H, Cheng B, Cao H, Wu S, Qi L (2009) Cloudlet: towards mapreduce implementation on virtual machines. In: Proceedings of the 18th ACM international symposium on high performance distributed computing (HPDC-18), Garching, pp 65–66Google Scholar
  28. 28.
    Zaharia M, Borthakur D, Sen Sarma J, Elmeleegy K, Shenker S, Stoica I (2010) Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th ACM European conference on computer systems (EuroSys’10), Paris, pp 265–278Google Scholar
  29. 29.
    Ibrahim S, Jin H, Lu L, He B, Antoniu G, Wu S (2012) Maestro: replica-aware map scheduling for mapreduce. In: Proceedings of the 12th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGrid 2012), OttawaGoogle Scholar
  30. 30.
    Ibrahim S, Jin H, Lu L, He B, Wu S (2011) Adaptive disk i/o scheduling for mapreduce in virtualized environment. In: Proceedings of the 2011 international conference on parallel processing (ICPP’11), Taipei, pp 335–344Google Scholar
  31. 31.
    Menon RK, Bhat GP, Schatz MC (2011) Rapid parallel genome indexing with MapReduce. In: Proceedings of the 2nd international workshop on MapReduce and its applications, San Jose, pp 51–58Google Scholar
  32. 32.
    Ekanayake J, Pallickara S, Fox G (2008) Mapreduce for data intensive scientific analyses. In: Proceedings of the 2008 fourth IEEE international conference on eScience, pp 277–284Google Scholar
  33. 33.
    Gunarathne T, Wu T-L, Qiu J, Fox G (2010) MapReduce in the clouds for science. In: Proceedings of the 2010 IEEE second international conference on cloud computing technology and science, pp 565–572Google Scholar
  34. 34.
    Ganjisaffar Y, Debeauvais T, Javanmardi S, Caruana R, Lopes CV (2011) Distributed tuning of machine learning algorithms using MapReduce clusters. In: Proceedings of the 3rd workshop on large scale data mining: theory and applications, San Diego, pp 2:1–2:8Google Scholar
  35. 35.
    Blanas S, Patel JM, Ercegovac V, Rao J, Shekita EJ, Tian Y (2010) A comparison of join algorithms for log processing in mapreduce. In: Proceedings of the 2010 international conference on Management of data, Indianapolis, pp 975–986Google Scholar
  36. 36.
    Logothetis D, Trezzo C, Webb KC, Yocum K (2011) In-situ mapreduce for log processing. In: Proceedings of the 2011 USENIX conference on USENIX annual technical conference, Portland, pp 9–9Google Scholar
  37. 37.
    Seo S, Jang I, Woo K, Kim I, Kim J-S, Maeng S (2009) Hpmr: prefetching and pre-shuffling in shared mapreduce computation environment. In: Proceedings of the 2009 IEEE international conference on cluster computing (CLUSTER’09), New OrleansGoogle Scholar
  38. 38.
    Su Y-L, Chen P-C, Chang J-B, Shieh C-K (2011) Variable-sized map and locality-aware reduce on public-resource grids. Futur Gener Comput Syst 27(6):843–849CrossRefGoogle Scholar
  39. 39.
    DeWitt D, Gray J (1992) Parallel database systems: the future of high performance database systems. Commun ACM 35:85–98CrossRefGoogle Scholar
  40. 40.
    Chen S, Schlosser SW (2008) Map-reduce meets wider varieties of applications, Tech. Rep. IRP-TR-08-05, Technical Report. Intel Research PittsburghGoogle Scholar
  41. 41.
    Ananthanarayanan G, Kandula S, Greenberg A, Stoica I, Lu Y, Saha B, Harris E (2010) Reining in the outliers in map-reduce clusters using mantri. In: Proceedings of the 9th USENIX conference on operating systems design and implementation (OSDI’10), Vancouver, pp 1–16Google Scholar
  42. 42.
    Kwon Y, Balazinska M, Howe B, Rolia J (2010) Skew-resistant parallel processing of feature-extracting scientific user-defined functions. In: Proceedings of the 1st ACM symposium on Cloud computing (SoCC ’10)Google Scholar
  43. 43.
    Gufler B, Augsten N, Reiser A, Kemper A (2012) Load balancing in mapreduce based on scalable cardinality estimates. In: Proceedings of the 2012 IEEE 28th international conference on data engineering (ICDE ’12)Google Scholar
  44. 44.
    Kwon Y, Balazinska M, Howe B, Rolia J (2012) Skewtune: mitigating skew in mapreduce applications. In: Proceedings of the 2012 ACM SIGMOD international conference on management of data (SIGMOD ’12)Google Scholar
  45. 45.
    He B, Yang M, Guo Z, Chen R, Lin W, Su B, Wang H, Zhou L (2009) Wave computing in the cloud. In: Proceedings of the 12th conference on hot topics in operating systems (HotOS’09)Google Scholar
  46. 46.
    He B, Yang M, Guo Z, Chen R, Su B, Lin W, Zhou L (2010) Comet: batched stream processing for data intensive distributed computing. In: Proceedings of the 1st ACM symposium on Cloud computing (SoCC ’10)Google Scholar
  47. 47.
    Ibrahim S, He B, Jin H (2011) Towards pay-as-you-consume cloud computing. In: Proceedings of the 2011 IEEE international conference on services computing (SCC’11), Washington, DC, pp 370–377Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Shadi Ibrahim
    • 1
    Email author
  • Hai Jin
    • 2
  • Lu Lu
    • 2
  • Bingsheng He
    • 3
  • Gabriel Antoniu
    • 1
  • Song Wu
    • 2
  1. 1.INRIA Rennes-Bretagne AtlantiqueRennesFrance
  2. 2.Cluster and Grid Computing Lab, Services Computing Technology and System LabHuazhong University of Science and TechnologyWuhanChina
  3. 3.School of Computer EngineeringNanyang Technological UniversitySingaporeSingapore

Personalised recommendations