Analysis and improvement of map-reduce data distribution in read mapping applications
- 388 Downloads
The map-reduce paradigm has shown to be a simple and feasible way of filtering and analyzing large data sets in cloud and cluster systems. Algorithms designed for the paradigm must implement regular data distribution patterns so that appropriate use of resources is ensured. Good scalability and performance on Map-Reduce applications greatly depend on the design of regular intermediate data generation-consumption patterns at the map and reduce phases. We describe the data distribution patterns found in current Map-Reduce read mapping bioinformatics applications and show some data decomposition principles to greatly improve their scalability and performance
KeywordsBioinformatics Read mapping Map reduce Scalability
We want to thank Eduard Ayguade, David Carrera and the staff at Barcelona Supercomputing Center (BSC) for their help and support to the usage of the IBM Blade computer cluster.
This paper was supported by Consolider Project CSD2007-00050 of the Spanish Ministerio de Ciencia y Tecnologia.
- 2.Bialecki A, Cafarella M, Cutting D, O’Malley O (2005) Hadoop: a framework for running applications on large clusters built of commodity hardware. Wiki at http://hadoop.apache.org/
- 3.Shi X (2009) Evaluating MapReduce on virtual machines: the Hadoop case. In: CloudCom 2009. LNCS, vol 5931. Springer, Berlin, pp 519–528 Google Scholar
- 7.Ranger C, Raghurama R, Penmetsa A, Bradski G, Kozykaris C (2007) Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of the 13th international symposium on high-performance computer architecture (HPCA), Phoenix, AZ Google Scholar
- 8.Mao Y, Morris R, Kaashoek MF (2010) Optimizing MapReduce for multicore architectures. Tech Rep, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology Google Scholar
- 13.Babu S (2010) Towards automatic optimization of MapReduce programs. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, New York Google Scholar
- 14.Palla K (2009) A comparative analysis of join algorithms using the Hadoop Map/Reduce framework. Master of science thesis. School of informatics, University of Edinburgh Google Scholar