Skip to main content
SpringerLink
Log in

Research and implementation of scalable parallel computing based on Map-Reduce

  • Information Technology
  • Published:
Journal of Shanghai University (English Edition)

Abstract

As a parallel programming model, Map-Reduce is used for distributed computing of massive data. Map-Reduce model encapsulates the details of parallel implementation, fault-tolerant processing, local computing and load balancing, etc., provides a simple but powerful interface. In case of having no clear idea about distributed and parallel programming, this interface can be utilized to save development time. This paper introduces the method of using Hadoop, the open-source Map-Reduce software platform, to combine PCs to carry out scalable parallel computing. Our experiment using 12 PCs to compute N-body problem based on Map-Reduce model shows that we can get a 9.8x speedup ratio. This work indicates that the Map-Reduce can be applied in scalable parallel computing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wikipedia. SETI@home [EB/OL]. (2011-7-10) [2011-7-15]. http://en.wikipedia.org/wiki/SETI@home.

  2. Wikipedia. Berkeley open infrastructure for network computing [EB/OL]. (2011-6-14) [2011-6-20]. http://en.wikipedia.org/wiki/Berkeley Open Infrastructure for Network Computing.

  3. Boinc’s official website. How BOINC works [EB/OL]. (2011-7-1) [2011-7-15]. http://boinc.berkeley.edu/wiki/How BOINC works.

  4. Wikipedia. MapReduce [EB/OL]. (2011-7-14) [2011-7-15]. http://en.wikipedia.org/wiki/Map-reduce.

  5. Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters [J]. Communications of the ACM, 2008, 51(1): 107–113.

    Article  Google Scholar 

  6. Hadoop wiki. PoweredBy [EB/OL]. (2011-7-10) [2011-7-15]. http://wiki.apache.org/hadoop/PoweredBy.

  7. Blelloch G, Narlikar G. A practical comparison of N-body algorithms [M]// Parallel Algorithms (series in Discrete Mathematics and Theoretical Computer Science), Providence: American Mathematical Society. 1997: 1–16.

    Google Scholar 

  8. Nyland L, Harris M, Prins J. Fast N-body simulation with CUDA [M]// GPU Gems 3. Boston: Addison-Wesley Professional. 2007: 677–696.

  9. Google. Hadoop-eclipse-plugin [EB/OL]. (2011-5-8) [2011-7-15]. http://hadoop-eclipse-plugin.googlecode.com/files/hadoop-0.20.3-dev-eclipse-plugin.jar.

  10. Chen W Y. Programming Map-Reduce (Hadoop) with eclipse [EB/OL]. (2008-5-27) [2011-7-15]. http://www.trac.nchc.org.tw/cloud/export/256/hadoopeclipse.pdf.

  11. White T. Hadoop: The definitive guide [M]. US: O’Reilly Media. 2009: 1–38.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shanghai, 200072, P. R. China

    Thanh-cuong Nguyen  (阮青强), Wen-feng Shen  (沈文枫), Ya-hui Chai  (柴亚辉) & Wei-min Xu  (徐炜民)

  2. School of Information Engineering, East China Jiaotong University, Nanchang, 330013, P. R. China

    Ya-hui Chai  (柴亚辉)

Authors
  1. Thanh-cuong Nguyen  (阮青强)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wen-feng Shen  (沈文枫)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ya-hui Chai  (柴亚辉)
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei-min Xu  (徐炜民)
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen-feng Shen  (沈文枫).

Additional information

Project supported by the Shanghai Leading Academic Discipline Project (Grant No.J50103), the National High-Technology Research and Development Program of China (Grant No.2009AA012201), the Major Technology R&D Program of Shanghai (Grant No.08DZ501600), and the Science and Technology Pillar Project of Jiangxi (Grant No.2010BGB00604)

About this article

Cite this article

Nguyen, Tc., Shen, Wf., Chai, Yh. et al. Research and implementation of scalable parallel computing based on Map-Reduce. J. Shanghai Univ.(Engl. Ed.) 15, 426–429 (2011). https://doi.org/10.1007/s11741-011-0763-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11741-011-0763-3

Keywords

Search

Navigation