Skip to main content

The Implementation of MapReduce Scheduling Algorithm Based on Priority

  • Conference paper

Part of the Communications in Computer and Information Science book series (CCIS,volume 405)

Abstract

Nowadays cloud computing has become a popular platform for scientific applications. Cloud computing intends to share a large scale resources and equipments of computation, storage, information and knowledge for scientific researches. Job Scheduling problem is a core and challenging research issue in the current cloud computation area, and the aim is to the reasonable control of the job execution sequence as well as the allocation of computing resources, making the job total completion time of the shortest and resources are fully utilized. Data locality is one of the main factors to influence scheduling algorithm. The paper proposed an improved scheduling algorithm based on priority, after taking full account of data locality (IPDSA), which can distinguish the user’s job levels, so as to reduce the job execution time and avoid losing into locally optimal solution. The experimental results on the Hadoop platform show that the new scheduling algorithm can reduce the job average execution time, and raises the rate availability of resources.

Keywords

  • Cloud computing
  • Hadoop
  • MapReduce
  • Priority
  • Data locality

This work was supported by the National Natural Science Foundation of China (61103047), SKLSE Open Foundation (The Open Foundation of State Key Laboratory of Software Engineering, China, and SKLSE 2012-09-18).

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-53962-6_9
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-53962-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Lin, Q.: The cloud computing model based on Hadoop. Modern Computer, 114–116 (2010)

    Google Scholar 

  2. Chemawat, S., Gobioff, H., Leung, S.T.: The Google file system, http://labs.google.com/papers.gfs.html

  3. MapReduce, http://www.mongodb.org/display/DOCS/MapReduce

  4. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: Proceeding of the 6th Symposium on Operating Systems Design and Implementation (OSDI 2004), pp. 137–150. USENIX Association (2004)

    Google Scholar 

  5. Bansal, S., et al.: Dynamic Task Scheduling in Grid Computing Using Prioritized Round Robin Algorithm. IJCSI International Journal of Computer Science Issues 8(2), 472–477 (2011)

    Google Scholar 

  6. Zaharia, M., Borthakur, D., Sarma, J.S.: Job scheduling for multi-user mapreduce clusters. In: Proceedings of the 5th European Conference IEEE, pp. 145–161 (2009)

    Google Scholar 

  7. Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) JSSPP 2010. LNCS, vol. 6253, pp. 110–131. Springer, Heidelberg (2010)

    CrossRef  Google Scholar 

  8. Tatebe, O., et al.: Grid datafarm architecture for petascale data intensive computing. In: CCGRID 2002. IEEE Computer Society, Washington, DC (2002)

    Google Scholar 

  9. Zaharia, M., Borthakur, D., Sarma, J.S., Elmele-egy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: EuroSys 2010: Proceedings of the 5th European Conference on Computer Systems, pp. 265–278. ACM, New York (2010)

    Google Scholar 

  10. Isard, M., et al.: Quincy: fair scheduling for distributed computing clusters. In: SOSP 2009, pp. 261–276. ACM, New York (2009)

    Google Scholar 

  11. Xie, J., Yin, S., Ruan, X.J., Ding, Z.Y., Tian, Y.: Improving MapReduce performance through data placement in heterogeneous hadoop clusters. In: IEEE International Symposium on Parallel & Distributed Processing, Workshops and PhdForum, pp. 1–9 (2010)

    Google Scholar 

  12. Lin, X., Lu, Y., Deogun, J., Goddard, S.: Real-time divisible load scheduling for cluster computing. In: 13th IEEE Real Time and Embedded Technology and Applications Symposium, RTAS 2007, pp. 303–314, 3–6 (2007)

    Google Scholar 

  13. Yu, J., Buyya, R.: A budget constrained scheduling of workflow applications on utility grids using genetic algorithms. In: Workshop on Workflows in Support of Large-Scale Science, Proceedings of the 15th IEEE International Symposium on High Performance Distributed Computing (HPDC). IEEE CS Press (2006)

    Google Scholar 

  14. Marozzo, F., Talia, D., Trunfio, P.: Adapting MapReduce for Dynamic Environments Using a Peer-to-Peer Model, http://grid.deis.unical.it/papers/pdf/CCA08.pdf

  15. Yang, L., et al.: A new Class of Priority based Weighted Fair Scheduling Algorithm. Physics Procedia 33, 942–948 (2012)

    CrossRef  Google Scholar 

  16. Kyriaki, Z.: Multi-Criteria Job Scheduling in Grid Using an Accelerated Genetic Algorithm. J. Grid Computing 10, 311–323 (2012)

    CrossRef  Google Scholar 

  17. Torabzadeh, E.: Cloud Theory-based Simulated Annealing Approach for Scheduling in the Two-stage Assembly Flowshop. Advances in Engineering Software 41(10), 1243–1258 (2010)

    Google Scholar 

  18. Polo, J., De Nadal, D., Carrera, D., Becerra, Y., Beltran, V., Torres, J., Ayguad´e, E.: Adaptive task scheduling for multi-job mapreduce environments. Technical report UPC-DAC-RR-CAP-2009-28, Departament d’Arquitectura de Com-putadors, Universitat Polit‘ecnica de Catalunya (2009)

    Google Scholar 

  19. Phan, L.T., Zhang, Z., Lo, B.T., Lee, I.: Real-time mapreduce scheduling. Technical Report MS-CIS-10-32, Department of Computer and Information Science, University of Pennsylvania (2010)

    Google Scholar 

  20. Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Proc. IPDPS Workshops, Atlanta, GA (2010)

    Google Scholar 

  21. Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: 8th USENIX symposium on operating systems design and implementation, pp. 29–42. ACM, New York (2008)

    Google Scholar 

  22. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: HPCA 2007: Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24. IEEE Computer Society, Washington (2007)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Gu, L., Tang, Z., Xie, G. (2014). The Implementation of MapReduce Scheduling Algorithm Based on Priority . In: Li, K., Xiao, Z., Wang, Y., Du, J., Li, K. (eds) Parallel Computational Fluid Dynamics. ParCFD 2013. Communications in Computer and Information Science, vol 405. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53962-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-53962-6_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-53961-9

  • Online ISBN: 978-3-642-53962-6

  • eBook Packages: Computer ScienceComputer Science (R0)