Skip to main content
SpringerLink
Log in

An Adaptive MapReduce Scheduler for Scalable Heterogeneous Systems

  • Conference paper
  • First Online:
Proceedings of the International Conference on Data Engineering and Communication Technology

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 469))

Abstract

Hadoop MapReduce has been proved to be an efficient model for distributed data processing. This model is widely used by different service providers, which create a challenge of maintaining same efficiency and performance level in different systems. One of the most critical problems for this model is how to overcome heterogeneity and scalability in different systems. The decreases of performance in heterogeneous environment occur due to inefficient scheduling of Map and Reduce tasks. Another important problem is how to minimize master node overhead and network traffic created by scheduling algorithm. In this paper, we introduce a lightweight adaptive scheduler in which we provide the classifier with information about jobs requirement and node capabilities. The scheduler classifies jobs into executable and nonexecutable according to the nodes capabilities. Then the scheduler assigns the tasks to appropriate nodes in the cluster to get highest performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing on large clusters”, Communications of the ACM, VOL. 51, NO. I, pp. 107–113, 2008.

    Google Scholar 

  2. B. Thirumala Rao, Dr. L S S Reddy “Survey on Improved Scheduling in Hadoop MapReduce in Cloud Environments”, in International Journal of Computer Applications (0975-8887) Volume 34. No. 9, November 2011.

    Google Scholar 

  3. B. Thirumala Rao, V. Krishna Reddy. “Performance Issues of Heterogeneous Hadoop Clusters in Cloud Computing”, Global Journal of Computer Science and Technology, Volume XI, Issue VIII, May 2011.

    Google Scholar 

  4. Dr. J. Aghav and Shyam Deshmukh (2013),“Job Classification for MapReduce Scheduler in Heterogeneous Environment”, IEEE Cloud & Ubiquitous Computing & Emerging Technologies (CUBE), 15–16 Nov. 2013, Page: 26.

    Google Scholar 

  5. Dhok J, Varma V (2010), “Using pattern classification for task assignment in MapReduce”, Proceedings of the 8th IEEE International Conference on Grid and Cooperative Computing, Volume 34. No. 9, November 2011.

    Google Scholar 

  6. M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, and I. Stoica. “Improving mapreduce performance in heterogeneous environments”,. In Proc. Of USENIX OSDI, 2008.

    Google Scholar 

  7. Y. Yao, J. Tai, B. Sheng, and N. Mi, “Scheduling heterogeneous mapreduce jobs for efficiency improvement in enterprise clusters”, Integrated Network Management (1 M 2(13), 2013 IFlPIIEEE International Symposium on, pp. 872–875, 2013.

    Google Scholar 

  8. K. Kc and K. Anyanwu, “Scheduling Hadoop Jobs to Meet Deadlines”, in Proc. CloudCom, 2010, pp. 388–392.

    Google Scholar 

  9. Rasooli and D. G. Down, “A hybrid scheduling approach for scalable heterogeneous hadoop systems”, IEEE Computer Society, 2012, pp. 1284–1291.

    Google Scholar 

  10. J. S. Manjaly and V. S. Chooralil, ‘‘Tasktracker aware scheduling for hadoop mapreduce”, 2013 Third International Conference on Advances in Computing and Communications, pp. 278–281, Aug. 2013.

    Google Scholar 

  11. M. Hammoud and M. F. Sakr, “Locality-aware reduce task scheduling for Mapreduce”, in Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM ‘11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 570–576.

    Google Scholar 

  12. S. Humbetov, “Data-intensive computing with map-reduce and hadoop”, IEEE International Conference on Application of Information and Communication Technologies, 17–19 Oct. 2012, pp. 1–5.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, Pune University, Pune, India

    Mohammad Ghoneem &  Lalit Kulkarni

Authors
  1. Mohammad Ghoneem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lalit Kulkarni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Ghoneem .

Editor information

Editors and Affiliations

  1. Department of Computer Science & Engineering, ANITS, Visakhapatnam, Andhra Pradesh, India

    Suresh Chandra Satapathy

  2. Dept. of ECE, Shri Ramswaroop Mem. Group of Prof. Clg, Lucknow, Uttar Pradesh, India

    Vikrant Bhateja

  3. Sabar Institute of Technology, Tajpur, Sabarkantha, Gujarat, India

    Amit Joshi

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer Science+Business Media Singapore

About this paper

Cite this paper

Mohammad Ghoneem, Lalit Kulkarni (2017). An Adaptive MapReduce Scheduler for Scalable Heterogeneous Systems. In: Satapathy, S., Bhateja, V., Joshi, A. (eds) Proceedings of the International Conference on Data Engineering and Communication Technology. Advances in Intelligent Systems and Computing, vol 469. Springer, Singapore. https://doi.org/10.1007/978-981-10-1678-3_57

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-1678-3_57

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-1677-6

  • Online ISBN: 978-981-10-1678-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation