Skip to main content
SpringerLink
Log in

WASC: Adapting Scheduler Configurations for Heterogeneous MapReduce Workloads

  • Conference paper
  • First Online:
Big Scientific Data Benchmarks, Architecture, and Systems (SDBA 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 911))

Included in the following conference series:

  • 303 Accesses

Abstract

MapReduce has emerged as a popular programming paradigm for data intensive computing in both scientific and commercial applications. On a MapReduce cluster, modern resource negotiation frameworks like Hadoop YARN and Mesos support scheduling of jobs submitted by multiple tenants. However, existing job schedulers lacks the automatic adaption to workload variations in their scheduling configuration, which is crucial for the jobs’ latencies because it determines how to share resources among the latest jobs in the system. The major challenge here is, to a MapReduce cluster scheduler, The performance of different configurations depends not only on the number of jobs in different queues, but also on their workload characteristics, which refer to the type and size of jobs. We introduce a workload-adaptive scheduling configuration (WASC) framework for heterogeneous MapReduce jobs. WASC identifies the optimal configuration for them by reasoning about their performances under different configurations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Apache mesos. http://mesos.apache.org/

  2. Hadoop yarn. https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/YARN.html

  3. Statistical workload injector for mapreduce (swim). https://github.com/SWIMProjectUCB/SWIM/wiki

  4. Yarn capacity scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

  5. Yarn fair scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

  6. Alapati, S.R.: Expert Hadoop Administration: Managing, Tuning, and Securing Spark, YARN, and HDFS. Addison-Wesley Professional, Boston (2016)

    Google Scholar 

  7. Bhattacharya, A.A., Culler, D., Friedman, E., Ghodsi, A., Shenker, S., Stoica, I.: Hierarchical scheduling for diverse datacenter workloads. In: SoCC 2013, p. 4. ACM (2013)

    Google Scholar 

  8. Cai, C.X., Saeed, S., Gupta, I., et al.: Phurti: application and network-aware flow scheduling for multi-tenant MapReduce clusters. In: IC2E 2016, pp. 161–170. IEEE (2016)

    Google Scholar 

  9. Chen, Y., Alspaugh, S., Katz, R.: Interactive analytical processing in big data systems: a cross-industry study of MapReduce workloads. In: VLDB 2011, vol. 5, pp. 1802–1813 (2012)

    Article  Google Scholar 

  10. Cho, B., Rahman, M., Chajed, T., et al.: Natjam: design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In: SoCC 2013, p. 6. ACM (2013)

    Google Scholar 

  11. Davis, R., Hamscher, W.: Model-based reasoning: troubleshooting. Explor. Artif. Intell. 8, 297–346 (1988)

    Article  Google Scholar 

  12. Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: fair allocation of multiple resource types. In: NSDI 2011, vol. 11, pp. 24–24 (2011)

    Google Scholar 

  13. Herodotou, H., et al.: Starfish: a self-tuning system for big data analytics. In: CIDR 2011, pp. 261–272 (2011)

    Google Scholar 

  14. Kavulya, S., Tan, J., Gandhi, R., Narasimhan, P.: An analysis of traces from a production MapReduce cluster. In: CCGrid 2010, pp. 94–103. IEEE (2010)

    Google Scholar 

  15. Ren, Z., Xu, X., Wan, J., Shi, W., Zhou, M.: Workload characterization on a production Hadoop cluster: a case study on Taobao. In: IISWC 2012, pp. 3–13. IEEE (2012)

    Google Scholar 

  16. Srirama, S.N., Jakovits, P., Vainikko, E.: Adapting scientific computing problems to clouds using MapReduce. Future Gener. Comput. Syst. 28(1), 184–192 (2012)

    Article  Google Scholar 

  17. Verma, A., Cherkasova, L., Campbell, R.H.: ARIA: automatic resource inference and allocation for MapReduce environments. In: ICAC 2011, pp. 235–244. ACM (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Zhengzhou University, Zhengzhou, China

    Siyi Wang

  2. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Fan Zhang & Rui Han

Authors
  1. Siyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rui Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fan Zhang .

Editor information

Editors and Affiliations

  1. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Rui Ren

  2. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Chen Zheng

  3. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Jianfeng Zhan

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, S., Zhang, F., Han, R. (2019). WASC: Adapting Scheduler Configurations for Heterogeneous MapReduce Workloads. In: Ren, R., Zheng, C., Zhan, J. (eds) Big Scientific Data Benchmarks, Architecture, and Systems. SDBA 2018. Communications in Computer and Information Science, vol 911. Springer, Singapore. https://doi.org/10.1007/978-981-13-5910-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-5910-1_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-5909-5

  • Online ISBN: 978-981-13-5910-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation