Resource-Aware Adaptive Scheduling for MapReduce Clusters

  • Jordà Polo
  • Claris Castillo
  • David Carrera
  • Yolanda Becerra
  • Ian Whalley
  • Malgorzata Steinder
  • Jordi Torres
  • Eduard Ayguadé
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7049)

Abstract

We present a resource-aware scheduling technique for MapReduce multi-job workloads that aims at improving resource utilization across machines while observing completion time goals. Existing MapReduce schedulers define a static number of slots to represent the capacity of a cluster, creating a fixed number of execution slots per machine. This abstraction works for homogeneous workloads, but fails to capture the different resource requirements of individual jobs in multi-user environments. Our technique leverages job profiling information to dynamically adjust the number of slots on each machine, as well as workload placement across them, to maximize the resource utilization of the cluster. In addition, our technique is guided by user-provided completion time goals for each job. Source code of our prototype is available at [1].

Keywords

MapReduce scheduling resource-awareness performance management 

References

  1. 1.
  2. 2.
    Dean, J., Ghemawat, S.: MapReduce: Simplified data processing on large clusters. In: OSDI 2004, San Francisco, CA, pp. 137–150 (December 2004)Google Scholar
  3. 3.
  4. 4.
    Thusoo, A., Shao, Z., Anthony, S., Borthakur, D., Jain, N., Sen Sarma, J., Murthy, R., Liu, H.: Data warehousing and analytics infrastructure at facebook. In: Proceedings of the 2010 International Conference on Management of Data, SIGMOD 2010, pp. 1013–1020. ACM, New York (2010)CrossRefGoogle Scholar
  5. 5.
    Ananthanarayanan, G., Kandula, S., Greenberg, A., Stoica, I., Lu, Y., Saha, B., Harris, E.: Reining in the outliers in map-reduce clusters using mantri. In: OSDI 2010, pp. 1–16. USENIX Asoc., Berkeley (2010)Google Scholar
  6. 6.
    Polo, J., Carrera, D., Becerra, Y., Steinder, M., Whalley, I.: Performance-driven task co-scheduling for MapReduce environments. In: Network Operations and Management Symposium, NOMS, pp. 373–380. IEEE, Osaka (2010)CrossRefGoogle Scholar
  7. 7.
    Wolf, J., Rajan, D., Hildrum, K., Khandekar, R., Kumar, V., Parekh, S., Wu, K.-L., Balmin, A.: Flex: A Slot Allocation Scheduling Optimizer for Mapreduce Workloads. In: Gupta, I., Mascolo, C. (eds.) Middleware 2010. LNCS, vol. 6452, pp. 1–20. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Verma, A., Cherkasova, L., Campbell, R.H.: ARIA: Automatic Resource Inference and Allocation for MapReduce Environments. In: 8th IEEE International Conference on Autonomic Computing, Karlsruhe, Germany (June 2011)Google Scholar
  9. 9.
    Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: OSDI 2008, pp. 29–42. USENIX Association, Berkeley (2008)Google Scholar
  10. 10.
    Tang, C., Steinder, M., Spreitzer, M., Pacifici, G.: A scalable application placement controller for enterprise data centers. In: Procs. of the 16th Intl. Conference on World Wide Web, pp. 331–340. ACM, NY (2007)CrossRefGoogle Scholar
  11. 11.
    Herodotou, H., Babu, S.: Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs. In: VLDB (2010)Google Scholar
  12. 12.
    Pacifici, G., Segmuller, W., Spreitzer, M., Tantawi, A.N.: Dynamic estimation of cpu demand of web traffic. In: Lenzini, L., Cruz, R.L. (eds.) VALUETOOLS. ACM International Conference Proceeding Series, vol. 180, p. 26. ACM (2006)Google Scholar
  13. 13.
    Tesauro, G., Jong, N.K., Das, R., Bennani, M.N.: A hybrid reinforcement learning approach to autonomic resource allocation. In: Proceedings of the 2006 IEEE International Conference on Autonomic Computing, pp. 65–73. IEEE Computer Society, Washington, DC (2006)CrossRefGoogle Scholar
  14. 14.
  15. 15.
    Isard, M., Prabhakaran, V., Jon Currey, U.W., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: SOSP 2009 (2009)Google Scholar
  16. 16.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, EuroSys 2007, pp. 59–72. ACM, New York (2007)Google Scholar
  17. 17.
    Dhok, J., Varma, V.: Using pattern classification for task assignment, http://researchweb.iiit.ac.in/~jaideep/jd-thesis.pdf
  18. 18.

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Jordà Polo
    • 1
  • Claris Castillo
    • 2
  • David Carrera
    • 1
  • Yolanda Becerra
    • 1
  • Ian Whalley
    • 2
  • Malgorzata Steinder
    • 2
  • Jordi Torres
    • 1
  • Eduard Ayguadé
    • 1
  1. 1.Barcelona Supercomputing Center (BSC) and Technical University of Catalonia (UPC)Spain
  2. 2.IBM T.J. Watson Research CenterUSA

Personalised recommendations