Workshop on Job Scheduling Strategies for Parallel Processing

JSSPP 2007: Job Scheduling Strategies for Parallel Processing pp 169-187

A Self-optimized Job Scheduler for Heterogeneous Server Clusters

  • Elad Yom-Tov
  • Yariv Aridor
Conference paper

DOI: 10.1007/978-3-540-78699-3_10

Volume 4942 of the book series Lecture Notes in Computer Science (LNCS)
Cite this paper as:
Yom-Tov E., Aridor Y. (2008) A Self-optimized Job Scheduler for Heterogeneous Server Clusters. In: Frachtenberg E., Schwiegelshohn U. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2007. Lecture Notes in Computer Science, vol 4942. Springer, Berlin, Heidelberg

Abstract

Heterogeneous clusters and grid infrastructures are becoming increasingly popular. In these computing infrastructures, machines have different resources, including memory sizes, disk space, and installed software packages. These differences give rise to a problem of over-provisioning, that is, sub-optimal utilization of a cluster due to users requesting resource capacities greater than what their jobs actually need. Our analysis of a real workload file (LANL CM5) revealed differences of up to two orders of magnitude between requested memory capacity and actual memory usage. This paper presents an algorithm to estimate actual resource capacities used by batch jobs. Such an algorithm reduces the need for users to correctly predict the resources required by their jobs, while at the same time managing the scheduling system to obtain superior utilization of available hardware. The algorithm is based on the Reinforcement Learning paradigm; it learns its estimation policy on-line and dynamically modifies it according to the overall cluster load. The paper includes simulation results which indicate that our algorithm can yield an improvement of over 30% in utilization (overall throughput) of heterogeneous clusters.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Elad Yom-Tov
    • 1
  • Yariv Aridor
    • 1
  1. 1.IBM Haifa Research Lab HaifaIsrael