Dynamically Scheduling a Component-Based Framework in Clusters

  • Aleksandra KuzmanovskaEmail author
  • Rudolf H. Mak
  • Dick Epema
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8828)


In many clusters and data centers, application frameworks are used that offer programming models such as Dryad and MapReduce, and jobs submitted to the clusters or data centers may be targeted at specific instances of these frameworks, for example because of the presence of certain data. An important question that then arises is how to allocate resources to framework instances that may have highly fluctuating workloads over their lifetimes. Static resource allocation, a traditional approach for scheduling jobs, may result in inefficient resource allocation because of poor resource utilization during off-peak hours. We address this issue with a strategy for the dynamic deployment of a component-based framework by extending a resource manager responsible for scheduling jobs in multi-cluster environments. This extension allows scheduling multiple concurrent instances of the framework as long-running utility jobs that share computational resources of the cluster. In order to accommodate the fluctuating resource demands of frameworks, we consider two provisioning policies for dynamic resource allocation: OnDemand and Proactive provisioning. We evaluate the effectiveness of both policies by comparing them with static resource allocation on the das4 multi-cluster system. Our results show that dynamic resource allocation gives at least 30 % improvement over the static resource allocation with respect to both the utilization of the resources and the reject rate of the applications within the framework.


Cluster Datacenter Framework Scheduling Dynamic deployment Resource utilization 


  1. 1.
    Buisson, J., Sonmez, O.O., Mohamed, H.H., Lammers, W., Epema, D.H.J.: Scheduling malleable applications in multicluster systems. In: IEEE International Conference on Cluster Computing, pp. 372–381 (2007)Google Scholar
  2. 2.
    David, I., Orlic, B., Mak, R.H., Lukkien, J.J.: Towards resource-aware runtime reconfigurable component-based systems. In: 2010 6th World Congress on Services, SERVICES 2010, pp. 465—466. IEEE Computer Society, Washington, DC, USA (2010).
  3. 3.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008). CrossRefGoogle Scholar
  4. 4.
    Ghit, B., Yigitbasi, N., Epema, D.: Resource management for dynamic mapreduce clusters in multicluster systems. In: Proceedings of the 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) co-located with Supercomputing (SC). IEEE (2012)Google Scholar
  5. 5.
    Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A.D., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: 8th USENIX Conference on Networked Systems Design and Implementation, NSDI 2011, pp. 22–22 (2011).
  6. 6.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev. 41(3), 59–72 (2007). CrossRefGoogle Scholar
  7. 7.
    Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: 22nd ACM Symposium on Operating Systems Principles, SOSP 2009, pp. 261–276 (2009).
  8. 8.
    Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., Wilkes, J.: Omega: flexible, scalable schedulers for large compute clusters. In: Eurosys 2013 (2013)Google Scholar
  9. 9.
    Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: A system for large-scale graph processing. In: 2010 ACM SIGMOD International Conference on Management of Data, pp. 135–146 (2010)Google Scholar
  10. 10.
    Mohamed, H., Epema, D.: Koala: a co-allocating grid scheduler. Concurr. Comput. Pract. Experience 20(16), 1851–1876 (2008)CrossRefGoogle Scholar
  11. 11.
    The distributed asci supercomputer 4.
  12. 12.
    A utility for detailed resource inspection of applications.
  13. 13.
    Vicomo website.
  14. 14.
    Orlic, B., David, I., Mak, R.H., Lukkien, J.J.: Dynamically reconfigurable resource-aware component framework: architecture and concepts. In: Crnkovic, I., Gruhn, V., Book, M. (eds.) ECSA 2011. LNCS, vol. 6903, pp. 212–215. Springer, Heidelberg (2011). CrossRefGoogle Scholar
  15. 15.
    Raman, R., Livny, M., Solomon, M.: Matchmaking: an extensible framework for distributed resource management. Cluster Comput. 2(2), 129–138 (1999). CrossRefGoogle Scholar
  16. 16.
    Sonmez, O.O., Grundeken, B., Mohamed, H.H., Iosup, A., Epema, D.H.J.: Scheduling strategies for cycle scavenging in multicluster grid systems. In: 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009, pp. 12–19 (2009)Google Scholar
  17. 17.
    Sonmez, O.O., Yigitbasi, N., Abrishami, S., Iosup, A., Epema, D.H.J.: Performance analysis of dynamic workflow scheduling in multicluster grids. In: ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), pp. 49–60 (2010)Google Scholar
  18. 18.
    Staples, G.: Torque resource manager. In: 2006 ACM/IEEE conference on Supercomputing (SC 2006) (2006)Google Scholar
  19. 19.
    Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurr. Comput. Pract. Experience 17(2–4), 323–356 (2005). CrossRefGoogle Scholar
  20. 20.
    Vavilapalli, V.K.: Apache hadoop yarn: yet another resource negotiator. In: ACM Symposium on Cloud Computing (2013)Google Scholar
  21. 21.
    White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly Media Inc., Sebastopol (2009) Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Aleksandra Kuzmanovska
    • 1
    Email author
  • Rudolf H. Mak
    • 1
  • Dick Epema
    • 1
  1. 1.Department of Mathematics and Computer ScienceEindhoven University of TechnologyEindhovenThe Netherlands

Personalised recommendations