Dynamic Resource Management in a HPC and Cloud Hybrid Environment

  • Miao Chen
  • Fang Dong
  • Junzhou Luo
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8285)

Abstract

Recently, the large-scale cluster of data center is usually constructed to support both HPC and Cloud computing. It can be explained from two aspects: (1) The data center is typically a sharing environment for all the users, users may submit different types of jobs (HPC and Cloud computing) for processing currently; (2) Some applications can be divided into two parts of subtasks which are suitable to HPC and Cloud computing respectively, e.g. the AMS (Alpha Magnetic Spectrometer) experiment is such a typical application. Thus in order to provide good service for both computing models, it is needed to construct a HPC and Cloud hybrid environment. An existing management mechanism is to allocate fixed proportions of resources for different application environments. However, this approach has a significant performance drawback that is the low resource utilization. In order to overcome this drawback, we propose a dynamic resource management framework and mechanism to satisfy the requirements of both HPC and Cloud computing. Firstly we present a prediction model that is used to predict the arrival rate of all kinds of jobs (HPC types and Cloud types). Based on the prediction results, we propose a dynamic resource allocation algorithm, which manages dynamic resources allocation by using queuing theory. Finally, we evaluate our mechanism by real data sets from AMS experiment and Cloud tasks running on the HPC center in Southeast University. The results show that the proposed mechanism can effectively improve resource utilization at least 30% in this hybrid environment.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kleinrack, L.: Queueing Systems, Volume 11: Computer Applications. Wiley (1976)Google Scholar
  2. 2.
    Yongwei, W., Yulai, Y.: Load Prediction Using Hybrid Model for Computational Grid. In: 8th Grid Computing Conference, pp. 235–242 (2008)Google Scholar
  3. 3.
    He, Q., Zhou, S., Kobler, B., Duffy, D., McGlynn, T.: Case Study for Running HPC Applications in Public Clouds. In: Proc. 19th ACM International Symposium on High Performance Distributed Computing (HPDC), pp. 395–401 (2010)Google Scholar
  4. 4.
    Rehr, J.J., Vila, F.D., Gardner, J.P., Svec, L., Prange, M.: Scientic Computing in the Cloud. Computing in Science & Engineering 12, 34–43 (2010)CrossRefGoogle Scholar
  5. 5.
    Chen, L., Agrawal, G.: A static resource allocation framework for Grid-based streaming applications. Concurrency and Computation: Practice and Experience, 653–666 (2006)Google Scholar
  6. 6.
    Kim, H., El-Khamra, Y., Jha, S., Parashar, M.: Exploring application and infrastructure adaptation on hybrid grid-cloud infrastructure. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, June 21-25, 2010, pp. 402–412 (2010)Google Scholar
  7. 7.
    Braun, T.D., Siegel, H.J., Maciejewski, A., Hong, Y.: Static resource allocation for heterogeneous computing environments with tasks having dependencies, priorities, deadlines, and multiple versions. Journal of Parallel and Distributed Computing, 1504–1516 (2008)Google Scholar
  8. 8.
    Assuncao, M.D., Costanzo, A., Buyya, R.: Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters. In: Pro. the 18th ACM International Symposium on High Performance Distributed Computing, pp. 141–150. ACM, New York (2009)CrossRefGoogle Scholar
  9. 9.
    Martinaitis, P., Patten, C., Wendelborn, A.: Remote interaction and scheduling aspects of cloud based streams. In: 2009 5th IEEE International Conference on E-Science Workshops, pp. 39–47 (December 2009)Google Scholar
  10. 10.
    Nie, L., Xu, Z.: An adaptive scheduling mechanism for elastic grid computing. In: International Conference on Semantics, Knowledge and Grid, pp. 184–191 (2009)Google Scholar
  11. 11.
    Dornemann, T., Juhnke, E., Freisleben, B.: On-demand resource provisioning for bpel workflows using amazon’s elastic compute cloud. In: The 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 140–147. IEEE Computer Society, Washington, DC (2009)CrossRefGoogle Scholar
  12. 12.
    Ozer, A., Ozturan, C.: An auction based mathematical model and heuristics for resource co-allocation problem in grids and clouds. In: Fifth International Conference on Soft Computing, Computing with Words and Perceptions in System Analysis, Decision and Control, ICSCCW 2009, pp. 1–4 (September 2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Miao Chen
    • 1
  • Fang Dong
    • 1
  • Junzhou Luo
    • 1
  1. 1.School of Computer Science and EngineeringSoutheast UniversityNanjingChina

Personalised recommendations