On HPC and Cloud Environments Integration

Abstract

Recently in many scientific disciplines, e.g. physics, chemistry, biology and multidisciplinary research have shifted to computational modelling. The main instrument for such numerical experiments has been supercomputing. However, the number of supercomputers and their performance grows significantly slower than the growth of user’s demands. As a result, users of supercomputers may wait for weeks until their job will be done. At the same time the computational power of cloud computing recently grows up considerably represented by heterogeneous DC network with plenty of available resources for numerical experiments. In these circumstances, it may turn out that the time spent by the task in the system, i.e. the time spent in the queue $+$ computing time, in the cloud environment may be shorter than in HPC installation. There are several problems related to cloud and supercomputer environments integration. First, is how to make a decision where to send a computational task: to a supercomputer or to cloud. Secondly, these environments may have significantly different APIs, so moving a computational task from one environment to another may require a lot of code modification. Another significant problem is an automatic provisioning of virtual environment to execute the task properly. The third one is how to organize effectively migration data, computational tasks, applications and services in DC network, between DC and HPC installation? Saying effectively, we mean that network can allocate shortly, on demand, the necessary capacity in order to transfer the necessary amount of data for the right time. It is called ‘Capacity on Demand’ service. In this chapter an environment for academic multidisciplinary research – Meta Cloud Computing Environment (MC2E) is presented. This environment demonstrates the possible solutions and approaches to the problems listed above.