Integration of High-Performance Computing into Cloud Computing Services
High-Performance Computing (HPC) projects span a spectrum of computer hardware implementations ranging from peta-flop supercomputers, high-end tera-flop facilities running a variety of operating systems and applications, to mid-range and smaller computational clusters used for HPC application development, pilot runs and prototype staging clusters. What they all have in common is that they operate as a stand-alone system rather than a scalable and shared user re-configurable resource. The advent of cloud computing has changed the traditional HPC implementation. In this article, we will discuss a very successful production-level architecture and policy framework for supporting HPC services within a more general cloud computing infrastructure. This integrated environment, called Virtual Computing Lab (VCL), has been operating at NC State since fall 2004. Nearly 8,500,000 HPC CPU-Hrs were delivered by this environment to NC State faculty and students during 2009. In addition, we present and discuss operational data that show that integration of HPC and non-HPC (or general VCL) services in a cloud can substantially reduce the cost of delivering cloud services (down to cents per CPU hour).
KeywordsCloud Computing Virtual Machine Message Passing Interface Cloud Computing Environment Cloud Computing System
This work is supported in part by IBM Corp., Intel Corp., SAS Institute, NetApp, EMC, NC State University, State of North Carolina, UNC General Administration, and DOE (DE-FC02-07)ER25809. The authors would like thank the NC State VCL team for their advice, support and input.
- Apache VCL (2010). Retrieved March 2010, from http://cwiki.apache.org/VCL/.
- Armbrust, M., Fox, A., Griffith, R., Joseph, A., Katz, R., Konwinski, A., Lee, Gunho., Patterson, D., Rabkin, A., Stoica, I., & Zaharia, M. (February 2009). Above the clouds: A Berkeley view of cloud computing (Tech. Rep. No. UCB/EECS-2009-28), at http://www.berkeley.edu/Pubs/TechRpts.EECS-2009-28.html.
- Averitt, S., Bugaev, M., Peeler, A., Schaffer, H., Sills, E., Stein, S., et al. (May 2007). The virtual computing laboratory. Proceedings of the International Conference on Virtual Computing Initiative, IBM Corporation, Research Triangle Park, NC, http://vcl.ncsu.edu/news/papers-publications/virtual-computing-laboratory-vcl-whitepaper, 1–16.Google Scholar
- Dreher, P., Vouk, M. A., Sills, E., & Averitt, S. (2009). Evidence for a cost effective cloud computing implementation based upon the NC state virtual computing laboratory model. In W. Gentzsch, L. Grandinetti, & G. Joubert (Eds.), Advances in parallel computing, high speed and large scale scientific computing (Vol. 18, pp. 236–250), ISBN 978-1-60750-073-5.Google Scholar
- Dreher, P., Vouk, M., Averitt, S., & Sills, E. “An Open Source Option for Cloud Computing in Education and Research”, In: S. Murugesan (ed.) Cloud Computing: Technologies, Business Models, Opportunities and Challenges, CRC Press/Chapman and Hall, 2010.Google Scholar
- Golden, B. (2009). The case against cloud computing, CIO Magazine.Google Scholar
- IBM (2007). IBM introduces ready-to-use cloud computing. Retrieved November 15, 2007, from http://www-03.ibm.com/press/us/en/pressrelease/22613.wss.
- Lohr, S., Google, & I.B.M. (October 2007). Join in ‘Cloud Computing’ research, The New York Times.Google Scholar
- Schaffer, H. E., Averitt, S. F., Hoit, M. I., Peeler, A., Sills, E. D., & Vouk, M. A. (July 2009). NCSUs virtual computing lab: A cloud computing solution. IEEE Computer, 42(7), 94–97.Google Scholar
- Vouk, M., Averitt, S., Bugaev, M., Kurth, A., Peeler, A., Rindos, A., et al. (May 2008). ‘Powered by VCL’ – Using virtual computing laboratory (VCL) technology to power cloud computing. Proceedings of the 2nd International Conference on Virtual Computing (ICVCI), RTP, NC, May 15–16, 1–10. http://vcl.ncsu.edu/news/papers-publications/powered-vcl-using-virtual-computing-laboratory-vcl.
- Vouk, M. (2008). Cloud computing – Issues, research and implementations. Journal of Computing and Information Technology, 16(4), 235–246.Google Scholar
- Walker, E. (2008). Benchmarking Amazon EC2 for high-performance scientific computing. Usenix Magazine, 33(5). http://www.usenix.org/publications/login/2008-10/openpdfs/walker.pdf. Accessed Sep 3, 2010.