Skip to main content

Efficient execution of the WRF model and other HPC applications in the cloud


There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 % overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16


  1. Currently, access to our WRF Web Portal is only provided to Mexican meteorologists that request an account.

  2. This situation does not hold in some of our experimental scenarios in which some Cgroups have bound more than two processes in order to perform CPU stress tests. Also, there are certain HPC applications that require running more than one process per VM. For instance, mpiBLAST requires at least 3 processes per VM in our experimental setup.


  • (Linux-VServer 2013) Linux-VServer (2013) Accessed 9 May 2013

  • Altschul S, Gish W, Miller W, Myers E, Lipman D (1990) Basic local alignment search tool. J Mol Biol 215(3):403–410. doi:10.1006/jmbi.1990.9999

    Article  Google Scholar 

  • Antypas K, Shalf J, Wasserman H (2008) NERSC-6 workload analysis and benchmark selection process. LBNL, Tech Rep

  • Barham P, Dragovic B, Fraser K, Hand S, Harris T, Ho A, Neugebar R, Pratt I, Warfield A (2003) Xen and the art of virtualization. In: ACM Symposium on Operating Systems Principles (SOSP)

  • Cgroup (2013) Accessed 9 May 2013

  • Dai Y, Qi Y, Ren J, Shi Y, Wang X, Yu X (2013) A lightweight VMM on many core for high performance computing. In: Proceedings of the 9th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE '13). ACM, New York, NY, USA, pp 111–120

  • Darling A, Carey L, Feng W (2003) The design, implementation, and evaluation of mpiBLAST. In: the 4th International Conference on Linux Clusters: the HPC Revolution 2003 in conjunction with ClusterWorld Conference & Expo, June 2003

  • Devera M (2013) Hierarchical token bucket. Accessed 9 May 2013

  • DiViNE (2015) Disaster mItigation on VIrtualised eNvironmEnts. Accessed 15 Dec 2015

  • Docker (2015) Accessed 15 Dec 2015

  • Duran-Limon H, Siller M, Blair G, Lopez A, Lombera-Landa J (2011a) Using lightweight virtual machines to achieve resource adaptation in middleware. IET Softw 5:229

    Article  Google Scholar 

  • Duran-Limon H, Silva-Bañuelos L, Tellez-Valdez V, Parlavantzas N, Zhao M (2011) Using Lightweight virtual machines to run high performance computing applications: the case of the weather research and forecasting model. In: Proceedings of the 4th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2011), Melbourne, Australia, December 2011

  • Ekanayake J, Fox G (2009) High performance parallel computing with clouds and cloud technologies. In: 1st International Conference on Cloud Computing (CloudComp09)

  • Ekanayake J, Gunarathne T, Qiu J (2011) Cloud technologies for bioinformatics applications. IEEE Trans Parallel Distrib Syst 22(6)

  • Evangelinos C and Hill C (2008). Cloud computing for parallel scientific HPC applications: feasibility of running coupled atmosphere-ocean climate models on Amazon’s EC2. In: Proceedings of Cloud Computing and Its Applications

  • Expósito R, Taboada G, Ramos S, Touriño J, Doallo R (2013a) Performance analysis of HPC applications in the cloud. Futur Gener Comput Syst 29(1):218–229

    Article  Google Scholar 

  • Expósito R, Taboada G, Ramos S, González-Domínguez J, Touriño J, Doallo R (2013b) Analysis of I/O performance on an Amazon EC2 cluster compute and high I/O platform. J Grid Comput

  • Fernández-Quiruelas V, Fernández J, Baeza C, Cofiño A, Gutiérrez J (2009) Execution management in the GRID for sensitivity studies of global climate simulations. Earth Sci Inf 2:75–82

    Article  Google Scholar 

  • LA Grid (2013) Latin American grid. Accessed 9 May 2013

  • Hoffa C, Mehta G, Freeman G, Deelman E, Keahey K, Berriman B, Good J (2008) On the Use of Cloud Computing for Scientific Workflows. SWBES 2008

  • Huang W, Liu J, Abali B, Panda D (2006) A case for high performance computing with virtual machines. In: Proceedings of the 20th annual international conference on Supercomputing (ICS '06). ACM, New York, pp 125–134

  • Hubert B, Maxwell G, van Mook R, van Oosterhout M, Schroeder P, Spaans J, Larroy P (2013) Linux advanced routing & traffic control HOWTO. Accessed 9 May 2013.

  • HWRF (2015) The hurricane weather research and forecast system. Accessed 15 Dec 2015

  • InfiniBand (2007) InfiniBand architecture specification volume 1, Release 1.2.1, InfiniBand Trade Association, 2007

  • Jackson K, Ramakrishnan L, Muriki K, Canon S, Cholia S, Shalf J, Wasserman H and Wright N (2010) Performance analysis of high performance computing applications on the Amazon Web Services Cloud Amazon Web Services Cloud. CloudCom 2010

  • JTG (2013) Jugi’s traffic generator. Accessed 9 May 2013.

  • Kroeker K (2009) The evolution of virtualization. Commun ACM 52(3):18–20

    Article  Google Scholar 

  • Lange J, Pedretti K, Dinda P, Bridges P, Bae C, Soltero P, Merritt A (2011) Minimal overhead virtualization of a large scale supercomputer. Proceedings of the 2011 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2011), March, 2011

  • LXC (2013) The LXC Linux containers. Accessed 9 May 2013

  • Martinez J, Wang L, Zhao M, Masoud S (2009) Experimental study of large-scale computing on virtualized resources. In: Proceedings of the 3rd international workshop on virtualization technologies in distributed computing (VTDC '09). ACM, New York, pp 35–42

  • Mauch V, Kunze M, Hillenbrand M (2013) High performance cloud computing. Futur Gener Comput Syst 29(6):1408–1416. doi:10.1016/j.future.2012.03.011, ISSN 0167-739X

    Article  Google Scholar 

  • Mell P, Grance T (2011) The NIST definition of cloud computing (800-145). National Institute of Standards and Technology (NIST) National Institute of Standards and Technology (NIST)

  • MiniFE (2013) Accessed 9 May 2013

  • Motika G, Weiss S (2012) Virtio network paravirtualization driver: implementation and performance of a de-facto standard. Comput Stand Interfaces 34(1):36–47

    Article  Google Scholar 

  • NCBI (2013) National Center for Biotechnology Information (NCBI). Accessed 9 May 2013

  • Nova (2015) Accessed 15 Dec 2015

  • OpenStack (2015) Accessed 15 Dec 2015

  • OpenVZ (2013) Accessed 9 May 2013

  • Raj H, Schwan K (2007) High performance and scalable I/O virtualization via self-virtualized devices. In: Proceedings of the 16th International Symposium on High Performance Distributed Computing, pp 179–188

  • Russell R (2008) Virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Syst Rev 42(5):95–103

    Article  Google Scholar 

  • SAP (2013) SAP SD standard benchmark application results, Two-tier internet configuration. Accessed 9 May 2013

  • SPECvirt_sc2010 (2013) Results published by SPEC. Accessed 9 May 2013

  • Sun C, Nishimura H, James S, Song K, Muriki K, Qin Y (2011) HPC cloud applied to lattice optimization. Proceedings of 2011 Particle Accelerator Conference, New York, NY, USA

  • VMware (2006) VMware infrastructure architecture overview. Technical paper. 2006

  • VMware (2007) A performance comparison of hypervisors, 2007

  • Wang G, Ng T (2010) The impact of virtualization on network performance of Amazon EC2 data center. INFOCOM 2010

  • WRF (2013) The weather research and forecasting model. Accessed 9 May 2013

  • Xavier M, Neves M, Rossi F, Ferreto T, Lange T, De Rose C (2013) Performance evaluation of container-based virtualization for high performance computing environments. 21st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp 233–240, Feb. 2013

  • XenSource (2007) A performance comparison of commercial hypervisors, 2007

  • Younge A, Henschel R, Brown J, von Laszewski G, Qiu J, Fox G (2011) Analysis of virtualization technologies for high performance computing environments. 2011 I.E. International Conference on Cloud Computing (CLOUD), pp 9–16, 4–9 July 2011

  • Zhai Y, Liu M, Zhai J, Ma X, Chen W (2011). Cloud versus in-house cluster: evaluating Amazon cluster compute instances for running MPI applications. In: State of the practice reports (SC '11). ACM, New York, Article 11, 10 p, 2011

Download references


Hector A. Duran-Limon would like to thank the State Council of Science and Technology of the State of Jalisco (COECYTJAL) (grant 495-2008), IBM (Faculty Award 2008), RedesClim-CONACYT, and the Mexican’s Public Education Ministry (grant PROMEP-Thematic Networks of Collaboration, call 2011) for supporting this work. Ming Zhao’s research is sponsored by National Science Foundation under grant CCF-0938045 and Department of Homeland Security under grant 2010-ST-062-000039. This work is part of the Latin American Grid (LA Grid) initiative (LA Grid 2013). We also thank Erick Corona for his valuable work to carry out some of the experiments. The authors are also thankful to the anonymous reviewers for their useful comments. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Hector A. Duran-Limon.

Additional information

Communicated by: H. A. Babaie

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duran-Limon, H.A., Flores-Contreras, J., Parlavantzas, N. et al. Efficient execution of the WRF model and other HPC applications in the cloud. Earth Sci Inform 9, 365–382 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: