Abstract
In this paper, we present the design and implementation of Neptune, a simple, domain-specific language based on the Ruby programming language. Neptune automates the configuration and deployment of scientific software frameworks over disparate cloud computing systems. Neptune integrates support for MPI, MapReduce, UPC, X10, StochKit, and others. We implement Neptune as a software overlay for the AppScale cloud platform and extend AppScale with support for elasticity and hybrid execution for scientific computing applications. Neptune imposes no overhead on application execution, yet significantly simplifies the application deployment process, enables portability across cloud systems, and promotes lock-in avoidance by specific cloud vendors.
Similar content being viewed by others
References
Amazon Simple Storage Service (Amazon S3): http://aws.amazon.com/s3/. Last accessed 31 December 2011
Armstrong, J., Virding, R., Wikström, C., Williams, M.: Concurrent Programming in ERLANG (1993)
Boto: http://code.google.com/p/boto/. Last accessed 31 December 2011
Bunch, C., Chohan, N., Krintz, C., Chohan, J., Kupferman, J., Lakhina, P., Li, Y., Nomura, Y.: An evaluation of distributed datastores using the AppScale Cloud Platform. In: IEEE International Conference on Cloud Computing (2010)
Bunch, C., Chohan, N., Krintz, C., Shams, K.: Neptune: a domain specific language for deploying HPC software on cloud platforms. In: ACM Workshop on Scientific Cloud Computing (2011)
Cassandra Operations: http://wiki.apache.org/cassandra/Operations
Charles, P., Grothoff, C., Saraswat, V., Donawa, C., Kielstra, A., Ebcioglu, K., von Praun, C., Sarkar, V.: X10: an object-oriented approach to non-uniform cluster computing. SIGPLAN Not. 40, 519–538 (2005)
Chohan, N., Bunch, C., Krintz, C., Nomura, Y.: Database-agnostic transaction support for cloud infrastructures. In: IEEE International Conference on Cloud Computing (2011)
Chohan, N., Bunch, C., Pang, S., Krintz, C., Mostafa, N., Soman, S., Wolski, R.: AppScale: scalable and open AppEngine application development and deployment. In: ICST International Conference on Cloud Computing (2009)
Daigle, B.J., Roh, M.K., Gillespie, D.T., Petzold, L.R.: Automated estimation of rare event probabilities in biochemical systems. J. Phys. Chem. 134, 044110 (2011)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of 6th Symposium on Operating System Design and Implementation (OSDI), pp. 137–150 (2004)
Drawert, B., Lawson, M.J., Petzold, L., Khammash, M.: The diffusive finite state projection algorithm for effficient simulation of the stochastic reaction-diffusion master equation. J. Phys. Chem. 132(7), 074101-1 (2010)
El-Ghazawi, T., Cantonnet, F.: UPC performance and potential: a NPB experimental study. In: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing. Supercomputing ’02, pp. 1–26. IEEE Computer Society Press, Los Alamitos, CA, USA (2002)
El-Samad, H., Kurata, H., Doyle, J.C., Gross, C.A., Khammash, M.: Surviving heat shock: control strategies for robustness and performance. Proc. Natl. Acad. Sci. USA 102(8), 2736–2741 (2005)
Engaging the Missing Middle: http://www.hpcinthecloud.com/features/Engaging-the-Missing-Middle-in-HPC-95750644.html. Last accessed 31 December 2011
Gropp, W., Lusk, E., Doss, N., Skjellum, A.: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Comput. 22(6), 789–828 (1996)
Hadoop Distributed File System: http://hadoop.apache.org. Last accessed 31 December 2011
Heroku Learns from Amazon EC2 Outage: http://searchcloudcomputing.techtarget.com/news/1378426/Heroku-learns-from-Amazon-EC2-outage. Last accessed 31 December 2011
Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: Networked Systems Design and Implementation (2011)
Hucka, M., Finney, A., Sauro, H.M., Bolouri, H., Doyle, J.C., Kitano, H., the rest of the SBML Forum, Arkin, A.P., Bornstein, B.J., Bray, D., Cornish-Bowden, A., Cuellar, A.A., Dronov, S., Gilles, E.D., Ginkel, M., Gor, V., Goryanin, I.I., Hedley, W.J., Hodgman, T.C., Hofmeyr, J.-H., Hunter, P.J., Juty, N.S., Kasberger, J.L., Kremling, A., Kummer, U., Le, N., NovÃĺre, Loew, L.M., Lucio, D., Mendes, P., Minch, E., Mjolsness, E.D., Nakayama, Y., Nelson, M.R., Nielsen, P.F., Sakurada, T., Schaff, J.C., Shapiro, B.E., Shimizu, T.S., Spence, H.D., Stelling, J., Takahashi, K., Tomita, M., Wagner, J., Wang, J.: The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4), 524–531 (2003)
Kaiser, H., Merzky, A., Hirmer, S., Allen, G., Seidel, E.: The SAGA C++ reference implementation: a milestone toward new high-level Grid applications. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06. ACM, New York, NY, USA (2006)
Keahey, K., Freeman, T.: Nimbus or an open source cloud platform or the best open source EC2 no money can buy. In: Supercomputing (2008)
Koslovski, G., Huu, T.T., Montagnat, J., Vicat-Blanc, P.: Executing distributed applications on virtualized infrastructures specified with the VXDL language and managed by the HIPerNET framework. In: ICST International Conference on Cloud Computing (2009)
Krintz, C., Bunch, C., Chohan, N.: AppScale: Open-Source Platform-A s-A-Service. Technical Report 2011-01, University of California, Santa Barbara (2011)
Lustre: http://www.lustre.org/. Last accessed 31 December 2011
Nagel, W.E., Arnold, A., Weber, M., Hoppe, H.-Ch., Solchenbach, K.: VAMPIR: visualization and analysis of MPI resources. Supercomputer 12, 69–80 (1996)
Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: IEEE International Symposium on Cluster Computing and the Grid. http://open.eucalyptus.com/documents/ccgrid2009.pdf (2009). Last accessed 31 December 2011
Pbspro home page: http://www.altair.com/software/pbspro.htm. Last accessed 31 December 2011
RightScale: RightScale Gems http://rightaws.rubyforge.org/. Last accessed 31 December 2011
Rolfe, T.J.: A specimen MPI application: N-Queens in parallel. Inroads (bulletin of the ACM SIG on Computer Science Education), vol. 40(4) (2008)
Ruby language: http://www.ruby-lang.org. Last accessed 31 December 2011
Ruby on Rails: http://www.rubyonrails.org. Last accessed 31 December 2011
Sanft, K.R., Wu, S., Roh, M., Fu, J., Lim, R.K., Petzold, L.R.: StochKit2: software for discrete stochastic simulation of biochemical systems with events. Bioinformatics (2011)
Shen, K., Tang, H., Yang, T.: Adaptive two-level thread Management for fast MPI execution on shared memory machines. In: Proceedings of ACM/IEEE SuperComputing ’99 (1999)
StochKit: http://www.cs.ucsb.edu/cse/StochKit/. Last accessed 31 December 2011
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bunch, C., Drawert, B., Chohan, N. et al. Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics. J Grid Computing 10, 23–46 (2012). https://doi.org/10.1007/s10723-012-9213-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-012-9213-8