Journal of Grid Computing

, Volume 10, Issue 1, pp 23–46 | Cite as

Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics

  • Chris BunchEmail author
  • Brian Drawert
  • Navraj Chohan
  • Chandra Krintz
  • Linda Petzold
  • Khawaja Shams


In this paper, we present the design and implementation of Neptune, a simple, domain-specific language based on the Ruby programming language. Neptune automates the configuration and deployment of scientific software frameworks over disparate cloud computing systems. Neptune integrates support for MPI, MapReduce, UPC, X10, StochKit, and others. We implement Neptune as a software overlay for the AppScale cloud platform and extend AppScale with support for elasticity and hybrid execution for scientific computing applications. Neptune imposes no overhead on application execution, yet significantly simplifies the application deployment process, enables portability across cloud systems, and promotes lock-in avoidance by specific cloud vendors.


Cloud platform Service placement Domain specific language 

Mathematics Subject Classifications (2010)

D.3.2 C.2.4 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amazon Simple Storage Service (Amazon S3): Last accessed 31 December 2011
  2. 2.
    Armstrong, J., Virding, R., Wikström, C., Williams, M.: Concurrent Programming in ERLANG (1993)Google Scholar
  3. 3.
    Boto: Last accessed 31 December 2011
  4. 4.
    Bunch, C., Chohan, N., Krintz, C., Chohan, J., Kupferman, J., Lakhina, P., Li, Y., Nomura, Y.: An evaluation of distributed datastores using the AppScale Cloud Platform. In: IEEE International Conference on Cloud Computing (2010)Google Scholar
  5. 5.
    Bunch, C., Chohan, N., Krintz, C., Shams, K.: Neptune: a domain specific language for deploying HPC software on cloud platforms. In: ACM Workshop on Scientific Cloud Computing (2011)Google Scholar
  6. 6.
  7. 7.
    Charles, P., Grothoff, C., Saraswat, V., Donawa, C., Kielstra, A., Ebcioglu, K., von Praun, C., Sarkar, V.: X10: an object-oriented approach to non-uniform cluster computing. SIGPLAN Not. 40, 519–538 (2005)CrossRefGoogle Scholar
  8. 8.
    Chohan, N., Bunch, C., Krintz, C., Nomura, Y.: Database-agnostic transaction support for cloud infrastructures. In: IEEE International Conference on Cloud Computing (2011)Google Scholar
  9. 9.
    Chohan, N., Bunch, C., Pang, S., Krintz, C., Mostafa, N., Soman, S., Wolski, R.: AppScale: scalable and open AppEngine application development and deployment. In: ICST International Conference on Cloud Computing (2009)Google Scholar
  10. 10.
    Daigle, B.J., Roh, M.K., Gillespie, D.T., Petzold, L.R.: Automated estimation of rare event probabilities in biochemical systems. J. Phys. Chem. 134, 044110 (2011)CrossRefGoogle Scholar
  11. 11.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of 6th Symposium on Operating System Design and Implementation (OSDI), pp. 137–150 (2004)Google Scholar
  12. 12.
    Drawert, B., Lawson, M.J., Petzold, L., Khammash, M.: The diffusive finite state projection algorithm for effficient simulation of the stochastic reaction-diffusion master equation. J. Phys. Chem. 132(7), 074101-1 (2010)CrossRefGoogle Scholar
  13. 13.
    El-Ghazawi, T., Cantonnet, F.: UPC performance and potential: a NPB experimental study. In: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing. Supercomputing ’02, pp. 1–26. IEEE Computer Society Press, Los Alamitos, CA, USA (2002)Google Scholar
  14. 14.
    El-Samad, H., Kurata, H., Doyle, J.C., Gross, C.A., Khammash, M.: Surviving heat shock: control strategies for robustness and performance. Proc. Natl. Acad. Sci. USA 102(8), 2736–2741 (2005)CrossRefGoogle Scholar
  15. 15.
    Engaging the Missing Middle: Last accessed 31 December 2011
  16. 16.
    Gropp, W., Lusk, E., Doss, N., Skjellum, A.: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Comput. 22(6), 789–828 (1996)zbMATHCrossRefGoogle Scholar
  17. 17.
    Hadoop Distributed File System: Last accessed 31 December 2011
  18. 18.
    Heroku Learns from Amazon EC2 Outage: Last accessed 31 December 2011
  19. 19.
    Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: Networked Systems Design and Implementation (2011)Google Scholar
  20. 20.
    Hucka, M., Finney, A., Sauro, H.M., Bolouri, H., Doyle, J.C., Kitano, H., the rest of the SBML Forum, Arkin, A.P., Bornstein, B.J., Bray, D., Cornish-Bowden, A., Cuellar, A.A., Dronov, S., Gilles, E.D., Ginkel, M., Gor, V., Goryanin, I.I., Hedley, W.J., Hodgman, T.C., Hofmeyr, J.-H., Hunter, P.J., Juty, N.S., Kasberger, J.L., Kremling, A., Kummer, U., Le, N., NovÃĺre, Loew, L.M., Lucio, D., Mendes, P., Minch, E., Mjolsness, E.D., Nakayama, Y., Nelson, M.R., Nielsen, P.F., Sakurada, T., Schaff, J.C., Shapiro, B.E., Shimizu, T.S., Spence, H.D., Stelling, J., Takahashi, K., Tomita, M., Wagner, J., Wang, J.: The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4), 524–531 (2003)CrossRefGoogle Scholar
  21. 21.
    Kaiser, H., Merzky, A., Hirmer, S., Allen, G., Seidel, E.: The SAGA C++ reference implementation: a milestone toward new high-level Grid applications. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06. ACM, New York, NY, USA (2006)Google Scholar
  22. 22.
    Keahey, K., Freeman, T.: Nimbus or an open source cloud platform or the best open source EC2 no money can buy. In: Supercomputing (2008)Google Scholar
  23. 23.
    Koslovski, G., Huu, T.T., Montagnat, J., Vicat-Blanc, P.: Executing distributed applications on virtualized infrastructures specified with the VXDL language and managed by the HIPerNET framework. In: ICST International Conference on Cloud Computing (2009)Google Scholar
  24. 24.
    Krintz, C., Bunch, C., Chohan, N.: AppScale: Open-Source Platform-A s-A-Service. Technical Report 2011-01, University of California, Santa Barbara (2011)Google Scholar
  25. 25.
    Lustre: Last accessed 31 December 2011
  26. 26.
    Nagel, W.E., Arnold, A., Weber, M., Hoppe, H.-Ch., Solchenbach, K.: VAMPIR: visualization and analysis of MPI resources. Supercomputer 12, 69–80 (1996)Google Scholar
  27. 27.
    Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: IEEE International Symposium on Cluster Computing and the Grid. (2009). Last accessed 31 December 2011
  28. 28.
    Pbspro home page: Last accessed 31 December 2011
  29. 29.
    RightScale: RightScale Gems Last accessed 31 December 2011
  30. 30.
    Rolfe, T.J.: A specimen MPI application: N-Queens in parallel. Inroads (bulletin of the ACM SIG on Computer Science Education), vol. 40(4) (2008)Google Scholar
  31. 31.
    Ruby language: Last accessed 31 December 2011
  32. 32.
    Ruby on Rails: Last accessed 31 December 2011
  33. 33.
    Sanft, K.R., Wu, S., Roh, M., Fu, J., Lim, R.K., Petzold, L.R.: StochKit2: software for discrete stochastic simulation of biochemical systems with events. Bioinformatics (2011)Google Scholar
  34. 34.
    Shen, K., Tang, H., Yang, T.: Adaptive two-level thread Management for fast MPI execution on shared memory machines. In: Proceedings of ACM/IEEE SuperComputing ’99 (1999)Google Scholar
  35. 35.
    StochKit: Last accessed 31 December 2011

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  • Chris Bunch
    • 1
    Email author
  • Brian Drawert
    • 1
  • Navraj Chohan
    • 1
  • Chandra Krintz
    • 1
  • Linda Petzold
    • 1
  • Khawaja Shams
    • 2
  1. 1.Computer Science DepartmentUniversity of CaliforniaSanta BarbaraUSA
  2. 2.Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadenaUSA

Personalised recommendations