Skip to main content
SpringerLink
Log in

Enhancing the Grid with Cloud Computing

ARC-CC: ARC Cluster in the Cloud

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Scientific computing has evolved considerably in recent years. Scientific applications have become more complex and require an increasing number of computing resources to perform on a large scale. Grid computing has become widely used and is the chosen infrastructure for many scientific calculations and projects, even though it demands a steep learning curve. The computing and storage resources in the Grid are limited, heterogeneous and often overloaded. This heterogeneity is not only present in the hardware setups, but also in the software composition, where configuration permissions are limited. It also has a negative effect on the portability of scientific applications. The use of Cloud resources could eliminate those constraints. In the Cloud, resources are provisioned on demand and can be scaled up and down, while scientists can easily customize their execution environments in the form of virtual machines. Extending the Grid with Cloud resources would improve the utilization of shared resources and would enable the use of additional resources when the Grid resources are overloaded – known as Cloud bursting. We propose an integration model of the Grid and the Cloud using the HTCondor batch system and the NorduGrid ARC middleware. This model enables batch job execution in any public or private Cloud by deploying a virtualized Grid cluster using the ARC middleware - PaaS model for running Grid applications. An evaluation of the virtual Grid cluster was made and compared with the physical one by running NAMD simulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Salnikov, A.: RAINBOW - ARC in the cloud. In: RAINBOW - ARC in the Cloud. Helsinki, Finland (2014)

  2. Apoa1 NAMD benchmark. http://www.ks.uiuc.edu/Research/namd/utilities

  3. Arsuaga-Ríos, M., Heikkilä, S.S., Duellmann, D., Meusel, R., Blomer, J., Couturier, B.: Using S3 cloud storage with ROOT and cvmFS. J. Phys. Conf. Ser. 664(2), 022001 (2015). https://doi.org/10.1088/1742-6596/664/2/022001

    Article  Google Scholar 

  4. Brandic, I., Buyya, R.: Special section: recent advances in utility and cloud computing. Futur. Gener. Comput. Syst. 28(1), 36–38 (2012). https://doi.org/10.1016/j.future.2011.06.001

    Article  Google Scholar 

  5. Burrows, M.: The chubby lock service for loosely-coupled distributed systems. In: Proceedings of the 7th Symposium on Operating Systems Design And Implementation, pp. 335–350. USENIX Association (2006)

  6. Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009). https://doi.org/10.1016/j.future.2008.12.001

    Article  Google Scholar 

  7. Cafaro, M., Aloisio, G. (eds.): Grids, Clouds and Virtualization. Computer Communications And Networks. Springer, London (2011)

  8. Cameron, D., Ellert, M., Jönemo, J., Konstantinov, A., Marton, I., Mohn, B., Nilsen, J.K., Nórden, M., Qiang, W., Roczei, G.: The hosting environment of the advanced resource connector middleware NORDUGRID-TECH-19 (2013)

  9. Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12 (1), 3–27 (2014). https://doi.org/10.1007/s10723-013-9283-2

    Article  Google Scholar 

  10. Consul. https://www.consul.io/

  11. CoreOS/etcd. https://github.com/coreos/etcd

  12. Cunha, J.C., Medeiros, P.D.: Special issue: parallel and distributed computing (EuroPar 2005). Concurrency and Computation: Practice and Experience 19(17), 2183–2184 (2007). https://doi.org/10.1002/cpe.1267

    Article  Google Scholar 

  13. DIRAC interware. http://diracgrid.org/

  14. Eerola, P., Ekelof, T., Ellert, M., Hansen, J.R., Konstantinov, A., Konya, B., Nielsen, J.L., Ould-Saada, F., Smirnova, O., Waananen, A.: The NorduGrid architecture and tools. arXiv:physics/0306002 (2003)

  15. EUgridPMA. https://www.eugridpma.org/

  16. European grid initiative. http://www.egi.eu

  17. Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud computing and grid computing 360-degree compared. In: Cloud Computing and Grid Computing 360-Degree Comparedgrid Computing Environments Workshop, 2008. GCE’08, pp. 1–10 (2008)

  18. Furht, B., Escalante, A. (eds.): Handbook of Cloud Computing. Springer, Boston (2010)

  19. Hariri, S., Varma, A.: High-performance distributed computing: promises and challenges. Concurrency: Practice and Experience 5(4), 233–238 (1993)

    Article  Google Scholar 

  20. Haug, S, Sciacca, FG, Collaboration, A: ATLAS computing on swiss cloud SWITCHengines. J Phys Conf Ser 898(5), 052017 (2017). http://stacks.iop.org/1742-6596/898/i=5/a=052017

    Article  Google Scholar 

  21. Hunter, A.A., Macgregor, A.B., Szabo, T.O., Wellington, C.A., Bellgard, M.I.: Yabi: an online research environment for grid, high performance and cloud computing. Source Code Biol. Med. 7(1), 1 (2012)

    Article  Google Scholar 

  22. Gable, I.: HEP Cloud Production Using the Cloudscheduler/HTCondor architecture. In: CHEP, Okinawa (2015)

  23. Li, K.: Optimal load distribution for multiple heterogeneous blade servers in a cloud computing environment. Journal of Grid Computing 11(1), 27–46 (2013). https://doi.org/10.1007/s10723-012-9239-y

    Article  Google Scholar 

  24. Litzkow, M., Livny, M., Mutka, M.: Condor - a hunter of idle workstations. In: Proceedings of the 8th International Conference of Distributed Computing Systems, pp. 104–111 (1988)

  25. Livny, M., Basney, J., Raman, R., Tannenbaum, T.: Mechanisms for high throughput computing. SPEEDUP Journal 11(1), 36–40 (1997)

    Google Scholar 

  26. Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5th International Conference on on Cloud Computing Technology and Science Cloud Computing Technology and Science (CloudCom), vol. 1, pp. 42–48 (2013)

  27. Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Voms-aware identity service for Openstack. In: EGI Community Forum (2013)

  28. Mateescu, G., Gentzsch, W., Ribbens, C.J.: Hybrid computing—Where HPC meets grid and cloud computing. Futur. Gener. Comput. Syst. 27(5), 440–453 (2011). https://doi.org/10.1016/j.future.2010.11.003

    Article  Google Scholar 

  29. Ellert, M., Michael Grønager, A.K., Kónya, B, Lindemann, J, Livenson, I, Nielsen, JL, Niinimäki, M., Smirnova, O., Wäänänen, A.: Advanced resource connector middleware for lightweight computational grids. Futur. Gener. Comput. Syst. 23(2), 219–240 (2007)

    Article  Google Scholar 

  30. Montero, R.S., Huedo, E., Llorente, I.M.: Dynamic deployment of custom execution environments in grids. In: Dynamic Deployment of Custom Execution Environments in Grids, pp. 33–38, IEEE (2008). https://doi.org/10.1109/ADVCOMP.2008.8

  31. NAMD - Scalable molecular dynamics. http://www.ks.uiuc.edu/Research/namd/

  32. Nordugrid - Grid Research and Development collaboration. www.nordugrid.org

  33. OpenStack - open source cloud computing software. http://www.openstack.org/

  34. Openstack cinder. https://docs.openstack.org/cinder/latest/

  35. OpenStack cloud sofware - openstack training guide. http://docs.openstack.org/training-guides/content/module001-ch004-openstack-architecture.html (2014)

  36. Openstack swift. https://docs.openstack.org/swift/latest/

  37. Osmani, L., Tarkoma, S., Eerola, P., Komu, M., Kortelainen, M.J., Kraemer, O., Lindén, T., Toor, S., White, J.: An overview of the DII-HEP OpenStack based CMS data analysis. J. Phys. Conf. Ser. 608(1), 012010 (2015). http://stacks.iop.org/1742-6596/608/i=1/a=012010

    Article  Google Scholar 

  38. Ostermann, S., Plankensteiner, K., Prodan, R.: Using a new event-based simulation framework for investigating resource provisioning in clouds. Sci. Program. 19(2), 161–178 (2011)

    Google Scholar 

  39. Ostermann, S., Prodan, R., Fahringer, T.: Extending grids with cloud resource management for scientific computing. In: 2009 10Th IEEE/ACM International Conference On Grid Computing, pp. 42–49. IEEE (2009)

  40. Packer. https://packer.io/

  41. Partnership for advanced computing in europe. http://www.prace-ri.eu/

  42. Phillips, J.C., Zheng, G., Kumar, S: NAMD: biomolecular simulation on thousands of processors. In: ACM/IEEE 2002 Conference on Supercomputing, pp. 36–36 (2002)

  43. Phoronix test suite. http://www.phoronix-test-suite.com/

  44. Alfieri, R., Cecchini, R., Ciaschini, V., dell’Agnello, L., Frohner, Á., Gianoli, A.: VOMS, an Authorization System for Virtual Organizations. In: VOMS, an Authorization System for Virtual Organizations, vol. 2970, pp. 33–40. Springer, Berlin (2004)

  45. Ranganathan, A., Campbell, R.H.: What is the complexity of a distributed computing system? Complexity 12(6), 37–45 (2007). https://doi.org/10.1002/cplx.20189

    Article  MathSciNet  Google Scholar 

  46. Montero, R.S., Moreno-Vozmediano, R, Llorente, I.M.: An elasticity model for high throughput computing clusters. J. Parallel Distrib. Comput., Special issue on cloud computing 71(6), 750–757 (2011)

    Google Scholar 

  47. Schwiegelshohn, U., Badia, R.M., Bubak, M., Danelutto, M., Dustdar, S., Gagliardi, F., Geiger, A., Hluchy, L., Kranzlmüller, D., Laure, E., Priol, T., Reinefeld, A., Resch, M., Reuter, A., Rienhoff, O., Rüter, T., Sloot, P., Talia, D., Ullmann, K., Yahyapour, R., von Voigt, G.: Perspectives on grid computing. Futur. Gener. Comput. Syst. 26(8), 1104–1115 (2010). https://doi.org/10.1016/j.future.2010.05.010

    Article  Google Scholar 

  48. Slovenian national supercomputing network. https://www.sling.si

  49. Slurm workload manager. https://slurm.schedmd.com/

  50. Sysbench benchmarking tool. https://launchpad.net/sysbench

  51. Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency-Practice and Experience 17(2-4), 323–356 (2005)

    Article  Google Scholar 

  52. The Atlas Collaboration: The ATLAS experiment at the CERN large hadron collider. J Instrum 3(08), S08,003 (2008)

    Google Scholar 

  53. Worldwide lhc computing grid. http://wlcg.web.cern.ch/

  54. Zookeeper. https://zookeeper.apache.org/

Download references

Acknowledgements

The authors would like to thank Jan Jona Javoršek and Farid Ould-Saada for their help and advice on the subject.

Author information

Authors and Affiliations

  1. Networking Infrastructure Centre, Jožef Stefan Institute, Jamova 39, Ljubljana, Slovenia

    Barbara Krašovec

  2. Department for Experimental Particle Physics, Jožef Stefan Institute, Jamova 39, Ljubljana, Slovenia

    Andrej Filipčič

Authors
  1. Barbara Krašovec
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrej Filipčič
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Barbara Krašovec.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Krašovec, B., Filipčič, A. Enhancing the Grid with Cloud Computing. J Grid Computing 17, 119–135 (2019). https://doi.org/10.1007/s10723-018-09472-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-018-09472-w

Keywords

Search

Navigation