Advertisement

Enhancing the Grid with Cloud Computing

ARC-CC: ARC Cluster in the Cloud
  • Barbara KrašovecEmail author
  • Andrej Filipčič
Article
  • 7 Downloads

Abstract

Scientific computing has evolved considerably in recent years. Scientific applications have become more complex and require an increasing number of computing resources to perform on a large scale. Grid computing has become widely used and is the chosen infrastructure for many scientific calculations and projects, even though it demands a steep learning curve. The computing and storage resources in the Grid are limited, heterogeneous and often overloaded. This heterogeneity is not only present in the hardware setups, but also in the software composition, where configuration permissions are limited. It also has a negative effect on the portability of scientific applications. The use of Cloud resources could eliminate those constraints. In the Cloud, resources are provisioned on demand and can be scaled up and down, while scientists can easily customize their execution environments in the form of virtual machines. Extending the Grid with Cloud resources would improve the utilization of shared resources and would enable the use of additional resources when the Grid resources are overloaded – known as Cloud bursting. We propose an integration model of the Grid and the Cloud using the HTCondor batch system and the NorduGrid ARC middleware. This model enables batch job execution in any public or private Cloud by deploying a virtualized Grid cluster using the ARC middleware - PaaS model for running Grid applications. An evaluation of the virtual Grid cluster was made and compared with the physical one by running NAMD simulations.

Keywords

Grid computing Cloud computing Parallel applications Distributed computing NAMD Interoperability HPC ARC 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgements

The authors would like to thank Jan Jona Javoršek and Farid Ould-Saada for their help and advice on the subject.

References

  1. 1.
    Salnikov, A.: RAINBOW - ARC in the cloud. In: RAINBOW - ARC in the Cloud. Helsinki, Finland (2014)Google Scholar
  2. 2.
  3. 3.
    Arsuaga-Ríos, M., Heikkilä, S.S., Duellmann, D., Meusel, R., Blomer, J., Couturier, B.: Using S3 cloud storage with ROOT and cvmFS. J. Phys. Conf. Ser. 664(2), 022001 (2015).  https://doi.org/10.1088/1742-6596/664/2/022001 CrossRefGoogle Scholar
  4. 4.
    Brandic, I., Buyya, R.: Special section: recent advances in utility and cloud computing. Futur. Gener. Comput. Syst. 28(1), 36–38 (2012).  https://doi.org/10.1016/j.future.2011.06.001 CrossRefGoogle Scholar
  5. 5.
    Burrows, M.: The chubby lock service for loosely-coupled distributed systems. In: Proceedings of the 7th Symposium on Operating Systems Design And Implementation, pp. 335–350. USENIX Association (2006)Google Scholar
  6. 6.
    Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009).  https://doi.org/10.1016/j.future.2008.12.001 CrossRefGoogle Scholar
  7. 7.
    Cafaro, M., Aloisio, G. (eds.): Grids, Clouds and Virtualization. Computer Communications And Networks. Springer, London (2011)Google Scholar
  8. 8.
    Cameron, D., Ellert, M., Jönemo, J., Konstantinov, A., Marton, I., Mohn, B., Nilsen, J.K., Nórden, M., Qiang, W., Roczei, G.: The hosting environment of the advanced resource connector middleware NORDUGRID-TECH-19 (2013)Google Scholar
  9. 9.
    Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12 (1), 3–27 (2014).  https://doi.org/10.1007/s10723-013-9283-2 CrossRefGoogle Scholar
  10. 10.
  11. 11.
  12. 12.
    Cunha, J.C., Medeiros, P.D.: Special issue: parallel and distributed computing (EuroPar 2005). Concurrency and Computation: Practice and Experience 19(17), 2183–2184 (2007).  https://doi.org/10.1002/cpe.1267 CrossRefGoogle Scholar
  13. 13.
    DIRAC interware. http://diracgrid.org/
  14. 14.
    Eerola, P., Ekelof, T., Ellert, M., Hansen, J.R., Konstantinov, A., Konya, B., Nielsen, J.L., Ould-Saada, F., Smirnova, O., Waananen, A.: The NorduGrid architecture and tools. arXiv:physics/0306002 (2003)
  15. 15.
  16. 16.
    European grid initiative. http://www.egi.eu
  17. 17.
    Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud computing and grid computing 360-degree compared. In: Cloud Computing and Grid Computing 360-Degree Comparedgrid Computing Environments Workshop, 2008. GCE’08, pp. 1–10 (2008)Google Scholar
  18. 18.
    Furht, B., Escalante, A. (eds.): Handbook of Cloud Computing. Springer, Boston (2010)Google Scholar
  19. 19.
    Hariri, S., Varma, A.: High-performance distributed computing: promises and challenges. Concurrency: Practice and Experience 5(4), 233–238 (1993)CrossRefGoogle Scholar
  20. 20.
    Haug, S, Sciacca, FG, Collaboration, A: ATLAS computing on swiss cloud SWITCHengines. J Phys Conf Ser 898(5), 052017 (2017). http://stacks.iop.org/1742-6596/898/i=5/a=052017 CrossRefGoogle Scholar
  21. 21.
    Hunter, A.A., Macgregor, A.B., Szabo, T.O., Wellington, C.A., Bellgard, M.I.: Yabi: an online research environment for grid, high performance and cloud computing. Source Code Biol. Med. 7(1), 1 (2012)CrossRefGoogle Scholar
  22. 22.
    Gable, I.: HEP Cloud Production Using the Cloudscheduler/HTCondor architecture. In: CHEP, Okinawa (2015)Google Scholar
  23. 23.
    Li, K.: Optimal load distribution for multiple heterogeneous blade servers in a cloud computing environment. Journal of Grid Computing 11(1), 27–46 (2013).  https://doi.org/10.1007/s10723-012-9239-y CrossRefGoogle Scholar
  24. 24.
    Litzkow, M., Livny, M., Mutka, M.: Condor - a hunter of idle workstations. In: Proceedings of the 8th International Conference of Distributed Computing Systems, pp. 104–111 (1988)Google Scholar
  25. 25.
    Livny, M., Basney, J., Raman, R., Tannenbaum, T.: Mechanisms for high throughput computing. SPEEDUP Journal 11(1), 36–40 (1997)Google Scholar
  26. 26.
    Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5th International Conference on on Cloud Computing Technology and Science Cloud Computing Technology and Science (CloudCom), vol. 1, pp. 42–48 (2013)Google Scholar
  27. 27.
    Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Voms-aware identity service for Openstack. In: EGI Community Forum (2013)Google Scholar
  28. 28.
    Mateescu, G., Gentzsch, W., Ribbens, C.J.: Hybrid computing—Where HPC meets grid and cloud computing. Futur. Gener. Comput. Syst. 27(5), 440–453 (2011).  https://doi.org/10.1016/j.future.2010.11.003 CrossRefGoogle Scholar
  29. 29.
    Ellert, M., Michael Grønager, A.K., Kónya, B, Lindemann, J, Livenson, I, Nielsen, JL, Niinimäki, M., Smirnova, O., Wäänänen, A.: Advanced resource connector middleware for lightweight computational grids. Futur. Gener. Comput. Syst. 23(2), 219–240 (2007)CrossRefGoogle Scholar
  30. 30.
    Montero, R.S., Huedo, E., Llorente, I.M.: Dynamic deployment of custom execution environments in grids. In: Dynamic Deployment of Custom Execution Environments in Grids, pp. 33–38, IEEE (2008).  https://doi.org/10.1109/ADVCOMP.2008.8
  31. 31.
    NAMD - Scalable molecular dynamics. http://www.ks.uiuc.edu/Research/namd/
  32. 32.
    Nordugrid - Grid Research and Development collaboration. www.nordugrid.org
  33. 33.
    OpenStack - open source cloud computing software. http://www.openstack.org/
  34. 34.
  35. 35.
  36. 36.
  37. 37.
    Osmani, L., Tarkoma, S., Eerola, P., Komu, M., Kortelainen, M.J., Kraemer, O., Lindén, T., Toor, S., White, J.: An overview of the DII-HEP OpenStack based CMS data analysis. J. Phys. Conf. Ser. 608(1), 012010 (2015). http://stacks.iop.org/1742-6596/608/i=1/a=012010 CrossRefGoogle Scholar
  38. 38.
    Ostermann, S., Plankensteiner, K., Prodan, R.: Using a new event-based simulation framework for investigating resource provisioning in clouds. Sci. Program. 19(2), 161–178 (2011)Google Scholar
  39. 39.
    Ostermann, S., Prodan, R., Fahringer, T.: Extending grids with cloud resource management for scientific computing. In: 2009 10Th IEEE/ACM International Conference On Grid Computing, pp. 42–49. IEEE (2009)Google Scholar
  40. 40.
  41. 41.
    Partnership for advanced computing in europe. http://www.prace-ri.eu/
  42. 42.
    Phillips, J.C., Zheng, G., Kumar, S: NAMD: biomolecular simulation on thousands of processors. In: ACM/IEEE 2002 Conference on Supercomputing, pp. 36–36 (2002)Google Scholar
  43. 43.
  44. 44.
    Alfieri, R., Cecchini, R., Ciaschini, V., dell’Agnello, L., Frohner, Á., Gianoli, A.: VOMS, an Authorization System for Virtual Organizations. In: VOMS, an Authorization System for Virtual Organizations, vol. 2970, pp. 33–40. Springer, Berlin (2004)Google Scholar
  45. 45.
    Ranganathan, A., Campbell, R.H.: What is the complexity of a distributed computing system? Complexity 12(6), 37–45 (2007).  https://doi.org/10.1002/cplx.20189 MathSciNetCrossRefGoogle Scholar
  46. 46.
    Montero, R.S., Moreno-Vozmediano, R, Llorente, I.M.: An elasticity model for high throughput computing clusters. J. Parallel Distrib. Comput., Special issue on cloud computing 71(6), 750–757 (2011)Google Scholar
  47. 47.
    Schwiegelshohn, U., Badia, R.M., Bubak, M., Danelutto, M., Dustdar, S., Gagliardi, F., Geiger, A., Hluchy, L., Kranzlmüller, D., Laure, E., Priol, T., Reinefeld, A., Resch, M., Reuter, A., Rienhoff, O., Rüter, T., Sloot, P., Talia, D., Ullmann, K., Yahyapour, R., von Voigt, G.: Perspectives on grid computing. Futur. Gener. Comput. Syst. 26(8), 1104–1115 (2010).  https://doi.org/10.1016/j.future.2010.05.010 CrossRefGoogle Scholar
  48. 48.
    Slovenian national supercomputing network. https://www.sling.si
  49. 49.
    Slurm workload manager. https://slurm.schedmd.com/
  50. 50.
    Sysbench benchmarking tool. https://launchpad.net/sysbench
  51. 51.
    Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency-Practice and Experience 17(2-4), 323–356 (2005)CrossRefGoogle Scholar
  52. 52.
    The Atlas Collaboration: The ATLAS experiment at the CERN large hadron collider. J Instrum 3(08), S08,003 (2008)Google Scholar
  53. 53.
    Worldwide lhc computing grid. http://wlcg.web.cern.ch/
  54. 54.

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Networking Infrastructure CentreJožef Stefan InstituteLjubljanaSlovenia
  2. 2.Department for Experimental Particle PhysicsJožef Stefan InstituteLjubljanaSlovenia

Personalised recommendations