Advertisement

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

  • D. Salomoni
  • I. Campos
  • L. Gaido
  • J. Marco de Lucas
  • P. Solagna
  • J. Gomes
  • L. Matyska
  • P. Fuhrman
  • M. Hardt
  • G. Donvito
  • L. Dutka
  • M. Plociennik
  • R. Barbera
  • I. Blanquer
  • A. Ceccanti
  • E. Cetinic
  • M. David
  • C. Duma
  • A. López-García
  • G. Moltó
  • P. Orviz
  • Z. Sustr
  • M. Viljoen
  • F. Aguilar
  • L. Alves
  • M. Antonacci
  • L. A. Antonelli
  • S. Bagnasco
  • A. M. J. J. Bonvin
  • R. Bruno
  • Y. Chen
  • A. Costa
  • D. Davidovic
  • B. Ertl
  • M. Fargetta
  • S. Fiore
  • S. Gallozzi
  • Z. Kurkcuoglu
  • L. Lloret
  • J. Martins
  • A. Nuzzo
  • P. Nassisi
  • C. Palazzo
  • J. Pina
  • E. Sciacca
  • D. Spiga
  • M. Tangaro
  • M. Urbaniak
  • S. Vallero
  • B. Wegh
  • V. Zaccolo
  • F. Zambelli
  • T. Zok
Open Access
Article

Abstract

This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.

Keywords

Cloud computing Platform as a service Containers Software management Advanced user interfaces Authorization and authentication 

Notes

Acknowledgments

INDIGO-Datacloud has been funded by the European Commision H2020 research and innovation program under grant agreement RIA 653549.

References

  1. 1.
    García, A.L., Castillo, E.F.-d., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5Th International Conference on Cloud Computing Technology and Science, pp 42–48 (2013)Google Scholar
  2. 2.
    Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12(1), 3–27 (2014)CrossRefGoogle Scholar
  3. 3.
    Craig, A.L.: A design space review for general federation management using keystone. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp 720–725. IEEE Computer Society (2014)Google Scholar
  4. 4.
    Pustchi, N., Krishnan, R., Sandhu, R.: Authorization federation in iaas multi cloud. In: Proceedings of the 3rd International Workshop on Security in Cloud Computing, pp 63–71. ACM (2015)Google Scholar
  5. 5.
    Lee, C.A., Desai, N., Brethorst, A.: A Keystone-Based Virtual Organization Management System. In: 2014 IEEE 6Th International Conference On Cloud Computing Technology and Science (Cloudcom), pp 727–730. IEEE (2014)Google Scholar
  6. 6.
    Castillo, E.F.-d., Scardaci, D., García, A.L.: The EGI Federated Cloud e-Infrastructure. Procedia Computer Science 68, 196–205 (2015)CrossRefGoogle Scholar
  7. 7.
    AARC project: AARC Blueprint Architecture, see https://aarc-project.eu/architecture. Technical report (2016)
  8. 8.
    Oesterle, F., Ostermann, S., Prodan, R., Mayr, G.J.: Experiences with distributed computing for meteorological applications: grid computing and cloud computing. Geosci. Model Dev. 8(7), 2067–2078 (2015)CrossRefGoogle Scholar
  9. 9.
    Plasencia, I.C., Castillo, E.F.-d., Heinemeyer, S., García, A.L., Pahlen, F., Borges, G.: Phenomenology tools on cloud infrastructures using OpenStack. The European Physical Journal C 73(4), 2375 (2013)CrossRefGoogle Scholar
  10. 10.
    Boettiger, C.: An introduction to docker for reproducible research. ACM SIGOPS Operating Systems Review 49(1), 71–79 (2015)CrossRefGoogle Scholar
  11. 11.
    Docker: http://www.docker.com (2013)
  12. 12.
    Gomes, J., Campos, I., Bagnaschi, E., David, M., Alves, L., Martins, J., Pina, J., Alvaro, L.-G., Orviz, P.: Enabling rootless linux containers in multi-user environments: the udocker tool. Computing Physics Communications.  https://doi.org/10.1016/j.cpc.2018.05.021 (2018)
  13. 13.
    Zhang, Z., Chuan, W., Cheung, D.W.L.: A survey on cloud interoperability taxonomies, standards, and practice. SIGMETRICS perform. Eval. Rev. 40(4), 13–22 (2013)CrossRefGoogle Scholar
  14. 14.
    Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments. Journal of Grid Computing 12(4), 559–592 (2014)CrossRefGoogle Scholar
  15. 15.
    Nyrén, R., Metsch, T., Edmonds, A., Papaspyrou, A.: Open Cloud Computing Interface–Core. Technical report, Open Grid Forum (2010)Google Scholar
  16. 16.
    Metsch, T., Edmonds, A.: Open Cloud Computing Interface-Infrastructure. Technical report, Open Grid Forum (2010)Google Scholar
  17. 17.
    Metsch, T., Edmonds, A.: Open Cloud Computing Interface-RESTful HTTP Rendering. Technical report, Open Grid Forum (2011)Google Scholar
  18. 18.
    (Ca Technologies) Lipton, P., (Ibm) Moser, S., (Vnomic) Palma, D., (Ibm) Spatzier, T.: Topology and Orchestration Specification for Cloud Applications. Technical report, OASIS Standard (2013)Google Scholar
  19. 19.
    Teckelmann, R., Reich, C., Sulistio, A.: Mapping of cloud standards to the taxonomy of interoperability in IaaS. In: Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, pp 522–526 (2011)Google Scholar
  20. 20.
    García, A.L., Castillo, E.F.-d., Fernández, P.O.: Standards for enabling heterogeneous IaaS cloud federations. Computer Standards & Interfaces 47, 19–23 (2016)CrossRefGoogle Scholar
  21. 21.
    Caballer, M., Zala, S., García, A.L., Montó, G., Fernández, P.O., Velten, M.: Orchestrating complex application architectures in heterogeneous clouds. Journal of Grid Computing 16 (1), 3–18 (2018)CrossRefGoogle Scholar
  22. 22.
    Hardt, M., Jejkal, T., Plasencia, I.C., Castillo, E.F.-d., Jackson, A., Weiland, M., Palak, B., Plociennik, M., Nielsson, D.: Transparent Access to Scientific and Commercial Clouds from the Kepler Workflow Engine. Computing and Informatics 31(1), 119 (2012)MATHGoogle Scholar
  23. 23.
    Fakhfakh, F., Kacem, H.H., Kacem, A.H.: Workflow Scheduling in Cloud Computing a Survey. In: IEEE 18Th International Enterprise Distributed Object Computing Conference Workshops and Demonstrations (EDOCW), 2014, Vol. 71, pp. 372–378. Springer, New York (2014)Google Scholar
  24. 24.
    Stockton, D.B., Santamaria, F.: Automating NEURON simulation deployment in cloud resources. Neuroinformatics 15(1), 51–70 (2017)CrossRefGoogle Scholar
  25. 25.
    Plóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level Dynamic Workflow Orchestration in the INDIGO DataCloud for Large-scale, Climate Change Data Analytics Experiments. Procedia Computer Science 80, 722–733 (2016)CrossRefGoogle Scholar
  26. 26.
    Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Multicloud deployment of computing clusters for loosely coupled mtc applications. IEEE transactions on parallel and distributed systems 22(6), 924–930 (2011)CrossRefGoogle Scholar
  27. 27.
    Katsaros, G., Menzel, M., Lenk, A.: Cloud Service Orchestration with TOSCA, Chef and Openstack. In: Ic2e (2014)Google Scholar
  28. 28.
    Garcia, A.L., Zangrando, L., Sgaravatto, M., Llorens, V., Vallero, S., Zaccolo, V., Bagnasco, S., Taneja, S., Dal Pra, S., Salomoni, D., Donvito, G.: Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers. J. Phys. Conf. Ser. 898(9), 92010 (2017)CrossRefGoogle Scholar
  29. 29.
    Singh, S., Chana, I.: A survey on resource scheduling in cloud computing issues and challenges. Journal of Grid Computing, pp. 1–48 (2016)Google Scholar
  30. 30.
    García, A.L., Castillo, E.F.-d., Fernández, P.O., Plasencia, I.C., de Lucas, J.M.: Resource provisioning in Science Clouds: Requirements and challenges. Software: Practice and Experience 48(3), 486–498 (2018)Google Scholar
  31. 31.
    Chauhan, M.A., Babar, M.A., Benatallah, B.: Architecting cloud-enabled systems: a systematic survey of challenges and solutions. Software - Practice and Experience 47(4), 599–644 (2017)Google Scholar
  32. 32.
    Somasundaram, T.S., Govindarajan, K.: CLOUDRB A Framework for scheduling and managing High-Performance Computing (HPC) applications in science cloud. Futur. Gener. Comput. Syst. 34, 47–65 (2014)CrossRefGoogle Scholar
  33. 33.
    Sotomayor, B., Keahey, K., Foster, I.: Overhead Matters: A Model for Virtual Resource Management. In: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing SE - VTDC ’06, p 5. IEEE Computer Society, Washington (2006)Google Scholar
  34. 34.
    SS, S.S., Shyam, G.K., Shyam, G.K.: Resource management for Infrastructure as a Service (IaaS) in cloud computing SS Manvi A survey. J. Netw. Comput. Appl. 41, 424–440 (2014)CrossRefGoogle Scholar
  35. 35.
  36. 36.
    Europen open science cloud: https://ec.europa.eu/research/openscience (2015)
  37. 37.
  38. 38.
  39. 39.
  40. 40.
    Pérez, A., Moltó, G., Caballer, M., Calatrava, A.: Serverless computing for container-based architectures Future Generation Computer Systems (2018)Google Scholar
  41. 41.
    de Vries, K.J.: Global fits of supersymmetric models after LHC run 1. Phd thesis Imperial College London (2015)Google Scholar
  42. 42.
    Openstack: https://www.openstack.org/ (2015)
  43. 43.
  44. 44.
  45. 45.
  46. 46.
    Opennebula: http://opennebula.org/ (2018)
  47. 47.
    Redhat openshift: http://www.opencityplatform.eu (2011)
  48. 48.
    The cloud foundry foundation: https://www.cloudfoundry.org/ (2015)
  49. 49.
    Caballer, M., Blanquer, I., Moltó, G., de Alfonso, C.: Dynamic management of virtual infrastructures. Journal of Grid Computing 13(1), 53–70 (2015)CrossRefGoogle Scholar
  50. 50.
  51. 51.
  52. 52.
    Opencitiy platform: http://www.opencityplatform.eu (2014)
  53. 53.
    Onedata: https://onedata.org/ (2018)
  54. 54.
  55. 55.
  56. 56.
    Fernández, P.O., García, A.L., Duma, D.C., Donvito, G., David, M., Gomes, J.: A set of common software quality assurance baseline criteria for research projects, see http://hdl.handle.net/10261/160086. Technical report
  57. 57.
    Httermann, M.: Devops for developers Apress (2012)Google Scholar
  58. 58.
    EOSC-Hub: ”Integrating and managing services for the European Open Science Cloud” Funded by H2020 research and innovation pr ogramme under grant agreement No. 777536. See http://eosc-hub.eu (2018)
  59. 59.
    Apache License: author = https://www.apache.org/licenses/LICENSE-2.0 (2004)
  60. 60.
    INDIGO Package Repo: http://repo.indigo-datacloud.eu/ (2017)
  61. 61.
  62. 62.
  63. 63.
    Van Zundert, G.C., Bonvin, A.M.: Disvis: quantifying and visualizing the accessible interaction space of distance restrained biomolecular complexes. Bioinformatics 31(19), 3222–3224 (2015)CrossRefGoogle Scholar
  64. 64.
    Van Zundert, G.C., Bonvin, A.M.: Fast and sensitive rigid–body fitting into cryo–em density maps with powerfit. AIMS Biophys. 2(0273), 73–87 (2015)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • D. Salomoni
    • 1
  • I. Campos
    • 2
  • L. Gaido
    • 3
  • J. Marco de Lucas
    • 2
  • P. Solagna
    • 14
  • J. Gomes
    • 9
  • L. Matyska
    • 15
  • P. Fuhrman
    • 5
  • M. Hardt
    • 8
  • G. Donvito
    • 4
  • L. Dutka
    • 13
  • M. Plociennik
    • 7
  • R. Barbera
    • 11
    • 21
  • I. Blanquer
    • 6
  • A. Ceccanti
    • 1
  • E. Cetinic
    • 18
  • M. David
    • 9
  • C. Duma
    • 1
  • A. López-García
    • 2
  • G. Moltó
    • 6
  • P. Orviz
    • 2
  • Z. Sustr
    • 15
  • M. Viljoen
    • 14
  • F. Aguilar
    • 2
  • L. Alves
    • 9
  • M. Antonacci
    • 4
  • L. A. Antonelli
    • 17
  • S. Bagnasco
    • 3
  • A. M. J. J. Bonvin
    • 16
  • R. Bruno
    • 11
  • Y. Chen
    • 14
  • A. Costa
    • 17
  • D. Davidovic
    • 18
  • B. Ertl
    • 8
  • M. Fargetta
    • 11
  • S. Fiore
    • 10
  • S. Gallozzi
    • 17
  • Z. Kurkcuoglu
    • 16
  • L. Lloret
    • 2
  • J. Martins
    • 9
  • A. Nuzzo
    • 10
  • P. Nassisi
    • 10
  • C. Palazzo
    • 10
  • J. Pina
    • 9
  • E. Sciacca
    • 17
  • D. Spiga
    • 12
  • M. Tangaro
    • 19
  • M. Urbaniak
    • 7
  • S. Vallero
    • 3
  • B. Wegh
    • 8
  • V. Zaccolo
    • 3
  • F. Zambelli
    • 19
    • 20
  • T. Zok
    • 7
  1. 1.INFN - CNAFBolognaItaly
  2. 2.IFCAConsejo Superior de Investigaciones Cientificas-CSICSantanderSpain
  3. 3.INFN - TorinoTorinoItaly
  4. 4.INFN - BariBariItaly
  5. 5.Deutsches Elektronen Synchrotron (DESY)HamburgGermany
  6. 6.Institute of Instrumentation for Molecular Imaging - Universitat Politècnica de ValènciaValenciaSpain
  7. 7.PSNC IBCh PASPoznańPoland
  8. 8.Karlsruhe Institute of Technology (KIT)KarlsruheGermany
  9. 9.Laboratory of Instrumentation and Experimental Particle Physics (LIP)LisbonPortugal
  10. 10.Fondazione Centro Euro-Mediterraneo sui Cambiamenti ClimaticiLecceItaly
  11. 11.INFN - CataniaCataniaItaly
  12. 12.INFN - PerugiaPerugiaItaly
  13. 13.Cyfronet AGHKrakowPoland
  14. 14.EGI FoundationAmsterdamNetherlands
  15. 15.CESNETPragueCzech Republic
  16. 16.University of UtrechtUtrechtThe Netherlands
  17. 17.Istituto Nazionale di AstrofisicaRomeItaly
  18. 18.Ruder Boskovic InstituteZagrebCroatia
  19. 19.Consiglio Nazionale delle RicercheIstituto di Biomembrane, Bioenergetica e Biotecnologie MolecolariBariItaly
  20. 20.Department of BiosciencesUniversity of MilanoMilanItaly
  21. 21.Department of Physics and AstronomyUniversity of CataniaCataniaItaly

Personalised recommendations