Advertisement

The NCI High Performance Computing and High Performance Data Platform to Support the Analysis of Petascale Environmental Data Collections

  • Ben Evans
  • Lesley Wyborn
  • Tim Pugh
  • Chris Allen
  • Joseph Antony
  • Kashif Gohar
  • David Porter
  • Jon Smillie
  • Claire Trenham
  • Jingbo Wang
  • Alex Ip
  • Gavin Bell
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 448)

Abstract

The National Computational Infrastructure (NCI) at the Australian National University (ANU) has co-located a priority set of over 10 PetaBytes (PBytes) of national data collections within a HPC research facility. The facility provides an integrated high-performance computational and storage platform, or a High Performance Data (HPD) platform, to serve and analyse the massive amounts of data across the spectrum of environmental collections – in particular from the climate, environmental and geoscientific domains. The data is managed in concert with the government agencies, major academic research communities and collaborating overseas organisations. By co-locating the vast data collections with high performance computing environments and harmonising these large valuable data assets, new opportunities have arisen for Data-Intensive interdisciplinary science at scales and resolutions not hitherto possible.

Keywords

high performance computing high performance data cloud Computting data-intensive science scalable data services data cube virtual laboratories 

References

  1. 1.
    Department of Education: National Collaboration Research Infrastructure Strategy, retrieved from https://education.gov.au/national-collaborative-research-infrastructure-strategy-ncrison (retrieve on November 27, 2014)
  2. 2.
    Department of Education: Information Sheet Research Data Storage Initiative (RDSI) Australian Government, http://docs.education.gov.au/documents/information-sheet-research-data-storage-initiative-rdsi (modified September 24, 2014)
  3. 3.
    Hey, T., Tansley, S., Tolle, K.: Jim Grey on eScience: a transformed scientific method. In: Hey, T., Tansley, S., Tolle, K. (eds.) The Fourth Paradigm: Data-Intensive Science Discovery, pp. xvii–xxxi. Microsoft Corporation, USA (2009)Google Scholar
  4. 4.
    National Computational Infrastructure: National Collections, retrieved from http://nci.org.au/data-collections/data-collections/, and the GeoNetwork catalogue http://nci.org.au/data-collections/data-collections/and (November 27, 2014)
  5. 5.
    Taylor, K.E., Stouffer, R.J., Meehl, G.A.: An Overview of CMIP5 and the experiment design. Bull. Amer. Meteor. Soc. 93, 485–498 (2012), doi:10.1175/BAMS-D-11-00094.1CrossRefGoogle Scholar
  6. 6.
    Japan Meteorological Agency: Himarwari-8/9 of the Meteorological Satellite Center (MSC) of the JMA, retrieved from http://www.data.jma.go.jp/mscweb/en/himawari89/index.html (November 27, 2014)
  7. 7.
    European Space Agency: The Copernicus programme, retrieved from http://www.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Overview4 (November 27, 2014)
  8. 8.
    Open Geospatial Consortium Web Processing Service, OGC, retrieved from http://www.opengeospatial.org/standards/wps (November 27, 2014)
  9. 9.
    OpenStack, retrieved from http://www.openstack.org/on (November 27, 2014)
  10. 10.
    Lustre file system, retrieved from http://wiki.lustre.org/index.php/Main_Page (November 27, 2014)
  11. 11.
    PuppetLabs Inc., retrieved from http://puppetlabs.com/ (November 27, 2014)
  12. 12.
    Git, retrieved from http://git-scm.com/ (November 27, 2014)
  13. 13.
    Docker, retrieved from https://www.docker.com/ (November 27, 2014)
  14. 14.
    Open Geospatial Consortium: Web Map Service, retrieved from http://www.opengeospatial.org/standards/wms (November 27, 2014)
  15. 15.
    Open Geospatial Consortium: Web Coverage Service, OGC, retrieved from http://www.opengeospatial.org/standards/wcs (November 27, 2014)
  16. 16.
    Open Geospatial Consortium: Web Feature Service, OGC, retrieved from http://www.opengeospatial.org/standards/wfs (November 27, 2014)
  17. 17.
    Open Source Geospatial Foundation:GeoNetwork, retrieved from http://geonetwork-opensource.org/ (modified January 31, 2014)
  18. 18.
    ElasticSearch, retrieved from http://www.elasticsearch.org/ (November 27, 2014)
  19. 19.
    THREDDS, retrieved from http://www.unidata.ucar.edu/software/thredds/current/tds/ (November 27, 2014)
  20. 20.
    OpenDAPInc: Hyrax, retrieved from http://docs.opendap.org/index.php/Hyrax (November 27, 2014)
  21. 21.
    Open Source Geospatial Foundation: GeoServer, retrieved from http://geoserver.org/ (November 27, 2014)
  22. 22.
    JSON, retrieved from http://www.json.org/ (November 27, 2014)
  23. 23.
    Australian Geoscience Data Cube (AGDC), retrieved from https://github.com/GeoscienceAustralia/agdc/wikion (November 27, 2014)
  24. 24.
    Earth Systems Grid Federation: About the Earth System Grid, retrieved, from https://www.earthsystemgrid.org/about/overview.htm (November 27, 2014)
  25. 25.
    Oliver, H.: The Cylc Suite Engine, retrieved from http://cylc.github.io/cylc/html/single/cug-html.html (November 27, 2014)
  26. 26.
    UK Met Office: Rose, retrieved from https://github.com/metomi/rose/ and http://www.metoffice.gov.uk/research/collaboration/rose (November 27, 2014)
  27. 27.
    Williams, D.N., et al.: UV-CDAT (2014), doi:10.5281/zenodo.12251, http://uvcdat.llnl.gov/
  28. 28.
    Oceans Data Interoperability Platform (ODIP), retrieved from http://www.odip.org/ (November 27, 2014)

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Ben Evans
    • 1
  • Lesley Wyborn
    • 1
  • Tim Pugh
    • 2
  • Chris Allen
    • 1
  • Joseph Antony
    • 1
  • Kashif Gohar
    • 1
  • David Porter
    • 1
  • Jon Smillie
    • 1
  • Claire Trenham
    • 1
  • Jingbo Wang
    • 1
  • Alex Ip
    • 3
  • Gavin Bell
    • 4
  1. 1.National Computational Infrastructure (NCI)Australian National UniversityCanberraAustralia
  2. 2.Bureau of MeteorologyMelbourneAustralia
  3. 3.Geoscience AustraliaCanberraAustralia
  4. 4.The 6th Column ProjectBerlinGermany

Personalised recommendations