Earth Science Informatics

, Volume 7, Issue 1, pp 1–12 | Cite as

Cloud computing and virtualization within the regional climate model and evaluation system

  • Chris A. Mattmann
  • Duane Waliser
  • Jinwon Kim
  • Cameron Goodale
  • Andrew Hart
  • Paul Ramirez
  • Dan Crichton
  • Paul Zimdars
  • Maziyar Boustani
  • Kyo Lee
  • Paul Loikith
  • Kim Whitehall
  • Chris Jack
  • Bruce Hewitson
Research Article

Abstract

The Regional Climate Model Evaluation System (RCMES) facilitates the rapid, flexible inclusion of NASA observations into climate model evaluations. RCMES provides two fundamental components. A database (RCMED) is a scalable point-oriented cloud database used to elastically store remote sensing observations and to make them available using a space time query interface. The analysis toolkit (RCMET) is a Python-based toolkit that can be delivered as a cloud virtual machine, or as an installer package deployed using Python Buildout to users in order to allow for temporal and spatial regridding, metrics calculation (RMSE, bias, PDFs, etc.) and end-user visualization. RCMET is available to users in an “offline”, lone scientist mode based on a virtual machine dynamically constructed with model outputs and observations to evaluate; or on an institution’s computational cluster seated close to the observations and model outputs. We have leveraged RCMES within the content of the Coordinated Regional Downscaling Experiment (CORDEX) project, working with the University of Cape Town and other institutions to compare the model output to NASA remote sensing data; in addition we are also working with the North American Regional Climate Change Assessment Program (NARCCAP). In this paper we explain the contribution of cloud computing to RCMES’s specifically describing studies of various cloud databases we evaluated for RCMED, and virtualization toolkits for RCMET, and their potential strengths in delivering user-created dynamic regional climate model evaluation virtual machines for our users.

Keywords

RCMES Regional Climate Modeling Apache OODT Hadoop Sqoop MongoDB Hive 

Notes

Acknowledgments

This work was conducted at the Jet Propulsion Laboratory, managed by the California Institute of Technology, for the National Aeronautics and Space Administration. The research reported on herein was sponsored by the NASA Advanced Information Systems (AIST) program (AIST-QRS-12-0002-T). Thanks are due to the RCMES team including Duane Waliser, Jinwon Kim, Cameron Goodale, Andrew Hart, Paul Ramirez, Paul Zimdars, Kim Whitehall, Jesslyn Whittell and Dan Crichton. The author also wishes to thank Michael Seablom for his support in this effort.

References

  1. Banker K (2011) MongoDB in Action. Manning Publications Co., Greenwich, CT, USAGoogle Scholar
  2. Capriolo E, Wampler D, Rutherglen J (2012) Programming hive. O’ReillyGoogle Scholar
  3. Cinquini L, Crichton D, Mattmann C, Harney J, Shipman G, Wang F, Ananthakrishnan R, Miller N, Denvil S, Morgan M, Pobre Z, Bell G, Drach B, Williams D, Kershaw P, Pascoe S, Gonzalez E, Fore S, Schweitzer R (2012) The Earth System Grid Federation: An Open Infrastructure for Access to Distributed Geospatial Data. In Proceedings of the 8th IEEE International Conference on eScience 2012, Chicago, IL, October 8–12Google Scholar
  4. Cornillon P, Gallagher J, Sgouros T (2003) OPeNDAP: Accessing data in a distributed, heterogeneous environment. Data Sci J 2:164–174CrossRefGoogle Scholar
  5. Crichton D, Mattmann C, Cinquini L, Braverman A, Waliser D, Hart A, Goodale C, Lean P (2012) Sharing Satellite Observations with the Climate Modeling Community: Software and Architecture. IEEE Software, Vol. 29, No. 5., September/October, pp 63–71Google Scholar
  6. Fielding RT, Taylor RN (2002) Principled design of the modern Web architecture. ACM Trans Internet Technol (TOIT) 2(2):115–150CrossRefGoogle Scholar
  7. B Fortner (1998) “HDF: The Hierarchical Data Format,” Dr Dobb’s J. Software Tools and Pro- fessional Programming, www.drdobbs.com/database/184410553
  8. Giorgi F, Jones C, Asrar G (2009) Addressing climate information needs at the regional level: the CORDEX framework. WMO Bulletin 58(3):175–183Google Scholar
  9. Hart A, Goodale C, Mattmann C, Zimdars P, Crichton D, Lean P, Kim J, Waliser D (2011) A Cloud-Enabled Regional Climate Model Evaluation System. In Proceedings of the ICSE 2011 Workshop on Software Engineering for Cloud Computing - SECLOUD, Honolulu, HI, May 22Google Scholar
  10. Hüttermann M (2012) Infrastructure as Code. In DevOps for Developers, Part IV. Springer, pp 135–156Google Scholar
  11. Mattmann C, Crichton D, Medvidovic N, Hughes S (2006) A Software Architecture-Based Framework for Highly Distributed and Data Intensive Scientific Applications. In Proceedings of the 28th International Conference on Software Engineering (ICSE06), Software Engineering Achievements Track, pp 721–730, Shanghai, China, May 20th-28thGoogle Scholar
  12. Mattmann C, Freeborn D, Crichton D, Foster B, Hart A, Woollard D, Hardman S, Ramirez P, Kelly S, Chang AY, Miller CE (2009) A reusable process control system framework for the orbiting carbon observatory and NPP sounder PEATE missions. In proceedings of the 3rd IEEE Intl Conference on Space Mission Challenges for Information Technology (SMC-IT 2009), pp 165–172, July 19–23Google Scholar
  13. Mearns L, Arritt R, Biner S, Bukovsky MS, McGinnis S, Sain S, Caya D, Correia J, Flory D, Gutowski W, Takle ES, Jones R, Leung R, M-Okia W, McDaniel L, Nunes AMB, Qian Y, Roads J, Sloan L, Synder M. The North American regional climate change assessment program: overview of phase I results. Bull Amer Meteor Soc 93: 1337–1362Google Scholar
  14. Obe R, Hsu L (2011) PostGIS in Action. Manning Publications Co., Greenwich, CT, USAGoogle Scholar
  15. Rew RK, Davis GP (1990) NetCDF: An Interface for Scientific Data Access. IEEE Comput Graph Appl 10(4):76–82CrossRefGoogle Scholar
  16. Rew RK et al (2007) “The CF Conventions: Gover- nance and Community Issues in Establishing Standards for Representing Climate, Forecast, and Observational Data,” slide presentation, Am. Geophysical Union Fall Meeting, abstract #IN52A-07Google Scholar
  17. T. White (2011) Hadoop: the Definitive Guide. 2nd Edition, O’ReillyGoogle Scholar
  18. Whitehall K, Mattmann C, Waliser D, Kim J, Goodale C, Hart A, Ramirez P, Zimdars P, Crichton D, Jenkins G, Jones C, Asrar G, Hewitson B (2012) Building model evaluation and decision support capacity for CORDEX. WMO Bulletin, Vol. 61, No. 2, pp 29–34, http://www.wmo.int/pages/publications/bulletin_en/61_2_cordex_en.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Chris A. Mattmann
    • 1
    • 2
    • 4
  • Duane Waliser
    • 1
    • 2
  • Jinwon Kim
    • 2
  • Cameron Goodale
    • 1
  • Andrew Hart
    • 1
  • Paul Ramirez
    • 1
  • Dan Crichton
    • 1
  • Paul Zimdars
    • 1
  • Maziyar Boustani
    • 1
  • Kyo Lee
    • 1
  • Paul Loikith
    • 1
  • Kim Whitehall
    • 1
    • 3
  • Chris Jack
    • 5
  • Bruce Hewitson
    • 5
  1. 1.Jet Propulsion Laboratory, California Institute of TechnologyPasadenaUSA
  2. 2.UCLA JIFRESSELos AngelesUSA
  3. 3.Howard UniversityWashingtonUSA
  4. 4.University of Southern CaliforniaLos AngelesUSA
  5. 5.University of Cape Town, South AfricaCape TownSouth Africa

Personalised recommendations