Scientific Computing and Big Data Analytics: Application in Climate Science

Abstract

The nature of scientific and engineering problems of real world is often very complex. These problems are intrinsically multi-dimensional, multivariate, nonlinear and non-stationary in their dynamics. Solutions to these problems often necessitate the use of complex mathematical modeling, simulation and analysis which are traditionally achieved by the use of expensive high performance computing, more commonly known as Super-Computers. In this chapter we discuss the nature of problems involved in Scientific Computations, a type of use cases under the genre of Big Data Analytics, that require Super-Computers to solve, and further discuss the possibilities of addressing the same using Cloud based Distributed Computing technologies. We discuss the same using the example of Climate Analytics, which represents typical challenges of Scientific Computing to a considerable extent. In particular, we delve into the details of how significantly large-sized data from the output of a complex fluid dynamics based Earth’s Climate Model can be processed using Distributed Technology framework, Spark, in an integrated manner with the final analytics results accessed by an web application for the end users.