Effects of atmospheric dynamics and ocean resolution on bi-stability of the thermohaline circulation examined using the Grid ENabled Integrated Earth system modelling (GENIE) framework

Abstract

We have used the Grid ENabled Integrated Earth system modelling (GENIE) framework to undertake a systematic search for bi-stability of the ocean thermohaline circulation (THC) for different surface grids and resolutions of 3-D ocean (GOLDSTEIN) under a 3-D dynamical atmosphere model (IGCM). A total of 407,000 years were simulated over a three month period using Grid computing. We find bi-stability of the THC despite significant, quasi-periodic variability in its strength driven by variability in the dynamical atmosphere. The position and width of the hysteresis loop depends on the choice of surface grid (longitude-latitude or equal area), but is less sensitive to changes in ocean resolution. For the same ocean resolution, the region of bi-stability is broader with the IGCM than with a simple energy-moisture balance atmosphere model (EMBM). Feedbacks involving both ocean and atmospheric dynamics are found to promote THC bi-stability. THC switch-off leads to increased import of freshwater at the southern boundary of the Atlantic associated with meridional overturning circulation. This is counteracted by decreased freshwater import associated with gyre and diffusive transports. However, these are localised such that the density gradient between North and South is reduced tending to maintain the THC off state. THC switch-off can also generate net atmospheric freshwater input to the Atlantic that tends to maintain the off state. The ocean feedbacks are present in all resolutions, across most of the bi-stable region, whereas the atmosphere feedback is strongest in the longitude–latitude grid and around the transition where the THC off state is disappearing. Here the net oceanic freshwater import due to the overturning mode weakens, promoting THC switch-on, but the atmosphere counteracts this by increasing net freshwater input. This increases the extent of THC bi-stability in this version of the model.

Appendix: Grid computing performance

To obtain the results presented here we undertook a total of 319 GENIE-2 simulations totalling 407,000 model years and comprising 3,736 compute tasks, taking 46,992 CPU hours (some timings estimated). This was achieved in around 3 months, including time spent considering the initial results and designing the restart experiments. The throughput is equivalent to having about 20 single-processor experiments running continuously throughout the time, highlighting the power of the Grid to enable ensemble studies with Earth system models.

Typically, a 1,000 year simulation of GENIE-2 requires ≈4 days of continuous compute time on a desktop machine (P4 3.0 GHz). To perform large ensembles of such runs presents a challenge in the traditional academic environment where compute clusters are often configured to limit single CPU jobs to less than 4 days and an individual user is unlikely to be permitted exclusive use of the resource. Furthermore, the effort involved in configuring, managing and executing a study of this nature on a single cluster can present a significant overhead. The GENIE data management solution provided the means for the project to harness the shared pool of compute resource available to individual project members and undertake the study with relative ease.

To contribute computational resource to a model study defined in the database a user simply requires a single piece of information; the unique identifier of the experiment entity in the database. The identifier is typically distributed by the experiment coordinator to interested parties although a query on the database can also be performed to obtain this information. Armed with the identifier, the user instantiates a time-limited proxy certificate to authenticate their actions on the Grid. They then create a data structure describing the computational resource(s) that they wish to submit work units to. This descriptive metadata provides information including hostname, resource type (Globus, Condor) and the remote brokering system (Portable Batch System, Sun Grid Engine, Condor, etc.) which the system can then use to manage job submissions. With this information defined in the workspace the user simply invokes a single function call in order to contribute to the study. The scripted workflows of the GENIE Toolbox then perform queries on the database to find available work units and submit jobs to the specified resource. Post-processing scripts are also invoked on completed work units in order to archive output data to the database.

In total five client installations were used to progress the simulations to completion. Each client queried the shared database for work units and submitted those compute tasks to resource available to the user in each case (see Fig. 15). Each client was configured by the user to exploit both local and national resource available to them using their X.509 e-Science certificate. The resource used included five nodes of the UK National Grid Service (NGS), three institutional High Performance Computing (HPC) clusters and one large university Condor pool. The database system enabled users to specify how much work was submitted, for example, short 5–10 h jobs were typically submitted on Condor as the chances of an interruption increase with time (we cannot exploit Condor’s checkpointing facility because our pool consists of only Windows nodes). For the NGS and HPC clusters we were able to submit jobs that run close to the maximum allowable wall time of the scheduling system. Throughput is in principle only limited by the amount of available resource.

Fig. 15

Collaborative study mediated by the GENIE database. Client installations at member institutions contribute to an ensemble study by retrieving work from the database and either submitting it to a resource within the institutional administrative domain or to a resource on the computational Grid

The daily breakdown of resource usage is shown in Fig. 16. Periods of relative inactivity reflect development work on the system and transitions between experiments where the science team were analysing results from the first phase of experiments and configuring the second phase of the study. The computational throughput was not limited by the system and the rate of progression was determined by the amount of available work. The five clients functioning in tandem reached a sustained peak in excess of 90 model runs per day and each made a significant contribution to the studies. The breakdown of the number of jobs completed by each client installation (Fig. 17a) reflects the amount of resource available to the user on each of these clients. The client responsible for 50% of the work was the only submission node on the large Condor pool and was also used to submit jobs to the National Grid Service. The distribution of jobs across the computational resources (Fig. 17b) illustrates that the studies made even use of the available platforms. The jobs were well spread across the National Grid Service, the large Condor pool and the three dedicated clusters. The usage of different clusters broadly reflects the relative numbers of jobs a user could reasonably hope to run concurrently on each of these systems.

Fig. 16

Daily resource usage for the 12 ensemble studies

Fig. 17

Distributions of a client job submissions and b resource usage measured as total number of timesteps performed