Development of an ensemble data assimilation system with LMDZ5 AGCM for regional reanalysis

Abstract

The present study describes a newly developed LMDZ–DART data assimilation system based on the Ensemble Kalman Filter (EnKF) with a stretched grid atmospheric model and evaluates the potential of the system for generating high-resolution reanalysis products over the Indian region. This system is composed using the LMDZ5 (Laboratoire de Meteorologie Dynamique, Zoom, version 5) global atmospheric model, the Data Assimilation Research Testbed (DART), and a set of interface routines. The interface routines enable assimilation of observations with the model using the EnKF assimilation method. The stretched grid capability of the LMDZ5 model has been utilized to generate computationally efficient high-resolution experimental reanalysis data over the Indian region. The assimilation experiments have been conducted for 4 months during the Indian summer monsoon (ISM) period using LMDZ–DART system with two different configurations: REG (1° × 1° regular grid size over the globe) and ZOOM (~ 0.35° × 35° over the Indian region and coarser resolution outside). The results of the assimilation experiments are validated by comparing with observations as well as with an independent atmospheric reanalysis (ERA-Interim). It has been found that LMDZ–DART system is robust and reliable throughout the investigated ISM period for both REG and ZOOM configurations. In the REG experiment, spatial patterns and magnitude of the global fields portray a high degree of resemblance with ERA-Interim. The spatial patterns are also well reproduced in the ZOOM reanalysis. However, at some places outside the zoom area, the differences of the ZOOM reanalysis with ERA-Interim are slightly higher than the regular grid. Over the Indian region, the ZOOM configuration provides higher quality analysis and better regional features than REG. These results thus provide confidence in the stretched grid LMDZ–DART system to serve as a basis for generating computationally economical high-resolution regional reanalysis.

Appendix

1.1 A1. Changes made in LMDZ5 source codes

Usually, the implementation of EnKF data assimilation does not require any change in model source codes. However, the LMDZ GCM, which is created for climate purposes are now converted to prediction models with data assimilation, requires some modification in the source codes to run the model in the assimilation framework. Documenting these modifications here will hopefully make it easier for others to develop assimilation systems in similar models or other LMDZ implementations in the future. Following are the changes made in LMDZ’s source codes:

• 1. LMDZ’s source code flags allow a model restart time of at least 24 h. It prevents the use of LMDZ for sub-daily (e.g., 3-h or 6-h) assimilation cycle. Therefore, some routines of LMDZ have been modified to run it with user-specified sub-daily restart cycles. The changes have been made in both parallel and serial versions of LMDZ codes.

• 2. LMDZ is formulated in terms of covariant/contravariant wind vector and potential temperature; and, the same variables are written to the model restart file. The restart file is an input to LMDZ-DART interface codes where variables get updated by assimilation of observations, and then the updated variables are written in the same restart files for a subsequent model simulation run. The current version of the interface has been developed to assimilate natural wind and temperature model variables instead of covariant/contravariant wind and potential temperature. Although there can be a forward operator, like for the refractivity, which can allow the inclusion of covariant/contravariant wind and potential temperature variables in the assimilation process. However the current version of the interface does not have such forward operator for the same and, therefore, it is necessary to convert the model restart variables to the natural wind and temperature at each assimilation/forecast cycle and then updated variables are converted back to the covariant wind and potential temperature for next LMDZ model run. To accomplish this task, a conversion code has been added to the model source codes (both serial and parallel).

• 3. In a cold start simulation, the LMDZ model reads column air mass from an input data file and it is recomputed inside the model using surface pressure after each step of time integration. Specifically, column air mass computation requires vertical pressure levels, which are constructed using surface pressure. Inside the model, the air mass is computed only after the start of time integration steps. However, surface pressure is a component to the state variables which is updated after each assimilation cycle and the same is written in the model restart input file. After the assimilation step, the forecast step begins, and the model simply reads the column air mass, which is not updated by assimilation, and updated surface pressure along with other variables from restart file and recompute the air mass only after the start of time integration steps. Therefore after each assimilation cycle, it is necessary to recompute the air mass inside the model from updated/analysis surface pressure before the start of the time integrations. It rebalances the vertical structure of each column of the model due to a change in the surface pressure after assimilation. This is important because it was found that without this, model surface pressure started decreasing after each assimilation cycle. Surface pressure decrease was visible after a few months of assimilation period (not shown).

1.2 A2. LMDZ-DART parallel execution

The data assimilation process involves running multiple copies of LMDZ5 model simulations coupled with computations in the DART assimilation module. Indeed, there are many possible execution options to run LMDZ-DART; e.g., performing DART parallel computations (i.e., using multiple processors) but execute each model ensemble simulations sequentially (i.e., using a single processor); or, run both DART and all LMDZ5 ensemble members simultaneously using parallel computations. Given the grid dimension 360 × 180 × 29 (longitude, latitude, level) of LMDZ5, it is a very challenging task to run each member of the ensemble sequentially. Hence both DART and LMDZ5 are executed using a parallel process. However, this choice of parallelism at the model execution level demands several processors and massive memory to run all ensemble members simultaneously. It requires some node management on the HPC (High Performance Computing) server to achieve the required memory allocation and processing of each member of the ensemble. To perform the experiments on HPC, firstly, a total of 60 nodes have been exclusively allocated in an MPI (Message Passing Interface) run script, and then an assimilation executable (“filter”) is divided into 60 MPI tasks (see Fig. 3). Here total numbers of allocated nodes are equal to the number of ensemble members. Now, each MPI task of the “filter” is distributed to an allocated node to perform assimilation by the DART module. On completion of the ensemble assimilation, each “filter” MPI task calls LMDZ5 separately to perform the next 6-h ensemble forecast. This mechanism allows to execute LMDZ5 for each member using multiple processors simultaneously on their assigned node where it gets enough span for required memory and processors. Here shared memory parallelization has been used to perform independent model simulation at their assigned node for each member. In this study, all assimilation experiments have been performed at the HPC Facility (http://supercomputing.iitd.ac.in/) of Indian Institute of Technology Delhi, New Delhi where each node has 24 processors and 64 GB of memory.