Evaluation of five high-resolution global model rainfall forecasts over India during monsoon 2020

Abstract

This study aims to evaluate the performance of five global medium-range operational NWP model rainfall forecasts, namely NCUM, UKMO, IMD GFS, NCEP GFS and ECMWF to provide an intercomparison of rainfall forecasts over India in terms of skill in predicting daily rainfall (24-hr accumulated rainfall). Verification and intercomparison of rainfall forecasts over India during monsoon 2020 (JJAS) are carried out using both (i) standard traditional verification methods (POD, FAR, RMSE, etc.) and (ii) advanced spatial verification methods (MODE, FSS). The evaluation also includes assessment of large-scale mean patterns, temporal evolution of spells during the season, dominant modes using spectral analysis, basin-scale rainfall time series and isolated heavy rainfall cases. Our analysis suggests that some of the key large-scale aspects of monsoon (seasonal mean, active/break spells, and northward propagation) are realistically represented in all the models, with slight discrepancies. In addition, the spectral analysis of rainfall is in association with observed rainfall in Day-1 forecast and deteriorates with lead times. Synoptic variance in NCUM on longer leading times is closer to observations. While the standard categorical verification over India as a whole (spatial averaged) suggests that ECMWF forecast skill is relatively high among the five models, the verification over the sub-regions shows mixed results with no clear unique higher performer among the models. In addition, basin-scale verification of rainfall forecasts for five rivers over the Indian subcontinent shows a fairly good amount of skill in terms of CC and RMSE up to Day-3 with comparable scores among the models. The advanced spatial verification metrics, like MODE and FSS, applied to the models show varying skills with different attributes. However, for FSS, forecast skill was high (low) for lower (higher) rainfall thresholds of 20 mm/day (100 mm/day). Though different models with different spatial resolutions show reasonable skill scores for larger regions, for high-impact heavy rainfall events, which are generally localised, the models have very comparable poor skill with no clear edge by a model among the five models.

Appendix

1.1 A1. Method of object-based diagnostics evaluation (MODE)

Object identification forms the first step in MODE analysis. In this step, objects are identified in the accumulated rainfall fields (observed and model forecast). The identification of objects is done by using a smoothing operator, ‘convolution operator’. This is governed by a convolution radius (CR) and a threshold (CT) on the intensity of the field. CR is expressed in terms of grid units and CT in terms of rainfall threshold. This step is necessary (a) to make areas more contiguous and (b) to filter out small/insignificant precipitation amounts. Thus, this step effectively selects the portion of the field that is of greatest interest to the user. There is no universally optimal choice for these parameters (CR & CT). Minimal smoothing and a very low threshold will result in a large number of objects, many of them small. Heavy smoothing and a high threshold will result in very few intense rain areas (Davis et al. 2009). In this study, R is chosen as 2 grid spacing (~20 km) as minimum spatial scale of interest. The observed and forecast rainfall data both are at high grid resolution of 0.1°×0.1°; CR = 2 is considered suitable for smoothing out tiny isolated thunderstorms. CT values indicating rainfall intensity of interest are carefully chosen at 20, 40, 60, 80, 100, and 120 mm to cover low medium and heavy rainfall events. During the monsoon, heavy rains (>80 mm/day) are common over different parts of India.

1.2 A2. Fractions skill score (FSS)

Fractions skill score (FSS) belongs to the category of fuzzy or neighborhood spatial verification methods. Like the spatial verification methods, these also do not look for a grid-to-grid match but relax the criteria by verifying forecasts in the local neighbourhood of the observations. FSS is a scale-selective method that can provide a measure of how the forecast skill varies with the chosen spatial scale; in other words, it can help in determining the scales at which the forecasts become useful. It is usually used for the verification of rainfall forecasts (Roberts and Lean 2008; Roberts 2008). A brief description of the methodology is presented here:

Step 1: The forecast probabilities and the corresponding binary observations are generated for a suitable threshold and these are used to generate fractions.

Step 2: For every grid point which has a forecast associated with it, a fraction of surrounding points is computed within a given square of length ‘n’ that has a value of 1 (i.e., where the forecasts from ensemble members have exceeded the threshold). This is presented in the equations below:

$$ O\left( n \right)\left( {i,j} \right) = \frac{1}{{n^{2} }}\mathop \sum \limits_{k = 1}^{n} \mathop \sum \limits_{l = 1}^{n} I_{0} \left[ {i + k - 1 - \frac{{\left( {n - 1} \right)}}{2},j + l - 1 - \frac{{\left( {n - 1} \right)}}{2}} \right], $$

(A1)

$$ M\left( n \right)\left( {i,j} \right) = \frac{1}{{n^{2} }}\mathop \sum \limits_{k = 1}^{n} \mathop \sum \limits_{l = 1}^{n} I_{M} \left[ {i + k - 1 - \frac{{\left( {n - 1} \right)}}{2},j + l - 1 - \frac{{\left( {n - 1} \right)}}{2}} \right] ,\quad (I = 1, \ldots , N_{x} {\text{ and }} j = 1, \ldots , N_{y} ),$$

(A2)

where O(n)(i, j) is the resultant field of observed fractions for a square of length ‘n’ obtained from the field of binary observation I_o and M(n)(i, j) is the resultant field of forecast fractions obtained from binary field I_M. i and j represent the number of columns and rows in the domain, respectively. Fractions are generated for different spatial scales by changing the value of ‘n’, which can be an odd number to a maximum of 2N – 1, where N is the number of points along the longest side of the domain.

Step 3: The FSS is then calculated as a variation on the Brier score (Brier 1950):

$$ {\text{FSS}} = 1 - \frac{{{\text{FBS}}}}{{{\text{FBS}}_{{{\text{worst}}}} }} ,$$

(A3)

where FBS is the Fractions Brier score and is calculated as follows:

$$ {\text{FBS}} = \frac{1}{N}\mathop \sum \limits_{j = 1}^{N} \left( {O_{j} - M_{j} } \right)^{2}, $$

(A4)

where M_j and O_j are the forecast and observed fractions, respectively, at each point j and have values between 0 and 1, and N is the number of pixels in the verification area.

FBS_worst is given by:

$$ {\text{FBS}}_{{{\text{worst}}}} = \frac{1}{N}\left[ {\mathop \sum \limits_{j = 1}^{N} O_{j}^{2} + \mathop \sum \limits_{j = 1}^{N} M_{j}^{2} } \right] .$$

(A5)

It represents the largest value that can be obtained from the forecast and observed fractions when there is no closeness between the non-zero fractions.

The FSS values range from 0 (for the worst forecast) to 1 for the perfect forecast. FSS has the lowest value when forecasts are verified at the grid-scale only, i.e., the neighbourhood has only a single point. As the size of the neighbourhood increases, the skill also increases until it assumes an asymptotic value (AFSS) when the value of ‘n’ becomes its highest at 2N – 1.