Dependence of skill and spread of the ensemble forecasts on the type of perturbation and its relationship with long-term norms of precipitation and temperature

A new computing cluster has been operating since 2016 at the Institute of Meteorology and Water Management – National Research Institute. Increasing computing power enabled the implementation of ensemble prediction system forecasts in the operational mode and the use of a new computer for research purposes. As part of the priority project on “Study of Disturbances in the Representation of Modeling Uncertainty in Ensemble Development” and the earlier project entitled “COSMO Towards Ensemble in Km in Our Countries), implemented in the Working Group 7 (Predictability and Ensemble Methods) as part of the COSMO modeling consortium, specific studies were carried out to test ensemble forecasts. This research concerned the impact of variability of physical fields characterizing the soil surface (a selected parameter determining evaporation from the soil surface and soil surface temperature) using various methods of perturbation. Numerical experiments were completed for the warm period (from June to September) 2013.


Introduction
Under the current state of knowledge, it is not possible to prepare a perfect forecast of the state of the atmosphere. However, one can determine to what extent the forecast is reliable. Some of the tools used to determine the "confidence" of the forecast are ensemble forecasts-a set of forecasts, starting from the space of initial states, defined with accuracy within the measurement error. Since 2015, a new (at this time, actually) computing cluster has been operating in IMWM-NRI. Linux cluster "Grad" ("Hail" in Polish) consists of six management nodes, each with two eight-core CPUs and of 139 computation nodes, with two ten-cores CPUs each. The overall performance according to HP-Linpack test achieved 61 Tflops, i.e., approximately 25 times higher in comparison with the prior machine. This increased computing power has enabled the IMWM-NRI to introduce the EPS for operational forecasts since the end of 2016.There are various methods to obtain probabilistic information (e.g., a stochastic Bayesian model, Cano et al. 2004), but the simplest tool for this task is the EPS forecast-set of forecasts: (a) in which each member starts from a different initial state, defined with a change of initial/boundary condi-1 3 tions and/or change of the method of solving differential equations; (b) where each member encloses implemented a different parameterization of the physical processes occurring in the atmosphere (see, e.g., Stensrud 2007); (c) for each member of the ensemble, a different numerical scheme is used to solve differential equations that describes physical processes in the atmosphere.
An additional advantage of the EPS approach is the wellknown fact that the ensemble mean is closer to the observed, actual state of the atmosphere (see for instance Zhao et al. 2016 or Hong andBishop 2013) than any single deterministic forecast from this ensemble.

Ensemble prediction system at IMWM-NRI: idea and configuration
At IMWM-NRI in the basic setup, the COSMO model works in deterministic mode using the initial-(IC) and boundary conditions (BC) from the global ICON model (Zängl et al. 2015). The non-hydrostatic ICON model is being launched in Deutscher WetterDienst, using an icosahedral-hexagonal grid, with the highest spatial resolution in Europe of 13 km and a time of at least 78 h, four times a day. It generates the IC/BCs set for COSMO mesoscale models with a basic spatial resolution of 7 km and a domain covering central Europe with the time horizon of seventy eight hours. Subsequently, the COSMO model with a horizontal resolution of 7 km (COSMO-7 km) uses ICs and BCs together with data assimilation of the latest set of meteorological data obtained from the GTS/ WMO network. Eventually, forecast results from the COSMO-7 km model are used as IC/BCs for a nested instance of a COSMO model with a higher resolution of 2.8 km (convection-permitting scale) and 48-h forecasts. Operationally, the model runs four times a day, with a forecast time horizon of up to 48 h. The settings are the same as in the deterministic COSMO model with a 2.8 km grid (cf. Fig. 1, see also Duniec et al. 2017).
From general experience, ensemble predictions give significantly varied results. This diversity is defined with socalled spread (Eq. 1), described in general as a deviation of individual values from ensemble members forecasts from ensemble mean, and calculated-most commonly-as equal to a standard deviation over the ensemble. In turn, the quality of the forecast-defined as skill (Eq. 2)-is measured as the average absolute error of forecasts (ensemble means) vs. observations, done at sixty-one Polish SYNOP stations.
In many cases, perturbations are effective (notable) only after a few hours of starting (the so-called "spin-up effect"). For this reason, twenty ensemble members are functionally divided into four groups (see Fig. 2). The collection of the first five members begins 18 h before the start of the current EPS run (formally starting at 00:00 UTC). The next five members will start 12 h earlier, the next five-6 h earlier, and at the end of the last five members will start at 00:00 UTC. This configuration is called time-lagged ensemble, in combination with model-varied ensemble (TLE-MVE; Duniec et al. 2017) (Fig. 2).
In this paper, the ensemble was prepared in such a way that each member started from a different value of the initial state of c_soil-i.e., the parameter describing evaporation from soil-and/or the temperature of the soil surface (for both cf. Doms et al. 2011). The value of this c_soil parameter was perturbed using a new random number generator (RNG; Duniec et al. 2017). The following cases were considered: (a) c_soil-disturbance of the dimensionless parameter describing evaporation from soil, soil evaporation index (Doms et al. 2011); (b) laf_pert-perturbation of initial soil surface temperature values (laf-taken from analysis files-see Doms et al. 2011); (c) laf-c_soil-disturbance of the initial value of soil surface temperature simultaneously with evaporation from the soil surface, i.e., a combination of points (a) and (b).
The aim of the study was to examine the dependence of the spread and skill-see above-values of EPS forecasts on fields such as air temperature, dew point temperature and wind speed, soil surface temperature, soil moisture and soil type. Evaporation from the soil surface depends on the soil water content value, and the evaporation rate depends on the soil surface temperature. Because there were no data on soil water content and soil surface temperature values available, these data were obtained indirectly, using information about the average monthly rainfall and air temperature-assuming a linear relationships between these quantities. The uppermost group-members #01 to #05-is finalized and ready for post-processing 18 h before the nominal EPS start-up time in the current time window; the second groupmembers #06 to #10-are gathered 12 h before the current time win-dow; the third group-members #11 to #15-are prepared 6 h before the current time window; and finally the lowermost group-members no. #16 to #20-starts and is finalized in the current time window, completing the whole EPS forecast

Methods
A warm period (June-September) of 2013 was selected for numerical simulations. Research focused on ensemble forecasts of three meteorological fields: air temperature 2 m above ground level (agl., TE2M), dew point temperature 2 m agl. (TD2M) and wind speed at 10 m agl. (U10M). For each meteorological field, average monthly values of spread and skill were computed (see, e.g., Jolliffe and Stephenson 2012). In the next step, the average skill and spread values were compared with average air temperature values, deviation of the average monthly air temperature in a given month in relation to the long-term average from 1971 to 2000 (Bulletin 2013), the monthly sum of precipitation and its deviation from the long-term norm for 1971-2000(Bulletin 2013. The results were checked for correlation with average skill and spread values for a given soil type that occurred in the analyzed area. The authors wanted to answer the question what factor-apart from the method of perturbation-and to what extent it affected the values of spread and skill. For example, it is known that these values can be influenced by a synoptic situation, e.g., through its dynamics-or lack of it. Similarly, even in a relatively low-dynamic situation an atmosphere has a major impact on soil condition via air temperature or precipitation. Spatial distributions of values of spread and skill were compared with the spatial distribution of average monthly air temperature values and monthly precipitation, taking into account the obvious fact that rainfall had an impact on soil moisture, i.e., soil water contents (SWC). For this reason, an attempt was made to establish the relationship between air temperature and/or rainfall amount and values of skill and spread. It has to be underlined that this relationship should be understood as spatial, not temporal, dependence between the two considered fields.
Values of spread and skill were obtained for all (c_soil, laf_pert and laf-c_soil) evaluated perturbations. Assuming the ability to transfer of heat, water, etc., is related with the compactness of the soil, general perturbation scheme had the following form with random_number being from 0 to 1, obtained from pseudo-random number generator (machine-related). If the soil is more "loose," akin to sand, then amplitude of perturbation equals one, then, the more compact structure, like peat or clay, the lower the amplitude became. For example, for sand the amplitude was set to 1.0, for clay to 0.6 and for peat to 0.5. Thus, for sand the perturbation varied from 0.5 to 1.5 of original (unperturbed) value. (3)

Results
The analysis of the results was divided into three groups separately for each meteorological field. Detailed results-for all types of perturbation, the entire period of June to September 2013 and for major cities (regions) with SYNOP stations, are presented in Tables 1, 2, 3, 4, 5 and 6. SYNOP stations are located in Wrocław, Katowice, Kraków, Rzeszów (as representative ones for southern Poland), Warszawa, Łódź, Poznań (central Poland), Szczecin, Gdańsk, Olsztyn and Suwałki (northern Poland, respectively).

Results for dew point temperature at 2 m agl. (TD2M)
For c_soil perturbation in July, August and September, spread values are increasing with increase in air temperature (see, e.g., Fig. 5 for August). However in June, this relationship could not be clearly identified (Fig. 3). Analyzing the climatological maps of the distribution of the average temperature value and qualitatively comparing the relevant distributions, a certain relationship between the distribution of the spread value in Poland and the distribution of the average monthly temperature value in Poland can be seen.
In July, August and September in the southwestern area of the country, average temperature values were higher than in the rest of the country. In July, average temperature values in western part of Poland were 21 °C, and in eastern part-only 18 °C. In August in Warsaw or in Poznań, the average temperature value was 19 °C. For example, in northern Poland the average temperature value was 17 °C.
In September, again in western part of Poland, the average temperature was 12 °C while in central and eastern part-only 11 °C. A higher average spread value was also observed in the vicinity of the mentioned areas. For example, in August in Poznań there were the highest temperature and spread (with the value of 0.74 °C). In northern Poland, there was the lowest spread value 0.24 °C. In turn in Warsaw, at the highest observed air temperature the spread value was 0.34 °C.
In July, a similar tendency was observed, namely in Poznań the maximum temperature was 21 °C and the spread was also quite large-0.76 °C. In southwestern Poland, the average temperature was also about 21 °C and the spread was slightly smaller-around 0.5 °C. While in northern part of the country, the temperature was lower and equal to 18 °C and the spread was about 0.4 °C. Analyzing, as before, the spatial tendency of the spread and the average temperature value, some correlation of their values can be seen. In areas with higher values of average monthly temperature, an increased spread was observed, with an analogous stipulation as to the deviation from the above conclusions.       In September, the situation was less clear to describe. This is, of course, related to the reduced dynamics of physical processes occurring in the soil. A lower amount of heat reaching the surface, and lower intensity of convective phenomena imply also that intensity of precipitation will be lower. The variation in the average temperature value in the country was small, up to 2 °C (excluding mountainous regions). In southern Poland, the average temperature was around 12 °C, while the spread was 0.5 °C. In contrary, in the north the average temperature was 13 °C with an increased spread value of 0.4 to 0.5 °C. A different tendency was observed in southern Poland, where despite the higher monthly temperature value (12 °C), the spread was lower, in the range of 0.3 °C.
In September, there was no clear relationship between temperature and TD2M spread. Only locally there some connection could be found; on the other hand, there were areas where this relationship was not fulfilled.
In June, as in September, the temperature differentiation is smaller. The highest values are observed in the eastern part of the country. In Poznań and Białystok, the average temperature was at 17 °C, while TD2M spread was at 0.8 °C.
In this case, the comparative analysis of the spatial distribution of temperature and TD2M spread allows for some conclusion-namely that TD2M spread decreases with the increase in temperature.
Comparative analysis of the spatial distribution of temperature and skill values showed the similar relationship in June, (yet, it subjected to local deviations). In central and northeastern Poland, the average temperature was roughly 17 °C and the skill value was at the level of 1.6-2.1 °C. However, there were some stations (e.g., Radom), where the temperature was higher (18 °C) and the skill value was lower (1.2 °C).
On the contrary, in July in the western part of the country this relationship was reversed. In Poznań, the average temperature was about 21 °C; similar thermal conditions prevailed in Wrocław. In these regions, the skill values were 2.4 °C for Poznań and 1.8 °C for Wrocław. For example, in northern Poland the average temperature was from 18 °C to 20 °C and the skill value reached from 1.7 to 2.4 °C.
In August, the average air temperature in the southwest, locally in the south and center of the country temperatures, was higher than in the rest. For example, in central and southern Poland the average air temperature was of 19 °C, while in the north the values were lower. As far as the skill values of TD2M were distributed as follows: in Poznań-2.6 °C, in Wrocław, Kraków-2.1 °C, while in Warsaw, despite the higher temperature, the skill value was lower (about 1.7 °C). Over the coastline (northern Poland), the skill value was from 1.3 °C to 1.7 °C. Analyzing the spatial distribution of average temperature and skill values, it seemed that both quantities were related.
In September, the highest values of average monthly temperature occurred in the western part of the country, locally in the center and in the south of Poland. In southern part of the country, the temperature values were around 13 °C, in central Poland-12 °C, in northeastern Poland 11 °C. The values of skill were from 1.2 °C to 1.8 °C, respectively. From the comparative analysis of the spatial distribution of average temperature and skill values, however, no relation cannot be clearly detected.
Skill values increased with increase in air temperature in July and August, while the inverse correlation was observed in June. In September, it was difficult to clearly determine the relationship between the skill value and air temperature.
July was a month characterized by heavy rainfall (see Fig. 4)-60-80 mm of rainfall was recorded in central and western Poland, and even above 100 mm in mountain areas. In regions of large rainfall (central Poland), the value of spread of TD2M was about from 0.4 °C (eastern and central Poland) to 0.8 °C (western Poland). For the areas with the low precipitation (in the northern part of Poland), the spread value was from 0.5 °C (northwest) to 0.8 °C (northeast). It is not possible, however, to determine an unambiguous relationship between the spread value for TD2M spread value and the precipitation amount by analyzing the spatial distribution of the spread and comparing with the results with spatial distribution of precipitation amount.
The values of skill in the area with higher precipitation amount were from 1.6 °C to 2.1 °C. On the opposite, with the lower precipitation the skill values were from only 1.2 °C to 2.1 °C. A comparative analysis of the spatial distribution of monthly precipitation and the average skill value does not allow for explicit conclusions. In the south-eastern area (and in the northwestern part) of Poland, some relationship could be observed, while in the southwest, as in the northeast, one can draw a conclusion about reversing this relationship.
An increased amount of rainfall was also observed in July. Spatial analysis allowed to observe the relationship between the spread value and the amount of precipitation. There is also such a relation between the skill value and the amount of precipitation, while in the northern part of the country this relation seems to be reversed (i.e., spread decreases as precipitation increases). It should also be remembered that there may be local deviations from the indicated dependencies.
In August, places where monthly precipitation values were high were also observed. The analysis, however, did not allow to clearly indicate the relationship. Both in areas with a high or small amount of precipitation, spread values are large and small. The analysis of the spatial distribution of precipitation and the spatial distribution of the spread value allowed to state a relationship between the values of precipitation field and the spread value. Similarly, the comparative analysis of the spatial distribution of the precipitation field and the skill value indicates an analogous relation, except for the north of the country, where this relationship seems to be reversed.
In turn, a comparative analysis of the spatial distribution of precipitation and the spread value for September indicated relationship (again, with the increase in precipitation, spread grows) except for northeastern part of Poland. However, it is not possible to clearly describe the relationship between the spatial distribution of precipitation and values of skill.
To sum up, it should be stated that, basically, skill values increased with increase in air temperature in July and August, while the reversed relation was observed in June. In September, it was difficult to clearly determine the relationship between the skill value and air temperature. The relationship between the spread values and the amount of precipitation was determined for some regions and/or months. For example, no connection of this kind could be established in June. In July, the relationship between the spread value and the amount of precipitation strongly depended on the area. In southwestern Poland, the spread increased with increase in rainfall, while in northern Poland-decreased (Fig. 4).
In August (Fig. 5) and September, the spread value increases with increase in rainfall, while in September in the north of the country the spread value decreased with increase in rainfall.
As far the skill is concerned, in June and August it was not possible to clearly determine any dependence. Skill increased with increase in rainfall in July and August (Fig. 6), except in the northern areas of the country. In the northern part of the country, this relationship seems to be reversed.
The dependence of the skill value on the air temperature was the same as in the case of perturbation of the c_soil parameter.
In June and September, there was no clear correlation between values of skill and the amount of precipitation, although there was some relation in July and August (except for the northern areas of the country). The relation between the values of spread with the amount of precipitation was more complicated. In July, it was the spread decreased with increase in rainfall, (see Fig. 7) in contrast to August (where it increased, except for the northern areas on the Baltic coast). In June and September, the situation was more complicated and difficult to assess it unambiguously.
An equivalent comparative analysis was also carried out for other meteorological fields, namely air temperature at 2 m agl. and wind speed at 10 m agl.; however, only final conclusions are included in the paper.

Results for air temperature forecasts 2 m agl. (TE2M)
In the case of perturbation of the c_soil parameter, it was found that in July and September the spread values are qualitatively related to air temperature (Figs. 8 and 9), i.e., the spread usually increases as the temperature increases (except for a small area in the northeast of Poland). In June and August, no clear dependence could be established. In August, the spread depended directly on the temperature in the southern regions of the country, while in the northern regions the relationship was reversed. In July, August and September, the skill value increased with the increase in air temperature. A reversed relationship was observed in June.
The average values of spread increased with the increase in the amount of precipitation throughout the entire analyzed period except for the northern area of the country in the Baltic Sea coast region. In June and July, there was no clear relationship to identify between average values of skill and precipitation. In August, skill increased with increase in precipitation amount (again, except for the northern part of Poland, near the Baltic coast), while in September the relationship-even qualitative one-between the amount of rainfall and the average spread value varies depending on the region.
In the case of laf_pert perturbation, in June and July a direct connection could be found, i.e., spread values increased with the increase in the average temperature. In August and September, this dependency was reversed. In the case of skill, the reversed dependency was observed only in June. The spread values increased with the amount of rainfall in June and August. In August, the relation was more complex-difficult to clearly assess in the south, direct relationship in the east, reversed one in the north. This reversed link also occurred in July. In September, again, it was difficult to clearly identify any relationship.
Correlations between the average skill values and the monthly amount of precipitation could be clearly established only in August, when the skill increased with the amount of precipitation in the southern regions of the country (in the north the relation was reversed; see Fig. 10).

Results for wind speed at 10 m agl. (U10M)
In the case of c_soil perturbation, it was difficult to establish a clear correlation between values of both spread and skill and air temperature or rainfall amount.
In the case of laf_pert perturbation, some correlation was found between skill values and average air temperature. In June in the eastern part of Poland (Fig. 11), the skill increased with rise in air temperature, while in the western part of the country the opposite relation was observed. And vice versa, in July, the skill increased with rise in air temperature in western Poland, while the relationship was reversed in eastern part. The connection between spread values and air temperature was more complicated. In June and July, no clear correlation was found between the various distributions and perturbation options. In August, spread values 1 3   dropped with increase in air temperature. Again, no explicit relationship was found in September. In addition, it was not possible to determine any correlation between skill value and precipitation amount every month. In August (Fig. 12), skill increased with the precipitation amount, while in September, in the northeastern part of Poland, the spread value was the highest in areas with higher rainfall. In the south of the country, on the other hand, the relation was reversed.

Results obtained for combined c_soil-laf_pert perturbation
Analysis of the results from the model showed that the spatial distribution of spread and skill values was similar to the results obtained for the perturbation of the c_soil parameter. The spread values obtained for laf_pert perturbation were slightly higher in comparison with the spread values obtained with c_soil and laf-c_soil perturbations. Therefore, the conclusions are similar for all analyzed meteorological fields.

Discussion
It can be concluded from the comparative analysis that the value of the T2M spread is directly proportional to the climatological water balance (CWB; the difference between precipitation depth and the depth of potential evapotranspiration at a given site), of course with regional deviations from the rule. The correlation obtained is observed regardless of the type of perturbation. A similar correlation was obtained for the T2M skill value and the CWB value. For some regions, certain variations from the rule could be seen.
The comparative analysis showed that, in general, spread of TD2M gets bigger with the increase in the CWB value. This relationship prevailed (with the same exception for certain areas). A similar correlation was obtained for values of the TD2M skill and the CWB. Obtained dependencies were independent of the type of perturbation.
Comparing the spatial distribution of the spread/skill value for U10M with the spatial distribution of CWB, it could be seen that the direct proportionality between the values was most common, while the correlation between the values of spread and CWB showed diversity from the scheme. Yet, the most frequent relation between these two values was that spread/skill grows together with growing CWB (this dependence, of course, subjected to regional variations). Conclusions for spread/skill obtained for U10M were independent of the type of perturbation.
The analyzed data showed no correlation between values of skill/spread and the type of soil. In general, it can be assumed that skill/spread values are related to air temperature values. If the air temperatures are higher, the evaporation from the soil is higher and the soil water content decreases faster, so the spread values are bigger. The forecast is more sensitive to initial conditions and to changes in initial conditions related to soil moisture. Similar conclusions can be drawn for the relationship between values of spread/skill and soil water content. It should be emphasized that there have been some local differences since the above assumption. Comparing the spatial distribution of the temperature field and the precipitation field with the spatial distribution of the spread and skill values, one can notice a tendency of proportionality between the observed fields and values of the skill or spread. However, there are also differences and deviations. For example, in June for c_soil and laf_pert perturbation, the value of skill for T2M decreases as the air temperature increases. Similarly, in August and September, for the laf_pert perturbation, the spread value for T2M increases with decrease in air temperature. Similar situation occurred for TD2M spread obtained as a result of laf_pert perturbation in June and September. Comparable differences were observed for the dependence of the spread/skill on the precipitation amount. In August, for c_soil, spread of TD2M decreased with increase in amount of soil hydration. A very alike situation occurred in July and September, but only for the northern part of the country. In July, for the laf_pert perturbation, there is also a deviation from this relationship between the T2M spread value and soil moisture.
Certainly, the skill and spread values are influenced by the type of perturbation of initial condition, as indicated by the results of numerical experiments. Since the amplitude of the perturbation was related to the type of soil, it should somehow manifest itself in the results. However, other factors, including the statistical and stochastic nature of EPS calculations, may somewhat mask this effect. Of course, these conclusions should be treated as preliminary results. To be absolutely sure, it would be necessary to conduct local experiments, measuring in detail the temperature and humidity profiles (in the area where a particular soil type exists) and compare the results with the spread and skill values of specific meteorological fields for this particular region.
In certain months, it was not possible to establish any significant relationship between spread/skill values and the analyzed meteorological fields. For example, in June or August it was not possible to determine the correlation between spread of TE2M and air temperature for c_soil perturbation, similarly-in September for the correlation between skill of TD2M and air temperature for laf_pert perturbation. In June and July, it was also not possible to establish the relationship between the skill of TE2M and the amount of precipitation (i.e., higher soil water content) for c_soil perturbation.

Conclusions
In summary, the aim of this work was to find a relationship between the value of spread and skill, and the content of water in the soil, soil temperature and soil type. The study could not show the relationship between soil type and spread and skill values. No direct (nor sensible) impact of soil type on values of spread and skill was found. Preliminary analyses confirmed the existence of an empirical relationship between air temperature (which affects the rate of evaporation and eventually the profile of soil water content) and the value of skill and spread. Similar conclusions can be drawn for the relation between spread and skill values and soil moisture. These conclusions, assessed for warm period of one year, will be verified in subsequent numerical experiments which will cover a interval of 5 years taking into account the greater variety of synoptic situations. With the incoming further studies, authors plan to utilize other archive measurement data (if available) on soil temperature and humidity (water contents). Since this paper pertainedto some extent-to the climatology of EPS, future study in author's intentions would concern the daily variability of spread/skill values depending on specific meteorological parameters, similarly, temperature or precipitation height.