# The relationship among probabilistic, deterministic and potential skills in predicting the ENSO for the past 161 years

## Abstract

Here, we explored in depth the relationship among the deterministic prediction skill, the probabilistic prediction skill and the potential predictability. This was achieved by theoretical analyses and, in particular, by an analysis of long-term ensemble ENSO hindcast over 161 years from 1856 to 2016. First, a nonlinear monotonic relationship between the deterministic prediction skill and the probabilistic prediction skill, derived by theoretical analysis, was examined and validated using the ensemble hindcast. Further, the co-variability between the potential predictability and the deterministic prediction skill was explored in both perfect model assumption and actual model scenario. On these bases, we investigated the relationship between the potential predictability and probabilistic prediction skill from both the practice of ENSO forecast and theoretical perspective. The results of the study indicate that there are nonlinear monotonic relationships among these three kinds of measures. The potential predictability is considered to be a good indicator for the actual prediction skill in terms of both the deterministic measures and the probabilistic framework. The relationships identified here exhibit considerable significant practical sense to conduct predictability researches, which provide an inexpensive and moderate approach for inquiring prediction uncertainties without the requirement of costly ensemble experiments.

## Keywords

ENSO ensemble forecast Predictability Nonlinear monotonic relationship## 1 Introduction

Usually, it involves two branches in the field of predictability researches on the El Niño-Southern Oscillation (ENSO) prediction. The first one is to investigate the ENSO potential predictability, which will reveal what the upper limit of the prediction skill might be and how much room would leave for the improvement of ENSO prediction systems (Tang et al. 2005, 2008; Cheng et al. 2010a; Kumar and Hu 2014; Kumar and Chen 2015). The common measures of the potential predictability can be categorized into variance-based metric and information-based metric, both without using the observations. Specifically, the signal-to-total variance ratio (STR) and signal-to-noise ratio (SNR) are two variance-based measures that are extensively employed (Kumar and Hoerling 2000; Peng et al. 2011; Hu and Huang 2012; Kumar et al. 2016), whereas several commonly used information-based measures include the predictive power (PP), predictive information (PI), relative entropy (RE), and mutual information (MI) (Tang et al. 2008). The studies that use these methods have revealed that the ENSO predictability is mainly dominated by signal component (Tang et al. 2008; Cheng et al. 2011; Kumar and Hu 2014).

The second one is to improve the actual prediction skill through the model development (Zebiak and Cane 1987; Tang and Hsieh 2002; Zhang et al. 2013), data assimilation (Chen et al. 2004; Zheng et al. 2007; Deng et al. 2010; Zheng and Zhu 2010; Zhu et al. 2014; Tang et al. 2016), and ensemble prediction etc. (Tang et al. 2006, 2008; Zheng et al. 2009a; Cheng et al. 2010a; Hou et al. 2018). Many efforts have been paid and significant progresses have been made toward this goal in last decades, as summarized in a recent review paper by Tang et al. (2018). Typically, the actual prediction skill can be measured in a deterministic manner and a probabilistic way. The former is widely measured using ensemble mean that is able to filter out the unpredictable feature and provide a nearly unbiased estimate for the future state of the climate element, whereas the latter intends to provide an estimated likelihood for the future state of climate variable, which can be commonly verified using Brier skill score (BSS).

The connection between the actual prediction skill and the potential predictability has been an interesting issue in the study of ENSO predictability. Several efforts have been devoted for investigating their relationship in the framework of the deterministic measures. For instance, it was found that the potential predictability metrics are good indicators in quantifying the deterministic actual skill in many ENSO models (Kumar and Hoerling 2000; Kumar et al. 2001; Tang et al. 2008; Cheng et al. 2011; Kumar et al. 2001; Kumar and Chen 2015). Recently, the linkage between probabilistic actual skill and deterministic actual skill has been addressed. Cheng et al. (2010a) found that the deterministic correlation skill is nonlinearly related with BSS probabilistic skill in the ENSO prediction. Yang et al. (2016, 2018) further theoretically derived their relationship and found that this relationship originated from the effect of the resolution term, while the reliability did not make contribution.

An interesting issue that has not been explored yet is the possible relationship between the probabilistic prediction skill and the potential predictability. Such a relationship, if existed and found, would be considerably interesting and important, since it offers practical confidence and estimate how much reliable the issued prediction is. Given the potential predictability-deterministic prediction skill relationship and the connection between probabilistic predictability and deterministic prediction skill, one can expect such a relationship existing.

In this paper, we will focus on investigating the relationship among the deterministic skill, probabilistic skill and potential predictability based on a long-term ensemble hindcast production of ENSO. Emphasis is placed on examining the potential predictability and the probabilistic prediction skill from both practical hindcast experiments and theoretical analysis. In Sect. 2, a concise description of the coupled model and the predictability measures are presented. Section 3 provides detailed information on the construction of the ensemble prediction and evaluates the forecast skill. In Sect. 4, we emphatically explore the relationships among the deterministic skill, the probabilistic skill and the potential predictability, in particular, the relationship between the probabilistic skill and the potential predictability. The main conclusion and discussion are summarized and followed in the final section.

## 2 Model and predictability metrics

### 2.1 The LDEO5 model

In this study, we employ the latest edition of the Zebiak–Cane (ZC) model, which is also named as LDEO5 model (Chen et al. 2004). The ZC model is an intermediate coupled model with a linear reduced-gravity ocean model and a Gill-type atmospheric model which driven by the anomalous heating combined sea surface temperature anomaly (SSTA) and low-level moisture convergence (Zebiak 1986). ZC model can reproduces certain key feature of the ENSO phenomenon and is the first coupled model used for ENSO prediction. It has high efficiency in calculation and been extensively applied to investigate the ENSO predictability. The domain of this model spans the tropical Pacific Ocean (124°E–80°W and 28.75°S–28.75°N), with a temporal resolution of 10 days. To initialize the long-term retrospection prediction, we assimilate the monthly reconstructed Kaplan SST V2 datasets (Kaplan et al. 1998) from 1856 to 2016 via a coupled nudging scheme (Chen et al. 2004). We also employ two model output statistic (MOS) procedures as present by Chen et al. (2004) at each integration step to rectify the systematic errors of the model.

### 2.2 Measures for ensemble prediction

- 1.
Deterministic prediction skill

- 2.
Probabilistic prediction skill

- 3.
Potential predictability

To quantify the potential predictability, we employed the STR measure and an information-based metric (MI). Both of the two measures do not involve with the observation, which are different from the actual prediction skill.

*M*indicates the total number of the initial conditions. Given the influence of the sampling error on measuring the signal variance, the more reasonable estimation of the signal variance can be given as follows (Rowell 1998):

For more details, see relevant literatures (Kleeman and Moore 1997; DelSole 2004; Tang et al. 2013).

## 3 Construction of the ensemble prediction

Generally, uncertainties for the ENSO prediction derived from the uncertainties in initial condition and the deficiencies in model formulation. The practice of the ensemble prediction is a useful strategy for sampling and evaluating these uncertainties (Wilks and Vannitsem 2018). From the perspective of optimal error growth, we combined two kinds of the optimal perturbation schemes for the initial condition and the stochastic atmospheric noise that has not been well considered in the framework of model.

## 4 Relationship among the deterministic skill, the probabilistic skill and the potential skill

### 4.1 Relationship between the deterministic skill and the probabilistic skill

In short, the monotonic nonlinear deterministic skill-probabilistic skill relationship which was found in previous studies (Yang et al. 2018), also holds well in the ENSO ensemble prediction system. This implies that the improvement of the ENSO resolution prediction also enhances its deterministic skill, and vice versa.

### 4.2 Relationship between the potential predictability skill and the deterministic prediction skill

### 4.3 Relationship between the potential predictability skill and the probabilistic prediction skill

In spite of these insignificant imperfections, the theoretical nonlinear monotonic potential predictability–probability prediction skill relationship can still be well verified in our ENSO ensemble prediction. As the deterministic prediction skill also exhibits covariability with the potential predictability skill in our ENSO ensemble forecast system (Fig. 5b), then indicating that a high potential predictability MI always corresponds to more accurate actual prediction skill in both the deterministic manner and probabilistic way.

## 5 Summary and discussion

The practice of evaluating and understanding the ENSO predictability involves two perspectives: actual prediction skill and the potential predictability. The former one can be investigated either from the deterministic viewpoint or probabilistic angle. Further, the inherent relationship between these two actual prediction skills has been validated from both the theoretical derivation and the practice of the seasonal climate prediction (Wang et al. 2009; Alessandri et al. 2011; Yang et al. 2016, 2018). In addition, the potential predictability measures are considered to be potential candidate indicators for the actual deterministic skill in a perfect model scenario (Tang et al. 2008; Cheng et al. 2010a, b). However, it still lacks systematic research on the issue about the coherent relationship among the deterministic skill, the probabilistic skill and the potential predictability in the ENSO prediction. In this study, we investigated the relationship among them based on the ENSO ensemble hindcast prediction over the 161 year, and especially discussed the connection between the probabilistic skill and the potential predictability skill from the practical application and theoretical identification, which has never been involved.

First, we constructed the ensemble forecast system with the combination of the optimal perturbations of the initial condition and the model stochastic physical processes. Based on this forecast system, we performed long-term hindcast over the past 161 year (1856–2016). The evaluation of the prediction indicated that this joint ensemble construction strategy could provide the skillful long-term ENSO prediction at lead 12 months. The further analysis demonstrated that the deterministic skill exhibited a nonlinear monotonic relationship with its probabilistic counterpart, in which the resolution property made the dominant contribution. These further confirm the theoretical result reported by Yang et al. (2018) with respect to the practice of the ENSO prediction and to imply that the enhancement of the ENSO resolution skill can also correspond to the improvement of the correlation skill.

In addition, our analysis also demonstrated that there was a nonlinear monotonic relationship between the deterministic prediction skill and the information-based potential predictability skill (MI) whether under the perfect model scenario or actual model prediction. These relationships could be approximately explained by a Gaussian-based theoretical equation, implying that MI has potential to indicate the actual prediction skill in this system.

Given that both the probabilistic skill and the potential predictability skill exhibit covariability with its deterministic compatriot in our ENSO ensemble forecast system, it’s naturally motivated us to further investigate the possible relationship between the probabilistic prediction skill and potential predictability skill. The result revealed the existence of a nonlinear monotonic link between the two in our ENSO ensemble forecast. Specifically, the high potential predictability skill always corresponds with high probabilistic skill. Under the assumptions that the forecast probabilistic density function is Gaussian, we derived a theoretical expression for the \(BSS_{RES}\)-MI relationship for the perfect model scenario. This theoretical demonstration was also practically well confirmed in our ENSO ensemble prediction.

All in all, the nonlinear monotonic relationships among the deterministic skill, the probabilistic skill and the potential predictability that are discussed in this study implies that we can quantify the model prediction skill (deterministic correlation or probabilistic resolution) by estimating the MI while issuing the prediction, which offers useful information for the end-users of prediction. According to the resolution-correlation relationship, it can be inferred that a possible effective way to improve the deterministic skill of ENSO prediction is to perform the multi-model ensemble prediction (Tippett and Barnston 2008), which has the potential to improve the resolution (Yang et al. 2016, 2018).

In short, the theoretical relationships among the deterministic skill, probabilistic skill and the potential predictability can approximately hold for the ENSO ensemble forecast. However, there are still some imperfections, especially with regard to the short lead time, where the ensemble spread is not sufficiently developed. This may be related to the linear-based perturbations we employed here. In further, we will focus on improving the ENSO actual skill through a combination of multiple-model ensemble prediction approach and some nonlinear-based perturbation methods, such as stochastic model-error perturbations (Zheng et al. 2009b) and Conditional Nonlinear Optimal Perturbation (Duan and Mu 2004) and so on.

## Notes

### Acknowledgements

This work was jointly supported by the National Key Research and Development Program (2017YFA0604202), the grants from the Scientific Research Fund of the Second Institute of Oceanography (JG1810), the National Natural Science Foundation of China (41690124, 41705049, 41690120, 41530961, 41621064). Y. Tang is also supported by NSERC (Natural Sciences and Engineering Research Council of Canada) Discovery program.

## References

- Alessandri A, Borrelli A, Navarra A, Arribas A, Déqué M, Rogel P, Weisheimer A (2011) Evaluation of probabilistic quality and value of the ENSEMBELS multimodel seasonal forecasts: comparison with DEMETER. Mon Weather Rev 139:581–607Google Scholar
- Atger F (2004) Estimation of the reliability of ensemble-based probabilistic forecasts. Q J R Meteorol Soc 130:627–646Google Scholar
- Chen D, Cane MA, Kaplan A, Zebiak SE, Huang D (2004) Predictability of El Niño over the past 148 years. Nature 428:733–736Google Scholar
- Chen D, Lian T, Fu C, Cane MA, Tang YM, Murtugudde R, Song XS, Wu QY, Zhou L (2015) Strong influence of westerly wind bursts on El Niño diversity. Nat Geosci 8:339–345Google Scholar
- Cheng YJ, Tang YM, Zhou XB, Jackson P, Chen D (2009) Further analysis of singular vector and ENSO predictability from 1856 to 2003—Part I: singular vector and control factors. Clim Dyn 35:807–826Google Scholar
- Cheng YJ, Tang YM, Jackson P, Chen D, Deng ZW (2010a) Ensemble construction and verification of the probabilistic ENSO prediction in the LDEO5 model. J Clim 23:5476–5479Google Scholar
- Cheng YJ, Tang YM, Jackson P, Chen D, Zhou XB, Deng ZW (2010b) Further analysis of singular vector and ENSO predictability from 1856 to 2003—Part II: singular value and predictability. Clim Dyn 35:827–840Google Scholar
- Cheng YJ, Tang YM, Chen D (2011) Relationship between predictability and forecast skill of ENSO on various time scales. J Geophys Res 116(C12). https://doi.org/10.1029/2011JC007249
- DelSole T (2004) Predictability and information theory. Part I: measures of predictability. J Atmos Sci 61:2425–2440Google Scholar
- DelSole T, Tippett MK (2007) Predictability: recent insights from information theory. Rev Geophys 45:4002Google Scholar
- Deng ZW, Tang YM, Wang G (2010) Assimilation of Argo temperature and salinity profiles using a bias-aware localized EnKF system for the Pacific Ocean. Ocean Model 35:187–205Google Scholar
- Duan WS, Mu M (2004) Conditional nonlinear optimal perturbations as the optimal precursors for El Nino–Southern Oscillation events. J Geophys Res Atmos 109(D23). https://doi.org/10.1029/2004jd004756
- Fan Y, Allen M, Anderson D, Balmaseda M (2000) How predictability depends on the nature of uncertainty in initial conditions in a coupled model of ENSO. J Clim 13:3298–3313Google Scholar
- Gebbie G, Eisenman I, Wittenberg A, Tziperman E (2007) Modulation of westerly wind bursts by sea surface temperature: a semistochastic feedback for ENSO. J Atmos Sci 64:3281–3295Google Scholar
- Hou ZL, Li JP, Ding RQ, Feng J, Duan WS (2018) The application of nonlinear local Lyapunov vectors to the Zebiak–Cane model and their performance in ensemble prediction. Clim Dyn 51:283–304Google Scholar
- Hu Z, Huang B (2012) The predictive skill and the most predictable pattern in the tropical atlantic: the effect of ENSO. Mon Weather Rev 135:1786–1806Google Scholar
- Kaplan A, Cane M, Kushnir Y, Clement A, Blumenthal M, Rajagopalan B (1998) Analysis of global sea surface temperature 1856–1991. J Geophys Res 103:18567–18589Google Scholar
- Kharin V, Zwiers F (2003) Improved seasonal probability forecasts. J Clim 16:1684–1701Google Scholar
- Kleeman R, Moore A (1997) A theory for the limitation of ENSO predictability due to stochastic atmospheric transients. J Atmos Sci 54:753–767Google Scholar
- Kumar A (2009) Finite samples and uncertainty estimates for skill measures for seasonal prediction. Mon Weather Rev 137:2622–2631Google Scholar
- Kumar A, Chen M (2015) Inherent predictability, requirements on the ensemble size, and complementarity. Mon Weather Rev 143:3192–3203Google Scholar
- Kumar A, Hoerling M (2000) Analysis of a conceptual model of seasonal climate variability and implications for seasonal prediction. Bull Am Meteorol Soc 81:255–264Google Scholar
- Kumar A, Hu Z (2014) How variable is the uncertainty in ENSO sea surface temperature prediction? J Clim 27:2779–2788Google Scholar
- Kumar A, Barnston A, Peng P, Hoerling M, Goddard L (2000) Changes in the spread of the variability of the seasonal mean atmospheric states associated with ENSO. J Clim 13:3139–3151Google Scholar
- Kumar A, Barnston A, Hoerling M (2001) Seasonal predictions, probabilistic verifications, and ensemble size. J Clim 14:671–1676Google Scholar
- Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142:2220–2227Google Scholar
- Kumar A, Hu Z, Jha B, Peng P (2016) Estimating ENSO predictability based on multi-model hindcasts. Clim Dyn 48:1–13Google Scholar
- Lian T, Chen D, Tang YM (2014) Effects of westerly wind bursts on El Niño: a new perspective. Geophys Res Lett 41:3522–3527Google Scholar
- Lorenz EN (1965) A study of the predictability of a 28-variable atmospheric model. Tellus 3:321–333Google Scholar
- Palmer T, Branković Č, Richardson D (2000) A probability and decision-model analysis of PROVOST seasonal multi-model ensemble integrations. Q J R Meteorol Soc 126:2013–2033Google Scholar
- Peng P, Kumar A, Wang W (2011) An analysis of seasonal predictability in coupled model forecasts. Clim Dyn 36:637–648Google Scholar
- Penland C, Sardeshmukh PD (1995) The optimal growth of tropical sea surface temperature anomalies. J Clim 8:1999–2024Google Scholar
- Rowell DP (1998) Assessing potential seasonal predictability with an ensemble of multidecadal GCM simulations. J Clim 11(2):109–120Google Scholar
- Tang YM, Hsieh W (2002) Hybrid coupled models of the tropical Pacific—II ENSO prediction. Clim Dyn 19:343–353Google Scholar
- Tang YM, Kleeman R, Moore A (2005) On the reliability of ENSO dynamical predictions. J Atmos Sci 62:1770–1791Google Scholar
- Tang YM, Kleeman R, Miller S (2006) ENSO predictability of a fully coupled GCM model using singular vector analysis. J Clim 19:3361–3377Google Scholar
- Tang YM, Kleeman R, Moore A (2008) Comparison of information-based measures of forecast uncertainty in ensemble ENSO prediction. J Clim 21:230–247Google Scholar
- Tang YM, Chen D, Yang D (2013) Methods of estimating uncertainty of climate prediction and climate change projection, in Climate Change-Realities, Impacts Over Ice Cap, Sea Level and Risks. InTech, RijekaGoogle Scholar
- Tang YM, Shen ZQ, Gao YQ (2016) An introduction to ensemble-based data assimilation method in the earth sciences, in nonlinear systems-design, analysis, estimation and control. InTech, RijekaGoogle Scholar
- Tang YM, Zhang RH, Liu T, Duan WS, Yang DJ, Zheng F, Ren HL, Lian T, Gao C, Chen D, Mu M (2018) Progress in ENSO prediction and predictability study. Natl Sci Rev 5:826–839Google Scholar
- Thompson C, Battisti D (2000) A linear stochastic dynamical model of ENSO. Part I: Model development. J Clim 13:2818–2832Google Scholar
- Tippett MK, Barnston A (2008) Skill of multimodel ENSO probability forecasts. Mon Weather Rev 136:3933–3946Google Scholar
- Tippett MK, Barnston A, Delsole T (2010) Comments on “finite samples and uncertainty estimates for skill measures for seasonal prediction”. Mon Weather Rev 138:1487–1493Google Scholar
- Toth Z, Kalnay E (1997) Ensemble forecasting at NCEP and the breeding method. Mon Weather Rev 125:3297–3319Google Scholar
- Toth Z, Talagrand O, Candille G, Zhu Y (2003) Probability and ensemble forecasts. In: Jolliffe IT, Stephenson DB (eds) Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Hoboken, pp 137–163Google Scholar
- Wang B, Lee J-Y, Kang I-S, Shukla J, Park C-K, Kumar A, Schemm J, Cocke S, Kug J-S, Luo J-J (2009) Advance and prospectus of seasonal prediction: assessment of the APCC/CliPAS 14-model ensemble retrospective seasonal prediction (1980–2004). Clim Dyn 33:93–117Google Scholar
- Wilks DS (2011) Statistical methods in the atmospheric sciences, Int. Geophys. Ser., vol. 100, 3rd edn. Academic, San DiegoGoogle Scholar
- Wilks DS, Vannitsem S (2018) Uncertain forecasts from deterministic dynamics. In: Vannitsem et al (eds) Statistical postprocessing of ensemble forecasts, pp 1–13. https://doi.org/10.1016/B978-0-12-812372-0.00001-7 Google Scholar
- Xue Y, Cane MA, Zebiak S (1997) Predictability of a coupled model of ENSO using singular vector analysis, Part I: optimal growth in seasonal background and ENSO cycles. Mon Weather Rev 125:2043–2056Google Scholar
- Yang DJ, Tang YM, Zhang YC, Yang XQ (2012) Information‐based potential predictability of the Asian summer monsoon in a coupled model. J Geophys Res Atmos 117(D3). https://doi.org/10.1029/2011JD016775 Google Scholar
- Yang DJ, Yang XQ, Xie Q, Zhang YC, Ren XJ, Tang YM (2016) Probabilistic versus deterministic skill in predicting the western North Pacific-East Asian summer monsoon variability with multimodel ensembles. J Geophys Res Atmos 121:1079–1103Google Scholar
- Yang DJ, Yang XQ, Ye D, Sun X, Fang J, Chu C, Feng T, Jiang Y, Liang J, Ren XJ, Zhang Y, Tang YM (2018) On the relationship between probabilistic and deterministic skills in dynamical seasonal climate prediction. J Geophys Res Atmos 123:5261–5283Google Scholar
- Zavala J, Zhang C, Moore A, Kleeman R (2005) The linear response of ENSO to the Madden–Julian oscillation. J Clim 18:2441–2459Google Scholar
- Zebiak S (1986) Atmospheric convergence feedback in a simple model for El Niño. Mon Weather Rev 114:1263–1271Google Scholar
- Zebiak S, Cane M (1987) A model El Niño-Southern oscillation. Mon Weather Rev 115:2262–2278Google Scholar
- Zhang RH, Zheng F, Zhu J, Wang Z (2013) A successful real-time forecast of the 2010–11 La Niña event. Sci Rep 3:1108Google Scholar
- Zheng F, Zhu J (2010) Coupled assimilation for an intermediated coupled ENSO prediction model. Ocean Dyn 60:1061–1073Google Scholar
- Zheng F, Zhu J, Zhang RH (2007) Impact of altimetry data on ENSO ensemble initializations and predictions. Geophys Res Lett 34(13). https://doi.org/10.1029/2007gl030451 Google Scholar
- Zheng F, Zhu J, Wang H, Zhang RH (2009a) Ensemble hindcasts of ENSO events over the past 120 years using a large number of ensembles. Adv Atmos Sci 26:359–372Google Scholar
- Zheng F, Wang H, Zhu J (2009b) ENSO ensemble prediction: initial error perturbations vs. model error perturbations. Chin Sci Bull 54:2516–2523Google Scholar
- Zhu J, Huang B, Zhang R, Hu Z, Kumar A, Balmaseda M, Marx L, Kinter J III (2014) Salinity anomaly as a trigger for ENSO events. Sci Rep 4:6821Google Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.