Skill of real-time operational forecasts with the APCC multi-model ensemble prediction system during the period 2008–2015

Climate Dynamics volume 49, pages4141–4156(2017)Cite this article

Abstract

This paper assesses the real-time 1-month lead forecasts of 3-month (seasonal) mean temperature and precipitation on a monthly basis issued by the Asia-Pacific Economic Cooperation Climate Center (APCC) for 2008–2015 (8 years, 96 forecasts). It shows the current level of the APCC operational multi-model prediction system performance. The skill of the APCC forecasts strongly depends on seasons and regions that it is higher for the tropics and boreal winter than for the extratropics and boreal summer due to direct effects and remote teleconnections from boundary forcings. There is a negative relationship between the forecast skill and its interseasonal variability for both variables and the forecast skill for precipitation is more seasonally and regionally dependent than that for temperature. The APCC operational probabilistic forecasts during this period show a cold bias (underforecasting of above-normal temperature and overforecasting of below-normal temperature) underestimating a long-term warming trend. A wet bias is evident for precipitation, particularly in the extratropical regions. The skill of both temperature and precipitation forecasts strongly depends upon the ENSO strength. Particularly, the highest forecast skill noted in 2015/2016 boreal winter is associated with the strong forcing of an extreme El Nino event. Meanwhile, the relatively low skill is associated with the transition and/or continuous ENSO-neutral phases of 2012–2014. As a result the skill of real-time forecast for boreal winter season is higher than that of hindcast. However, on average, the level of forecast skill during the period 2008–2015 is similar to that of hindcast.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    The ONI is defined as 3-month running mean of ERSST.v3b SST anomalies in the Nino 3.4 region (5oN–5oS, 120o–170oW), based on centered 30-year based periods (1971–2000) and the episodes are defined when the threshold is met for a minimum of 5 consecutive over-lapping seasons. It is defined as a strong/moderate/weak El Nino (La Nina) based on a threshold of +1.5/1.0/0.5 °C (−1.5/1.0/0.5 °C) for the ONI.

  2. 2.

    http://www.cpc.ncep.noaa.gov/products/analysis_monitoring/ensostuff/ensoyears.shtml.

References

  1. Ahn JB, Kim HJ (2013) Improvement of one-month lead predictability of the wintertime AO using a realistically varying solar constant for a CGCM. Meteorol Appl 21:415–418. doi:10.1002/met.1372

    Article  Google Scholar 

  2. Alessandri A, Borrelli A, Masina S, Di Pietro P, Carril AF, Cherchi A, Gualdi S, Navarra A (2010) The INGV-CMCC seasonal prediction system: improved ocean initial conditions. Mon Weather Rev 138:2930–2952. doi:10.1175/2010MWR3178.1

    Article  Google Scholar 

  3. Alessandri A et al (2011) Evaluation of probabilistic quality and value of the ENSEMBELS multi-model seasonal forecasts: comparison with DEMETER. Mon Weather Rev 139:581–607. doi:10.1175/2010MWR3417.1

    Article  Google Scholar 

  4. Alexander MA, Blade I, Newman M, Lanzante JD, Lau NC, Scott JD (2002) The atmospheric bridge: the influence of ENSO teleconnections on air-sea interaction over the global oceans. J Clim 15:2205–2231. doi:10.1175/1520-0442(2002)015<2205:TABTIO>2.0.CO;2

    Article  Google Scholar 

  5. Atger F (2003) Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: consequences for calibration. Mon Weather Rev 131:1509–1523. doi:10.1175//1520-0493(2003)131<1509:SAIVOT>2.0.CO;2

    Article  Google Scholar 

  6. Barnston AG, Mason SJ, Goddard L, Dewitt DG, Zebiak SE (2003) Multimodel ensembling in seasonal climate forecasting at IRI. Bull Am Meteorol Soc 84:1783–1796. doi:10.1175/BAMS-84-12-1783

    Article  Google Scholar 

  7. Barnston AG, Shuhua L, Mason SJ, DeWitt DG, Goddard L, Gong X (2010) Verification of the first 11 years of IRI’s seasonal climate forecast. J Appl Meteorol Clim 49:493–520. doi:10.1175//1520-0493(2003)131<1509:SAIVOT>2.0.CO;2

    Article  Google Scholar 

  8. Becker E, Van Den Dool H (2016) Probabilistic seasonal forecasts in the North American multi-model ensemble: a baseline skill assessment. J Clim 15:3015–3026. doi:10.1175/JCLI-D-14-00862.1

    Article  Google Scholar 

  9. Boer GJ (2005) An evolving seasonal forecasting system using Bayes’ theorem. Atmos Ocean 43:129–143. doi:10.3137/ao.430202

    Article  Google Scholar 

  10. Collins WD et al (2006) The community climate system model version 3 (CCSM3). J Clim 19:2122–2143. doi:10.1175/JCLI3761.1

    Article  Google Scholar 

  11. DelSole T, Shukla J (2009) Artificial skill due to predictor screening. J Clim 22:331–345. doi:10.1175/2008JCLI2414.1

    Article  Google Scholar 

  12. DelSole T, Nattala J, Tippett MK (2014) Skill improvement from increased ensemble size and model diversity. Geophys Res Lett 41:7331–7342. doi:10.1002/2014GL060133

    Article  Google Scholar 

  13. Ding Y, Ni Y, Zhang X, Li M, Dong W, Zhao ZC, Li Z, Shen W (2000) Introduction to the short-term climate prediction model system. China Meteorological Press, Beijing (Chinese)

    Google Scholar 

  14. Doblas-Rebes FJ, Deque M, Piedelievre JP (2000) Multi-model spread and probabilistic seasonal forecast in PROVOST. Q J R Meteorol Soc 126:2069–2088

    Article  Google Scholar 

  15. Doblas-Reyes FJ, Hagedorn R, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting: II. Calibration and combination. Tellus 57A:234–252. doi:10.1111/j.1600-0870.2005.00104.x

    Google Scholar 

  16. Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J Appl Meteorol 8:985–987

    Article  Google Scholar 

  17. Gong XF, Barnston AG, Ward MN (2003) The effect of spatial aggregation on the skill of seasonal precipitation forecasts. J Clim 16:3059–3071

    Article  Google Scholar 

  18. Hagedorn R, Doblas-Reyes FJ, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting: I. Basic concept. Tellus 57A:219–233. doi:10.1111/j.1600-0870.2005.00103.x

    Google Scholar 

  19. Janowiak JE, Xie P (1999) CAMS_OPI: a global satellite-raingauge merged product for real-time precipitation monitoring applications. J Clim 12:3335–3342. doi:10.1175/1520-0442(1999)012<3335:COAGSR>2.0.CO;2

    Article  Google Scholar 

  20. Jeong HI, Ashok K, Song BG, Min YM (2008) Experimental 6-month Hindcast and forecast simulation using CCSM3. APCC 2008 Technical Report. APEC Climate Center, Korea

    Google Scholar 

  21. Jeong HI, Lee DY, Ashok K, Ahn JB et al (2012) Assessment of the APCC coupled MME suite in predicting the distinctive climate impacts of two flavors of ENSO during boreal winter. Clim Dyn 39:475–493. doi:10.1007/s00382-012-1359-3

    Article  Google Scholar 

  22. Jin EK, Kinter JL, Wang B et al (2008) Current status of ENSO prediction skill in coupled ocean-atmosphere model. Clim Dyn 31:647–664. doi:10.1007/s00382-008-0397-3

    Article  Google Scholar 

  23. Jolliffe IT, Stephenson DB (2003) Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Chichester

    Google Scholar 

  24. Kanamitsu M et al (2002) NCEP-DOE AMIP-II reanalysis (R-2). Bull Am Meteorol Soc 83:1631–1643. doi:10.1175/BAMS-83-11-1631(2002)083<1631:NAR>2.3.CO;2

    Article  Google Scholar 

  25. Kang HS, Boo KO, Cho CH (2011) Introduction to KMA-Met office joint seasonal forecasting system and evaluation of its hindcast ensemble simulations. NOAA/NWS Science and Technology Infusion Climate Bulletin

  26. Kharin VV, Zwiers FW (2003) Improved seasonal probability forecasts. J Clim 16:1684–1701. doi:10.1175/1520-0442(2003)016<1684:ISPF>2.0.CO;2

    Article  Google Scholar 

  27. Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137:2908–2930. doi:10.3137/Ao1101.2009

    Article  Google Scholar 

  28. Kryjov VN (2002) The influence of the winter Arctic oscillation on the northern Russia spring temperature. Int J Climatol 22:779–785. doi:10.1002/joc.746

    Article  Google Scholar 

  29. Kryjov VN et al (2006) Assessment of the climate forecasts produced by individual models and MME methods. APCC Technical Report 2006. APEC Climate Center, Korea

    Google Scholar 

  30. Kumar A (2009) Finite samples and uncertainty estimates for skill measures for seasonal predictions. Mon Weather Rev 137:2622–2631

    Article  Google Scholar 

  31. Kumar A, Hoerling MP (2003) The nature and causes for the delayed atmospheric response to El Nino. J Clim 16:1391–1403. doi:10.1175/2771.1

    Article  Google Scholar 

  32. Lee JY, Wang B (2012) Seasonal climate prediction and predictability of atmospheric circulation. In: L. M. Druyan (ed) Climate models. InTech, Rijeka, pp 19–42. doi:10.5772/33782

    Google Scholar 

  33. Lee DY, Ashok K, Ahn JB (2011) Toward enhancement of prediction skills of multimodel ensemble seasonal prediction: a climate filter concept. J Geophys Res 116:D06116. doi:10.1029/2010JD014610

    Google Scholar 

  34. Lim EP, Hendon HH, Langford S, Alves O (2012) Improvements in POAMA2 for the prediction of major climate drivers and south eastern Australian rainfall. CAWCR Tech. Rep. No. 051. http://www.cawcr.gov.au/publications/technicalreports.php

  35. Liou CS, Chen JH, Terng CT, Wang FJ, Fong CT, Rosmond TE, Kuo HC, Shiao CH, Cheng MD (1997) The second generation global forecast system at the central weather bureau in Taiwan. Weather Forecast 12:653–663. doi:10.1175/1520-0434-12.3.653

    Article  Google Scholar 

  36. MacLachlan C et al (2015) Global Seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Q J R Meteorol Soc 141:1072–1084

    Article  Google Scholar 

  37. Mason I (1982) A model for assessment of weather forecasts. Aust Meteorol Mag 30:291–303

    Google Scholar 

  38. Mason SJ (2004) On using “climatology” as a reference strategy in the Brier and ranked probability skill scores. Mon Weather Rev 132:1891–1895

    Article  Google Scholar 

  39. Mason SJ, Goddard L (2001) Probabilistic precipitation anomalies associated with ENSO. Bull Am Meteorol Soc 82:619–638. doi:10.1175/1520-0477(2001)082<0619:PPAAWE>2.3.CO;2

    Article  Google Scholar 

  40. Merryfield WJ et al (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141:2910–2945. doi:10.1175/MWR-D-12-00216.1

    Article  Google Scholar 

  41. Min YM (2015) Development of a real-time verification system for the APCC operational multi-model prediction. APCC Research Report 2015-03

  42. Min YM, Kryjov VN, Park CK (2009) A probabilistic multimodel ensemble approach to seasonal prediction. Weather Forecast 24:812–828. doi:10.1175/2008WAF2222140.1

    Article  Google Scholar 

  43. Min YM, Kryjov VN, Oh JH (2011) Probabilistic interpretation of regression-based downscaled seasonal ensemble predictions with the estimation of uncertainty. J Geophys Res 116:D08101. doi:10.1029/2010JD015284

    Article  Google Scholar 

  44. Min YM, Kryjov VN, Oh SM (2014) Assessment of APCC multimodel ensemble prediction in seasonal climate forecasting: retrospective (1983–2003) and real-time forecasts (2008–2013). J Geophys Res 199:12132–12150

    Google Scholar 

  45. Molod A et al (2012) The GEOS-5 atmospheric general circulation model: mean climate and development from MERRA to Fortuna. Technical Report Series on Global Modeling and Data Assimilation, 28. http://gmao.gsfc.nasa.gov/pubs/docs/tm28.pdf

  46. Murphy AH (1960) On the “ranked probability score”. J Appl Meteorol 8:988–989

    Article  Google Scholar 

  47. Murphy AH (1971) A note on the ranked probability score. J Appl Meteorol 10:155–156

    Article  Google Scholar 

  48. Murphy AH (1973) A new vector partition of the probability score. J Appl Meteorol 12:595–600. doi:10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2

    Article  Google Scholar 

  49. Palmer TN et al (2004) Development of a European multi-model ensemble system for seasonal to inter-annual prediction. Bull Am Meteorol Soc 85:853–872. doi:10.1175/BAMS-85-6-853

    Article  Google Scholar 

  50. Reynolds RW, Rayner NA, Smith TM, Stokes DC, Wang W (2002) An improved in situ and satellite SST analysis for climate. J Clim 15:1609–1625. doi:10.1175/1520-0442(2002)015<1609:AIISAS>2.0.CO;2

    Article  Google Scholar 

  51. Richardson DS (2000) Skill and economic value of the ECMWF ensemble prediction system. Q J R Meteorol Soc 126:649–668. doi:10.1002/qj.49712656313

    Article  Google Scholar 

  52. Rigor IG, Colony RL, Martin S (2000) Variations in surface air temperature observations in the Arctic, 1979–97. J Clim 13:896–914

    Article  Google Scholar 

  53. Rodrigues LRL, Doblas-Reyes FJ, Coelho CAS (2014) Multi-model calibration and combination of tropical seasonal sea surface temperature forecasts. Clim Dyn 42:597–616. doi:10.1007/s00382-013-1779-8

    Article  Google Scholar 

  54. Ropelewski CF, Halpert MS (1989) Precipitation patterns associated with the high index phase of the Southern Oscillation. J Clim 2:268–284. doi:10.1175/1520-0442(1989)002<0268:PPAWTH>2.0.CO;2

    Article  Google Scholar 

  55. Saha S et al (2014) The NCEP climate forecast system version 2. J Clim 27:2185–2208. doi:10.1175/JCLI-D-12-00823.1

    Article  Google Scholar 

  56. Scinocca JF, Mcfarlane NA, Lazare M, Li J, Plummer D (2008) The CCCma third generation AGCM and its extension into the middle atmosphere. Atmos Chem Phys 8:7055–7074. doi:10.5194/acp-8-7055-2008

    Article  Google Scholar 

  57. Shneerov BE et al (2002) The current status of the MGO global atmospheric circulation model (version-MGO-03). MGO Proc 550:3–43

    Google Scholar 

  58. Smith TM et al (2008) Improvements to NOAA’s historical merged land–ocean surface temperature analysis (1880–2006). J Clim 21:2283–2296. doi:10.1175/2007JCLI2100.1

    Article  Google Scholar 

  59. Sohn SJ, Min YM, Lee JY, Tam CY, Kang IS, Wang B, Ahn JB, Yamagata T (2012) Assessment of the long-lead probabilistic prediction for the Asian summer monsoon precipitation (1983–2011) based on the APCC multimodel system and a statistical model. J Geophys Res 117:D04102. doi:10.1029/20011JD016308

    Article  Google Scholar 

  60. Stefanova L, Krishnamurti TN (2002) Interpretation of seasonal climate forecast using Brier skill score, the Florida State University superensemble, and the AMIP-I data set. J Clim 15:537–544. doi:10.1175/1520-0442(2002)015<0537:IOSCFU>2.0.CO;2

    Article  Google Scholar 

  61. Takaya Y, Yasuda T, Ose T, Nakaegawa T (2010) Predictability of the mean location of typhoon formation in a seasonal prediction experiment with a coupled general circulation model. J Meteorol Soc Japan 88:799–812. doi:10.2151/jmsj.2010-502

    Article  Google Scholar 

  62. Tang Y, Deng Z, Zhou X, Cheng Y, Chen D (2008) Inter-decadal variation of ENSO predictability in multiple models. J Clim 21:4811–4833

    Article  Google Scholar 

  63. Tippett MK, Barnston AG, Li S (2012) Performance of recent multimodel ENSO forecasts. J Appl Meteorol Clim 9:637–654. doi:10.1175/JAMC-D-11-093.1

    Article  Google Scholar 

  64. Treshnikov AF, Korotkevich ES, Kruchinin YA, Markov VF (eds) (1985) Atlas of Arctic. Glavnoe Upravlenie Geodesii i Kartografii, Moscow

    Google Scholar 

  65. Trosnikov IV, Kaznacheeva VD, Kiktev DB, Tolstikh MA (2005) Assessment of potential predictability of meteorological variables in dynamical seasonal modeling of atmospheric circulation on the basis of semi-Lagrangian model SL-AV. Russian Meteorol Hydrol 12:5–17

    Google Scholar 

  66. Wang B et al (2009) Advance and prospectus of seasonal prediction: assessment of the APCC/CliPAS 14-model ensemble retrospective seasonal prediction (1980–2004). Clim Dyn 33:93–117. doi:10.1007/s00382-008-0460-0

    Article  Google Scholar 

  67. Wang W, Chen M, Kumar A (2010) An assessment of the CFS real-time seasonal forecasts. Weather Forecast 25: 950–969. doi:10.1175/2010WAF2222345.1

    Article  Google Scholar 

  68. Weigel AP, Liniger MA, Appenzeller C (2008) Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Q J R Meteorol Soc 134:241–260

    Article  Google Scholar 

  69. Wilks DS (1995) Statistical methods in the atmospheric sciences: an introduction. Academic, San Diego

    Google Scholar 

  70. Wilks DS (2000) Diagnostic verification of the Climate Prediction Center long-lead outlooks, 1995–98. J Clim 13:2389–2403. doi:10.1175/1520-0442(2000)013<2389:DVOTCP>2.0.CO;2

    Article  Google Scholar 

  71. Wilks DS, Godfrey CM (2002) Diagnostic verification of the IRI Net Assessment forecasts, 1997–2000. J Clim 15:1369–1377

    Article  Google Scholar 

  72. World Meteorological Organization (2002) Standardised verification system (SVS) for long-range forecasts (LRF). New attachment II-9 to the manual on the GDPS, vol 1. WMO 485, Geneva

    Google Scholar 

  73. Yang D, Yang XO, Xie Q, Zhang Y, Ren X, Tang Y (2016) Probabilistic versus deterministic skill in predicting the western North Pacific-East Asian summer monsoon variability with multimodel ensembles. J Geophys Res 121:1079–1103

    Google Scholar 

  74. Yoo JH, Kang IS (2005) Theoretical examination of a multi-model composite for seasonal prediction. J Geophys Res 32:L18707. doi:10.1029/2005GL023513

    Google Scholar 

  75. Zhu Y, Toth Z, Wobus R, Richardson D, Mylne K (2002) The economic value of ensemble-based weather forecasts. Bull Am Meteorol Soc 83:73–83. doi:10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the APEC Climate Center. The authors are very grateful to the APCC MME Producing Centers for making their hindcast/forecast data available for analysis and the APEC Climate Center for collecting and archiving them and for organizing APCC MME prediction.

Author information

Affiliations

  1. Climate Prediction Department, APEC Climate Center, 12 Centum 7-ro, Haeundae-gu, Busan, 48058, Republic of Korea

    Young-Mi Min, Vladimir N. Kryjov, Sang Myeong Oh & Hyun-Ju Lee

Authors
  1. Young-Mi Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir N. Kryjov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sang Myeong Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hyun-Ju Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Young-Mi Min.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Min, YM., Kryjov, V.N., Oh, S.M. et al. Skill of real-time operational forecasts with the APCC multi-model ensemble prediction system during the period 2008–2015. Clim Dyn 49, 4141–4156 (2017). https://doi.org/10.1007/s00382-017-3576-2

Download citation

Keywords

  • APEC Climate Center
  • Multi-model prediction
  • Seasonal forecast
  • Skill of real-time forecast