Climate Dynamics

, Volume 49, Issue 11–12, pp 4141–4156 | Cite as

Skill of real-time operational forecasts with the APCC multi-model ensemble prediction system during the period 2008–2015

  • Young-Mi Min
  • Vladimir N. Kryjov
  • Sang Myeong Oh
  • Hyun-Ju Lee


This paper assesses the real-time 1-month lead forecasts of 3-month (seasonal) mean temperature and precipitation on a monthly basis issued by the Asia-Pacific Economic Cooperation Climate Center (APCC) for 2008–2015 (8 years, 96 forecasts). It shows the current level of the APCC operational multi-model prediction system performance. The skill of the APCC forecasts strongly depends on seasons and regions that it is higher for the tropics and boreal winter than for the extratropics and boreal summer due to direct effects and remote teleconnections from boundary forcings. There is a negative relationship between the forecast skill and its interseasonal variability for both variables and the forecast skill for precipitation is more seasonally and regionally dependent than that for temperature. The APCC operational probabilistic forecasts during this period show a cold bias (underforecasting of above-normal temperature and overforecasting of below-normal temperature) underestimating a long-term warming trend. A wet bias is evident for precipitation, particularly in the extratropical regions. The skill of both temperature and precipitation forecasts strongly depends upon the ENSO strength. Particularly, the highest forecast skill noted in 2015/2016 boreal winter is associated with the strong forcing of an extreme El Nino event. Meanwhile, the relatively low skill is associated with the transition and/or continuous ENSO-neutral phases of 2012–2014. As a result the skill of real-time forecast for boreal winter season is higher than that of hindcast. However, on average, the level of forecast skill during the period 2008–2015 is similar to that of hindcast.


APEC Climate Center Multi-model prediction Seasonal forecast Skill of real-time forecast 



This research was supported by the APEC Climate Center. The authors are very grateful to the APCC MME Producing Centers for making their hindcast/forecast data available for analysis and the APEC Climate Center for collecting and archiving them and for organizing APCC MME prediction.


  1. Ahn JB, Kim HJ (2013) Improvement of one-month lead predictability of the wintertime AO using a realistically varying solar constant for a CGCM. Meteorol Appl 21:415–418. doi: 10.1002/met.1372 CrossRefGoogle Scholar
  2. Alessandri A, Borrelli A, Masina S, Di Pietro P, Carril AF, Cherchi A, Gualdi S, Navarra A (2010) The INGV-CMCC seasonal prediction system: improved ocean initial conditions. Mon Weather Rev 138:2930–2952. doi: 10.1175/2010MWR3178.1 CrossRefGoogle Scholar
  3. Alessandri A et al (2011) Evaluation of probabilistic quality and value of the ENSEMBELS multi-model seasonal forecasts: comparison with DEMETER. Mon Weather Rev 139:581–607. doi: 10.1175/2010MWR3417.1 CrossRefGoogle Scholar
  4. Alexander MA, Blade I, Newman M, Lanzante JD, Lau NC, Scott JD (2002) The atmospheric bridge: the influence of ENSO teleconnections on air-sea interaction over the global oceans. J Clim 15:2205–2231. doi: 10.1175/1520-0442(2002)015<2205:TABTIO>2.0.CO;2 CrossRefGoogle Scholar
  5. Atger F (2003) Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: consequences for calibration. Mon Weather Rev 131:1509–1523. doi: 10.1175//1520-0493(2003)131<1509:SAIVOT>2.0.CO;2 CrossRefGoogle Scholar
  6. Barnston AG, Mason SJ, Goddard L, Dewitt DG, Zebiak SE (2003) Multimodel ensembling in seasonal climate forecasting at IRI. Bull Am Meteorol Soc 84:1783–1796. doi: 10.1175/BAMS-84-12-1783 CrossRefGoogle Scholar
  7. Barnston AG, Shuhua L, Mason SJ, DeWitt DG, Goddard L, Gong X (2010) Verification of the first 11 years of IRI’s seasonal climate forecast. J Appl Meteorol Clim 49:493–520. doi: 10.1175//1520-0493(2003)131<1509:SAIVOT>2.0.CO;2 CrossRefGoogle Scholar
  8. Becker E, Van Den Dool H (2016) Probabilistic seasonal forecasts in the North American multi-model ensemble: a baseline skill assessment. J Clim 15:3015–3026. doi: 10.1175/JCLI-D-14-00862.1 CrossRefGoogle Scholar
  9. Boer GJ (2005) An evolving seasonal forecasting system using Bayes’ theorem. Atmos Ocean 43:129–143. doi: 10.3137/ao.430202 CrossRefGoogle Scholar
  10. Collins WD et al (2006) The community climate system model version 3 (CCSM3). J Clim 19:2122–2143. doi: 10.1175/JCLI3761.1 CrossRefGoogle Scholar
  11. DelSole T, Shukla J (2009) Artificial skill due to predictor screening. J Clim 22:331–345. doi: 10.1175/2008JCLI2414.1 CrossRefGoogle Scholar
  12. DelSole T, Nattala J, Tippett MK (2014) Skill improvement from increased ensemble size and model diversity. Geophys Res Lett 41:7331–7342. doi: 10.1002/2014GL060133 CrossRefGoogle Scholar
  13. Ding Y, Ni Y, Zhang X, Li M, Dong W, Zhao ZC, Li Z, Shen W (2000) Introduction to the short-term climate prediction model system. China Meteorological Press, Beijing (Chinese) Google Scholar
  14. Doblas-Rebes FJ, Deque M, Piedelievre JP (2000) Multi-model spread and probabilistic seasonal forecast in PROVOST. Q J R Meteorol Soc 126:2069–2088CrossRefGoogle Scholar
  15. Doblas-Reyes FJ, Hagedorn R, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting: II. Calibration and combination. Tellus 57A:234–252. doi: 10.1111/j.1600-0870.2005.00104.x Google Scholar
  16. Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J Appl Meteorol 8:985–987CrossRefGoogle Scholar
  17. Gong XF, Barnston AG, Ward MN (2003) The effect of spatial aggregation on the skill of seasonal precipitation forecasts. J Clim 16:3059–3071CrossRefGoogle Scholar
  18. Hagedorn R, Doblas-Reyes FJ, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting: I. Basic concept. Tellus 57A:219–233. doi: 10.1111/j.1600-0870.2005.00103.x Google Scholar
  19. Janowiak JE, Xie P (1999) CAMS_OPI: a global satellite-raingauge merged product for real-time precipitation monitoring applications. J Clim 12:3335–3342. doi: 10.1175/1520-0442(1999)012<3335:COAGSR>2.0.CO;2 CrossRefGoogle Scholar
  20. Jeong HI, Ashok K, Song BG, Min YM (2008) Experimental 6-month Hindcast and forecast simulation using CCSM3. APCC 2008 Technical Report. APEC Climate Center, KoreaGoogle Scholar
  21. Jeong HI, Lee DY, Ashok K, Ahn JB et al (2012) Assessment of the APCC coupled MME suite in predicting the distinctive climate impacts of two flavors of ENSO during boreal winter. Clim Dyn 39:475–493. doi: 10.1007/s00382-012-1359-3 CrossRefGoogle Scholar
  22. Jin EK, Kinter JL, Wang B et al (2008) Current status of ENSO prediction skill in coupled ocean-atmosphere model. Clim Dyn 31:647–664. doi: 10.1007/s00382-008-0397-3 CrossRefGoogle Scholar
  23. Jolliffe IT, Stephenson DB (2003) Forecast verification: a practitioner’s guide in atmospheric science. Wiley, ChichesterGoogle Scholar
  24. Kanamitsu M et al (2002) NCEP-DOE AMIP-II reanalysis (R-2). Bull Am Meteorol Soc 83:1631–1643. doi: 10.1175/BAMS-83-11-1631(2002)083<1631:NAR>2.3.CO;2 CrossRefGoogle Scholar
  25. Kang HS, Boo KO, Cho CH (2011) Introduction to KMA-Met office joint seasonal forecasting system and evaluation of its hindcast ensemble simulations. NOAA/NWS Science and Technology Infusion Climate BulletinGoogle Scholar
  26. Kharin VV, Zwiers FW (2003) Improved seasonal probability forecasts. J Clim 16:1684–1701. doi: 10.1175/1520-0442(2003)016<1684:ISPF>2.0.CO;2 CrossRefGoogle Scholar
  27. Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137:2908–2930. doi: 10.3137/Ao1101.2009 CrossRefGoogle Scholar
  28. Kryjov VN (2002) The influence of the winter Arctic oscillation on the northern Russia spring temperature. Int J Climatol 22:779–785. doi: 10.1002/joc.746 CrossRefGoogle Scholar
  29. Kryjov VN et al (2006) Assessment of the climate forecasts produced by individual models and MME methods. APCC Technical Report 2006. APEC Climate Center, KoreaGoogle Scholar
  30. Kumar A (2009) Finite samples and uncertainty estimates for skill measures for seasonal predictions. Mon Weather Rev 137:2622–2631CrossRefGoogle Scholar
  31. Kumar A, Hoerling MP (2003) The nature and causes for the delayed atmospheric response to El Nino. J Clim 16:1391–1403. doi: 10.1175/2771.1 CrossRefGoogle Scholar
  32. Lee JY, Wang B (2012) Seasonal climate prediction and predictability of atmospheric circulation. In: L. M. Druyan (ed) Climate models. InTech, Rijeka, pp 19–42. doi: 10.5772/33782 Google Scholar
  33. Lee DY, Ashok K, Ahn JB (2011) Toward enhancement of prediction skills of multimodel ensemble seasonal prediction: a climate filter concept. J Geophys Res 116:D06116. doi: 10.1029/2010JD014610 Google Scholar
  34. Lim EP, Hendon HH, Langford S, Alves O (2012) Improvements in POAMA2 for the prediction of major climate drivers and south eastern Australian rainfall. CAWCR Tech. Rep. No. 051.
  35. Liou CS, Chen JH, Terng CT, Wang FJ, Fong CT, Rosmond TE, Kuo HC, Shiao CH, Cheng MD (1997) The second generation global forecast system at the central weather bureau in Taiwan. Weather Forecast 12:653–663. doi: 10.1175/1520-0434-12.3.653 CrossRefGoogle Scholar
  36. MacLachlan C et al (2015) Global Seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Q J R Meteorol Soc 141:1072–1084CrossRefGoogle Scholar
  37. Mason I (1982) A model for assessment of weather forecasts. Aust Meteorol Mag 30:291–303Google Scholar
  38. Mason SJ (2004) On using “climatology” as a reference strategy in the Brier and ranked probability skill scores. Mon Weather Rev 132:1891–1895CrossRefGoogle Scholar
  39. Mason SJ, Goddard L (2001) Probabilistic precipitation anomalies associated with ENSO. Bull Am Meteorol Soc 82:619–638. doi: 10.1175/1520-0477(2001)082<0619:PPAAWE>2.3.CO;2 CrossRefGoogle Scholar
  40. Merryfield WJ et al (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141:2910–2945. doi: 10.1175/MWR-D-12-00216.1 CrossRefGoogle Scholar
  41. Min YM (2015) Development of a real-time verification system for the APCC operational multi-model prediction. APCC Research Report 2015-03Google Scholar
  42. Min YM, Kryjov VN, Park CK (2009) A probabilistic multimodel ensemble approach to seasonal prediction. Weather Forecast 24:812–828. doi: 10.1175/2008WAF2222140.1 CrossRefGoogle Scholar
  43. Min YM, Kryjov VN, Oh JH (2011) Probabilistic interpretation of regression-based downscaled seasonal ensemble predictions with the estimation of uncertainty. J Geophys Res 116:D08101. doi: 10.1029/2010JD015284 CrossRefGoogle Scholar
  44. Min YM, Kryjov VN, Oh SM (2014) Assessment of APCC multimodel ensemble prediction in seasonal climate forecasting: retrospective (1983–2003) and real-time forecasts (2008–2013). J Geophys Res 199:12132–12150Google Scholar
  45. Molod A et al (2012) The GEOS-5 atmospheric general circulation model: mean climate and development from MERRA to Fortuna. Technical Report Series on Global Modeling and Data Assimilation, 28.
  46. Murphy AH (1960) On the “ranked probability score”. J Appl Meteorol 8:988–989CrossRefGoogle Scholar
  47. Murphy AH (1971) A note on the ranked probability score. J Appl Meteorol 10:155–156CrossRefGoogle Scholar
  48. Murphy AH (1973) A new vector partition of the probability score. J Appl Meteorol 12:595–600. doi: 10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2 CrossRefGoogle Scholar
  49. Palmer TN et al (2004) Development of a European multi-model ensemble system for seasonal to inter-annual prediction. Bull Am Meteorol Soc 85:853–872. doi: 10.1175/BAMS-85-6-853 CrossRefGoogle Scholar
  50. Reynolds RW, Rayner NA, Smith TM, Stokes DC, Wang W (2002) An improved in situ and satellite SST analysis for climate. J Clim 15:1609–1625. doi: 10.1175/1520-0442(2002)015<1609:AIISAS>2.0.CO;2 CrossRefGoogle Scholar
  51. Richardson DS (2000) Skill and economic value of the ECMWF ensemble prediction system. Q J R Meteorol Soc 126:649–668. doi: 10.1002/qj.49712656313 CrossRefGoogle Scholar
  52. Rigor IG, Colony RL, Martin S (2000) Variations in surface air temperature observations in the Arctic, 1979–97. J Clim 13:896–914CrossRefGoogle Scholar
  53. Rodrigues LRL, Doblas-Reyes FJ, Coelho CAS (2014) Multi-model calibration and combination of tropical seasonal sea surface temperature forecasts. Clim Dyn 42:597–616. doi: 10.1007/s00382-013-1779-8 CrossRefGoogle Scholar
  54. Ropelewski CF, Halpert MS (1989) Precipitation patterns associated with the high index phase of the Southern Oscillation. J Clim 2:268–284. doi: 10.1175/1520-0442(1989)002<0268:PPAWTH>2.0.CO;2 CrossRefGoogle Scholar
  55. Saha S et al (2014) The NCEP climate forecast system version 2. J Clim 27:2185–2208. doi: 10.1175/JCLI-D-12-00823.1 CrossRefGoogle Scholar
  56. Scinocca JF, Mcfarlane NA, Lazare M, Li J, Plummer D (2008) The CCCma third generation AGCM and its extension into the middle atmosphere. Atmos Chem Phys 8:7055–7074. doi: 10.5194/acp-8-7055-2008 CrossRefGoogle Scholar
  57. Shneerov BE et al (2002) The current status of the MGO global atmospheric circulation model (version-MGO-03). MGO Proc 550:3–43Google Scholar
  58. Smith TM et al (2008) Improvements to NOAA’s historical merged land–ocean surface temperature analysis (1880–2006). J Clim 21:2283–2296. doi: 10.1175/2007JCLI2100.1 CrossRefGoogle Scholar
  59. Sohn SJ, Min YM, Lee JY, Tam CY, Kang IS, Wang B, Ahn JB, Yamagata T (2012) Assessment of the long-lead probabilistic prediction for the Asian summer monsoon precipitation (1983–2011) based on the APCC multimodel system and a statistical model. J Geophys Res 117:D04102. doi: 10.1029/20011JD016308 CrossRefGoogle Scholar
  60. Stefanova L, Krishnamurti TN (2002) Interpretation of seasonal climate forecast using Brier skill score, the Florida State University superensemble, and the AMIP-I data set. J Clim 15:537–544. doi: 10.1175/1520-0442(2002)015<0537:IOSCFU>2.0.CO;2 CrossRefGoogle Scholar
  61. Takaya Y, Yasuda T, Ose T, Nakaegawa T (2010) Predictability of the mean location of typhoon formation in a seasonal prediction experiment with a coupled general circulation model. J Meteorol Soc Japan 88:799–812. doi: 10.2151/jmsj.2010-502 CrossRefGoogle Scholar
  62. Tang Y, Deng Z, Zhou X, Cheng Y, Chen D (2008) Inter-decadal variation of ENSO predictability in multiple models. J Clim 21:4811–4833CrossRefGoogle Scholar
  63. Tippett MK, Barnston AG, Li S (2012) Performance of recent multimodel ENSO forecasts. J Appl Meteorol Clim 9:637–654. doi: 10.1175/JAMC-D-11-093.1 CrossRefGoogle Scholar
  64. Treshnikov AF, Korotkevich ES, Kruchinin YA, Markov VF (eds) (1985) Atlas of Arctic. Glavnoe Upravlenie Geodesii i Kartografii, MoscowGoogle Scholar
  65. Trosnikov IV, Kaznacheeva VD, Kiktev DB, Tolstikh MA (2005) Assessment of potential predictability of meteorological variables in dynamical seasonal modeling of atmospheric circulation on the basis of semi-Lagrangian model SL-AV. Russian Meteorol Hydrol 12:5–17Google Scholar
  66. Wang B et al (2009) Advance and prospectus of seasonal prediction: assessment of the APCC/CliPAS 14-model ensemble retrospective seasonal prediction (1980–2004). Clim Dyn 33:93–117. doi: 10.1007/s00382-008-0460-0 CrossRefGoogle Scholar
  67. Wang W, Chen M, Kumar A (2010) An assessment of the CFS real-time seasonal forecasts. Weather Forecast 25: 950–969. doi: 10.1175/2010WAF2222345.1 CrossRefGoogle Scholar
  68. Weigel AP, Liniger MA, Appenzeller C (2008) Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Q J R Meteorol Soc 134:241–260CrossRefGoogle Scholar
  69. Wilks DS (1995) Statistical methods in the atmospheric sciences: an introduction. Academic, San DiegoGoogle Scholar
  70. Wilks DS (2000) Diagnostic verification of the Climate Prediction Center long-lead outlooks, 1995–98. J Clim 13:2389–2403. doi: 10.1175/1520-0442(2000)013<2389:DVOTCP>2.0.CO;2 CrossRefGoogle Scholar
  71. Wilks DS, Godfrey CM (2002) Diagnostic verification of the IRI Net Assessment forecasts, 1997–2000. J Clim 15:1369–1377CrossRefGoogle Scholar
  72. World Meteorological Organization (2002) Standardised verification system (SVS) for long-range forecasts (LRF). New attachment II-9 to the manual on the GDPS, vol 1. WMO 485, GenevaGoogle Scholar
  73. Yang D, Yang XO, Xie Q, Zhang Y, Ren X, Tang Y (2016) Probabilistic versus deterministic skill in predicting the western North Pacific-East Asian summer monsoon variability with multimodel ensembles. J Geophys Res 121:1079–1103Google Scholar
  74. Yoo JH, Kang IS (2005) Theoretical examination of a multi-model composite for seasonal prediction. J Geophys Res 32:L18707. doi: 10.1029/2005GL023513 Google Scholar
  75. Zhu Y, Toth Z, Wobus R, Richardson D, Mylne K (2002) The economic value of ensemble-based weather forecasts. Bull Am Meteorol Soc 83:73–83. doi: 10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.Climate Prediction DepartmentAPEC Climate CenterBusanRepublic of Korea

Personalised recommendations