How Do We Know Whether Seasonal Climate Forecasts are Any Good?

When seasonal climate forecasts are expressed probabilistically, it is not possible to answer simple questions such as “how often are the forecasts correct?” The simpler attributes of forecast quality, such as “accuracy” or “correctness” are not applicable to probabilistic forecasts, and instead the main attributes of interest are: reliability, which defines whether the confidence communicated in the forecasts is appropriate; resolution, which defines whether there is any usable information in the forecasts; discrimination, which defines whether the forecasts are discernibly different given different outcomes (somewhat similar to the attribute of resolution); and sharpness, which defines the level of confidence that is communicated in the forecasts (regardless of whether that level is appropriate). How these attributes are measured depends on how the forecasts are expressed. In this chapter these attributes are explained in detail, and representation by various graphical procedures and scoring metrics is described. Partly because there is more than one desirable attribute to good probabilistic forecasts, it is argued that there is no single scoring metric that can adequately summarise forecast quality, and that in many cases graphical procedures also hide important aspects of forecast quality. The aim in this chapter is to provide some guidelines for interpreting and recognising the strengths and limitations of the most important verification tools as applied to seasonal climate forecasts.