Journal of Meteorological Research

, Volume 32, Issue 1, pp 135–145 | Cite as

Assessing Hourly Precipitation Forecast Skill with the Fractions Skill Score

  • Bin Zhao
  • Bo Zhang
Regular Articles


Statistical methods for category (yes/no) forecasts, such as the Threat Score, are typically used in the verification of precipitation forecasts. However, these standard methods are affected by the so-called “double-penalty” problem caused by slight displacements in either space or time with respect to the observations. Spatial techniques have recently been developed to help solve this problem. The fractions skill score (FSS), a neighborhood spatial verification method, directly compares the fractional coverage of events in windows surrounding the observations and forecasts. We applied the FSS to hourly precipitation verification by taking hourly forecast products from the GRAPES (Global/Regional Assimilation Prediction System) regional model and quantitative precipitation estimation products from the National Meteorological Information Center of China during July and August 2016, and investigated the difference between these results and those obtained with the traditional category score. We found that the model spin-up period affected the assessment of stability. Systematic errors had an insignificant role in the fraction Brier score and could be ignored. The dispersion of observations followed a diurnal cycle and the standard deviation of the forecast had a similar pattern to the reference maximum of the fraction Brier score. The coefficient of the forecasts and the observations is similar to the FSS; that is, the FSS may be a useful index that can be used to indicate correlation. Compared with the traditional skill score, the FSS has obvious advantages in distinguishing differences in precipitation time series, especially in the assessment of heavy rainfall.


neighborhood spatial verification method fractions skill score traditional category score hourly precipitation heavy rainfall 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ahijevych, D., E. Gilleland, B. G. Brown, et al., 2009: Application of spatial verification methods to idealized and NWPgridded precipitation forecasts. Wea. Forecasting, 24, 1485–1497, doi: 10.1175/2009WAF2222298.1.CrossRefGoogle Scholar
  2. Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8, 401–417, doi: 10.5194/npg-8-401-2001.CrossRefGoogle Scholar
  3. Baldwin, M. E., and J. S. Kain, 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636–648, doi: 10.1175/WAF933.1.CrossRefGoogle Scholar
  4. Brill, K. F., and F. Mesinger, 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24, 1748–1754, doi: 10.1175/2009WAF2222272.1.CrossRefGoogle Scholar
  5. Casati, B., 2010: New developments of the intensity-scale technique within the Spatial Verification Methods Intercomparison Project. Wea. Forecasting, 25, 113–143, doi: 10.1175/2009WAF2222257.1.CrossRefGoogle Scholar
  6. Casati, B., G. Ross, and D. B. Stephenson, 2004: A new intensityscale approach for the verification of spatial precipitation forecasts. Meteor. Appl., 11, 141–154, doi: 10.1017/S1350482704001239.CrossRefGoogle Scholar
  7. Casati, B., L. J. Wilson, D. B. Stephenson, et al., 2008: Forecast verification: Current status and future directions. Meteor. Appl., 15, 3–18, doi: 10.1002/(ISSN)1469-8080.CrossRefGoogle Scholar
  8. Chen, J., Y. Wang, L. Li, et al., 2013: A unified verification system for operational models from Regional Meteorological Centres of China Meteorological Administration. Meteor. Appl., 20, 140–149, doi: 10.1002/met.1406.CrossRefGoogle Scholar
  9. Davis, C. A., B. G. Brown, R. Bullock, et al., 2009: The method for Object-Based Diagnostic Evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24, 1252–1267, doi: 10.1175/2009WAF2222241.1.CrossRefGoogle Scholar
  10. Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 51–64, doi: 10.1002/(ISSN)1469-8080.CrossRefGoogle Scholar
  11. Ebert, E. E., 2009: Neighborhood verification: A strategy for rewarding close forecasts. Wea. Forecasting, 24, 1498–1510, doi: 10.1175/2009WAF2222251.1.CrossRefGoogle Scholar
  12. Ebert, E. E., and W. A. Gallus Jr., 2009: Toward better understanding of the contiguous rain area (CRA) method for spatial forecast verification. Wea. Forecasting, 24, 1401–1415, doi: 10.1175/2009WAF2222252.1.CrossRefGoogle Scholar
  13. Gilleland, E., 2016: A new characterization in the spatial verification framework for false alarms, misses, and overall patterns. Wea. Forecasting, 32, 187–198, doi: 10.1175/WAF-D-16-0134.1.CrossRefGoogle Scholar
  14. Lorenz, E. N., 1969: Atmospheric predictability as revealed by naturally occurring analogues. J. Atmos. Sci, 26, 636–646, doi: 10.1175/1520-0469(1969)26<636:APARBN>2.0.CO;2.CrossRefGoogle Scholar
  15. Mass, C. F., D. Ovens, K. Westrick, et al., 2002: Does increasing horizontal resolution produce more skillful forecasts? Bull Amer. Meteor. Soc, 83, 407–430, doi: 10.1175/1520-0477(2002)083<0407:DIHRPM>2.3.CO;2.CrossRefGoogle Scholar
  16. Mittermaier, M., and N. Roberts, 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score. Wea. Forecasting, 25, 343–354, doi: 10.1175/2009WAF2222260.1.CrossRefGoogle Scholar
  17. Murphy, A. H., 1971: A note on the ranked probability score. J. Appl. Meteor., 10, 155–156, doi: 10.1175/1520-0450(1971)010<0155:ANOTRP>2.0.CO;2.CrossRefGoogle Scholar
  18. Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting Wea. Forecasting, 8, 281–293, doi: 10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.CrossRefGoogle Scholar
  19. Roberts, N., 2008: Assessing the spatial and temporal variation in the skill of precipitation forecasts from an NWP model. Meteor. Appl., 15, 163–169, doi: 10.1002/(ISSN)1469-8080.CrossRefGoogle Scholar
  20. Roberts, N., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of con-vective events. Mon. Wea. Rev., 136, 78–97, doi: 10.1175/2007MWR2123.1.CrossRefGoogle Scholar
  21. Taylor, K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Georphys. Res., 106, 7183–7192, doi: 10.1029/2000JD900719.CrossRefGoogle Scholar
  22. Weisman, M. L., C. Davis, W. Wang, et al., 2008: Experiences with 0-36-h explicit convective forecasts with the WRF-ARW model. Wea. Forecasting, 23, 407–437, doi: 10.1175/2007WAF2007005.1.CrossRefGoogle Scholar
  23. Yates, E., S. Anquetin, V. Ducrocq, et al., 2006: Point and areal validation of forecast precipitation fields. Meteor. Appl., 13, 1–20.CrossRefGoogle Scholar
  24. Zepeda-Arce, J., E. Foufoula-Georgiou, and K. K. Droegemeier, 2000: Space-time rainfall organization and its role in validating quantitative precipitation forecasts. J. Geophys. Res., 105, 10129–10146, doi: 10.1029/1999JD901087.CrossRefGoogle Scholar
  25. Zhao, B., and B. Zhang, 2017: Application of neighborhood spatial verification method on precipitation evaluation. Torrential Rain and Disasters, 36, 497–504, doi: 10.3969/j.issn.1004-045.2017.06.004. (in Chinese)Google Scholar

Copyright information

© The Chinese Meteorological Society and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.National Meteorological CenterBeijingChina
  2. 2.China Meteorological Administration Numerical Weather Prediction CenterBeijingChina

Personalised recommendations