Validation and Evaluation of Predictive Models in Hazard Assessment and Risk Management Article Received: 01 June 2004 Accepted: 04 April 2005 DOI:
Cite this article as: Beguería, S. Nat Hazards (2006) 37: 315. doi:10.1007/s11069-005-5182-6 Abstract
The paper deals with the validation and evaluation of mathematical models in natural hazard analysis, with a special focus on establishing their predictive power. Although most of the tools and statistics available are common to general classification models, some peculiarities arise in the case of hazard assessment. This is due to the fact that the target for validation, the propensity to develop a dangerous characteristic, is not really known and must be estimated from a (usually) very small sample. This implies that the two types of errors (false positives and false negatives) should be given different meanings. Related to this, a very frequent situation is the presence of prevalence (different proportion of positive and negative cases) in the sample. It is shown that sample prevalence can have a dramatic effect in some very common validation statistics, like the confusion matrix and model efficiency. Here some statistics based on the confusion matrix are presented and discussed, and the use of threshold-independent approaches (especially the ROC plot) is shown. The ROC plot is also proposed as a convenient tool for decision-taking in a risk management context. A general scheme for hazard predictive modeling is finally proposed.
Keywords geomorphological hazard modelling probabilistic models prediction errors accuracy assessment decision support decision threshold ROC plot References Bledsoe, B. P., Watson, C. C. 2001 Logistic analysis of channel pattern thresholds: meandering, braiding, and incising Geomorphology 38 281 300 Google Scholar Carrara, A. 1983 Multivariate models for landslide hazard evaluation Math. Geol. 15 403 426 Google Scholar Carrara, A., Cardinali, M., Detti, R., Guzzetti, F., Pasqui, V., Reichenbach, P. 1991 GIS techniques and statistical models in evaluating landslide hazard Earth Surf. Process. Landforms 16 427 445 Google Scholar Chung, C. F., Fabbri, A. G. 1999 Probabilistic prediction models for landslide hazard mapping Photogram. Eng. Remote Sensing 65 1389 1399 Google Scholar Chung, C. F., Fabbri, A. G. 2003 Validation of spatial prediction models for landslide hazard mapping Nat. Hazards 30 451 472 CrossRef Google Scholar
Chung, C. F., Fabbri, A., and Van Westen, C. J.: 1995. Multivariate regression analysis for landslide hazard zonation. In A. Carrara & F. Guzzetti (Eds.), The Nederlands: Kluwer Academic Publishers (pp. 107–133).
Clerici, A., Parego, S., Tellini, C., Vescovi, P. 2002 A procedure for landslide susceptibility zonation by the conditional analysis method Geomorphology 48 349 364 CrossRef Google Scholar Dai, F. C., Lee, C. F. 2002 Landslide characteristics and slope instability modeling using GIS, Lantau Island, Hong Kong Geomorphology 42 213 228 Google Scholar
Deleo, J. M.: 1993, Receiver operating characteristic laboratory (ROCLAB): software for developing decision strategies that account for uncertainty. In:
Proceedings of the Second International Symposium on Uncertainty Modelling and Analysis, Computer Society Press, College Park, pp. 318–325 Fielding, A. H., Bell, J. F. 1997 A review of methods for the assessment of prediction errors in conservation presence/absence models Environ. Conserv. 24 38 49 Google Scholar Floyer, J. A., McClung, D. M. 2003 Numerical avalanche prediction: Bear Pass, British Columbia, Canada Cold Regions Sci. Technol. 37 333 342 Google Scholar Forbes, A. D. 1995 Classification-algorithm evaluation: five performance measures based on confusion matrices J. Clin. Monitor. 11 189 206 Google Scholar Furbish, D. J., Rice, R. M. 1983 Predicting landslides related to clearcut logging, Northwestern California, USA Mount. Res. Dev. 3 253 259 Google Scholar Lee, S., Choi, J., Min, K. 2002 Landslide susceptibility analysis and verification using the Bayesian probability model Environ. Geol. 43 120 131 Google Scholar Lorente, A., García-Ruiz, J. M., Beguería, S., Arnáez, J. 2002 Factors explaining the spatial distribution of hillslope debris flows: a case study in the Flysch Sector of the Central Spanish Pyrenees Mount. Res. Dev. 22 32 39 Google Scholar Martínez-Casasnovas, J. A., Ramos, M. C., Poesen, J. 2003 Assessment of sidewall erosion in large gullies using multi-temporal DEMs and logistic regression analysis Geomorphology 58 305 321 Google Scholar Massie, D. D., White, K. D., Daly, S. F. 2002 Application of neural networks to predict ice jam occurrence Cold Regions Sci. Technol. 35 115 122 Google Scholar Morgan, R. P. C., Mngomezulu, D. 2003 Threshold conditions for initiation of valley-side gullies in the Middle Veld of Swaziland Catena 50 401 414 Google Scholar Neuland, H. 1976 A prediction model of landslips Catena 3 215 230 CrossRef Google Scholar Perry, F. V., Valentine, G. A., Desmarais, E. K., WoldeGabriel, G. 2001 Probabilistic assessment of volcanic hazard to radioactive waste repositories in Japan: intersection by a dike from a nearby composite volcano Geology 29 255 258 CrossRef Google Scholar Remondo, J., González-Díez, A., Terán, J. R. D. D., Cendrero, A. 2003 Landslide susceptibility models utilising spatial data analysis techniques. A case study from the Lower Deba Valley, Guipuzcoa (Spain) Nat. Hazards 30 233 249 Google Scholar
Rice, R. M. and Pillsbury, N. H.: 1982, Predicting landslides in clearcut patches.
Symposium on Recent Development in the Explanation and Prediction of Erosion and Sediment Yield. International Association of Hydrological Sciences, pp. 303–311 Rowbotham, D. N., Dudycha, D. 1998 GIS modelling of slope stability in Phewa Tal watershed, Nepal Geomorphology 26 151 170 Google Scholar Beek, R., Asch, T. 2004 Regional assessment of the effects of land-use change on landslide hazard by means of physically based modelling Nat. Hazards 31 289 304 Google Scholar Westen, C. J., Rengers, N., Terlien, M. T. J., Soeters, R. 1997 Prediction of the occurrence of slope instability phenomenal through GIS-based hazard zonation Geol. Runds. (Int. J. Earth Sci.) 86 404 414 Google Scholar