Calibration and validation of a model of forest disturbance in the Western Ghats, India 1920–1990

Abstract

This paper introduces a new statistical method that we recommend should become standard procedure to quantify the goodness-of-fit of calibration and validation for land-use change models. We present a multiple-resolution Relative Operating Characteristic (ROC) that measures the goodness-of-fit between a reference map, which is considered reality, and a simulated map, which is a model’s output. This proposed ROC is based on: (1) multiple-resolutions, (2) soft classification, (3) sampling without replacement, and 4) explicit separation of factors of quantity versus location of land-use change. We illustrate the method with a case study in India’s Western Ghats, a biodiversity hotspot, where we have maps of cumulative forest disturbance for 1920 and 1990. We use a predictive modeling approach similar to GEOMOD in which we calibrate the model with the map of 1920, then predict the map of 1990, at which point we subject the model to validation. We show that the fit of calibration tends to be much larger than the fit of validation. Thus if a modeler assess a model by goodness-of-fit of calibration only, then the modeler will likely be over confident in the model’s predictive ability.