Skip to main content

Validating models of one-way land change: an example case of forest insect disturbance



Validation of models of Land Use and Cover Change often involves comparing maps of simulated and reference change. The interpretation of differences between simulated and reference change depends on the characteristics of the process being studied. Our paper focuses on validation of models of one-way land change processes that spread in space.


Our objective is to develop a method for validation of one-way land change models, such that the method provides objective information about the spatial distribution of errors.


Using distance analysis on reference data, we build a baseline model for comparison with simulations. We then simultaneously compare the four maps of reference at initial time, reference at final time, simulation at final time, and baseline at final time. We also use Total Operating Characteristic curves and multiple-resolution map comparison. We illustrate the methods with a simulation of forest insect infestations.


The methods give insights concerning the reference data and the spatial distribution of misses, hits, and false alarms with respect to initial points of infestations. The new methods reveal that the simulations underestimated change near initial points of spread.


The spatial distribution of errors is a topic of land change models that deserves attention. For models of one-way, geographically-spreading processes, we recommend that validation should distinguish between near and far allocation errors with respect to initial points of spread.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Data availability

The datasets generated and/or analysed are available in the Open Science Framework repository, via


  • Batty M, Torrens PM (2005) Modelling and prediction in a complex world. Futures 37:745–766

    Article  Google Scholar 

  • Brown DG, Walker R, Manson S, Seto K (2012) Modeling land use and land cover change. In: Gutman G, Janetos AC, Justice CO et al (eds) Land change science. Springer, Dordrecht, pp 395–409

    Chapter  Google Scholar 

  • Brown DG, Verburg PH, Pontius RG Jr, Lange MD (2013) Opportunities to improve impact, integration, and evaluation of land change models. Curr Opin Environ Sustain 5:452–457

    Article  Google Scholar 

  • Chen H, Pontius RG Jr (2010) Diagnostic tools to evaluate a spatial land change projection along a gradient of an explanatory variable. Landsc Ecol 25:1319–1331

    Article  Google Scholar 

  • Congalton RG (2004) Putting the map back in map accuracy assessment. In: Lunetta RS, Lyon JG (eds) Remote sensing and GIS accuracy assessment. CRC Press, Boca Raton, pp 1–11

    Google Scholar 

  • Cushman SA, Macdonald EA, Landguth EL et al (2017) Multiple-scale prediction of forest loss risk across Borneo. Landsc Ecol 32:1581–1598

    Article  Google Scholar 

  • de Sousa-Neto ER, Gomes L, Nascimento N et al (2018) Land use and land cover transition in Brazil and their effects on greenhouse gas emissions. Soil management and climate change. Academic Press, Cambridge, pp 309–321

    Chapter  Google Scholar 

  • Di Gregorio S, Serra R, Villani M (1997) A cellular automata model of soil bioremediation. Complex Syst 11:31–54

    Google Scholar 

  • ESRI (2015) ArcGIS 10.4.1 for desktop

  • Foody GM (2004) Thematic map comparison: evaluating the statistical significance of differences in classification accuracy. Photogramm Eng Remote Sens 70:627–633

    Article  Google Scholar 

  • Gaudreau J, Perez L, Drapeau P (2016) BorealFireSim: a GIS-based cellular automata model of wildfires for the boreal forest of Quebec in a climate change paradigm. Ecol Inform 32:12–27

    Article  Google Scholar 

  • Hagen-Zanker A (2006) Map comparison methods that simultaneously address overlap and structure. J Geogr Syst 8:165–185

    Article  Google Scholar 

  • Harati S, Perez L, Molowny-Horas R (2020) Integrating neighborhood effect and supervised machine learning techniques to model and simulate forest insect outbreaks in British Columbia, Canada. Forests 11:1–23

    Article  Google Scholar 

  • Hermoso V, Morán-Ordóñez A, Brotons L (2018) Assessing the role of Natura 2000 at maintaining dynamic landscapes in Europe over the last two decades: implications for conservation. Landsc Ecol 33:1447–1460

    Article  Google Scholar 

  • Hijmans RJ (2019) raster: geographic data analysis and modeling. R package version 2.9–5

  • Lambin EF, Geist H, Rindfuss RR (2006) Introduction: local processes with global impacts. In: Lambin EF, Geist H (eds) Land-use and land-cover change. Springer, Berlin, pp 1–8

    Chapter  Google Scholar 

  • Liu Z (2020) TOC Curve Generator.

  • Li Z, Huffman T, Zhang A et al (2012) Spatially locating soil classes within complex soil polygons – Mapping soil capability for agriculture in Saskatchewan Canada. Agric Ecosyst Environ 152:59–67

    Article  Google Scholar 

  • Moulds S, Buytaert W, Mijic A (2015) An open and extensible framework for spatially explicit land use change modelling: the lulcc R package. Geosci Model Dev 8:3215–3229

    Article  Google Scholar 

  • National Research Council (2014) Advancing land change modeling: opportunities and research requirements. National Academies Press, Washington, D.C

    Google Scholar 

  • Natural Resources Canada (2019) Mountain pine beetle. Accessed 2 June 2020

  • Paudel S, Yuan F (2012) Assessing landscape changes and dynamics using patch analysis and GIS modeling. Int J Appl Earth Obs Geoinf 16:66–76

    Article  Google Scholar 

  • Pérez L, Dragićević S, White R (2013) Model testing and assessment: perspectives from a swarm intelligence, agent-based model of forest insect infestations. Comput Environ Urban Syst 39:121–135

    Article  Google Scholar 

  • Perez L, Molowny-Horas R, Harati S (2016) Modelling forest insect outbreaks: efforts towards an inverse approach to model calibration. In: Sauvage S, Sánchez-Pérez JM, Rizzoli AE (eds) Proceddings of the 8th International Congress on Environmental Modelling and Software (iEMSs). Toulouse, France, p 688

  • Pijanowski BC, Pithadia S, Shellito BA, Alexandridis K (2005) Calibrating a neural network-based urban change model for two metropolitan areas of the Upper Midwest of the United States. Int J Geogr Inf Sci 19:197–215

    Article  Google Scholar 

  • Pontius RG Jr (2000) Quantification error versus location error in comparison of categorical maps. Photogramm Eng Remote Sens 66:1011–1016

    Google Scholar 

  • Pontius RG Jr (2002) Statistical methods to partition effects of quantity and location during comparison of categorical maps at multiple resolutions. Photogramm Eng Remote Sensing 68:1041–1050

    Google Scholar 

  • Pontius RG Jr (2018) Criteria to confirm models that simulate deforestation and carbon disturbance. Land 7:1–14

    Article  Google Scholar 

  • Pontius RG Jr, Millones M (2011) Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int J Remote Sens 32:4407–4429

    Article  Google Scholar 

  • Pontius RG Jr, Parmentier B (2014) Recommendations for using the relative operating characteristic (ROC). Landsc Ecol 29:367–382

    Article  Google Scholar 

  • Pontius RG Jr, Si K (2014) The total operating characteristic to measure diagnostic ability for multiple thresholds. Int J Geogr Inf Sci 28:570–583

    Article  Google Scholar 

  • Pontius RG Jr, Huffaker D, Denman K (2004) Useful techniques of validation for spatially explicit land-change models. Ecol Modell 179:445–461

    Article  Google Scholar 

  • Pontius RG Jr, Boersma W, Castella JC et al (2008) Comparing the input, output, and validation maps for several models of land change. Ann Reg Sci 42:11–37

    Article  Google Scholar 

  • Pontius RG Jr, Peethambaram S, Castella JC (2011) Comparison of three maps at multiple resolutions: a case study of land change simulation in Cho Don district, Vietnam. Ann Assoc Am Geogr 101:45–62

    Article  Google Scholar 

  • Pontius RG Jr, Santacruz A, Tayyebi A, et al (2015) TOC: total operating characteristic curve and ROC curve. R package version 0.0–4

  • Pontius RG Jr, Castella J-C, de Nijs T et al (2018) Lessons and challenges in land change modeling derived from synthesis of cross-case comparisons. In: Behnisch M, Meinel G (eds) Possible urban futures: the impact of planners and developers on urban dynamics. Springer International Publishing, Cham, pp 143–164

    Google Scholar 

  • Pontius RG Jr, Walker R, Yao-kumah R et al (2007) Accuracy assessment for a simulation model of Amazonian deforestation. Ann Assoc Am Geogr 97:677–695

    Article  Google Scholar 

  • Province of British Columbia (2015) BC MPB observed cumulative Kill - Vol.12

  • Province of British Columbia (2020) Aerial Overview Survey Methods. Accessed 2 June 2020

  • R Core Team (2019) R: a language and environment for statistical computing

  • Rollins MG, Keane RE, Parsons RA (2004) Mapping fuels and fire regimes using remote sensing, ecosystem simulation, and gradient modeling. Ecol Appl 14:75–95

    Article  Google Scholar 

  • Rykiel EJ (1996) Testing ecological models: the meaning of validation. Ecol Modell 90:229–244.

    Article  Google Scholar 

  • Tobler WR (1970) A computer movie simulating urban growth in the detroit region. Econ Geogr 46:234

    Article  Google Scholar 

  • van Vliet J, Bregt AK, Hagen-Zanker A (2011) Revisiting Kappa to account for change in the accuracy assessment of land-use change models. Ecol Modell 222:1367–1375

    Article  Google Scholar 

  • van Vliet J, Bregt AK, Brown DG et al (2016) A review of current calibration and validation practices in land-change modeling. Environ Model Softw 82:174–182

    Article  Google Scholar 

  • Verburg PH, Kok K, Pontius RG Jr, Veldkamp A (2006) Modeling land-use and land-cover change. In: Lambin EF, Geist H (eds) Land-use and land-cover change. Springer, Berlin, pp 117–135

    Chapter  Google Scholar 

  • White R (2006) Pattern based map comparisons. J Geogr Syst 8:145–164

    Article  Google Scholar 

Download references


We are thankful to the Natural Sciences and Engineering Research Council (NSERC) of Canada for partial support of this study under the Discovery Grant Program awarded to LP, and to the Université de Montréal’s International Affairs Office (IAO) for their financial support through the International Partnership Development program, which allowed the collaboration between researchers from UdeM and CREAF. RMH received financial support from the NEWFOREST (PIRSES-GA-2013-612645) program of the European Union’s Seventh Framework Programme. The United States National Science Foundation supported RGP through its Long Term Ecological Research Network via grant OCE-1637630 for Plum Island Ecosystems. We thank four anonymous reviewers who provided constructive comments on this paper

Author information

Authors and Affiliations


Corresponding author

Correspondence to Saeed Harati.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Harati, S., Perez, L., Molowny-Horas, R. et al. Validating models of one-way land change: an example case of forest insect disturbance. Landscape Ecol 36, 2919–2935 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Model validation
  • One-way change
  • Total operating characteristic
  • Multiple resolution
  • Distance analysis
  • Area partition