Comparison of model selection technique performance in predicting the spread of newly invasive species: a case study with Batrachochytrium salamandrivorans Original Paper First Online: 01 March 2018 Abstract
Species distribution models (SDMs) increasingly have been used to anticipate the spread of invasive species. However, these models are powerful conservation tools only if they are biologically relevant, and thus validation of these models is essential. Here, we evaluate four model selection frameworks for their ability to identify a best fit model of spread under low data conditions early in an invasion, specifically testing the efficacy of methods that utilize absence data in addition to presence data in evaluating models. We test this question using a simulation where we generated data with varying confidence in the accuracy of the absence data, as absences in early invasions may become presences in the future, and increasing quantity of observation data to test the models. We create these simulations based on a real-world example of a newly emergent, invasive fungal pathogen,
Batrachochytrium salamandrivorans ( Bsal). The simulation demonstrated that AIC and Likelihood consistently outperform both Kappa and AUC in selecting the true model as the best model when data are limited and absence data are low quality, with AIC providing the most conservative results due to penalties for overparameterization. With these results, we then used these techniques to compare five candidate models for predicting the spread of Bsal. Consistent with the simulation, the best fit model of the candidate models for Bsal was inconsistent across the four metrics. However, AIC, which performed best in the simulation study, suggested that the spread of Bsal into Western Europe was best predicted by a combination of bioclimatic suitability, salamander richness, and number of salamander imports. Our results highlight the difficulty in evaluating predictive models when data are limited and of low quality, as with a newly invasive species, but show that these challenges can be partially addressed with the appropriate model selection approach. Use of this approach is critical as SDMs of invasive species are often used to inform conservation policy and efforts before the invasion spreads, when limited data are available. Keywords AUC AIC Kappa Likelihood Amphibian conservation Chytrid Maxent Species distribution model Electronic supplementary material
The online version of this article (
) contains supplementary material, which is available to authorized users. https://doi.org/10.1007/s10530-018-1690-7 Notes Acknowledgements
We thank the Occidental College Undergraduate Research Center for funding. Thanks to J. McCormack and E. Braker for helpful comments on the manuscript. Thanks also to T. Yap for additional information on previously proposed bioclimatic suitability models.
Allouche O, Tsoar A, Kadmon R (2006) Assessing the accuracy of species distribution models: prevalence, Kappa and the true skill statistic (TSS). J Appl Ecol 43:1223–1232.
https://doi.org/10.1111/j.1365-2664.2006.01214.x CrossRef Google Scholar
Anderson RP, Lew D, Peterson AT (2003) Evaluating predictive models of species’ distributions: criteria for selecting optimal models. Ecol Model 162:211–232.
https://doi.org/10.1016/S0304-3800(02)00349-6 CrossRef Google Scholar
Bales EK, Hyman OJ, Loudon AH et al (2015) Pathogenic chytrid fungus
, but not
, detected on eastern hellbenders. PLoS ONE 10:e0116405.
https://doi.org/10.1371/journal.pone.0116405 CrossRef PubMed PubMedCentral Google Scholar
Bosso L, De Conno C, Russo D (2017) Modelling the risk posed by the Zebra Mussel
: Italy as a case study. Environ Manage 60:304–313.
https://doi.org/10.1007/s00267-017-0882-8 CrossRef PubMed Google Scholar
Burnham KP, Anderson DR (2003) Model selection and multimodel inference. Springer, New York
Carlos-Júnior LA, Barbosa NPU, Moulton TP, Creed JC (2015) Ecological niche model used to examine the distribution of an invasive, non-indigenous coral. Mar Environ Res 103:115–124.
https://doi.org/10.1016/j.marenvres.2014.10.004 CrossRef PubMed Google Scholar
CITES (2016) CITES trade database.
. Accessed 8 May 2017
Diao C, Wang L (2014) Development of an invasive species distribution model with fine-resolution remote sensing. Int J Appl Earth Obs Geoinf 30:65–75.
https://doi.org/10.1016/j.jag.2014.01.015 CrossRef Google Scholar
Elith J, Graham CH, Anderson RP et al (2006) Novel methods improve prediction of species’ distributions from occurrence data. Ecography (Cop) 29:129–151.
https://doi.org/10.1111/j.2006.0906-7590.04596.x CrossRef Google Scholar
Elith J, Kearney M, Phillips S (2010) The art of modelling range-shifting species. Methods Ecol Evol 1:330–342.
https://doi.org/10.1111/j.2041-210X.2010.00036.x CrossRef Google Scholar
Elith J, Phillips SJ, Hastie T et al (2011) A statistical explanation of MaxEnt for ecologists. Divers Distrib 17:43–57.
https://doi.org/10.1111/j.1472-4642.2010.00725.x CrossRef Google Scholar
Feldmeier S, Schefczyk L, Wagner N et al (2016) Exploring the distribution of the spreading lethal salamander chytrid fungus in its invasive range in Europe—a macroecological approach. PLoS ONE 11:e0165682.
https://doi.org/10.1371/journal.pone.0165682 CrossRef PubMed PubMedCentral Google Scholar
Ficetola GF, Thuiller W, Miaud C (2007) Prediction and validation of the potential global distribution of a problematic alien invasive species—the American bullfrog. Divers Distrib 13:476–485.
https://doi.org/10.1111/j.1472-4642.2007.00377.x CrossRef Google Scholar
Fielding AH, Bell JF (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ Conserv 24:38–49
CrossRef Google Scholar
Freeman EA, Moisen GG (2008) A comparison of the performance of threshold criteria for binary classification in terms of predicted prevalence and Kappa. Ecol Model 217:48–58.
https://doi.org/10.1016/j.ecolmodel.2008.05.015 CrossRef Google Scholar
Golicher D, Ford A, Cayuela L, Newton A (2012) Pseudo-absences, pseudo-models and pseudo-niches: pitfalls of model selection based on the area under the curve. Int J Geogr Inf Sci 26:2049–2063.
https://doi.org/10.1080/13658816.2012.719626 CrossRef Google Scholar
Gray MJ, Lewis JP, Nanjappa P et al (2015)
: the North American response and a call for action. PLoS Pathog 11:e1005251.
https://doi.org/10.1371/journal.ppat.1005251 CrossRef PubMed PubMedCentral Google Scholar
Hastie T, Tibshirani R, Friedman JH (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New York
CrossRef Google Scholar
Hijmans RJ, Cameron SE, Parra JL et al (2005) Very high resolution interpolated climate surfaces for global land areas. Int J Climatol 25:1965–1978.
https://doi.org/10.1002/joc.1276 CrossRef Google Scholar
IUCN (2017) The IUCN red list of threatened species.
. Accessed 8 May 2017
Kramer AM, Annis G, Wittmann ME et al (2017) Suitability of Laurentian Great Lakes for invasive species based on global species distribution models and local habitat. Ecosphere 8:e01883.
https://doi.org/10.1002/ecs2.1883 CrossRef Google Scholar
Lobo JM, Jiménez-Valverde A, Real R (2008) AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr 17:145–151.
https://doi.org/10.1111/j.1466-8238.2007.00358.x CrossRef Google Scholar
Martel A, Spitzen-van der Sluijs A, Blooi M et al (2013)
sp. nov. causes lethal chytridiomycosis in amphibians. Proc Natl Acad Sci USA 110:15325–15329.
https://doi.org/10.1073/pnas.1307356110 CrossRef PubMed Google Scholar
Martel A, Blooi M, Adriaensen C et al (2014) Recent introduction of a chytrid fungus endangers Western Palearctic salamanders. Science 346:630–631.
https://doi.org/10.1126/science.1258268 CrossRef PubMed PubMedCentral Google Scholar
Merow C, Smith MJ, Silander JA (2013) A practical guide to MaxEnt for modeling species’ distributions: what it does, and why inputs and settings matter. Ecography (Cop) 36:1058–1069.
https://doi.org/10.1111/j.1600-0587.2013.07872.x CrossRef Google Scholar
Muscarella R, Galante PJ, Soley-Guardia M et al (2014) ENMeval: an R package for conducting spatially independent evaluations and estimating optimal model complexity for MAXENT ecological niche models. Methods Ecol Evol 5:1198–1205.
https://doi.org/10.1111/2041-210X.12261 CrossRef Google Scholar
Nakazawa M (2017) fmsb: functions for medical statistics book with some demographic data. R package version 0.6.1
Nguyen TT, Van Nguyen T, Ziegler T et al (2017) Trade in wild anurans vectors the urodelan pathogen
into Europe. Amphibia Reptil.
https://doi.org/10.1038/srep44443.Martel CrossRef Google Scholar
Parrott JC, Shepack A, Burkart D et al (2016) Survey of pathogenic chytrid fungi (
) in salamanders from three mountain ranges in Europe and the Americas. EcoHealth 14:296–302.
https://doi.org/10.1007/s10393-016-1188-7 CrossRef PubMed Google Scholar
Peterson AT, Vieglais D (2001) Predicting species invasions using ecological niche modeling: new approaches from bioinformatics attack a pressing problem. Bioscience 51:363–371.
https://doi.org/10.1641/0006-3568(2001)051[0363:PSIUEN]2.0.CO;2 CrossRef Google Scholar
Phillips SJ (2008) Transferability, sample selection bias and background data in presence-only modelling: a response to Peterson et al. (2007). Ecography (Cop) 31:272–278.
https://doi.org/10.1111/j.2007.0906-7590.05378.x CrossRef Google Scholar
Phillips SJ, Anderson RP, Schapire RE (2006) Maximum entropy modeling of species geographic distributions. Ecol Model 190:231–259
CrossRef Google Scholar
Phillips SJ, Dudík M, Schapire RE (2010) [Internet] Maxent software for modeling species niches and distributions (Version 3.3.3k). Available from url:
. Accessed 1 May 2017
Phillips SJ, Anderson RP, Dudík M et al (2017) Opening the black box: an open-source release of Maxent. Ecography (Cop) 40:887–893.
https://doi.org/10.1111/ecog.03049 CrossRef Google Scholar
R Development Core Team (2014) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
Richgels KLD, Russell RE, Adams J et al (2016) Spatial variation in risk and consequence of
introduction in the USA. R Soc Open Sci 3:150616.
https://doi.org/10.1098/rsos.150616 CrossRef PubMed PubMedCentral Google Scholar
Robin X, Turck N, Hainard A et al (2011) pROC: an open-source package for R and S + to analyze and compare ROC curves. BMC Bioinformatics 12:77
CrossRef PubMed PubMedCentral Google Scholar
Robinson TP, van Klinken RD, Metternicht G (2010) Comparison of alternative strategies for invasive species distribution modeling. Ecol Model 221:2261–2269.
https://doi.org/10.1016/j.ecolmodel.2010.04.018 CrossRef Google Scholar
Rodríguez JP, Brotons L, Bustamante J, Seoane J (2007) The application of predictive modelling of species distribution to biodiversity conservation. Divers Distrib 13:243–251.
https://doi.org/10.1111/j.1472-4642.2007.00356.x CrossRef Google Scholar
Somodi I, Lepesi N, Botta-Dukát Z (2017) Prevalence dependence in model goodness measures with special emphasis on true skill statistics. Ecol Evol 7:863–872.
https://doi.org/10.1002/ece3.2654 CrossRef PubMed PubMedCentral Google Scholar
Spitzen-van der Sluijs A, Martel A, Asselberghs J et al (2016) Expanding distribution of lethal amphibian fungus
in Europe. Emerg Infect Dis 22:1286–1288.
https://doi.org/10.3201/eid2207.160109 CrossRef PubMed PubMedCentral Google Scholar
Stegen G, Pasmans F, Schmidt BR et al (2017) Drivers of salamander extirpation mediated by
. Nature 544:353–356.
https://doi.org/10.1038/nature22059 CrossRef PubMed Google Scholar
Václavík T, Meentemeyer RK (2009) Invasive species distribution modeling (iSDM): are absence data and dispersal constraints needed to predict actual distributions? Ecol Model 220:3248–3258.
https://doi.org/10.1016/j.ecolmodel.2009.08.013 CrossRef Google Scholar
Václavík T, Kupfer JA, Meentemeyer RK (2012) Accounting for multi-scale spatial autocorrelation improves performance of invasive species distribution modeling (iSDM). J Biogeogr 39:42–55.
https://doi.org/10.1111/j.1365-2699.2011.02589.x CrossRef Google Scholar
Vall-llosera M, Woolnough AP, Anderson D, Cassey P (2017) Improved surveillance for early detection of a potential invasive species: the alien Rose-ringed parakeet
in Australia. Biol Invasions 19:1273–1284.
https://doi.org/10.1007/s10530-016-1332-x CrossRef Google Scholar
VanDerWal J, Falconi L, Januchowski S et al (2015) SDMTools: species distribution modelling tools: tools for processing data associated with species distribution modelling exercises. R package version 1.1–221
Warren DL, Seifert SN (2011) Ecological niche modeling in Maxent: the importance of model complexity and the performance of model selection criteria. Ecol Appl 21:335–342
CrossRef PubMed Google Scholar
Warren DL, Wright AN, Seifert SN, Shaffer HB (2014) Incorporating model complexity and spatial sampling bias into ecological niche models of climate change risks faced by 90 California vertebrate species of concern. Divers Distrib 20:334–343.
https://doi.org/10.1111/ddi.12160 CrossRef Google Scholar
Yap TA, Koo MS, Ambrose RF et al (2015) Averting a North American biodiversity crisis. Science 349:481–482.
https://doi.org/10.1126/science.aab1052 CrossRef PubMed Google Scholar Copyright information
© Springer International Publishing AG, part of Springer Nature 2018