Comparison of model selection technique performance in predicting the spread of newly invasive species: a case study with Batrachochytrium salamandrivorans

Abstract

Species distribution models (SDMs) increasingly have been used to anticipate the spread of invasive species. However, these models are powerful conservation tools only if they are biologically relevant, and thus validation of these models is essential. Here, we evaluate four model selection frameworks for their ability to identify a best fit model of spread under low data conditions early in an invasion, specifically testing the efficacy of methods that utilize absence data in addition to presence data in evaluating models. We test this question using a simulation where we generated data with varying confidence in the accuracy of the absence data, as absences in early invasions may become presences in the future, and increasing quantity of observation data to test the models. We create these simulations based on a real-world example of a newly emergent, invasive fungal pathogen, Batrachochytrium salamandrivorans (Bsal). The simulation demonstrated that AIC and Likelihood consistently outperform both Kappa and AUC in selecting the true model as the best model when data are limited and absence data are low quality, with AIC providing the most conservative results due to penalties for overparameterization. With these results, we then used these techniques to compare five candidate models for predicting the spread of Bsal. Consistent with the simulation, the best fit model of the candidate models for Bsal was inconsistent across the four metrics. However, AIC, which performed best in the simulation study, suggested that the spread of Bsal into Western Europe was best predicted by a combination of bioclimatic suitability, salamander richness, and number of salamander imports. Our results highlight the difficulty in evaluating predictive models when data are limited and of low quality, as with a newly invasive species, but show that these challenges can be partially addressed with the appropriate model selection approach. Use of this approach is critical as SDMs of invasive species are often used to inform conservation policy and efforts before the invasion spreads, when limited data are available.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

Reproduced with permission from Spitzen-van der Sluijs et al. (2016) and RAVON (2017)

References

  1. Allouche O, Tsoar A, Kadmon R (2006) Assessing the accuracy of species distribution models: prevalence, Kappa and the true skill statistic (TSS). J Appl Ecol 43:1223–1232. https://doi.org/10.1111/j.1365-2664.2006.01214.x

    Article  Google Scholar 

  2. Anderson RP, Lew D, Peterson AT (2003) Evaluating predictive models of species’ distributions: criteria for selecting optimal models. Ecol Model 162:211–232. https://doi.org/10.1016/S0304-3800(02)00349-6

    Article  Google Scholar 

  3. Bales EK, Hyman OJ, Loudon AH et al (2015) Pathogenic chytrid fungus Batrachochytrium dendrobatidis, but not B. salamandrivorans, detected on eastern hellbenders. PLoS ONE 10:e0116405. https://doi.org/10.1371/journal.pone.0116405

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  4. Baums J (2015a) project_maxent.R. https://github.com/johnbaums/things/blob/master/R/project_maxent.R. Accessed 1 Jan 2017

  5. Baums J (2015b) parse_lambdas.R. https://github.com/johnbaums/things/blob/master/R/parse_lambdas.R. Accessed 1 Jan 2017

  6. Bosso L, De Conno C, Russo D (2017) Modelling the risk posed by the Zebra Mussel Dreissena polymorpha: Italy as a case study. Environ Manage 60:304–313. https://doi.org/10.1007/s00267-017-0882-8

    Article  PubMed  Google Scholar 

  7. Burnham KP, Anderson DR (2003) Model selection and multimodel inference. Springer, New York

    Google Scholar 

  8. Carlos-Júnior LA, Barbosa NPU, Moulton TP, Creed JC (2015) Ecological niche model used to examine the distribution of an invasive, non-indigenous coral. Mar Environ Res 103:115–124. https://doi.org/10.1016/j.marenvres.2014.10.004

    Article  PubMed  CAS  Google Scholar 

  9. CITES (2016) CITES trade database. https://trade.cites.org/. Accessed 8 May 2017

  10. Diao C, Wang L (2014) Development of an invasive species distribution model with fine-resolution remote sensing. Int J Appl Earth Obs Geoinf 30:65–75. https://doi.org/10.1016/j.jag.2014.01.015

    Article  Google Scholar 

  11. Elith J, Graham CH, Anderson RP et al (2006) Novel methods improve prediction of species’ distributions from occurrence data. Ecography (Cop) 29:129–151. https://doi.org/10.1111/j.2006.0906-7590.04596.x

    Article  Google Scholar 

  12. Elith J, Kearney M, Phillips S (2010) The art of modelling range-shifting species. Methods Ecol Evol 1:330–342. https://doi.org/10.1111/j.2041-210X.2010.00036.x

    Article  Google Scholar 

  13. Elith J, Phillips SJ, Hastie T et al (2011) A statistical explanation of MaxEnt for ecologists. Divers Distrib 17:43–57. https://doi.org/10.1111/j.1472-4642.2010.00725.x

    Article  Google Scholar 

  14. Feldmeier S, Schefczyk L, Wagner N et al (2016) Exploring the distribution of the spreading lethal salamander chytrid fungus in its invasive range in Europe—a macroecological approach. PLoS ONE 11:e0165682. https://doi.org/10.1371/journal.pone.0165682

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  15. Ficetola GF, Thuiller W, Miaud C (2007) Prediction and validation of the potential global distribution of a problematic alien invasive species—the American bullfrog. Divers Distrib 13:476–485. https://doi.org/10.1111/j.1472-4642.2007.00377.x

    Article  Google Scholar 

  16. Fielding AH, Bell JF (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ Conserv 24:38–49

    Article  Google Scholar 

  17. Freeman EA, Moisen GG (2008) A comparison of the performance of threshold criteria for binary classification in terms of predicted prevalence and Kappa. Ecol Model 217:48–58. https://doi.org/10.1016/j.ecolmodel.2008.05.015

    Article  Google Scholar 

  18. Golicher D, Ford A, Cayuela L, Newton A (2012) Pseudo-absences, pseudo-models and pseudo-niches: pitfalls of model selection based on the area under the curve. Int J Geogr Inf Sci 26:2049–2063. https://doi.org/10.1080/13658816.2012.719626

    Article  Google Scholar 

  19. Gray MJ, Lewis JP, Nanjappa P et al (2015) Batrachochytrium salamandrivorans: the North American response and a call for action. PLoS Pathog 11:e1005251. https://doi.org/10.1371/journal.ppat.1005251

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  20. Hastie T, Tibshirani R, Friedman JH (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New York

    Google Scholar 

  21. Hijmans RJ, Cameron SE, Parra JL et al (2005) Very high resolution interpolated climate surfaces for global land areas. Int J Climatol 25:1965–1978. https://doi.org/10.1002/joc.1276

    Article  Google Scholar 

  22. IUCN (2017) The IUCN red list of threatened species. https://www.iucn.org/. Accessed 8 May 2017

  23. Kramer AM, Annis G, Wittmann ME et al (2017) Suitability of Laurentian Great Lakes for invasive species based on global species distribution models and local habitat. Ecosphere 8:e01883. https://doi.org/10.1002/ecs2.1883

    Article  Google Scholar 

  24. Lobo JM, Jiménez-Valverde A, Real R (2008) AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr 17:145–151. https://doi.org/10.1111/j.1466-8238.2007.00358.x

    Article  Google Scholar 

  25. Martel A, Spitzen-van der Sluijs A, Blooi M et al (2013) Batrachochytrium salamandrivorans sp. nov. causes lethal chytridiomycosis in amphibians. Proc Natl Acad Sci USA 110:15325–15329. https://doi.org/10.1073/pnas.1307356110

    Article  PubMed  Google Scholar 

  26. Martel A, Blooi M, Adriaensen C et al (2014) Recent introduction of a chytrid fungus endangers Western Palearctic salamanders. Science 346:630–631. https://doi.org/10.1126/science.1258268

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  27. Merow C, Smith MJ, Silander JA (2013) A practical guide to MaxEnt for modeling species’ distributions: what it does, and why inputs and settings matter. Ecography (Cop) 36:1058–1069. https://doi.org/10.1111/j.1600-0587.2013.07872.x

    Article  Google Scholar 

  28. Muscarella R, Galante PJ, Soley-Guardia M et al (2014) ENMeval: an R package for conducting spatially independent evaluations and estimating optimal model complexity for MAXENT ecological niche models. Methods Ecol Evol 5:1198–1205. https://doi.org/10.1111/2041-210X.12261

    Article  Google Scholar 

  29. Nakazawa M (2017) fmsb: functions for medical statistics book with some demographic data. R package version 0.6.1

  30. Nguyen TT, Van Nguyen T, Ziegler T et al (2017) Trade in wild anurans vectors the urodelan pathogen Batrachochytrium salamandrivorans into Europe. Amphibia Reptil. https://doi.org/10.1038/srep44443.Martel

    Article  Google Scholar 

  31. Parrott JC, Shepack A, Burkart D et al (2016) Survey of pathogenic chytrid fungi (Batrachochytrium dendrobatidis and B. salamandrivorans) in salamanders from three mountain ranges in Europe and the Americas. EcoHealth 14:296–302. https://doi.org/10.1007/s10393-016-1188-7

    Article  PubMed  Google Scholar 

  32. Peterson AT, Vieglais D (2001) Predicting species invasions using ecological niche modeling: new approaches from bioinformatics attack a pressing problem. Bioscience 51:363–371. https://doi.org/10.1641/0006-3568(2001)051[0363:PSIUEN]2.0.CO;2

    Article  Google Scholar 

  33. Phillips SJ (2008) Transferability, sample selection bias and background data in presence-only modelling: a response to Peterson et al. (2007). Ecography (Cop) 31:272–278. https://doi.org/10.1111/j.2007.0906-7590.05378.x

    Article  Google Scholar 

  34. Phillips SJ, Anderson RP, Schapire RE (2006) Maximum entropy modeling of species geographic distributions. Ecol Model 190:231–259

    Article  Google Scholar 

  35. Phillips SJ, Dudík M, Schapire RE (2010) [Internet] Maxent software for modeling species niches and distributions (Version 3.3.3k). Available from url: http://biodiversityinformatics.amnh.org/open_source/maxent/. Accessed 1 May 2017

  36. Phillips SJ, Anderson RP, Dudík M et al (2017) Opening the black box: an open-source release of Maxent. Ecography (Cop) 40:887–893. https://doi.org/10.1111/ecog.03049

    Article  Google Scholar 

  37. R Development Core Team (2014) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/

  38. RAVON (2017) RAVON. http://www.ravon.nl/English/Research/Bsal/tabid/3820/Default.aspx. Accessed 13 Jan 2018

  39. Richgels KLD, Russell RE, Adams J et al (2016) Spatial variation in risk and consequence of Batrachochytrium salamandrivorans introduction in the USA. R Soc Open Sci 3:150616. https://doi.org/10.1098/rsos.150616

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  40. Robin X, Turck N, Hainard A et al (2011) pROC: an open-source package for R and S + to analyze and compare ROC curves. BMC Bioinformatics 12:77

    Article  PubMed  PubMed Central  Google Scholar 

  41. Robinson TP, van Klinken RD, Metternicht G (2010) Comparison of alternative strategies for invasive species distribution modeling. Ecol Model 221:2261–2269. https://doi.org/10.1016/j.ecolmodel.2010.04.018

    Article  Google Scholar 

  42. Rodríguez JP, Brotons L, Bustamante J, Seoane J (2007) The application of predictive modelling of species distribution to biodiversity conservation. Divers Distrib 13:243–251. https://doi.org/10.1111/j.1472-4642.2007.00356.x

    Article  Google Scholar 

  43. Somodi I, Lepesi N, Botta-Dukát Z (2017) Prevalence dependence in model goodness measures with special emphasis on true skill statistics. Ecol Evol 7:863–872. https://doi.org/10.1002/ece3.2654

    Article  PubMed  PubMed Central  Google Scholar 

  44. Spitzen-van der Sluijs A, Martel A, Asselberghs J et al (2016) Expanding distribution of lethal amphibian fungus Batrachochytrium salamandrivorans in Europe. Emerg Infect Dis 22:1286–1288. https://doi.org/10.3201/eid2207.160109

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  45. Stegen G, Pasmans F, Schmidt BR et al (2017) Drivers of salamander extirpation mediated by Batrachochytrium salamandrivorans. Nature 544:353–356. https://doi.org/10.1038/nature22059

    Article  PubMed  CAS  Google Scholar 

  46. Václavík T, Meentemeyer RK (2009) Invasive species distribution modeling (iSDM): are absence data and dispersal constraints needed to predict actual distributions? Ecol Model 220:3248–3258. https://doi.org/10.1016/j.ecolmodel.2009.08.013

    Article  Google Scholar 

  47. Václavík T, Kupfer JA, Meentemeyer RK (2012) Accounting for multi-scale spatial autocorrelation improves performance of invasive species distribution modeling (iSDM). J Biogeogr 39:42–55. https://doi.org/10.1111/j.1365-2699.2011.02589.x

    Article  Google Scholar 

  48. Vall-llosera M, Woolnough AP, Anderson D, Cassey P (2017) Improved surveillance for early detection of a potential invasive species: the alien Rose-ringed parakeet Psittacula krameri in Australia. Biol Invasions 19:1273–1284. https://doi.org/10.1007/s10530-016-1332-x

    Article  Google Scholar 

  49. VanDerWal J, Falconi L, Januchowski S et al (2015) SDMTools: species distribution modelling tools: tools for processing data associated with species distribution modelling exercises. R package version 1.1–221

  50. Warren DL, Seifert SN (2011) Ecological niche modeling in Maxent: the importance of model complexity and the performance of model selection criteria. Ecol Appl 21:335–342

    Article  PubMed  Google Scholar 

  51. Warren DL, Wright AN, Seifert SN, Shaffer HB (2014) Incorporating model complexity and spatial sampling bias into ecological niche models of climate change risks faced by 90 California vertebrate species of concern. Divers Distrib 20:334–343. https://doi.org/10.1111/ddi.12160

    Article  Google Scholar 

  52. Yap TA, Koo MS, Ambrose RF et al (2015) Averting a North American biodiversity crisis. Science 349:481–482. https://doi.org/10.1126/science.aab1052

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgements

We thank the Occidental College Undergraduate Research Center for funding. Thanks to J. McCormack and E. Braker for helpful comments on the manuscript. Thanks also to T. Yap for additional information on previously proposed bioclimatic suitability models.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Amanda J. Zellmer.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 3120 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Katz, T.S., Zellmer, A.J. Comparison of model selection technique performance in predicting the spread of newly invasive species: a case study with Batrachochytrium salamandrivorans. Biol Invasions 20, 2107–2119 (2018). https://doi.org/10.1007/s10530-018-1690-7

Download citation

Keywords

  • AUC
  • AIC
  • Kappa
  • Likelihood
  • Amphibian conservation
  • Chytrid
  • Maxent
  • Species distribution model