Environmental Modeling & Assessment

, Volume 14, Issue 2, pp 221–230 | Cite as

Performance Evaluation of the SLEUTH Model in the Shenyang Metropolitan Area of Northeastern China

  • Xiaoqing Wu
  • Yuanman HuEmail author
  • Hong S. He
  • Rencang Bu
  • Jeff Onsted
  • Fengming Xi


Performance evaluation is crucial for the development and improvement of an urban cellular automata model, such as SLEUTH. In this paper, we employed multiple methods for map comparison and model validation to evaluate the simulation performance of the SLEUTH urban growth model in the Shenyang metropolitan area of China. These multiple methods included the relative operating characteristic (ROC) curve statistic, multiple-resolutions error budget, and landscape metrics. They were used to quantitatively examine model performance in terms of the amount and spatial location of urban development, urban spatial pattern and prediction ability. The assessment results showed that SLEUTH performed well in the way of the quantitative simulation of urban growth for this case study. Similar to other urban growth models, however, the simulation accuracy for spatial location of new development at the pixel scale and urban spatial pattern still needs to be improved greatly. These inaccuracies might be attributed to the structure and nature of SLEUTH, local urban development characteristics, and the temporal and spatial scale of its application. Finally, many valuable suggestions had been put forward to improve simulation performance of SLEUTH model for spatial location of urban development in the Shenyang metropolitan area.


Multiple methods SLEUTH model Model performance Urban growth 



We thank Professor Keith C. Clarke, University of California Santa-Barbara, for providing helpful suggestions. This work was jointly sponsored by the Knowledge Innovation Programs of the Chinese Academy of Sciences (KSCX2-SW-133) and the Knowledge Innovation Programs of Institute of Applied Ecology, Chinese Academy of Sciences (O6LYQY1001). We are grateful to the anonymous reviewers for their constructive suggestions and comments.


  1. 1.
    Alberti, M., & Waddell, P. (2000). An integrated urban development and ecological simulation model. Integrated Assessment, 1, 215–227. doi: 10.1023/A:1019140101212.CrossRefGoogle Scholar
  2. 2.
    Batty, M., & Xie, Y. (1994). From cells to cities. Environment and Planning B: Planning and Design, 21, 31–48. doi: 10.1068/b21s031.CrossRefGoogle Scholar
  3. 3.
    Candau, J. C. (2002). Temporal Calibration Sensitivity of the SLEUTH Urban Growth Model. M.A. Thesis, University of California, Santa Barbara, USAGoogle Scholar
  4. 4.
    Carletta, J. (1996). Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2), 249–254.Google Scholar
  5. 5.
    Claggett, P. R., Jantz, C. A., Goetz, S. J., & Bisland, C. (2004). Assessing development pressure in the Chesapeake Bay watershed: an evaluation of two land use change models. Environmental Monitoring and Assessment, 94, 129–146. doi: 10.1023/B:EMAS.0000016884.96098.77.CrossRefGoogle Scholar
  6. 6.
    Clarke, K. C., Hoppen, S., & Gaydos, L. (1997). A self-modifying cellular automaton model of historical urbanization in the San Francisco Bay Area. Environment and Planning B: Planning and Design, 24, 247–261. doi: 10.1068/b240247.CrossRefGoogle Scholar
  7. 7.
    Clarke, K. C., & Dietzel, C. (2004). A Decade of SLEUTHing: Lessons learned from applications of a cellular automaton land use change model. Paper presented at the Joint International Workshop on Integrated Assessment of the Land System: The Future of Land Use. Institute for Environmental Studies, Amsterdam.Google Scholar
  8. 8.
    Dietzel, C. (2004). Spatio-temporal difference in model outputs and parameter space as determined by calibration extent. In P. M. Atkinson, G. M. Foody, S. E. Darby, & F. Wu (Eds.), Geodynamics pp. 233–247. Boca Raton: CRC.Google Scholar
  9. 9.
    Dietzel, C., & Clarke, K. C. (2004). Spatial differences in multi-resolution urban automata modeling. Transactions in GIS, 8(4), 479–492. doi: 10.1111/j.1467-9671.2004.00197.x.CrossRefGoogle Scholar
  10. 10.
    Dietzel, C., & Clarke, K. C. (2006). The effect of disaggregating land use categories in cellular automata during model calibration and forecasting. Computers, Environment and Urban Systems, 30, 78–101. doi: 10.1016/j.compenvurbsys.2005.04.001.CrossRefGoogle Scholar
  11. 11.
    Goldstein, N. C., Candau, J. T., & Clarke, K. C. (2004). Approaches to simulating the “March of Bricks and Mortar”. Computers, Environment and Urban Systems, 28, 125–147. doi: 10.1016/S0198-9715(02)00046-7.CrossRefGoogle Scholar
  12. 12.
    Gustafson, E. J. (1998). Quantifying landscape spatial pattern: what is the state of the art? Ecosystems, 1, 143–156. doi: 10.1007/s100219900011.CrossRefGoogle Scholar
  13. 13.
    Hagen, A. (2002). Multi-method assessment of map similarity. Paper presented at the 5th AGILE Conference on Geographic Information Science, Palma, Spain.Google Scholar
  14. 14.
    Hagen-Zanker, A. (2006). Map comparison methods that simultaneously address overlap and structure. Journal of Geographical Systems, 8(2), 165–185. doi: 10.1007/s10109-006-0024-y.CrossRefGoogle Scholar
  15. 15.
    Herold, M., Goldstein, N. C., & Clarke, K. C. (2003). The spatiotemporal form of urban growth: measurement, analysis and modeling. Remote Sensing of Environment, 86, 286–302. doi: 10.1016/S0034-4257(03)00075-0.CrossRefGoogle Scholar
  16. 16.
    Herold, M., Couclelis, H., & Clarke, K. C. (2005). The role of spatial metrics in the analysis and modeling of urban land use change. Computers, Environment and Urban Systems, 29, 369–399. doi: 10.1016/j.compenvurbsys.2003.12.001.CrossRefGoogle Scholar
  17. 17.
    Jantz, C. A., Goetz, S. J., & Shelley, M. K. (2003). Using the SLEUTH urban growth model to simulate the impacts of future policy scenarios on urban land use in the Baltimore–Washington metropolitan area. Environment and Planning B: Planning and Design, 31, 251–271. doi: 10.1068/b2983.CrossRefGoogle Scholar
  18. 18.
    Jantz, C. A., & Goetz, S. J. (2005). Analysis of scale dependencies in an urban land use change model. International Journal of Geographical Information Science, 19(2), 217–241. doi: 10.1080/13658810410001713425.CrossRefGoogle Scholar
  19. 19.
    Klostermann, R. E. (1999). The what if? Collaborative planning support system. Environment and Planning B: Planning and Design, 26, 393–408. doi: 10.1068/b260393.CrossRefGoogle Scholar
  20. 20.
    Li, X., & Yeh, A. (2000). Modeling sustainable urban development by the integration of constrained cellular automata and GIS. International Journal of Geographical Information Science, 14(2), 131–152. doi: 10.1080/136588100240886.CrossRefGoogle Scholar
  21. 21.
    Li, X., & Yeh, A. (2004). Analyzing spatial restructuring of land use patterns in a fast growing region using remote sensing and GIS. Landscape and Urban Planning, 69, 335–354. doi: 10.1016/j.landurbplan.2003.10.033.CrossRefGoogle Scholar
  22. 22.
    McGarigal, K., & Marks, B. J. (1995). FRAGSTATS: spatial pattern analysis program for quantifying landscape structure. Gen. Tech. Rep. PNW-GTR-351. Portland, OR: U.S. Department of Agriculture, Forest Service, Pacific Northwest Research Station. 122 p.Google Scholar
  23. 23.
    Pontius Jr., R. G. (2000). Quantification error versus location error in comparison of categorical maps. Photogrammetric Engineering and Remote Sensing, 66(8), 1011–1016.Google Scholar
  24. 24.
    Pontiusx Jr., R. G., & Schneider, L. C. (2001). Land-cover change model validation by an ROC method for the Ipswich watershed, Massachusetts, USA. Agriculture, Ecosystems & Environment, 85, 239–248. doi: 10.1016/S0167-8809(01)00187-6.CrossRefGoogle Scholar
  25. 25.
    Pontius Jr., R. G. (2002). Statistical methods to partition effects of quantity and location during comparison of categorical maps at multiple resolutions. Photogrammetric Engineering and Remote Sensing, 68(10), 1041–1049.Google Scholar
  26. 26.
    Pontius Jr., R. G., & Batchu, K. (2003). Using the relative operating characteristic to quantify certainty in prediction of location of land cover change in India. Transactions in GIS, 7(4), 467–484. doi: 10.1111/1467-9671.00159.CrossRefGoogle Scholar
  27. 27.
    Pontius Jr., R. G., & Suedmeyer, B. (2004a). Components of agreement between categorical maps at multiple resolutions. In R. S. Lunetta, & J. G. Lyon (Eds.), Remote Sensing and GIS Accuracy Assessment (pp. 233–251). Boca Raton: CRC.Google Scholar
  28. 28.
    Pontius Jr., R. G., Huffaker, D., & Denman, K. (2004b). Useful techniques of validation for spatially explicit land-change models. Ecological Modelling, 179(4), 445–461. doi: 10.1016/j.ecolmodel.2004.05.010.CrossRefGoogle Scholar
  29. 29.
    Project Gigalopolis: Urban and Land Cover Modeling. URL:
  30. 30.
    Silva, E. A., & Clarke, K. C. (2002). Calibration of the SLEUTH urban growth model for Lisbon and Porto, Portugal. Computers, Environment and Urban Systems, 26, 525–552. doi: 10.1016/S0198-9715(01)00014-X.CrossRefGoogle Scholar
  31. 31.
    Sui, D. Z., & Zeng, H. (2001). Modeling the dynamics of landscape structure in Asia's emerging desakota regions: a case study in Shenzhen. Landscape and Urban Planning, 53, 37–52. doi: 10.1016/S0169-2046(00)00136-5.CrossRefGoogle Scholar
  32. 32.
    Swets, J. A. (1986). Indices of discrimination or diagnostic accuracy: their ROC’s and implied models. Psychological Bulletin, 99, 100–17. doi: 10.1037/0033-2909.99.1.100.CrossRefGoogle Scholar
  33. 33.
    Torrens, P. M., & O’Sullivan, D. (2001). Cellular automata and urban simulation: where do we go from here. Environment and Planning B: Planning and Design, 28, 163–168. doi: 10.1068/b2802ed.CrossRefGoogle Scholar
  34. 34.
    Verburg, P. H., de Nijs, T. C. M., Ritsema van Eck, J. R., Visser, H., & de Jong, K. (2004). A method to analyse neighborhood characteristics of land use patterns. Computers, Environment and Urban Systems, 28(6), 667–690. doi: 10.1016/j.compenvurbsys.2003.07.001.CrossRefGoogle Scholar
  35. 35.
    White, R., & Engelen, G. (1993). Cellular automata and fractal urban form: a cellular modeling approach to the evolution of urban land use patterns. Environment and Planning A, 25, 1175–1199. doi: 10.1068/a251175.CrossRefGoogle Scholar
  36. 36.
    Wu, F. (1998). Simulating urban encroachment on rural land with fuzzy logic controlled cellular automata in a geographical information system. Journal of Environmental Management, 53, 293–308. doi: 10.1006/jema.1998.0195.CrossRefGoogle Scholar
  37. 37.
    Xian, G., Crane, M., & Steinwand, D. (2005). Dynamic modeling of Tampa Bay urban development using parallel computing. Computers & Geosciences, 31(7), 920–928. doi: 10.1016/j.cageo.2005.03.006.CrossRefGoogle Scholar
  38. 38.
    Yang, X., & Lo, C. P. (2003). Modelling urban growth and landscape change in the Atlanta metropolitan area. International Journal of Geographical Information Science, 17(5), 463–488. doi: 10.1080/1365881031000086965.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Xiaoqing Wu
    • 1
  • Yuanman Hu
    • 1
    Email author
  • Hong S. He
    • 1
  • Rencang Bu
    • 1
  • Jeff Onsted
    • 2
  • Fengming Xi
    • 1
  1. 1.Institute of Applied EcologyChinese Academy of SciencesShenyangChina
  2. 2.Department of International Relations and GeographyFlorida International UniversityMiamiUSA

Personalised recommendations