Abstract
The classification of land use and land cover (LULC) from remotely sensed imagery in semi-arid Mediterranean areas is a challenging task due to the fragmentation of the landscape and the diversity of spatial patterns. Recently, the use of deep learning (DL) for image analysis has increased compared to commonly used machine learning (ML) methods. This paper compares the performance of four algorithms, Random Forest (RF), Support Vector Machine (SVM), Multilayer Perceptron (MLP) and Convolutional Network (CNN), using multi-source data, applying an exhaustive optimisation process of the hyperparameters. The usual approach in the optimisation process of a LULC classification model is to keep the best model in terms of accuracy without analysing the rest of the results. In this study, we have analysed such results, discovering noteworthy patterns in a space defined by the mean and standard deviation of the validation accuracy estimated in a 10-fold cross validation (CV). The point distributions in such a space do not appear to be completely random, but show clusters of points that facilitate the discovery of hyperparameter values that tend to increase the mean accuracy and decrease its standard deviation. RF is not the most accurate model, but it is the less sensitive to changes in hyperparameters. Neural Networks, tend to increase commission and omission errors of the less represented classes because their optimisation lead the model to learn better the most frequent classes. On the other hand, RF and MLP prediction layers are the most accurate from a general qualitative point of view.
Article PDF
Similar content being viewed by others
Availability of data and materials
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
References
Ali, A.M., Abouelghar, M., Belal, A., et al.: Crop yield prediction using multi sensors remote sensing (review article). Egypt. J. Remote Sens. Space Sci. 25, 711–716 (2022). https://doi.org/10.1016/j.ejrs.2022.04.006, https://linkinghub.elsevier.com/retrieve/pii/S1110982322000527
Ali, K., Johnson, B.A.: Land-use and land-cover classification in semi-arid areas from medium-resolution remote-sensing imagery: A deep learning approach. Sensors 22, 8750 (2022) https://doi.org/10.3390/s22228750, https://www.mdpi.com/1424-8220/22/22/8750
Alonso-Sarria, F., Valdivieso-Ros, C., Gomariz-Castillo, F.: Isolation forests to evaluate class separability and the representativeness of training and validation areas in land cover classification. Remote Sensing 11, 3000 (2019). https://doi.org/10.3390/rs11243000
Amoakoh, A.O., Aplin, P., Awuah, K.T., et al.: Testing the contribution of multi-source remote sensing features for random forest classification of the greater amanzule tropical peatland. Sensors 21,(2021). https://doi.org/10.3390/s21103399
Atef, I., Ahmed, W., Abdel-Maguid, R.H.: Modelling of land use land cover changes using machine learning and gis techniques: a case study in el-fayoum governorate, egypt. Environ. Monit. Assess. 195, 637 (2023). https://doi.org/10.1007/s10661-023-11224-7, https//link.springer.com/10.1007/s10661-023-11224-7
Baddeley, A., Rubak, E., Turner, R.: Spatial Point Patterns: Methodology and Applications with R. Chapman and Hall/CRC Press, London (2015). https://doi.org/10.1201/b19708
Berberoglu, S., Curran, P.J., Lloyd, C.D., et al.: Texture classification of mediterranean land cover. Int. J. Appl. Earth Obs. Geoinf. 9,(2007). https://doi.org/10.1016/j.jag.2006.11.004
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001). https://doi.org/10.1023/A:1010933404324
Campos, J.C., Sillero, N., Brito, J.C.: Normalized difference water indexes have dissimilar performances in detecting seasonal and permanent water in the sahara-sahel transition zone. J. Hydrol. 464–465 (2012). https://doi.org/10.1016/j.jhydrol.2012.07.042
CARM (2021) Estadística agraria regional. Comunidad Autónoma de la Región de Murcia, accessed: 2021-04-15
Castelo-Cabay, M., Piedra-Fernandez, J.A., Ayala, R.: Deep learning for land use and land cover classification from the ecuadorian paramo. Int. J. Digit. Earth 15, 1001–1017 (2022). https://doi.org/10.1080/17538947.2022.2088872, www.tandfonline.com/doi/full/10.1080/17538947.2022.2088872
Charrad M, Ghazzali N, Boiteau V, et al: Nbclust: An r package for determining the relevant number of clusters in a data set. J Stat. Softw. 61 (2014). https://doi.org/10.18637/jss.v061.i06
Chen, L., Ren, C., Bao, G., et al.: Improved object-based estimation of forest aboveground biomass by integrating lidar data from gedi and icesat-2 with multi-sensor images in a heterogeneous mountainous region. Remote Sens 14, 2743 (2022). https://doi.org/10.3390/rs14122743, www.mdpi.com/2072-4292/14/12/2743
Chen, X.L., Zhao, H.M., Li, P.X., et al.: Remote sensing image-based analysis of the relationship between urban heat island and land use/cover changes. Remote Sens. Environ. 104,(2006). https://doi.org/10.1016/j.rse.2005.11.016
Cánovas-García, F., Alonso-Sarría, F.: Optimal combination of classification algorithms and feature ranking methods for object-based classification of submeter resolution z/i-imaging dmc imagery. Remote Sens 7, 4651–4677 (2015). https://doi.org/10.3390/rs70404651
Council NR: Research Strategies for the U.S. Global Change Research Program. The National Academies Press, (1990). https://doi.org/10.17226/1743, https://www.nap.edu/catalog/1743/research-strategies-for-the-us-global-change-research-program
Davranche, A., Lefebvre, G., Poulin, B.: Wetland monitoring using classification trees and spot-5 seasonal time series. Remote Sens. Environ. 114, 552–562 (2010). https://doi.org/10.1016/j.rse.2009.10.009
Dobrinić, D., Medak, D., Gašparović, M.: Integration of multitemporal sentinel-1 and sentinel-2 imagery for land-cover classification using machine learning methods. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B1-2020:91–98. (2020).https://doi.org/10.5194/isprs-archives-XLIII-B1-2020-91-2020, https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLIII-B1-2020/91/2020/
Dong, P., Chen, Q.: LiDAR Remote Sensing and Applications. CRC Press (2017). https://doi.org/10.4324/9781351233354
Ghayour, L., Neshat, A., Paryani, S., et al.: Performance evaluation of sentinel-2 and landsat 8 oli data for land cover/use classification using a comparison between machine learning algorithms. Remote Sens 13, 1349 (2021). https://doi.org/10.3390/rs13071349, https://www.mdpi.com/2072-4292/13/7/1349
Giménez-Casalduero, F., Gomariz-Castillo, F., Alonso-Sarría, F., et al.: Pinna nobilis in the mar menor coastal lagoon: a story of colonization and uncertainty. Mar. Ecol. Prog. Ser. 652, 77–94 (2020). https://doi.org/10.3354/meps13468
Gomariz-Castillo, F., Alonso-Sarría, F., Cánovas-García, F.: Improving classification accuracy of multi-temporal landsat images by assessing the use of different algorithms, textural and ancillary information for a mediterranean semiarid area from 2000 to 2015. Remote Sens 9, 1058 (2017). https://doi.org/10.3390/rs9101058
Hall-Beyer, M.: Practical guidelines for choosing glcm textures to use in landscape classification tasks over a range of moderate spatial scales. Int. J. Remote Sens. 38, 1312–1338 (2017). https://doi.org/10.1080/01431161.2016.1278314
Haralick, R.: Statistical and structural approaches to texture. Proc. IEEE 67, 786–804 (1979). https://doi.org/10.1109/PROC.1979.11328
Haralick, R.M., Shanmugam, K., Dinstein, I.: Textural features for image classification. IEEE Trans. Syst. Man Cybern 3, 610–621 (1973). https://doi.org/10.1109/TSMC.1973.4309314
Hong, C., Jin, X., Ren, J., et al.: Satellite data indicates multidimensional variation of agricultural production in land consolidation area. Sci. Total Environ. 653,(2019). https://doi.org/10.1016/j.scitotenv.2018.10.415
Hopkins, B., Skellam, J.G.: A new method for determining the type of distribution of plant individuals. Ann. Bot. 18, 213–227 (1954). https://doi.org/10.1093/oxfordjournals.aob.a083391
Hu, Y., Zhang, Q., Zhang, Y., et al.: A deep convolution neural network method for land cover mapping: A case study of qinhuangdao, china. Remote Sensing 10, 2053 (2018). https://doi.org/10.3390/rs10122053
Huete, A.: A soil-adjusted vegetation index (savi). Remote Sens. Environ. 25, 295–309 (1988). https://doi.org/10.1016/0034-4257(88)90106-X
IGN: Plan Nacional de Ortofotgrafía Aérea. (2023) . https://pnoa.ign.es
James, G., Witten, D., Hastie, T., et al: An Introduction to Statistical Learning, vol 103. Springer New York, (2013). https://doi.org/10.1007/978-1-4614-7138-7, http://link.springer.com/10.1007/978-1-4614-7138-7
Kauth, R.J., Thomas, G.S.: The Tasselled Cap – a graphic description of the spectral-temporal development of agricultural crops as seen by LANDSAT. In: for Applications of Remote Sensing TL (ed) LARS Symposia, vol 159. pp. 4B–41–4B–51. Purdue University, West Lafayette, Indiana, (1976)
Klein, I., Gessner, U., Dietz, A.J., et al.: Global waterpack - a 250 m resolution dataset revealing the daily dynamics of global inland water bodies. Remote Sens. Environ. 198, 345–362 (2017). https://doi.org/10.1016/j.rse.2017.06.045
Liaw, A., Wiener, M.: Classification and regression by randomforest. R News 2. (2002)
Liaw, A., Yan, J., Li, W., et al: Package ’randomforest’. R news XXXIX. (2014)
Liu, S., Qi, Z., Li, X., et al.: Integration of convolutional neural networks and object-based post-classification refinement for land use and land cover mapping with optical and sar data. Remote Sensing 11, 690 (2019). https://doi.org/10.3390/rs11060690
Liu, T., Abd-Elrahman, A., Morton, J., et al.: Comparing fully convolutional networks, random forest, support vector machine, and patch-based deep convolutional neural networks for object-based wetland mapping using images from small unmanned aircraft system. GISci Remote Sens 55, 243–264 (2018). https://doi.org/10.1080/15481603.2018.1426091
Lloyd C (2010) Spatial Data Analysis: An Introduction for GIS Users. Oxford University Press
Luca, G.D., Silva, J.M.N., Fazio, S.D., et al.: Integrated use of sentinel-1 and sentinel-2 data and open-source machine learning algorithms for land cover mapping in a mediterranean region. Eur. J Remote Sens. 55, 52–70 (2022). https://doi.org/10.1080/22797254.2021.2018667, https://www.tandfonline.com/doi/full/10.1080/22797254.2021.2018667
Martínez, J., Esteve, M., Martínez-Paz, J., et al: Simulating management options and scenarios to control nutrient load to mar menor, southeast spain. Transitional Waters Monographs TWM, Transit Waters Monogr 1. (2007). https://doi.org/10.1285/i18252273v1n1p53
Masiza, W., Chirima, J.G., Hamandawana, H., et al.: Enhanced mapping of a smallholder crop farming landscape through image fusion and model stacking. Int. J. Remote Sens. 41, 8739–8756 (2020). https://doi.org/10.1080/01431161.2020.1783017
Mason, P.J., Manton, M., Harrison, D.E., et al : The second report on the adequacy of the global observing systems for climate in support of the unfccc. GCOS Rep 82. (2003). https://library.wmo.int/doc_num.php?explnum_id=3931
Mostafiz, C., Chang, N.B.: Tasseled cap transformation for assessing hurricane landfall impact on a coastal watershed. Int. J. Appl. Earth Obs. Geoinf. 73, 736–745 (2018). https://doi.org/10.1016/j.jag.2018.08.015
Periasamy, S.: Significance of dual polarimetric synthetic aperture radar in biomass retrieval: An attempt on sentinel-1. Remote Sens. Environ. 217, 537–549 (2018). https://doi.org/10.1016/j.rse.2018.09.003
Qin, R., Liu, T.: A review of landcover classification with very-high resolution remotely sensed optical images-analysis unit, model scalability and transferability. Remote Sensing 14, 646 (2022). https://doi.org/10.3390/rs14030646
Reichstein, M., Camps-Valls, G., Stevens, B., et al.: Deep learning and process understanding for data-driven earth system science. Nature 566, 195–204 (2019). https://doi.org/10.1038/s41586-019-0912-1
Rossiter, D.: Statistical methods for accuracy assesment of classified thematic maps. Tech. rep., Department of Earth Systems Analysis International Institute for Geo-information Science & Earth Observation (ITC), Enschede (NL). (2004)
Rouse, J.W., Haas, R.H., Schell, J.A., et al: Monitoring the vernal advancement and retrogradation (green wave effect) of natural vegetation. Progress Report RSC 1978-1 (1973) https://ntrs.nasa.gov/citations/19740022555
Singh, P.G., Bordu, N., Singh, D., et al.: Permuted spectral and permuted spectral-spatial cnn models for polsar-multispectral data based land cover classification. Int. J. Remote Sens. 42, 1096–1120 (2021). https://doi.org/10.1080/01431161.2020.1823041, https://www.tandfonline.com/doi/full/10.1080/01431161.2020.1823041
Tsai, M.D., Tseng, K.W., Lai, C.C., et al.: Exploring airborne lidar and aerial photographs using machine learning for land cover classification. Remote Sensing 15, 2280 (2023). https://doi.org/10.3390/rs15092280, https://www.mdpi.com/2072-4292/15/9/2280
Valdivieso-Ros, C., Alonso-Sarria, F., Gomariz-Castillo, F.: Effect of different atmospheric correction algorithms on sentinel-2 imagery classification accuracy in a semiarid mediterranean area. Remote Sensing 13, 1770 (2021). https://doi.org/10.3390/rs13091770, https://www.mdpi.com/2072-4292/13/9/1770
Valdivieso-Ros, C., Alonso-Sarria, F., Gomariz-Castillo, F.: Effect of the synergetic use of sentinel-1, sentinel-2, lidar and derived data in land cover classification of a semiarid mediterranean area using machine learning algorithms. Remote Sensing 15, 312 (2023). https://doi.org/10.3390/rs15020312, https://www.mdpi.com/2072-4292/15/2/312
Vanhellemont, Q.: Adaptation of the dark spectrum fitting atmospheric correction for aquatic applications of the landsat and sentinel-2 archives. Remote Sens. Environ. 225, 175–192 (2019). https://doi.org/10.1016/j.rse.2019.03.010
Vanhellemont, Q., Ruddick, K. Acolite for sentinel-2: Aquatic applications of msi imagery. Paper presented at the Living Planet Symposium, Proceedings of the conference held 9-13 May 2016. (2016)
Vanhellemont, Q., Ruddick, K.: Atmospheric correction of metre-scale optical satellite data for inland and coastal water applications. Remote Sens. Environ. 216, 586–597 (2018). https://doi.org/10.1016/j.rse.2018.07.015
Xu, H.: Modification of normalised difference water index (ndwi) to enhance open water features in remotely sensed imagery. Int. J. Remote Sens. 27, 3025–3033 (2006). https://doi.org/10.1080/01431160600589179
Yang, X., Qin, Q., Grussenmeyer, P., et al.: Urban surface water body detection with suppressed built-up noise based on water indices from sentinel-2 msi imagery. Remote Sens. Environ. 219, 259–270 (2018). https://doi.org/10.1016/j.rse.2018.09.016
YiLan L, RuTong Z (2022) clustertend: Check the Clustering Tendency. https://CRAN.R-project.org/package=clustertend, r package version 1.4
Yuh, Y.G., Tracz, W., Matthews, H.D., et al.: Application of machine learning approaches for land cover monitoring in northern cameroon. Eco. Inform. 74, 101955 (2023). https://doi.org/10.1016/j.ecoinf.2022.101955, https://linkinghub.elsevier.com/retrieve/pii/S1574954122004058
Zhang, H., Xu, R.: Exploring the optimal integration levels between sar and optical data for better urban land cover mapping in the pearl river delta. Int. J. Appl. Earth Obs. Geoinf. 64, 87–95 (2018). https://doi.org/10.1016/j.jag.2017.08.013
Acknowledgements
We thank the referees and editors for their constructive feedback regarding the initial version of the manuscript.
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was supported by the Spanish Agencia Estatal de Investigación (Grant number TED2021-131131B-I00). Carmen Valdivieso-Ros is grateful for the financing of the pre-doctoral research by the Ministerio de Ciencia, Innovación y Universidades from the Government of Spain (FPU18/01447).
Author information
Authors and Affiliations
Contributions
Conceptualization: Francisco Alonso-Sarría and Carmen Valdivieso-Ros; Methodology: Francisco Alonso-Sarría and Carmen Valdivieso-Ros; Formal analysis and investigation: Francisco Alonso-Sarría, Carmen Valdivieso-Ros and Francisco Gomariz-Castillo; Writing - original draft preparation: Francisco Alonso-Sarría, Carmen Valdivieso-Ros and Francisco Gomariz-Castillo; Writing - review and editing: Francisco Alonso-Sarría, Carmen Valdivieso-Ros and Francisco Gomariz-Castillo; Funding acquisition: Francisco Alonso-Sarría and Francisco Gomariz-Castillo; Resources: Francisco Alonso-Sarría and Francisco Gomariz-Castillo; Software: Francisco Alonso-Sarría and Francisco Gomariz-Castillo; Data curation: Carmen Valdivieso-Ros; Supervision: Francisco Alonso-Sarría. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Ethics approval
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Alonso-Sarría, F., Valdivieso-Ros, C. & Gomariz-Castillo, F. Analysis of the hyperparameter optimisation of four machine learning satellite imagery classification methods. Comput Geosci (2024). https://doi.org/10.1007/s10596-024-10285-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10596-024-10285-y
Keywords
- Machine learning
- LULC
- Convolutional neuronal networks
- Random forest
- Support vector machines
- Hyperparameter optimisation