Analysis of remote sensing imagery for disaster assessment using deep learning: a case study of flooding event

Abstract

This paper proposes a methodology that integrates deep learning and machine learning for automatically assessing damage with limited human input in hundreds of thousands of aerial images. The goal is to develop a system that can help automatically identifying damaged areas in massive amount of data. The main difficulty consists in damaged infrastructure looking very different from when undamaged, likely resulting in an incorrect classification because of their different appearance, and the fact that deep learning and machine learning training sets normally only include undamaged infrastructures. In the proposed method, a deep learning algorithm is firstly used to automatically extract the presence of critical infrastructure from imagery, such as bridges, roads, or houses. However, because damaged infrastructure looks very different from when undamaged, the set of features identified can contain errors. A small portion of the images are then manually labeled if they include damaged areas, or not. Multiple machine learning algorithms are used to learn attribute–value relationships on the labeled data to capture the characteristic features associated with damaged areas. Finally, the trained classifiers are combined to construct an ensemble max-voting classifier. The selected max-voting model is then applied to the remaining unlabeled data to automatically identify images including damaged infrastructure. Evaluation results (85.6% accuracy and 89.09% F1 score) demonstrated the effectiveness of combining deep learning and an ensemble max-voting classifier of multiple machine learning models to analyze aerial images for damage assessment.

Hyperparameter tuning settings and tuned results

In this appendix, we provide some details of the hyperparameter tuning process and results for the multiple machine learning algorithms introduced in Sect. 4.5. Appendix A.1 provides the hyperparameter settings including hyperparameter grids for each ML algorithms we tuned, and Appendix A.2 presents the optimized hyperparameters for each ML algorithm and how many percentage of accuracy each algorithm improved compared with its corresponding baseline model that uses the default hyperparameters in Scikit-Learn (version 0.19.1).

Table 2 Hyperparameter settings

1.1 Hyperparameter settings for tuning multiple ML algorithms

While machine learning model parameters are learned during training (such as weights and bias in a neural network), hyperparameters need to be set before training by researchers or data scientists. Taking a neural network as an example, its hyperparameters include the number of hidden layers, number of neurons in each hidden layer, how many iterations, and what is activation function to use. Scikit-Learn has implemented a set of sensible default hyperparameters for all models, but these are not guaranteed to be optimal for a specific problem. The best hyperparameters are in general impossible to be pre-determined, and tuning hyperparameters of a model is where machine learning turns from science into trial-and-error-based engineering.

Hyperparameter tuning relies more on experimental results than theory, and thus, the best practice to determine the optimal settings is to run ML models on many different combinations of hyperparameters and evaluate the performance of each model. Evaluating each model only on the training set can lead to one of the most fundamental problems in machine learning: overfitting. An overfit model may look impressive on the training set, but will most probably not be able to generalize to new unseen data in a real-world application. The standard procedure for hyperparameter optimization avoiding overfitting is through cross-validation.

In this research, we use 5-fold cross-validation while tuning the ML models (as introduced in Sect. A.2, 10-fold is a better setting. However, considering the expensive computation of tuning many combinations of hyperparameters, we chose 5-fold for the tuning process.) We also used all CPU cores (20 cores) available on our Linux Server. Table 2 provides the hyperparameter settings we used to search best parameter for each ML model using Scikit-Learn GridSearch. In Table 2, Combinations refers to the number of hyperparameter combinations and Fits refers to the number of model fits calculated based on the fold (in our case, the fold number = 5) used for cross-validation and the hyperparameter combinations.

Note that we used nine algorithms in total in Sect. 4.5, but we just tune seven ML models here. The reasons are as follows: for NB, there is no hyperparameter to be tuned; LDA has a closed-form solution and therefore has no hyperparameters to be tuned either.

1.2 Hyperparameter tuning results and comparison with baseline ML models

Using the hyperparameter settings we provided in Table 2, we tuned the ML models, and Table 3 provides the best parameters from our hyperparameter tuning and also the performance improvement compared with corresponding baseline ML models that used default hyperparameter settings.

In Table 3, in the training accuracy and testing accuracy columns, left value refers to accuracy score of the best model and right refers to that of baseline model. We can see that almost all ML models have better performance when using best hyperparameters from tuning, and some models are improved substantially (e.g., SVM and NN).

Table 3 Hyperparameter tuning results

While comparing the model improvement using best hyperparameters from tuning against baseline line model, we used 10-fold cross-validation. The main reason we set k as 10-fold is based on conclusion in the literature, as Witten et al. (2011) introduced, tests on different datasets, with different learning techniques, have shown that 10 is about the right number of folds to get the best estimate of error.