Analysis of remote sensing imagery for disaster assessment using deep learning: a case study of flooding event

Abstract

This paper proposes a methodology that integrates deep learning and machine learning for automatically assessing damage with limited human input in hundreds of thousands of aerial images. The goal is to develop a system that can help automatically identifying damaged areas in massive amount of data. The main difficulty consists in damaged infrastructure looking very different from when undamaged, likely resulting in an incorrect classification because of their different appearance, and the fact that deep learning and machine learning training sets normally only include undamaged infrastructures. In the proposed method, a deep learning algorithm is firstly used to automatically extract the presence of critical infrastructure from imagery, such as bridges, roads, or houses. However, because damaged infrastructure looks very different from when undamaged, the set of features identified can contain errors. A small portion of the images are then manually labeled if they include damaged areas, or not. Multiple machine learning algorithms are used to learn attribute–value relationships on the labeled data to capture the characteristic features associated with damaged areas. Finally, the trained classifiers are combined to construct an ensemble max-voting classifier. The selected max-voting model is then applied to the remaining unlabeled data to automatically identify images including damaged infrastructure. Evaluation results (85.6% accuracy and 89.09% F1 score) demonstrated the effectiveness of combining deep learning and an ensemble max-voting classifier of multiple machine learning models to analyze aerial images for damage assessment.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    USGS Hazards Data Distribution System (HDDS) Explorer https://hddsexplorer.usgs.gov/.

  2. 2.

    Using AI For Good: A New Data Challenge To Use AI To Triage Natural Disaster Aerial Imagery https://www.forbes.com/sites/kalevleetaru/2018/01/20/using-ai-for-good-a-new-data-challenge-to-use-ai-to-triage-natural-disaster-aerial-imagery.

Abbreviations

ML:

Machine learning

DL:

Deep learning

CAP:

Civilian Air Patrol

AI:

Artificial intelligence

CNN:

Convolutional neural network

RNN:

Recurrent neural network

MLP:

Multilayer perceptron

SVM:

Support vector machine

RBF:

Radial basis function

DT:

Decision tree

NB:

Naive Bayes

KNN:

k-Nearest neighbors

RF:

Random forest

GB:

Gradient boosting

GBC:

Gradient boosting classifier

LR:

Logistic regression

LDA:

Linear discriminant analysis

NN:

Neural networks

USGS:

United States Geological Survey

USGS HDDS:

USGS Hazards Data Distribution System

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283

    Google Scholar 

  2. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185

    MathSciNet  Google Scholar 

  3. Amancio DR, Comin CH, Casanova D, Travieso G, Bruno OM, Rodrigues FA, da Fontoura Costa L (2014) A systematic comparison of supervised classifiers. PLoS ONE 9(4):e94–137

    Article  Google Scholar 

  4. Ben-Hur A, Horn D, Siegelmann HT, Vapnik V (2001) Support vector clustering. J Mach Learn Res 2(Dec):125–137

    MATH  Google Scholar 

  5. Bird S, Klein E, Loper E (2009) Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., Newton

    Google Scholar 

  6. Bishop MC (2006) Pattern recognition and machine learning. Springer, New York

    Google Scholar 

  7. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  Google Scholar 

  8. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    MATH  Article  Google Scholar 

  9. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, Cambridge

    Google Scholar 

  10. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2(2):121–167

    Article  Google Scholar 

  11. Caruana R, Niculescu-Mizil A (2006) An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd international conference on machine learning. ACM, pp 161–168

  12. Cervone G, Sava E, Huang Q, Schnebele E, Harrison J, Waters N (2016) Using Twitter for tasking remote-sensing data collection and damage assessment: 2013 Boulder flood case study. Int J Remote Sens 37(1):100–124

    Article  Google Scholar 

  13. Daelemans W, Van den Bosch A (2005) Memory-based language processing. Cambridge University Press, Cambridge

    Google Scholar 

  14. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. CVPR 2009, IEEE, pp 248–255

  15. Domingos P (2012) A few useful things to know about machine learning. Commun ACM 55(10):78–87

    Article  Google Scholar 

  16. Domingos P (2015) The master algorithm: how the quest for the ultimate learning machine will remake our world. Basic Books, New York

    Google Scholar 

  17. Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29(2–3):103–130

    MATH  Article  Google Scholar 

  18. Dubitzky W, Granzow M, Berrar DP (2007) Fundamentals of data mining in genomics and proteomics. Springer, Berlin

    Google Scholar 

  19. Elman JL (1990) Finding structure in time. Cognit Sci 14(2):179–211

    Article  Google Scholar 

  20. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139

    MathSciNet  MATH  Article  Google Scholar 

  21. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29:1189–1232

    MathSciNet  MATH  Article  Google Scholar 

  22. Gislason PO, Benediktsson JA, Sveinsson JR (2006) Random forests for land cover classification. Pattern Recognit Lett 27(4):294–300

    Article  Google Scholar 

  23. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J et al (2017) Recent advances in convolutional neural networks. Pattern Recognit 77:354

    Article  Google Scholar 

  24. Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier, Amsterdam

    Google Scholar 

  25. Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New York

    Google Scholar 

  26. Kamiński B, Jakubczyk M, Szufel P (2018) A framework for sensitivity analysis of decision trees. Cent Eur J Oper Res 26(1):135–159

    MathSciNet  MATH  Article  Google Scholar 

  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  28. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  29. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  30. Li Z, Wang C, Emrich CT, Guo D (2018) A novel approach to leveraging social media for rapid flood mapping: a case study of the 2015 South Carolina floods. Cartogr Geogr Inf Sci 45(2):97–110

    Article  Google Scholar 

  31. Liong CY, Foo SF (2013) Comparison of linear discriminant analysis and logistic regression for data classification. In: AIP conference proceedings, AIP, vol 1522, pp 1159–1165

  32. Murthy SK (1998) Automatic construction of decision trees from data: a multi-disciplinary survey. Data Mining Knowl Discov 2(4):345–389

    Article  Google Scholar 

  33. Ng AY, Jordan MI (2002) On discriminative vs. generative classifiers: a comparison of logistic regression and Naive Bayes. In: Advances in neural information processing systems, pp 841–848

  34. Opitz DW, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res (JAIR) 11:169–198

    MATH  Article  Google Scholar 

  35. Panteras G, Cervone G (2018) Enhancing the temporal resolution of satellite-based flood extent generation using crowdsourced data for disaster monitoring. Int J Remote Sens 39(5):1459–1474

    Article  Google Scholar 

  36. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

  37. Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Syst Mag 6(3):21–45

    Article  Google Scholar 

  38. Press SJ, Wilson S (1978) Choosing between logistic regression and discriminant analysis. J Am Stat Assoc 73(364):699–705

    MATH  Article  Google Scholar 

  39. Provost F, Kohavi R (1998) Glossary of terms. J Mach Learn 30(2–3):271–274

    Google Scholar 

  40. Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106

    Google Scholar 

  41. Rokach L (2010) Ensemble-based classifiers. Artif Intell Rev 33(1–2):1–39

    Article  Google Scholar 

  42. Russell SJ, Norvig P, Canny JF, Malik JM, Edwards DD (2003) Artificial intelligence: a modern approach, vol 2. Prentice Hall, Upper Saddle River

    Google Scholar 

  43. Salzberg SL (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Mining Knowl Discov 1(3):317–328

    Article  Google Scholar 

  44. Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge

    Google Scholar 

  45. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  46. Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427–437

    Article  Google Scholar 

  47. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A, et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  48. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  49. Wainer J (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets. arXiv preprint arXiv:1606.00930

  50. Weiss GM, Provost F (2003) Learning when training data are costly: the effect of class distribution on tree induction. J Artif Intell Res 19:315–354

    MATH  Article  Google Scholar 

  51. Witten IH, Frank E, Hall MA (2011) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, Burlington

    Google Scholar 

  52. Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8(7):1341–1390

    Article  Google Scholar 

  53. Xiao T, Xia T, Yang Y, Huang C, Wang X (2015) Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2691–2699

  54. Yang L, MacEachren AM, Mitra P, Onorati T (2018) Visually-enabled active deep learning for (geo) text and image classification: a review. ISPRS Int J Geo-Inf 7(2):65

    Article  Google Scholar 

  55. Zhu XX, Tuia D, Mou L, Xia GS, Zhang L, Xu F, Fraundorfer F (2017) Deep learning in remote sensing: a review. arXiv preprint arXiv:1710.03959

Download references

Acknowledgements

This work was partially supported by the Office of Naval Research (ONR) award no. N00014-16-1-2543 (PSU no. 171570) and by the NVIDIA Corporation. We acknowledge Dr. Davide Del Vento from NCAR CISL and Dr. Chuck Pavloski at the Penn State Institute for CyberScience (ICS). The authors wish to thank Elena Sava for useful discussions and for providing the data and initial results relative to the Texas flood event.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Liping Yang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by V. Loia.

Hyperparameter tuning settings and tuned results

Hyperparameter tuning settings and tuned results

In this appendix, we provide some details of the hyperparameter tuning process and results for the multiple machine learning algorithms introduced in Sect. 4.5. Appendix A.1 provides the hyperparameter settings including hyperparameter grids for each ML algorithms we tuned, and Appendix A.2 presents the optimized hyperparameters for each ML algorithm and how many percentage of accuracy each algorithm improved compared with its corresponding baseline model that uses the default hyperparameters in Scikit-Learn (version 0.19.1).

Table 2 Hyperparameter settings

Hyperparameter settings for tuning multiple ML algorithms

While machine learning model parameters are learned during training (such as weights and bias in a neural network), hyperparameters need to be set before training by researchers or data scientists. Taking a neural network as an example, its hyperparameters include the number of hidden layers, number of neurons in each hidden layer, how many iterations, and what is activation function to use. Scikit-Learn has implemented a set of sensible default hyperparameters for all models, but these are not guaranteed to be optimal for a specific problem. The best hyperparameters are in general impossible to be pre-determined, and tuning hyperparameters of a model is where machine learning turns from science into trial-and-error-based engineering.

Hyperparameter tuning relies more on experimental results than theory, and thus, the best practice to determine the optimal settings is to run ML models on many different combinations of hyperparameters and evaluate the performance of each model. Evaluating each model only on the training set can lead to one of the most fundamental problems in machine learning: overfitting. An overfit model may look impressive on the training set, but will most probably not be able to generalize to new unseen data in a real-world application. The standard procedure for hyperparameter optimization avoiding overfitting is through cross-validation.

In this research, we use 5-fold cross-validation while tuning the ML models (as introduced in Sect. A.2, 10-fold is a better setting. However, considering the expensive computation of tuning many combinations of hyperparameters, we chose 5-fold for the tuning process.) We also used all CPU cores (20 cores) available on our Linux Server. Table 2 provides the hyperparameter settings we used to search best parameter for each ML model using Scikit-Learn GridSearch. In Table 2, Combinations refers to the number of hyperparameter combinations and Fits refers to the number of model fits calculated based on the fold (in our case, the fold number = 5) used for cross-validation and the hyperparameter combinations.

Note that we used nine algorithms in total in Sect. 4.5, but we just tune seven ML models here. The reasons are as follows: for NB, there is no hyperparameter to be tuned; LDA has a closed-form solution and therefore has no hyperparameters to be tuned either.

Hyperparameter tuning results and comparison with baseline ML models

Using the hyperparameter settings we provided in Table 2, we tuned the ML models, and Table 3 provides the best parameters from our hyperparameter tuning and also the performance improvement compared with corresponding baseline ML models that used default hyperparameter settings.

In Table 3, in the training accuracy and testing accuracy columns, left value refers to accuracy score of the best model and right refers to that of baseline model. We can see that almost all ML models have better performance when using best hyperparameters from tuning, and some models are improved substantially (e.g., SVM and NN).

Table 3 Hyperparameter tuning results

While comparing the model improvement using best hyperparameters from tuning against baseline line model, we used 10-fold cross-validation. The main reason we set k as 10-fold is based on conclusion in the literature, as Witten et al. (2011) introduced, tests on different datasets, with different learning techniques, have shown that 10 is about the right number of folds to get the best estimate of error.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, L., Cervone, G. Analysis of remote sensing imagery for disaster assessment using deep learning: a case study of flooding event. Soft Comput 23, 13393–13408 (2019). https://doi.org/10.1007/s00500-019-03878-8

Download citation

Keywords

  • Spatiotemporal data
  • Image classification
  • TensorFlow
  • Machine learning
  • Deep learning
  • Damage assessment