Advertisement

Localizing and quantifying infrastructure damage using class activation mapping approaches

  • Xukun Li
  • Doina CarageaEmail author
  • Huaiyu Zhang
  • Muhammad Imran
Original Article
  • 48 Downloads

Abstract

Traditional post-disaster assessment of damage heavily relies on expensive geographic information system (GIS) data, especially remote sensing image data. In recent years, social media have become a rich source of disaster information that may be useful in assessing damage at a lower cost. Such information includes text (e.g., tweets) or images posted by eyewitnesses of a disaster. Most of the existing research explores the use of text in identifying situational awareness information useful for disaster response teams. The use of social media images to assess disaster damage is limited. We have recently proposed a novel approach, based on convolutional neural networks and class activation mapping, to locate building damage in a disaster image and to quantify the degree of the damage. In this paper, we study the usefulness of the proposed approach for other categories of infrastructure damage, specifically bridge and road damage, and compare two-class activation mapping approaches in this context. Experimental results show that our proposed approach enables the use of social network images for post-disaster infrastructure damage assessment and provides an inexpensive and feasible alternative to the more expensive GIS approach.

Keywords

Image analysis Convolutional neural networks (CNN) Class activation mapping (CAM) approaches Damage localization Bridge, building, and road damage 

Notes

Acknowledgements

The computing for this project was performed using Amazon Web Services (AWS). We thank the National Science Foundation and Amazon Web Services for support from Grant IIS-1741345, which enabled the research and the computation in this study. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either express or implied, of the National Science Foundation or AWS.

References

  1. Alam F, Imran M, Ofli F (2017) Image4act: online social media image processing for disaster response. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. ACM, pp 601–604Google Scholar
  2. Ashktorab Z, Brown C, Nandi M, Culotta A (2014) Tweedr: mining twitter to inform disaster response. In: Proceedings of ISCRAMGoogle Scholar
  3. Attari N, Ofli F, Awad M, Lucas J, Chawla S (2017) Nazr-CNN: fine-grained classification of UAV imagery for damage assessment. In: 2017 IEEE international conference on data science and advanced analytics (DSAA). IEEE, pp 50–59Google Scholar
  4. Bica M, Palen L, Bopp C (2017) Visual representations of disaster. In: Proceedings of the 2017 ACM CSCW. ACM, NY, pp. 1262–1276Google Scholar
  5. Caragea C, Silvescu A, Tapia AH (2016) Identifying informative messages in disasters using convolutional neural networks. In: Proceedings of the ISCRAM, BrazilGoogle Scholar
  6. Castillo C (2016) Big crisis data: social media in disasters and time-critical situations. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  7. Cha YJ, Choi W, Büyüköztürk O (2017) Deep learning-based crack damage detection using convolutional neural networks. Comput Aided Civ Infrastruct Eng 32(5):361–378CrossRefGoogle Scholar
  8. Chattopadhyay A, Sarkar A, Howlader P, Balasubramanian VN (2017) Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. CoRR arXiv:abs/1710.11063
  9. Enenkel M, Saenz SM, Dookie DS, Braman L, Obradovich N, Kryvasheyeu Y (2018) Social media data analysis and feedback for advanced disaster risk management. CoRR arXiv:abs/1802.02631
  10. Everingham M, Gool L, Williams CK, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338CrossRefGoogle Scholar
  11. Fan Y, Wen Q, Wang W, Wang P, Li L, Zhang P (2017) Quantifying disaster physical damage using remote sensing data—a technical work flow and case study of the 2014 Ludian Earthquake in china. Int J Disaster Risk Sci 8(4):471–488CrossRefGoogle Scholar
  12. Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning, vol 1. MIT Press, CambridgezbMATHGoogle Scholar
  13. Guan X, Chen C (2014) Using social media data to understand and assess disasters. Nat Hazards 74(2):837–850MathSciNetCrossRefGoogle Scholar
  14. Gueguen L, Hamid R (2015) Large-scale damage detection using satellite imagery. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1321–1328Google Scholar
  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  16. Huang Q, Xiao Y (2015) Geographic situational awareness: mining tweets for disaster preparedness, emergency response, impact, and recovery. ISPRS Int J Geo Inf 4(3):1549–1568CrossRefGoogle Scholar
  17. Imran M, Castillo C, Diaz F, Vieweg S (2015) Processing social media messages in mass emergency: a survey. ACM Comput Surv (CSUR) 47(4):67CrossRefGoogle Scholar
  18. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  19. Kryvasheyeu Y, Chen H, Obradovich N, Moro E, Van Hentenryck P, Fowler J, Cebrian M (2016) Rapid assessment of disaster damage using social media activity. Sci Adv 2(3):e1500779CrossRefGoogle Scholar
  20. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551CrossRefGoogle Scholar
  21. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436CrossRefGoogle Scholar
  22. Li H, Caragea D, Caragea C, Herndon N (2017) Disaster response aided by tweet classification with a domain adaptation approach. J Conting Crisis Manag (JCCM) 26(1):16–27CrossRefGoogle Scholar
  23. Liu W, Anguelov D, Erhan D, Szegedy C, Reed SE, Fu C, Berg AC (2016) SSD: single shot multibox detector. In: In European conference on computer vision. Springer, pp 21–37Google Scholar
  24. Li X, Zhang H, Caragea D, Imran M (2018) Localizing and quantifying damage in social media images. In: IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 194–201Google Scholar
  25. Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H (2018) Road damage detection using deep neural networks with images captured through a smartphone. arXiv preprint arXiv:1801.09454
  26. Meier P (2015) Digital humanitarians: how big data is changing the face of humanitarian response. CRC Press Inc, Boca RatonCrossRefGoogle Scholar
  27. Nguyen DT, Al-Mannai K, Joty SR, Sajjad H, Imran M, Mitra P (2016) Rapid classification of crisis-related data on social networks using convolutional neural networks. CoRR arXiv:abs/1608.03902
  28. Nguyen DT, Ofli F, Imran M, Mitra P (2017) Damage assessment from social media imagery data during disasters. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. ACM, pp 569–576Google Scholar
  29. Nia KR, Mori G (2017) Building damage assessment using deep learning and ground-level image data. In: 14th conference on computer and robot vision (CRV). IEEE, pp 95–102Google Scholar
  30. Palen L, Anderson KM (2016) Crisis informatics-new data for extraordinary times. Science 353(6296):224–225CrossRefGoogle Scholar
  31. Resch B, Usländer F, Havas C (2017) Combining machine-learning topic models and spatiotemporal analysis of social media data for disaster footprint and damage assessment. In: Cartography and geographic information science, pp 1–15CrossRefGoogle Scholar
  32. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: IEEE international conference on computer vision (ICCV)Google Scholar
  33. Sen A, Rudra K, Ghosh S (2015) Extracting situational awareness from microblogs during disaster events. In: 7th international conference on communication systems and networks (COMSNETS). IEEE, pp 1–6Google Scholar
  34. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  35. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Computer vision and pattern recognition (CVPR)Google Scholar
  36. Vetrivel A, Gerke M, Kerle N, Nex F, Vosselman G (2018) Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J Photogramm Remote Sens 140:45–59Google Scholar
  37. Xie S, Duan J, Liu S, Dai Q, Liu W, Ma Y, Guo R, Ma C (2016) Crowdsourcing rapid assessment of collapsed buildings early after the earthquake based on aerial remote sensing image: a case study of Yushu earthquake. Remote Sens 8(9):759CrossRefGoogle Scholar
  38. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp 3320–3328Google Scholar
  39. Yuan F, Liu R (2018) Feasibility study of using crowdsourcing to identify critical affected areas for rapid damage assessment: Hurricane Matthew case study. Int J Disaster Risk Reduct 28:758–767CrossRefGoogle Scholar
  40. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2921–2929Google Scholar

Copyright information

© Springer-Verlag GmbH Austria, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceKansas State UniversityManhattanUSA
  2. 2.Department of StatisticsKansas State UniversityManhattanUSA
  3. 3.Qatar Computing Research InstituteHamad Bin Khalifa UniversityDohaQatar

Personalised recommendations