Localizing and quantifying infrastructure damage using class activation mapping approaches

Abstract

Traditional post-disaster assessment of damage heavily relies on expensive geographic information system (GIS) data, especially remote sensing image data. In recent years, social media have become a rich source of disaster information that may be useful in assessing damage at a lower cost. Such information includes text (e.g., tweets) or images posted by eyewitnesses of a disaster. Most of the existing research explores the use of text in identifying situational awareness information useful for disaster response teams. The use of social media images to assess disaster damage is limited. We have recently proposed a novel approach, based on convolutional neural networks and class activation mapping, to locate building damage in a disaster image and to quantify the degree of the damage. In this paper, we study the usefulness of the proposed approach for other categories of infrastructure damage, specifically bridge and road damage, and compare two-class activation mapping approaches in this context. Experimental results show that our proposed approach enables the use of social network images for post-disaster infrastructure damage assessment and provides an inexpensive and feasible alternative to the more expensive GIS approach.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Alam F, Imran M, Ofli F (2017) Image4act: online social media image processing for disaster response. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. ACM, pp 601–604

  2. Ashktorab Z, Brown C, Nandi M, Culotta A (2014) Tweedr: mining twitter to inform disaster response. In: Proceedings of ISCRAM

  3. Attari N, Ofli F, Awad M, Lucas J, Chawla S (2017) Nazr-CNN: fine-grained classification of UAV imagery for damage assessment. In: 2017 IEEE international conference on data science and advanced analytics (DSAA). IEEE, pp 50–59

  4. Bica M, Palen L, Bopp C (2017) Visual representations of disaster. In: Proceedings of the 2017 ACM CSCW. ACM, NY, pp. 1262–1276

  5. Caragea C, Silvescu A, Tapia AH (2016) Identifying informative messages in disasters using convolutional neural networks. In: Proceedings of the ISCRAM, Brazil

  6. Castillo C (2016) Big crisis data: social media in disasters and time-critical situations. Cambridge University Press, Cambridge

    Google Scholar 

  7. Cha YJ, Choi W, Büyüköztürk O (2017) Deep learning-based crack damage detection using convolutional neural networks. Comput Aided Civ Infrastruct Eng 32(5):361–378

    Article  Google Scholar 

  8. Chattopadhyay A, Sarkar A, Howlader P, Balasubramanian VN (2017) Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. CoRR arXiv:abs/1710.11063

  9. Enenkel M, Saenz SM, Dookie DS, Braman L, Obradovich N, Kryvasheyeu Y (2018) Social media data analysis and feedback for advanced disaster risk management. CoRR arXiv:abs/1802.02631

  10. Everingham M, Gool L, Williams CK, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  11. Fan Y, Wen Q, Wang W, Wang P, Li L, Zhang P (2017) Quantifying disaster physical damage using remote sensing data—a technical work flow and case study of the 2014 Ludian Earthquake in china. Int J Disaster Risk Sci 8(4):471–488

    Article  Google Scholar 

  12. Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning, vol 1. MIT Press, Cambridge

    Google Scholar 

  13. Guan X, Chen C (2014) Using social media data to understand and assess disasters. Nat Hazards 74(2):837–850

    Article  Google Scholar 

  14. Gueguen L, Hamid R (2015) Large-scale damage detection using satellite imagery. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1321–1328

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  16. Huang Q, Xiao Y (2015) Geographic situational awareness: mining tweets for disaster preparedness, emergency response, impact, and recovery. ISPRS Int J Geo Inf 4(3):1549–1568

    Article  Google Scholar 

  17. Imran M, Castillo C, Diaz F, Vieweg S (2015) Processing social media messages in mass emergency: a survey. ACM Comput Surv (CSUR) 47(4):67

    Article  Google Scholar 

  18. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  19. Kryvasheyeu Y, Chen H, Obradovich N, Moro E, Van Hentenryck P, Fowler J, Cebrian M (2016) Rapid assessment of disaster damage using social media activity. Sci Adv 2(3):e1500779

    Article  Google Scholar 

  20. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  21. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  22. Li H, Caragea D, Caragea C, Herndon N (2017) Disaster response aided by tweet classification with a domain adaptation approach. J Conting Crisis Manag (JCCM) 26(1):16–27

    Article  Google Scholar 

  23. Liu W, Anguelov D, Erhan D, Szegedy C, Reed SE, Fu C, Berg AC (2016) SSD: single shot multibox detector. In: In European conference on computer vision. Springer, pp 21–37

  24. Li X, Zhang H, Caragea D, Imran M (2018) Localizing and quantifying damage in social media images. In: IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 194–201

  25. Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H (2018) Road damage detection using deep neural networks with images captured through a smartphone. arXiv preprint arXiv:1801.09454

  26. Meier P (2015) Digital humanitarians: how big data is changing the face of humanitarian response. CRC Press Inc, Boca Raton

    Google Scholar 

  27. Nguyen DT, Al-Mannai K, Joty SR, Sajjad H, Imran M, Mitra P (2016) Rapid classification of crisis-related data on social networks using convolutional neural networks. CoRR arXiv:abs/1608.03902

  28. Nguyen DT, Ofli F, Imran M, Mitra P (2017) Damage assessment from social media imagery data during disasters. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. ACM, pp 569–576

  29. Nia KR, Mori G (2017) Building damage assessment using deep learning and ground-level image data. In: 14th conference on computer and robot vision (CRV). IEEE, pp 95–102

  30. Palen L, Anderson KM (2016) Crisis informatics-new data for extraordinary times. Science 353(6296):224–225

    Article  Google Scholar 

  31. Resch B, Usländer F, Havas C (2017) Combining machine-learning topic models and spatiotemporal analysis of social media data for disaster footprint and damage assessment. In: Cartography and geographic information science, pp 1–15

  32. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: IEEE international conference on computer vision (ICCV)

  33. Sen A, Rudra K, Ghosh S (2015) Extracting situational awareness from microblogs during disaster events. In: 7th international conference on communication systems and networks (COMSNETS). IEEE, pp 1–6

  34. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  35. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Computer vision and pattern recognition (CVPR)

  36. Vetrivel A, Gerke M, Kerle N, Nex F, Vosselman G (2018) Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J Photogramm Remote Sens 140:45–59

    Google Scholar 

  37. Xie S, Duan J, Liu S, Dai Q, Liu W, Ma Y, Guo R, Ma C (2016) Crowdsourcing rapid assessment of collapsed buildings early after the earthquake based on aerial remote sensing image: a case study of Yushu earthquake. Remote Sens 8(9):759

    Article  Google Scholar 

  38. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp 3320–3328

  39. Yuan F, Liu R (2018) Feasibility study of using crowdsourcing to identify critical affected areas for rapid damage assessment: Hurricane Matthew case study. Int J Disaster Risk Reduct 28:758–767

    Article  Google Scholar 

  40. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2921–2929

Download references

Acknowledgements

The computing for this project was performed using Amazon Web Services (AWS). We thank the National Science Foundation and Amazon Web Services for support from Grant IIS-1741345, which enabled the research and the computation in this study. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either express or implied, of the National Science Foundation or AWS.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Doina Caragea.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, X., Caragea, D., Zhang, H. et al. Localizing and quantifying infrastructure damage using class activation mapping approaches. Soc. Netw. Anal. Min. 9, 44 (2019). https://doi.org/10.1007/s13278-019-0588-4

Download citation

Keywords

  • Image analysis
  • Convolutional neural networks (CNN)
  • Class activation mapping (CAM) approaches
  • Damage localization
  • Bridge, building, and road damage