Advertisement

Detecting Natural Disasters, Damage, and Incidents in the Wild

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)

Abstract

Responding to natural disasters, such as earthquakes, floods, and wildfires, is a laborious task performed by on-the-ground emergency responders and analysts. Social media has emerged as a low-latency data source to quickly understand disaster situations. While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes. However, no large-scale image datasets for incident detection exists. In this work, we present the Incidents Dataset, which contains 446,684 images annotated by humans that cover 43 incidents across a variety of scenes. We employ a baseline classification model that mitigates false-positive errors and we perform image filtering experiments on millions of social media images from Flickr and Twitter. Through these experiments, we show how the Incidents Dataset can be used to detect images with incidents in the wild. Code, data, and models are available online at http://incidentsdataset.csail.mit.edu.

Keywords

Image classification Visual recognition Scene understanding Image dataset Social media Disaster analysis Incident detection 

Notes

Acknowledgments

This work is supported by the CSAIL-QCRI collaboration project and RTI2018-095232-B-C22 grant from the Spanish Ministry of Science, Innovation and Universities.

References

  1. 1.
    Abavisani, M., Wu, L., Hu, S., Tetreault, J., Jaimes, A.: Multimodal categorization of crisis events in social media. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  2. 2.
    Abdur Rehman, N., Saif, U., Chunara, R.: Deep landscape features for improving vector-borne disease prediction. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  3. 3.
    Alam, F., Ofli, F., Imran, M.: Processing social media images by combining human and machine computing during crises. Int. J. Hum. Comput. Interact. 34(4), 311–327 (2018)CrossRefGoogle Scholar
  4. 4.
    Arietta, S.M., Efros, A.A., Ramamoorthi, R., Agrawala, M.: City forensics: using visual elements to predict non-visual city attributes. IEEE Trans. Visual Comput. Graphics 20(12), 2624–2633 (2014)CrossRefGoogle Scholar
  5. 5.
    Attari, N., Ofli, F., Awad, M., Lucas, J., Chawla, S.: Nazr-CNN: object detection and fine-grained classification in crowdsourced UAV images. In: IEEE International Conference on Data Science and Advanced Analytics (DSAA) (2016)Google Scholar
  6. 6.
    Avvenuti, M., Cresci, S., Marchetti, A., Meletti, C., Tesconi, M.: Ears (earthquake alert and report system) a real time decision support system for earthquake crisis management. In: SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1749–1758. ACM (2014)Google Scholar
  7. 7.
    Ben-Haim, Z., et al.: Inundation modeling in data scarce regions. In: NeurIPS Workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response (2019)Google Scholar
  8. 8.
    Bischke, B., Helber, P., Schulze, C., Venkat, S., Dengel, A., Borth, D.: The multimedia satellite task at mediaeval 2017: emergency response for flooding events. In: Proceedings of the MediaEval 2017 Workshop, pp. 1–3 (2017)Google Scholar
  9. 9.
    Bischke, B., Helber, P., Zhao, Z., De Bruijn, J., Borth, D.: The multimedia satellite task at mediaeval 2018: emergency response for flooding events. In: Proceedings of the MediaEval 2018 Workshop, pp. 1–3 (2018)Google Scholar
  10. 10.
    Bonafilia, D., Gill, J., Basu, S., Yang, D.: Building high resolution maps for humanitarian aid and development with weakly- and semi-supervised learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  11. 11.
    Can, G., Benkhedda, Y., Gatica-Perez, D.: Ambiance in social media venues: visual cue interpretation by machines and crowds. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2018)Google Scholar
  12. 12.
    Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997).  https://doi.org/10.1023/A:1007379606734MathSciNetCrossRefGoogle Scholar
  13. 13.
    Castillo, C.: Big Crisis Data. Cambridge University Press, New York (2016)CrossRefGoogle Scholar
  14. 14.
    Chehata, N., Orny, C., Boukir, S., Guyon, D., Wigneron, J.: Object-based change detection in wind storm-damaged forest using high-resolution multispectral images. Int. J. Remote Sens. 35(13), 4758–4777 (2014)CrossRefGoogle Scholar
  15. 15.
    Chen, T., Lu, D., Kan, M.Y., Cui, P.: Understanding and classifying image tweets. In: ACM International Conference on Multimedia, pp. 781–784 (2013)Google Scholar
  16. 16.
    Daly, S., Thom, J.: Mining and classifying image posts on social media to analyse fires. In: 13th International Conference on Information Systems for Crisis Response and Management (ISCRAM), pp. 1–14 (2016)Google Scholar
  17. 17.
    Dell’Acqua, F., Gamba, P.: Remote sensing and earthquake damage assessment: experiences, limits, and perspectives. Proc. IEEE 100(10), 2876–2890 (2012)CrossRefGoogle Scholar
  18. 18.
    Deng, J., Dong, W., Socher, R., Li, L., Kai, L., Li, F.-F.: ImageNet: a large-scale hierarchical image database. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)Google Scholar
  19. 19.
    Dong, L., Shan, J.: A comprehensive review of earthquake-induced building damage detection with remote sensing techniques. ISPRS J. Photogrammetry Remote Sens. 84, 85–99 (2013)CrossRefGoogle Scholar
  20. 20.
    Doshi, J., Basu, S., Pang, G.: From satellite imagery to disaster insights. In: NeurIPS Workshop on Artificial Intelligence for Social Good (2018)Google Scholar
  21. 21.
    Doshi, J., et al.: FireNet: real-time segmentation of fire perimeter from aerial video. In: NeurIPS Workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response (2019)Google Scholar
  22. 22.
    Durand, T., Mehrasa, N., Mori, G.: Learning a deep convnet for multi-label classification with partial labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  23. 23.
    Efremova, N., West, D., Zausaev, D.: AI-based evaluation of the SDGs: the case of crop detection with earth observation data. In: ICLR Workshop on Artificial Intelligence for Social Good (2019)Google Scholar
  24. 24.
    Fernandez Galarreta, J., Kerle, N., Gerke, M.: UAV-based urban structural damage assessment using object-based image analysis and semantic reasoning. Nat. Hazards Earth Syst. Sci. 15(6), 1087–1101 (2015)CrossRefGoogle Scholar
  25. 25.
    Gamba, P., Dell’Acqua, F., Trianni, G.: Rapid damage detection in the bam area using multitemporal SAR and exploiting ancillary data. IEEE Trans. Geosci. Remote Sens. 45(6), 1582–1589 (2007)CrossRefGoogle Scholar
  26. 26.
    Gebru, T., et al.: Using deep learning and google street view to estimate the demographic makeup of neighborhoods across the United States. Proc. Nat. Acad. Sci. 114(50), 13108–13113 (2017)CrossRefGoogle Scholar
  27. 27.
    Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Fei-Fei, L.: Fine-grained car detection for visual census estimation. In: The AAAI Conference on Artificial Intelligence (2017)Google Scholar
  28. 28.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  29. 29.
    Gritta, M., Pilevar, M.T., Collier, N.: A pragmatic guide to geoparsing evaluation: toponyms, named entity recognition and pragmatics. Lang. Resour. Eval. (2019).  https://doi.org/10.1007/s10579-019-09475-3CrossRefGoogle Scholar
  30. 30.
    Gueguen, L., Hamid, R.: Large-scale damage detection using satellite imagery. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1321–1328 (2015)Google Scholar
  31. 31.
    Gupta, R., et al.: Creating xBD: a dataset for assessing building damage from satellite imagery. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  32. 32.
    Gupta, R., et al.: xBD: a dataset for assessing building damage from satellite imagery. arXiv preprint arXiv:1911.09296 (2019)
  33. 33.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  34. 34.
    Helber, P., et al.: Mapping informal settlements in developing countries with multi-resolution, multi-spectral data. In: ICLR Workshop on Artificial Intelligence for Social Good (2019)Google Scholar
  35. 35.
    Imran, M., Castillo, C., Diaz, F., Vieweg, S.: Processing social media messages in mass emergency: a survey. ACM Comput. Surv. 47(4), 67 (2015)CrossRefGoogle Scholar
  36. 36.
    Jean, N., Burke, M., Xie, M., Davis, W.M., Lobell, D.B., Ermon, S.: Combining satellite imagery and machine learning to predict poverty. Science 353(6301), 790–794 (2016)CrossRefGoogle Scholar
  37. 37.
    Joyce, K.E., Belliss, S.E., Samsonov, S.V., McNeill, S.J., Glassey, P.J.: A review of the status of satellite remote sensing and image processing techniques for mapping natural hazards and disasters. Prog. Phys. Geogr. 33(2), 183–207 (2009)CrossRefGoogle Scholar
  38. 38.
    Kalliatakis, G., Ehsan, S., Fasli, M., D McDonald-Maier, K.: DisplaceNet: recognising displaced people from images by exploiting dominance level. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  39. 39.
    Kalliatakis, G., Ehsan, S., Leonardis, A., Fasli, M., McDonald-Maier, K.D.: Exploring object-centric and scene-centric CNN features and their complementarity for human rights violations recognition in images. IEEE Access 7, 10045–10056 (2019)CrossRefGoogle Scholar
  40. 40.
    Kaneko, A., et al.: Deep learning for crop yield prediction in Africa. In: ICML Workshop on Artificial Intelligence for Social Good (2019)Google Scholar
  41. 41.
    Kataoka, H., Satoh, Y., Abe, K., Minoguchi, M., Nakamura, A.: Ten-million-order human database for world-wide fashion culture analysis. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  42. 42.
    Kellenberger, B., Marcos, D., Tuia, D.: When a few clicks make all the difference: improving weakly-supervised wildlife detection in UAV images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  43. 43.
    Lee, K., Lee, H., Lee, K., Shin, J.: Training confidence-calibrated classifiers for detecting out-of-distribution samples. In: International Conference on Learning Representations (2018)Google Scholar
  44. 44.
    Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 7167–7177. Curran Associates, Inc. (2018)Google Scholar
  45. 45.
    Li, X., Caragea, D., Caragea, C., Imran, M., Ofli, F.: Identifying disaster damage images using a domain adaptation approach. In: 16th International Conference on Information Systems for Crisis Response and Management (ISCRAM) (2019)Google Scholar
  46. 46.
    Li, Y., Hu, W., Dong, H., Zhang, X.: Building damage detection from post-event aerial imagery using single shot multibox detector. Appl. Sci. 9(6), 1128 (2019)CrossRefGoogle Scholar
  47. 47.
    Li, Y., Ye, S., Bartoli, I.: Semisupervised classification of hurricane damage from postevent aerial imagery using deep learning. J. Appl. Remote Sens. 12(4), 045008 (2018)Google Scholar
  48. 48.
    Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 (2017)
  49. 49.
    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  50. 50.
    McKinney, S.M., et al.: International evaluation of an AI system for breast cancer screening. Nature 577(7788), 89–94 (2020)CrossRefGoogle Scholar
  51. 51.
    Nachmany, Y., Alemohammad, H.: Detecting roads from satellite imagery in the developing world. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  52. 52.
    Naik, N., Philipoom, J., Raskar, R., Hidalgo, C.: Streetscore - predicting the perceived safety of one million streetscapes. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 793–799 (2014)Google Scholar
  53. 53.
    Naik, N., Kominers, S.D., Raskar, R., Glaeser, E.L., Hidalgo, C.A.: Computer vision uncovers predictors of physical urban change. Proc. Nat. Acad. Sci. (2017).  https://doi.org/10.1073/pnas.1619003114CrossRefGoogle Scholar
  54. 54.
    Nguyen, D.T., Ofli, F., Imran, M., Mitra, P.: Damage assessment from social media imagery data during disasters. In: International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 1–8 (2017)Google Scholar
  55. 55.
    Nia, K.R., Mori, G.: Building damage assessment using deep learning and ground-level image data. In: 14th Conference on Computer and Robot Vision (CRV), pp. 95–102. IEEE (2017)Google Scholar
  56. 56.
    Nogueira, K., et al.: Exploiting convnet diversity for flooding identification. IEEE Geosci. Remote Sens. Lett. 15(9), 1446–1450 (2018)CrossRefGoogle Scholar
  57. 57.
    Oh, J., Hebert, M., Jeon, H.G., Perez, X., Dai, C., Song, Y.: Explainable semantic mapping for first responders. In: NeurIPS Workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response (2019)Google Scholar
  58. 58.
    Peters, R., de Albuquerque, J.P.: Investigating images as indicators for relevant social media messages in disaster management. In: 12th International Conference on Information Systems for Crisis Response and Management (ISCRAM) (2015)Google Scholar
  59. 59.
    Piaggesi, S., et al.: Predicting city poverty using satellite imagery. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  60. 60.
    Plank, S.: Rapid damage assessment by means of multi-temporal SAR–a comprehensive review and outlook to sentinel-1. Remote Sens. 6(6), 4870–4906 (2014)CrossRefGoogle Scholar
  61. 61.
    Pouyanfar, S., et al.: Unconstrained flood event detection using adversarial data augmentation. In: IEEE International Conference on Image Processing (ICIP), pp. 155–159 (2019)Google Scholar
  62. 62.
    Pryzant, R., Ermon, S., Lobell, D.: Monitoring ethiopian wheat fungus with satellite imagery and deep feature learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)Google Scholar
  63. 63.
    Radhika, S., Tamura, Y., Matsui, M.: Cyclone damage detection on building structures from pre-and post-satellite images using wavelet based pattern recognition. J. Wind Eng. Ind. Aerodyn. 136, 23–33 (2015)CrossRefGoogle Scholar
  64. 64.
    Radke, D., Hessler, A., Ellsworth, D.: FireCast: leveraging deep learning to predict wildfire spread. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 4575–4581 (2019)Google Scholar
  65. 65.
    Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 506–516 (2017)Google Scholar
  66. 66.
    Reuter, C., Kaufhold, M.A.: Fifteen years of social media in emergencies: a retrospective review and future directions for crisis informatics. J. Contingencies Crisis Manage. 26(1), 41–57 (2018)CrossRefGoogle Scholar
  67. 67.
    Rudner, T.G.J., et al.: Multi\(^3\)Net: segmenting flooded buildings via fusion of multiresolution, multisensor, and multitemporal satellite imagery. In: The AAAI Conference on Artificial Intelligence, pp. 702–709 (2019)Google Scholar
  68. 68.
    Rustowicz, R., Cheong, R., Wang, L., Ermon, S., Burke, M., Lobell, D.: Semantic segmentation of crop type in Africa: a novel dataset and analysis of deep learning methods. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  69. 69.
    Said, N., et al.: Natural disasters detection in social media and satellite imagery: a survey. Multimedia Tools Appl. 78, 31267–31302 (2019)CrossRefGoogle Scholar
  70. 70.
    Schmidt, V., et al.: Visualizing the consequences of climate change using cycle-consistent adversarial networks. In: ICLR Workshop on Artificial Intelligence for Social Good (2019)Google Scholar
  71. 71.
    Seo, J., Lee, S., Kim, B., Jeon, T.: Revisiting classical bagging with modern transfer learning for on-the-fly disaster damage detector. In: NeurIPS Workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response (2019)Google Scholar
  72. 72.
    Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  73. 73.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568–576 (2014)Google Scholar
  74. 74.
    Skakun, S., Kussul, N., Shelestov, A., Kussul, O.: Flood hazard and flood risk assessment using a time series of satellite images: a case study in Namibia. Risk Anal. 34(8), 1521–1537 (2014)CrossRefGoogle Scholar
  75. 75.
    Sublime, J., Kalinicheva, E.: Automatic post-disaster damage mapping using deep-learning techniques for change detection: Case study of the Tohoku Tsunami. Remote Sens. 11(9), 1123 (2019)CrossRefGoogle Scholar
  76. 76.
    Thomee, B., et al.: YFCC100M: the new data in multimedia research. Commun. ACM 59(2), 64–73 (2016)CrossRefGoogle Scholar
  77. 77.
    Tingzon, I., et al.: Mapping poverty in the Philippines using machine learning, satellite imagery, and crowd-sourced geospatial information. In: ICML Workshop on Artificial Intelligence for Social Good (2019)Google Scholar
  78. 78.
    Turker, M., San, B.T.: Detection of collapsed buildings caused by the 1999 Izmit, Turkey earthquake through digital analysis of post-event aerial photographs. Int. J. Remote Sens. 25(21), 4701–4714 (2004)CrossRefGoogle Scholar
  79. 79.
    de Vries, T., Misra, I., Wang, C., van der Maaten, L.: Does object recognition work for everyone? In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  80. 80.
    Watmough, G.R., et al.: Socioecologically informed use of remote sensing data to predict rural household poverty. Proc. Nat. Acad. Sci. 116(4), 1213–1218 (2019)CrossRefGoogle Scholar
  81. 81.
    Workman, S., Zhai, M., Crandall, D.J., Jacobs, N.: A unified model for near and remote sensing. In: The IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  82. 82.
    Wu, N., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39(1), 1184–1194 (2019)Google Scholar
  83. 83.
    Xu, J.Z., Lu, W., Li, Z., Khaitan, P., Zaytseva, V.: Building damage detection in satellite imagery using convolutional neural networks. In: NeurIPS Workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response (2019)Google Scholar
  84. 84.
    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452–1464 (2018)CrossRefGoogle Scholar
  85. 85.
    Zhou, B., Liu, L., Oliva, A., Torralba, A.: Recognizing city identity via attribute analysis of geo-tagged images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 519–534. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10578-9_34CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Massachusetts Institute of TechnologyCambridgeUSA
  2. 2.Qatar Computing Research Institute, HBKUAr-RayyanQatar
  3. 3.Universitat Oberta de CatalunyaBarcelonaSpain

Personalised recommendations