Abstract
Deep learning has revolutionized event detection in social media and remote sensing, allowing for more precise and efficient analysis of immense amounts of data to unveil hidden patterns and insights. Moreover, Merging both data sources has proven efficient despite the absence of multi-sensed datasets. In today’s data-driven globe, it is becoming increasingly critical to process and explore heterogeneous data and to design models handling such data. This paper proposes a new multi-sensed fusion approach that leverages satellite images and tweets as input. We combined two open datasets to obtain a multi-sensed dataset concerning the 2017 hurricane Harvey. We extracted features from satellite imagery using Resnet34 and generated embeddings from tweets using Bert. We fused the embeddings and the features using a selective attention module incorporating cross and self-attention. Our module can filter misleading features from weak modalities on a sample-by-sample basis. We demonstrate that our approach surpasses unimodal models based on tweets or satellite imagery. We compared our results to a few baselines associated with hurricane Harvey and proved that our model surpasses them in accuracy, precision, recall, and F1 measure.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alam, F., Ofli, F., Imran, M.: CrisisMMD: multimodal Twitter datasets from natural disasters. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 12 (2018)
Avvenuti, M., Cresci, S., Del Vigna, F., Fagni, T., Tesconi, M.: CrisMap: a big data crisis mapping system based on damage detection and geoparsing. Inf. Syst. Front. 20(5), 993–1011 (2018)
Bischke, B., Bhardwaj, P., Gautam, A., Helber, P., Borth, D., Dengel, A.: Detection of flooding events in social multimedia and satellite imagery using deep neural networks. In: MediaEval (2017)
Cao, Q.D., Choe, Y.: Building damage annotation on post-hurricane satellite imagery based on convolutional neural networks. Nat. Hazards 103(3), 3357–3376 (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, X., Wang, C., Li, Z.: A near real-time flood-mapping approach by integrating social media and post-event satellite imagery. Ann. GIS 24(2), 113–123 (2018)
Kashif, A., Pogorelov, K., Riegler, M., Conci, N., Halvorsen, P.: Social media and satellites. Multimed. Tools Appl. 78(3), 2837–2875 (2019)
Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)
Li, Y., et al.: CBANet: an end-to-end cross band 2-D attention network for hyperspectral change detection in remote sensing. IEEE Trans. Geosci. Remote Sens. (2023)
Lu, J., Batra, D., Parikh, D., Lee, S.: VilBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Madichetty, S., Sridevi, M.: Detecting informative tweets during disaster using deep neural networks. In: 2019 11th International Conference on Communication Systems & Networks (COMSNETS), pp. 709–713. IEEE (2019)
Pittaras, N., et al.: Geosensor: semantifying change and event detection over big data. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pp. 2259–2266. ACM (2019)
Qi, L., Li, J., Wang, Y., Gao, X.: Urban observation: integration of remote sensing and social media data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 12(11), 4252–4264 (2019)
Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 (2019)
Thangavel, K., et al.: Autonomous satellite wildfire detection using hyperspectral imagery and neural networks: a case study on Australian wildfire. Remote Sens. 15(3), 720 (2023)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Yang, T., et al.: Monitoring ecological conditions by remote sensing and social media data—Sanya city (China) as case study. Remote Sens. 14(12), 2824 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bouabid, M., Farah, M. (2023). Crisis Detection by Social and Remote Sensing Fusion: A Selective Attention Approach. In: Nguyen, N.T., et al. Computational Collective Intelligence. ICCCI 2023. Lecture Notes in Computer Science(), vol 14162. Springer, Cham. https://doi.org/10.1007/978-3-031-41456-5_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-41456-5_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41455-8
Online ISBN: 978-3-031-41456-5
eBook Packages: Computer ScienceComputer Science (R0)