Skip to main content

Advertisement

SpringerLink
Log in

Multi-modal classification of Twitter data during disasters for humanitarian response

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

In recent days, humanitarian organizations rely on social media like Twitter for situational awareness during the disaster. Millions of tweets are posted on Twitter in the form of text or images or both. Existing works showed that both image and text give complementary information during the disaster. Multi-modal informative tweet detection is helpful to both government and non-government organizations which remains a challenging task during the disaster. However, most of the existing works focused on either text or image content, but not both. In this paper, we propose a novel method based on the combination of fine-tuned BERT and DenseNet models for identifying the multi-modal informative tweets during the disaster. Fine-tuned BERT model is used to extract the linguistic, syntactic and semantic features which help deep understanding of the informative text present in the multi-modal tweet. On the other hand, the fine-tuned DenseNet model is used to extract the sophisticated features from the image. Different experiments are performed on a vast number of data-sets such as Hurricane Harvey, Hurricane Irma, Hurricane Maria, California Wildfire, Sri Lanka floods and Iraq–Iran Earthquake. Experimental results demonstrate that the proposed method outperforms the state-of-the-art method on different parameters. It is the first attempt, to the best of our knowledge, to detect multi-modal informative tweets using the combination of fine-tuned BERT and DenseNet models, where at least any text or image is informative during the disaster.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Agarwal M, Leekha M, Sawhney R, Shah RR (2020) Crisis-Dias: towards multimodal damage analysis-deployment, challenges and assessment. Proc AAAI Conf Artif Intell 34:346–353

    Google Scholar 

  • Alam F, Ofli F, Imran M (2018) Crisismmd: multimodal Twitter datasets from natural disasters. In: Proceedings of the 12th international AAAI conference on web and social media (ICWSM)

  • Alam F, Ofli F, Imran M (2019) Descriptive and visual summaries of disaster events using artificial intelligence techniques: case studies of hurricanes Harvey, Irma, and Maria. Behaviour & Information Technology pp 1–31

  • Alberti C, Lee K, Collins M (2019) A BERT baseline for the natural questions. arXiv preprint arXiv:190108634

  • Basu M, Shandilya A, Khosla P, Ghosh K, Ghosh S (2019) Extracting resource needs and availabilities from microblogs for aiding post-disaster relief operations. IEEE Trans Comput Soc Syst 6(3):604–618

    Article  Google Scholar 

  • Caragea C, Silvescu A, Tapia AH (2016) Identifying informative messages in disaster events using convolutional neural networks. In: International conference on information systems for crisis response and management

  • Chien T, Kalita J (2019) Adversarial analysis of natural language inference systems. arXiv preprint arXiv:191203441

  • Cresci S, Tesconi M, Cimino A, Dell’Orletta F (2015) A linguistically-driven approach to cross-event damage assessment of natural disasters from social media messages. In: Proceedings of the 24th international conference on world wide web, ACM, pp 1195–1200

  • Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805

  • Gautam AK, Misra L, Kumar A, Misra K, Aggarwal S, Shah RR (2019) Multimodal analysis of disaster tweets. In: 2019 IEEE Fifth international conference on multimedia big data (BigMM). IEEE, pp 94–103

  • Goldberg Y (2019) Assessing BERT’s syntactic abilities. arXiv preprint arXiv:190105287

  • Hao H, Wang Y (2019) Hurricane damage assessment with multi-, crowd-sourced image data: a case study of Hurricane Irma in the city of Miami. In: ISCRAM

  • Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  • Hughes AL, Palen L (2009) Twitter adoption and use in mass convergence and emergency events. Int J Emerg Manag 6(3–4):248–260

    Article  Google Scholar 

  • Imran M, Castillo C, Lucas J, Meier P, Vieweg S (2014) Aidr: artificial intelligence for disaster response. In: Proceedings of the 23rd international conference on world wide web. ACM, pp 159–162

  • Imran M, Castillo C, Diaz F, Vieweg S (2015) Processing social media messages in mass emergency: a survey. ACM Comput Surv (CSUR) 47(4):67

    Article  Google Scholar 

  • Jiang N, de Marneffe MC (2019) Evaluating BERT for natural language inference: a case study on the commitmentbank. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 6088–6093

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • Li X, Caragea D, Caragea C, Imran M, Ofli F (2019) Identifying disaster damage images using a domain adaptation approach. In: ISCRAM

  • Lin Y, Tan YC, Frank R (2019) Open sesame: getting inside BERT’s linguistic knowledge. arXiv preprint arXiv:190601698

  • Madichetty S, Muthukumarasamy S (2020) Detection of situational information from Twitter during disaster using deep learning models. Sādhanā 45(1):1–13

    Article  Google Scholar 

  • Madichetty S, Sridevi M (2019) Disaster damage assessment from the tweets using the combination of statistical features and informative words. Soc Netw Anal Min 9(1):42

    Article  Google Scholar 

  • Madichetty S, Sridevi M (2020a) Identification of medical resource tweets using majority voting-based ensemble during disaster. Soc Netw Anal Min 10(1):1–18

    Article  Google Scholar 

  • Madichetty S, Sridevi M (2020b) A novel method for identifying the damage assessment tweets during disaster. Future Gener Comput Syst 116:440–454

    Article  Google Scholar 

  • Madichetty S et al (2018) Re-ranking feature selection algorithm for detecting the availability and requirement of resources tweets during disaster. Int J Comput Intell IoT 1(2):207–211

    Google Scholar 

  • Mouzannar H, Rizk Y, Awad M (2018) Damage identification in social media posts using multimodal deep learning. In: ISCRAM

  • Nalluru G, Pandey R, Purohit H (2019) Classifying relevant social media posts during disasters using ensemble of domain-agnostic and domain-specific word embeddings. arXiv preprint arXiv:191105165

  • Nguyen DT, Mannai KAA, Joty S, Sajjad H, Imran M, Mitra P (2016) Rapid classification of crisis-related data on social networks using convolutional neural networks. arXiv preprint arXiv:160803902

  • Nguyen DT, Ofli F, Imran M, Mitra P (2017) Damage assessment from social media imagery data during disasters. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. ACM, pp 569–576

  • Rudra K, Ganguly N, Goyal P, Ghosh S (2018a) Extracting and summarizing situational information from the Twitter social media during disasters. ACM Trans Web (TWEB) 12(3):17

    Google Scholar 

  • Rudra K, Goyal P, Ganguly N, Mitra P, Imran M (2018b) Identifying sub-events and summarizing disaster-related information from microblogs. In: The 41st international ACM SIGIR conference on research & development in information retrieval. ACM, pp 265–274

  • Sreenivasulu M, Sridevi M (2017) Mining informative words from the tweets for detecting the resources during disaster. In: International conference on mining intelligence and knowledge exploration. Springer, pp 348–358

  • Sreenivasulu M, Sridevi M (2020) Comparative study of statistical features to detect the target event during disaster. Big Data Min Anal 3(2):121–130

    Article  Google Scholar 

  • Xie S, Duan J, Liu S, Dai Q, Liu W, Ma Y, Guo R, Ma C (2016) Crowdsourcing rapid assessment of collapsed buildings early after the earthquake based on aerial remote sensing image: a case study of yushu earthquake. Remote Sens 8(9):759

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NIT, Tiruchirappalli, India

    Sreenivasulu Madichetty, Sridevi Muthukumarasamy & P. Jayadev

Authors
  1. Sreenivasulu Madichetty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sridevi Muthukumarasamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. Jayadev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sreenivasulu Madichetty.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Madichetty, S., Muthukumarasamy, S. & Jayadev, P. Multi-modal classification of Twitter data during disasters for humanitarian response. J Ambient Intell Human Comput 12, 10223–10237 (2021). https://doi.org/10.1007/s12652-020-02791-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02791-5

Keywords

Search

Navigation