A deep multi-modal neural network for informative Twitter content classification during emergencies

  • Abhinav Kumar
  • Jyoti Prakash Singh
  • Yogesh K. DwivediEmail author
  • Nripendra P. Rana
S.I. : Design and Management of Humanitarian Supply Chains


People start posting tweets containing texts, images, and videos as soon as a disaster hits an area. The analysis of these disaster-related tweet texts, images, and videos can help humanitarian response organizations in better decision-making and prioritizing their tasks. Finding the informative contents which can help in decision making out of the massive volume of Twitter content is a difficult task and require a system to filter out the informative contents. In this paper, we present a multi-modal approach to identify disaster-related informative content from the Twitter streams using text and images together. Our approach is based on long-short-term-memory and VGG-16 networks that show significant improvement in the performance, as evident from the validation result on seven different disaster-related datasets. The range of F1-score varied from 0.74 to 0.93 when tweet texts and images used together, whereas, in the case of only tweet text, it varies from 0.61 to 0.92. From this result, it is evident that the proposed multi-modal system is performing significantly well in identifying disaster-related informative social media contents.


Disaster Twitter LSTM VGG-16 Social media Tweets 



  1. Aipe, A., Ekbal, A., Mukuntha, N., & Kurohashi, S. (2018). Linguistic feature assisted deep learning approach towards multi-label classification of crisis related tweets. In Proceedings of the 15th ISCRAM Conference (pp. 705–717).Google Scholar
  2. Akter, S., & Wamba, S. F. (2017). Big data and disaster management: A systematic review and agenda for future research. Annals of Operations Research. Scholar
  3. Alalwan, A. A., Rana, N. P., Dwivedi, Y. K., & Algharabat, R. (2017). Social media in marketing: A review and analysis of the existing literature. Telematics and Informatics,34(7), 1177–1190.CrossRefGoogle Scholar
  4. Alam, F., Imran, M., & Ofli, F. (2017). Image4act: Online social media image processing for disaster response. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 601–604). ACM.Google Scholar
  5. Alam, F., Ofli, F., & Imran, M. (2018). Crisismmd: Multimodal Twitter datasets from natural disasters. In Proceedings of the 12th international AAAI conference on web and social media (ICWSM). Google Scholar
  6. Ashktorab, Z., Brown, C., Nandi, M., & Culotta, A. (2014). Tweedr: Mining Twitter to inform disaster response. In Proceedings of the 11th ISCRAM conference (pp. 354–358).Google Scholar
  7. Atefeh, F., & Khreich, W. (2015). A survey of techniques for event detection in Twitter. Computational Intelligence,31, 132–164.CrossRefGoogle Scholar
  8. Cameron, M. A., Power, R., Robinson, B., & Yin, J. (2012). Emergency situation awareness from Twitter for crisis management. In Proceedings of the 21st international conference on world wide web (pp. 695–698). ACM.Google Scholar
  9. Caragea, C., Mcneese, N., Jaiswal, A., Traylor, G., Woo Kim, H., Mitra, P., et al. (2011). Classifying text messages for the Haiti earthquake. In Proceedings of the 8th international conference on information systems for crisis response and management (ISCRAM2011). Google Scholar
  10. Caragea, C., Silvescu, A., & Tapia, A. H. (2016). Identifying informative messages in disaster events using convolutional neural networks. In International conference on information systems for crisis response and management (pp. 137–147).Google Scholar
  11. Chaudhuri, N., & Bose, I. (2019). Application of image analytics for disaster response in smart cities. In Proceedings of the 52nd Hawaii international conference on system sciences.
  12. Daly, S., & Thom, J. A. (2016). Mining and classifying image posts on social media to analyse fires. In Proceedings of the 13th ISCRAM conference (pp. 1–14).Google Scholar
  13. Dubey, R. (2019). Developing an integration framework for crowdsourcing and internet of things with applications for disaster response. In Social entrepreneurship: Concepts, methodologies, tools, and applications (pp. 274–283). IGI Global.Google Scholar
  14. Dubey, R., Ali, S. S., Aital, P., Venkatesh, V., et al. (2014). Mechanics of humanitarian supply chain agility and resilience and its empirical validation. International Journal of Services and Operations Management,17, 367–384.CrossRefGoogle Scholar
  15. Dubey, R., Altay, N., & Blome, C. (2017). Swift trust and commitment: The missing links for humanitarian supply chain coordination? Annals of Operations Research. Scholar
  16. Dubey, R., Gunasekaran, A., Childe, S. J., Roubaud, D., Wamba, S. F., Giannakis, M., et al. (2019). Big data analytics and organizational culture as complements to swift trust and collaborative performance in the humanitarian supply chain. International Journal of Production Economics,210, 120–136.CrossRefGoogle Scholar
  17. Dwivedi, Y. K., Kapoor, K. K., & Chen, H. (2015). Social media marketing and advertising. The Marketing Review,15(3), 289–309.CrossRefGoogle Scholar
  18. Dwivedi, Y. K., Shareef, M. A., Mukerji, B., Rana, N. P., & Kapoor, K. K. (2018). Involvement in emergency supply chain for disaster management: A cognitive dissonance perspective. International Journal of Production Research,56, 6758–6773.CrossRefGoogle Scholar
  19. Elbanna, A., Bunker, D., Levine, L., & Sleigh, A. (2019). Emergency management in the changing world of social media: Framing the research agenda with the stakeholders through engaged scholarship. International Journal of Information Management,47, 112–120.CrossRefGoogle Scholar
  20. Graf, D., Retschitzegger, W., Schwinger, W., Pröll, B., & Kapsammer, E. (2018). Cross-domain informativeness classification for disaster situations. In Proceedings of the 10th international conference on management of digital ecosystems (pp. 183–190). ACM.Google Scholar
  21. Guha-Sapir, D., Vos, F., Below, R., & Ponserre, S. (2012). Annual disaster statistical review 2011: The number sandtrends. Technical Report Centre for Research on the Epidemiology of Disasters (CRED).Google Scholar
  22. Gupta, A., Lamba, H., Kumaraguru, P., & Joshi, A. (2013, May). Faking sandy: Characterizing and identifying fake images on Twitter during hurricane sandy. In Proceedings of the 22nd international conference on world wide web (pp. 729–736). ACM.Google Scholar
  23. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation,9, 1735–1780.CrossRefGoogle Scholar
  24. Huang, Q., & Xiao, Y. (2015). Geographic situational awareness: Mining tweets for disaster preparedness, emergency response, impact, and recovery. ISPRS International Journal of Geo-Information,4, 1549–1568.CrossRefGoogle Scholar
  25. Imran, M., Castillo, C., Diaz, F., & Vieweg, S. (2015). Processing social media messages in mass emergency: A survey. ACM Computing Surveys (CSUR),47, 67.CrossRefGoogle Scholar
  26. Imran, M., Castillo, C., Lucas, J., Meier, P., & Vieweg, S. (2014). Aidr: Artificial intelligence for disaster response. In Proceedings of the 23rd international conference on world wide web (pp. 159–162). ACM.Google Scholar
  27. Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013a). Extracting information nuggets from disaster-related messages in social media. In Proceedings of the 10th ISCRAM conference (pp. 791–801).Google Scholar
  28. Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013b). Practical extraction of disaster-relevant information from social media. In Proceedings of the 22nd international conference on world wide web (pp. 1021–1024). ACM.Google Scholar
  29. Jabbour, C. J. C., Sobreiro, V. A., de Sousa Jabbour, A. B. L., de Souza Campos, L. M., Mariano, E. B., & Renwick, D. W. S. (2017). An analysis of the literature on humanitarian logistics and supply chain management: Paving the way for future studies. Annals of Operations Research. Scholar
  30. Jamali, M., Nejat, A., Ghosh, S., Jin, F., & Cao, G. (2019). Social media data and post-disaster recovery. International Journal of Information Management,44, 25–37.CrossRefGoogle Scholar
  31. Jin, S., Jeong, S., Kim, J., & Kim, K. (2015). A logistics model for the transport of disaster victims with various injuries and survival probabilities. Annals of Operations Research,230, 17–33.CrossRefGoogle Scholar
  32. John, L., Gurumurthy, A., Soni, G., & Jain, V. (2018). Modelling the inter-relationship between factors affecting coordination in a humanitarian supply chain: A case of Chennai flood relief. Annals of Operations Research. Scholar
  33. Kapoor, K. K., Tamilmani, K., Rana, N. P., Patil, P., Dwivedi, Y. K., & Nerur, S. (2018). Advances in social media research: Past, present and future. Information Systems Frontiers,20(3), 531–558.CrossRefGoogle Scholar
  34. Kim, J., Bae, J., & Hastak, M. (2018). Emergency information diffusion on online social media during storm Cindy in US. International Journal of Information Management,40, 153–165.CrossRefGoogle Scholar
  35. Kim, J., & Hastak, M. (2018). Social network analysis: Characteristics of online social networks after a disaster. International Journal of Information Management,38(1), 86–96.CrossRefGoogle Scholar
  36. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  37. Kumar, A., & Singh, J. P. (2019). Location reference identification from tweets during emergencies: A deep learning approach. International Journal of Disaster Risk Reduction,33, 365–375.CrossRefGoogle Scholar
  38. Kumar, A., Singh, J. P., & Rana, N. P. (2017). Authenticity of geo-location and place name in tweets. In Proceedings of the 23rd Americas conference on information systems (AMCIS) (pp. 1–9)Google Scholar
  39. Kumar, A., & Singh, J. P., Saumya, S. (2019). A comparative analysis of machine learning techniques for disaster related tweet classification. In IEEE region 10 humanitarian technology conference, (pp. 222–227).Google Scholar
  40. Lagerstrom, R., Arzhaeva, Y., Szul, P., Obst, O., Power, R., Robinson, B., et al. (2016). Image classification to support emergency situation awareness. Frontiers in Robotics and AI,3, 54.CrossRefGoogle Scholar
  41. Li, H., Guevara, N., Herndon, N., Caragea, D., Neppalli, K., Caragea, C., et al. (2015). Twitter mining for disaster response: A domain adaptation approach. In Proceedings of the 12th ISCRAM conference.Google Scholar
  42. Mouzannar, H., Rizk, Y., & Awad, M. (2018). Damage identification in social media posts using multimodal deep learning. In ISCRAM.Google Scholar
  43. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807–814).Google Scholar
  44. Nguyen, D. T., Al Mannai, K. A., Joty, S., Sajjad, H., Imran, M., & Mitra, P. (2017a). Robust classification of crisis-related data on social networks using convolutional neural networks. In 11th international conference on web and social media, ICWSM 2017 (pp. 632–635). AAAI Press.Google Scholar
  45. Nguyen, D. T., Alam, F., Ofli, F., & Imran, M. (2017b). Automatic image filtering on social networks using deep learning and perceptual hashing during crises. arXiv preprint arXiv:1704.02602.
  46. Nguyen, D. T., Joty, S., Imran, M., Sajjad, H., & Mitra, P. (2016). Applications of online deep learning for crisis response using social media information. arXiv preprint arXiv:1610.01030.
  47. Nguyen, D. T., Ofli, F., Imran, M., & Mitra, P. (2017c). Damage assessment from social media imagery data during disasters. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 569–576). ACM.Google Scholar
  48. Olteanu, A., Castillo, C., Diaz, F., & Vieweg, S. (2014). Crisislex: A lexicon for collecting and filtering microblogged communications in crises. In Eighth international AAAI conference on weblogs and social media.Google Scholar
  49. Papadopoulos, T., Gunasekaran, A., Dubey, R., Altay, N., Childe, S. J., & Fosso-Wamba, S. (2017). The role of big data in explaining disaster resilience in supply chains for sustainability. Journal of Cleaner Production,142, 1108–1118.CrossRefGoogle Scholar
  50. Paul, J. A., & Hariharan, G. (2012). Location-allocation planning of stockpiles for effective disaster mitigation. Annals of Operations Research,196, 469–490.CrossRefGoogle Scholar
  51. Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).Google Scholar
  52. Ragini, J. R., Anand, P. R., & Bhaskar, V. (2018). Big data analytics for disaster response and recovery through sentiment analysis. International Journal of Information Management,42, 13–24.CrossRefGoogle Scholar
  53. Rizk, Y., Jomaa, H. S., Awad, M., & Castillo, C. (2019). A computationally efficient multi-modal classification approach of disaster-related Twitter images. In Proceedings of the 34th ACM/SIGAPP symposium on applied computing (pp. 2050–2059). ACM.Google Scholar
  54. Rudra, K., Banerjee, S., Ganguly, N., Goyal, P., Imran, M., & Mitra, P. (2016). Summarizing situational tweets in crisis scenario. In Proceedings of the 27th ACM conference on hypertext and social media (pp. 137–147). ACM.Google Scholar
  55. Sakaki, T., Okazaki, M., & Matsuo, Y. (2013). Tweet analysis for real-time event detection and earthquake reporting system development. IEEE Transactions on Knowledge and Data Engineering,25, 919–931.CrossRefGoogle Scholar
  56. Shareef, M. A., Dwivedi, Y. K., Mahmud, R., Wright, A., Rahman, M. M., Kizgin, H., et al. (2018). Disaster management in Bangladesh: Developing an effective emergency supply chain network. Annals of Operations Research. Scholar
  57. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  58. Singh, J. P., Dwivedi, Y. K., Rana, N. P., Kumar, A., & Kapoor, K. K. (2019). Event classification and location prediction from tweets during disasters. Annals of Operations Research,283(1–2), 737–757. Scholar
  59. Sinha, A., Kumar, P., Rana, N. P., Islam, R., & Dwivedi, Y. K. (2019). Impact of internet of things (IoT) in disaster management: A task-technology fit perspective. Annals of Operations Research,283, 759–794. Scholar
  60. Son, J., Lee, H. K., Jin, S., & Lee, J. (2019). Content features of tweets for effective communication during disasters: A media synchronicity theory perspective. International Journal of Information Management,45, 56–68.CrossRefGoogle Scholar
  61. Sundermeyer, M., Schlüter, R., & Ney, H. (2012). LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association. Google Scholar
  62. Verma, S., Vieweg, S., Corvey, W. J., Palen, L., Martin, J. H., Palmer, M., et al. (2011). Natural language processing to the rescue? Extracting” situational awareness” tweets during mass emergency. In Fifth international AAAI conference on weblogs and social media. Google Scholar
  63. Yu, M., Huang, Q., Qin, H., Scheele, C., & Yang, C. (2019). Deep learning for real-time social media text classification for situation awareness using hurricanes sandy, Harvey, and Irma as case studies. International Journal of Digital Earth. Scholar
  64. Zhang, C., Fan, C., Yao, W., Hu, X., & Mostafavi, A. (2019). Social media for intelligent public information and warning in disasters: An interdisciplinary review. International Journal of Information Management,49, 190–207.CrossRefGoogle Scholar
  65. Zheng, X., Han, J., & Sun, A. (2018). A survey of location prediction on Twitter. IEEE Transactions on Knowledge and Data Engineering,30, 1652–1671.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  • Abhinav Kumar
    • 1
  • Jyoti Prakash Singh
    • 1
  • Yogesh K. Dwivedi
    • 2
    Email author
  • Nripendra P. Rana
    • 3
  1. 1.Department of Computer Science and EngineeringNational Institute of Technology PatnaPatnaIndia
  2. 2.School of ManagementEmerging Markets Research Centre (EMaRC)SwanseaUK
  3. 3.School of ManagementUniversity of BradfordBradfordUK

Personalised recommendations