A deep multi-modal neural network for informative Twitter content classification during emergencies
People start posting tweets containing texts, images, and videos as soon as a disaster hits an area. The analysis of these disaster-related tweet texts, images, and videos can help humanitarian response organizations in better decision-making and prioritizing their tasks. Finding the informative contents which can help in decision making out of the massive volume of Twitter content is a difficult task and require a system to filter out the informative contents. In this paper, we present a multi-modal approach to identify disaster-related informative content from the Twitter streams using text and images together. Our approach is based on long-short-term-memory and VGG-16 networks that show significant improvement in the performance, as evident from the validation result on seven different disaster-related datasets. The range of F1-score varied from 0.74 to 0.93 when tweet texts and images used together, whereas, in the case of only tweet text, it varies from 0.61 to 0.92. From this result, it is evident that the proposed multi-modal system is performing significantly well in identifying disaster-related informative social media contents.
KeywordsDisaster Twitter LSTM VGG-16 Social media Tweets
- Aipe, A., Ekbal, A., Mukuntha, N., & Kurohashi, S. (2018). Linguistic feature assisted deep learning approach towards multi-label classification of crisis related tweets. In Proceedings of the 15th ISCRAM Conference (pp. 705–717).Google Scholar
- Alam, F., Imran, M., & Ofli, F. (2017). Image4act: Online social media image processing for disaster response. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 601–604). ACM.Google Scholar
- Alam, F., Ofli, F., & Imran, M. (2018). Crisismmd: Multimodal Twitter datasets from natural disasters. In Proceedings of the 12th international AAAI conference on web and social media (ICWSM). Google Scholar
- Ashktorab, Z., Brown, C., Nandi, M., & Culotta, A. (2014). Tweedr: Mining Twitter to inform disaster response. In Proceedings of the 11th ISCRAM conference (pp. 354–358).Google Scholar
- Cameron, M. A., Power, R., Robinson, B., & Yin, J. (2012). Emergency situation awareness from Twitter for crisis management. In Proceedings of the 21st international conference on world wide web (pp. 695–698). ACM.Google Scholar
- Caragea, C., Mcneese, N., Jaiswal, A., Traylor, G., Woo Kim, H., Mitra, P., et al. (2011). Classifying text messages for the Haiti earthquake. In Proceedings of the 8th international conference on information systems for crisis response and management (ISCRAM2011). Google Scholar
- Caragea, C., Silvescu, A., & Tapia, A. H. (2016). Identifying informative messages in disaster events using convolutional neural networks. In International conference on information systems for crisis response and management (pp. 137–147).Google Scholar
- Chaudhuri, N., & Bose, I. (2019). Application of image analytics for disaster response in smart cities. In Proceedings of the 52nd Hawaii international conference on system sciences. https://doi.org/10.24251/hicss.2019.367
- Daly, S., & Thom, J. A. (2016). Mining and classifying image posts on social media to analyse fires. In Proceedings of the 13th ISCRAM conference (pp. 1–14).Google Scholar
- Dubey, R. (2019). Developing an integration framework for crowdsourcing and internet of things with applications for disaster response. In Social entrepreneurship: Concepts, methodologies, tools, and applications (pp. 274–283). IGI Global.Google Scholar
- Dubey, R., Gunasekaran, A., Childe, S. J., Roubaud, D., Wamba, S. F., Giannakis, M., et al. (2019). Big data analytics and organizational culture as complements to swift trust and collaborative performance in the humanitarian supply chain. International Journal of Production Economics,210, 120–136.CrossRefGoogle Scholar
- Graf, D., Retschitzegger, W., Schwinger, W., Pröll, B., & Kapsammer, E. (2018). Cross-domain informativeness classification for disaster situations. In Proceedings of the 10th international conference on management of digital ecosystems (pp. 183–190). ACM.Google Scholar
- Guha-Sapir, D., Vos, F., Below, R., & Ponserre, S. (2012). Annual disaster statistical review 2011: The number sandtrends. Technical Report Centre for Research on the Epidemiology of Disasters (CRED).Google Scholar
- Gupta, A., Lamba, H., Kumaraguru, P., & Joshi, A. (2013, May). Faking sandy: Characterizing and identifying fake images on Twitter during hurricane sandy. In Proceedings of the 22nd international conference on world wide web (pp. 729–736). ACM.Google Scholar
- Imran, M., Castillo, C., Lucas, J., Meier, P., & Vieweg, S. (2014). Aidr: Artificial intelligence for disaster response. In Proceedings of the 23rd international conference on world wide web (pp. 159–162). ACM.Google Scholar
- Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013a). Extracting information nuggets from disaster-related messages in social media. In Proceedings of the 10th ISCRAM conference (pp. 791–801).Google Scholar
- Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013b). Practical extraction of disaster-relevant information from social media. In Proceedings of the 22nd international conference on world wide web (pp. 1021–1024). ACM.Google Scholar
- Jabbour, C. J. C., Sobreiro, V. A., de Sousa Jabbour, A. B. L., de Souza Campos, L. M., Mariano, E. B., & Renwick, D. W. S. (2017). An analysis of the literature on humanitarian logistics and supply chain management: Paving the way for future studies. Annals of Operations Research. https://doi.org/10.1007/s10479-017-2536-x.CrossRefGoogle Scholar
- Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Kumar, A., Singh, J. P., & Rana, N. P. (2017). Authenticity of geo-location and place name in tweets. In Proceedings of the 23rd Americas conference on information systems (AMCIS) (pp. 1–9)Google Scholar
- Kumar, A., & Singh, J. P., Saumya, S. (2019). A comparative analysis of machine learning techniques for disaster related tweet classification. In IEEE region 10 humanitarian technology conference, (pp. 222–227).Google Scholar
- Li, H., Guevara, N., Herndon, N., Caragea, D., Neppalli, K., Caragea, C., et al. (2015). Twitter mining for disaster response: A domain adaptation approach. In Proceedings of the 12th ISCRAM conference.Google Scholar
- Mouzannar, H., Rizk, Y., & Awad, M. (2018). Damage identification in social media posts using multimodal deep learning. In ISCRAM.Google Scholar
- Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807–814).Google Scholar
- Nguyen, D. T., Al Mannai, K. A., Joty, S., Sajjad, H., Imran, M., & Mitra, P. (2017a). Robust classification of crisis-related data on social networks using convolutional neural networks. In 11th international conference on web and social media, ICWSM 2017 (pp. 632–635). AAAI Press.Google Scholar
- Nguyen, D. T., Alam, F., Ofli, F., & Imran, M. (2017b). Automatic image filtering on social networks using deep learning and perceptual hashing during crises. arXiv preprint arXiv:1704.02602.
- Nguyen, D. T., Joty, S., Imran, M., Sajjad, H., & Mitra, P. (2016). Applications of online deep learning for crisis response using social media information. arXiv preprint arXiv:1610.01030.
- Nguyen, D. T., Ofli, F., Imran, M., & Mitra, P. (2017c). Damage assessment from social media imagery data during disasters. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 569–576). ACM.Google Scholar
- Olteanu, A., Castillo, C., Diaz, F., & Vieweg, S. (2014). Crisislex: A lexicon for collecting and filtering microblogged communications in crises. In Eighth international AAAI conference on weblogs and social media.Google Scholar
- Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).Google Scholar
- Rizk, Y., Jomaa, H. S., Awad, M., & Castillo, C. (2019). A computationally efficient multi-modal classification approach of disaster-related Twitter images. In Proceedings of the 34th ACM/SIGAPP symposium on applied computing (pp. 2050–2059). ACM.Google Scholar
- Rudra, K., Banerjee, S., Ganguly, N., Goyal, P., Imran, M., & Mitra, P. (2016). Summarizing situational tweets in crisis scenario. In Proceedings of the 27th ACM conference on hypertext and social media (pp. 137–147). ACM.Google Scholar
- Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
- Sundermeyer, M., Schlüter, R., & Ney, H. (2012). LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association. Google Scholar
- Verma, S., Vieweg, S., Corvey, W. J., Palen, L., Martin, J. H., Palmer, M., et al. (2011). Natural language processing to the rescue? Extracting” situational awareness” tweets during mass emergency. In Fifth international AAAI conference on weblogs and social media. Google Scholar
- Yu, M., Huang, Q., Qin, H., Scheele, C., & Yang, C. (2019). Deep learning for real-time social media text classification for situation awareness using hurricanes sandy, Harvey, and Irma as case studies. International Journal of Digital Earth. https://doi.org/10.1080/17538947.2019.1574316.CrossRefGoogle Scholar