Skip to main content
SpringerLink
Log in

Social media and satellites

Disaster event detection, linking and summarization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time. In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data. To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. http://landsat.usgs.gov/

  2. https://www.planet.com

  3. http://www.multimediaeval.org/mediaeval2017/multimediasatellite/index.html

  4. http://www.acmmm.org/2016/wp-content/uploads/2016/03/ACMMM16_GC_Sky_and_the_Social_Eye_latest.pdf

  5. Real-time in the context of information retrieval means in our case that JORD continuously monitors various information sources and retrieves the information as soon as a query match is found.

  6. https://landsat.gsfc.nasa.gov/landsat-1/

  7. http://www.satimagingcorp.com/applications/environmental-impact-studies/natural-disasters/

  8. https://cloud.google.com/translate/docs/

  9. http://caffe.berkeleyvision.org/

  10. http://www.vlfeat.org/matconvnet/

  11. https://textblob.readthedocs.io/en/dev/

  12. https://ttv.microworkers.com

  13. https://textblob.readthedocs.io/en/dev/

  14. http://www.multimediaeval.org/mediaeval2017/

References

  1. Ahmad K, De Natale F, Boato G, Rosani A (2016) A hierarchical approach to event discovery from single images using mil framework. In: 2016 IEEE global conference on Signal and information processing (globalSIP). IEEE, pp 1223–1227

  2. Ahmad K, Konstantin P, Riegler M, Conci N, Halvorsen p. (2017) CNN and GAN based satellite and social media data fusion for disaster detection. In: Working notes proceedings of the Mediaeval workshop, pp 2

  3. Ahmad K, Riegler M, Pogorelov K, Conci N, Halvorsen P, De Natale F (2017) Jord: a system for collecting information and monitoring natural disasters by linking social media with satellite imagery. In: Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing. ACM, p 12

  4. Ahmad S, Ahmad K, Ahmad N, Conci N (2017) Convolutional neural networks for disaster images retrieval. In: Proceedings of the MediaEval. Workshop, Dublin

  5. Ahmad K, Conci N, De Natale F (2018) A saliency-based approach to event recognition. Signal Process Image Commun 60:42–51

    Article  Google Scholar 

  6. Amit SNKB, Shiraishi S, Inoshita T, Aoki Y (2016) Analysis of satellite images for disaster detection. In: 2016 IEEE international Geoscience and remote sensing symposium (IGARSS). IEEE, pp 5189–5192

  7. Atefeh F, Khreich W (2015) A survey of techniques for event detection in twitter. Comput Intell 31(1):132–164

    Article  MathSciNet  Google Scholar 

  8. Bakillah M, Li RY, Liang SH (2015) Geo-located community detection in twitter with enhanced fast-greedy optimization of modularity: the case study of typhoon haiyan. Int J Geogr Inf Sci 29(2):258– 279

    Article  Google Scholar 

  9. Bischke B, Borth D, Schulze C, Dengel A (2016) Contextual enrichment of remote-sensed events with social media streams. Proceedings of the 2016 ACM on Multimedia Conference. ACM, pp 1077–1081

  10. Bischke B, Bhardwaj P, Gautam A, Helber P, Borth D, Dengel A (2017) Detection of flooding events in social multimedia and satellite imagery using deep neural networks

  11. Bischke B, Helber P, Schulze C, Venkat S, Dengel A, Borth D (2017) The multimedia satellite task at mediaeval 2017: Emergence response for flooding events. In: Proceedings of the MediaEval. Workshop, Dublin

  12. Campbell JB, Wynne RH (2011) Introduction to remote sensing. Guilford Press, New York

    Google Scholar 

  13. Chang SF (2016) New frontiers of large scale multimedia information retrieval. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. ACM, pp. 5–5

  14. Chavez P, Sides SC, Anderson JA (1991) Comparison of three different methods to merge multiresolution and multispectral data- landsat tm and spot panchromatic. Photogramm Eng Remote Sens 57(3):295–303

    Google Scholar 

  15. Chen T, Borth D, Darrell T, Chang SF (2014) Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv:1410.8586

  16. Cheong M, Lee V (2010) Twittering for earth: a study on the impact of microblogging activism on earth hour 2009 in australia. In: Springer ACIIDS. Springer, pp. 114–123

  17. Crooks A, Croitoru A, Stefanidis A, Radzikowski J (2013) # earthquake: Twitter as a distributed sensor system. Trans GIS 17(1):124–147

    Article  Google Scholar 

  18. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009. CVPR 2009. IEEE conference on Computer vision and pattern recognition. IEEE, pp. 248–255

  19. Du R, Yu Z, Mei T, Wang Z, Wang Z, Guo B (2014) Predicting activity attendance in event-based social networks: Content, context and social influence. In: Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp. 425–434

  20. Earle PS, Bowden DC, Guy M (2012) Twitter earthquake detection: earthquake monitoring in a social world. Annals of Geophysics 54(6):708–715

  21. Erol B, Hull JJ, Lee DS (2003) Linking multimedia presentations with their symbolic source documents: algorithm and applications. In: Proceedings of ACM MM, pp 498–507

  22. Fisher A, Flood N, Danaher T (2016) Comparing landsat water index methods for automated water classification in eastern australia. IJRSE 175:167–182

    Google Scholar 

  23. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  24. Guha-Sapir D, Below R, Hoyois P (2015) Em-dat: International disaster database. Catholic University of Louvain, Brussels

    Google Scholar 

  25. Guille A, Favre C (2015) Event detection, tracking, and visualization in twitter: a mention-anomaly-based approach. Soc Netw Anal Min 5(1):1–18

    Article  Google Scholar 

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  27. Joyce KE, Belliss SE, Samsonov SV, McNeill SJ, Glassey PJ (2009) A review of the status of satellite remote sensing and image processing techniques for mapping natural hazards and disasters. Progress in Physical Geography

  28. Jung M, Henkel K, Herold M, Churkina G (2006) Exploiting synergies of global land cover products for carbon cycle modeling. IJRSE 101(4):534–553

    Google Scholar 

  29. Kamilaris A, Prenafeta-Boldú FX Disaster monitoring using unmanned aerial vehicles and deep learning

  30. Kansas J, Vargas J, Skatter HG, Balicki B, McCullum K (2016) Using landsat imagery to backcast fire and post-fire residuals in the boreal shield of saskatchewan: implications for woodland caribou management. IJWF 25(5):597–607

    Google Scholar 

  31. Keiller N, Samuel F, Ícaro D, Rafael W, Javier M, Otávio P, Rodrigo C (2017) Data-driven flood detection using neural networks. In: Proceedings of the MediaEval. Workshop, Dublin

  32. Konstantinos A, Anastasia M, Andreadis S, Emmanouil M, Ilias G, Stefanos V, Ioannis K (2017) Visual and textual analysis of social media and satellite images for flood detection @ multimedia satellite task mediaeval 2017. In: Proceedings of the MediaEval. Workshop, Dublin

  33. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  34. Laura LF, Joost W, Marc B, Harald S (2017) Multi-modal deep learning approach for flood detection. In: Proceedings of the MediaEval. Workshop, Dublin

  35. Li C, Sun A, Datta A (2012) Twevent: segment-based event detection from tweets. In: Proceedings of ACM IKM. ACM, pp 155–164

  36. Liu Y, Wu L (2016) Geological disaster recognition on optical remote sensing images using deep learning. Proced Comput Sci 91:566–575

    Article  Google Scholar 

  37. Lux M, Riegler M, Halvorsen P, Pogorelov K, Anagnostopoulos N (2016) Lire: open source visual information retrieval. In: Proceedings of ACM MMSys

  38. Manjunath T, Hegadi RS, Ravikumar G (2010) A survey on multimedia data mining and its relevance today. IJCSNS 10(11):165–170

    Google Scholar 

  39. Mathioudakis M, Koudas N (2010) Twittermonitor: trend detection over the twitter stream. In: Proceedings of ACM SIGMOD, pp 1155–1158

  40. Meladianos P, Nikolentzos G, Rousseau F, Stavrakas Y, Vazirgiannis M (2015) Degeneracy-based real-time sub-event detection in twitter stream. In: Ninth international AAAI conference on web and social media, pp 248–257

  41. Minh-Son D, Quang-Nhat-Minh P, Duc-Tien DN (2017) A domain-based late-fusion for disaster image retrieval from social media. In: Proceedings of the MediaEval. Workshop. Dublin

  42. Muhammad H, Muhammad A, Mahrukh K, Mohammad R (2017) Flood detection using social media data and spectral regression based kernel discriminant analysis. In: Proceedings of the MediaEval. Workshop, Dublin

  43. Nataliya T, Arkaitz Z, Procter R (2017) Wisc at mediaeval 2017: Multimedia satellite task. In: Proceedings of the MediaEval. Workshop, Dublin

  44. Nurwidyantoro A, Winarko E (2013) Event detection in social media: a survey. In: 2013 international conference on ICT For smart society (ICISS). IEEE, pp 1–5

  45. Paul F, Andreassen LM (2009) A new glacier inventory for the svartisen region, norway, from landsat etm+ data: challenges and change assessment. J Glaciol 55(192):607–618

    Article  Google Scholar 

  46. Pogorelov K, Randel KR, Griwodz C, Eskeland SL, de Lange T, Johansen D, Spampinato C, Dang-Nguyen DT, Lux M, Schmidt PT, Riegler M, Halvorsen P (2017) Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, MMSys’17. ACM, New York, pp 164–169. https://doi.org/10.1145/3083187.3083212

  47. Pogorelov K, Riegler M, Eskeland SL, de Lange T, Johansen D, Griwodz C, Schmidt PT, Halvorsen P (2017) Efficient disease detection in gastrointestinal videos–global features versus neural networks. Multimedia Tools and Applications 76:1–33

  48. Popescu AM, Pennacchiotti M (2010) Detecting controversial events from twitter. In: Proceedings of ACM IKM. ACM, pp 1873–1876

  49. Riegler M, Gaddam VR, Larson M, Eg R, Halvorsen P, Griwodz C (2016) Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks. In: 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI), pp 1–6. https://doi.org/10.1109/CBMI.2016.7500256

  50. Sakaki T, Okazaki M, Matsuo Y (2010) Earthquake shakes twitter users: real-time event detection by social sensors. In: Proceedings of the 19th international conference on World wide web. ACM, pp 851–860

  51. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  52. Son J, Park SJ, Jung KH (2017) Retinal vessel segmentation in fundoscopic images with generative adversarial networks. arXiv:1706.09318

  53. Stelter B, Cohen N (2008) Citizen journalists provided glimpses of mumbai attacks. The New York Times:30

  54. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  55. Takahashi B, Tandoc EC, Carmichael C (2015) Communicating on twitter during a disaster: an analysis of tweets during typhoon haiyan in the philippines. Comput Hum Behav 50:392–398

    Article  Google Scholar 

  56. Team P (2016) Planet application program interface: In space for life on earth. san francisco, ca

  57. Tzelepis C, Ma Z, Mezaris V, Ionescu B, Kompatsiaris I, Boato G, Sebe N, Yan S (2016) Event-based media processing and analysis: A survey of the literature. Image and Vision Computing

  58. Wood H (2002) The use of earth observing satellites for hazard support: Assessments and scenarios. Final Report of the CEOS Disaster Management Support Group. Available from < http://disaster.ceos.org/legal. cfm

  59. Xu Z, Zhang H, Sugumaran V, Choo KKR, Mei L, Zhu Y (2016) Participatory sensing-based semantic and spatial analysis of urban emergency events using mobile social media. EURASIP J Wirel Commun Netw 2016(1):44

    Article  Google Scholar 

  60. Yan C, Xie H, Liu S, Yin J, Zhang Y, Dai Q (2017) Effective uyghur language text detection in complex background images for traffic prompt identification. IEEE transactions on intelligent transportation systems

  61. Yan C, Xie H, Yang D, Yin J, Zhang Y, Dai Q (2017) Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE transactions on intelligent transportation systems

  62. Yin J, Lampert A, Cameron M, Robinson B, Power R (2012) Using social media to enhance emergency situation awareness. IJIS 27(6):52–59

    Google Scholar 

  63. Zhengyu Z, Larson M (2017) Retrieving social flooding images based on multimodal information. In: Proceedings of the MediaEval. Workshop, Dublin

  64. Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. In: Proceedings of the NIPS, pp 487–495

Download references

Author information

Authors and Affiliations

  1. DISI-University of Trento, Via Sammarive 5, Trento, Italy

    Kashif Ahmad, Konstantin Pogorelov, Michael Riegler, Nicola Conci & Pål Halvorsen

Authors
  1. Kashif Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Konstantin Pogorelov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Riegler
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicola Conci
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pål Halvorsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kashif Ahmad.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmad, K., Pogorelov, K., Riegler, M. et al. Social media and satellites. Multimed Tools Appl 78, 2837–2875 (2019). https://doi.org/10.1007/s11042-018-5982-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5982-9

Keywords

Search

Navigation