Data Fusion Across Traditional and Social Media

  • Werner Bailer
  • Gert Kienast
  • Georg Thallinger
  • Gerhard Backfried


Crises and disasters are covered continuously and without interruption by today’s media, especially social media. There is not a single significant occurrence within the flow of events which they do not document. Consequently, the information contained in media—especially social media like Facebook and Twitter—provides an often neglected potential which should not be overlooked. Through fusion of sources, diverse, mixed, and complementary types of information can be tapped into and combined. The difficulty of this process is to view, channel, prepare, and exploit this inhomogeneous and enormous amount of information. Automatic monitoring of traditional as well as social media sources allows to deriving risk factors and risk indicators for crises and disaster events quickly. Intelligence derived from this process allows for earlier and swifter reaction to potential situations of crisis and interrelationships. Current publicly described technical and electronic infrastructure for national and international crisis and disaster management is not able to perform comprehensive analyses of all media channels automatically. The continuous developments in the areas of multimedia and social media demand the creation of adequate methods of processing. Relevant manifestations of events are to be identified automatically from documents from traditional (TV, radio, web) as well as social media and document clusters of the examined multimedia documents are to be presented to situational awareness experts. The focus of the Quelloffene IntegrierteMultimedia Analyse (QuOIMA) project is on the research on and development of algorithms and methods to achieve this goal. Automatic analysis of content in the multimedia and social media domain forms a fundamental innovation. From a technical, social studies, and scientific point of view, the targeted insights and findings of this project, form a fundamental contribution to security research, reaching far beyond the quality of existing systems. The integration of findings regarding situational awareness will provide more realistic risk assessment increasing their possibilities to (re)act. End users extend their expertise and as a consequence the ability of the overall organizations to act.


  1. Bailer W, Weiss W, Kienast G, Thallinger G, Haas W (2010) A video browsing tool for content management in post-production. Int J Digit Multimed Broadcast 2010, 856761. doi: 10.1155/2010/856761
  2. Bailer W, Lokaj M, Stiegler H (2014) Context in video search: is close-by good enough when using linking? In: Proceedings of ACM International Conference on Multimedia Retrieval, Glasgow, UK, pp 209–217Google Scholar
  3. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359. doi:10.1016/j.cviu.2007.09.014 CrossRefGoogle Scholar
  4. Bikel D, Miller S, Schwartz R, Weischedel R (1997) Nymble: a high-performance learning name-finder, ANLC’97 Proceedings of the fifth conference on applied natural language processing, Stroudsburg, PAGoogle Scholar
  5. Boser BE, Guyon I, Vapnik V (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the 5th Annual Workshop on Computational learning theory. ACM Press, New York, NY, pp 144–152Google Scholar
  6. Brugghemans B, Van de Walle B, Milis K (2013) Impact of the distribution and enrichment of information on the coordination of a human-made fast-burning crisis. In: Proceedings of the 10th International Conference on Information Systems for Crisis Responses and Management (ISCRAM 2013), Baden-Baden, Germany, pp 89–93Google Scholar
  7. Bruno E, Marchand-Maillet S (2009) Multiview clustering: a late fusion approach using latent models. In: Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval (SIGIR’09), Boston, MAGoogle Scholar
  8. Bruns A, Burgess J, Crawford K, Shaw F (2012) #qldfloods and @QPSMedia: crisis communication on Twitter in the 2011 South East Queensland Floods. ARC Centre of Excellence for Creative Industries and Innovation, Brisbane, QLDGoogle Scholar
  9. Costantini L, Nicolussi R (2011) Image clustering fusion technique based on BFS. In: Proceedings of the 20th ACM international conference on Information and knowledge management (CIKM’11), Glasgow, UKGoogle Scholar
  10. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Computer Society Conference on Computer vision and pattern recognition (CVPR’05), vol 1, San Diego, CA, pp 886–893Google Scholar
  11. Duan L-Y, Gao F, Chen J, Lin J, Huang T (2013) Compact descriptors for mobile visual search and MPEG CDVS standardization. In: IEEE International Symposium on Circuits and Systems (ISCAS), Beijing, China, 19–23 May 2013, pp 885–888Google Scholar
  12. Duda RO, Hart PE, Stork DG (2000) Pattern classification. Wiley-Interscience, New York, NYGoogle Scholar
  13. Fei-Fei L, Perona P (2005) A Bayesian hierarchical model for learning natural scene categories. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol 2, San Diego, CA, pp 524Google Scholar
  14. Fred ALN, Jain AK (2005) Combining multiple clusterings using evidence accumulation. IEEE Trans Pattern Anal Mach Intell 27(6):835–850CrossRefGoogle Scholar
  15. Frossyniotis DS, Pertselakis M, Stafylopatis A (2002) A multi-clustering fusion algorithm. In: Proceedings of the Second Hellenic Conference on AI: methods and applications of artificial intelligence (SETN’02), Thessaloniki, Greece, April 11–12, 2002Google Scholar
  16. Gao B, Liu T-Y, Qin T, Zheng X, Cheng Q-S, Ma W-Y (2005) Web image clustering by consistent utilization of visual features and surrounding texts. In: Proceedings of ACM Multimedia, Singapore, November 6–11, 2005, pp 112–121Google Scholar
  17. Hecht R, Riedler J, Backfried G, German broadcast news transcription. In: Proceedings of the International Conference on Spoken language processing, ICSLP 2002, Denver, CO, USA, September 2002Google Scholar
  18. Hsu WH, Chang S-F (2006) Topic tracking across broadcast news videos with visual duplicates and semantic concepts. In: IEEE International Conference on Image Processing, October 8–11, 2006, pp 141–144Google Scholar
  19. IPTC. International Press Telecommunications Council. Available from: Accessed on 24 June, 2014
  20. Johansson F, Brynielsson J, Narganes Quijano M (2012) Estimating citizen alertness in crises using social media monitoring and analysis, EISIC 2012, Odense, DenmarkGoogle Scholar
  21. Kohlschütter C, Frankhauser P, Nejdl W (2010) Boilerplate detection using shallow text features. In: Proceedings of the third ACM international conference on Web search and data mining (WSDM 2010), New York, NYGoogle Scholar
  22. Kubala F, Jin H, Nguyen L, Schwartz R, Matsoukas S (1997) Broadcast News Transcription. In: Proc of the ICASSP’97, vol 1, pp 203–206Google Scholar
  23. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRefGoogle Scholar
  24. Mendoza M, Poblete B, Castillo C (2010) Twitter under crisis: can we trust what we RT? In: SOMA 2010, Washington, DCGoogle Scholar
  25. Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10):1615–1630, CrossRefGoogle Scholar
  26. Mimaroglu S, Erdil E (2011) Combining multiple clusterings using similarity graph. Pattern Recogn 44(3):694–703CrossRefMATHGoogle Scholar
  27. Mimaroglu S, Erdil E (2013) An efficient and scalable family of algorithms for combining clusterings. Eng Appl Artif Intel 26(10):2525–2539CrossRefGoogle Scholar
  28. Muhren WJ, Van de Walle B (2010) Sense-making and information management in emergency response. Bull Am Soc Inf Sci Technol 36(5):30–33CrossRefGoogle Scholar
  29. Nilsson J et al. (2012) Making use of new Media for Pan-European crisis communication, ISCRAM 2012, Vancouver, CanadaGoogle Scholar
  30. Perronnin F, Dance C (2007) Fisher kernels on visual vocabularies for image categorization. In: Proc IEEE Conference on Computer vision and pattern recognition, Minneapolis, MN, pp 1–8Google Scholar
  31. Picard D, Gosselin P-H (2011) Improving image similarity with vectors of locally aggregated tensors. In: IEEE International Conference on Image Processing, Brussels, Belgium, pp669–672Google Scholar
  32. Rege M, Dong M, Hua J (2008) Graph theoretical framework for simultaneously integrating visual and textual features for efficient web image clustering. In: Proceedings of the 17th international conference on World Wide Web, Beijing, China, pp 317–326Google Scholar
  33. Shalunts G., Backfried G, Prinz K, Sentiment Analysis of German Social Media for Natural Disasters, , 11th International Conference of Information Systems for Crisis Response and Management, ISCRAM, 2014, University Park, PGoogle Scholar
  34. Snoek CGM, Worring M, Smeulders AWM (2005) Early versus late fusion in semantic video analysis. In: Proceedings of the 13th annual ACM international conference on Multimedia (MULTIMEDIA’05), Singapore, November 06–12, 2005Google Scholar
  35. Thelwall M, Buckley K, Paltoglou G, Cai D, Kappas A (2010) Sentiment strength detection in short informal text. J Am Soc Inf Sci Technol 61(12):2544–2558CrossRefGoogle Scholar
  36. Vega-Pons S, Ruiz-Shulcloper J (2011) A Survey of Clustering Ensemble Algorithms. Int J Pattern Recogn Artif Intell 25(3):337–372CrossRefMathSciNetGoogle Scholar
  37. Von Luxburg U (2007) A tutorial on spectral clustering. Stat Comput 17(4):395–416CrossRefMathSciNetGoogle Scholar
  38. Voorhees EM, Gupta NK, Johnson-Laird B (1995) Learning collection fusion strategies. In: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR’95), Seattle, WAGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Werner Bailer
    • 1
  • Gert Kienast
    • 1
  • Georg Thallinger
    • 1
  • Gerhard Backfried
    • 2
  2. 2.SAIL LABS TechnologyViennaAustria

Personalised recommendations