Multimedia Tools and Applications

, Volume 76, Issue 6, pp 7897–7919 | Cite as

Graph-based multimodal clustering for social multimedia

  • Georgios Petkos
  • Manos Schinas
  • Symeon Papadopoulos
  • Yiannis Kompatsiaris
Article

Abstract

Real world datasets often consist of data expressed through multiple modalities. Clustering such datasets is in most cases a challenging task as the involved modalities are often heterogeneous. In this paper we propose a graph-based multimodal clustering approach. The proposed approach utilizes an example relevant clustering in order to learn a model of the “same cluster” relationship between a pair of items. This model is subsequently used in order to organize the items of the collection to be clustered in a graph, where the nodes represent the items and a link between a pair of nodes exists if the model predicted that the corresponding pair of items belong to the same cluster. Eventually, a graph clustering algorithm is applied on the graph in order to produce the final clustering. The proposed approach is applied on two problems that are typically treated using clustering techniques; in particular, it is applied on the problem of detecting social events and to the problem of discovering different landmark views in collections of social multimedia.

Keywords

Multimodal clustering Social multimedia Social event detection Multimedia 

References

  1. 1.
    Aggarwal C, Subbian K (2014) Evolutionary network analysis: a survey. ACM Comput Surv 47(1):10:1–10:36. doi:10.1145/2601412 CrossRefMATHGoogle Scholar
  2. 2.
    Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-Up Robust features (SURF). Comp Vis Image Underst 110(3):346–359. doi:10.1016/j.cviu.2007.09.014 CrossRefGoogle Scholar
  3. 3.
    Becker H, Naaman M, Gravano L (2010) Learning similarity metrics for event identification in social media. In: Proceedings of the third ACM international conference on Web search and data mining, WSDM ’10. ACM, New York, pp 291–300. doi:10.1145/1718487.1718524 CrossRefGoogle Scholar
  4. 4.
    Bekkerman R, Jeon J (2007) Multi-modal clustering for multimedia collections. In: CVPRGoogle Scholar
  5. 5.
    Blei D , Ng A, Jordan M (2003) Latent dirichlet allocation. J Mach Learn Res 3(1532–4435):993–1022. http://dl.acm.org/citation.cfm?id=944919.944937 MATHGoogle Scholar
  6. 6.
    Boteanu B, Mironica I, Ionescu B (2015) Hierarchical clustering pseudo-relevance feedback for social image search result diversification. In: 2015 13th international workshop on Content-based multimedia indexing (CBMI), pp 1–6Google Scholar
  7. 7.
    Brenner M, Izquierdo E (2011) Mediaeval benchmark: social event detection in collaborative photo collections. In: Mediaeval, CEUR workshop proceedingsGoogle Scholar
  8. 8.
    Cai X, Nie F, Huang H, Kamangar F (2011) Heterogeneous image feature integration via multi-modal spectral clustering. In: 2011 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1977–1984. doi:10.1109/CVPR.2011.5995740
  9. 9.
    Clauset A, Newman MEJ, Moore C (2004) Finding community structure in very large networks. Phys Rev E:1–6. doi:10.1103/PhysRevE.70.066111
  10. 10.
    Dang-Nguyen D, Piras L, Giacinto G, Boato G, Natale FGBD (2014) Retrieval of diverse images by pre-filtering and hierarchical clustering. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  11. 11.
    Fortunato S (2010) Community detection in graphs. Phys Rep 486(3-5):75–174MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gînsca A, Popescu A, Rekabsaz N (2014) CEA List’s participation at the mediaeval 2014 retrieving diverse social images task. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  13. 13.
    Goder A, Filkov V (2008) Consensus clustering algorithms: Comparison and refinement. In: Munro JI, Wagner D (eds) Proceedings of the workshop on algorithm engineering and experiments, ALENEX 2008. SIAM, San Francisco, pp 109–117Google Scholar
  14. 14.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explorations Newsletter 11(1):10–18. doi:10.1145/1656274.1656278 CrossRefGoogle Scholar
  15. 15.
    Ionescu B, Popescu A, Lupu M, Gînsca A, Müller H (2014) Retrieving diverse social images at mediaeval 2014: challenge, dataset and evaluation. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  16. 16.
    Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image representation. In: 23rd IEEE conference on computer vision & pattern recognition (CVPR ’10). doi:10.1109/CVPR.2010.5540039. IEEE Computer Society, San Francisco, pp 3304–3311
  17. 17.
    Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128CrossRefGoogle Scholar
  18. 18.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. arXiv:1408.5093
  19. 19.
    Jian M, Dong J, Ma J (2011) Image retrieval using wavelet-based salient regions. Imaging Sci J 59(4):219–231. doi:10.1179/136821910X12867873897355 CrossRefGoogle Scholar
  20. 20.
    Khalidov V, Forbes F, Horaud RP (2011) Conjugate mixture models for clustering multimodal data. Neural Comput 23(2):517–557. http://perception.inrialpes.fr/Publications/2011/KFH11 MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Li Y, Crandall DJ, Huttenlocher DP (2009) Landmark classification in large-scale image collections. In: IEEE 12th international conference on computer vision, ICCV 2009. IEEE, Kyoto, pp 1957–1964. doi:10.1109/ICCV.2009.5459432 Google Scholar
  22. 22.
    Liu X, Troncy R, Huet B (2011) Using social media to identify events. In: WSM’11, ACM Multimedia 3rd workshop on social media, november 18-december 1st, 2011, ScottsdaleGoogle Scholar
  23. 23.
    Manning CD, Raghavan P, Schütze H (2008) Introduction to information retrieval. Cambridge University Press, New YorkCrossRefMATHGoogle Scholar
  24. 24.
    Nguyen NP, Dinh TN, Xuan Y, Thai MT (2011) Adaptive algorithms for detecting community structure in dynamic social networks. In: INFOCOM 2011. 30th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 10-15 April 2011. IEEE, Shanghai, pp 2282–2290. doi:10.1109/INFCOM.2011.5935045 Google Scholar
  25. 25.
    Papadopoulos S, Kompatsiaris Y, Vakali A, Spyridonos P (2012) Community detection in social media. Data Min Knowl Disc 24(3):515–554CrossRefGoogle Scholar
  26. 26.
    Papadopoulos S, Schinas E, Mezaris V, Troncy R, Kompatsiaris Y (2012) Social Event Detection at MediaEval 2012: challenges, dataset and evaluation. In: Mediaeval. Workshop, Pisa, p 2012Google Scholar
  27. 27.
    Papadopoulos S, Troncy R, Mezaris V, Huet B, Kompatsiaris I (2011) Social event detection at mediaeval 2011: challenges, dataset and evaluation Mediaeval, CEUR workshop proceedingsGoogle Scholar
  28. 28.
    Papadopoulos S, Zigkolis C, Kompatsiaris Y, Vakali A (2011) CERTH@Mediaeval 2011 social event detection task. In: Mediaeval, CEUR workshop proceedingsGoogle Scholar
  29. 29.
    Papadopoulos S, Zigkolis C, Kompatsiaris Y, Vakali A (2011) Cluster-based landmark and event detection for tagged photo collections. Multimedia, IEEE 18 (1):52–63. doi:10.1109/MMUL.2010.68 CrossRefGoogle Scholar
  30. 30.
    Petkos G, Papadopoulos S, Kompatsiaris Y (2012) Social event detection using multimodal clustering and integrating supervisory signals. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, ICMR ’12. ACM, New York, pp 23:1–23:8. doi:10.1145/2324796.2324825
  31. 31.
    Petkos G, Papadopoulos S, Mezaris V, Kompatsiaris Y (2014) Social event detection at mediaeval 2014: challenges, datasets, and evaluation. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  32. 32.
    Petkos G, Papadopoulos S, Schinas E, Kompatsiaris Y (2014) Graph-based multimodal clustering for social event detection in large collections of images Multimedia modeling - 20th anniversary international conference, MMM 2014, Dublin, Ireland, January 6–10, 2014, Proceedings, Part I, pp 146–158Google Scholar
  33. 33.
    Phuvipadawat S, Murata T (2010) Breaking news detection and tracking in twitter. IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology 3:120–123. http://doi.ieeecomputersociety.org/10.1109/WI-IAT.2010.205 CrossRefGoogle Scholar
  34. 34.
    Rendle S, Schmidt-Thieme L (2008) Scaling record linkage to non-uniform distributed class sizes. In: Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining, PAKDD’08. Springer, Berlin, pp 308–319. http://dl.acm.org/citation.cfm?id=1786574.1786605 CrossRefGoogle Scholar
  35. 35.
    Reuter T, Cimiano P (2012) Event-based classification of social media streams. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, ICMR ’12. ACM, New York, pp 22:1–22:8. doi:10.1145/2324796.2324824 Google Scholar
  36. 36.
    Reuter T, Papadopoulos S, Petkos G, Mezaris V, Kompatsiaris Y, Cimiano P, Vries CMD, Geva S (2013) Social event detection at mediaeval 2013: challenges, datasets, and evaluation. In: Proceedings of the mediaeval 2013 multimedia benchmark workshop, BarcelonaGoogle Scholar
  37. 37.
    Samangooei S, Hare JS, Dupplaw D, Niranjan M, Gibbins N, Lewis PH, Davies J, Jain N, Preston J (2013) Social event detection via sparse multi-modal feature selection and incremental density based clustering. In: Proceedings of the mediaeval 2013 multimedia benchmark workshop, BarcelonaGoogle Scholar
  38. 38.
    Snoek CGM, Worring M, Smeulders AWM (2005) Early versus late fusion in semantic video analysis. In: Proceedings of the 13th annual ACM international conference on Multimedia, MULTIMEDIA ’05. ACM, New York, pp 399–402. doi:10.1145/1101149.1101236 CrossRefGoogle Scholar
  39. 39.
    Spiliopoulou M., Aggarwal CC (2011) Evolution in social networks: a survey. In: Social network data analytics, Springer, pp 149–175Google Scholar
  40. 40.
    Xu X, Yuruk N, Feng Z, Schweiger TAJ (2007) Scan: a structural clustering algorithm for networks. In: Proceedings of the 13th ACM SIGKDD, KDD ’07. ACM, NY, pp 824–833. doi:10.1145/1281192.1281280
  41. 41.
    Ye Z, Hu S, Yu J (2008) Adaptive clustering algorithm for community detection in complex networks. Phys Rev E 78(4)Google Scholar
  42. 42.
    Zaharieva M, Schopfhauser D, del Fabro M, Zeppelzauer M (2014) Clustering and retrieval of social events in flickr. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  43. 43.
    Zaharieva M, Schwab P (2014) A unified framework for retrieving diverse social images. In: Working notes proceedings of the mediaeval 2014 workshop, BarcelonaGoogle Scholar
  44. 44.
    Zhang T, Ramakrishnan R, Livny M (1996) Birch: An efficient data clustering method for very large databases. SIGMOD Rec 25(2):103–114. doi:10.1145/235968.233324 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.CERTH - ITIThessalonikiGreece

Personalised recommendations