Multimedia Tools and Applications

, Volume 76, Issue 2, pp 2861–2889

Gaze movement-driven random forests for query clustering in automatic video annotation

  • Stefanos Vrochidis
  • Ioannis Patras
  • Ioannis Kompatsiaris
Article

DOI: 10.1007/s11042-015-3221-1

Cite this article as:
Vrochidis, S., Patras, I. & Kompatsiaris, I. Multimed Tools Appl (2017) 76: 2861. doi:10.1007/s11042-015-3221-1

Abstract

In the recent years, the rapid increase of the volume of multimedia content has led to the development of several automatic annotation approaches. In parallel, the high availability of large amounts of user interaction data, revealed the need for developing automatic annotation techniques that exploit the implicit user feedback during interactive multimedia retrieval tasks. In this context, this paper proposes a method for automatic video annotation by exploiting implicit user feedback during interactive video retrieval, as this is expressed with gaze movements, mouse clicks and queries submitted to a content-based video search engine. We exploit this interaction data to represent video shots with feature vectors based on aggregated gaze movements. This information is used to train a classifier that can identify shots of interest for new users. Subsequently, we propose a framework that during testing: a) identifies topics (expressed by query clusters), for which new users are searching for, based on a novel clustering algorithm and b) associates multimedia data (i.e., video shots) to the identified topics using supervised classification. The novel clustering algorithm is based on random forests and is driven by two factors: first, by the distance measures between different sets of queries and second by the homogeneity of the shots viewed during each query cluster defined by the clustering procedure; this homogeneity is inferred from the performance of the gaze-based classifier on these shots. The evaluation shows that the use of aggregated gaze data can be exploited for video annotation purposes.

Keywords

Implicit feedback Eye-tracking Interactive video retrieval Clustering Random forests 

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Stefanos Vrochidis
    • 1
  • Ioannis Patras
    • 2
  • Ioannis Kompatsiaris
    • 1
  1. 1.Centre for Research and Technology Hellas - Information Technologies InstituteThessalonikiGreece
  2. 2.Queen MaryUniversity of LondonLondonUK

Personalised recommendations