MILC²: A Multi-Layer Multi-Instance Learning Approach to Video Concept Detection

Abstract

Video is a kind of structured data with multi-layer (ML) information, e.g., a shot is consisted of three layers including shot, key-frame, and region. Moreover, multi-instance (MI) relation is embedded along the consecutive layers. Both the ML structure and MI relation are essential for video concept detection. The previous work [5] dealt with ML structure and MI relation by constructing a MLMI kernel in which each layer is assumed to have equal contribution. However, such equal weighting technique cannot well model MI relation or handle ambiguity propagation problem, i.e., the propagation of uncertainty of sub-layer label through multiple layers, as it has been proved that different layers have different contributions to the kernel. In this paper, we propose a novel algorithm named MILC² (Multi-Layer Multi-Instance Learning with Inter-layer Consistency Constraint.) to tackle the ambiguity propagation problem, in which an inter-layer consistency constraint is explicitly introduced to measure the disagreement of inter-layers, and thus the MI relation is better modeled. This learning task is formulated in a regularization framework with three components including hyper-bag prediction error, inter-layer inconsistency measure, and classifier complexity. We apply the proposed MILC² to video concept detection over TRECVID 2005 development corpus, and report better performance than both standard Support Vector Machine based and MLMI kernel methods.

This work was performed when the first author visited Microsoft Research Asia as an intern.