Describing Images with Ontology-Aware Dictionary Learning

Abstract

In this paper, we focus on the generation of contextual descriptions for images by learning an ontology-aware dictionary. Ontology deals with questions concerning what entities exist and how such entities can be related with a hierarchy. Thus, if we incorporate the semantic hierarchies of visual concepts into a learned visual dictionary, which consists of visual atoms, we can generate contextual descriptions of testing images through the reconstruction. This paper proposes to learn the ontology-aware dictionary by integrating hierarchical dictionary learning and multi-task regression into a joint framework. By utilizing a hierarchical regularization term defined on the multiple semantic categories, the hierarchical structures are introduced into the multi-task regression. The joint optimization of the sparse coding and multi-task regression makes the semantic hierarchies embedded into the learned dictionary. Experiments on two benchmark datasets show the better performance of the proposed algorithm. Examples of the ontology-aware dictionary and generated image descriptions successfully demonstrate the effectiveness of the proposed framework.

Y. Han was partly supported by the NSFC (under Grant 61202166 and 61472276) and the Major Project of National Social Science Fund of China (under Grant 14ZDB153).