Fusion Models for Improved Image Captioning

Abstract

Visual captioning aims to generate textual descriptions given images or videos. Traditionally, image captioning models are trained on human annotated datasets such as Flickr30k and MS-COCO, which are limited in size and diversity. This limitation hinders the generalization capabilities of these models while also rendering them liable to making mistakes. Language models can, however, be trained on vast amounts of freely available unlabelled data and have recently emerged as successful language encoders [10] and coherent text generators [4]. Meanwhile, several unimodal and multimodal fusion techniques have been proven to work well for natural language generation [11] and automatic speech recognition [30]. Building on these recent developments, and with the aim of improving the quality of generated captions, the contribution of our work in this paper is two-fold: First, we propose a generic multimodal model fusion framework for caption generation as well as emendation where we utilize different fusion strategies to integrate a pretrained Auxiliary Language Model (AuxLM) within the traditional encoder-decoder visual captioning frameworks. Next, we employ the same fusion strategies to integrate a pretrained Masked Language Model (MLM), namely BERT, with a visual captioning model, viz. Show, Attend, and Tell, for emending both syntactic and semantic errors in captions. Our caption emendation experiments on three benchmark image captioning datasets, viz. Flickr8k, Flickr30k, and MSCOCO, show improvements over the baseline, indicating the usefulness of our proposed multimodal fusion strategies. Further, we perform a preliminary qualitative analysis on the emended captions and identify error categories based on the type of corrections.

A Appendix

Here we provide examples of (token) corrections made by the fusion models and categorize the edits into one of the following five categories: (i) Gender (ii) Color (iii) Specificity (iv) Syntactic (v) Semantic, based on the nature of change.

The above classification has been provided only for the purpose of preliminary illustration. For a thorough understanding of the trends in caption emendations and to draw conclusions, a detailed study using human evaluation should be performed on all three datasets. We leave this aspiration for future work. In the following examples, we color the incorrect tokens in , the correct replacements in , and the equally valid tokens in .

Semantic Correction. This section presents examples where the fusion models have corrected few tokens in the baseline captions so as to make them semantically valid with respect to the image. Edits to achieve semantic correctness may include emendation of visual attributes such as colors, objects, object size, etc. (Figs. 3 and 4).

Fig. 3.

An example illustrating the correction of semantic errors in the captions by our simple fusion model.

Fig. 4.

Another example to show correction of semantic errors with cold fusion.

Gender Alteration. This section provides an illustration of the case where the fusion models corrected the wrong gender of captions from the baseline model (Fig. 5).

Fig. 5.

An example of cold fusion approach achieving gender correction.

Specificity. This deals with emendations of fusion models where the corrected captions end up describing the images more precisely than the baseline captions (Fig. 6).

Fig. 6.

An example to show achievement of specificity with hierarchical fusion.

Syntactic Correction. In this section, we show an example to demonstrate the case where syntactic errors such as token repetitions in the baseline captions are correctly emended by the fusion models (Fig. 7).

Fig. 7.

Replacement of repetitive tokens with a correct alternative.

Color Correction. In this part, we show an example to illustrate the case where the fusion models emended color attributes in the captions of the baseline model (Fig. 8).

Fig. 8.

An example of cold fusion achieving emendation of color attribute.