Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning

Abstract

Observing a set of images and their corresponding paragraph-captions, a challenging task is to learn how to produce a semantically coherent paragraph to describe the visual content of an image. Inspired by recent successes in integrating semantic topics into this task, this paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework, which couples a visual extractor with a deep topic model to guide the learning of a language model. To capture the correlations between the image and text at multiple levels of abstraction and learn the semantic topics from images, we design a variational inference network to build the mapping from image features to textual captions. To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model, including Long Short-Term Memory and Transformer, and jointly optimized. Experiments on public datasets demonstrate that the proposed models, which are competitive with many state-of-the-art approaches in terms of standard evaluation metrics, can be used to both distill interpretable multi-layer semantic topics and generate diverse and coherent captions.

A Appendix

1.1 A.1 The Variational Topic Encoder of VTCM

Inspired by Zhang et al. (2018), to approximate the gamma distributed topic weight vector $\varvec{\theta } ^{l}$ with a Weibull distribution, we assume the topic encoder as $q(\varvec{\theta } ^{l}|{\overline{{\varvec{v}} }}) = \text{ Weibull }({\varvec{k}} ^{l},\varvec{\lambda } ^{l})$, where the parameters ${\varvec{k}} ^{l}$ and $\varvec{\lambda } ^{l}$ of $\varvec{\theta } ^{l}$ can be denoted as

$$\begin{aligned}&{\varvec{k}} ^{l} = \ln [1+\exp (\mathbf{W} _{hk}^{l}{\varvec{h}} ^{l}+ {\varvec{b}} _1^{l})] , \end{aligned}$$

(15)

$$\begin{aligned}&\varvec{\lambda } ^{l} = \ln [1+\exp (\mathbf{W} _{h \lambda }^{l}{\varvec{h}} ^{l}+ {\varvec{b}} _2^{l})] , \end{aligned}$$

(16)

where ${\varvec{h}} ^{l}$ are deterministically nonlinearly transformed from the image pooled representation, stated as ${\varvec{h}} ^{0}={\overline{{\varvec{v}} }}$ and ${\varvec{h}} ^{l}\!=\!\text{ tanh } \left( \mathbf{W} _v^{l}{\varvec{h}} ^{l-1}+{\varvec{b}} _v^{l}\right)$.

1.2 A.2 The Gating Unit in VTCM-LSTM

Note the input ${\varvec{u}} _{j,t}^{l}$ of sentence-level LSTM at layer l combines the topic weight vectors $\varvec{\theta } ^{l}$ and hidden output of the sentence-level LSTM ${\varvec{h}} _{j,t}^{s,l}$ at each time step t. To realize ${\varvec{u}} _{j,t}^{l}=g\left( {\varvec{h}} _{j,t}^{s,l}, \varvec{\theta } ^{l}\right)$, we adopt a gating unit similar to the gated recurrent unit (GRU) (Cho et al. 2014), defined as

$$\begin{aligned} {\varvec{u}} _{j,t}^{l}&=\left( 1-{\varvec{z}} _{j,t}^{l}\right) \odot {\varvec{h}} _{j,t}^{s,l} + {\varvec{z}} _{j,t}^{l} \odot {\hat{{\varvec{h}} }}_{j,t}^{s,l}\,\, . \end{aligned}$$

(17)

where

$$\begin{aligned} {\varvec{z}} _{j,t}^{l}&= \sigma \left( \mathbf{W} _{z}^{l} \varvec{\theta } ^{l} + \mathbf{U} _{z}^{l} {\varvec{h}} _{j,t}^{s,l} + {\varvec{b}} _{z}^{l} \right) , \nonumber \\ {\varvec{r}} _{j,t}^{l}&= \sigma \left( \mathbf{W} _{r}^{l} \varvec{\theta } ^{l} + \mathbf{U} _{r}^{l} {\varvec{h}} _{j,t}^{s,l} + {\varvec{b}} _{r}^{l} \right) , \nonumber \\ {\hat{{\varvec{h}} }}_{j,t}^{s,l}&=\tanh \left( \mathbf{W} _{h}^{l} \varvec{\theta } ^{l} + \mathbf{U} _{h}^{l}\left( {\varvec{r}} _{j,t}^{l} \odot {\varvec{h}} _{j, t}^{s,l}\right) +{\varvec{b}} _{h}^{l}\right) . \end{aligned}$$

(18)

Define ${\varvec{u}} _{j,t}^{1:L}$ as the concatenation of ${\varvec{u}} _{j,t}^{l}$ across all layers and $\mathbf{W} _o$ as a weight matrix with V rows, the conditional distribution probability $p\left( w_{j,t} \,|\,w_{j,<t}, Img \right)$ of $w_{j,t}$ becomes

$$\begin{aligned} p\left( w_{j,t} \,|\,w_{j,<t}, {\varvec{v}} _{1:M},\varvec{\theta } ^{1:L}\right) = {\text {softmax}}\left( \mathbf{W} _o {\varvec{u}} _{j,t}^{1:L}\right) . \end{aligned}$$

(19)

There are two advantages to combine ${\varvec{u}} _{j,t}^{l}$ at all layers for language generation. First, the combination can enhance representation power because of different statistical properties at different stochastic layers of the deep topic model. Second, owing “skip connections” from all hidden layers to the output, one can reduce the number of processing steps between the bottom of the network and the top, mitigating the “vanishing gradient” problem (Graves et al. 2013).

1.3 A.3 Likelihood and Inference of VTCM-Transformer

Given an image $Img$, we can also represent the paragraph as ${\varvec{Y}}=\{y_1,...,y_I\}$, which is suitable for flat language model, such as Transformer-based model. Under the deep topic model (VTCM) and Transformer-based LM, the joint likelihood of the target ground truth paragraph ${\varvec{Y}}$ of $Img$ and its corresponding BoW count vector ${\varvec{d}}$ is defined as

(20)

Since we introduce a variational topic encoder to learn the multi-layer topic weight vectors $\varvec{\theta } ^{1:L}$ with the image features ${\overline{{\varvec{v}} }}$ as the input. Thus, a lower bound of the log of (20) can be constructed as

$$\begin{aligned} L_\text {all}&= {\mathbb {E}}_{q(\varvec{\theta } ^{1}|{\overline{{\varvec{v}} }})}\left[ \ln p\left( {\varvec{d}} \,|\,\, \varvec{\varPhi }^{1}\varvec{\theta } ^{1} \right) \right] \nonumber \\&- \sum _{l=1}^L {\mathbb {E}}_{q(\varvec{\theta } ^{l}|{\overline{{\varvec{v}} }})} \left[ \ln \frac{q\left( \varvec{\theta } ^{l}\,|\,{\overline{{\varvec{v}} }} \right) }{p \left( \varvec{\theta } ^{l} \,|\,\varvec{\varPhi }^{l + 1}\varvec{\theta } ^{l+ 1}\right) } \right] \nonumber \\&+\sum _{l=1}^L {\mathbb {E}}_{q(\varvec{\theta } ^{l}|{\overline{{\varvec{v}} }})} \left[ \sum _{i=1}^{I} \ln p\left( {y_{i}}\,|\,y_{<i},{\varvec{v}} _{1:M},\varvec{\theta } _j^{1:L}\right) \right] , \end{aligned}$$

(21)

which unites the first two terms primarily responsible for training the topic model component, and the last term for training the Transformer-based LM component. The parameters $\varvec{\Omega }_{\text {TM}}$ of the variational topic encoder and the parameters $\varvec{\Omega }_\text {Trans}$ of Transformer-based LM can be jointly updated by maximizing $L_\text {all}$. Besides, the global parameters $\varvec{\varPhi }^{1:L}$ of the topic decoder can be sampled with TLASGR-MCMC in Cong et al. (2017) and presented below. The training strategy of VTCM-Transformer is similar to that of VTCM-LSTM.

Table 4 Comparisons of our proposed VTCM-M-Transformer with M-Transformer on the test sets of IU X-RAY and MIMIC-CXR, where RG-L is ROUGE-L

1.4 A.4 Inference of Global Parameters $\varvec{\varPhi }^{1:L}$ of VTCM

For scale identifiability and ease of inference and interpretation, the Dirichlet prior is placed on each column of $\varvec{\varPhi }^{l} \in$ ${\mathbb {R}}_{+}^{K_{l-1} \times K_{l}}$, which means $0 \le \varPhi ^{l}_{k^{\prime }, k} \le 1$ and $\sum _{k^{\prime }=1}^{K_{l-1}} \varPhi ^{l}_{k^{\prime }, k}= 1$. To allow for scalable inference, we apply the topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC algorithm described in Cong et al. (2017); Zhang et al. (2018), which can be used to sample simplex-constrained global parameters in a mini-batch based manner. It improves its sampling efficiency via the use of the Fisher information matrix (FIM), with adaptive step-sizes for the topics at different layers. Here, we discuss how to update the global parameters $\{\varvec{\varPhi }^{l}\}_{l=1}^L$ of VTCM in detail and give a complete one in Algorithm 1.

Sample the auxiliary counts: This step is about the “upward” pass. For the given mini-batch $\{ Img _n, P_n, {\varvec{d}} _n \}_{n=1}^{N}$ in the training set, ${\varvec{d}} _n$ is the bag of words (BoW) count vector of paragraph $P_n$ for input image $Img _n$ and $\varvec{\theta } _n^{1:L}$ denotes the latent features of the nth image. By transforming standard uniform noises ${\varvec{\epsilon } _n^{l}}$, we can sample $\varvec{\theta } _n^{l}$ as

$$\begin{aligned} \varvec{\theta } _n^{l}&\small = {\varvec{\lambda } _n^{l}} \left( -\ln (1-{\varvec{\epsilon } _n^{l}})\right) ^ {1/{{\varvec{k}} _n^{l}}}. \end{aligned}$$

(22)

Working upward for $l = 1,...,L$, we can propagate the latent counts $x_{vn}^{l}$ of layer l upward to layer $l + 1$ as

(23)

(24)

(25)

where $x_{vn}^{1}=d_{vn}$, ${\varvec{d}} _{n}=\{d_{1n},..,d_{vn},..,d_{{V_c}n}\}$, $V_c$ is the size of vocabulary in VTCM, and $x_{kn}^{(l+1)}$ denotes the latent counts at layer $l+1$.

Fig. 8

Illustrations of reports from ground-truth, M-Transformer and VTCM-M-Transformer models for two X-ray chest images

Sample the hierarchical components $\{\varvec{\varPhi }^{l}\}_{l=1}^L$: For $\varvec{\phi } _k^{l}$, the kth column of the loading matrix $\varvec{\varPhi }^{l}$ of layer l, its sampling can be efficiently realized as

(26)

where $\varepsilon _q$ denotes the learning rate at the qth iteration, $\rho$ the ratio of the dataset size to the mini-batch size, $P_k^{l}$ is calculated using the estimated FIM, $\tilde{A}_{{k'}k\varvec{\cdot }}^{{l}} = \sum _{{n} = 1}^{N} {A_{{k'}kn}^{ {l}}}, {{{\tilde{{\varvec{A}} }}}_{:k\varvec{\cdot }}^{l }} = \{{\tilde{A}}_{{1}k\varvec{\cdot }}^{{l}},\cdots , {\tilde{A}}_{{K'}k\varvec{\cdot }}^{{l}} \}^{T}$ and ${{\tilde{A}}_{\varvec{\cdot }k\varvec{\cdot }}^{l}}= \sum _{{k'} = 1}^{K'} {\tilde{A}}_{{k'}k\varvec{\cdot }}^{{l}}$, ${A_{{k'}kn}^{ {l}}}$ comes from the augmented latent counts $A^{l}$ in (23), $\eta _{0}^{l}$ is the prior of ${\varvec{\phi } _k^{l}}$, and $[\cdot ]_\angle$ denotes a simplex constraint. More details about TLASGR-MCMC for our proposed model can be found in the Equations (18–19) of Cong et al. (2017).

1.5 A.5 Additional Experimental Results

To validate the generalizability of our proposed model, we also conducted experiments on the task of generating the radiology reports for the chest X-ray images, which is an important task to apply artificial intelligence to the medical domain. We consider the memory-driven Transformer (M-Transformer) designed for the radiology report generation task (Chen et al. 2020) as our baseline, which introduces a relational memory (RM) to record the information from previous generation processes and a memory-driven conditional layer normalization (MCLN) to incorporate the memory into Transformer. For a fair comparison, we adopt the same implementation for our model; see Chen et al. (2020) for more details. Our experiments are performed on two prevailing radiology report datasets. IUX-RAY (Demner-Fushman et al. 2016) is collected by the Indiana University and consists of 7,471 chest X-ray images and 3,955 reports; MIMIC-CXR (Johnson et al. 2019) includes 473,057 chest X-ray images and 206,563 reports from 63,478 patients. Following Chen et al. (2020) , we exclude the samples without reports. Note that we can flexibly select the language model for our plug-and-play system, since we pay more attention to assimilating the multi-layer semantic topic weight vectors into the paragraph generator. We here adopt the same Transformer encoder with the M-Transformer and introduce the three-layer semantic topics into its Transformer decoder, where we only add the concatenated topic proportion $\varvec{\theta } ^{1:L}$ to the embedding vector of the input token $y_{t-1}$, which are then embedded to calculate the keys ${\varvec{K}}$ and values ${\varvec{V}}$ of the decoder. As summarized in Table 4, our model (VTCM-M-Transformer) outperforms the M-Transformer on METEOR, ROUGE-L, BLEU-1, BLEU-2 and BLEU-3 and is competitive on the BLEU-4. It indicates that the multi-layer semantic topics with VTCM can enhance the radiology report generation despite without designing the complex language model on purpose, proving the generalizability and flexibility of our model. To qualitatively show the effectiveness of our proposed method, we show the reports of two randomly sampled X-ray chest images generated by different methods in Fig. 8, as well as the ground truth reports. Compared with M-Transformer, our VTCM-M-Transformer produces more detailed and coherent reports. For the first image, our VTCM-M-Transformer describes the “heart” and “mediastinal silhouette” in a natural way, while the M-Transformer ignores the “heart”. For the second image, the report generated by our model is closer to the ground-truth report, which summarizes the “lungs are hyperrexpanded”. The above observations show that using the hierarchical semantic topics can enhance the radiology report generation.