Inductive hierarchical nonnegative graph embedding for “verb–object” image classification

Abstract

Most existing image classification algorithms mainly focus on dealing with images with only “object” concepts. However, in real-world cases, a great variety of images contain “verb–object” concepts, rather than only “object” ones. The hierarchical structure embedded in these “verb–object” concepts can help to enhance classification. However, traditional feature representation methods cannot utilize it. To tackle this problem, we present in this paper a novel approach, called inductive hierarchical nonnegative graph embedding. By assuming that those “verb–object” concept images which share the same “object” part but different “verb” part have a specific hierarchical structure, we integrate this hierarchical structure into the nonnegative graph embedding technique, together with the definition of inductive matrix, to (1) conduct effective feature extraction from hierarchical structure, (2) easily transfer each new testing sample into its low-dimensional nonnegative representation, and (3) perform image classification of “verb–object” concept images. Extensive experiments compared with the state-of-the-art algorithms on nonnegative data factorization demonstrate the classification power of proposed approach on “verb–object” concept images classification.

Appendix

Here we present the convergence proof of update rule for both matrix $W$ and matrix $C$.

1.1 Preliminaries

First of all, we introduce the concept of auxiliary function and the lemma which will be used for algorithmic derivation.

Definition 1

Function $G(A,A')$ is an auxiliary function for function $F(A)$ if the following conditions are satisfied

$$\begin{aligned} G(A,A') \ge F(A), \qquad G(A,A) = F(A) \end{aligned}$$

(44)

From this definition, we have the following lemma with proof omitted [12].

Lemma 1

If $G$ is an auxiliary function, then $F$ is non-increasing under the update

$$\begin{aligned} A^{t+1} = \mathrm{arg\, min}_A G(A,A') \end{aligned}$$

(45)

where $t$ denotes the $t$th iteration.

1.2 Convergence proof of update rule for W

Let $F_{ij}$ as the part of $F(W)$ relevant to $W_{ij}$, we have

$$\begin{aligned} F'_{ij}(W)&= [W(2Q^1 +2Q^2 +2Q^3) +2\lambda W C X X^T C^T \nonumber \\&- 2\lambda X X^T C^T]_{ij}\end{aligned}$$

(46)

$$\begin{aligned} F''_{ij}(W)&= [2(Q^1 +Q^2 +Q^3) +2\lambda C X X^T C^T]_{jj} \end{aligned}$$

(47)

The auxiliary function of $F_{ij}$ is then designed as

$$\begin{aligned} G(W_{ij}, W^t_{ij})&= F_{ij}(W^t_{ij}) + F'_{ij}(W_{ij})(W_{ij} - W^t_{ij}) \nonumber \\&+ \frac{[W^t (Q^1_+ \!+\!Q^2_+ \!+\!Q^3_+) \!+\! \lambda \!W^t \!C X X^T C^T]_{ij}}{\!W^t_{ij}} \nonumber \\&\times (W_{ij} - W^t_{ij})^2 \end{aligned}$$

(48)

Lemma 2

Equation (48) is an auxiliary function for $F_{ij}$, which is the part of $F(W)$ relevant to $W_{ij}$.

Proof

Obviously, $G(W_{ij}, W_{ij}) = F_{ij}(W_{ij})$. We only need to prove that $G(W_{ij}, W^t_{ij}) \ge F_{ij}(W_{ij})$.

First, we have the Taylor series expansion of $F_{ij}$

$$\begin{aligned} F_{ij}(W_{ij})&=F_{ij}(W^t_{ij}) + F'_{ij}(W^t_{ij})(W_{ij} - W^t_{ij}) \nonumber \\&\quad + \frac{1}{2} F''_{ij}(W^t_{ij})(W_{ij} - W^t_{ij})^2 \end{aligned}$$

(49)

Then, it is easy to verify that

$$\begin{aligned}&[\lambda W^t C X X^T C^T]_{ij} \ge W^t_{ij} [\lambda C X X^T C^T]_{jj}\end{aligned}$$

(50)

$$\begin{aligned}&[W^t (Q^1_+ +Q^2_+ +Q^3_+)]_{ij} \ge W^t_{ij} [(Q^1_+ +Q^2_+ +Q^3_+)]_{jj}\nonumber \\ \end{aligned}$$

(51)

Thus we have

$$\begin{aligned}&\frac{[W^t (Q^1_+ +Q^2_+ +Q^3_+) + \lambda W^t C X X^T C^T]_{ij}}{W^t_{ij}} \nonumber \\&\quad \ge [(Q^1 +Q^2 +Q^3) +\lambda C X X^T C^T]_{jj} \end{aligned}$$

(52)

Then, $G(W_{ij}, W^t_{ij}) \ge F_{ij}(W_{ij})$ holds.

Lemma 3

Equation (34) could be obtained by minimizing the auxiliary function $G(W_{ij}, W^t_{ij})$.

Proof

Let $\partial G(W_{ij}, W^t_{ij}) / \partial W_{ij} = 0$, we have

$$\begin{aligned}&F'_{ij}(W^t_{ij}) \!+\! 2 \frac{[W^t (Q^1_+ \!+\!Q^2_+ +Q^3_+) \!+\! \lambda W^t C X X^T C^T]_{ij}}{W^t_{ij}}\times \nonumber \\&\quad (W_{ij} - W^t_{ij}) = 0 \end{aligned}$$

(53)

Finally we can obtain the update rule for $W$

$$\begin{aligned} W^{t+1}_{ij} \leftarrow W^t_{ij} \frac{[\lambda X X^T C^T + W^t(Q^1_- + Q^2_- + Q^3_-)]_{ij}}{[\lambda W^t C X X^T C^T + W^t(Q^1_+ +Q^2_+ + Q^3_+)]_{ij}}\nonumber \\ \end{aligned}$$

(54)

and the lemma is proved.

1.3 Convergence proof of update rule for C

Let $F_{ij}$ as the part of $F(C)$ relevant to $C_{ij}$, we have

$$\begin{aligned} F'_{ij}(C)&= [2{R^1}^T R^1 C X L^u X^T + 2{R^2}^T R^2 C X \tilde{L} X^T \nonumber \\&\quad + 2{R^3}^T R^3 C X \tilde{L}^u X^T - 2 \lambda W^T X X^T \nonumber \\&\quad + 2 \lambda W^T W C X X^T]_{ij}\end{aligned}$$

(55)

$$\begin{aligned} F''_{ij}(C)&= 2[{R^1}^T R^1]_{ii} [X L^u X^T]_{jj} + 2[{R^2}^T R^2]_{ii} [X \tilde{L} X^T]_{jj} \nonumber \\&\quad + 2[{R^3}^T R^3]_{ii} [X \tilde{L}^u X^T]_{jj} + 2 \lambda [W^T W]_{ii} [X X^T]_{jj} \nonumber \\ \end{aligned}$$

(56)

The auxiliary function of $F_{ij}$ is then designed as

$$\begin{aligned}&G(C_{ij}, C^t_{ij}) \nonumber \\&\quad = F_{ij}(C^t_{ij}) + F'_{ij}(C_{ij})(C_{ij} - C^t_{ij}) \nonumber \\&\quad \quad + [{R^1}^T R^1 C^t X D^u X^T + {R^2}^T R^2 C^t X \tilde{D} X^T \nonumber \\&\quad \quad + {R^3}^T R^3 C^t X \tilde{D}^u X^T + \lambda W^T W C^t X X^T]_{ij} / C^t_{ij} \nonumber \\&\quad \quad \times (C_{ij} - C^t_{ij})^2 \end{aligned}$$

(57)

Lemma 4

Equation (57) is an auxiliary function for $F_{ij}$, which is the part of $F(C)$ relevant to $C_{ij}$.

Proof

Obviously, $G(C_{ij}, C_{ij}) = F_{ij}(C_{ij})$. We only need to prove that $G(C_{ij}, C^t_{ij}) \ge F_{ij}(C_{ij})$.

First, we have the Taylor series expansion of $F_{ij}$

$$\begin{aligned} F_{ij}(C_{ij})&= F_{ij}(C^t_{ij}) + F'_{ij}(C^t_{ij})(C_{ij} - C^t_{ij}) \nonumber \\&\quad + \frac{1}{2} F''_{ij}(C^t_{ij})(C_{ij} - C^t_{ij})^2 \end{aligned}$$

(58)

Then, it is easy to verify that

$$\begin{aligned}&[W^T W C X X^T]_{ij} \ge C^t_{ij} [W^T W]_{ii} [X X^T]_{jj}\end{aligned}$$

(59)

$$\begin{aligned}&[{R^1}^T R^1 C^t X D^u X^T]_{ij} \ge [{R^1}^T R^1]_{ii} C^t_{ij} [X L^u X^T]_{jj}\end{aligned}$$

(60)

$$\begin{aligned}&[{R^2}^T R^2 C^t X \tilde{D} X^T]_{ij} \ge [{R^2}^T R^2]_{ii} C^t_{ij} [X \tilde{L} X^T]_{jj}\end{aligned}$$

(61)

$$\begin{aligned}&[{R^3}^T R^3 C X \tilde{D}^u X^T]_{ij} \ge [{R^3}^T R^3]_{ii} C^t_{ij} [X \tilde{L}^u X^T]_{jj} \end{aligned}$$

(62)

Thus we have $G(C_{ij}, C^t_{ij}) \ge F_{ij}(C_{ij})$.

Lemma 5

Equation (43) could be obtained by minimizing the auxiliary function $G(C_{ij}, C^t_{ij})$.

Proof

Let $\partial G(C_{ij}, C^t_{ij}) \ / \ \partial C_{ij} = 0$, we have

$$\begin{aligned}&F'_{ij}(C_{ij}) + [{R^1}^T R^1 C^t X D^u X^T + {R^2}^T R^2 C^t X \tilde{D} X^T \nonumber \\&\quad + {R^3}^T R^3 C^t X \tilde{D}^u X^T + \lambda W^T W C^t X X^T]_{ij} / C^t_{ij} \nonumber \\&\quad \cdot (C_{ij} - C^t_{ij}) = 0 \end{aligned}$$

(63)

Finally we can obtain the update rule for $C$

$$\begin{aligned}&C^{t+1}_{ij} \leftarrow C^t_{ij} \cdot [\lambda W^T X X^T + {R^1}^T R^1 C^t X S^u X^T \nonumber \\&\quad + {R^2}^T R^2 C^t X \tilde{S} X^T + {R^3}^T R^3 C^t X \tilde{S}^u X^T]_{ij} \nonumber \\&\quad / \ [\lambda W^T W C^t X X^T + {R^1}^T R^1 C^t X D^u X^T \nonumber \\&\quad + {R^2}^T R^2 C^t X \tilde{D} X^T + {R^3}^T R^3 C^t X \tilde{D}^u X^T]_{ij} \end{aligned}$$

(64)

and the lemma is proved. $\square$