Deep Hashing with Hash-Consistent Large Margin Proxy Embeddings

Abstract

Image hash codes are produced by binarizing the embeddings of convolutional neural networks (CNN) trained for either classification or retrieval. While proxy embeddings achieve good performance on both tasks, they are non-trivial to binarize, due to a rotational ambiguity that encourages non-binary embeddings. The use of a fixed set of proxies (weights of the CNN classification layer) is proposed to eliminate this ambiguity, and a procedure to design proxy sets that are nearly optimal for both classification and hashing is introduced. The resulting hash-consistent large margin (HCLM) proxies are shown to encourage saturation of hashing units, thus guaranteeing a small binarization error, while producing highly discriminative hash-codes. A semantic extension (sHCLM), aimed to improve hashing performance in a transfer scenario, is also proposed. Extensive experiments show that sHCLM embeddings achieve significant improvements over state-of-the-art hashing procedures on several small and large datasets, both within and beyond the set of training classes.

Appendix: Relations Between Classification and Metric Learning

Although seemingly different, metric learning and classification are closely related. To see this, consider the Bayes rule

$$\begin{aligned} P_{Y|\mathbf {X}}(y|\mathbf {x})= & {} \frac{P_{\mathbf {X}|Y}(\mathbf {x}|y) P_Y(y)}{\sum _k P_{\mathbf {X}|Y}(\mathbf {x}|k) P_Y(k)}. \end{aligned}$$

(27)

It follows from (2) that

$$\begin{aligned} P_{\mathbf {X}|Y}(\mathbf {x}|y) P_Y(y) \propto _\mathbf {x}{e}^{\mathbf {w}^{T}_y \nu (\mathbf {x}) + b_y} \end{aligned}$$

(28)

where $\propto _\mathbf {x}$ denotes a proportional relation for each value of $\mathbf{x}$. This holds when

$$\begin{aligned} P_{\mathbf {X}|Y}(\mathbf {x}|y)= & {} q(\mathbf {x}) {e}^{\mathbf {w}^{T}_y \nu (\mathbf {x}) - \psi (\mathbf {w}_y)} \end{aligned}$$

(29)

$$\begin{aligned} P_Y(y)= & {} \frac{{e}^{b_y + \psi (\mathbf {w}_y)}}{\sum _k {e}^{b_k + \psi (\mathbf {w}_k)}}, \end{aligned}$$

(30)

where $q(\mathbf {x})$ is any non-negative function and $\psi (\mathbf {w}_y)$ a constant such that (29) integrates to one. In this case, $P_{\mathbf {X}|Y}(\mathbf {x}|y)$ is an exponential family distribution of canonical parameter $\mathbf {w}_y$, sufficient statistic $\nu (\mathbf {x})$ and cumulant function $\psi (\mathbf {w}_y)$ Barndorff-Nielsen (2014). Further assuming, for simplicity, that the classes are balanced, i.e., $P_Y(y) =\frac{1}{C} \forall y$, leads to

$$\begin{aligned} b_y = -\psi (\mathbf {w}_y) + \log K \end{aligned}$$

(31)

where K is a constant.

The cumulant $\psi (\mathbf {w}_y)$ has several important properties Barndorff-Nielsen (2014); Nelder and Wedderburn (1972); Banerjee et al. (2005). First, $\psi (\cdot )$ is a convex function of $\mathbf {w}_y$. Second, its first and second order derivatives are the mean $\nabla \psi (\mathbf {w}_y) = {{\mu }}^\nu _y$ and co-variance $\nabla ^2 \psi (\mathbf {w}_y) = {\varvec{\Sigma }}^\nu _y$ of $\nu (\mathbf {x})$ under class y. Third, $\psi (\cdot )$ has a conjugate function, convex on ${{\mu }}^\nu _y$, given by

$$\begin{aligned} \phi ({{\mu }}^\nu _y) = \mathbf {w}^{T}_y {{\mu }}^\nu _y - \psi (\mathbf {w}_y). \end{aligned}$$

(32)

It follows that the exponent of (29) can be re-written as

$$\begin{aligned} \mathbf {w}^{T}_y \nu (\mathbf {x}) - \psi (\mathbf {w}_y)= & {} \mathbf {w}^{T}_y {{\mu }}^\nu _y - \psi (\mathbf {w}_y) + \mathbf {w}^{T}_y (\nu (\mathbf {x}) - {{\mu }}^\nu _y) \nonumber \\= & {} \phi ({{\mu }}^\nu _y) + \mathbf {w}^{T}_y (\nu (\mathbf {x}) - {{\mu }}^\nu _y) \nonumber \\= & {} \phi ({{\mu }}^\nu _y) + \nabla \phi ({{\mu }}^\nu _y)^{T} (\nu (\mathbf {x}) - {{\mu }}^\nu _y) \nonumber \\= & {} -d_{\phi }(\nu (\mathbf {x}),{{\mu }}^\nu _y) + \phi (\nu (\mathbf {x})) \end{aligned}$$

(33)

where

$$\begin{aligned} d_{\phi }(\mathbf{a},\mathbf{b}) = \phi (\mathbf{a}) - \phi (\mathbf{b}) - \langle \nabla \phi (\mathbf{b}), \mathbf{a} - \mathbf{b} \rangle \end{aligned}$$

(34)

is the Bregman divergence between $\mathbf{a}$ and $\mathbf{b}$ associated with $\phi$. Thus, (29) can be written as

$$\begin{aligned} P_{\mathbf {X}|Y}(\mathbf {x}|y) = u(\mathbf {x}) {e}^{-d_{\phi }(\nu (\mathbf {x}),{{\mu }}^\nu _y)} \end{aligned}$$

(35)

where $u(\mathbf {x}) = q(\mathbf {x}) {e}^{\phi (\nu (\mathbf {x}))}$ and using (31), (30) and (27),

$$\begin{aligned} P_{Y|\mathbf {X}}(y|\mathbf {x})= & {} \frac{{e}^{-d_{\phi }(\nu (\mathbf {x}),{{\mu }}^\nu _y)}}{\sum _{k} {e}^{-d_{\phi }(\nu (\mathbf {x}),{{\mu }}^\nu _k)}}. \end{aligned}$$

(36)

Hence, learning the embedding $\nu (\mathbf {x})$ with the softmax classifier of (2) endows $\mathcal V$ with the Bregman divergence $d_{\phi }(\nu (\mathbf {x}),{{\mu }}^\nu _y)$. From (32), it follows that

$$\begin{aligned} \nabla \psi (\mathbf {w}_y) = {{\mu }}_y^\nu \quad \quad \quad \nabla \phi ({{\mu }}_y^\nu ) = \mathbf {w}_y. \end{aligned}$$

(37)

Hence,

$$\begin{aligned} {{\mu }}_y^\nu = \mathbf {w}_y \end{aligned}$$

(38)

if and only if

$$\begin{aligned} \nabla \psi (\mathbf {w}_y) = \mathbf {w}_y \quad \quad \quad \nabla \phi ({{\mu }}_y^\nu ) = {{\mu }}_y^\nu , \end{aligned}$$

(39)

which holds when

$$\begin{aligned} \psi (\mathbf{a}) = \phi (\mathbf{a}) = \frac{1}{2}||\mathbf{a}||^2. \end{aligned}$$

(40)

It can be shown that the corresponding exponential family model is the Gaussian of identity covariance and the corresponding Bregman divergence the squared Euclidean distance. Hence, ${{\mu }}_y^g = \mathbf {w}_y$ if only if $d_\phi$ is the $L_2$ distance. In this case, (36) reduces to

$$\begin{aligned} P_{Y|\mathbf {X}}(y|\mathbf {x})= & {} \frac{{e}^{-d(\nu (\mathbf {x}),\mathbf {w}_y)}}{\sum _{k} {e}^{-d(\nu (\mathbf {x}),\mathbf {w}_k)}}. \end{aligned}$$

(41)