Image Classification with the Fisher Vector: Theory and Practice

Abstract

A standard approach to describe an image for classification and retrieval purposes is to extract a set of local patch descriptors, encode them into a high dimensional vector and pool them into an image-level signature. The most common patch encoding strategy consists in quantizing the local descriptors into a finite set of prototypical elements. This leads to the popular Bag-of-Visual words representation. In this work, we propose to use the Fisher Kernel framework as an alternative patch encoding strategy: we describe patches by their deviation from an “universal” generative Gaussian mixture model. This representation, which we call Fisher vector has many advantages: it is efficient to compute, it leads to excellent results even with efficient linear classifiers, and it can be compressed with a minimal loss of accuracy using product quantization. We report experimental results on five standard datasets—PASCAL VOC 2007, Caltech 256, SUN 397, ILSVRC 2010 and ImageNet10K—with up to 9M images and 10K classes, showing that the FV framework is a state-of-the-art patch encoding technique.

Appendices

Appendix 1 : An Approximation of the Fisher Information Matrix

In this appendix we show that, under the assumption that the posterior distribution $\gamma _x(k) = w_k u_k(x) / u_\lambda (x)$ is sharply peaked, the normalization with the FIM takes a diagonal form. Throughout this appendix we assume the data $x$ to be one dimensional. The extension to the multidimensional data case is immediate for the mixtures of Gaussians with diagonal covariance matrices that we are interested in.

Under some mild regularity conditions on $u_\lambda (x)$, the entries of the FIM can be expressed as:

$$\begin{aligned} {\left[ {F_\lambda }\right] }_{i,j} = \mathbb{E {\left[ { - \frac{\partial ^2 \log u_\lambda (x)}{\partial \lambda _i \partial \lambda _j}}\right] }}. \end{aligned}$$

(57)

First, let us consider the partial derivatives of the posteriors w.r.t. the mean and variance parameters. If we use $\theta _s$ to denote one such parameter associated with $u_s(x)$, i.e. mixture component number $s$, then:

$$\begin{aligned}&\frac{\partial \gamma _x(k)}{\partial \theta _s} = \gamma _x(k) \frac{\partial \log \gamma _x(k)}{\partial \theta _s} \end{aligned}$$

(58)

$$\begin{aligned}&\quad = \gamma _x(k) \frac{\partial }{\partial \theta _s}\big [ \log w_k + \log u_k(x) - \log u_\lambda (x)\big ]\end{aligned}$$

(59)

$$\begin{aligned}&\quad = \gamma _x(k) \left[ [\![k=s]\!] \frac{\partial \log u_s(x)}{\partial \theta _s} - \frac{\partial \log u_\lambda (x)}{\partial \theta _s} \right] \end{aligned}$$

(60)

$$\begin{aligned}&\quad = \gamma _x(k) \left[ [\![k=s]\!] \frac{\partial \log u_s(x)}{\partial \theta _s} - \gamma _x(s) \frac{\partial \log u_s(x)}{\partial \theta _s} \right] \end{aligned}$$

(61)

$$\begin{aligned}&\quad = \gamma _x(k) \Big ( [\![k=s]\!] - \gamma _x(s)\Big ) \frac{\partial \log u_s(x)}{\partial \theta _s} \approx 0, \end{aligned}$$

(62)

where $[\![\cdot ]\!]$ is the Iverson bracket notation which equals one if the argument is true, and zero otherwise. It is easy to verify that the assumption that the posterior is sharply peaked implies that the partial derivative is approximately zero, since the assumption implies that (i) $\gamma _x(k)\gamma _x(s)\approx 0$ if $k\ne s$ and (ii) $\gamma _x(k) \approx \gamma _x(k)\gamma _x(s)$ if $k=s$.

From this result and Eqs. (12), (13), and (14), it is then easy to see that second order derivatives are zero if (i) they involve mean or variance parameters corresponding to different mixture components ($k\ne s$), or if (ii) they involve a mixing weight parameter and a mean or variance parameter (possibly from the same component).

To see that the cross terms for mean and variance of the same mixture component are zero, we again rely on the observation that $\partial \gamma _x(k) / \partial \theta _s\approx 0$ to obtain:

$$\begin{aligned}&\frac{\partial ^2 \log u_\lambda (x)}{\partial \sigma _k \partial \mu _k} \approx \gamma _x(k)(x-\mu _k) \frac{\partial \sigma _k^{-2}}{\partial \sigma _k}\nonumber \\&\quad = -2\sigma _k^{-3}\gamma _x(k) (x-\mu _k) \end{aligned}$$

(63)

Then by integration we obtain:

$$\begin{aligned} {\left[ {F_\lambda }\right] }_{\sigma _k,\mu _k}&= -\int \limits _x u_\lambda (x) \frac{\partial ^2 \log u_\lambda (x)}{\partial \sigma _k \partial \mu _k} {\mathrm{d }}x\end{aligned}$$

(64)

$$\begin{aligned}&\approx 2\sigma _k^{-3}\int \limits _x u_\lambda (x)\gamma _x(k) (x-\mu _k) {\mathrm{d }}x\end{aligned}$$

(65)

$$\begin{aligned}&= 2\sigma _k^{-3}w_k\int \limits _x u_k(x) (x-\mu _k) {\mathrm{d }}x = 0 \end{aligned}$$

(66)

We now compute the second order derivatives w.r.t. the means:

$$\begin{aligned} \frac{\partial ^2 \log u_\lambda (x)}{(\partial \mu _k)^2} \approx \sigma _k^{-2}\gamma _x(k)\frac{\partial (x-\mu _k)}{\partial \mu _k} = -\sigma _k^{-2}\gamma _x(k) \end{aligned}$$

(67)

Integration then gives:

$$\begin{aligned} {\left[ {F_\lambda }\right] }_{\mu _k,\mu _k}&= -\int \limits _x u_\lambda (x) \frac{\partial ^2 \log u_\lambda (x)}{(\partial \mu _k)^2} {\mathrm{d }}x \approx \sigma _k^{-2}\int \limits _x u_\lambda (x) \gamma _x(k) {\mathrm{d }}x\end{aligned}$$

(68)

$$\begin{aligned}&= \sigma _k^{-2}w_k\int \limits _x u_k(x) {\mathrm{d }} x = \sigma _k^{-2}w_k, \end{aligned}$$

(69)

and the corresponding entry in $L_\lambda$ equals $\sigma _k / \sqrt{w_k}$. This leads to the normalized gradients as presented in (17).

Similarly, for the variance parameters we obtain:

$$\begin{aligned} \frac{\partial ^2 \log u_\lambda (x)}{(\partial \sigma _k)^2} \approx \sigma _k^{-2}\gamma _x(k)\Big (1 - 3(x-\mu _k)^2 / \sigma _k^2\Big ) \end{aligned}$$

(70)

Integration then gives:

$$\begin{aligned}&{\left[ {F_\lambda }\right] }_{\sigma _k,\sigma _k}\nonumber \\&\quad \approx \sigma _k^{-2}\int \limits _x u_\lambda (x) \gamma _x(k) \Big (3(x-\mu _k)^2 / \sigma _k^2 -1\Big ) {\mathrm{d }}x\end{aligned}$$

(71)

$$\begin{aligned}&\quad = \sigma _k^{-2}w_k\int \limits _x u_k(x) \Big (3(x-\mu _k)^2 / \sigma _k^2 -1\Big ) {\mathrm{d }}x = 2\sigma _k^{-2}w_k,\nonumber \\ \end{aligned}$$

(72)

which leads to a corresponding entry in $L_\lambda$ of $\sigma _k / \sqrt{2w_k}$. This leads to the normalized gradients as presented in (18).

Finally, the computation of the normalization coefficients for the mixing weights is somewhat more involved. To compute the second order derivatives involving mixing weight parameters only, we will make use of the partial derivative of the posterior probabilities $\gamma _x(k)$:

$$\begin{aligned} \frac{\partial \gamma _x(k)}{\partial \alpha _s} = \gamma _x(k)\Big ([\![k=s]\!] - \gamma _x(s)\Big ) \approx 0, \end{aligned}$$

(73)

where the approximation follows from the same observations as used in (62). Using this approximation, the second order derivatives w.r.t. mixing weights are:

$$\begin{aligned} \frac{\partial ^2 \log u_\lambda (x)}{\partial \alpha _s \partial \alpha _k}&= \frac{\partial \gamma _x(k)}{\partial \alpha _s} - \frac{\partial w_k}{\partial \alpha _s}\approx - \frac{\partial w_k}{\partial \alpha _s}\nonumber \\&= w_k\big ( [\![k=s]\!] - w_s\big ) \end{aligned}$$

(74)

Since this result is independent of $x$, the corresponding block of the FIM is simply obtained by collecting the negative second order gradients in matrix form:

$$\begin{aligned} {\left[ {F_\lambda }\right] }_{\alpha ,\alpha } = { w w^{\prime }} - {\mathrm{d iag}}(w), \end{aligned}$$

(75)

where we used $w$ and $\alpha$ to denote the vector of all mixing weights, and mixing weight parameters respectively.

Since the mixing weights sum to one, it is easy to show that this matrix is non-invertible by verifying that the constant vector is an eigenvector of this matrix with associated eigenvalue zero. In fact, since there are only $K-1$ degrees of freedom in the mixing weights, we can fix $\alpha _K=0$ without loss of generality, and work with a reduced set of $K-1$ mixing weight parameters. Now, let us make the following definitions: let $\tilde{\alpha } = (\alpha _1, \ldots , \alpha _{K-1})^T$ denote the vector of the first $K-1$ mixing weight parameters, let $G_{\tilde{\alpha }}^x$ denote the gradient vector with respect to these, and $F_{\tilde{\alpha }}$ the corresponding matrix of second order derivatives. Using this definition $F_{\tilde{\alpha }}$ is invertible, and using Woodburry’s matrix inversion lemma, we can show that

$$\begin{aligned} G_{\tilde{\alpha }}^x F_{\tilde{\alpha }}^{-1} G_{\tilde{\alpha }}^y = \sum \limits _{k=1}^K (\gamma _x(k)-w_k)(\gamma _y(k)-w_k) / w_k. \end{aligned}$$

(76)

The last form shows that the inner product, normalized by the inverse of the non-diagonal $K-1$ dimensional square matrix $F_{\tilde{\alpha }}$, can in fact be obtained as a simple inner product between the normalized version of the $K$ dimensional gradient vectors as defined in (12), i.e. with entries $\big (\gamma _x(k)-w_k\big )/\sqrt{w_k}$. This leads to the normalized gradients as presented in (16).

Note also that, if we consider the complete data likelihood:

$$\begin{aligned} p(x,z|\lambda ) = u_\lambda (x) p(z|x,\lambda ) \end{aligned}$$

(77)

the Fisher information decomposes as:

$$\begin{aligned} F_{c} = F_\lambda + F_{r}, \end{aligned}$$

(78)

where $F_{c}, F_\lambda$ and $F_{r}$ denote the FIM of the complete, marginal (observed) and conditional terms. Using the $1\mathrm -of- K$ formulation for $z$, it can be shown that $F_c$ has a diagonal form with entries given by (68), (71) and (76), respectively. Therefore, $F_r$ can be seen as the amount of “information” lost by not knowing the true mixture component generating each of the $x$’s Titterington et al. (1985). By requiring the distribution of the $\gamma _x(k)$ to be “sharply peaked” we are making the approximation $z_k\approx \gamma _x(k)$.

From this derivation we conclude that the assumption of sharply peaked posteriors leads to a diagonal approximation of the FIM, which can therefore be taken into account by a coordinate-wise normalization of the gradient vectors.

Appendix 2 : Good Practices for Gaussian Mixture Modeling

We now provide some good practices for GMM. For a public GMM implementation and for more details on how to train and test GMMs, we refer the reader to the excellent HMM ToolKit (HTK) Young et al. (2002)¹⁰.

Computation in the Log Domain We first describe how to compute in practice the likelihood (8) and the soft-assignment (15). Since the low-level descriptors are quite high-dimensional (typically $D=64$ in our experiments), the likelihood values $u_k(x)$ for each Gaussian can be extremely small (and even fall below machine precision if using floating point values) because of the $\exp$ of Eq. (9). Hence, for a stable implementation, it is of utmost importance to perform all computations in the log domain. In practice, for descriptor $x$, one never computes $u_k(x)$ but

$$\begin{aligned}&\log u_k(x) \nonumber \\&\quad = -\frac{1}{2} \sum \limits _{d=1}^D \left[ \log (2\pi ) + \log (\sigma _{kd}^2) + \frac{(x_d - \mu _{kd})^2)}{\sigma _{kd}^2} \right] \end{aligned}$$

(79)

where the subscript $d$ denotes the $d$-th dimension of a vector. To compute the log-likelihood $\log u_\lambda (x) = \log \sum \nolimits _{k=1}^K w_k u_k(x)$, one does so incrementally by writing $\log u_\lambda (x) = \log \big ( w_1 u_1(x) + \sum \nolimits _{k=2}^K w_k u_k(x) \big )$ and by using the fact that $\log (a+b) = \log (a) + \log \left( 1+ \exp (\log (b) - \log (a)\right)$ to remain in the log domain.

Similarly, to compute the posterior probability (15), one writes $\gamma _k = \exp \left[ \log (w_k u_k(x)) - \log (u_\lambda (x)) \right]$ to operate in the log domain.

Variance Flooring Because the variance $\sigma _k^2$ appears in a $\log$ and as a denominator in Eq. (79), too small values of the variance can lead to instabilities in the Gaussian computations. In our case, this is even more likely to happen since we extract patches densely and we do not discard uniform patches. In our experience, such patches tend to cluster in a Gaussian mixture component with a tiny variance. To avoid this issue, we use variance flooring: we compute the global covariance matrix over all our training set and we enforce the variance of each Gaussian to be no smaller than a constant $\alpha$ times the global variance. Such an operation is referred to as variance flooring. HTK suggest a value $\alpha = 0.01$.

Posterior Thresholding To reduce the cost of training GMMs as well as the cost of computing FVs, we assume that all the posteriors $\gamma (k)$ which are below a given threshold $\theta$ are equal to exactly zero. In practice, we use a quite conservative threshold $\theta =10^{-4}$ and for a GMM with 256 Gaussians, 5–10 Gaussians maximum exceed this threshold. After discarding some of the $\gamma (k)$ values, we renormalize the $\gamma ^{\prime }$s to ensure that we still have $\sum \nolimits _{k=1}^K \gamma (k) = 1$.

Note that this operation does not only reduce the computational cost, it also sparsifies the FV (see Sect. 4.2). Without such a posterior thresholding, the FV (or the soft-BOV) would be completely dense.

Incremental Training It is well known that the ML estimation of a GMM is a non-convex optimization problem for more than one Gaussian. Hence, different initializations might lead to different solutions. While in our experience, we have never observed a drastic influence of the initialization on the end result, we strongly advise the use of an iterative process as suggested for instance in Young et al. (2002). This iterative procedure consists in starting with a single Gaussian (for which a closed-form formula exists), splitting all Gaussians by slightly perturbing the mean and then re-estimating the GMM parameters with EM. This iterative splitting-training strategy enables cross-validating and monitoring the influence of the number of Gaussians $K$ in a consistent way.