Detector Performance Prediction Using Set Annotations

Abstract

Content-based videos search engines often use the output of concept detectors to answer queries. The improvement of detectors requires computational power and human labor. It is therefore important to predict detector performance economically and improve detectors adaptively. Detector performance prediction, however, has not received much research attention so far. In this paper, we propose a prediction approach that uses human annotators. The annotators estimate the number of images in a grid in which a concept is present, a task that can be performed efficiently. Using these estimations, we define a model for the posterior probability of a concept being present given its confidence score. We then use the model to predict the average precision of a detector. We evaluate our approach using a TRECVid collection of Internet archive videos, comparing it to an approach that labels individual images. Our approach requires fewer resources while achieving good prediction quality.

A Appendix - Optimization of Maximum Likelihood

In this section, we present a procedure for the maximization problem of finding the maximum likelihood weights for the logistic regression model defined in Sect. 3.2. The maximization problem was formulated as follows:

$$ {\mathbf {w}}_{ml} = \mathop {\mathrm{argmax}}_{{\mathbf {w}}}{\; p(H=h | {\mathbf {w}})} $$

where $h$ is the estimate given by the annotator. We assume that $x({\mathbf {w}})$ is Gaussian distributed around a mean $\mu _h$ with variance $v_h$, where both depend on the annotator’s estimation $h$ and possibly the annotator himself. In this paper, we choose a simple method to come from $h$ to $\mu _h$ by choosing $\mu _h=h$ for $1\le h < N$ and $\mu _0=1$ and $\mu _N=N-1$ modeling the case where the annotator oversees at least one example when annotating extreme values. We ignore $v_h$ because it does not play a role in the optimization. Therefore, for the likelihood a weight vector ${\mathbf {w}}$ given an annotation $h$, we have:

$$ p(H|{\mathbf {w}}) = \mathcal {N}\!\left( x({\mathbf {w}}), \mu _h, v_h\right) $$

where $\mathcal {N}\!$ is the Gaussian density function. Taking the log of $\mathcal {N}\!$ yields:

$$ log(\mathcal {N}\!\left( x({\mathbf {w}}), \mu _h, v_h\right) ) = log\left( \frac{1}{\sqrt{2\pi \; v_h}}\right) + \frac{- (x({\mathbf {w}})-\mu _h)^2}{2\; v_h}. $$

By expanding $(\cdot )^2$, leaving out constant terms and factors, and multiplying by ${-}1$ to convert the maximization to a minimization problem, we get:

$$ x({\mathbf {w}})^2 - 2 \mu _h x({\mathbf {w}}) + \mu _h^2 $$

By expanding the definition of the expected number of positive examples in (2), leaving out the constant $\mu _h^2$ and combining factors we get:

$$ \left( \sum _i^N{\sigma _i({\mathbf {w}})}\right) ^2 - 2 \mu _h \sum _i^N{\sigma _i({\mathbf {w}})} $$

And by expanding the square of the expectation $x({\mathbf {w}})^2$:

$$\begin{aligned} y({\mathbf {w}}) = \underbrace{\left( \sum _i^N{\sum _j^N{\sigma _i({\mathbf {w}}) \sigma _j({\mathbf {w}})}}\right) }_{u_{i,j}({\mathbf {w}})} - 2 \mu _h \sum _i^N{\sigma _i({\mathbf {w}})} \end{aligned}$$

(5)

To optimize this function we use the gradient decent method with the update rule:

$$\begin{aligned} {\mathbf {w}}^{t+1} = {\mathbf {w}}^{t} - \lambda \triangledown y({\mathbf {w}}^{t}) \end{aligned}$$

(6)

where $t$ refers to the $t$th iteration, $\lambda$ is the “update speed” of the method (in this paper we chose $\lambda = 0.03$) and $\triangledown y({\mathbf {w}}^{t})$ is gradient of the method with respect to ${\mathbf {w}}$. The gradient $\triangledown y$ is the vector of partial derivatives:

$$\begin{aligned} \triangledown y = \left[ \frac{\partial y}{\partial w_1}, \frac{\partial y}{\partial w_2}\right] \end{aligned}$$

(7)

To calculate the two partial derivations of $\triangledown y$, we start by calculating the gradient $\triangledown \sigma$ which used in the second expression of (5). As an intermediate step, we give the derivation of a general sigmoid function $\sigma (s)$:

$$\begin{aligned} \sigma '(s) = \sigma (s) (1-\sigma (s)) \end{aligned}$$

(8)

Given this relationship we get the partial derivatives for $\triangledown \sigma$:

$$\begin{aligned} \frac{\partial \sigma _i}{\partial w_1} = \sigma _i({\mathbf {w}}) (1-\sigma _i({\mathbf {w}})) \qquad \frac{\partial \sigma _i}{\partial w_2} = \sigma _i({\mathbf {w}}) (1-\sigma _i({\mathbf {w}})) s_i \end{aligned}$$

(9)

Furthermore, for the derivation of the products of two sigmoid functions $u_{ij}({\mathbf {w}}) = \sigma _i({\mathbf {w}}) \sigma _j({\mathbf {w}})$ in (5), we use the product rule and the results from (9). For $w_1$ we have:

$$\begin{aligned} \frac{\partial u_{ij}}{\partial w_1}&= \sigma _i({\mathbf {w}}) (1-\sigma _i({\mathbf {w}})) \sigma _j({\mathbf {w}}) \\&+ \sigma _i({\mathbf {w}}) \sigma _j({\mathbf {w}}) (1-\sigma _j({\mathbf {w}})) \end{aligned}$$

and for $w_2$:

$$\begin{aligned} \frac{\partial u_{ij}}{\partial w_2}&= \left[ \sigma _i({\mathbf {w}}) (1-\sigma _i({\mathbf {w}})) s_i\right] \sigma _j({\mathbf {w}}) \\&+ \sigma _i({\mathbf {w}}) \left[ \sigma _j({\mathbf {w}}) (1-\sigma _j({\mathbf {w}})) s_j\right] \end{aligned}$$

Therefore, the partial derivatives for the gradient $\triangledown y$ in (7) are:

$$ \frac{\partial y}{\partial w_1} = \left( \sum _i^N{\sum _j^N{\frac{\partial u_{ij}}{\partial w_1}}}\right) - 2 \mu _h \sum _i{\frac{\partial \sigma _i}{\partial w_1}} $$

and

$$ \frac{\partial y}{\partial w_2} = \left( \sum _i^N{\sum _j^N{\frac{\partial u_{ij}}{\partial w_2}}}\right) - 2 \mu _h \sum _i^N{\frac{\partial \sigma _i}{\partial w_2}} $$

Note that although quadratic in the number of images, the gradient can be calculated efficiently by memorizing the values for $\sigma _i({\mathbf {w}}^t)$ for $1\le i \le N$.