Online adaptive feature weighting for spatiogram-bank tracking

Abstract

In this paper, we introduce a new adaptive feature weighting framework for multi-modal tracking. Our proposed tracker can compactly and efficiently handle multiple sources of tracking data, such as colour, brightness, gradient orientation and thermal infrared, and adaptively weight the sources based on their reliability for tracking. The adaptive weight selection mechanism is inspired by the state-of-the-art Collins tracker, but instead of treating the tracked object as a bag of features, it takes advantage of the spatial information using a global object model. Additionally, our tracker incorporates scale into the weight selection process and is shown to outperform the Collins tracker in an extensive video evaluation.

Appendix: Derivation of gradient of OD ratio

Before computing the object-to-distractor ratio (OD ratio), $D$ in Eq. (6), all scores are replaced by their $log$ values, which makes the derivation more straightforward. Given a set of $K$ log-similarity surfaces, $s_k(p) = \log \rho _k(p), k \in \{1,2,\ldots ,K\},$ with $p$ being the candidate position, and $K$ representing the number of features, the fused surface is given by

$$\begin{aligned} f(p) = \sum _{i=1}^{K} w_{i}s_i(p) \end{aligned}$$

(7)

where the weights sum to $1,$ so that $\sum _{i=1}^{K}w_i = 1.$ The weights, $w_k,$ are computed from $K$ independent variables, denoted by $\{W_1,W_2,\ldots ,W_K\},$ as follows:

$$\begin{aligned} w_k = \frac{W_k}{\sum _{i=1}^{K} W_i} \end{aligned}$$

(8)

Since subtraction in $log$ space is the same as division, the object-to-distractor ratio (OD ratio) is computed as:

$$\begin{aligned} g = f(p_0) - f(p_1) \end{aligned}$$

(9)

where $p_0$ is the object position and $p_1$ is the position of the strongest distractor. The goal is to change the independent $W_k$ variables in order to maximise the OD ratio, $g.$ We can see how the $W_k$ variables affect the weights as follows:

$$\begin{aligned} \frac{\partial w_k}{\partial W_k} = \frac{\left(\sum _{i=1}^{K}W_i\right) - W_k}{\left(\sum _{i=1}^{K}W_i\right)^2} \end{aligned}$$

(10)

And if $a \ne k$

$$\begin{aligned} \frac{\partial w_a}{\partial W_k} = \frac{-W_a}{\left(\sum _{i=1}^{K}W_i\right)^2} \end{aligned}$$

(11)

Since the weights sum to one, they are not independent and influence each other. If we have $a, a \ne k,$ this gives:

$$\begin{aligned} \frac{\partial w_a}{\partial w_k} = \frac{\partial w_a}{\partial W_k}\frac{\partial W_k}{\partial w_k} = \frac{-W_a}{\left(\sum _{i=1}^{K}W_i\right) - W_k} \end{aligned}$$

(12)

Using this result, the partial derivative of the fused score with respect to the weights is given by

$$\begin{aligned} \frac{\partial f(p)}{\partial w_k} = \sum _{i=1,i \ne k}^{K}\frac{-W_i}{\left(\sum _{j=1}^{K}W_j\right) - W_k}s_i(p) + s_k(p) \end{aligned}$$

(13)

Now the partial derivative of the fused score with respect to the independent $W_k$ terms is given by:

$$\begin{aligned} \frac{\partial f(p)}{\partial W_k} = \frac{\partial f}{\partial w_k}\frac{\partial w_k}{\partial W_k} = \end{aligned}$$

(14)

$$\begin{aligned} \left[\sum _{i=1,i \ne k}^{K}\frac{-W_i}{\left(\sum _{j=1}^{K}W_j\right) - W_k}s_i(p) + s_k(p)\right]\left[\frac{\left(\sum _{i=1}^{K}W_i\right) - W_k}{\left(\sum _{i=1}^{K}W_i\right)^2}\right] \end{aligned}$$

(15)

$$\begin{aligned} =\sum _{i=1,i \ne k}^{K}\frac{-W_i}{\left(\sum _{j=1}^{K}W_j\right)^2}s_i(p) + s_k(p)\left(\frac{\sum _{j=1}^{K}W_j - W_k}{\left(\sum _{j=1}^{K}W_j\right)^2}\right) \end{aligned}$$

(16)

If we now define

$$\begin{aligned} S_k = s_k(p_0) - s_k(p_1) \end{aligned}$$

(17)

then we can write $\partial g / \partial W_k$ as

$$\begin{aligned} \frac{\partial g}{\partial W_k} = \sum _{i=1,i \ne k}^{K}\frac{-W_i}{\left(\sum _{j=1}^{K}W_j\right)^2}S_i + \left(\frac{\sum _{j=1}^{K}W_j - W_k}{\left(\sum _{j=1}^{K}W_j\right)^2}\right)S_k \end{aligned}$$

(18)

This represents the effect each of the independent $W_k$ variables have on the OD ratio, $g.$ The gradient vector is then simply

$$\begin{aligned} V = \left[\frac{\partial g}{\partial W_1}, \frac{\partial g}{\partial W_2}, \ldots , \frac{\partial g}{\partial W_K} \right] \end{aligned}$$

(19)

This is used in a gradient ascent procedure, recomputing the distractor position, $p_1,$ at each step by choosing the position of the background sample with the largest score using the current weights.