Discriminative Correlation Filter Tracker with Channel and Spatial Reliability

Abstract

Short-term tracking is an open and challenging problem for which discriminative correlation filters (DCF) have shown excellent performance. We introduce the channel and spatial reliability concepts to DCF tracking and provide a learning algorithm for its efficient and seamless integration in the filter update and the tracking process. The spatial reliability map adjusts the filter support to the part of the object suitable for tracking. This both allows to enlarge the search region and improves tracking of non-rectangular objects. Reliability scores reflect channel-wise quality of the learned filters and are used as feature weighting coefficients in localization. Experimentally, with only two simple standard feature sets, HoGs and colornames, the novel CSR-DCF method—DCF with channel and spatial reliability—achieves state-of-the-art results on VOT 2016, VOT 2015 and OTB100. The CSR-DCF runs close to real-time on a CPU.

Appendix 1: Derivation of the Augmented Lagrangian Minimizer

This section provides a complete derivation of the relations (12, 13) in the Sect. 3.1. The augmented Lagrangian from Eq. (8) is

$$\begin{aligned} \mathcal {L}\left( \hat{\mathbf {h}}_c, \mathbf {h}, \hat{\mathbf {l}}\right)= & {} \left\| \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \right\| ^2 + \frac{\lambda }{2} \Vert \mathbf {h}_m \Vert ^2 \nonumber \\&+\,\left[ \hat{\mathbf {l}}^H\big ( \hat{\mathbf {h}}_c - \hat{\mathbf {h}}_m\big ) + \overline{\hat{\mathbf {l}}^H\big ( \hat{\mathbf {h}}_c -\hat{\mathbf {h}}_m\big )}\right] \nonumber \\&+\, \mu \big \Vert \hat{\mathbf {h}}_c - \hat{\mathbf {h}}_m \big \Vert ^2, \end{aligned}$$

(20)

with $\mathbf {h}_m = (\mathbf {m} \odot \mathbf {h})$. For the purposes of derivation we will rewrite (20) into a fully vectorized form

$$\begin{aligned} \mathcal {L}(\hat{\mathbf {h}}_c, \mathbf {h}, \hat{\mathbf {l}})= & {} \left\| \mathrm {diag}\left( \hat{\mathbf {f}}\right) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \right\| ^2 + \frac{\lambda }{2} \left\| \mathbf {h}_m \right\| ^2 \nonumber \\&+\, \left[ \hat{\mathbf {l}}^H\bigg ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h}\bigg ) + \overline{\hat{\mathbf {l}}^H( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h})}\right] \nonumber \\&+\, \mu \bigg \Vert \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \bigg \Vert ^2, \end{aligned}$$

(21)

where $\mathbf {F}$ denotes $D\times D$ orthonormal matrix of Fourier coefficients, such that the Fourier transform is defined as $\hat{\mathbf {x}} = \mathcal {F}(\mathbf {x}) = \sqrt{D}\mathbf {F}\mathbf {x}$ and $\mathbf {M}=\mathrm {diag}(\mathbf {m})$. For clearer representation we denote the four terms in the summation (21) as

$$\begin{aligned} \mathcal {L}\bigg (\hat{\mathbf {h}}_c, \mathbf {h}, \hat{\mathbf {l}}\bigg ) = \mathcal {L}_{1} + \mathcal {L}_{2} + \mathcal {L}_{3} + \mathcal {L}_{4}, \end{aligned}$$

(22)

where

$$\begin{aligned} \mathcal {L}_1 = \overline{\Big ( \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \Big )}^{T} \Big ( \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \Big ), \end{aligned}$$

(23)

$$\begin{aligned} \mathcal {L}_2 = \frac{\lambda }{2} \Vert \mathbf {h}_m \Vert ^2, \end{aligned}$$

(24)

$$\begin{aligned} \mathcal {L}_3 = \hat{\mathbf {l}}^H( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h}) + \overline{\hat{\mathbf {l}}^H( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h})}, \end{aligned}$$

(25)

$$\begin{aligned} \mathcal {L}_4 = \mu \Vert \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Vert ^2. \end{aligned}$$

(26)

Minimization of Eq. (8) in Sect. 3.1 is an iterative process at which the following minimizations are required:

$$\begin{aligned} \hat{\mathbf {h}}_c^\mathrm {opt} = \mathop {\arg \min }\limits _\mathbf {h_c} \mathcal {L} \bigg (\hat{\mathbf {h}}_c, \mathbf {h}, \hat{\mathbf {l}}\bigg ),\end{aligned}$$

(27)

$$\begin{aligned} \mathbf {h}^\mathrm {opt} = \mathop {\arg \min }\limits _\mathbf {h} \mathcal {L} \bigg (\hat{\mathbf {h}}_c^\mathrm {opt}, \mathbf {h}, \hat{\mathbf {l}}\bigg ). \end{aligned}$$

(28)

Minimization w.r.t. to $\hat{\mathbf {h}}_c$ is derived by finding $\hat{\mathbf {h}}_c$ at which the complex gradient of the augmented Lagrangian vanishes, i.e.,

$$\begin{aligned}&\nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L} \equiv 0, \end{aligned}$$

(29)

$$\begin{aligned}&\nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{1} + \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{2} + \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{3} + \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{4} \equiv 0. \end{aligned}$$

(30)

The partial complex gradients are:

$$\begin{aligned} \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{1}= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \overline{\Big ( \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \Big )}^{T} \Big ( \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {g}} \Big ) \bigg ] \nonumber \\= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \hat{\mathbf {h}}_c^T \mathrm {diag}(\hat{\mathbf {f}})^H \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {h}}_{c}^{T} \mathrm {diag}(\hat{\mathbf {f}})^{H} \hat{\mathbf {g}} \nonumber \\&-\, \hat{\mathbf {g}}^{H} \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {h}}}_c + \hat{\mathbf {g}}^{H}\hat{\mathbf {g}} \bigg ] \nonumber \\= & {} \mathrm {diag}(\hat{\mathbf {f}})^{H} \mathrm {diag}(\hat{\mathbf {f}}) \hat{\mathbf {h}}_c - \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {g}}}, \end{aligned}$$

(31)

$$\begin{aligned} \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{2}= & {} 0,\end{aligned}$$

(32)

$$\begin{aligned} \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{3}= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \hat{\mathbf {l}}^H \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big ) + \overline{\hat{\mathbf {l}}^H \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big )} \bigg ] \nonumber \\= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \hat{\mathbf {l}}^H \hat{\mathbf {h}}_c - \hat{\mathbf {l}}^H \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} + \hat{\mathbf {l}}^T \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {l}}^T \sqrt{D}\overline{\mathbf {F}}\overline{\mathbf {M}}\overline{\mathbf {h}} \bigg ] \nonumber \\= & {} \hat{\mathbf {l}}, \end{aligned}$$

(33)

$$\begin{aligned} \nabla _{\overline{\hat{\mathbf {h}}}_c} \mathcal {L}_{4}= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \mu \overline{\Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big )}^{T} \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big ) \bigg ] \nonumber \\= & {} \frac{\partial }{\partial \overline{\hat{\mathbf {h}}}_c} \bigg [ \mu \Big ( \hat{\mathbf {h}}_c^{H} \hat{\mathbf {h}}_c - \hat{\mathbf {h}}_c^{H}\sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \nonumber \\&-\, \sqrt{D}\mathbf {h}^T\mathbf {M}\mathbf {F}^{H}\hat{\mathbf {h}}_c + D\mathbf {h}^T\mathbf {M}\mathbf {F}^{H}\mathbf {F}\mathbf {M}\mathbf {h} \Big ) \bigg ] \nonumber \\= & {} \mu \hat{\mathbf {h}}_c - \mu \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h}. \end{aligned}$$

(34)

Note that $\sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} = \hat{\mathbf {h}}_m$ according to our original definition of $\hat{\mathbf {h}}_m$. Plugging (31–34) into (30) yields

$$\begin{aligned}&\mathrm {diag}(\hat{\mathbf {f}})^{H} \mathrm {diag}(\hat{\mathbf {f}}) \hat{\mathbf {h}}_c - \mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {g}}} + \hat{\mathbf {l}} + \mu \hat{\mathbf {h}}_c - \mu \hat{\mathbf {h}}_m = 0,\\&\hat{\mathbf {h}}_c = \frac{\mathrm {diag}(\hat{\mathbf {f}}) \overline{\hat{\mathbf {g}}} + \mu \hat{\mathbf {h}}_m - \hat{\mathbf {l}}}{\mathrm {diag}(\hat{\mathbf {f}})^{H} \mathrm {diag}(\hat{\mathbf {f}}) + \mu },\nonumber \end{aligned}$$

(35)

which can be rewritten into

$$\begin{aligned} \hat{\mathbf {h}}_c = \frac{\hat{\mathbf {f}} \odot \overline{\hat{\mathbf {g}}} + \mu \hat{\mathbf {h}}_m - \hat{\mathbf {l}}}{\overline{\hat{\mathbf {f}}} \odot \hat{\mathbf {f}} + \mu }. \end{aligned}$$

(36)

Next we derive the closed-form solution of (28). The optimal $\mathbf {h}$ is obtained when the complex gradient w.r.t. $\mathbf {h}$ vanishes, i.e.,

$$\begin{aligned}&\nabla _{\overline{\mathbf {h}}} \mathcal {L} \equiv 0 \end{aligned}$$

(37)

$$\begin{aligned}&\nabla _{\overline{\mathbf {h}}} \mathcal {L}_{1} + \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{2} + \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{3} + \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{4} \equiv 0. \end{aligned}$$

(38)

The partial gradients are

$$\begin{aligned} \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{1}= & {} 0, \end{aligned}$$

(39)

$$\begin{aligned} \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{2}= & {} \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \frac{\lambda }{2} \overline{(\mathbf {M} \mathbf {h})}^T (\mathbf {M} \mathbf {h}) \bigg ] = \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \frac{\lambda }{2} \mathbf {h}^H \overline{\mathbf {M}} \mathbf {M} \mathbf {h} \bigg ]. \end{aligned}$$

(40)

Since we defined mask $\mathbf {m}$ as a binary mask, the product $\overline{\mathbf {M}} \mathbf {M}$ can be simplified into $\mathbf {M}$ and the result for $\nabla _{\overline{\mathbf {h}}} \mathcal {L}_{2}$ is

$$\begin{aligned} \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{2} = \frac{\lambda }{2} \mathbf {M} \mathbf {h}. \end{aligned}$$

(41)

The remaining gradients are as follows:

$$\begin{aligned} \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{3}= & {} \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \hat{\mathbf {l}}^H \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big ) + \overline{\hat{\mathbf {l}}^H \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big )} \bigg ] \nonumber \\= & {} \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \hat{\mathbf {l}}^H \hat{\mathbf {h}}_c - \hat{\mathbf {l}}^H \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} + \hat{\mathbf {l}}^T \overline{\hat{\mathbf {h}}}_c - \hat{\mathbf {l}}^T \sqrt{D}\overline{\mathbf {F}}\overline{\mathbf {M}}\overline{\mathbf {h}} \bigg ] \nonumber \\= & {} - \sqrt{D} \mathbf {M} \mathbf {F}^H \hat{\mathbf {l}}, \end{aligned}$$

(42)

$$\begin{aligned} \nabla _{\overline{\mathbf {h}}} \mathcal {L}_{4}= & {} \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \mu \overline{\Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big )}^{T} \Big ( \hat{\mathbf {h}}_c - \sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \Big ) \bigg ] \nonumber \\= & {} \frac{\partial }{\partial \overline{\mathbf {h}}} \bigg [ \mu \Big ( \hat{\mathbf {h}}_c^H \hat{\mathbf {h}}_c - \hat{\mathbf {h}}_c^H\sqrt{D}\mathbf {F}\mathbf {M}\mathbf {h} \nonumber \\&- \, \sqrt{D}\mathbf {h}^H\mathbf {M}\mathbf {F}^H\hat{\mathbf {h}}_c + D\mathbf {h}^H \mathbf {M}\mathbf {h} \Big ) \bigg ] \nonumber \\= & {} - \mu \sqrt{D}\mathbf {M}\mathbf {F}^H \hat{\mathbf {h}}_c + \mu D\mathbf {M}\mathbf {h}. \end{aligned}$$

(43)

Plugging (39–43) into (38) yields

$$\begin{aligned}&\frac{\lambda }{2} \mathbf {M} \mathbf {h} - \sqrt{D} \mathbf {M} \mathbf {F}^H \hat{\mathbf {l}} - \mu \sqrt{D}\mathbf {M}\mathbf {F}^H \hat{\mathbf {h}}_c + \mu D\mathbf {M}\mathbf {h} = 0, \nonumber \\&\mathbf {M}\mathbf {h} = \mathbf {M}\frac{\sqrt{D}\mathbf {F}^H(\hat{\mathbf {l}} + \mu \hat{\mathbf {h}}_c)}{\frac{\lambda }{2} + \mu D}. \end{aligned}$$

(44)

Using the definition of the inverse Fourier transform, i.e., $\mathcal {F}^{-1}(\hat{\mathbf {x}}) = \frac{1}{\sqrt{D}} \mathbf {F}^H \hat{\mathbf {x}}$, (44) can be rewritten into

$$\begin{aligned} \mathbf {m} \odot \mathbf {h} = \mathbf {m} \odot \frac{\mathcal {F}^{-1}(\hat{\mathbf {l}} + \mu \hat{\mathbf {h}}_c)}{\frac{\lambda }{2D} + \mu }. \end{aligned}$$

(45)

The values in $\mathbf {m}$ are either zero or one. Elements in $\mathbf {h}$ that correspond to the zeros in $\mathbf {m}$ can in principle not be recovered from (45) since this would result in division by zero. But our initial definition of the problem was to seek solutions for the filter that satisfies the following relation $\mathbf {h} \equiv \mathbf {h} \odot \mathbf {m}$. This means the values corresponding to zeros in $\mathbf {m}$ should be zero in $\mathbf {h}$. Thus the proximal solution to (45) is

$$\begin{aligned} \mathbf {h} = \mathbf {m} \odot \frac{\mathcal {F}^{-1}(\hat{\mathbf {l}} + \mu \hat{\mathbf {h}}_c)}{\frac{\lambda }{2D} + \mu }. \end{aligned}$$

(46)