Adaptive Correlation Filters with Long-Term and Short-Term Memory for Object Tracking

Abstract

Object tracking is challenging as target objects often undergo drastic appearance changes over time. Recently, adaptive correlation filters have been successfully applied to object tracking. However, tracking algorithms relying on highly adaptive correlation filters are prone to drift due to noisy updates. Moreover, as these algorithms do not maintain long-term memory of target appearance, they cannot recover from tracking failures caused by heavy occlusion or target disappearance in the camera view. In this paper, we propose to learn multiple adaptive correlation filters with both long-term and short-term memory of target appearance for robust object tracking. First, we learn a kernelized correlation filter with an aggressive learning rate for locating target objects precisely. We take into account the appropriate size of surrounding context and the feature representations. Second, we learn a correlation filter over a feature pyramid centered at the estimated target position for predicting scale changes. Third, we learn a complementary correlation filter with a conservative learning rate to maintain long-term memory of target appearance. We use the output responses of this long-term filter to determine if tracking failure occurs. In the case of tracking failures, we apply an incrementally learned detector to recover the target position in a sliding window fashion. Extensive experimental results on large-scale benchmark datasets demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods in terms of efficiency, accuracy, and robustness.

Appendix

In this appendix, we present two additional ablation studies on the OTB2013 dataset. First, we show the results of directly minimizing the errors over all the tracked results to update the correlation filters. Second, we analyze the robustness of the proposed method by spatially shifting the ground truth bounding boxes.

By directly minimizing the errors over all the tracked results, we consider all the extracted appearances $\{x_j,j=1,2,\ldots ,p\}$ of the target object from the first frame up to the current frame p. The cost function is the weighted average quadratic error over these p frames. We assign each frame j with a weight $\beta _j \ge 0$ and learn correlation filter w by minimizing the following objective function:

Fig. 21

Performance of different update schemes on the OTB2013 dataset (Wu et al. 2013a) under one pass evaluation (OPE). Considering all the tracked results (ours-all-update) to update the translation filter does not improve tracking performance. The legend of precision plots shows the distance precision scores at 20 pixels. The legend of success plots contains the overlap success scores with the area under the curve (AUC)

Fig. 22

Spatial shifts. The amount of shift is 10% of width or height of the ground-truth bounding box

$$\begin{aligned} \min _w\sum _{j=1}^p\beta _j\left( \sum _{m,n}\left| \left\langle \phi \left( x_{m,n}^j\right) , w^j \right\rangle -y^j(m,n) \right| ^2 +\lambda \left\langle w^j,w^j\right\rangle \right) , \end{aligned}$$

(23)

where $w^j=\sum _{k,l}a(k,l)\phi (x_{k,l}^j)$. We have the solution to (23) in the Fourier domain as:

$$\begin{aligned} \mathcal {A}^p=\frac{\sum _{j=1}^p\beta _j\mathcal {K}_x^j\odot \overline{\mathcal {Y}}}{\sum _{j=1}^p\beta _j\mathcal {K}_x^j\odot \left( \mathcal {K}_x^j +\lambda \right) }, \end{aligned}$$

(24)

where $\mathcal {K}_x^j=\mathscr {F}\left\{ k_x^j\right\}$ and $k_x^j(m,n)=k(x_{m,n}^j,x^j)$. We perform a grid search and set the weight $\beta _j=0.01$ and the update rate $\lambda =10^{-4}$ for the best accuracy. We restore the parameter $\{\mathcal {K}_x^j\}$, $j=1,2,\ldots ,p-1$, to update the correlation filter in frame j.

Note that such an update scheme is not applicable in practice as it requires a linearly increasing computation and memory storage over the increase of frame number p. The average tracking speed is 2.5 frames per second (fps) versus 20.8 fps (ours) on the OTB2013 dataset. However, Fig. 21 shows that this update scheme does not improve performance. The average distance precision is 83.5 versus 84.8% (ours), and the average overlap success is 62.0 versus 62.8% (ours).

Fig. 23

Tracking performance with spatially shifted ground truth bounding boxes on the OTB2013 dataset (Wu et al. 2013a) under one pass evaluation (OPE)

We spatially shift the ground truth bounding boxes with eight directions (Fig. 22) and rescale the ground truth bounding boxes with scaling factors 0.8, 0.9, 1.1 and 1.2. Figure 23 shows that slightly enlarge the ground truth bounding boxes (with scaling factor 1.1) does not significantly affect the tracking performance.