International Journal of Computer Vision

, Volume 91, Issue 3, pp 303–327 | Cite as

Incremental Tensor Subspace Learning and Its Applications to Foreground Segmentation and Tracking

  • Weiming Hu
  • Xi Li
  • Xiaoqin Zhang
  • Xinchu Shi
  • Stephen Maybank
  • Zhongfei Zhang
Open Access
Article

Abstract

Appearance modeling is very important for background modeling and object tracking. Subspace learning-based algorithms have been used to model the appearances of objects or scenes. Current vector subspace-based algorithms cannot effectively represent spatial correlations between pixel values. Current tensor subspace-based algorithms construct an offline representation of image ensembles, and current online tensor subspace learning algorithms cannot be applied to background modeling and object tracking. In this paper, we propose an online tensor subspace learning algorithm which models appearance changes by incrementally learning a tensor subspace representation through adaptively updating the sample mean and an eigenbasis for each unfolding matrix of the tensor. The proposed incremental tensor subspace learning algorithm is applied to foreground segmentation and object tracking for grayscale and color image sequences. The new background models capture the intrinsic spatiotemporal characteristics of scenes. The new tracking algorithm captures the appearance characteristics of an object during tracking and uses a particle filter to estimate the optimal object state. Experimental evaluations against state-of-the-art algorithms demonstrate the promise and effectiveness of the proposed incremental tensor subspace learning algorithm, and its applications to foreground segmentation and object tracking.

Keywords

Incremental learning Tensor subspace Foreground segmentation Tracking 

Supplementary material

Results of the DTA-based algorithm for Example 1 (4.717 kb)

Results of the DTA-based algorithm for Example 2 (1.366 kb)

Results of the DTA-based algorithm for Example 3 (3.251 kb)

11263_2010_399_MOESM4_ESM.avi (4.1 mb)
Results of the DTA-based algorithm for Example 4 (4.246 kb)

Results of the DTA-based algorithm for Example 5 (2.661 kb)

Results of the Riemannian metric-based algorithm for Example 2 (3.568 kb)

Results of the Riemannian metric-based algorithm for Example 3 (3.768 kb)

11263_2010_399_MOESM8_ESM.avi (3.8 mb)
Results of the Riemannian metric-based algorithm for Example 4 (3.916 kb)

Results of the Riemannian metric-based algorithm for Example 5 (3.013 kb)

References

  1. Black, M. J., & Jepson, A. D. (1998). EigenTracking: robust matching and tracking of articulated objects using a view-based representation. International Journal of Computer Vision, 26(1), 63–84. CrossRefGoogle Scholar
  2. Black, M. J., Fleet, D. J., & Yacoob, Y. (1998). A framework for modeling appearance change in image sequence. In Proc. of IEEE international conference on computer vision (pp. 660–667), Jan. 1998. Google Scholar
  3. Chen, D., & Yang, J. (2007). Robust object tracking via online dynamic spatial bias appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(12), 2157–2169. CrossRefGoogle Scholar
  4. Elgammal, A., Duraiswami, R., Harwood, M., & Davis, L. S. (2002). Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proceedings of the IEEE, 99(7), 1151–1163. CrossRefGoogle Scholar
  5. Gall, J., Rosenhahn, B., & Seidel, H.-P. (2008). Drift-free tracking of rigid and articulated objects. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 1–8), June 2008. Google Scholar
  6. Golub, G. H., & Van Loan, C. F. (1996). Matrix computations. Baltimore: Johns Hopkins University Press. MATHGoogle Scholar
  7. Grabner, H., Leistner, C., & Bischof, H. (2008). Semi-supervised on-line boosting for robust tracking. In Proc. of European conference on computer vision (pp. 234–247). Google Scholar
  8. Gu, M., & Eisenstat, S. C. (1993). A stable and fast algorithm for updating the singular value decomposition. Research report YALEU/DCS/RR-966, Department of Computer Science, Yale University, New Haven, June 1993. Google Scholar
  9. Gu, M., & Eisenstat, S. C. (1995). Downdating the singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 16(3), 793–810. MATHCrossRefMathSciNetGoogle Scholar
  10. Hager, G. D., & Belhumeur, P. N. (1996). Real-time tracking of image regions with changes in geometry and illumination. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 403–410), June 1996. Google Scholar
  11. Han, B., Zhu, Y., Comaniciu, D., & Davis, L. S. (2009). Visual tracking by continuous density propagation in sequential bayesian filtering framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 919–930. CrossRefGoogle Scholar
  12. Haritaoglu, I., Harwood, D., & Davis, L. S. (2000). W4: real-time surveillance of people and their activities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 809–830. CrossRefGoogle Scholar
  13. He, X., Cai, D., & Niyogi, P. (2005). Tensor subspace analysis. In Proc. of annual conference on neural information processing systems Dec. 2005 Cambridge: MIT Press. Google Scholar
  14. Ho, J., Lee, K., Yang, M., & Kriegman, D. (2004). Visual tracking using learned linear subspaces. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 782–789). Google Scholar
  15. Isard, M., & Blake, A. (1996). Contour tracking by stochastic propagation of conditional density. In Proc. of European conference on computer vision (vol. 2, pp. 343–356). Google Scholar
  16. Jacques, J. C. S. Jr., Jung, C. R., & Musse, S. R. (2006). A background subtraction model adapted to illumination changes. In Proc. of IEEE international conference on image processing (pp. 1817–1820), Oct. 2006. Google Scholar
  17. Jepson, A. D., Fleet, D. J., & El-Maraghi, T. F. (2003). Robust online appearance models for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10), 1296–1311. CrossRefGoogle Scholar
  18. Kwon, J., & Lee, K. M. (2009). Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping Monte Carlo sampling. In Proc. of IEEE conference on computer vision and pattern recognition workshops (pp. 1208–1215), June 2009. Google Scholar
  19. Lathauwer, L. D., Moor, B. D., & Vandewalle, J. (2000). On the best Rank-1 and Rank-(R1,R2,…,Rn) approximation of higher-order tensors. SIAM Journal of Matrix Analysis and Applications, 21(4), 1324–1342. MATHCrossRefGoogle Scholar
  20. Lee, K., & Kriegman, D. (2005). Online learning of probabilistic appearance manifolds for video-based recognition and tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 852–859). Google Scholar
  21. Levy, A., & Lindenbaum, M. (2000). Sequential Karhunen-Loeve basis extraction and its application to images. IEEE Transactions on Image Processing, 9, 1371–1374. MATHCrossRefGoogle Scholar
  22. Li, Y. (2004). On incremental and robust subspace learning. Pattern Recognition, 37(7), 1509–1518. MATHCrossRefGoogle Scholar
  23. Li, J., Zhou, S. K., & Chellappa, R. (2005). Appearance modeling under geometric context. In Proc. of IEEE international conference on computer vision (vol. 2, pp. 1252–1259). Google Scholar
  24. Li, X., Hu, W. M., Zhang, Z. F., Zhang, X. Q., & Luo, G. (2007). Robust visual tracking based on incremental tensor subspace learning. In Proc. of IEEE international conference on computer vision (pp. 1–8), Oct. 2007. Google Scholar
  25. Lim, H., Morariu, V. I., Camps, O. I., & Sznaier, M. (2006). Dynamic appearance modeling for human tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 751–757). Google Scholar
  26. Mahadevan, V., & Vasconcelos, N. (2009). Saliency-based discriminant tracking. In Proc. of IEEE conference on computer vision and pattern recognition workshops (pp. 1007–1013), June 2009. Google Scholar
  27. Matthews, I., Ishikawa, T., & Baker, S. (2004). The template update problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(4), 810–815. CrossRefGoogle Scholar
  28. Nickel, K., & Stiefelhagen, R. (2008). Dynamic integration of generalized cues for person tracking. In Proc. of European conference on computer vision, Part IV. Lecture notes in computer science (vol. 5305, pp. 514–526), Oct. 2008. Google Scholar
  29. Nummiaroa, K., Koller-Meierb, E., & Gool, I. V. (2003). An adaptive color-based particle filter. Image and Vision Computing, 21(1), 99–110. CrossRefGoogle Scholar
  30. Papadimitriou, S., Sun, J., & Faloutsos, C. (2005). Streaming pattern discovery in multiple timeseries. In Proc. of international conference on very large data bases (pp. 697–708). Google Scholar
  31. Patwardhan, K., Morellas, V., & Sapiro, G. (2008). Robust foreground detection in video using pixel layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4), 746–751. CrossRefGoogle Scholar
  32. Perez, P., Hue, C., Vermaak, J., & Gangnet, M. (2002). Color-based probabilistic tracking. In Proc. of European conference on computer vision, Part I. Lecture notes in computer science (vol. 2350, pp. 661–675). Google Scholar
  33. Porikli, F., Tuzel, O., & Meer, P. (2006). Covariance tracking using model update based on Lie algebra. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 728–735). Google Scholar
  34. Ramanan, D., Forsyth, D. A., & Zisserman, A. (2007). Tracking people by learning their appearance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(1), 65–81. CrossRefGoogle Scholar
  35. Ross, D. A., Lim, J., Lin, R.-S., & Yang, M.-H. (2008). Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(2), 125–141. CrossRefGoogle Scholar
  36. Sheikh, Y., & Shah, M. (2005). Bayesian object detection in dynamic scenes. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 74–79). Google Scholar
  37. Skocaj, D., & Leonardis, A. (2003). Weighted and robust incremental method for subspace learning. In Proc. of IEEE international conference on computer vision (vol. 2, pp. 1494–1501), Oct. 2003. Google Scholar
  38. Stauffer, C., & Grimson, W. E. L. (1999). Adaptive background mixture models for real-time tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 246–252). Google Scholar
  39. Sun, J., Tao, D., & Faloutsos, C. (2006a). Beyond streams and graphs: dynamic tensor analysis. In Proc. of ACM SIGKDD international conference on knowledge discovery and data mining (pp. 374–383), Aug. 2006. Google Scholar
  40. Sun, J., Papadimitriou, S., & Yu, P. S. (2006b). Window-based tensor analysis on high-dimensional and multi-aspect streams. In Proc. of international conference on data mining (pp. 1076–1080), Dec. 2006. Google Scholar
  41. Sun, J., Tao, D., Papadimitriou, S., Yu, P. S., & Faloutsos, C. (2008). Incremental tensor analysis: theory and applications. ACM Transactions on Knowledge Discovery from Data, 2(3), 1–37. CrossRefGoogle Scholar
  42. Tian, Y., Lu, M., & Hampapur, A. (2005). Robust and efficient foreground analysis for real-time video surveillance. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 1182–1187). Google Scholar
  43. Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear subspace analysis of image ensembles: TensorFaces. In Proc. of European conference on computer vision (pp. 447–460), May 2002. Google Scholar
  44. Vasilescu, M. A. O., & Terzopoulos, D. (2003). Multilinear subspace analysis of image ensembles. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 93–99), June 2003. Google Scholar
  45. Wang, H., & Ahuja, N. (2005). Rank-R approximation of tensors using image-as-matrix representation. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 346–353). Google Scholar
  46. Wang, H., & Ahuja, N. (2008). A tensor approximation approach to dimensionality reduction. International Journal of Computer Vision, 76(3), 217–229. CrossRefGoogle Scholar
  47. Wang, Y., Tan, T., Loe, K. F., & Wu, J. K. (2005). A probabilistic approach for foreground and shadow segmentation in monocular image sequences. Pattern Recognition, 38(11), 1937–1946. CrossRefGoogle Scholar
  48. Wang, Y., Loe, K., & Wu, J. (2006). A dynamic conditional random field model for foreground and shadow segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(2), 279–289. CrossRefGoogle Scholar
  49. Wang, H., Yan, S., Huang, T., & Tang, X. (2007). A convergent solution to tensor subspace learning. In Proc. of international joint conference on artificial intelligence (pp. 629–634). Google Scholar
  50. Wong, S., Wong, K. K., & Cipolla, R. (2006). Robust appearance-based tracking using a sparse bayesian classifier. In Proc. of international conference on pattern recognition (vol. 3, pp. 47–50). Google Scholar
  51. Wu, Y., & Huang, T. S. (2004). Robust visual tracking by integrating multiple cues based on co-inference learning. International Journal of Computer Vision, 58(1), 55–71. CrossRefGoogle Scholar
  52. Yan, S., Xu, D., Yang, Q., Zhang, L., Tang, X., & Zhang, H. (2005). Discriminant analysis with tensor representation. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 526–532), June 2005. Google Scholar
  53. Yan, S., Shan, S., Chen, X., Gao, W., & Chen, J. (2007). Matrix-structural learning (MSL) of cascaded classifier from enormous training set. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 1–7), June 2007. Google Scholar
  54. Yang, J., Zhang, D., Frangi, A. F., & Yang, J. (2004). Two-dimensional PCA a new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), 131–137. CrossRefGoogle Scholar
  55. Yang, C., Duraiswami, R., & Davis, L. S. (2005). Efficient mean-shift tracking via a new similarity measure. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 176–183), June 2005. Google Scholar
  56. Yang, M., Fan, Z., Fan, J., & Wu, Y. (2009). Tracking nonstationary visual appearances by data-driven adaptation. IEEE Transactions on Image Processing, 18(7), 1633–1644. CrossRefGoogle Scholar
  57. Ye, J. (2005). Generalized low rank approximations of matrices. Machine Learning, 61(1–3), 167–191. MATHCrossRefGoogle Scholar
  58. Ye, J., Janardan, R., & Li, Q. (2004a). Two-dimensional linear discriminant nalysis. In Proc. of neural information processing systems conference (pp. 1569–1576). Cambridge: MIT Press. Google Scholar
  59. Ye, J., Janardan, R., & Li, Q. (2004b). GPCA: an efficient dimension reduction scheme for image compression and retrieval. In Proc. of ACM SIGKDD international conference on knowledge discovery and data mining (pp. 354–363), Aug. 2004. Google Scholar
  60. Yu, T., & Wu, Y. (2006). Differential tracking based on spatial-appearance model (SAM). In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 720–727), June 2006. Google Scholar
  61. Zhou, S. K., Chellappa, R., & Moghaddam, B. (2004). Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Transactions on Image Processing, 13(11), 1491–1506. CrossRefGoogle Scholar
  62. Zhou, X., Hu, W. M., Chen, Y., & Hu, W. (2007). Markov random field modeled level sets method for object tracking with moving cameras. In Proc. of Asian conference on computer vision, Part I (pp. 832–842). Google Scholar

Copyright information

© The Author(s) 2010

Authors and Affiliations

  • Weiming Hu
    • 1
  • Xi Li
    • 1
  • Xiaoqin Zhang
    • 2
  • Xinchu Shi
    • 1
  • Stephen Maybank
    • 3
  • Zhongfei Zhang
    • 4
  1. 1.National Laboratory of Pattern Recognition, Institute of AutomationChinese Academy of SciencesBeijingChina
  2. 2.College of Mathematics & Information ScienceWenzhou UniversityWenzhouChina
  3. 3.Department of Computer Science and Information SystemsBirkbeck CollegeLondonUK
  4. 4.State University of New YorkBinghamtonUSA

Personalised recommendations