Fast Covariance Computation and Dimensionality Reduction for Sub-window Features in Images

  • Vivek Kwatra
  • Mei Han
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6312)


This paper presents algorithms for efficiently computing the covariance matrix for features that form sub-windows in a large multi-dimensional image. For example, several image processing applications, e.g. texture analysis/synthesis, image retrieval, and compression, operate upon patches within an image. These patches are usually projected onto a low-dimensional feature space using dimensionality reduction techniques such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), which in-turn requires computation of the covariance matrix from a set of features. Covariance computation is usually the bottleneck during PCA or LDA (O(nd 2) where n is the number of pixels in the image and d is the dimensionality of the vector). Our approach reduces the complexity of covariance computation by exploiting the redundancy between feature vectors corresponding to overlapping patches. Specifically, we show that the covariance between two feature components can be reduced to a function of the relative displacement between those components in patch space. One can then employ a lookup table to store covariance values by relative displacement. By operating in the frequency domain, this lookup table can be computed in O(n logn) time. We allow the patches to sub-sample the image, which is useful for hierarchical processing and also enables working with filtered responses over these patches, such as local gist features. We also propose a method for fast projection of sub-window patches onto the low-dimensional space.


Feature Vector Linear Discriminant Analysis Image Patch Texture Synthesis Covariance Computation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adams, A., Gelfand, N., Dolson, J., Levoy, M.: Gaussian kd-trees for fast high-dimensional filtering. ACM Trans. Graph., SIGGRAPH 28(3), 1–12 (2009)CrossRefGoogle Scholar
  2. 2.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 60–65. IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  3. 3.
    Cao, G., Bouman, C.A.: Covariance estimation for high dimensional data vectors using the sparse matrix transform. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) NIPS, pp. 225–232. MIT Press, Cambridge (2008)Google Scholar
  4. 4.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13 (2004)Google Scholar
  5. 5.
    Darlington, R.B., Weinberg, S., Herbert, W.: Canonical variate analysis and related techniques. Review of Educational Research, 453–454 (1973)Google Scholar
  6. 6.
    Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annals of Eugenics 7, 179–188 (1936)Google Scholar
  7. 7.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super resolution. IEEE Comput. Graph. Appl. (2002)Google Scholar
  8. 8.
    Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Computer Science and Scientific Computing Series. Academic Press, London (1990)zbMATHGoogle Scholar
  9. 9.
    Higham, N.J.: Computing the nearest correlation matrix a problem from finance. IMA Journal of Numerical Analysis 22(3), 329–343 (2002)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1), 4–37 (2000)CrossRefGoogle Scholar
  11. 11.
    Jones, J.P., Palmer, L.A.: An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. J Neurophysiol 58(6), 1233–1258 (1987)Google Scholar
  12. 12.
    Ke, Y., Sukthankar, R.: Pca-sift: a more distinctive representation for local image descriptors. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 506–513 (2004)Google Scholar
  13. 13.
    Kim, K.I., Franz, M., Schölkopf, B.: Kernel hebbian algorithm for single-frame super-resolution. In: Leonardis, A., Bischof, H. (eds.) Statistical Learning in Computer Vision, pp. 135–149. Springer, Berlin (2004)Google Scholar
  14. 14.
    Kim, K.I., Franz, M.O., Schölkopf, B.: Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(9), 1351–1366 (2005)CrossRefGoogle Scholar
  15. 15.
    Korah, T., Rasmussen, C.: Pca-based recognition for efficient inpainting. In: IEEE Asian Conference on Computer Vision (2006)Google Scholar
  16. 16.
    Kwatra, V., Essa, I., Bobick, A., Kwatra, N.: Texture optimization for example-based synthesis. ACM Trans. Graph., SIGGRAPH 24(3), 795–802 (2005)CrossRefGoogle Scholar
  17. 17.
    Lefebvre, S., Hoppe, H.: Appearance-space texture synthesis. In: Proc. of SIGGRAPH 2006, pp. 541–548 (2006)Google Scholar
  18. 18.
    Lefebvre, S., Hoppe, H.: Parallel controllable texture synthesis. ACM Transactions on Graphics, SIGGRAPH, 777–786 (2005)Google Scholar
  19. 19.
    Liang, L., Liu, C., Xu, Y., Guo, B., Shum, H.Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001)CrossRefGoogle Scholar
  20. 20.
    Liu, J., Wu, F., Yao, L., Zhuang, Y.: A prediction error compression method with tensor-pca in video coding. In: MACM, pp. 493–500 (2007)Google Scholar
  21. 21.
    Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Progress in Brain Research 155, 23–36 (2006)CrossRefGoogle Scholar
  22. 22.
    Pearson, K.: On lines and planes of closest fit to systems of points in space. Philosophical Magazine 2(6), 559–572 (1901)Google Scholar
  23. 23.
    Porikli, W.F., Tuzel, O.: Fast construction of covariance matrices for arbitrary size image. In: Proc. Intl. Conf. on Image Processing, pp. 1581–1584 (2006)Google Scholar
  24. 24.
    Qi, J., Leahy, R.M.: Fast computation of the covariance of map reconstructions of pet images. Proceedings of SPIE 3661(1), 344–355 (1999)CrossRefGoogle Scholar
  25. 25.
    Stein, C., Efron, B., Morris, C.: Improving the usual estimator of a normal covariance matrix. Dept. of Statistics, Stanford University, Report 37 (1972)Google Scholar
  26. 26.
    Wang, Q., Tang, X., Shum, H.Y.: Patch based blind image super resolution. In: ICCV (2005)Google Scholar
  27. 27.
    Wei, L.Y., Lefebvre, S., Kwatra, V., Turk, G.: State of the art in example-based texture synthesis. In: Eurographics 2009, State of the Art Report, EG-STAR. Eurographics Association (2009)Google Scholar
  28. 28.
    Yu, Y.D., Kang, D.S., Kim, D.: Color image compression based on vector quantization using pca and lebld. In: Proc. of the IEEE Region 10 Conference, vol. 2, pp. 1259–1262 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Vivek Kwatra
    • 1
  • Mei Han
    • 1
  1. 1.Google ResearchMountain View

Personalised recommendations